add user-agent header to template downloader request by sudo87 · Pull Request #12791 · apache/cloudstack

sudo87 · 2026-03-11T12:12:55Z

Description

This PR adds user-agent header to download request.

Some template providers reject HTTP requests without a User-Agent
header and return HTTP 403 instead of redirecting to the actual
download location. This causes template downloads to fail.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sudo87 · 2026-03-11T12:34:17Z

@blueorangutan package

blueorangutan · 2026-03-11T12:36:05Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-03-11T12:41:50Z

Codecov Report

❌ Patch coverage is 60.00000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.61%. Comparing base (7aa0558) to head (0f31e3e).
⚠️ Report is 14 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...torage/template/SimpleHttpMultiFileDownloader.java	0.00%	2 Missing ⚠️
...cloud/utils/net/HttpClientCloudStackUserAgent.java	81.81%	1 Missing and 1 partial ⚠️
...cloud/storage/template/HttpTemplateDownloader.java	0.00%	1 Missing ⚠️
...d/storage/template/MetalinkTemplateDownloader.java	0.00%	1 Missing ⚠️
utils/src/main/java/com/cloud/utils/UriUtils.java	0.00%	1 Missing ⚠️
.../main/java/com/cloud/utils/storage/QCOW2Utils.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #12791      +/-   ##
============================================
- Coverage     17.61%   17.61%   -0.01%     
- Complexity    15664    15676      +12     
============================================
  Files          5917     5918       +1     
  Lines        531402   531557     +155     
  Branches      64971    64986      +15     
============================================
+ Hits          93596    93611      +15     
- Misses       427252   427389     +137     
- Partials      10554    10557       +3

Flag	Coverage Δ
uitests	`3.70% <ø> (-0.01%)`	⬇️
unittests	`18.68% <60.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-03-11T13:53:32Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17096

weizhouapache · 2026-03-11T14:02:41Z

@sudo87
code looks ok

any linked issue ?

core/src/main/java/com/cloud/storage/template/HttpTemplateDownloader.java

sureshanaparti · 2026-03-12T05:16:15Z

@sudo87 code looks ok

any linked issue ?

@weizhouapache it seems no linked issue for this.

sureshanaparti · 2026-03-12T05:34:06Z

@sudo87 check if the user-agent update is relavent in createRequest() at MetalinkTemplateDownloader, SimpleHttpMultiFileDownloader, HttpDirectTemplateDownloader.

sureshanaparti

clgtm

sudo87 · 2026-03-12T10:45:08Z

@sudo87 check if the user-agent update is relavent in createRequest() at MetalinkTemplateDownloader, SimpleHttpMultiFileDownloader, HttpDirectTemplateDownloader.

Thanks @sureshanaparti for your feedback on the pr. I think there is no harm adding user-agent in the GetMethod request, updated the pr accordingly.

sureshanaparti · 2026-03-12T10:49:22Z

@blueorangutan package

blueorangutan · 2026-03-12T10:50:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-12T12:02:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17116

sudo87 · 2026-03-13T05:36:58Z

@blueorangutan test

blueorangutan · 2026-03-13T05:38:04Z

@sudo87 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-03-13T20:24:50Z

[SF] Trillian test result (tid-15638)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50270 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12791-t15638-kvm-ol8.zip
Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestListIdsParams>:teardown	`Error`	1.14	test_list_ids_parameter.py
test_01_snapshot_root_disk	`Error`	3.90	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	46.72	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	46.72	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	`Error`	27.49	test_snapshots.py
test_01_snapshot_usage	`Error`	32.91	test_usage.py
test_01_vpn_usage	`Error`	1.12	test_usage.py

vishesh92

The code looks good. My only concern (not sure about the impact) is the value for the user agent. We should ensure it matches the format as per the RFC. below is the output from quick web search.
Ref:
https://www.rfc-editor.org/rfc/rfc9110.html#section-10.1.5
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent

sudo87 · 2026-03-24T10:45:01Z

Thank you @vishesh92 for review and suggestion. Changed the user agent to follow RFC format, e.g.:

CloudStack-Agent/4.22.1.0-SNAPSHOT (Apache CloudStack)

Please let me know if this makes sense.

sudo87 · 2026-03-24T11:05:28Z

@blueorangutan package

blueorangutan · 2026-03-24T11:06:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-24T12:00:24Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17231

vishesh92

clgtm

blueorangutan · 2026-03-26T06:53:26Z

[LL] Trillian Build Failed (tid-7181)

sureshanaparti · 2026-03-26T09:08:24Z

@blueorangutan package

blueorangutan · 2026-03-26T09:10:08Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

rajujith · 2026-03-26T10:05:45Z

@sudo87

There is a URL validity check done by the management server, which still shows up on the webserver as "Java/17.0.18". The SSVM file download shows "CloudStack-Agent/4.22.1.0-SNAPSHOT (Apache CloudStack)" . Do you think the management server also needs to be fixed?

blueorangutan · 2026-03-26T10:13:16Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17252

sudo87 · 2026-03-26T11:09:07Z

@sudo87

There is a URL validity check done by the management server, which still shows up on the webserver as "Java/17.0.18". The SSVM file download shows "CloudStack-Agent/4.22.1.0-SNAPSHOT (Apache CloudStack)" . Do you think the management server also needs to be fixed?

Thank you @rajujith for suggestion, it would be wise to add user-agent in HEAD request as well. I have updated the pr accordingly.

sudo87 · 2026-03-26T11:09:38Z

@blueorangutan package

blueorangutan · 2026-03-26T11:10:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-26T12:05:27Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17253

sureshanaparti · 2026-03-27T05:17:02Z

@blueorangutan package

blueorangutan · 2026-03-27T05:18:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sudo87 · 2026-03-27T05:23:02Z

@blueorangutan package

blueorangutan · 2026-03-27T05:24:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-27T06:28:46Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17262

blueorangutan · 2026-03-27T06:32:54Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17263

rajujith

LGTM
Tested

10.x.x.x- - [27/Mar/2026:11:28:09 +0000] "HEAD /jithintemplates/alpine-router.qcow2 HTTP/1.1" 200 271 "-" "CloudStack-Agent/4.22.1.0-SNAPSHOT (Apache CloudStack)"
10.x.x.x - - [27/Mar/2026:11:28:09 +0000] "GET /jithintemplates/alpine-router.qcow2 HTTP/1.1" 200 336462039 "-" "CloudStack-Agent/4.22.1.0-SNAPSHOT (Apache CloudStack)"

sureshanaparti · 2026-03-27T11:40:08Z

@blueorangutan test

blueorangutan · 2026-03-27T11:42:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-03-28T02:48:45Z

[SF] Trillian test result (tid-15754)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51344 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12791-t15754-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

add user-agent header to template downloader request

1fe48e1

sudo87 requested a review from sureshanaparti March 11, 2026 12:34

sureshanaparti reviewed Mar 12, 2026

View reviewed changes

core/src/main/java/com/cloud/storage/template/HttpTemplateDownloader.java Outdated Show resolved Hide resolved

handle review comment

c1eb8a5

boring-cyborg bot added the component:api label Mar 12, 2026

sureshanaparti approved these changes Mar 12, 2026

View reviewed changes

DaanHoogland added the ready-for-testing label Mar 16, 2026

vishesh92 reviewed Mar 20, 2026

View reviewed changes

add user agent adhereing to RFC format

6fd80d6

sudo87 force-pushed the addUserAgentTemplateDownload branch from 7a13c88 to 6fd80d6 Compare March 24, 2026 11:05

sureshanaparti added status:needs-testing and removed ready-for-testing labels Mar 24, 2026

sureshanaparti added this to Apache CloudStack 4.22.1 Mar 24, 2026

github-project-automation bot moved this to Todo in Apache CloudStack 4.22.1 Mar 24, 2026

sureshanaparti moved this from Todo to In Progress in Apache CloudStack 4.22.1 Mar 24, 2026

vishesh92 approved these changes Mar 24, 2026

View reviewed changes

sureshanaparti added the ready-for-testing label Mar 24, 2026

rajujith self-assigned this Mar 26, 2026

add user agent in HEAD request as well

e7cf205

add user-agent for httpurlconnection

0f31e3e

rajujith approved these changes Mar 27, 2026

View reviewed changes

rajujith removed their assignment Mar 27, 2026

rajujith removed status:needs-testing ready-for-testing labels Mar 27, 2026

Conversation

sudo87 commented Mar 11, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sudo87 commented Mar 11, 2026

Uh oh!

blueorangutan commented Mar 11, 2026

Uh oh!

codecov bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Mar 11, 2026

Uh oh!

weizhouapache commented Mar 11, 2026

Uh oh!

Uh oh!

sureshanaparti commented Mar 12, 2026

Uh oh!

sureshanaparti commented Mar 12, 2026

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

sudo87 commented Mar 12, 2026

Uh oh!

sureshanaparti commented Mar 12, 2026

Uh oh!

blueorangutan commented Mar 12, 2026

Uh oh!

blueorangutan commented Mar 12, 2026

Uh oh!

sudo87 commented Mar 13, 2026

Uh oh!

blueorangutan commented Mar 13, 2026

Uh oh!

blueorangutan commented Mar 13, 2026

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

sudo87 commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sudo87 commented Mar 24, 2026

Uh oh!

blueorangutan commented Mar 24, 2026

Uh oh!

blueorangutan commented Mar 24, 2026

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

sureshanaparti commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

rajujith commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

sudo87 commented Mar 26, 2026

Uh oh!

sudo87 commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

blueorangutan commented Mar 26, 2026

Uh oh!

sureshanaparti commented Mar 27, 2026

Uh oh!

codecov bot commented Mar 11, 2026 •

edited

Loading

sudo87 commented Mar 24, 2026 •

edited

Loading