Add ROOT CAs to the trust store by vishesh92 · Pull Request #12911 · apache/cloudstack

vishesh92 · 2026-03-30T07:06:17Z

Description

This PR adds ROOT CAs to the trust store in System VMs & the management
This PR also adds another parameter forced to provisionCertificate command which allows provisioning certs via SSH to KVM hosts & system VMs.

Details

This pull request introduces several improvements and new features to the CloudStack Certificate Authority (CA) management system, with a focus on enhancing certificate provisioning flexibility, supporting user-provided CA material, and improving truststore management. The most significant changes add support for forced certificate re-provisioning via SSH for disconnected agents, clarify and enforce requirements for custom CA keys, and enable automatic truststore injection for outgoing HTTPS connections.

Certificate Provisioning Enhancements:

Added a forced boolean parameter to the ProvisionCertificateCmd API and the CAManager.provisionCertificate method, allowing administrators to re-provision agent certificates via SSH when agents are disconnected (e.g., after a CA change). This is supported for KVM hosts and SystemVMs. The implementation includes a new provisionCertificateViaSsh method for KVM hosts. [1] [2] [3] [4] [5] [6]

CA Provider and Configuration Improvements:

Expanded and clarified configuration key documentation for user-provided CA material in the RootCAProvider, including PEM format requirements and the need to set all three keys together. Enhanced error handling and logging when loading user-provided CA keys/certificates fails, with fallback to auto-generation. [1] [2]
Improved the description of the ca.framework.provider.plugin configuration key to clarify its purpose and the requirements for custom CA material.

Truststore Management:

Introduced a new configuration key ca.framework.inject.default.truststore to control whether the CA provider's certificate is injected into the JVM default truststore on management server startup, enabling outgoing HTTPS connections from the management server to trust servers signed by the configured CA.
Updated the keystore-cert-import script to also import the CA certificate into the realhostip.keystore used by the SSVM JVM, ensuring trust for CloudStack CA-signed servers.

Script and Utility Fixes:

Improved the keystore-cert-import script to handle certificate splitting and file cleanup more robustly, preventing errors if no certificates are present.

Code Cleanup:

Removed unused imports and code from LibvirtServerDiscoverer and related classes as part of the refactoring for SSH-based certificate provisioning. [1] [2] [3] [4] [5] [6] [7]

These changes collectively improve CloudStack's certificate management flexibility, security, and ease of integration with external CA infrastructures.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

vishesh92 · 2026-03-30T07:10:11Z

@blueorangutan package

blueorangutan · 2026-03-30T07:12:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-03-30T07:19:29Z

Codecov Report

❌ Patch coverage is 36.78161% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.02%. Comparing base (1bff543) to head (a77ed3e).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/java/org/apache/cloudstack/ca/CAManagerImpl.java	41.96%	56 Missing and 9 partials ⚠️
.../apache/cloudstack/ca/provider/RootCAProvider.java	13.63%	37 Missing and 1 partial ⚠️
.../api/command/admin/ca/ProvisionCertificateCmd.java	0.00%	4 Missing ⚠️
...rvisor/kvm/discoverer/LibvirtServerDiscoverer.java	0.00%	1 Missing ⚠️
utils/src/main/java/com/cloud/utils/nio/Link.java	0.00%	1 Missing ⚠️
...rg/apache/cloudstack/utils/security/CertUtils.java	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #12911    +/-   ##
==========================================
  Coverage     18.02%   18.02%            
- Complexity    16460    16476    +16     
==========================================
  Files          5968     5973     +5     
  Lines        537213   537590   +377     
  Branches      65975    66014    +39     
==========================================
+ Hits          96825    96922    +97     
- Misses       429469   429734   +265     
- Partials      10919    10934    +15

Flag	Coverage Δ
uitests	`3.52% <ø> (-0.01%)`	⬇️
unittests	`19.19% <36.78%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

weizhouapache · 2026-03-30T07:24:38Z

server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java

+
+            provisionCertificateViaSsh(sshConnection, hostIp, host.getName());
+
+            SSHCmdHelper.sshExecuteCmd(sshConnection, "sudo service cloudstack-agent restart");


sudo could be a problem, depends on which user is used on the kvm host. please check.

may use systemctl instead of service, please check and keep consistent with other commands.

weizhouapache · 2026-03-30T07:25:38Z

plugins/ca/root-ca/src/main/java/org/apache/cloudstack/ca/provider/RootCAProvider.java

            null,
-            "The ROOT CA certificate.", true);
+            "The ROOT CA X.509 certificate in PEM format (must start with '-----BEGIN CERTIFICATE-----'). " +
+            "Required when providing a custom CA. Restart management server(s) when changed.", true);


test and update description if an intermediate certificate exists ?

blueorangutan · 2026-03-30T07:55:52Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17282

blueorangutan · 2026-03-30T08:06:22Z

[SF] Trillian Build Failed (tid-15758)

Copilot

Pull request overview

This PR extends CloudStack’s CA/certificate provisioning to support forced re-provisioning via SSH (for disconnected KVM hosts and SystemVMs) and improves truststore handling so CloudStack CA roots can be trusted by management and SystemVM JVM processes.

Changes:

Add a forced flag to provisionCertificate (API + manager) and implement SSH-based provisioning paths for KVM hosts and SystemVMs.
Inject configured CA certificate(s) into the management server JVM default truststore on startup (configurable via a new global setting).
Update SystemVM/scripts truststore import logic to also populate realhostip.keystore with CA certs; improve cert-chain splitting robustness.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
utils/src/main/java/org/apache/cloudstack/utils/security/KeyStoreUtils.java	Removes an unused SystemVM import-script constant.
utils/src/main/java/com/cloud/utils/nio/Link.java	Adjusts handshake wrap error handling to fail the handshake on SSL wrap exceptions.
systemvm/patch-sysvms.sh	Imports CA cert chain into `realhostip.keystore` during live patch for CPVM/SSVM.
server/src/test/java/org/apache/cloudstack/ca/CAManagerImplTest.java	Updates tests for the new `provisionCertificate(..., forced)` signature.
server/src/test/java/org/apache/cloudstack/ca/CABackgroundTaskTest.java	Updates mocks/verifications for the new method signature.
server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java	Implements forced provisioning via SSH + JVM truststore injection feature and wires new dependencies.
server/src/main/java/com/cloud/hypervisor/kvm/discoverer/LibvirtServerDiscoverer.java	Refactors KVM discovery provisioning to call `caManager.provisionCertificateViaSsh(...)`.
scripts/util/keystore-cert-import	Improves cert split loop robustness; imports CA into `realhostip.keystore` for SystemVMs.
plugins/ca/root-ca/src/main/java/org/apache/cloudstack/ca/provider/RootCAProvider.java	Clarifies config docs, improves fallback logging, and fixes startup loading sequence.
api/src/main/java/org/apache/cloudstack/ca/CAManager.java	Adds `forced` support + introduces `CaInjectDefaultTruststore` config key + adds `provisionCertificateViaSsh`.
api/src/main/java/org/apache/cloudstack/api/command/admin/ca/ProvisionCertificateCmd.java	Adds `forced` API parameter and passes it through to CA manager.

Comments suppressed due to low confidence (1)

scripts/util/keystore-cert-import:78

The certificate-chain splitting uses temp files named cloudca.* in the current working directory. This can fail or be unsafe if the script runs from an unexpected directory (clobbering existing files or following symlinks). Consider using a dedicated temporary directory via mktemp -d and writing the split certs there, then cleaning up that directory.

awk 'BEGIN{n=0} /-----BEGIN CERTIFICATE-----/{n++}{print > "cloudca." n }' "$CACERT_FILE"
for caChain in $(ls cloudca.* 2>/dev/null); do
    keytool -delete -noprompt -alias "$caChain" -keystore "$KS_FILE" -storepass "$KS_PASS" > /dev/null 2>&1 || true
    keytool -import -noprompt -storepass "$KS_PASS" -trustcacerts -alias "$caChain" -file "$caChain" -keystore "$KS_FILE" > /dev/null 2>&1
done
rm -f cloudca.*

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java

scripts/util/keystore-cert-import

systemvm/patch-sysvms.sh

api/src/main/java/org/apache/cloudstack/ca/CAManager.java

server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java

scripts/util/keystore-cert-import

blueorangutan · 2026-03-30T08:19:31Z

[SF] Trillian Build Failed (tid-15759)

vishesh92 · 2026-03-30T12:01:49Z

@blueorangutan package

blueorangutan · 2026-03-30T12:04:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-30T12:54:34Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17290

vishesh92 · 2026-03-31T07:38:58Z

@blueorangutan package

blueorangutan · 2026-03-31T07:40:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-31T08:23:30Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17300

vishesh92 · 2026-03-31T09:49:20Z

@blueorangutan package

blueorangutan · 2026-03-31T09:50:05Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-31T10:54:59Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17301

vishesh92 added 3 commits March 30, 2026 12:23

Add root CA to the truststore for System VMs

c901d51

Add root CA certs to default truststore for MS

c23231f

Update provision certificate command to allow provisioning via SSH

516f1f4

boring-cyborg bot added component:api component:kvm component:security labels Mar 30, 2026

weizhouapache reviewed Mar 30, 2026

View reviewed changes

DaanHoogland requested a review from Copilot March 30, 2026 07:43

Copilot started reviewing on behalf of DaanHoogland March 30, 2026 07:58 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

vishesh92 added 2 commits March 30, 2026 14:52

Address comments

d3b0026

Add support for intermediate certificate chain in RootCAProvider

6c7fffb

Add some unit tests

3ebc750

Add some more unit tests

a77ed3e

apache deleted a comment from blueorangutan Mar 31, 2026


		provisionCertificateViaSsh(sshConnection, hostIp, host.getName());

		SSHCmdHelper.sshExecuteCmd(sshConnection, "sudo service cloudstack-agent restart");

Conversation

vishesh92 commented Mar 30, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

vishesh92 commented Mar 30, 2026

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

codecov bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

weizhouapache Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

vishesh92 commented Mar 30, 2026

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

blueorangutan commented Mar 30, 2026

Uh oh!

vishesh92 commented Mar 31, 2026

Uh oh!

blueorangutan commented Mar 31, 2026

Uh oh!

blueorangutan commented Mar 31, 2026

Uh oh!

vishesh92 commented Mar 31, 2026

Uh oh!

blueorangutan commented Mar 31, 2026

Uh oh!

blueorangutan commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 30, 2026 •

edited

Loading