feat: node-exporter into vhd build #7704

chmill-zz · 2026-01-22T00:14:23Z

What this PR does / why we need it:

this is adding node-exporter into the vhdbuild by default. At the end of the vhdbuild we systemctl disable to allow a fresh start during cse letting the node-exporter-startup script run and gather up node specific details needed.

Which issue(s) this PR fixes:

Fixes #

cameronmeissner · 2026-01-23T23:17:07Z

parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/node-exporter.d/web-config.yml

@@ -0,0 +1,5 @@
+tls_server_config:
+  cert_file: "/etc/kubernetes/certs/kubeletserver.crt"


these paths aren't necessarily correct - they depend on whether kubelet serving certificate rotation is enabled - when it's disabled these paths are correct, however when it's enabled both cert_file and key_file should point towards: /var/lib/kubelet/pki/kubelet-server-current.pem

interesting this is copy paste from the aks-vm-extension repo and what is on my node today. I don't see any where it's touched, just a static file.

root@aks-sys-41317600-vmss000000:/etc/node-exporter.d# cat web-config.yml tls_server_config: cert_file: "/etc/kubernetes/certs/kubeletserver.crt" key_file: "/etc/kubernetes/certs/kubeletserver.key" client_auth_type: "RequireAndVerifyClientCert" client_ca_file: "/etc/kubernetes/certs/ca.crt"

i think we could address this in the node-exporter-startup.sh and check for the existence of /var/lib/kubelet/pki/kubelet-server-current.pem and if it exists use it. And if not use /etc/kubernetes/certs/kubeletserver.crt

cameronmeissner · 2026-01-23T23:17:55Z

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path

@@ -0,0 +1,5 @@
+[Path]
+PathModified=/var/lib/kubelet/pki/kubelet.crt


are you trying to point towards kubelet's client cert? if so this should be: /var/lib/kubelet/pki/kubelet-client-current.pem

i'm starting to question if this node-exporter-restart.path and node-exporter-restart.service are even needed.

https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md

its reading like node-exporter will be reading the web-config with each request so changes are picked up right away.

This is another config that is copy paste from aks-vm-extensions, and looking like lacking real ownership/maintaining.

cameronmeissner · 2026-01-23T23:19:22Z

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

+
+# specify tls config path, if it exists
+if [ -f $PUBLIC_SETTINGS_PATH ]
+then


nit: we tend to follow the style of if [ condition ]; then

cameronmeissner · 2026-01-23T23:21:17Z

parts/linux/cloud-init/artifacts/cse_config.sh

+configureNodeExporter() {
+    echo "Configuring Node Exporter"
+    # Check for skip file to determine if node-exporter was installed on this VHD
+    if [ ! -f /etc/node-exporter.d/skip_vhd_node_exporter ]; then


if the skip file isn't there, meaning we don't want to skip, we end up skipping? this seems backwards

also, rather than commit and manage an empty flag file in version control, can we instead pivot off some other node exporter asset file's existence?

skip file is for the extension logic as a signal that the component is agentbaker managed and extension should skip all managing.

existing nodes should have all components from extension previously. If we were to check for node-exporter running it should always be there vhd or extension install originally and we wouldn't be able to tell which. The skip file becomes a clear sign that this was vhd baked and extension should ignore.

once we get to a point where extension isn't needed anymore we can easily remove it

cameronmeissner · 2026-01-23T23:23:39Z

parts/linux/cloud-init/artifacts/cse_helpers.sh

-            mv "/opt/bin/${packageName}-${packageVersion}" "/opt/bin/${packageName}"
-            chmod a+x /opt/bin/${packageName}
-            rm -rf /opt/bin/${packageName}-* &
+        elif [ -f "/usr/local/bin/${packageName}-${packageVersion}" ]; then


why the change here? we should be using /opt/bin

yeah i think a few things got messed up when i decided to involve master in some rebasing and not main... everyone is pretty upset by that move.

cameronmeissner · 2026-01-23T23:24:01Z

vhdbuilder/packer/test/linux-vhd-content-test.sh

+  # Check for the skip sentinel file - this is essential for e2e testing
+  # If this file doesn't exist, e2e tests will silently skip node-exporter validation. Which is bad.
+  local skip_file="/etc/node-exporter.d/skip_vhd_node_exporter"
+  if [ ! -f "$skip_file" ]; then


again, this seems backwards

cameronmeissner · 2026-01-23T23:24:50Z

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

@@ -0,0 +1,32 @@
+#!/bin/sh
+
+if [ "$(cat /etc/os-release | grep ^ID= | cut -c 4-)" = "flatcar" ]; then


do we need the flatcar check? it looks like we don't install node exporter on flatcar according to what's currently in install-dependencies.sh

yeah for the initial setup in AB i'm just looking at getting ubuntu and mariner working. Plan is to get all distros and this is copy paste from existing extension so eventually needed

cameronmeissner · 2026-01-23T23:25:26Z

vhdbuilder/packer/install-dependencies.sh

-        exit "${ret}"
-    }
+    # Limit parallel image pulls - count only image_pids, not BCC_PID
+    while [ "${#image_pids[@]}" -ge "$parallel_container_image_pull_limit" ]; do


unrelated changes?

r2k1 · 2026-01-24T06:54:03Z

vhdbuilder/packer/install-node-exporter.sh

+    [ -n "${VHD_LOGS_FILEPATH:-}" ] && echo "  - node-exporter ${NODE_EXPORTER_VERSION}-${NODE_EXPORTER_REVISION}" >> "${VHD_LOGS_FILEPATH}"
+
+    rm -rf "${NODE_EXPORTER_BUILD_ROOT}"
+}


A random question. Would you prefer to write this logic in bash or golang?

chmill-zz requested review from AbelHu, Devinwong, YaoC, awesomenix, cameronmeissner, djsly, ganeshkumarashok, juan-lee, junjiezhang1997, lilypan26, phealy, r2k1, timmy-wright, yewmsft and zachary-bailey as code owners January 22, 2026 00:14

chmill-zz changed the title ~~Nodeexportershift~~ feat: node-exporter into vhd build Jan 22, 2026

chmill-zz force-pushed the nodeexportershift branch from 0d52887 to 60cc001 Compare January 22, 2026 00:40

chmill-zz temporarily deployed to test January 22, 2026 00:41 — with GitHub Actions Inactive

github-actions bot added the components This pull request updates cached components on Linux or Windows VHDs label Jan 22, 2026

chmill-zz temporarily deployed to test January 22, 2026 02:09 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch from f0f23e8 to eb8e316 Compare January 22, 2026 19:34

chmill-zz temporarily deployed to test January 22, 2026 19:34 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch 2 times, most recently from 68a2704 to d494735 Compare January 23, 2026 21:59

chmill-zz requested a review from yagmurbaydogan as a code owner January 23, 2026 21:59

chmill-zz force-pushed the nodeexportershift branch from d494735 to 277e340 Compare January 23, 2026 22:21

chmill-zz temporarily deployed to test January 23, 2026 22:21 — with GitHub Actions Inactive

rebase again

1ae3542

chmill-zz force-pushed the nodeexportershift branch from 277e340 to 1ae3542 Compare January 23, 2026 23:16

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

chmill-zz added 3 commits January 24, 2026 02:34

fixing components, opt ref, startup logic

ec60291

component ans vhd-image-build fixing

3108c0a

maybe ill remember how to do this vhd-image thing one of these days

d9de63e

r2k1 reviewed Jan 24, 2026

View reviewed changes

seeing odd 99 time outs and think maybe exhaustion is happening

8b28957

		@@ -0,0 +1,5 @@
		tls_server_config:
		cert_file: "/etc/kubernetes/certs/kubeletserver.crt"

		@@ -0,0 +1,5 @@
		[Path]
		PathModified=/var/lib/kubelet/pki/kubelet.crt

		@@ -0,0 +1,32 @@
		#!/bin/sh

		if [ "$(cat /etc/os-release \| grep ^ID= \| cut -c 4-)" = "flatcar" ]; then

feat: node-exporter into vhd build #7704

Are you sure you want to change the base?

feat: node-exporter into vhd build #7704

Conversation

chmill-zz commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cameronmeissner Jan 23, 2026 •

edited

Loading

cameronmeissner Jan 23, 2026 •

edited

Loading