Skip to content

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chewi wants to merge 15 commits into
base: main
Choose a base branch
from
Open

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0306d0c
fix: explicitly exclude containerd from Flatcar in components.json
chewi Oct 1, 2025
3642583
fix: testPackagesInstalled logic for Flatcar and Azure Linux OS Guard
chewi Sep 30, 2025
2fddd42
style: make "isOS" shell helpers less cumbersome to use and add isUbuntu
chewi Aug 28, 2025
fbe78db
refactor: simplify configureKubeletAndKubectl logic
chewi Aug 28, 2025
2c6b56f
refactor: simplify credential provider installation logic
chewi Sep 23, 2025
904153b
style: rename installKubeletKubectlPkgFromPMC to *FromPkg
chewi Aug 28, 2025
c91d847
rename: installCredentialProviderFromPMC to *FromPkg
chewi Sep 23, 2025
99d3007
feat: install kubelet and kubectl to Flatcar from sysexts on MAR
chewi Aug 28, 2025
d82f62b
feat: install credential provider to Flatcar from sysext on MAR
chewi Sep 23, 2025
17b84a6
test: don't mix distro scripts as they clash and use BeforeEach properly
chewi Jan 21, 2026
0f704dd
test: add shellspec tests for new Flatcar and sysext code
chewi Sep 1, 2025
5ad0d86
fix: adjust Flatcar aks-sysext to not use /etc or /opt/bin
chewi Jan 16, 2026
e1454ee
fix: move apt_get_download to Ubuntu-specific script file
chewi Jan 19, 2026
44b6428
fix: only provision Ubuntu with /opt/azure/manifest.json
chewi Jan 19, 2026
b96f8d1
chore: update test data
chewi Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/README-RENOVATE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,8 @@ where
- `${version}` will be resolved at runtime with the `latestVersion` and `previousLatestVersion` defined above.
- `${CPU_ARCH}` will be resolved at runtime depending on the CPU architecture of the Node (VM) under provisioning.

systemd system extensions (sysexts) are also hosted as MAR OCI artifacts, but they use a slightly different `extractVersion` rule and `downloadURL`. The distribution (e.g. `azlinux3`) is included in the version to allow different distributions within groups of artifacts. `${SYSTEMD_ARCH}` rather than `${CPU_ARCH}` is used in the URL, as systemd has different architecture names in some cases.

## `REVISION` in Dalec built container images
Dalec-built container images use static tags in the form `vMAJOR.MINOR.PATCH-REVISION` (see the Dalec FAQ https://github.com/Azure/dalec-build-defs/blob/main/faq.md#how-do-floating-vs-static-tags-work for details). For clarity and deterministic caching we represent these container images in Agent Baker's `components.json` using the exact static tag `vMAJOR.MINOR.PATCH-REVISION`.

Expand Down
13 changes: 12 additions & 1 deletion .github/renovate.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@
},
{
"matchPackageNames": [
"oss/v2/**"
"oss/v2/**",
"!oss/v2/kubernetes/*-sysext"
],
"versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-(?<prerelease>\\d+)$",
"ignoreUnstable": false
Expand Down Expand Up @@ -479,6 +480,16 @@
],
"extractVersion": "^(?P<version>.*?)-[^-]*-[^-]*$"
},
{
"matchDatasources": [
"docker"
],
"matchPackageNames": [
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who would be the owner of such packages ? I'm not sure if they would have automatic assigned in the PR

Copy link
Contributor

@cameronmeissner cameronmeissner Jan 21, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably Chewi + Jeremi as primary owners, and maybe us as secondary?

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we need to be owners. These sysexts are published in tandem with their .deb and .rpm counterparts from the very same Dalec YAML definition. Adding sysext support is usually just a case of adding one line.

"oss/v2/kubernetes/*-sysext"
],
"matchCurrentVersion": "/-azlinux3$/",
"extractVersion": "^(?P<version>.+-azlinux3)-"
},
{
"matchPackageNames": [
"aks/aks-gpu-cuda"
Expand Down
145 changes: 117 additions & 28 deletions parts/common/components.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1086,6 +1086,16 @@
]
}
},
"flatcar": {
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when should we expect udpated/patched version of containerd ? from base images only ?

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Jeremi took this decision a while back. Bear in mind that Flatcar releases every few weeks.

"current": {
"versionsV2": [
{
"renovateTag": "<DO_NOT_UPDATE>",
"latestVersion": "<SKIP>"
}
]
}
},
"windows": {
"ws2019": {
"versionsV2": [
Expand Down Expand Up @@ -1324,6 +1334,48 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.28",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.28.101-4-azlinux3"
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are those called azlinux3 ? can they also be used by azurelinux3 ? or maybe osguard ? I'm guessing they are built by azl3 team ? instead of upstream team ?

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may recall my Dalec sysext work. 😄 We were told we have to use the binaries already published to PMC for these components. Why? Honestly, I still haven't had a clear answer on that, I'm just following orders. But yes, these can be used with OS Guard. Using them with a mutable distro like Azure Linux (or Ubuntu with compatible sysexts) is possible, but you have to effectively ban subsequent writes to /usr. Mutable mode for sysexts is also a thing, but I'm not very familiar with it yet. Thilo can tell you more.

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, these are built by the upstream team. They are called -azlinux3 to signify that they were built against that, but you can use them with other distros as long as they have compatible libraries; basically, a matching glibc and OpenSSL in this case. If the sysext was truly universal with no dynamic executables, it wouldn't need such a suffix.

},
{
"k8sVersion": "1.29",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.29.100-2-azlinux3"
},
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.30.100-2-azlinux3"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.31.14-1-azlinux3"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.32.9-4-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.33.7-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/kubelet-sysext:${version}-${SYSTEMD_ARCH}"
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we check for azlinux3 for the versions when we overwrite the systemd arch when we donwload ?

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I don't understand the question.

}
}
}
},
Expand Down Expand Up @@ -1375,6 +1427,48 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.28",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.28.101-6-azlinux3"
},
{
"k8sVersion": "1.29",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.29.100-3-azlinux3"
},
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.30.100-3-azlinux3"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.31.14-1-azlinux3"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.32.9-5-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.33.7-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/kubectl-sysext:${version}-${SYSTEMD_ARCH}"
}
}
}
},
Expand Down Expand Up @@ -1485,34 +1579,7 @@
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.30.15"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.31.12"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.32.11"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.33.6"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.34.3"
}
],
"downloadURL": "mcr.microsoft.com/oss/binaries/kubernetes/azure-acr-credential-provider:${version}-linux-${CPU_ARCH}"
"versionsV2": []
}
}
}
Expand Down Expand Up @@ -1605,6 +1672,28 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.32.11-1-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.33.6-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/azure-acr-credential-provider-sysext:${version}-${SYSTEMD_ARCH}"
}
}
}
},
Expand Down
2 changes: 1 addition & 1 deletion parts/linux/cloud-init/artifacts/azlosguard/cse_install_osguard.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ stub() {
echo "${FUNCNAME[1]} stub"
}

installKubeletKubectlPkgFromPMC() {
installKubeletKubectlFromPkg() {
local desiredVersion="${1}"
installRPMPackageFromFile "kubelet" $desiredVersion || exit $ERR_KUBELET_INSTALL_FAIL
installRPMPackageFromFile "kubectl" $desiredVersion || exit $ERR_KUBECTL_INSTALL_FAIL
Expand Down
59 changes: 24 additions & 35 deletions parts/linux/cloud-init/artifacts/cse_config.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,8 @@ ExecStartPost=/sbin/iptables -P FORWARD ACCEPT
EOF

mkdir -p /etc/containerd
# Remove in case this is an existing symlink
rm -f /etc/containerd/config.toml
if [ "${GPU_NODE}" = "true" ]; then
# Check VM tag directly to determine if GPU drivers should be skipped
export -f should_skip_nvidia_drivers
Expand Down Expand Up @@ -550,27 +552,19 @@ EOF
}

configureKubeletAndKubectl() {
# Install kubelet and kubectl binaries from URL for Custom Kube binary and Private Kube binary
if [ -n "${CUSTOM_KUBE_BINARY_DOWNLOAD_URL}" ] || [ -n "${PRIVATE_KUBE_BINARY_DOWNLOAD_URL}" ]; then
# Install kubelet and kubectl binaries from URL:
# 1. For Custom Kube binary or Private Kube binary.
# 2. If k8s version < 1.34.0, skip_bypass_k8s_version_check != true, and not Flatcar (which falls back to URL later).
# 3. For Azure Linux v2 due to lack of PMC packages (if not network isolated).
if [ -n "${CUSTOM_KUBE_BINARY_DOWNLOAD_URL}" ] || [ -n "${PRIVATE_KUBE_BINARY_DOWNLOAD_URL}" ] ||
{ ! isFlatcar && [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != true ] && ! semverCompare "${KUBERNETES_VERSION:-0.0.0}" 1.34.0; } ||
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess with this logic SHOULD_ENFORCE_KUBE_PMC_INSTALL isn't really enforced anymore when
BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVERis defined and if not on the three OS in the last clause

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that was the case before either?

{ isMarinerOrAzureLinux && [ "${OS_VERSION}" = 2.0 ] && [ -z "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; }
then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
# only install kube pkgs from pmc if k8s version >= 1.34.0 or skip_bypass_k8s_version_check is true
elif [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && ! semverCompare ${KUBERNETES_VERSION:-"0.0.0"} "1.34.0"; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
else
if [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ] ; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromBootstrapProfileRegistry" "installKubeletKubectlFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}"
elif isMarinerOrAzureLinux "$OS"; then
if [ "$OS_VERSION" = "2.0" ]; then
# we do not publish packages to PMC for azurelinux V2
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
else
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlPkgFromPMC" "installKubeletKubectlPkgFromPMC ${KUBERNETES_VERSION}"
fi
elif [ "${OS}" = "${UBUNTU_OS_NAME}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlPkgFromPMC" "installKubeletKubectlPkgFromPMC ${KUBERNETES_VERSION}"
elif [ "${OS}" = "${FLATCAR_OS_NAME}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
fi
elif [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromBootstrapProfileRegistry" "installKubeletKubectlFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}"
elif isMarinerOrAzureLinux || isFlatcar || isUbuntu; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromPkg" "installKubeletKubectlFromPkg ${KUBERNETES_VERSION}"
fi
}

Expand Down Expand Up @@ -765,23 +759,18 @@ EOF
if [[ $KUBELET_FLAGS == *"image-credential-provider-config"* && $KUBELET_FLAGS == *"image-credential-provider-bin-dir"* ]]; then
echo "Configure credential provider for both image-credential-provider-config and image-credential-provider-bin-dir flags are specified in KUBELET_FLAGS"
logs_to_events "AKS.CSE.ensureKubelet.configCredentialProvider" configCredentialProvider
if { [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && ! semverCompare ${KUBERNETES_VERSION:-"0.0.0"} "1.34.0"; }; then
# Install credential provider from URL:
# 1. If k8s version < 1.34.0, skip_bypass_k8s_version_check != true, and not Flatcar (which falls back to URL later).
# 2. For Azure Linux v2 due to lack of PMC packages (if not network isolated).
if { ! isFlatcar && [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != true ] && ! semverCompare "${KUBERNETES_VERSION:-0.0.0}" 1.34.0; } ||
{ isMarinerOrAzureLinux && [ "${OS_VERSION}" = 2.0 ] && [ -z "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; }
then
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
elif [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; then
# For network isolated clusters, try distro packages first and fallback to binary installation
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromBootstrapProfileRegistry" installCredentialProviderPackageFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}
else
Copy link
Collaborator

@djsly djsly Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the else here while the elif isMarinerOrAzureLinux || isFlatcar || isUbuntu; for the other pacakge ?

Copy link
Contributor Author

@chewi chewi Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's just how it was written before. 🤷‍♂️ I didn't want to change the behaviour.

if [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ] ; then
# For network isolated clusters, try distro packages first and fallback to binary installation
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromBootstrapProfileRegistry" installCredentialProviderPackageFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}
elif isMarinerOrAzureLinux "$OS"; then
if [ "$OS_VERSION" = "2.0" ]; then # PMC package installation not supported for AzureLinux V2, only V3
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
else
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPMC" "installCredentialProviderFromPMC ${KUBERNETES_VERSION}"
fi
elif isFlatcar "$OS"; then # Flatcar cannot use PMC. It will use sysext soon.
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
else
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPMC" "installCredentialProviderFromPMC ${KUBERNETES_VERSION}"
fi
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPkg" "installCredentialProviderFromPkg ${KUBERNETES_VERSION}"
fi
fi

Expand Down
Loading
Loading