Upgrade from v1.31.3 to v1.32.1 failure #22
Description
I've checked out 8ee6ca9 to do the upgrade to v1.32.1 from v1.31.3. It's failed with this error
module.k8s_worker.null_resource.k8s_worker_upgrade[0]: Destroying... [id=6448651301635622870]
module.k8s_worker.null_resource.k8s_worker_upgrade[1]: Destroying... [id=2752923152628568573]
module.k8s_worker.null_resource.k8s_worker_upgrade[0]: Destruction complete after 0s
module.k8s_worker.null_resource.k8s_worker_upgrade[1]: Destruction complete after 0s
module.k8s_storage.brightbox_config_map.k8s_worker: Modifying... [id=cfg-ijpww]
module.k8s_master.null_resource.k8s_master_configure: Creating...
module.k8s_worker.brightbox_config_map.k8s_worker: Modifying... [id=cfg-bw90q]
module.k8s_master.null_resource.k8s_master_configure: Provisioning with 'file'...
module.k8s_storage.brightbox_config_map.k8s_worker: Modifications complete after 0s [id=cfg-ijpww]
module.k8s_worker.brightbox_config_map.k8s_worker: Modifications complete after 0s [id=cfg-bw90q]
module.k8s_master.null_resource.k8s_master_configure: Provisioning with 'file'...
module.k8s_master.null_resource.k8s_master_configure: Provisioning with 'remote-exec'...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Connecting to remote host via SSH...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Host: ipv4.cip-s7sbr.gb1.brightbox.com
module.k8s_master.null_resource.k8s_master_configure (remote-exec): User: ubuntu
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Password: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Private key: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Certificate: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): SSH Agent: true
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Checking Host Key: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Target Platform: unix
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Connected!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): {
module.k8s_master.null_resource.k8s_master_configure (remote-exec): "cniVersion": "0.3.1",
module.k8s_master.null_resource.k8s_master_configure (remote-exec): "type": "loopback"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): }
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.all.forwarding=1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.ip_forward=1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.bridge.bridge-nf-call-ip6tables=1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.bridge.bridge-nf-call-iptables=1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Resetting sysctl
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-console-messages.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.printk = 4 4 1 7
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.all.use_tempaddr = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.default.use_tempaddr = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-kernel-hardening.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.kptr_restrict = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-magic-sysrq.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.sysrq = 176
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-network-security.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.conf.default.rp_filter = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.conf.all.rp_filter = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-ptrace.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.yama.ptrace_scope = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/10-zeropage.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): vm.mmap_min_addr = 65536
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/40-kubernetes.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.all.forwarding = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.ip_forward = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.bridge.bridge-nf-call-ip6tables = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.bridge.bridge-nf-call-iptables = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /usr/lib/sysctl.d/50-default.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.core_uses_pid = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.conf.default.rp_filter = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.conf.default.accept_source_route = 0
module.k8s_master.null_resource.k8s_master_configure (remote-exec): sysctl: setting key "net.ipv4.conf.all.accept_source_route": Invalid argument
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.conf.default.promote_secondaries = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv4.ping_group_range = 0 2147483647
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.core.default_qdisc = fq_codel
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_hardlinks = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_symlinks = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_regular = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_fifos = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /usr/lib/sysctl.d/50-pid-max.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kernel.pid_max = 4194304
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/99-cloudimg-ipv6.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.all.use_tempaddr = 0
module.k8s_master.null_resource.k8s_master_configure (remote-exec): net.ipv6.conf.default.use_tempaddr = 0
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /usr/lib/sysctl.d/99-protect-links.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_fifos = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_hardlinks = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_regular = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fs.protected_symlinks = 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.d/99-sysctl.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): * Applying /etc/sysctl.conf ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Activating kubelet bootstrap services
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Installing bash completion
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Make containerd CRI v1 runtime use systemd cgroup
module.k8s_master.null_resource.k8s_master_configure (remote-exec): version = 2
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): runtime_type = "io.containerd.runc.v2"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): SystemdCgroup = true
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting up critools
module.k8s_master.null_resource.k8s_master_configure (remote-exec): runtime-endpoint: unix:///run/containerd/containerd.sock
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Making time sync run more often
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [Time]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): PollIntervalMaxSec=1024
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Restarting reconfigured services
module.k8s_master.null_resource.k8s_master_configure: Provisioning with 'remote-exec'...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Connecting to remote host via SSH...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Host: ipv4.cip-s7sbr.gb1.brightbox.com
module.k8s_master.null_resource.k8s_master_configure (remote-exec): User: ubuntu
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Password: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Private key: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Certificate: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): SSH Agent: true
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Checking Host Key: false
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Target Platform: unix
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Connected!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Installing kubeadm.conf
module.k8s_master.null_resource.k8s_master_configure (remote-exec): deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Kubeadm installed master detected - checking for upgrade
module.k8s_master.null_resource.k8s_master_configure: Still creating... [00m10s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Upgrading cri-tools
module.k8s_master.null_resource.k8s_master_configure: Still creating... [00m20s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 5%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 10%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 15%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 20%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 25%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 30%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 35%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 40%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 45%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 50%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 55%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 60%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 65%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 70%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 75%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 80%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 85%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 90%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 100%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95150 files and directories currently installed.)
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Preparing to unpack .../cri-tools_1.32.0-1.1_amd64.deb ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Unpacking cri-tools (1.32.0-1.1) over (1.31.1-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting up cri-tools (1.32.0-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): packages have been installed but needrestart is suspended
module.k8s_master.null_resource.k8s_master_configure (remote-exec): cri-tools set on hold.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Upgrading kubeadm
module.k8s_master.null_resource.k8s_master_configure: Still creating... [00m30s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 5%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 10%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 15%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 20%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 25%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 30%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 35%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 40%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 45%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 50%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 55%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 60%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 65%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 70%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 75%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 80%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 85%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 90%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 100%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95150 files and directories currently installed.)
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Preparing to unpack .../kubeadm_1.32.1-1.1_amd64.deb ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Unpacking kubeadm (1.32.1-1.1) over (1.31.3-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting up kubeadm (1.32.1-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): packages have been installed but needrestart is suspended
module.k8s_master.null_resource.k8s_master_configure: Still creating... [00m40s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kubeadm set on hold.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/preflight] Running preflight checks
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] Running cluster health checks
module.k8s_master.null_resource.k8s_master_configure: Still creating... [00m50s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/preflight] You have chosen to upgrade the cluster version to "v1.32.1"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/versions] Cluster version: v1.31.3
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/versions] kubeadm version: v1.32.1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
module.k8s_master.null_resource.k8s_master_configure (remote-exec): W0902 10:56:41.899381 452313 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m00s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m10s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.32.1" (timeout: 5m0s)...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4025462875"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Preparing for "etcd" upgrade
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing etcd-server certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing etcd-peer certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing etcd-healthcheck-client certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-09-02-10-57-04/etcd.yaml"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Waiting for the kubelet to restart the component
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] This can take up to 5m0s
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m20s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m30s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m40s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [01m50s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m00s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m10s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m20s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m30s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [apiclient] Found 1 Pods for label selector component=etcd
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m40s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Component "etcd" upgraded successfully!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/etcd] Waiting for etcd to become available
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Preparing for "kube-apiserver" upgrade
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing apiserver certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing apiserver-kubelet-client certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing front-proxy-client certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing apiserver-etcd-client certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-09-02-10-57-04/kube-apiserver.yaml"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Waiting for the kubelet to restart the component
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] This can take up to 5m0s
module.k8s_master.null_resource.k8s_master_configure: Still creating... [02m50s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m00s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m10s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m20s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [apiclient] Found 1 Pods for label selector component=kube-apiserver
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m30s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing controller-manager.conf certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-09-02-10-57-04/kube-controller-manager.yaml"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Waiting for the kubelet to restart the component
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] This can take up to 5m0s
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m40s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [apiclient] Found 1 Pods for label selector component=kube-controller-manager
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Preparing for "kube-scheduler" upgrade
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Renewing scheduler.conf certificate
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-09-02-10-57-04/kube-scheduler.yaml"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Waiting for the kubelet to restart the component
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] This can take up to 5m0s
module.k8s_master.null_resource.k8s_master_configure: Still creating... [03m50s elapsed]
module.k8s_master.null_resource.k8s_master_configure: Still creating... [04m00s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [apiclient] Found 1 Pods for label selector component=kube-scheduler
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/control-plane] The control plane instance for this node was successfully upgraded!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrad/kubeconfig] The kubeconfig files for this node were successfully upgraded!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): W0902 10:59:56.987945 452313 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1313518119 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1313518119/config.yaml
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [addons] Applied essential addon: CoreDNS
module.k8s_master.null_resource.k8s_master_configure: Still creating... [04m10s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [addons] Applied essential addon: kube-proxy
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.32.1".
module.k8s_master.null_resource.k8s_master_configure (remote-exec): [upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Upgrading kubelet
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 5%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 10%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 15%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 20%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 25%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 30%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 35%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 40%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 45%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 50%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 55%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 60%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 65%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 70%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 75%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 80%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 85%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 90%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 100%
module.k8s_master.null_resource.k8s_master_configure (remote-exec): (Reading database ... 95150 files and directories currently installed.)
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Preparing to unpack .../kubectl_1.32.1-1.1_amd64.deb ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Unpacking kubectl (1.32.1-1.1) over (1.31.3-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Preparing to unpack .../kubelet_1.32.1-1.1_amd64.deb ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Unpacking kubelet (1.32.1-1.1) over (1.31.3-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting up kubectl (1.32.1-1.1) ...
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting up kubelet (1.32.1-1.1) ...
module.k8s_master.null_resource.k8s_master_configure: Still creating... [04m20s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): packages have been installed but needrestart is suspended
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kubelet set on hold.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kubectl set on hold.
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Installing admin config
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Installing client cert generator
module.k8s_master.null_resource.k8s_master_configure (remote-exec): #!/usr/bin/env bash
module.k8s_master.null_resource.k8s_master_configure (remote-exec): if [[ "${#}" -lt 2 ]]; then
module.k8s_master.null_resource.k8s_master_configure (remote-exec): echo "Usage: ${0} user group [group ...]"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): exit 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fi
module.k8s_master.null_resource.k8s_master_configure (remote-exec): USER=${1}; shift
module.k8s_master.null_resource.k8s_master_configure (remote-exec): GROUP=""
module.k8s_master.null_resource.k8s_master_configure (remote-exec): for word
module.k8s_master.null_resource.k8s_master_configure (remote-exec): do
module.k8s_master.null_resource.k8s_master_configure (remote-exec): GROUP="${GROUP}/O=${word}"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): done
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CLUSTERENDPOINT=https://cip-s7sbr.gb1.brightbox.com:6443
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CLUSTERNAME=luzme-2024.cluster.local
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CACERT=/etc/kubernetes/pki/ca.crt
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CAKEY=/etc/kubernetes/pki/ca.key
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CLIENTCERTKEY=clients/${USER}/${USER}.key
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CLIENTCERTCSR=clients/${USER}/${USER}.csr
module.k8s_master.null_resource.k8s_master_configure (remote-exec): CLIENTCERTCRT=clients/${USER}/${USER}.crt
module.k8s_master.null_resource.k8s_master_configure (remote-exec): if [ ! -r "${CAKEY}" ]; then
module.k8s_master.null_resource.k8s_master_configure (remote-exec): echo "Insufficent permissions to read ${CAKEY}"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): exit 1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): fi
module.k8s_master.null_resource.k8s_master_configure (remote-exec): mkdir -p clients/${USER}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): openssl genrsa -out ${CLIENTCERTKEY} 4096
module.k8s_master.null_resource.k8s_master_configure (remote-exec): openssl req -new -key ${CLIENTCERTKEY} -out ${CLIENTCERTCSR} \
module.k8s_master.null_resource.k8s_master_configure (remote-exec): -subj "/CN=${USER}${GROUP}"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): openssl x509 -req -days 365 -sha256 -in ${CLIENTCERTCSR} -CA ${CACERT} -CAkey ${CAKEY} -set_serial 2 -out ${CLIENTCERTCRT}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): cat <<-EOF > clients/${USER}/kubeconfig
module.k8s_master.null_resource.k8s_master_configure (remote-exec): apiVersion: v1
module.k8s_master.null_resource.k8s_master_configure (remote-exec): kind: Config
module.k8s_master.null_resource.k8s_master_configure (remote-exec): preferences:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): colors: true
module.k8s_master.null_resource.k8s_master_configure (remote-exec): current-context: ${CLUSTERNAME}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusters:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): - name: ${CLUSTERNAME}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): cluster:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): server: ${CLUSTERENDPOINT}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): certificate-authority-data: |-
module.k8s_master.null_resource.k8s_master_configure (remote-exec): $(base64 --wrap=0 ${CACERT})
module.k8s_master.null_resource.k8s_master_configure (remote-exec): contexts:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): - context:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): cluster: ${CLUSTERNAME}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): user: ${USER}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): name: ${CLUSTERNAME}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): users:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): - name: ${USER}
module.k8s_master.null_resource.k8s_master_configure (remote-exec): user:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): client-certificate-data: |-
module.k8s_master.null_resource.k8s_master_configure (remote-exec): $(base64 --wrap=0 ${CLIENTCERTCRT})
module.k8s_master.null_resource.k8s_master_configure (remote-exec): client-key-data: |-
module.k8s_master.null_resource.k8s_master_configure (remote-exec): $(base64 --wrap=0 ${CLIENTCERTKEY})
module.k8s_master.null_resource.k8s_master_configure (remote-exec): EOF
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Setting permissions
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Refreshing Pod Disruption Budget for kube-dns
module.k8s_master.null_resource.k8s_master_configure (remote-exec): poddisruptionbudget.policy/coredns-pdb configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Installing Calico network controllers
module.k8s_master.null_resource.k8s_master_configure (remote-exec): poddisruptionbudget.policy/calico-kube-controllers configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): serviceaccount/calico-kube-controllers unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): serviceaccount/calico-node unchanged
module.k8s_master.null_resource.k8s_master_configure: Still creating... [04m30s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure: Still creating... [04m40s elapsed]
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/tiers.crd.projectcalico.org configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): customresourcedefinition.apiextensions.k8s.io/adminnetworkpolicies.policy.networking.k8s.io configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrole.rbac.authorization.k8s.io/calico-node unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrole.rbac.authorization.k8s.io/calico-cni-plugin unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): daemonset.apps/calico-node configured
module.k8s_master.null_resource.k8s_master_configure (remote-exec): deployment.apps/calico-kube-controllers unchanged
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Error from server (Forbidden): error when retrieving current configuration of:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Name: "calico-cni-plugin", Namespace: "kube-system"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): from server for: "STDIN": serviceaccounts "calico-cni-plugin" is forbidden: User "kubernetes-admin" cannot get resource "serviceaccounts" in API group "" in the namespace "kube-system"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Error from server (Forbidden): error when retrieving current configuration of:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Resource: "/v1, Resource=configmaps", GroupVersionKind: "/v1, Kind=ConfigMap"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Name: "calico-config", Namespace: "kube-system"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): from server for: "STDIN": configmaps "calico-config" is forbidden: User "kubernetes-admin" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Error from server (Forbidden): error when retrieving current configuration of:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Name: "bgpconfigurations.crd.projectcalico.org", Namespace: ""
module.k8s_master.null_resource.k8s_master_configure (remote-exec): from server for: "STDIN": customresourcedefinitions.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" is forbidden: User "kubernetes-admin" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Error from server (Forbidden): error when retrieving current configuration of:
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
module.k8s_master.null_resource.k8s_master_configure (remote-exec): Name: "bgpfilters.crd.projectcalico.org", Namespace: ""
module.k8s_master.null_resource.k8s_master_configure (remote-exec): from server for: "STDIN": customresourcedefinitions.apiextensions.k8s.io "bgpfilters.crd.projectcalico.org" is forbidden: User "kubernetes-admin" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
╷
│ Error: remote-exec provisioner error
│
│ with module.k8s_master.null_resource.k8s_master_configure,
│ on master/main.tf line 186, in resource "null_resource" "k8s_master_configure":
│ 186: provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_523202900.sh": Process exited with status 1
╵
Metadata
Metadata
Assignees
Labels
No labels