cAdvisor holds LVM device references, preventing TopoLVM from deleting logical volumes on RKE nodes

[darinvhs]

Summary

When a pod using a TopoLVM volume is deleted, the LV should be unmounted and removed. However, cAdvisor (installed via a Helm chart) keeps an open reference to the LV’s mount path, causing TopoLVM to fail volume deletion. The LV remains in “open” state (-wi-ao----) and lvremove will not proceed until the cAdvisor pod on that node is restarted.

This results in stuck PVCs, dangling LVs, and manual intervention.

Symptoms

• lvremove or lvchange returns device busy
• TopoLVM logs indicate the LV cannot be deleted
• mount, lsof, or lsblk shows the LV is still in use
• lvs displays the LV with the o (open) flag
• LV deletion succeeds only after the Helm-managed cAdvisor pod on that node is restarted

Steps to Reproduce

1. Run Kubernetes on RKE with TopoLVM and cAdvisor installed via Helm 
2. Create a pod using a TopoLVM-backed LV.
3. Delete the pod.
4. Check mount references:
   mount | grep <lv-id>
   sudo lsof | grep <lv-id>

5. Attempt LV deletion:
   sudo lvremove /dev/vg-topolvm-ssd-3/<lv-id>
   → fails with "Device or resource busy"

6. Identify the cAdvisor pod on that node:
   kubectl get pods -n kube-system -o wide | grep cadvisor

7. Restart cAdvisor (Helm-managed DaemonSet will recreate it):
   kubectl delete pod <cadvisor-pod> -n kube-system

8. Retry LV deletion — it succeeds.

Expected Behavior

cAdvisor should release mount and device references when the pod using the LV is removed, allowing TopoLVM to unmount and delete the LV without requiring a restart of the cAdvisor pod.

Actual Behavior

cAdvisor (deployed via Helm) keeps a stale mount reference under /var/lib/kubelet/pods/, causing the LV to stay active (o flag).
TopoLVM cannot unmount or delete the LV until the cAdvisor pod on that node is restarted.

Environment

cAdvisor version:

    image: gcr.io/cadvisor/cadvisor:v0.52.1
    imageID: gcr.io/cadvisor/cadvisor@sha256:f40e65878e25c2e78ea037f73a449527a0fb994e303dc3e34cb6b187b4b91435

Kernel Version

Linux baremetal01 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Host OS

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble

Container runtime: containerd