[Bug]: Pyxis + Enroot: Multi-layer images fail due to tmpfs whiteouts, PMIx mount errors for non-PMIx jobs, and NVIDIA hook EPERM (`nvidia-persistenced/socket`) — GPU unusable unless hooks disabled (OCI backend idea optional)

[narcotis]

Description

When launching containers via Pyxis + Enroot in a Slinky/Kubernetes-managed Slurm cluster,
we encountered three issues:

---

1) Multi-layer images (e.g., NVCR Triton) fail during whiteout conversion

Even after setting:

ENROOT_CACHE_PATH=/enroot-tmp/cache
ENROOT_DATA_PATH=/enroot-tmp/data
ENROOT_RUNTIME_PATH=/enroot-tmp/run
ENROOT_TEMP_PATH=/enroot-tmp/tmp

in the enroot.conf. /enroot-tmp is a emptyDir from Node Disk.

multi-layer images fail during AUFS → overlayfs whiteout conversion if Enroot internally uses /tmp:

enroot-aufs2ovlfs: failed to create opaque ovlfs whiteout:
/tmp/enroot.<id>/17/usr/src/python3.12/: Not supported

/tmp in the Slurm worker pods is an overlayfs/tmpfs, which cannot store overlay whiteouts/xattrs.
Simple images like alpine/ubuntu succeed; NVCR-based multi-layer images fail.

2) PMIx hook (50-slurm-pmi.sh) fails on non-MPI jobs

Slurm only creates PMIx directories when --mpi=pmix is used:

/var/spool/slurmd/pmix.<jobid>.<stepid>
/tmp/spmix_appdir_<uid>_<jobid>.<stepid>

But Pyxis’s PMIx hook always attempts to bind-mount these directories:

enroot-mount: failed to mount: /tmp/spmix_appdir_*: No such file or directory

→ All non-MPI jobs fail unless the PMIx hook is disabled.

With --mpi=pmix, PMIx works correctly.

3) NVIDIA hook (98-nvidia.sh) fails — GPU cannot be used at all

When GPU hooks are enabled:

nvidia-container-cli: mount error: mount operation failed:
/enroot-tmp/data/pyxis_<job>/run/nvidia-persistenced/socket: operation not permitted

Causing container startup to fail:

pyxis: couldn't start container
spank_pyxis.so: task_init() failed

If the hook is disabled, containers start — but no GPU is usable:

• /dev/nvidia* devices appear, BUT:
  • nvidia-smi is not injected
  • NVIDIA driver libraries (libcuda.so, libnvidia-ml.so, etc.) are not mounted
  • torch.cuda.is_available() → False

Because NVCR images rely on NVIDIA Container Toolkit to inject host GPUs and driver stack,
disabling the hook makes the container functional but CPU-only.

Steps to Reproduce

Environment

• Slinky-managed Slurm in Kubernetes
• Pyxis + Enroot 4.0.1
• NVIDIA H100 nodes (NVRM 580.95.05, CUDA 13.0)
• GPU plugin + NVIDIA toolkit installed
• ENROOT directories on local NVMe (/enroot-tmp)
• --mpi=pmix tested and verified
  (This is from my custom Docker Image which is built from login-pyxis:25.11-ubuntu24.04 and slurmd-pyxis.25.11-ubuntu24.04.)

1) Whiteout failure

srun --container-image <multi-layer-image> bash

2) PMIx hook failure

srun --container-image <any-image> bash # without --mpi=pmix

3) NVIDIA hook failure

srun --gpus 2 --mpi=pmix
--container-image <nvcr-image> bash

---

Expected Behavior

1. Enroot should honor ENROOT_TEMP_PATH fully and avoid /tmp on tmpfs/overlayfs when extracting layers.
2. PMIx hook should only run when the job actually uses PMIx (--mpi=pmix, or detect via environment).
3. NVIDIA hook should succeed in injecting:
   • /dev/nvidia*
   • nvidia-smi
   • driver libraries (libcuda.so, etc.)
   • /run/nvidia-persistenced/socket
     without EPERM errors.

Overall expectation:

GPU-enabled containers should start correctly under Pyxis + Enroot without disabling core hooks.

---

Additional Context

A) tmpfs/overlayfs whiteout limitations

Pyxis still directs Enroot to use /tmp in some stages of import, even when ENROOT_TEMP_PATH is set.
On Kubernetes worker pods, /tmp is overlayfs → whiteouts fail → multi-layer images cannot be imported.

B) PMIx directory creation behavior

Slurm only generates PMIx directories for jobs using --mpi=pmix.
Hook should skip PMIx mounts for non-MPI jobs to avoid failures.

C) NVIDIA hook mount EPERM

Pyxis runs Enroot inside a user namespace; nvidia-container-cli configure requires privileged bind-mounts into the Enroot rootfs.
This fails with EPERM even when the Slurm worker pod is privileged: true.

D) GPU works only if hook is disabled

Disabling 98-nvidia.sh allows containers to start (including multi-layer NVCR images),
but GPU libraries are missing → CUDA unavailable inside container.

---

Optional Suggestion: Support OCI backend (rootless docker/podman)

Optional — not required for resolving the bug.

If other OCI backends using rootless Docker or Podman, several issues might be avoided:
(via oci.conf + Authtype=auth/slurm). From the slurm documentation, it says that Authtype=auth/munge only supports rootless Docker or Podman.

• OCI runtimes already handle:
  • driver injection
  • mount permissions
  • whiteout semantics
• Would reduce the need for Pyxis to perform privileged mount operations inside user namespaces
• Could avoid tmpfs whiteout problems entirely
• Might eliminate the need for custom Enroot NVIDIA hooks

Not required, but may offer a clean long-term architectural path.

---

If more logs (Pyxis debug, Enroot mount trace, NVIDIA Toolkit debug) would be helpful,
I can provide them.

[narcotis]

This is my slurm.conf from helm values:

  cgroup.conf: |
    CgroupPlugin=cgroup/v2
    IgnoreSystemd=yes
    EnableControllers=yes
    ConstrainCores=yes
    ConstrainRAMSpace=yes
    ConstrainDevices=yes
    ConstrainSwapSpace=yes

  gres.conf: |
    AutoDetect=nvidia
  
  plugstack.conf: |
    include /usr/share/pyxis/*
  
  job_container.conf: |
    AutoBasePath=true
    EntireStepInNS=true
  # Ref: https://slurm.schedmd.com/mpi.conf.html
  mpi.conf: |
    PMIxDebug=0
  # Ref: https://slurm.schedmd.com/oci.conf.html
  oci.conf: |
    FileDebug=debug2

And this is the securityContext from login pod:

securityContext:
      privileged: true

And this is from worker pod:

    securityContext:
       capabilities:
        add:
        - BPF
        - NET_ADMIN
        - SYS_ADMIN
        - SYS_NICE
      privileged: true

[vivian-hafener]

Good afternoon @narcotis . I am going to tackle the three issues you present here in serial instead of in parallel.

Even after setting:

ENROOT_CACHE_PATH=/enroot-tmp/cache
ENROOT_DATA_PATH=/enroot-tmp/data
ENROOT_RUNTIME_PATH=/enroot-tmp/run
ENROOT_TEMP_PATH=/enroot-tmp/tmp

in the enroot.conf. /enroot-tmp is a emptyDir from Node Disk.

Where and how are you setting these?

As the SlinkyProject's Pyxis container images install Enroot and set up it's configuration, it is recommended that the following process be used to override enroot.conf with your own values: https://slinky.schedmd.com/projects/slurm-operator/en/release-1.0/usage/override-config-file.html

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener ,

I had previously bundled enroot.conf directly inside the Docker image during the dockerization process, and for testing/debugging I also manually updated the values on both the login node and the nodeset. I confirmed that the configuration values were being applied correctly in that setup.

The reason I initially took this approach is because we needed to provide isolated Enroot environments per user. Since /home is on NFS and /tmp is backed by tmpfs, the usual directory settings didn’t behave correctly in our environment, so I configured Enroot explicitly during image build and node setup.

That said, the method you suggested — overriding the config using Slinky/Pyxis’ recommended mechanism — is definitely a cleaner and more robust approach. I agree that this is the better way, and I’ll make sure to adopt this method going forward.

Thanks for the guidance!

Best,
Narcotis

[vivian-hafener]

Good morning @narcotis,

1. Enroot Overlayfs Whiteout Error

I believe that this is a configuration issue or an issue with enroot itself. Aside from the configuration files that are baked in to the published Slinky images, which I have provided a means to override, Slurm-operator should not interfere with the operator of enroot itself once a job allocation has been made. I would encourage you to double-check that your configurations are being picked up properly by enroot at runtime. If that does not resolve your issues, consider opening an issue with the enroot project.

2. PMIx Hook Failure

Slurm only creates PMIx directories when --mpi=pmix is used:

/var/spool/slurmd/pmix.<jobid>.<stepid>
/tmp/spmix_appdir_<uid>_<jobid>.<stepid>

But Pyxis’s PMIx hook always attempts to bind-mount these directories:

enroot-mount: failed to mount: /tmp/spmix_appdir_*: No such file or directory

→ All non-MPI jobs fail unless the PMIx hook is disabled.

This is an issue with either your configuration or enroot, and is not an issue with Slurm. Slurm should only create PMIx directories when --mpi=pmix is used. To do otherwise would add delay to job launch and increase filesystem churn for sites that do not use pmix.

3. NVIDIA hook failure

nvidia-container-cli: mount error: mount operation failed:
/enroot-tmp/data/pyxis_<job>/run/nvidia-persistenced/socket: operation not permitted

This error indicates that a mount error has occurred, likely due to filesystem permissions. I would confirm that the SlurmdUser, and the user you are running the job as both have proper permissions within that filesystem path. This seems to be a configuration issue, and not an issue with how Slurm-operator invokes enroot.

Are you using NVIDIA's GPU Operator for driver injection, or are you using the container toolkit separately from that? I have had success running Slurm-operator with the GPU Operator in the past.

---

Enroot and its NVIDIA hook are maintained by the enroot project, which is a distinct and separate entity from the Slinky project. Issues with enroot tooling that are not specifically due to interactions with Slinky components should be directed towards the enroot project, via their GitHub repo.

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener ,

Thanks for the detailed explanation — that helps a lot.
I understand point #1 now.

However, I still have some questions regarding #2 and #3, as there are a few parts that seem unclear from my side.

---

2. PMIx hook behavior

For this issue, I did not add any custom configuration on my side.
I am using the hooks exactly as they are provided in the Slinky *-pyxis image.

From the Slurm configuration side, I do not have any MPI-related settings enabled:

MpiDefault = (null)
MpiParams  = (null)

Given this, it seems that enroot is always expecting PMIx-related paths to exist, regardless of whether --mpi=pmix is actually used.

Specifically, the behavior appears to come from the enroot hook defined in:

/etc/enroot/hooks.d/50-slurm-pmi.sh

Since this hook attempts to bind-mount PMIx directories unconditionally, would it be reasonable to adjust this hook in the Slinky-provided container image so that it only runs when PMIx is actually enabled?

From my perspective, this feels less like a site-specific misconfiguration and more like a default hook behavior that may need to be guarded for non-MPI jobs.

---

3. NVIDIA hook and permission issue

Similarly for the NVIDIA hook, I am not doing any custom modification in the container image.

I am not using NVIDIA GPU Operator.
Instead, GPU support is provided via the standard nvidia-device-plugin DaemonSet, and GPU allocation itself works correctly for normal workloads.

The permission error only occurs when enroot is involved:

nvidia-container-cli: mount error:
.../run/nvidia-persistenced/socket: operation not permitted

The cluster uses OpenLDAP integration, with users authenticated via sssd and pam, so jobs are running as real Linux users.
Given that GPU workloads run correctly outside of enroot, I am trying to understand why this permission issue only manifests within enroot.

Could this be related to the behavior of the NVIDIA enroot hook here?

/etc/enroot/hooks.d/98-nvidia.sh

I am wondering whether the hook assumes certain filesystem permissions or runtime conditions that may not hold in this setup, even though no explicit customization was made on our side.

---

Overall, my main question is whether these hooks are expected to work out-of-the-box for non-MPI jobs and for setups using nvidia-device-plugin, or if some adjustment in the Slinky-provided enroot hooks is required in these cases.

Thanks again for your help.

Best regards,
Narco

[vivian-hafener]

Good morning @narcotis,

Overall, my main question is whether these hooks are expected to work out-of-the-box for non-MPI jobs and for setups using nvidia-device-plugin, or if some adjustment in the Slinky-provided enroot hooks is required in these cases.

These hooks are expected to work out-of-the-box, without modification to the enroot hooks.

I have gotten around to testing this on a cluster with H100s, which sounds like it matches your configuration. Testing each reproducer in order there:

1. Whiteout

root@slurm-login-pyxis-58fdbb84df-gzlqr:/tmp# srun --container-image=alpine:latest grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:latest
pyxis: imported docker image: alpine:latest
PRETTY_NAME="Alpine Linux v3.23"

2. PMIx hook

root@slurm-login-pyxis-58fdbb84df-gzlqr:/tmp# srun --container-image=alpine:latest grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:latest
pyxis: imported docker image: alpine:latest
PRETTY_NAME="Alpine Linux v3.23"

3. NVIDIA hook

root@slurm-login-pyxis-58fdbb84df-gzlqr:/tmp# srun --gpus 2 --mpi=pmix --container-image=alpine:latest grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:latest
pyxis: imported docker image: alpine:latest
PRETTY_NAME="Alpine Linux v3.23"

Best regards,
Vivian Hafener

[vivian-hafener]

Hey there @narcotis,

I have been looking into this further. Could you try running the following and sending me the output, once inside a Slurm allocation (using srun for each command) and once on the baremetal nodes of your cluster? This may help me identify what in your environment is resulting in those issues:

$ curl -fSsL -O https://github.com/NVIDIA/enroot/releases/download/v4.0.1/enroot-check_4.0.1_$(uname -m).run
$ chmod +x enroot-check_*.run

$ ./enroot-check_*.run --verify

$ ./enroot-check_*.run
Bundle ran successfully!

In the meantime, I will continue investigating this from my side of things.

Best,
Vivian Hafener

[narcotis]

Hi @vivian-hafener ,

Thanks for the suggestion. I tried the command you shared.

When I ran it as a regular Linux user, I got the following error:

[ERROR] Could not find kernel configuration

However, when running the same command as root, it worked as expected:

Extracting [####################] 100%
Bundle ran successfully!

So, as you mentioned earlier, this does look like a permission-related issue.

In my setup, users SSH into a login pod and submit Slurm jobs using srun. Given this workflow, I’m starting to wonder whether enroot is expected to work correctly when invoked via srun in a non-root (rootless) context, or if there are known limitations in this scenario.

I haven’t had a chance to dig much deeper yet due to other ongoing work, but my end goal is to allow users to run OCI-compatible Slurm jobs in a rootless way—either using enroot or an equivalent mechanism—without requiring root privileges.

If there’s any additional information or debugging details that would be helpful on my side, please feel free to let me know.

Thanks!

Best,
Narcotis

[vivian-hafener]

Hi @narcotis,

I believe that this resolves the issue that you were seeing: NVIDIA/nvidia-container-toolkit#1593

Best,
Vivian hafener