Summary
When nvidia-container-cli configure fails for any reason, its cleanup does not only unmount what it mounted — it also deletes the mount-target files: the fail path in nvc_driver_mount (nvc_mount.c#L800-L804) calls unmount() (nvc_mount.c#L509-L516), which is umount2() followed by file_remove(), and file_remove unlinks every regular file with st_size == 0 (utils.c#L606-L638) — i.e. exactly the placeholder files file_create makes as bind-mount targets.
On a single machine this is safe: unlinking a file that is a mountpoint in another mount namespace fails with EBUSY. But when the container rootfs lives on a shared network filesystem used by several nodes at once (Slurm + enroot-style setups: each node runs configure against the shared rootfs and bind-mounts its node-local driver files over the shared placeholders), the failing node's kernel knows nothing about the other nodes' mounts. The unlinks go through to the server, the other nodes' kernels notice the deletions on the next revalidation and detach their live bind mounts. One failing instance silently breaks GPU userspace (libnvidia-ml.so.1: cannot open shared object file, file too short) on every other node using that rootfs, and nothing on those nodes re-runs configure.
Observed in production (Slurm cluster, shared rootfs on a FUSE-based filesystem, driver 580.159.04, libnvidia-container v1.18.2): two nodes cold-start concurrently; one configure fails; its cleanup unlinks ~55 lib/bin placeholders; the node whose configure succeeded loses all its driver bind mounts seconds later (mountinfo count 9 → 0) and stays broken until pod restart.
Suggested fix
Don't delete mount targets in the failure path — keep the umount2(), drop the file_remove(). The leftover 0-byte placeholders are harmless and actually useful: a retry of configure skips and reuses them (file_create's existing-path handling), and on success they are invisible under the mounts. If leaving placeholders behind after a permanent failure is considered undesirable for private rootfs, an opt-out (e.g. --keep-mount-targets, or a config option) for shared-rootfs deployments would solve it as well.
Note that tracking ownership ("only delete files this run created") would not fix the shared case: concurrent cold-start instances interleave creation, so a failing instance legitimately owns files that other instances have already mounted over.
Happy to send a PR for whichever direction you prefer.
Summary
When
nvidia-container-cli configurefails for any reason, its cleanup does not only unmount what it mounted — it also deletes the mount-target files: the fail path innvc_driver_mount(nvc_mount.c#L800-L804) callsunmount()(nvc_mount.c#L509-L516), which isumount2()followed byfile_remove(), andfile_removeunlinks every regular file withst_size == 0(utils.c#L606-L638) — i.e. exactly the placeholder filesfile_createmakes as bind-mount targets.On a single machine this is safe: unlinking a file that is a mountpoint in another mount namespace fails with EBUSY. But when the container rootfs lives on a shared network filesystem used by several nodes at once (Slurm + enroot-style setups: each node runs
configureagainst the shared rootfs and bind-mounts its node-local driver files over the shared placeholders), the failing node's kernel knows nothing about the other nodes' mounts. The unlinks go through to the server, the other nodes' kernels notice the deletions on the next revalidation and detach their live bind mounts. One failing instance silently breaks GPU userspace (libnvidia-ml.so.1: cannot open shared object file,file too short) on every other node using that rootfs, and nothing on those nodes re-runsconfigure.Observed in production (Slurm cluster, shared rootfs on a FUSE-based filesystem, driver 580.159.04, libnvidia-container v1.18.2): two nodes cold-start concurrently; one
configurefails; its cleanup unlinks ~55 lib/bin placeholders; the node whoseconfiguresucceeded loses all its driver bind mounts seconds later (mountinfo count 9 → 0) and stays broken until pod restart.Suggested fix
Don't delete mount targets in the failure path — keep the
umount2(), drop thefile_remove(). The leftover 0-byte placeholders are harmless and actually useful: a retry ofconfigureskips and reuses them (file_create's existing-path handling), and on success they are invisible under the mounts. If leaving placeholders behind after a permanent failure is considered undesirable for private rootfs, an opt-out (e.g.--keep-mount-targets, or a config option) for shared-rootfs deployments would solve it as well.Note that tracking ownership ("only delete files this run created") would not fix the shared case: concurrent cold-start instances interleave creation, so a failing instance legitimately owns files that other instances have already mounted over.
Happy to send a PR for whichever direction you prefer.