The shared execution jail¶

On this cluster every srun and sbatch step runs inside a per-job jail — a chroot / pivot_root sandbox that Slurm sets up for the step on each worker node. This is the defining trait of the cluster: your job never runs directly on the worker's root filesystem, but in an isolated, consistent Ubuntu userspace that looks the same on every node.

You can confirm you are inside the jail from any step — the VERDA_SHARED_JAIL environment variable is set to 1:

$ srun bash -c 'echo $VERDA_SHARED_JAIL'
1

Why it exists¶

The jail gives you two things at once:

Process isolation — each step runs in its own PID and mount namespace with a private /tmp, so steps cannot see or disturb each other's processes on a shared worker.
A consistent userspace — the same Ubuntu environment, tools and library paths are presented on every worker node, so a job behaves identically whether it lands on slinky-0 or slinky-1.

The jail root is shared and writable

Every job on every node pivots into the same root filesystem at /home/.slinky-jail/rootfs. It is not per-job and not an overlay, and steps run as root — so writing outside /home, /tmp or /local (apt install, pip install into system paths) changes the environment for every other job on the cluster and persists after yours ends. Keep changes in /home, a virtualenv, or a container image.

What is visible inside the jail¶

The jail is not empty — the pieces you need for real workloads are bind-mounted in:

Your /home is shared into the jail. Home directories and Python virtualenvs work exactly as they do on the login node — install once under /home, use it from every job.
CUDA is available at /usr/local/cuda.
HPC-X is available at /opt/hpcx.
Scratch: /shared is the dataset area on the shared filesystem, /local is the worker's NVMe and persists across jobs on that node, and /tmp is a private per-job directory on the same NVMe, emptied when the job ends.

srun bash -c 'ls -d /usr/local/cuda /opt/hpcx'

GPUs are the exception: they appear only when you request them. A step with no --gpus (or --gres=gpu:N) sees no GPUs at all, because the job's cgroup fences them off:

$ srun bash -c 'nvidia-smi -L'
No devices found.

Request GPUs and exactly that many become visible inside the jail:

$ srun --gpus=1 bash -c 'nvidia-smi -L'
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)

Tip

Always size --gpus (or --gres=gpu:N) to what your step actually needs. Anything you do not request is invisible to the step.

Implications¶

Containers work. Apptainer / Singularity and Enroot are available inside the jail, including GPU passthrough with apptainer exec --nv for the GPUs you requested. See Containers for usage.
Pyxis is off. Because it is mutually exclusive with the jail, srun --container-image=... (pyxis) is not available — use Apptainer instead.
No in-job Docker daemon. The docker client binary exists, but there is no Docker daemon running inside a job, so docker run will not work. Use Apptainer or Enroot to run container images.
Lmod and HPC-X modules work. Lmod is present; in a login shell (bash -lc) module avail lists the HPC-X modulefiles and module load hpcx works:
```
srun bash -lc 'module load hpcx && echo loaded'
```

Where the NVIDIA userspace comes from¶

Inside a step, the NVIDIA driver userspace (nvidia-smi, libnvidia-*.so) comes from packages in the jail image — dpkg -S /usr/bin/nvidia-smi reports libnvidia-compute:

$ srun which nvidia-smi
/usr/bin/nvidia-smi

The NVIDIA Container Toolkit injection happens one level out, in the slurmd worker pod, where nvidia-smi and the libnvidia-*.so.<driver> libraries are read-only bind-mounts from the host. You will not see those bind-mounts from inside a step: mount | grep nvidia shows only the persistenced socket and a toolkit hook.

Neither the login node nor the login pod has a GPU. The login pod has no NVIDIA userspace at all; the login node has nvidia-smi installed but no driver behind it, so it exits with NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Run it under srun --gpus=N instead.