The shared execution jail¶

On this cluster every srun and sbatch step runs inside a per-job jail — a chroot / pivot_root sandbox that Slurm sets up for the step on each worker node. This is the defining trait of the cluster: your job never runs directly on the worker's root filesystem, but in an isolated, consistent Ubuntu userspace that looks the same on every node.

You can confirm you are inside the jail from any step — the VERDA_SHARED_JAIL environment variable is set to 1:

$ srun bash -c 'echo $VERDA_SHARED_JAIL'
1

Why it exists¶

The jail gives you two things at once:

• Isolation — each job gets its own root, so steps cannot see or disturb each other's processes and mounts on a shared worker.
• A consistent userspace — the same Ubuntu environment, tools and library paths are presented on every worker node, so a job behaves identically whether it lands on slinky-0 or slinky-1.

What is visible inside the jail¶

The jail is not empty — the pieces you need for real workloads are bind-mounted in:

• Your /home is shared into the jail. Home directories, Python virtualenvs and Spack environments all work exactly as they do on the login node — install once under /home, use it from every job.
• CUDA is available at /usr/local/cuda.
• HPC-X is available at /opt/hpcx.
• Scratch: /shared is the dataset area on the shared filesystem, /local is the worker's NVMe and persists across jobs on that node, and /tmp is a private per-job directory on the same NVMe, emptied when the job ends.

srun bash -c 'ls -d /usr/local/cuda /opt/hpcx'

GPUs are the exception: they appear only when you request them. A step with no --gpus (or --gres=gpu:N) sees no GPUs at all, because the job's cgroup fences them off:

$ srun bash -c 'nvidia-smi -L'
No devices found.

Request GPUs and exactly that many become visible inside the jail:

$ srun --gpus=1 bash -c 'nvidia-smi -L'
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)

Implications¶

• Containers work. Apptainer / Singularity and Enroot are available inside the jail, including GPU passthrough with apptainer exec --nv for the GPUs you requested. See Containers for usage.
• Pyxis is off. Because it is mutually exclusive with the jail, srun --container-image=... (pyxis) is not available — use Apptainer instead.
• No in-job Docker daemon. The docker client binary exists, but there is no Docker daemon running inside a job, so docker run will not work. Use Apptainer or Enroot to run container images.

• Lmod and HPC-X modules work. Lmod is present; in a login shell (bash -lc) module avail lists the HPC-X modulefiles and module load hpcx works:

  srun bash -lc 'module load hpcx && echo loaded'
  

Where the NVIDIA userspace comes from¶

Inside a step, the NVIDIA driver userspace (nvidia-smi, libnvidia-*.so) is injected by the NVIDIA Container Toolkit: the binaries and libraries are bind-mounted from the host, not installed from packages. So dpkg -l shows no nvidia-* entries even though the tools work:

$ srun which nvidia-smi
/usr/bin/nvidia-smi

You can see the bind-mounts with mount | grep nvidia inside a step. The login pod has no GPUs attached and therefore no injected NVIDIA binaries — nvidia-smi is only available under srun on a worker.