---
description: How the shared execution jail works on Slinky clusters and what is visible inside it.
revision_date: 27.07.2026
---

# The shared execution jail

On this cluster every `srun` and `sbatch` step runs inside a **per-job jail** — a `chroot` / `pivot_root` sandbox that Slurm sets up for the step on each worker node. This is the defining trait of the cluster: your job never runs directly on the worker's root filesystem, but in an isolated, consistent Ubuntu userspace that looks the same on every node.

You can confirm you are inside the jail from any step — the `VERDA_SHARED_JAIL` environment variable is set to `1`:

```console
$ srun bash -c 'echo $VERDA_SHARED_JAIL'
1
```

## Why it exists

The jail gives you two things at once:

* **Process isolation** — each step runs in its own PID and mount namespace with a private `/tmp`, so steps cannot see or disturb each other's processes on a shared worker.
* **A consistent userspace** — the same Ubuntu environment, tools and library paths are presented on every worker node, so a job behaves identically whether it lands on `slinky-0` or `slinky-1`.

!!! warning "The jail root is shared and writable"
    Every job on every node pivots into the same root filesystem at `/home/.slinky-jail/rootfs`. It is not per-job and not an overlay, and steps run as `root` — so writing outside `/home`, `/tmp` or `/local` (`apt install`, `pip install` into system paths) changes the environment for every other job on the cluster and persists after yours ends. Keep changes in `/home`, a virtualenv, or a container image.

## What is visible inside the jail

The jail is not empty — the pieces you need for real workloads are bind-mounted in:

* **Your `/home`** is shared into the jail. Home directories and Python virtualenvs work exactly as they do on the login node — install once under `/home`, use it from every job.
* **CUDA** is available at `/usr/local/cuda`.
* **HPC-X** is available at `/opt/hpcx`.
* **Scratch**: `/shared` is the dataset area on the shared filesystem, `/local` is the worker's NVMe and persists across jobs on that node, and `/tmp` is a private per-job directory on the same NVMe, emptied when the job ends.

```bash
srun bash -c 'ls -d /usr/local/cuda /opt/hpcx'
```

GPUs are the exception: they appear **only when you request them**. A step with no `--gpus` (or `--gres=gpu:N`) sees no GPUs at all, because the job's cgroup fences them off:

```console
$ srun bash -c 'nvidia-smi -L'
No devices found.
```

Request GPUs and exactly that many become visible inside the jail:

```console
$ srun --gpus=1 bash -c 'nvidia-smi -L'
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
```

!!! tip
    Always size `--gpus` (or `--gres=gpu:N`) to what your step actually needs. Anything you do not request is invisible to the step.

## Implications

* **Containers work.** [Apptainer / Singularity](https://docs.verda.com/clusters/instant-clusters/slinky/containers/) and Enroot are available inside the jail, including GPU passthrough with `apptainer exec --nv` for the GPUs you requested. See [Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/) for usage.
* **Pyxis is off.** Because it is mutually exclusive with the jail, `srun --container-image=...` (pyxis) is **not** available — use Apptainer instead.
* **No in-job Docker daemon.** The `docker` client binary exists, but there is no Docker daemon running inside a job, so `docker run` will not work. Use Apptainer or Enroot to run container images.
* **Lmod and HPC-X modules work.** [Lmod](https://lmod.readthedocs.io/) is present; in a login shell (`bash -lc`) `module avail` lists the HPC-X modulefiles and `module load hpcx` works:

    ```bash
    srun bash -lc 'module load hpcx && echo loaded'
    ```

## Where the NVIDIA userspace comes from

Inside a step, the NVIDIA driver userspace (`nvidia-smi`, `libnvidia-*.so`) comes from packages in the jail image — `dpkg -S /usr/bin/nvidia-smi` reports `libnvidia-compute`:

```console
$ srun which nvidia-smi
/usr/bin/nvidia-smi
```

The [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html) injection happens one level out, in the `slurmd` worker pod, where `nvidia-smi` and the `libnvidia-*.so.<driver>` libraries are read-only bind-mounts from the host. You will not see those bind-mounts from inside a step: `mount | grep nvidia` shows only the persistenced socket and a toolkit hook.

Neither the login node nor the login pod has a GPU. The login pod has no NVIDIA userspace at all; the login node has `nvidia-smi` installed but no driver behind it, so it exits with `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`. Run it under `srun --gpus=N` instead.