---
description: Where Slurm job steps run on a Slinky cluster, what software and storage they see, and how GPUs are fenced.
revision_date: 29.10.2026
---

# Job environment

Every `srun` and `sbatch` step runs inside the worker's `slurmd` pod, directly in that pod's Ubuntu 24.04 userspace and as **your own user** (the same UID as in the login pod). The pod image is identical on every worker, so a job behaves the same whichever node it lands on.

```console
$ srun bash -c 'hostname; id -un; grep PRETTY /etc/os-release'
slinky-0
ubuntu
PRETTY_NAME="Ubuntu 24.04.4 LTS"
```

!!! info "The shared jail is gone"
    Earlier Slinky clusters ran every step inside a shared `chroot` jail (`VERDA_SHARED_JAIL=1`). The jail is retired: steps run in the worker pod, and Pyxis (`srun --container-image`) works again. See [Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/).

## Storage inside a job

| Path | What it is |
|------|------------|
| `/home` | The cluster's shared filesystem, the same on the login pod, the login node and every worker. Keep job scripts, environments and SIF files here. |
| `/shared` | A world-writable dataset area on the same shared filesystem. |
| `/local` | The worker's local NVMe. Persists across jobs and node restarts, so a re-run on the same node finds its data again. |
| `/tmp` | Also on the worker's NVMe, and **shared by all jobs on that node**. It is not private to your job and is not emptied when the job ends, so clean up after yourself and keep anything valuable on `/local` or `/home`. Files untouched for 30 days are deleted. |
| `/<SFS_NAME>` | Any extra [shared filesystem](https://docs.verda.com/storage/shared-filesystems-sfs/shared-filesystem-for-a-cluster/) you attached when creating the cluster. Mounted in the login pod and inside jobs, at the same path as on the login node. |

!!! info "Extra shared filesystems in Pyxis containers"
    A Pyxis container gets `$HOME`, `/shared` and `/local` by default. To use an extra shared filesystem inside the container, pass it with `--container-mounts=/mnt/<SFS_NAME>:/mnt/<SFS_NAME>`. See [Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/).

## GPUs

GPUs appear **only when you request them**. The job's cgroup fences off everything else, so a step with no `--gpus` (or `--gres=gpu:N`) sees no GPU at all:

```console
$ srun nvidia-smi -L
No devices found.
$ srun --gpus=1 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
```

Size `--gpus` to what each step needs. `CUDA_VISIBLE_DEVICES` is set to the GPUs Slurm gave you.

The NVIDIA driver userspace (`nvidia-smi`, `libnvidia-*.so`) is injected into the worker pod from the host by the NVIDIA Container Toolkit. Neither the login node nor the login pod has a GPU: run `nvidia-smi` under `srun --gpus=N`.

## Software in the worker image

* **Python 3.12** with `venv` and `pip`, the same version as in the login pod.
* **CUDA 13.4 toolkit** at `/usr/local/cuda`. It is not on `PATH`; use `/usr/local/cuda/bin/nvcc` or add `/usr/local/cuda/bin` yourself.
* **Open MPI 4.1** (`mpirun`) from Ubuntu.
* **Containers**: Pyxis + Enroot and Apptainer, see [Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/). Enroot is installed only on the workers: the login pod has none, so run `enroot import` through `srun`.

**HPC-X and Lmod (`module load`) are not available** inside jobs. For an MPI or NCCL stack, use a container image that ships one (for example an NGC image) or install it under `/home`.

## Python environments

!!! warning "Known issue: use containers, not virtual environments on `/home`"
    A job can hang when it first reads files that were just written to the shared `/home` (or `/shared`) from the login pod or another node. The process cannot be killed and the node drains with `Kill task failed`. Python virtual environments are the usual victim: `python3 -m venv`, `pip` and `uv` write thousands of small files that the next job then reads. We are working on a fix.

    Until then, run Python workloads in a container with [Pyxis](https://docs.verda.com/clusters/instant-clusters/slinky/containers/), with your dependencies in the image:

    ```bash
    srun --gpus=8 --container-image=nvcr.io#nvidia/pytorch:25.10-py3 \
         --container-name=pytorch-25.10 --container-workdir="$PWD" \
         python train.py
    ```

    Do not `pip install` inside the container: pip puts packages in `~/.local` on `/home`, which has the same problem. Use an image that already contains what you need, or build one with your dependencies and push it to a registry.

The worker pods and the login pod run the same Ubuntu 24.04 and Python 3.12. The login node (port `2222`) runs Ubuntu 26.04 with a newer Python, so environments created there do not work inside jobs.