Job environment¶

Every srun and sbatch step runs inside the worker's slurmd pod, directly in that pod's Ubuntu 24.04 userspace and as your own user (the same UID as in the login pod). The pod image is identical on every worker, so a job behaves the same whichever node it lands on.

$ srun bash -c 'hostname; id -un; grep PRETTY /etc/os-release'
slinky-0
ubuntu
PRETTY_NAME="Ubuntu 24.04.4 LTS"

The shared jail is gone

Earlier Slinky clusters ran every step inside a shared chroot jail (VERDA_SHARED_JAIL=1). The jail is retired: steps run in the worker pod, and Pyxis (srun --container-image) works again. See Containers.

Storage inside a job¶

Path	What it is
`/home`	The cluster's shared filesystem, the same on the login pod, the login node and every worker. Keep job scripts, environments and SIF files here.
`/shared`	A world-writable dataset area on the same shared filesystem.
`/local`	The worker's local NVMe. Persists across jobs and node restarts, so a re-run on the same node finds its data again.
`/tmp`	Also on the worker's NVMe, and shared by all jobs on that node. It is not private to your job and is not emptied when the job ends, so clean up after yourself and keep anything valuable on `/local` or `/home`. Files untouched for 30 days are deleted.
`/<SFS_NAME>`	Any extra shared filesystem you attached when creating the cluster. Mounted in the login pod and inside jobs, at the same path as on the login node.

Extra shared filesystems in Pyxis containers

A Pyxis container gets $HOME, /shared and /local by default. To use an extra shared filesystem inside the container, pass it with --container-mounts=/mnt/<SFS_NAME>:/mnt/<SFS_NAME>. See Containers.

GPUs¶

GPUs appear only when you request them. The job's cgroup fences off everything else, so a step with no --gpus (or --gres=gpu:N) sees no GPU at all:

$ srun nvidia-smi -L
No devices found.
$ srun --gpus=1 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)

Size --gpus to what each step needs. CUDA_VISIBLE_DEVICES is set to the GPUs Slurm gave you.

The NVIDIA driver userspace (nvidia-smi, libnvidia-*.so) is injected into the worker pod from the host by the NVIDIA Container Toolkit. Neither the login node nor the login pod has a GPU: run nvidia-smi under srun --gpus=N.

Software in the worker image¶

Python 3.12 with venv and pip, the same version as in the login pod.
CUDA 13.4 toolkit at /usr/local/cuda. It is not on PATH; use /usr/local/cuda/bin/nvcc or add /usr/local/cuda/bin yourself.
Open MPI 4.1 (mpirun) from Ubuntu.
Containers: Pyxis + Enroot and Apptainer, see Containers. Enroot is installed only on the workers: the login pod has none, so run enroot import through srun.

HPC-X and Lmod (module load) are not available inside jobs. For an MPI or NCCL stack, use a container image that ships one (for example an NGC image) or install it under /home.

Python environments¶

Known issue: use containers, not virtual environments on /home

A job can hang when it first reads files that were just written to the shared /home (or /shared) from the login pod or another node. The process cannot be killed and the node drains with Kill task failed. Python virtual environments are the usual victim: python3 -m venv, pip and uv write thousands of small files that the next job then reads. We are working on a fix.

Until then, run Python workloads in a container with Pyxis, with your dependencies in the image:

srun --gpus=8 --container-image=nvcr.io#nvidia/pytorch:25.10-py3 \
     --container-name=pytorch-25.10 --container-workdir="$PWD" \
     python train.py

Do not pip install inside the container: pip puts packages in ~/.local on /home, which has the same problem. Use an image that already contains what you need, or build one with your dependencies and push it to a registry.

The worker pods and the login pod run the same Ubuntu 24.04 and Python 3.12. The login node (port 2222) runs Ubuntu 26.04 with a newer Python, so environments created there do not work inside jobs.