Skip to content

Containers

A Slinky cluster runs containers in two ways, both from a Slurm step and both as your own user:

  • Pyxis + Enroot: add --container-image to srun. This is the usual choice for multi-node GPU training.
  • Apptainer (formerly Singularity): run apptainer exec inside a step, for example with a pre-built SIF file.

The cluster ships Pyxis 0.24, Enroot 4.2 and Apptainer 1.5.5.

Pyxis: srun --container-image

$ srun --container-image=alpine:3 grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:3
pyxis: imported docker image: alpine:3
PRETTY_NAME="Alpine Linux v3.24"

Image names follow Enroot's syntax: Docker Hub images by name, other registries with # between the registry and the image, for example --container-image=nvcr.io#nvidia/pytorch:25.06-py3.

Without extra flags, the container gets:

  • your $HOME and the shared /shared directory, plus the worker's /local NVMe,
  • the GPUs you requested with --gpus (and only those),
  • the InfiniBand devices, so NCCL uses the fabric (NET/IB) from inside the container,
  • unlimited locked memory, which NCCL needs for InfiniBand.
$ srun --gpus=2 --container-image=nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
GPU 1: NVIDIA B300 SXM6 AC (UUID: GPU-...)

The container starts in the image's WORKDIR (/workspace for NGC images), not where you submitted. Pass --container-workdir="$PWD" to run your code from the current directory, and --container-mounts=SRC:DST to mount other paths. The full option list is in srun --help and the Pyxis documentation.

No enroot on the login pod

Enroot runs in the job step on the worker, not on the login pod you SSH into on port 22. Running enroot there prints the commands below and exits. Run enroot through srun, or on the login node (ssh -p 2222 ubuntu@<cluster-ip>), which has Enroot but no GPU.

On the login node, Enroot keeps its cache and unpacked containers in /var/lib/enroot on the node's own disk, not in /home. That directory has a quota of half the disk, at most 150 GiB, so imports cannot fill the node's disk; check usage with sudo repquota -P / 2>/dev/null. Cached layers unused for 14 days are deleted automatically. The .sqsh file from enroot import -o goes where you put it.

Import once, run many times

--container-image=<registry image> downloads and converts the image in every step. For large images (NGC PyTorch is about 16 GB), import it once into a squashfs file on the shared filesystem and start jobs from that file:

# Import once, inside a job
srun -p dev -c 32 enroot import -o /shared/pytorch-25.10.sqsh \
    docker://nvcr.io#nvidia/pytorch:25.10-py3

# Run from the file
srun --gpus=8 --container-image=/shared/pytorch-25.10.sqsh \
     --container-name=pytorch-25.10 --container-workdir="$PWD" \
     python train.py
  • --container-name keeps the unpacked image on the worker's NVMe, so later jobs on that node start in seconds instead of unpacking the .sqsh again. An existing name is reused as is, so put the image version in the name.
  • -p dev uses the short-job partition (8 h limit), which the scheduler considers before all. -c 32 only speeds up the squashfs build; the import also completes in a default-sized job.

Named containers stay on the node (one copy per user, name and node) until you remove them, see Clean up container storage.

Tip

Pick an image whose CUDA build matches the GPU. These are Blackwell B300 GPUs (compute_cap 10.3), so use CUDA 13.x and a recent framework build (e.g. a current PyTorch/NGC release). CUDA 12.x images predate B300 support and may fail to run kernels even though nvidia-smi lists the device.

Apptainer

Pull and run a public image directly. Apptainer converts the OCI image to a SIF on the fly:

$ srun apptainer exec docker://alpine:3 cat /etc/os-release
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.24"

singularity is an alias of apptainer, so either name works.

GPU containers

Request GPUs with --gpus=N on the step and pass --nv to Apptainer so the NVIDIA driver userspace is exposed inside the container. Exactly the requested number of GPUs appears:

$ srun --gpus=1 apptainer exec --nv docker://nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)

A step launched with no --gpus sees no GPUs: nvidia-smi -L prints No devices found. from inside the container.

Pull once, reuse many times

Converting an OCI image on every step is wasteful for large images. Pull to a SIF in the shared /home filesystem once, then point every step at that file:

# Pull once, inside a job
srun --time=00:10:00 bash -c '
  export APPTAINER_CACHEDIR=/local/apptainer-cache-$USER APPTAINER_TMPDIR=/local/apptainer-tmp-$USER
  mkdir -p "$APPTAINER_CACHEDIR" "$APPTAINER_TMPDIR"
  apptainer pull $HOME/cuda.sif docker://nvidia/cuda:13.0.0-base-ubuntu24.04'

# In any step: reuse the SIF (no re-download, no conversion)
srun --gpus=1 apptainer exec --nv $HOME/cuda.sif nvidia-smi -L

Because /home is shared, the same SIF is usable from any worker, including multi-node jobs.

Cache and temp directories

Set APPTAINER_CACHEDIR and APPTAINER_TMPDIR to node-local paths as above. The default cache lives under $HOME (~/.apptainer/cache), which puts multi-gigabyte layers on the shared filesystem. The cache is not reclaimed automatically, see Clean up container storage.

No Docker daemon

There is no Docker daemon in a job, so docker run does not work. Run Docker images with Pyxis (--container-image) or Apptainer (docker://...) instead.

Clean up container storage

Apart from the Enroot layer cache, nothing removes container images or caches for you. They stay until you delete them, and the node-local ones fill the worker's NVMe (/local), which your datasets and checkpoints share:

What Where Per
Pyxis named containers (--container-name) /local/enroot/data-<uid>/pyxis_<name> node
Enroot layer cache (--container-image=<registry image>, enroot import) /local/enroot/cache-<uid>; layers unused for 14 days are deleted automatically node
Apptainer cache $APPTAINER_CACHEDIR, or ~/.apptainer/cache if unset node, or shared
.sqsh and .sif files you created wherever you put them, e.g. /home, /shared shared

Check what you have on each worker now and then, and before large runs. Pass the number of workers to -N, or a single node with -w <node>:

srun -p dev -N <workers> --ntasks-per-node=1 bash -c '
  echo "== $(hostname)"; enroot list
  du -sh /local/enroot/*-$(id -u) /local/apptainer-cache-$USER 2>/dev/null
  df -h /local | tail -1'
du -sh ~/.apptainer/cache ~/*.sif /shared/*.sqsh 2>/dev/null

Free space on a node:

# A named container: the next job with that name unpacks the .sqsh again
srun -p dev -w <node> enroot remove -f pyxis_pytorch-25.10

# The Enroot layer cache: the next registry import downloads the layers again
srun -p dev -w <node> rm -rf /local/enroot/cache-$(id -u)

# The Apptainer cache (set APPTAINER_CACHEDIR as when you pulled)
srun -p dev -w <node> env APPTAINER_CACHEDIR=/local/apptainer-cache-$USER apptainer cache clean -f

Delete .sqsh and .sif files you no longer use with rm. On the shared filesystem they count against the space of the whole cluster.