---
description: Run containerized workloads on a Slinky cluster with Pyxis/Enroot (srun --container-image) or Apptainer inside Slurm steps.
revision_date: 03.11.2026
---

# Containers

A Slinky cluster runs containers in two ways, both from a Slurm step and both as your own user:

* **[Pyxis](https://github.com/NVIDIA/pyxis) + [Enroot](https://github.com/NVIDIA/enroot)**: add `--container-image` to `srun`. This is the usual choice for multi-node GPU training.
* **[Apptainer](https://apptainer.org/)** (formerly Singularity): run `apptainer exec` inside a step, for example with a pre-built SIF file.

The cluster ships Pyxis 0.24, Enroot 4.2 and Apptainer 1.5.5.

## Pyxis: `srun --container-image`

```console
$ srun --container-image=alpine:3 grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:3
pyxis: imported docker image: alpine:3
PRETTY_NAME="Alpine Linux v3.24"
```

Image names follow Enroot's syntax: Docker Hub images by name, other registries with `#` between the registry and the image, for example `--container-image=nvcr.io#nvidia/pytorch:25.06-py3`.

Without extra flags, the container gets:

* your `$HOME` and the shared `/shared` directory, plus the worker's `/local` NVMe,
* the GPUs you requested with `--gpus` (and only those),
* the InfiniBand devices, so NCCL uses the fabric (`NET/IB`) from inside the container,
* unlimited locked memory, which NCCL needs for InfiniBand.

```console
$ srun --gpus=2 --container-image=nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
GPU 1: NVIDIA B300 SXM6 AC (UUID: GPU-...)
```

The container starts in the image's `WORKDIR` (`/workspace` for NGC images), not where you submitted. Pass `--container-workdir="$PWD"` to run your code from the current directory, and `--container-mounts=SRC:DST` to mount other paths. The full option list is in `srun --help` and the [Pyxis documentation](https://github.com/NVIDIA/pyxis/wiki/Usage).

!!! info "No `enroot` on the login pod"
    Enroot runs in the job step on the worker, not on the login pod you SSH into on port `22`. Running `enroot` there prints the commands below and exits. Run `enroot` through `srun`, or on the login node (`ssh -p 2222 ubuntu@<cluster-ip>`), which has Enroot but no GPU.

    On the login node, Enroot keeps its cache and unpacked containers in `/var/lib/enroot` on the node's own disk, not in `/home`. That directory has a quota of half the disk, at most 150 GiB, so imports cannot fill the node's disk; check usage with `sudo repquota -P / 2>/dev/null`. Cached layers unused for 14 days are deleted automatically. The `.sqsh` file from `enroot import -o` goes where you put it.

### Import once, run many times

`--container-image=<registry image>` downloads and converts the image in every step. For large images (NGC PyTorch is about 16 GB), import it once into a squashfs file on the shared filesystem and start jobs from that file:

```bash
# Import once, inside a job
srun -p dev -c 32 enroot import -o /shared/pytorch-25.10.sqsh \
    docker://nvcr.io#nvidia/pytorch:25.10-py3

# Run from the file
srun --gpus=8 --container-image=/shared/pytorch-25.10.sqsh \
     --container-name=pytorch-25.10 --container-workdir="$PWD" \
     python train.py
```

* `--container-name` keeps the unpacked image on the worker's NVMe, so later jobs on that node start in seconds instead of unpacking the `.sqsh` again. An existing name is reused as is, so put the image version in the name.
* `-p dev` uses the short-job partition (8 h limit), which the scheduler considers before `all`. `-c 32` only speeds up the squashfs build; the import also completes in a default-sized job.

Named containers stay on the node (one copy per user, name and node) until you remove them, see [Clean up container storage](#clean-up-container-storage).

!!! tip
    Pick an image whose CUDA build matches the GPU. These are Blackwell **B300** GPUs (`compute_cap` 10.3), so use CUDA 13.x and a recent framework build (e.g. a current PyTorch/NGC release). CUDA 12.x images predate B300 support and may fail to run kernels even though `nvidia-smi` lists the device.

## Apptainer

Pull and run a public image directly. Apptainer converts the OCI image to a SIF on the fly:

```console
$ srun apptainer exec docker://alpine:3 cat /etc/os-release
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.24"
```

`singularity` is an alias of `apptainer`, so either name works.

### GPU containers

Request GPUs with `--gpus=N` on the step and pass `--nv` to Apptainer so the NVIDIA driver userspace is exposed inside the container. Exactly the requested number of GPUs appears:

```console
$ srun --gpus=1 apptainer exec --nv docker://nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
```

A step launched with **no** `--gpus` sees no GPUs: `nvidia-smi -L` prints `No devices found.` from inside the container.

### Pull once, reuse many times

Converting an OCI image on every step is wasteful for large images. Pull to a SIF in the shared `/home` filesystem once, then point every step at that file:

```bash
# Pull once, inside a job
srun --time=00:10:00 bash -c '
  export APPTAINER_CACHEDIR=/local/apptainer-cache-$USER APPTAINER_TMPDIR=/local/apptainer-tmp-$USER
  mkdir -p "$APPTAINER_CACHEDIR" "$APPTAINER_TMPDIR"
  apptainer pull $HOME/cuda.sif docker://nvidia/cuda:13.0.0-base-ubuntu24.04'

# In any step: reuse the SIF (no re-download, no conversion)
srun --gpus=1 apptainer exec --nv $HOME/cuda.sif nvidia-smi -L
```

Because `/home` is shared, the same SIF is usable from any worker, including multi-node jobs.

!!! tip "Cache and temp directories"
    Set `APPTAINER_CACHEDIR` and `APPTAINER_TMPDIR` to node-local paths as above. The default cache lives under `$HOME` (`~/.apptainer/cache`), which puts multi-gigabyte layers on the shared filesystem. The cache is not reclaimed automatically, see [Clean up container storage](#clean-up-container-storage).

## No Docker daemon

There is no Docker daemon in a job, so `docker run` does not work. Run Docker images with Pyxis (`--container-image`) or Apptainer (`docker://...`) instead.

## Clean up container storage

Apart from the Enroot layer cache, nothing removes container images or caches for you. They stay until you delete them, and the node-local ones fill the worker's NVMe (`/local`), which your datasets and checkpoints share:

| What | Where | Per |
|------|-------|-----|
| Pyxis named containers (`--container-name`) | `/local/enroot/data-<uid>/pyxis_<name>` | node |
| Enroot layer cache (`--container-image=<registry image>`, `enroot import`) | `/local/enroot/cache-<uid>`; layers unused for 14 days are deleted automatically | node |
| Apptainer cache | `$APPTAINER_CACHEDIR`, or `~/.apptainer/cache` if unset | node, or shared |
| `.sqsh` and `.sif` files you created | wherever you put them, e.g. `/home`, `/shared` | shared |

Check what you have on each worker now and then, and before large runs. Pass the number of workers to `-N`, or a single node with `-w <node>`:

```bash
srun -p dev -N <workers> --ntasks-per-node=1 bash -c '
  echo "== $(hostname)"; enroot list
  du -sh /local/enroot/*-$(id -u) /local/apptainer-cache-$USER 2>/dev/null
  df -h /local | tail -1'
du -sh ~/.apptainer/cache ~/*.sif /shared/*.sqsh 2>/dev/null
```

Free space on a node:

```bash
# A named container: the next job with that name unpacks the .sqsh again
srun -p dev -w <node> enroot remove -f pyxis_pytorch-25.10

# The Enroot layer cache: the next registry import downloads the layers again
srun -p dev -w <node> rm -rf /local/enroot/cache-$(id -u)

# The Apptainer cache (set APPTAINER_CACHEDIR as when you pulled)
srun -p dev -w <node> env APPTAINER_CACHEDIR=/local/apptainer-cache-$USER apptainer cache clean -f
```

Delete `.sqsh` and `.sif` files you no longer use with `rm`. On the shared filesystem they count against the space of the whole cluster.
