Containers¶
On this Slinky (Slurm-on-Kubernetes) cluster, the supported way to run containers is Apptainer (formerly Singularity) from inside a Slurm step. Every srun/sbatch step runs in a per-job shared jail, and Apptainer works inside that jail.
Info
singularity is an alias of apptainer (the singularity binary is a symlink to apptainer), so either name works. enroot is also installed for manual use.
Run a container in a Slurm step¶
Pull and run a public image directly. Apptainer converts the OCI image to a SIF on the fly:
$ srun apptainer exec docker://alpine:3 cat /etc/os-release
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.24"
See Slurm for the full srun/sbatch reference, including --time, --mem and partition selection.
GPU containers¶
Request GPUs with --gpus=N on the step and pass --nv to Apptainer so the host NVIDIA driver userspace is exposed inside the container. Exactly the requested number of GPUs appears:
$ srun --gpus=1 apptainer exec --nv docker://nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-...)
Warning
The step is GPU-cgroup fenced. A step launched with no --gpus sees no GPUs — nvidia-smi -L prints No devices found. from inside the container. Always request --gpus=N for GPU work.
Tip
Pick an image whose CUDA / framework build matches the GPU architecture. These are Blackwell B300 GPUs, so use a recent CUDA (12.x) and a recent framework build (e.g. a current PyTorch/NGC release). Older images built before Blackwell support may fail to run kernels even though nvidia-smi lists the device.
Pull once, reuse many times¶
Converting an OCI image on every step is wasteful for large images. Pull to a SIF in the shared /home filesystem once on the login node, then point every step at that file:
# On the login node — pull to /home (shared across all nodes)
apptainer pull /home/$USER/cuda.sif docker://nvidia/cuda:12.4.1-base-ubuntu22.04
# In any step — reuse the local SIF (no re-download, no conversion)
srun --gpus=1 apptainer exec --nv /home/$USER/cuda.sif nvidia-smi -L
Because /home is shared (and bind-mounted into the jail), the same SIF is usable from any worker, including multi-node jobs.
What is not available¶
pyxis / srun --container-image¶
NVIDIA pyxis is not enabled on this cluster, so srun --container-image=... does not work:
The shared jail owns the step's mount namespace, and pyxis would need to own the same namespace — the two are mutually exclusive. Use Apptainer instead. See Shared jail for details on how the jail works.
docker run¶
The docker client binary exists in the jail, but there is no in-job Docker daemon, so docker run (and any other daemon-backed command) fails inside a step:
$ srun docker run --rm alpine:3 true
failed to connect to the docker API at unix:///var/run/docker.sock ...
To run a Docker image, reference it with Apptainer (docker://...) as shown above, or pre-pull it to a SIF.