Tutorials¶
Run a containerized PyTorch job¶
Goal: run a GPU workload from an NGC container image as a normal Slurm batch job. The pattern is pull once, reuse everywhere: convert the image to a SIF file on the shared /home, then point every job at it.
1. Pull the image to a SIF (once)¶
Run the pull inside a job so the conversion happens on a worker. NGC PyTorch images are big: give the job memory and CPUs (the SIF compression is parallel) and point the Apptainer cache and temp directory at the node-local /local, instead of the slower shared /home. The cache on /local survives the job, so a repeat pull on the same node skips the download. Pick a current NGC PyTorch tag; Blackwell GPUs need a recent build.
srun --mem=64G --cpus-per-task=16 --time=01:00:00 bash -c '
export APPTAINER_CACHEDIR=/local/apptainer-cache-$USER APPTAINER_TMPDIR=/local/apptainer-tmp-$USER
mkdir -p "$APPTAINER_CACHEDIR" "$APPTAINER_TMPDIR"
apptainer pull /home/'"$USER"'/pytorch.sif docker://nvcr.io/nvidia/pytorch:25.06-py3
'
The pull takes about 10 minutes and the resulting SIF is around 13 GB. Because /home is shared with every worker, the same SIF is now usable from any worker, so this is a one-time step. The SIF and the cache stay until you delete them, see Clean up container storage.
2. Submit the job¶
Write the test script and the batch file, then submit. The batch file references absolute paths on /home (they are filled in when you create the file, so the job itself does not depend on any runtime environment):
cat > /home/$USER/torch-test.py <<'EOF'
import torch
n = torch.cuda.device_count()
print(f"visible GPUs: {n}")
for i in range(n):
print(" ", torch.cuda.get_device_name(i))
x = torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16)
y = x @ x
torch.cuda.synchronize()
print("matmul OK:", y.shape)
EOF
cat > /home/$USER/torch-test.sbatch <<EOF
#!/bin/bash
#SBATCH --job-name=torch-test
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --mem=128G
#SBATCH --time=00:10:00
apptainer exec --nv /home/$USER/pytorch.sif python3 /home/$USER/torch-test.py
EOF
sbatch /home/$USER/torch-test.sbatch
3. Check the result¶
The job takes under a minute. Watch it with squeue --me, then read its output:
$ cat slurm-<jobid>.out
visible GPUs: 8
NVIDIA B300 SXM6 AC
NVIDIA B300 SXM6 AC
...
matmul OK: torch.Size([8192, 8192])
Key points:
--nvexposes the host NVIDIA driver inside the container;--gres=gpu:8decides how many GPUs the step (and therefore the container) sees.- Request memory (
--mem) for both the pull and the job; image conversion and PyTorch both need it. -
For interactive experimentation, the same works under
srun --pty:
More container details (GPU passthrough, what is not available) are on the Containers page.