---
description: "Run containerised workloads on a Verda Instant Cluster, using Pyxis and Enroot to launch images across nodes for distributed training jobs at any scale."
revision_date: 04.08.2026
---

# Containers

!!! info
    This page covers containers under **native Slurm** using [Enroot](https://github.com/NVIDIA/enroot) and [Pyxis](https://github.com/NVIDIA/pyxis). If your cluster runs **Slinky** (Slurm-on-Kubernetes), Enroot/Pyxis are not used — run containers with Apptainer instead. See [Slinky → Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/).

We present here a basic test for containerized environments using [Enroot](https://github.com/NVIDIA/enroot) and [Pyxis](https://github.com/NVIDIA/pyxis), both from NVIDIA. 

First, for testing enroot:

```
enroot import docker://ubuntu
enroot create -n ubuntu ubuntu.sqsh
enroot start ubuntu sh -c 'grep PRETTY /etc/os-release'
> PRETTY_NAME="Ubuntu 24.04.2 LTS"
```

Secondly, we ensure we get the same results from testing Pyxis:

```
srun --container-image=ubuntu grep PRETTY /etc/os-release
> PRETTY_NAME="Ubuntu 24.04.2 LTS"
```

Alternatively to use a custom image built in dockerd:

1. Build a custom dockerfile with:

    ```bash
    docker build -f <file.dockerfile> -t <name:tag> .
    ```

2. [Import](https://github.com/NVIDIA/enroot/blob/master/doc/cmd/import.md) dockerd image to Enroot (Can be done with `docker://IMAGE:TAG` from registry)

    ```bash
    enroot import dockerd://<name:tag>
    ```

3. Use flag pointing to the [name:tag](name:tag).sqsh

    ```bash
    --container-image=<name:tag>.sqsh
    ```

### Example: torchtitan multi-node

We clone cluster-tests into `/home/ubuntu`:

```
git clone https://github.com/datacrunch-research/cluster-tests.git /home/ubuntu/cluster-tests
```

We build the image based on [torchtitan.dockerfile](https://github.com/datacrunch-research/cluster-tests/blob/main/containers/torchtitan.dockerfile):

> NOTE: we need to include the HF\_TOKEN in .bashrc or export it in the bash session with access granted for llama3 family models.

```
docker build -f torchtitan.dockerfile --build-arg HF_TOKEN="$HF_TOKEN" -t torchtitan_cuda128_torch27 .
```

Then we import the squash file, which Enroot will use:

```
enroot import -o /home/ubuntu/torchtitan_cuda128_torch27.sqsh dockerd://torchtitan_cuda128_torch27
```

Now, we execute [torchtitan\_multinode.sh](https://github.com/datacrunch-research/cluster-tests/blob/main/containers/torchtitan_multinode.sh):

```
sbatch torchtitan_multinode.sh
```