--- description: Run distributed multi-node GPU workloads on a Kubernetes Instant Cluster — MPIJob in depth, InfiniBand/NCCL configuration, and PyTorchJob via the Kubeflow Training Operator. revision_date: 27.07.2026 --- # Running multi-node workloads ## MPIJob (MPI Operator) An `MPIJob` is a Kubernetes custom resource provided by the pre-installed [MPI Operator](https://github.com/kubeflow/mpi-operator). It is the primary way to run distributed multi-node workloads on the cluster. When you submit an MPIJob, the operator creates: - A **launcher pod** that coordinates the job (similar to `mpirun`) - One or more **worker pods** that perform the actual computation The operator handles SSH key distribution and network setup between pods automatically. You define your container image, GPU resource requests, and the command to run — the operator takes care of the rest. ### MPI Operator API versions: `v1` vs `v2beta1` The MPI Operator supports two API versions. Your cluster uses **v2beta1**, which is the recommended version. | | `kubeflow.org/v1` | `kubeflow.org/v2beta1` | | ------------------- | ------------------------------------------- | ------------------------------------------------ | | Worker connectivity | `kubectl exec` (requires API server access) | SSH (direct pod-to-pod) | | Image requirement | No `sshd` needed | **Must include `sshd`** | | Launcher networking | Goes through Kubernetes API server | Direct SSH to workers — lower latency | | Hostfile | Managed via ConfigMap | Written to `/etc/mpi/hostfile` | | Status | Stable but older | Actively developed, recommended for new clusters | !!! info All examples in this documentation use `apiVersion: kubeflow.org/v2beta1`. If you see `v1` examples from external sources, the main difference to be aware of is the `sshd` requirement — `v2beta1` workers must have an SSH server in the container image. Use MPIJobs for: - NCCL communication tests (e.g. `all_reduce_perf`) - Distributed PyTorch training with `torchrun` - Any workload that needs to run across multiple nodes with GPU-to-GPU communication ### Complete example This runs an NCCL `all_reduce_perf` benchmark across 2 nodes with 8 GPUs each: ```yaml apiVersion: kubeflow.org/v2beta1 kind: MPIJob metadata: generateName: nccl-test-2n- spec: slotsPerWorker: 8 runPolicy: cleanPodPolicy: Running mpiReplicaSpecs: Launcher: replicas: 1 template: spec: containers: - name: launcher image: vccr.io/nccl-tests/nccl-tests:cuda13.1.1-nccl2.29.3-1-v2.17.9 env: - name: OMPI_ALLOW_RUN_AS_ROOT value: "1" - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM value: "1" command: ["/bin/bash", "-c"] args: - | echo "=== NCCL 16-GPU Test (2 nodes) ===" # Wait for MPI hostfile echo "Waiting for MPI hostfile..." while [ ! -f /etc/mpi/hostfile ] || [ ! -s /etc/mpi/hostfile ]; do sleep 2 done echo "Hostfile:" cat /etc/mpi/hostfile # Wait for workers to be reachable via SSH echo "Waiting for workers..." for worker in $(awk '{print $1}' /etc/mpi/hostfile); do retries=0 until ssh -o ConnectTimeout=2 "$worker" hostname >/dev/null 2>&1; do retries=$((retries + 1)) if [ "$retries" -ge 60 ]; then echo "TIMEOUT: $worker not reachable after 5 minutes" exit 1 fi echo " Waiting for $worker... (attempt $retries)" sleep 5 done echo " $worker ready" done echo "All workers ready" echo "" echo "==========================================" echo "Running: all_reduce_perf" echo "==========================================" mpirun \ -np 16 \ -bind-to none \ /opt/nccl-tests/build/all_reduce_perf -b 512M -e 8G -f 2 -g 1 echo "" echo "=== NCCL test completed ===" resources: requests: cpu: 2 memory: 256Mi Worker: replicas: 2 template: metadata: labels: app: nccl-test spec: containers: - name: worker image: vccr.io/nccl-tests/nccl-tests:cuda13.1.1-nccl2.29.3-1-v2.17.9 securityContext: capabilities: add: - IPC_LOCK resources: requests: cpu: 32 memory: 128Gi nvidia.com/gpu: 8 rdma/rdma_shared_device_a: 1 limits: nvidia.com/gpu: 8 rdma/rdma_shared_device_a: 1 volumeMounts: - mountPath: /dev/shm name: dshm affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: nccl-test topologyKey: kubernetes.io/hostname volumes: - name: dshm emptyDir: medium: Memory sizeLimit: 64Gi ``` Key details in this manifest: - **`generateName`** instead of `name` — each `kubectl create` generates a unique job name - **`rdma/rdma_shared_device_a`** — requests RDMA device access for InfiniBand communication - **`IPC_LOCK` capability** — required for RDMA memory registration - **`/dev/shm`** — large shared memory volume for NCCL inter-process communication - **`podAntiAffinity`** — ensures workers are scheduled on different physical nodes - **`NCCL_IB_PKEY`** — deliberately not set, which is correct for B300. On H200/B200 clusters add `-x NCCL_IB_PKEY=1 \` to the `mpirun` invocation (see below) - **`sshd` requirement** — the MPI Operator uses SSH to launch processes on workers, so your container image must include an SSH server !!! warning Your container image **must** include `/usr/sbin/sshd`. Standard NGC images (e.g. `nvcr.io/nvidia/pytorch:...`) do not ship with an SSH server and will fail with `StartError` when used in an MPIJob. Either build a custom image with `sshd` installed, or use `PyTorchJob` instead (see below). ## InfiniBand and NCCL configuration The cluster uses InfiniBand for high-speed GPU-to-GPU communication across nodes via [NCCL](https://developer.nvidia.com/nccl) (NVIDIA Collective Communications Library). Depending on the GPU generation, you may need to set the following environment variables in your job containers: | Environment Variable | Value | Purpose | | -------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `NCCL_IB_PKEY` | `1` | **Required on H200 and B200 clusters.** Tells NCCL which InfiniBand Partition Key to use — without it, cross-node GPU communication fails. **Do not set it on B300 clusters**: their IB partition sits at PKey index 0, which is NCCL's default. | | `NCCL_DEBUG` | `INFO` | Optional. Enables verbose NCCL logging, useful for troubleshooting communication issues. | !!! tip The pre-staged example jobs in `/home/ubuntu/` are rendered for your cluster's hardware — on B300 clusters the `NCCL_IB_PKEY` line is already removed, on H200/B200 it is already set. When in doubt, copy the environment block from those examples. ## PyTorchJob (Kubeflow Training Operator) The cluster ships with the **MPI Operator** (for `MPIJob`). If you want a higher-level abstraction for distributed training — such as `PyTorchJob`, which automatically injects environment variables like `MASTER_ADDR`, `WORLD_SIZE`, and `RANK` — install the Kubeflow Training Operator yourself: ```bash kubectl apply --server-side -k "github.com/kubeflow/training-operator.git/manifests/overlays/standalone?ref=v1.8.1" ``` Verify the installation: ```bash kubectl get crd | grep kubeflow ``` You should see `pytorchjobs.kubeflow.org` alongside the existing `mpijobs.kubeflow.org`. ### When to use MPIJob vs PyTorchJob | | MPIJob | PyTorchJob | | ----------------- | --------------------------------------- | --------------------------------------------------- | | Pre-installed | Yes | No (install Training Operator first) | | Communication | MPI (mpirun launches processes via SSH) | PyTorch Distributed (torchrun / elastic) | | Image requirement | Must include `sshd` | No `sshd` needed — standard NGC images work | | Env vars | Manual (`NCCL_IB_PKEY`, etc.) | Auto-injected (`MASTER_ADDR`, `RANK`, `WORLD_SIZE`) | | Best for | NCCL tests, MPI-native workloads | PyTorch training scripts using `torch.distributed` | ## Queueing your workloads Both MPIJobs and plain Jobs can be queued and quota-gated through the pre-installed Kueue — see [Job queueing (Kueue)](https://docs.verda.com/clusters/instant-clusters/kubernetes/queueing/). For a full gang-scheduled training pipeline, see the [SkyPilot + Kueue tutorial](https://docs.verda.com/clusters/instant-clusters/tutorials/kueue-gang-scheduling/).