Kubernetes¶

Choosing the Kubernetes job orchestrator provisions the Instant Cluster as a vanilla, kubeadm-based Kubernetes cluster ready to run multi-node GPU workloads over InfiniBand — no additional setup required. You get full cluster-admin access: install operators, run Helm charts, and use standard Kubernetes tooling as on any cluster you own.

Architecture¶

graph TD
    User([User])
    User -->|"SSH :22"| Login

    subgraph cluster [Cluster private network]
        Login["Jumphost (login node)<br/>kubectl · k9s · helm"]
        subgraph service [Service node — control plane]
            API["kube-apiserver, etcd,<br/>scheduler, controller-manager"]
            Mon["Monitoring stack<br/>(VictoriaMetrics · Grafana)"]
            Mgmt["Cluster add-on management<br/>(operators, dashboard, Kueue)"]
        end
        subgraph workers [GPU workers]
            W1["worker node · 8 GPUs<br/>your pods"]
            Wn["worker node · 8 GPUs<br/>your pods"]
        end
        Login --- service
        API --- W1
        API --- Wn
    end

The jumphost is the SSH entry point. kubectl, k9s and helm are pre-configured with admin credentials — see Getting started.
The service node runs the Kubernetes control plane and the management plane of every pre-installed add-on, plus the monitoring stack. It is tainted NoSchedule, so your workloads never compete with it.
Each GPU worker exposes 8 × nvidia.com/gpu plus an rdma/rdma_shared_device_a device for InfiniBand. Workers run nothing but your pods and the per-node agents (device plugin, exporters, health checks).
/home is the shared filesystem, mounted on every node — it also backs the shared-path ReadWriteMany StorageClass (see Storage).

What's included¶

The following components are pre-installed and ready to use:

Component	Purpose	Details
NVIDIA GPU Operator	GPU scheduling and lifecycle	Manages the device plugin, GPU Feature Discovery, Node Feature Discovery and validators. GPUs appear as `nvidia.com/gpu` resources. The driver and container toolkit are preinstalled in the image, so no driver containers run
MPI Operator	Distributed multi-node job orchestration	Provides the `MPIJob` custom resource — see Running multi-node workloads
Kueue	Job queueing and quota admission	Pre-wired with a `default` queue — see Job queueing
NVIDIA Network Operator	InfiniBand / RDMA networking	Configures high-speed InfiniBand networking for GPU-to-GPU communication across nodes
Cilium	Pod networking (CNI)	Handles standard Ethernet-based pod-to-pod and pod-to-service communication
Kubernetes Dashboard	Web UI	Full cluster web UI on NodePort `32443` — see Observability
metrics-server	Resource metrics API	Backs `kubectl top nodes` / `kubectl top pods`
Storage classes	Local and shared storage	`local-path` (default), `shared-path` (RWX), `local-disk` — see Storage

Info

The Kubernetes orchestrator is actively developed. Centralized cluster-user management (available on Slurm/Slinky via Kanidm) is not yet integrated on Kubernetes clusters — access is via the admin kubeconfig.

In this section¶

Getting started — access the cluster, run your first multi-node NCCL test, pull from registries.
Running multi-node workloads — MPIJob in depth, InfiniBand/NCCL configuration, PyTorchJob and the Training Operator.
Job queueing (Kueue) — queue workloads, enforce GPU quota, set priorities.
Storage — node-local NVMe, ReadWriteMany volumes on the shared filesystem.
Observability — Grafana, the Kubernetes Dashboard, health checks and profiling.
Tutorials — worked examples: gang-scheduled training with SkyPilot + Kueue and NVIDIA Dynamo inference.

Monitoring access and validation work as on any instant cluster; the dashboards and health checks are Kubernetes-aware.