--- description: What a Kubernetes Instant Cluster is, how it is put together, and which components come pre-installed — GPU Operator, MPI Operator, Kueue, storage classes and more. revision_date: 27.07.2026 --- # Kubernetes Choosing the Kubernetes job orchestrator provisions the Instant Cluster as a vanilla, kubeadm-based Kubernetes cluster ready to run multi-node GPU workloads over InfiniBand — no additional setup required. You get full `cluster-admin` access: install operators, run Helm charts, and use standard Kubernetes tooling as on any cluster you own. ![k8s-instant-cluster-os](https://docs.verda.com/assets/k8s-instant-cluster-os.png) ## Architecture ```mermaid graph TD User([User]) User -->|"SSH :22"| Login subgraph cluster [Cluster private network] Login["Jumphost (login node)
kubectl · k9s · helm"] subgraph service [Service node — control plane] API["kube-apiserver, etcd,
scheduler, controller-manager"] Mon["Monitoring stack
(VictoriaMetrics · Grafana)"] Mgmt["Cluster add-on management
(operators, Kueue)"] end subgraph workers [GPU workers] W1["worker node · 8 GPUs
your pods"] Wn["worker node · 8 GPUs
your pods"] end Login --- service API --- W1 API --- Wn end ``` * The **jumphost** is the SSH entry point. `kubectl`, `k9s` and `helm` are pre-configured with admin credentials — see [Getting started](https://docs.verda.com/clusters/instant-clusters/kubernetes/getting-started/). * The **service node** runs the Kubernetes control plane and the management plane of every pre-installed add-on, plus the [monitoring stack](https://docs.verda.com/clusters/instant-clusters/monitoring/). It is tainted `NoSchedule`, so your workloads never compete with it. * Each **GPU worker** exposes 8 × `nvidia.com/gpu` plus an `rdma/rdma_shared_device_a` device for InfiniBand. Workers run nothing but your pods and the per-node agents (device plugin, exporters, health checks). * `/home` is the shared filesystem, mounted on every node — it also backs the `shared-path` **ReadWriteMany** StorageClass (see [Storage](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/)). ## What's included The following components are pre-installed and ready to use: | Component | Purpose | Details | | ------------------------------------------------------------------------------------------------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/) | GPU scheduling and lifecycle | Manages the device plugin, GPU Feature Discovery, Node Feature Discovery and validators. GPUs appear as `nvidia.com/gpu` resources. The driver and container toolkit are preinstalled in the image, so no driver containers run | | [MPI Operator](https://github.com/kubeflow/mpi-operator) | Distributed multi-node job orchestration | Provides the `MPIJob` custom resource — see [Running multi-node workloads](https://docs.verda.com/clusters/instant-clusters/kubernetes/workloads/) | | [Kueue](https://kueue.sigs.k8s.io/) | Job queueing and quota admission | Pre-wired with a `default` queue — see [Job queueing](https://docs.verda.com/clusters/instant-clusters/kubernetes/queueing/) | | [NVIDIA Network Operator](https://github.com/Mellanox/network-operator) | InfiniBand / RDMA networking | Configures high-speed InfiniBand networking for GPU-to-GPU communication across nodes | | [Cilium](https://cilium.io/) | Pod networking (CNI) | Handles standard Ethernet-based pod-to-pod and pod-to-service communication | | [metrics-server](https://github.com/kubernetes-sigs/metrics-server) | Resource metrics API | Backs `kubectl top nodes` / `kubectl top pods` | | [Storage classes](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/) | Local and shared storage | [`local-path`](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/#node-local-scratch-local-path-default) (default), [`shared-path`](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/#shared-multi-node-volumes-shared-path) (**RWX**), [`local-disk`](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/#storage-classes) — see [Storage](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/) | !!! info The Kubernetes orchestrator is actively developed. Centralized cluster-user management (available on Slurm/Slinky via Kanidm) is not yet integrated on Kubernetes clusters — access is via the admin kubeconfig. ## In this section * **[Getting started](https://docs.verda.com/clusters/instant-clusters/kubernetes/getting-started/)** — access the cluster, run your first multi-node NCCL test, pull from registries. * **[Running multi-node workloads](https://docs.verda.com/clusters/instant-clusters/kubernetes/workloads/)** — MPIJob in depth, InfiniBand/NCCL configuration, PyTorchJob and the Training Operator. * **[Job queueing (Kueue)](https://docs.verda.com/clusters/instant-clusters/kubernetes/queueing/)** — queue workloads, enforce GPU quota, set priorities. * **[Storage](https://docs.verda.com/clusters/instant-clusters/kubernetes/storage/)** — node-local NVMe, ReadWriteMany volumes on the shared filesystem. * **[Observability](https://docs.verda.com/clusters/instant-clusters/kubernetes/observability/)** — Grafana, resource metrics, and profiling. * **[Health checks](https://docs.verda.com/clusters/instant-clusters/kubernetes/health-checks/)** — recurring node, fabric, storage, training, and inference checks; dashboard results; manual runs; and troubleshooting. * **[Tutorials](https://docs.verda.com/clusters/instant-clusters/tutorials/kueue-gang-scheduling/)** — gang-scheduled training with SkyPilot + Kueue, and [NVIDIA Dynamo inference](https://docs.verda.com/clusters/instant-clusters/tutorials/dynamo-inference/). [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/) access and [validation](https://docs.verda.com/clusters/instant-clusters/validation/) work as on any instant cluster; the dashboards and health checks are Kubernetes-aware.