--- description: "Validate a Verda Instant Cluster before training: NCCL bandwidth tests, GPU and InfiniBand health checks, and how to interpret the results that you get." revision_date: 04.08.2026 --- # Validation Every Instant Cluster is validated automatically before it is handed over. The validation tests we run depend on the **image type / orchestrator** you deploy: - **Native Slurm** — Slurm runs directly on the nodes. - **Kubernetes (k8s)** — Kubernetes only, no Slurm. - **Slinky** — Slurm running inside Kubernetes (the [Slinky](https://github.com/SlinkyProject) Slurm operator). The validation runs in two phases: - **Phase 1** runs while the cluster is in **validating** status. It must pass before the cluster transitions to **running**. - **Phase 2** runs after the cluster is **running**. Its tests are currently _informational_ and do not affect cluster status. --- ## Checks common to every cluster These run regardless of the orchestrator. ### Early checks (per node) When each node comes up we verify: - Kernel versions - InfiniBand card port status, configuration and firmware versions - ECC configuration consistency across all GPUs within each node ### Node health (NHC + gpud, Slurm-based images) On native Slurm and Slinky images, a node only becomes **idle** in Slurm after the node health check (NHC) passes (see `/etc/nhc/nhc.conf`), which verifies: - `/` partition has less than 90% disk usage - `dcgmi diag -r 1 -n gpu:8` passes - All NVLinks and InfiniBand ports are up - There are 8 InfiniBand ports at NDR or faster, all sharing the same P\_Key - All [gpud](https://github.com/leptonai/gpud) checks report Healthy On Kubernetes-only images the equivalent gate is the node reporting **Ready** in `kubectl get nodes`. gpud still runs on every compute node. ### Phase 1 readiness gates (all orchestrators) Before the orchestrator-specific Phase 1 tests run, the login node waits for and verifies: - The expected number of nodes report ready (`slurm_nodes_idle` for Slurm, `count(up{job="node_exporter"})` for k8s) in Prometheus - Prometheus has the expected number of healthy scrape targets - Grafana is responding - The object storage endpoint is reachable - For Kubernetes images: `kubectl get nodes` shows all nodes `Ready` - [kanidm](https://kanidm.com/) (cluster auth) reports online on the login node --- ## Native Slurm ### Phase 1 In addition to the common readiness gates above: - A Slurm job runs `nccl-tests` `all_reduce_perf` across **all** nodes. Phase 1 **fails** if the job fails or reports too little bus bandwidth (minimum **350 GB/s**). ### Phase 2 (informational) The following Slurm jobs run as the `ubuntu` user. Their results are recorded as Prometheus metrics: - **NCCL `all_reduce_perf`** (2-node) — must reach a minimum bus bandwidth (**380 GB/s** on H200, **680 GB/s** otherwise) - **`ucx_perftest`** — RDMA bandwidth between nodes (minimum **50000 MB/s**) - **iperf** node-to-node Ethernet bandwidth (minimum **50 Gbps**) - **iperf** all-nodes-to-`node-1` Ethernet bandwidth (minimum **50 Gbps** total) - **srun responsiveness** — `srun hostname` and `srun --gpus 8 nvidia-smi` each complete within 30s - **slurmrestd ping** — the Slurm REST API answers Details of these jobs are in `/home/ubuntu/slurm-*.out` and `/home/ubuntu/verda_validation/`, or via `journalctl -u verda-validation-phase-2.service`. --- ## Kubernetes (k8s) A Kubernetes-only image has no Slurm (no `slurmctld`, no worker pods, no login pod), so all Slurm-specific tests are skipped. ### Phase 1 Only the common readiness gates apply — most importantly that `kubectl get nodes` shows all nodes `Ready`. Once those pass, the cluster transitions to **running**. ### Phase 2 (informational) - Prometheus is responding - The Kubernetes RDMA networking is configured: `NicClusterPolicy` / `rdma_shared_device_a`, the local-disk and local-path StorageClasses, and the [MPI Operator](https://github.com/kubeflow/mpi-operator) - Validation metrics are written to Prometheus --- ## Slinky A Slinky image runs Slurm inside Kubernetes. Both the Kubernetes node checks and a set of Slurm-via-k8s smoke checks run. Beyond deploy-time validation, each worker also runs a Slurm `HealthCheckProgram` inside the `slurmd` pod on an interval: it checks GPU visibility, DCGM health and recent fatal NVIDIA Xid events, and drains the Slurm node when a check fails, so new jobs avoid unhealthy workers. ### Phase 1 In addition to the common readiness gates: - The Kubernetes RDMA policy for the Slinky workers is in place - `slurmctld` is up - The Slurm worker pods register and become ready - `scontrol reconfigure` succeeds from inside the login pod - An `srun` smoke check is responsive inside the login pod ### Phase 2 (informational) Run from inside the Slurm login pod / across the worker pods: - Prometheus is responding and the Kubernetes RDMA networking is configured - `slurmctld` is up and the worker pods are ready - **Multi-node smoke** — `srun` across nodes runs `hostname` and `nvidia-smi -L` - **Shared-jail smoke** — entering the shared jail, plus `sbatch` with a nested `srun` - **DNS / egress on every worker** — each worker has intact jail binaries, a working `resolv.conf`, DNS resolution and egress (regression guard for the shared-rootfs bind-detach race) - **Shared `/home` venv on every worker** — a Python venv on shared `/home` is usable from every worker - **Nested multi-node `srun`** — `sbatch` launching a nested multi-node `srun` inside the jail - **UCX 2-node RDMA wireup smoke** — _non-fatal_; a transient InfiniBand hiccup warns rather than fails validation - Validation metrics are written to Prometheus --- Phase 2 details for any image can be found in `/home/ubuntu/verda_validation/`, `/home/ubuntu/slurm-*.out`, or by running `journalctl -u verda-validation-phase-2.service` on the login node. --- ## Ongoing health checks Validation covers hand-over; after that, recurring health checks watch the cluster for the rest of its life. For the check catalog and operational commands for your orchestrator, see [Kubernetes health checks](https://docs.verda.com/clusters/instant-clusters/kubernetes/health-checks/) or [Slinky health checks](https://docs.verda.com/clusters/instant-clusters/slinky/health-checks/). All results land in the Grafana **Health Checks** folder (see [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/)), whose overview dashboard acts as a registry: one row per check with its last run, result and freshness — a check that stops reporting shows as **OVERDUE** rather than silently disappearing. ### Active checks (6-hourly) - **Per-node benchmark suite** — DCGM diagnostics, matmul, intra-node NCCL allreduce/alltoall, host↔device memcpy, kernel-launch latency and CPU memory bandwidth run on every idle GPU node. The first sweep runs minutes after provisioning. Checks only use idle nodes — they queue behind and never preempt your workloads (on Slurm flavours via exclusive Slurm jobs, on Kubernetes via GPU-requesting pods). - **Full-cluster NCCL AllReduce** — a single NCCL communicator spanning every GPU node runs an allreduce over InfiniBand, with correctness checking, and compares the measured bus bandwidth against a baseline for your cluster's exact size and hardware. Fabric degradation anywhere in the fleet is caught within hours instead of at your next big training run. Runs on Slinky (as Slurm jobs) and Kubernetes (as MPIJobs) clusters alike. ### Per-job checks (Slinky) - On Slinky clusters, lightweight prolog and epilog checks run at the boundaries of **every Slurm job**: GPU state and visibility before the job starts, and a lightweight GEMM benchmark plus an intra-node NCCL allreduce after it ends. The slowest GPU's TFLOPS is compared against a baseline for your GPU model, so a straggling or degraded GPU is flagged at the next job boundary — not at the next weekly benchmark. Results appear in the *Cluster Prolog Epilog Health Check* dashboard. ### Passive checks (continuous) - The passive suite, [gpud](https://github.com/leptonai/gpud) and the DCGM exporter watch every node continuously for XID/SXID events, ECC errors, NVLink and InfiniBand health, thermals and remapped rows. Hardware-related alerts are forwarded to Verda automatically. ### Weekly benchmarks Three heavier benchmarks run once a week on idle capacity. Like all health checks, they queue behind your workloads and never preempt them. On a freshly provisioned cluster these show **Pending first run** until their first weekly slot; that is expected. - **Training benchmark** — real [TorchTitan](https://github.com/pytorch/torchtitan) Llama-70B and Qwen3 training runs, as an end-to-end "does real training still converge at the expected TFLOPs" probe. Results (including measured TFLOPs per GPU and model FLOPs utilization) land in the *Training Benchmark Details* dashboard. - **Storage IO benchmark** — fio and mdtest measure bandwidth, IOPS and metadata latency on each node's local NVMe scratch and on the shared filesystem, and compare the results against pinned baselines with a ±15% drift gate. Results land in the *Storage IO Benchmark* dashboard. - **Inference benchmark** (Slinky and Kubernetes, B300) — the full DeepSeek-V4-Pro vLLM serving frontier (tensor- and expert-parallel modes across concurrency levels), following the SemiAnalysis InferenceX methodology. It runs with synthetic in-memory weights, so nothing large is downloaded, and each point is compared against a frozen baseline with a −4% regression gate. Throughput per GPU, interactivity and time-to-first-token land in the *Inference Benchmark Details* dashboard, including a side-by-side comparison table against the baseline.