Skip to content

Validation

Every Instant Cluster is validated automatically before it is handed over. The validation tests we run depend on the image type / orchestrator you deploy:

  • Native Slurm — Slurm runs directly on the nodes.
  • Kubernetes (k8s) — Kubernetes only, no Slurm.
  • Slinky — Slurm running inside Kubernetes (the Slinky Slurm operator).

The validation runs in two phases:

  • Phase 1 runs while the cluster is in validating status. It must pass before the cluster transitions to running.
  • Phase 2 runs after the cluster is running. Its tests are currently informational and do not affect cluster status.

Checks common to every cluster

These run regardless of the orchestrator.

Early checks (per node)

When each node comes up we verify:

  • Kernel versions
  • InfiniBand card port status, configuration and firmware versions
  • ECC configuration consistency across all GPUs within each node

Node health (NHC + gpud, Slurm-based images)

On native Slurm and Slinky images, a node only becomes idle in Slurm after the node health check (NHC) passes (see /etc/nhc/nhc.conf), which verifies:

  • / partition has less than 90% disk usage
  • dcgmi diag -r 1 -n gpu:8 passes
  • All NVLinks and InfiniBand ports are up
  • There are 8 InfiniBand ports at NDR or faster, all sharing the same P_Key
  • All gpud checks report Healthy

On Kubernetes-only images the equivalent gate is the node reporting Ready in kubectl get nodes. gpud still runs on every compute node.

Phase 1 readiness gates (all orchestrators)

Before the orchestrator-specific Phase 1 tests run, the login node waits for and verifies:

  • The expected number of nodes report ready (slurm_nodes_idle for Slurm, count(up{job="node_exporter"}) for k8s) in Prometheus
  • Prometheus has the expected number of healthy scrape targets
  • Grafana is responding
  • The object storage endpoint is reachable
  • For Kubernetes images: kubectl get nodes shows all nodes Ready
  • kanidm (cluster auth) reports online on the login node

Native Slurm

Phase 1

In addition to the common readiness gates above:

  • A Slurm job runs nccl-tests all_reduce_perf across all nodes. Phase 1 fails if the job fails or reports too little bus bandwidth (minimum 350 GB/s).

Phase 2 (informational)

The following Slurm jobs run as the ubuntu user. Their results are recorded as Prometheus metrics:

  • NCCL all_reduce_perf (2-node) — must reach a minimum bus bandwidth (380 GB/s on H200, 680 GB/s otherwise)
  • ucx_perftest — RDMA bandwidth between nodes (minimum 50000 MB/s)
  • iperf node-to-node Ethernet bandwidth (minimum 50 Gbps)
  • iperf all-nodes-to-node-1 Ethernet bandwidth (minimum 50 Gbps total)
  • srun responsiveness — srun hostname and srun --gpus 8 nvidia-smi each complete within 30s
  • slurmrestd ping — the Slurm REST API answers

Details of these jobs are in /home/ubuntu/slurm-*.out and /home/ubuntu/verda_validation/, or via journalctl -u verda-validation-phase-2.service.

Kubernetes (k8s)

A Kubernetes-only image has no Slurm (no slurmctld, no worker pods, no login pod), so all Slurm-specific tests are skipped.

Phase 1

Only the common readiness gates apply — most importantly that kubectl get nodes shows all nodes Ready. Once those pass, the cluster transitions to running.

Phase 2 (informational)

  • Prometheus is responding
  • The Kubernetes RDMA networking is configured: NicClusterPolicy / rdma_shared_device_a, the local-disk and local-path StorageClasses, and the MPI Operator
  • Validation metrics are written to Prometheus

Slinky

A Slinky image runs Slurm inside Kubernetes. Both the Kubernetes node checks and a set of Slurm-via-k8s smoke checks run.

Beyond deploy-time validation, each worker also runs a Slurm HealthCheckProgram inside the slurmd pod on an interval: it checks GPU visibility, DCGM health and recent fatal NVIDIA Xid events, and drains the Slurm node when a check fails, so new jobs avoid unhealthy workers.

Phase 1

In addition to the common readiness gates:

  • The Kubernetes RDMA policy for the Slinky workers is in place
  • slurmctld is up
  • The Slurm worker pods register and become ready
  • scontrol reconfigure succeeds from inside the login pod
  • An srun smoke check is responsive inside the login pod

Phase 2 (informational)

Run from inside the Slurm login pod / across the worker pods:

  • Prometheus is responding and the Kubernetes RDMA networking is configured
  • slurmctld is up and the worker pods are ready
  • Multi-node smoke — srun across nodes runs hostname and nvidia-smi -L
  • Shared-jail smoke — entering the shared jail, plus sbatch with a nested srun
  • DNS / egress on every worker — each worker has intact jail binaries, a working resolv.conf, DNS resolution and egress (regression guard for the shared-rootfs bind-detach race)
  • Shared /home venv on every worker — a Python venv on shared /home is usable from every worker
  • Nested multi-node srun — sbatch launching a nested multi-node srun inside the jail
  • UCX 2-node RDMA wireup smoke — non-fatal; a transient InfiniBand hiccup warns rather than fails validation
  • Validation metrics are written to Prometheus

Phase 2 details for any image can be found in /home/ubuntu/verda_validation/, /home/ubuntu/slurm-*.out, or by running journalctl -u verda-validation-phase-2.service on the login node.

Ongoing health checks

Validation covers hand-over; after that, recurring health checks watch the cluster for the rest of its life. For the check catalog and operational commands for your orchestrator, see Kubernetes health checks or Slinky health checks.

All results land in the Grafana Health Checks folder (see Monitoring), whose overview dashboard acts as a registry: one row per check with its last run, result and freshness — a check that stops reporting shows as OVERDUE rather than silently disappearing.

Active checks (6-hourly)

  • Per-node benchmark suite — DCGM diagnostics, matmul, intra-node NCCL allreduce/alltoall, host↔device memcpy, kernel-launch latency and CPU memory bandwidth run on every idle GPU node. The first sweep runs minutes after provisioning. Checks only use idle nodes — they queue behind and never preempt your workloads (on Slurm flavours via exclusive Slurm jobs, on Kubernetes via GPU-requesting pods).
  • Full-cluster NCCL AllReduce — a single NCCL communicator spanning every GPU node runs an allreduce over InfiniBand, with correctness checking, and compares the measured bus bandwidth against a baseline for your cluster's exact size and hardware. Fabric degradation anywhere in the fleet is caught within hours instead of at your next big training run. Runs on Slinky (as Slurm jobs) and Kubernetes (as MPIJobs) clusters alike.

Per-job checks (Slinky)

  • On Slinky clusters, lightweight prolog and epilog checks run at the boundaries of every Slurm job: GPU state and visibility before the job starts, and a lightweight GEMM benchmark plus an intra-node NCCL allreduce after it ends. The slowest GPU's TFLOPS is compared against a baseline for your GPU model, so a straggling or degraded GPU is flagged at the next job boundary — not at the next weekly benchmark. Results appear in the Cluster Prolog Epilog Health Check dashboard.

Passive checks (continuous)

  • The passive suite, gpud and the DCGM exporter watch every node continuously for XID/SXID events, ECC errors, NVLink and InfiniBand health, thermals and remapped rows. Hardware-related alerts are forwarded to Verda automatically.

Weekly benchmarks

Three heavier benchmarks run once a week on idle capacity. Like all health checks, they queue behind your workloads and never preempt them. On a freshly provisioned cluster these show Pending first run until their first weekly slot; that is expected.

  • Training benchmark — real TorchTitan Llama-70B and Qwen3 training runs, as an end-to-end "does real training still converge at the expected TFLOPs" probe. Results (including measured TFLOPs per GPU and model FLOPs utilization) land in the Training Benchmark Details dashboard.
  • Storage IO benchmark — fio and mdtest measure bandwidth, IOPS and metadata latency on each node's local NVMe scratch and on the shared filesystem, and compare the results against pinned baselines with a ±15% drift gate. Results land in the Storage IO Benchmark dashboard.
  • Inference benchmark (Slinky and Kubernetes, B300) — the full DeepSeek-V4-Pro vLLM serving frontier (tensor- and expert-parallel modes across concurrency levels), following the SemiAnalysis InferenceX methodology. It runs with synthetic in-memory weights, so nothing large is downloaded, and each point is compared against a frozen baseline with a −4% regression gate. Throughput per GPU, interactivity and time-to-first-token land in the Inference Benchmark Details dashboard, including a side-by-side comparison table against the baseline.