Skip to content

Health checks

Kubernetes Instant Clusters continuously monitor every worker and run scheduled checks as Kubernetes Jobs, CronJobs, and MPIJobs in the monitoring namespace. GPU checks request resources through the normal scheduler, queue behind your workloads, and do not preempt them.

View health-check results

Open Grafana as described in Monitoring, then open the Health Checks folder. Start with Cluster Active Health Check Overview.

The Registered checks table provides one row per check:

  • Check — the registry name. Select it to open the corresponding overview or detail dashboard.
  • Scope — whether the result covers a node or the whole fleet.
  • Kind — an observing check reports existing metrics; an owning check can also dispatch work.
  • Scheduler — k8s on Kubernetes clusters.
  • Result — Healthy, Degraded, or Failed.
  • Freshness and Age — whether the result is still within that check's reporting window.
  • Last run — the timestamp associated with the latest result.

The node table below it links each Instance to the corresponding node detail dashboard.

Info

A newly provisioned cluster can show states such as Pending first run, Awaiting registry, or Idle before a scheduled check has reported. These are not failed results. Overdue means that a check has not reported within its expected freshness window and should be reviewed.

Checks included with Kubernetes

Registry check Scope What it covers How it runs
active-node-suite Node DCGM diagnostics, GPU matrix multiplication, intra-node NCCL collectives, host/device memory copies, kernel-launch latency, and CPU memory bandwidth Recurring CronJob with one GPU pod per available worker
passive-node-suite Node Continuously collected GPU, NVLink, InfiniBand, thermal, ECC, and host-health signals DaemonSet on every worker
gpud-node-agent Node GPU incidents including XID/SXID events, remapped rows, and link health Node agent on every worker
dcgm-exporter Node DCGM GPU telemetry and health-watch metrics Exporter on every worker
inter-node-nccl Fleet Full-cluster NCCL AllReduce over InfiniBand, compared with the baseline for the cluster topology and GPU model Registry-dispatched MPIJob
weekly-training-benchmark Fleet TorchTitan Llama and Qwen training, including convergence, throughput, TFLOPs per GPU, and model FLOPs utilization Weekly Indexed Jobs on supported B300 clusters
weekly-io-benchmark Node fio and mdtest on local NVMe and the shared filesystem, compared with pinned baselines Registry-dispatched Jobs
weekly-inference-benchmark Fleet The DeepSeek-V4-Pro serving frontier across tensor- and expert-parallel configurations, compared with a frozen baseline Registry-dispatched Jobs on supported B300 clusters

A Degraded result generally means the check completed but one or more measurements crossed a review threshold. A Failed result means the check itself failed, produced an incomplete result, or reported a failed source verdict. Open the linked detail dashboard to see the affected nodes and measurements.

How checks are scheduled

The active suite and training benchmarks are Kubernetes CronJobs. The full-cluster NCCL, storage IO, and inference checks are owned by the health-check registry on the service node, which periodically dispatches work that is due. Passive agents and exporters run continuously.

Inspect the health-check resources from the jump host:

kubectl -n monitoring get cronjobs
kubectl -n monitoring get jobs
kubectl -n monitoring get mpijobs
kubectl -n monitoring get pods

For a compact live view:

kubectl -n monitoring get jobs,mpijobs,pods -w

Trigger a CronJob manually

You can create a one-off Job from the active or training CronJobs. These Jobs still follow normal scheduling and wait for the required GPU resources.

suffix=$(date +%s)

kubectl -n monitoring create job \
  --from=cronjob/cluster-health-check-active "chc-manual-${suffix}"

kubectl -n monitoring create job \
  --from=cronjob/tt-llama70b-weekly "tt-llama-manual-${suffix}"

kubectl -n monitoring create job \
  --from=cronjob/tt-qwen3-weekly "tt-qwen-manual-${suffix}"

The registry dispatches NCCL, storage IO, and inference checks automatically when they are due. Their Jobs and MPIJobs appear in the same monitoring namespace.

Inspect a run

Find the Job and its pods, then inspect events and logs:

kubectl -n monitoring describe job <job-name>
kubectl -n monitoring get pods -l job-name=<job-name> -o wide
kubectl -n monitoring logs job/<job-name>

For an MPIJob:

kubectl -n monitoring describe mpijob <mpijob-name>
kubectl -n monitoring get pods -l training.kubeflow.org/job-name=<mpijob-name> -o wide

If a pod is still Pending, inspect its scheduling events:

kubectl -n monitoring describe pod <pod-name>

A scheduled check can remain queued while customer workloads hold the GPUs. Do not delete or preempt customer workloads to make a health check run.

Troubleshooting

If a registry row needs review:

  1. Select the Check name to open its dashboard.
  2. Identify whether the issue is a failed result, a threshold regression, or an overdue report.
  3. Inspect the corresponding Job, MPIJob, pods, and scheduling events in monitoring.
  4. Review logs from the launcher or index-zero pod for distributed checks.
  5. For a node-level issue, select its Instance in the overview and compare it with the other workers.

Hardware-related alerts from the passive checks are also forwarded to Verda.