---
description: Health checks and recurring benchmarks on Kubernetes Instant Clusters, including dashboard results, Kubernetes resources, manual runs, and troubleshooting.
revision_date: 27.07.2026
---

# Health checks

Kubernetes Instant Clusters continuously monitor every worker and run scheduled checks as Kubernetes Jobs, CronJobs, and MPIJobs in the `monitoring` namespace. GPU checks request resources through the normal scheduler, queue behind your workloads, and do not preempt them.

## View health-check results

Open Grafana as described in [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/), then open the **Health Checks** folder. Start with **Cluster Active Health Check Overview**.

The **Registered checks** table provides one row per check:

* **Check** — the registry name. Select it to open the corresponding overview or detail dashboard.
* **Scope** — whether the result covers a node or the whole fleet.
* **Kind** — an observing check reports existing metrics; an owning check can also dispatch work.
* **Scheduler** — `k8s` on Kubernetes clusters.
* **Result** — `Healthy`, `Degraded`, or `Failed`.
* **Freshness** and **Age** — whether the result is still within that check's reporting window.
* **Last run** — the timestamp associated with the latest result.

The node table below it links each **Instance** to the corresponding node detail dashboard.

!!! info
    A newly provisioned cluster can show states such as **Pending first run**, **Awaiting registry**, or **Idle** before a scheduled check has reported. These are not failed results. **Overdue** means that a check has not reported within its expected freshness window and should be reviewed.

## Checks included with Kubernetes

| Registry check | Scope | What it covers | How it runs |
|---|---|---|---|
| `active-node-suite` | Node | DCGM diagnostics, GPU matrix multiplication, intra-node NCCL collectives, host/device memory copies, kernel-launch latency, and CPU memory bandwidth | Recurring CronJob with one GPU pod per available worker |
| `passive-node-suite` | Node | Continuously collected GPU, NVLink, InfiniBand, thermal, ECC, and host-health signals | DaemonSet on every worker |
| `gpud-node-agent` | Node | GPU incidents including XID/SXID events, remapped rows, and link health | Node agent on every worker |
| `dcgm-exporter` | Node | DCGM GPU telemetry and health-watch metrics | Exporter on every worker |
| `inter-node-nccl` | Fleet | Full-cluster NCCL AllReduce over InfiniBand, compared with the baseline for the cluster topology and GPU model | Registry-dispatched MPIJob |
| `weekly-training-benchmark` | Fleet | TorchTitan Llama and Qwen training, including convergence, throughput, TFLOPs per GPU, and model FLOPs utilization | Weekly Indexed Jobs on supported B300 clusters |
| `weekly-io-benchmark` | Node | fio and mdtest on local NVMe and the shared filesystem, compared with pinned baselines | Registry-dispatched Jobs |
| `weekly-inference-benchmark` | Fleet | The DeepSeek-V4-Pro serving frontier across tensor- and expert-parallel configurations, compared with a frozen baseline | Registry-dispatched Jobs on supported B300 clusters |

A `Degraded` result generally means the check completed but one or more measurements crossed a review threshold. A `Failed` result means the check itself failed, produced an incomplete result, or reported a failed source verdict. Open the linked detail dashboard to see the affected nodes and measurements.

## How checks are scheduled

The active suite and training benchmarks are Kubernetes CronJobs. The full-cluster NCCL, storage IO, and inference checks are owned by the health-check registry on the service node, which periodically dispatches work that is due. Passive agents and exporters run continuously.

Inspect the health-check resources from the jump host:

```bash
kubectl -n monitoring get cronjobs
kubectl -n monitoring get jobs
kubectl -n monitoring get mpijobs
kubectl -n monitoring get pods
```

For a compact live view:

```bash
kubectl -n monitoring get jobs,mpijobs,pods -w
```

## Trigger a CronJob manually

You can create a one-off Job from the active or training CronJobs. These Jobs still follow normal scheduling and wait for the required GPU resources.

```bash
suffix=$(date +%s)

kubectl -n monitoring create job \
  --from=cronjob/cluster-health-check-active "chc-manual-${suffix}"

kubectl -n monitoring create job \
  --from=cronjob/tt-llama70b-weekly "tt-llama-manual-${suffix}"

kubectl -n monitoring create job \
  --from=cronjob/tt-qwen3-weekly "tt-qwen-manual-${suffix}"
```

The registry dispatches NCCL, storage IO, and inference checks automatically when they are due. Their Jobs and MPIJobs appear in the same `monitoring` namespace.

## Inspect a run

Find the Job and its pods, then inspect events and logs:

```bash
kubectl -n monitoring describe job <job-name>
kubectl -n monitoring get pods -l job-name=<job-name> -o wide
kubectl -n monitoring logs job/<job-name>
```

For an MPIJob:

```bash
kubectl -n monitoring describe mpijob <mpijob-name>
kubectl -n monitoring get pods -l training.kubeflow.org/job-name=<mpijob-name> -o wide
```

If a pod is still Pending, inspect its scheduling events:

```bash
kubectl -n monitoring describe pod <pod-name>
```

A scheduled check can remain queued while customer workloads hold the GPUs. Do not delete or preempt customer workloads to make a health check run.

## Troubleshooting

If a registry row needs review:

1. Select the **Check** name to open its dashboard.
2. Identify whether the issue is a failed result, a threshold regression, or an overdue report.
3. Inspect the corresponding Job, MPIJob, pods, and scheduling events in `monitoring`.
4. Review logs from the launcher or index-zero pod for distributed checks.
5. For a node-level issue, select its **Instance** in the overview and compare it with the other workers.

Hardware-related alerts from the passive checks are also forwarded to Verda.