---
description: Health checks and recurring benchmarks on Slinky Instant Clusters, including dashboard results, scheduling behavior, manual runs, and troubleshooting.
revision_date: 27.07.2026
---

# Health checks

Slinky clusters continuously monitor worker health and run scheduled checks through ordinary Slurm jobs. The checks use the same scheduler as your workloads: they queue behind your jobs, do not preempt them, and only run when the required resources are available.

## View health-check results

Open Grafana as described in [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/), then open the **Health Checks** folder. Start with **Cluster Active Health Check Overview**.

The **Registered checks** table provides one row per check:

* **Check** — the registry name. Select it to open the corresponding overview or detail dashboard.
* **Scope** — whether the result covers a node or the whole fleet.
* **Kind** — an observing check reports existing metrics; an owning check can also dispatch work.
* **Scheduler** — `slurm` on Slinky clusters.
* **Result** — `Healthy`, `Degraded`, or `Failed`.
* **Freshness** and **Age** — whether the result is still within that check's reporting window.
* **Last run** — the timestamp associated with the latest result.

The node table below it links each **Instance** to the corresponding node detail dashboard.

!!! info
    A newly provisioned cluster can show states such as **Pending first run**, **Awaiting registry**, or **Idle** before a scheduled check has reported. These are not failed results. **Overdue** means that a check has not reported within its expected freshness window and should be reviewed.

## Checks included with Slinky

| Registry check | Scope | What it covers | When it runs |
|---|---|---|---|
| `active-node-suite` | Node | DCGM diagnostics, GPU matrix multiplication, intra-node NCCL collectives, host/device memory copies, kernel-launch latency, and CPU memory bandwidth | Recurring sweep on available workers |
| `passive-node-suite` | Node | Continuously collected GPU, NVLink, InfiniBand, thermal, ECC, and host-health signals | Continuous |
| `gpud-node-agent` | Node | GPU incidents including XID/SXID events, remapped rows, and link health | Continuous |
| `dcgm-exporter` | Node | DCGM GPU telemetry and health-watch metrics | Continuous |
| `inter-node-nccl` | Fleet | Full-cluster NCCL AllReduce over InfiniBand, compared with the baseline for the cluster topology and GPU model | Recurring |
| `prolog-epilog-job-checks` | Node | GPU state before a Slurm job and GPU/NCCL performance checks after it | At Slurm job boundaries |
| `weekly-training-benchmark` | Fleet | TorchTitan Llama and Qwen training, including convergence, throughput, TFLOPs per GPU, and model FLOPs utilization | Weekly |
| `weekly-io-benchmark` | Node | fio and mdtest on local NVMe and the shared filesystem, compared with pinned baselines | Weekly |
| `weekly-inference-benchmark` | Fleet | The DeepSeek-V4-Pro serving frontier across tensor- and expert-parallel configurations, compared with a frozen baseline | Weekly on supported B300 clusters |

A `Degraded` result generally means the check completed but one or more measurements crossed a review threshold. A `Failed` result means the check itself failed, produced an incomplete result, or reported a failed source verdict. Open the linked detail dashboard to see the affected nodes and measurements.

## Automatic worker gate

Every worker's `slurmd` runs a lightweight health program periodically and when the node state changes. It verifies GPU visibility, DCGM health, and recent fatal NVIDIA XID events. If the gate fails, Slurm drains the node so new jobs avoid it.

Inspect a drained node and return it to service after fixing the cause:

```bash
sinfo -R
scontrol show node slinky-0
scontrol update NodeName=slinky-0 State=RESUME
```

Hardware-related alerts are also forwarded to Verda.

## Inspect scheduled checks

Run these commands on the login node:

```bash
# Current and recent health-check jobs
squeue
sacct -X --name=chc-active,dsv4-xref --starttime today

# Services and schedules
systemctl list-timers --all | grep -E 'chc-active|io-bench|infbench|tb-weekly'
systemctl status chc-active.service

# Service logs
journalctl -u chc-active.service
journalctl -u io-bench-weekly.service
journalctl -u infbench-weekly.service
journalctl -u tb-weekly.service
```

Health-check Slurm jobs use recognizable names such as `chc-active`, `dsv4-xref`, and `io-bench-*`.

## Trigger a scheduled check manually

Run the relevant service as `root` on the login node. The submitted jobs still follow normal Slurm scheduling and wait for the required resources.

```bash
systemctl start chc-active.service
systemctl start io-bench-weekly.service
systemctl start infbench-weekly.service
systemctl start tb-weekly.service
```

Scheduled checks use marker files to avoid rerunning before they are due. If you intentionally need to repeat a check immediately, inspect the corresponding marker first:

```bash
ls -l /var/lib/cluster-health-check/
```

Remove only the marker for the check you intend to repeat, then start its service again. Removing a marker changes scheduling state; it does not remove existing Grafana results.

The prolog and epilog checks do not have a separate manual trigger. They run at the boundaries of normal GPU jobs. See [Observability](https://docs.verda.com/clusters/instant-clusters/slinky/observability/#custom-prolog-and-epilog-hooks) before adding your own Slurm hooks.

## Troubleshooting

If a registry row needs review:

1. Select the **Check** name to open its dashboard.
2. Identify whether the issue is a failed result, a threshold regression, or an overdue report.
3. Check `squeue` and `sacct` to see whether the underlying job is queued, running, or completed.
4. Inspect the corresponding systemd journal and Slurm output.
5. For a node-level issue, select its **Instance** in the overview and compare it with the other workers.

An overdue scheduled check can simply be waiting behind customer workloads. Do not cancel customer jobs to make a health check run.