Skip to content

Health checks

Slinky clusters continuously monitor worker health and run scheduled checks through ordinary Slurm jobs. The checks use the same scheduler as your workloads: they queue behind your jobs, do not preempt them, and only run when the required resources are available.

View health-check results

Open Grafana as described in Monitoring, then open the Health Checks folder. Start with Cluster Active Health Check Overview.

The Registered checks table provides one row per check:

  • Check — the registry name. Select it to open the corresponding overview or detail dashboard.
  • Scope — whether the result covers a node or the whole fleet.
  • Kind — an observing check reports existing metrics; an owning check can also dispatch work.
  • Scheduler — slurm on Slinky clusters.
  • Result — Healthy, Degraded, or Failed.
  • Freshness and Age — whether the result is still within that check's reporting window.
  • Last run — the timestamp associated with the latest result.

The node table below it links each Instance to the corresponding node detail dashboard.

Info

A newly provisioned cluster can show states such as Pending first run, Awaiting registry, or Idle before a scheduled check has reported. These are not failed results. Overdue means that a check has not reported within its expected freshness window and should be reviewed.

Checks included with Slinky

Registry check Scope What it covers When it runs
active-node-suite Node DCGM diagnostics, GPU matrix multiplication, intra-node NCCL collectives, host/device memory copies, kernel-launch latency, and CPU memory bandwidth Recurring sweep on available workers
passive-node-suite Node Continuously collected GPU, NVLink, InfiniBand, thermal, ECC, and host-health signals Continuous
gpud-node-agent Node GPU incidents including XID/SXID events, remapped rows, and link health Continuous
dcgm-exporter Node DCGM GPU telemetry and health-watch metrics Continuous
inter-node-nccl Fleet Full-cluster NCCL AllReduce over InfiniBand, compared with the baseline for the cluster topology and GPU model Recurring
prolog-epilog-job-checks Node GPU state before a Slurm job and GPU/NCCL performance checks after it At Slurm job boundaries
weekly-training-benchmark Fleet TorchTitan Llama and Qwen training, including convergence, throughput, TFLOPs per GPU, and model FLOPs utilization Weekly
weekly-io-benchmark Node fio and mdtest on local NVMe and the shared filesystem, compared with pinned baselines Weekly
weekly-inference-benchmark Fleet The DeepSeek-V4-Pro serving frontier across tensor- and expert-parallel configurations, compared with a frozen baseline Weekly on supported B300 clusters

A Degraded result generally means the check completed but one or more measurements crossed a review threshold. A Failed result means the check itself failed, produced an incomplete result, or reported a failed source verdict. Open the linked detail dashboard to see the affected nodes and measurements.

Automatic worker gate

Every worker's slurmd runs a lightweight health program periodically and when the node state changes. It verifies GPU visibility, DCGM health, and recent fatal NVIDIA XID events. If the gate fails, Slurm drains the node so new jobs avoid it.

Inspect a drained node and return it to service after fixing the cause:

sinfo -R
scontrol show node slinky-0
scontrol update NodeName=slinky-0 State=RESUME

Hardware-related alerts are also forwarded to Verda.

Inspect scheduled checks

Run these commands on the login node:

# Current and recent health-check jobs
squeue
sacct -X --name=chc-active,dsv4-xref --starttime today

# Services and schedules
systemctl list-timers --all | grep -E 'chc-active|io-bench|infbench|tb-weekly'
systemctl status chc-active.service

# Service logs
journalctl -u chc-active.service
journalctl -u io-bench-weekly.service
journalctl -u infbench-weekly.service
journalctl -u tb-weekly.service

Health-check Slurm jobs use recognizable names such as chc-active, dsv4-xref, and io-bench-*.

Trigger a scheduled check manually

Run the relevant service as root on the login node. The submitted jobs still follow normal Slurm scheduling and wait for the required resources.

systemctl start chc-active.service
systemctl start io-bench-weekly.service
systemctl start infbench-weekly.service
systemctl start tb-weekly.service

Scheduled checks use marker files to avoid rerunning before they are due. If you intentionally need to repeat a check immediately, inspect the corresponding marker first:

ls -l /var/lib/cluster-health-check/

Remove only the marker for the check you intend to repeat, then start its service again. Removing a marker changes scheduling state; it does not remove existing Grafana results.

The prolog and epilog checks do not have a separate manual trigger. They run at the boundaries of normal GPU jobs. See Observability before adding your own Slurm hooks.

Troubleshooting

If a registry row needs review:

  1. Select the Check name to open its dashboard.
  2. Identify whether the issue is a failed result, a threshold regression, or an overdue report.
  3. Check squeue and sacct to see whether the underlying job is queued, running, or completed.
  4. Inspect the corresponding systemd journal and Slurm output.
  5. For a node-level issue, select its Instance in the overview and compare it with the other workers.

An overdue scheduled check can simply be waiting behind customer workloads. Do not cancel customer jobs to make a health check run.