Observability¶

Grafana dashboards¶

Grafana access works as on any instant cluster, see Monitoring. On Slinky clusters the pre-provisioned set includes:

• Slurm folder: Slurm Native / Overview, / Nodes and Partitions, / Scheduler, plus a Slinky operator dashboard.
• Kubernetes folder: cluster, node, pod and workload views.
• Health Checks folder: the health-check registry and detail dashboards — active sweeps, full-cluster NCCL, per-job prolog/epilog GPU checks, and the weekly training, storage IO and inference benchmarks (see Health checks).
• The common dashboards: GPU Overview, GPUd Overview, NVIDIA DCGM Exporter, Node Exporter, and Cluster Log Explorer for centralized logs.

Health checks¶

Every worker's slurmd runs a lightweight health gate periodically and when its node state changes. It verifies GPU visibility, DCGM health, and recent fatal NVIDIA XID events. A failed gate drains the node automatically so new jobs avoid it.

Recurring node, fabric, storage, training, and inference checks run through the normal Slurm scheduler without preempting customer workloads. See Health checks for the full check catalog, dashboard statuses, manual triggers, and troubleshooting commands.

Custom prolog and epilog hooks¶

Slurm runs prolog scripts on every allocated node before a job starts and epilog scripts after it ends. Use them for per-job health gates, cleanup, or bookkeeping. On Slinky the scripts are distributed as ConfigMaps referenced from the controller resource; the operator wires them into slurm.conf and the workers fetch them automatically, no image changes or pod restarts needed.

Clusters ship with Verda-provided health-gate scripts preconfigured (visible with scontrol show config | grep -iE '^Prolog|^Epilog', names like 20-verda-prolog-gpustate.sh). They check GPU state and run a short NCCL all-reduce before each job, then a per-GPU GEMM benchmark after it, publishing per-GPU TFLOPS to the Cluster Prolog Epilog Health Check dashboard — keep them in place when you add your own.

1. Write the script and create a ConfigMap from it. The file name becomes the script name; when several scripts are configured they run in file-name order, so a numeric prefix keeps the order explicit:

   cat > 50-gpu-gate.sh <<'EOF'
   #!/bin/bash
   # refuse to start jobs on a node that lost its GPUs
   count=$(ls /dev/nvidia[0-9]* 2>/dev/null | wc -l)
   [ "${count}" -ge 8 ] || exit 1
   EOF
   kubectl -n slurm create configmap my-prolog --from-file=50-gpu-gate.sh
   

2. Reference it from the controller. This replaces the whole list, so check kubectl -n slurm get controller slurm -o yaml first and include any existing refs:

   kubectl -n slurm patch controller slurm --type merge \
     -p '{"spec":{"prologScriptRefs":[{"name":"my-prolog"}]}}'
   

   Epilogs work the same via epilogScriptRefs.

3. The operator reconfigures the cluster live. Verify:

   scontrol show config | grep -iE '^Prolog|^Epilog'