---
description: Grafana dashboards, automatic health checks and benchmarks, and custom prolog/epilog hooks on a Slinky cluster.
revision_date: 27.07.2026
---

# Observability

## Grafana dashboards

Grafana access works as on any instant cluster, see [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/). On Slinky clusters the pre-provisioned set includes:

* **Slurm folder**: *Slurm Native / Overview*, */ Nodes and Partitions*, */ Scheduler*, plus a Slinky operator dashboard.
* **Kubernetes folder**: cluster, node, pod and workload views.
* **Health Checks folder**: the health-check registry and detail dashboards — active sweeps, full-cluster NCCL, per-job prolog/epilog GPU checks, and the weekly training, storage IO and inference benchmarks (see [Health checks](https://docs.verda.com/clusters/instant-clusters/slinky/health-checks/)).
* The common dashboards: *GPU Overview*, *GPUd Overview*, *NVIDIA DCGM Exporter*, *Node Exporter*, and *Cluster Log Explorer* for centralized logs.

## Health checks

Every worker's `slurmd` runs a lightweight health gate periodically and when its node state changes. It verifies GPU visibility, DCGM health, and recent fatal NVIDIA XID events. A failed gate drains the node automatically so new jobs avoid it.

Recurring node, fabric, storage, training, and inference checks run through the normal Slurm scheduler without preempting customer workloads. See [Health checks](https://docs.verda.com/clusters/instant-clusters/slinky/health-checks/) for the full check catalog, dashboard statuses, manual triggers, and troubleshooting commands.

## Custom prolog and epilog hooks

Slurm runs **prolog** scripts on every allocated node before a job starts and **epilog** scripts after it ends. Use them for per-job health gates, cleanup, or bookkeeping. On Slinky the scripts are distributed as ConfigMaps referenced from the controller resource; the operator wires them into `slurm.conf` and the workers fetch them automatically, no image changes or pod restarts needed.

Clusters ship with Verda-provided health-gate scripts preconfigured (visible with `scontrol show config | grep -iE '^Prolog|^Epilog'`, names like `20-verda-prolog-gpustate.sh`). They check GPU state and run a short NCCL all-reduce before each job, then a per-GPU GEMM benchmark after it, publishing per-GPU TFLOPS to the *Cluster Prolog Epilog Health Check* dashboard — keep them in place when you add your own.

1. Write the script and create a ConfigMap from it. The file name becomes the script name; when several scripts are configured they run in file-name order, so a numeric prefix keeps the order explicit:

    ```bash
    cat > 50-gpu-gate.sh <<'EOF'
    #!/bin/bash
    # refuse to start jobs on a node that lost its GPUs
    count=$(ls /dev/nvidia[0-9]* 2>/dev/null | wc -l)
    [ "${count}" -ge 8 ] || exit 1
    EOF
    kubectl -n slurm create configmap my-prolog --from-file=50-gpu-gate.sh
    ```

2. Reference it from the controller. This **replaces** the whole list, so check `kubectl -n slurm get controller slurm -o yaml` first and include any existing refs:

    ```bash
    kubectl -n slurm patch controller slurm --type merge \
      -p '{"spec":{"prologScriptRefs":[{"name":"my-prolog"}]}}'
    ```

    Epilogs work the same via `epilogScriptRefs`.

3. The operator reconfigures the cluster live. Verify:

    ```bash
    scontrol show config | grep -iE '^Prolog|^Epilog'
    ```

!!! warning
    * The prolog runs in the start path of **every job**, keep it fast (well under a couple of seconds).
    * A non-zero exit from a prolog or epilog **drains the node**. That is the point (gate unhealthy nodes), but it means a buggy script drains the whole cluster job by job. Test on one job before rolling out.