---
description: Grafana dashboards, automatic health checks and benchmarks, and custom prolog/epilog hooks on a Slinky cluster.
revision_date: 29.10.2026
---

# Observability

## Grafana dashboards

Grafana access works as on any instant cluster, see [Monitoring](https://docs.verda.com/clusters/cluster-basics/monitoring/). On Slinky clusters the pre-provisioned set includes:

* **Slurm folder**: *Slurm Native / Overview*, */ Nodes and Partitions*, */ Scheduler*, plus a Slinky operator dashboard.
* **Kubernetes folder**: cluster, node, pod and workload views.
* **Health Checks folder**: the health-check registry and detail dashboards — active sweeps, full-cluster NCCL, the per-job prolog GPU check, and the weekly training, storage IO and inference benchmarks (see [Ongoing health checks](https://docs.verda.com/clusters/cluster-basics/validation/#ongoing-health-checks)).
* The common dashboards: *GPU Overview*, *GPUd Overview*, *NVIDIA DCGM Exporter*, *Node Exporter*, and *Cluster Log Explorer* for centralized logs.

## Health checks

Every worker's `slurmd` runs a lightweight health gate periodically and when its node state changes. It verifies GPU visibility, DCGM health, and recent fatal NVIDIA XID events. A failed gate drains the node automatically so new jobs avoid it.

Recurring node, fabric, storage, training, and inference checks run through the normal Slurm scheduler without preempting customer workloads. See [Ongoing health checks](https://docs.verda.com/clusters/cluster-basics/validation/#ongoing-health-checks) for the full check catalog and dashboard statuses.

## Custom prolog and epilog hooks

Slurm runs **prolog** scripts on every allocated node before a job starts and **epilog** scripts after it ends. Use them for per-job health gates, cleanup, or bookkeeping. On Slinky the scripts are distributed as ConfigMaps referenced from the controller resource; the operator wires them into `slurm.conf` and the workers fetch them automatically, no image changes or pod restarts needed.

Clusters ship with a Verda-provided health-gate prolog preconfigured (visible with `scontrol show config | grep -iE '^Prolog|^Epilog'`, named `20-verda-prolog-gpustate.sh`). Before each GPU job it checks that the allocated GPUs are visible and idle and that NVLink is up, and publishes the result to the *Cluster Prolog Epilog Health Check* dashboard — keep it in place when you add your own.

1. Write the script and create a ConfigMap from it. The file name becomes the script name; when several scripts are configured they run in file-name order, so a numeric prefix keeps the order explicit:

    ```bash
    cat > 50-gpu-gate.sh <<'EOF'
    #!/bin/bash
    # refuse to start jobs on a node that lost its GPUs
    count=$(ls /dev/nvidia[0-9]* 2>/dev/null | wc -l)
    [ "${count}" -ge 8 ] || exit 1
    EOF
    kubectl -n slurm create configmap my-prolog --from-file=50-gpu-gate.sh
    ```

2. Reference it from the controller. This **replaces** the whole list, so check `kubectl -n slurm get controller slurm -o yaml` first and include any existing refs:

    ```bash
    kubectl -n slurm patch controller slurm --type merge \
      -p '{"spec":{"prologScriptRefs":[{"name":"my-prolog"}]}}'
    ```

    Epilogs work the same via `epilogScriptRefs`.

3. The operator reconfigures the cluster live. Verify:

    ```bash
    scontrol show config | grep -iE '^Prolog|^Epilog'
    ```

!!! warning
    * The prolog runs in the start path of **every job**, keep it fast (well under a couple of seconds).
    * A non-zero exit from a prolog or epilog **drains the node**. That is the point (gate unhealthy nodes), but it means a buggy script drains the whole cluster job by job. Test on one job before rolling out.
