Observability¶

Grafana dashboards¶

Grafana access works as on any instant cluster, see Monitoring. On Slinky clusters the pre-provisioned set includes:

• Slurm folder: Slurm Native / Overview, / Nodes and Partitions, / Scheduler, plus a Slinky operator dashboard.
• Kubernetes folder: cluster, node, pod and workload views.
• The common dashboards: GPU Overview, GPUd Overview, NVIDIA DCGM Exporter, Node Exporter, and Cluster Log Explorer for centralized logs.

s9s, a terminal UI for Slurm¶

s9s is a k9s-style TUI for Slurm, preinstalled at /usr/local/bin/s9s and preconfigured for the cluster. From your bastion shell:

s9s

Navigation is vim-like (j/k move, Enter drills in, : switches views, ? help, q quit). You can watch jobs, nodes, partitions and the queue update live, and cancel/hold/release jobs from the list. It talks to the Slurm REST API (slurmrestd), so what it shows matches what the CLI reports.

Use s9s for watching a long-running job or the node states; use sbatch/srun and the rest of the Slurm CLI for submitting and for anything scripted.

Automatic worker health checks¶

Every worker's slurmd runs a health check every few minutes and at node-state changes. It verifies GPU visibility, DCGM health, and scans for recent fatal NVIDIA Xid events. When a check fails, the node is drained automatically so new jobs avoid it.

Inspect and recover:

sinfo -R                          # drained nodes with reasons
scontrol show node slinky-0       # details for one node
scontrol update NodeName=slinky-0 State=RESUME   # after the cause is fixed

Hardware-related alerts are also forwarded to Verda, see Monitoring.

Custom prolog and epilog hooks¶

Slurm runs prolog scripts on every allocated node before a job starts and epilog scripts after it ends. Use them for per-job health gates, cleanup, or bookkeeping. On Slinky the scripts are distributed as ConfigMaps referenced from the controller resource; the operator wires them into slurm.conf and the workers fetch them automatically, no image changes or pod restarts needed.

Clusters ship with Verda-provided health-gate scripts preconfigured (visible with scontrol show config | grep -iE '^Prolog|^Epilog', names like 10-verda-prolog-gpustate.sh). Keep them in place when you add your own.

1. Write the script and create a ConfigMap from it. The file name becomes the script name; when several scripts are configured they run in file-name order, so a numeric prefix keeps the order explicit:

   cat > 50-gpu-gate.sh <<'EOF'
   #!/bin/bash
   # refuse to start jobs on a node that lost its GPUs
   count=$(ls /dev/nvidia[0-9]* 2>/dev/null | wc -l)
   [ "${count}" -ge 8 ] || exit 1
   EOF
   kubectl -n slurm create configmap my-prolog --from-file=50-gpu-gate.sh
   

2. Reference it from the controller. This replaces the whole list, so check kubectl -n slurm get controller slurm -o yaml first and include any existing refs:

   kubectl -n slurm patch controller slurm --type merge \
     -p '{"spec":{"prologScriptRefs":[{"name":"my-prolog"}]}}'
   

   Epilogs work the same via epilogScriptRefs.

3. The operator reconfigures the cluster live. Verify:

   scontrol show config | grep -iE '^Prolog|^Epilog'