Monitoring¶
Our instant clusters come with dashboards, centralized logs and customizable alerts, to monitor the state of the cluster.
To access the dashboard navigate to the cluster dropdown and select View metric dashboard.
To access the Grafana portal, follow the provided instructions to obtain the address and login details. If prompted with a certificate warning (common with self-signed certificates), select Advanced and then Proceed to … to continue.
One-click read-only link¶
The welcome message shown at SSH login (on the login node, and on Slinky clusters also in the Slurm login pod) contains a link of the form https://<cluster-dns>/grafana-login/<token>. Opening it signs you in as a read-only viewer user, so anyone who can log in to the cluster can open the dashboards and logs without a password. The link rotates weekly; the current one is always in the welcome message, and an open Grafana session keeps working after a rotation. Treat the link like a password: it grants read access to everything in Grafana.
Admin login¶
Changing dashboards, alert rules or contact points needs the admin account. Open https://<cluster-dns>/ and sign in with admin and the password in /root/grafana_password.txt on the login node (readable with sudo; on Slinky clusters the same file is available in the Slurm login pod).
Once logged in, navigate to the Dashboards section in the side menu. The pre-configured dashboards include:
- GPU Overview – General GPU monitoring.
- GPUd Overview – GPU health as seen by gpud.
- NVIDIA DCGM Exporter – Metrics from the DCGM exporter.
- Node Exporter – Detailed hardware and OS-level system metrics.
- Cluster Log Explorer – Centralized logs from all cluster nodes.
- Slurm folder – Job and scheduler activity. Slinky clusters get the Slurm Native overview, nodes/partitions and scheduler dashboards plus a Slinky operator dashboard.
- Kubernetes folder (Kubernetes and Slinky clusters) – Cluster, node, pod and workload views from the Kubernetes collectors.
- Health Checks folder – Results of the automatic health checks. See Ongoing health checks for the check catalog. The Cluster Active Health Check Overview is the index: select a Check name to open its corresponding dashboard, or select a node's Instance to drill into per-node details. Detail dashboards cover full-cluster NCCL results, the per-job prolog GPU check on Slinky, and the weekly training, storage IO and inference benchmarks.
Behind Grafana, metrics and logs are collected by a VictoriaMetrics-based stack on the service node: VictoriaMetrics stores metrics and VictoriaLogs stores logs shipped from every node. The metrics datasource (named Metrics) is Prometheus-compatible, so custom dashboards and PromQL queries work as usual; the logs datasource is named Logs.
The cluster is also pre-configured with several alerting rules, which can be viewed under the Alerts tab. Hardware-related alerts are automatically forwarded to Verda for faster resolution. Additional alerts can be created and customized to notify through Grafana’s contact points by editing the grafana-default-email channel. This allows customer-specific alerts to be routed to any contact point defined by the customer directly within the Grafana UI.



