Skip to content

Monitoring

Our instant clusters come with dashboards, centralized logs and customizable alerts, to monitor the state of the cluster.

To access the dashboard navigate to the cluster dropdown and select View metric dashboard.

To access the Grafana portal, follow the provided instructions to obtain the address and login details. If prompted with a certificate warning (common with self-signed certificates), select Advanced and then Proceed to … to continue. The password can be retrieved from the jump host.

Once logged in, navigate to the Dashboards section in the side menu. The pre-configured dashboards include:

  • GPU Overview – General GPU monitoring.
  • GPUd Overview – GPU health as seen by gpud.
  • NVIDIA DCGM Exporter – Metrics from the DCGM exporter.
  • Node Exporter – Detailed hardware and OS-level system metrics.
  • Cluster Log Explorer – Centralized logs from all cluster nodes.
  • Slurm folder – Job and scheduler activity. Native Slurm clusters get the SLURM Dashboard and Slurm Job (GPU) dashboards; Slinky clusters get the Slurm Native overview, nodes/partitions and scheduler dashboards plus a Slinky operator dashboard.
  • Kubernetes folder (Kubernetes and Slinky clusters) – Cluster, node, pod and workload views from the Kubernetes collectors.
  • Health Checks folder – Results of the automatic health checks. See the dedicated Kubernetes or Slinky guide for the check catalog and operational commands. The Cluster Active Health Check Overview is the index: select a Check name to open its corresponding dashboard, or select a node's Instance to drill into per-node details. Detail dashboards cover full-cluster NCCL results, per-job prolog/epilog GPU checks, and the weekly training, storage IO and inference benchmarks.

Behind Grafana, metrics and logs are collected by a VictoriaMetrics-based stack on the service node: VictoriaMetrics stores metrics and VictoriaLogs stores logs shipped from every node. The metrics datasource (named Metrics) is Prometheus-compatible, so custom dashboards and PromQL queries work as usual; the logs datasource is named Logs.

The cluster is also pre-configured with several alerting rules, which can be viewed under the Alerts tab. Hardware-related alerts are automatically forwarded to Verda for faster resolution. Additional alerts can be created and customized to notify through Grafana’s contact points by editing the grafana-default-email channel. This allows customer-specific alerts to be routed to any contact point defined by the customer directly within the Grafana UI.