Skip to content

Observability

Grafana dashboards

Every cluster ships a Grafana instance on the service node, backed by a VictoriaMetrics stack — access and login details are described in Monitoring. On Kubernetes clusters, the most relevant folders are:

  • Kubernetes folder — cluster, node, pod and workload views (kube-state-metrics, kubelet/cAdvisor, control-plane metrics).
  • Health Checks folder — results of the automatic health checks. The Cluster Active Health Check Overview is the index: select a Check name to open its corresponding dashboard, or select a node's Instance to drill into per-node details.
  • The common dashboards: GPU Overview, GPUd Overview, NVIDIA DCGM Exporter, Node Exporter, and Cluster Log Explorer for centralized logs.

The metrics datasource is Prometheus-compatible — your own workloads can be scraped by adding a VMServiceScrape/VMPodScrape resource, and custom dashboards work as on any Grafana.

Health checks

Passive node agents run continuously, while active node, fabric, storage, training, and inference checks run as scheduled Kubernetes workloads in the monitoring namespace. GPU checks request resources through the normal scheduler and do not preempt customer workloads.

See Health checks for the full check catalog, dashboard statuses, Kubernetes resources, manual runs, and troubleshooting commands.

Live resource usage

The pre-installed metrics-server backs the standard resource commands:

kubectl top nodes
kubectl top pods -A

Profiling

Profiling works out of the box for unprivileged users, in pods and on the nodes:

  • Nsight Compute (ncu) — GPU hardware counters are unlocked (NVreg_RestrictProfilingToAdminUsers=0), so kernels can be profiled without root or SYS_ADMIN.
  • perf — kernel.perf_event_paranoid=1 and kernel.kptr_restrict=0 are set on the workers, so perf stat / perf top work for CPU-side profiling (data loading, launch overhead) inside pods.
  • dmesg is readable by unprivileged users for quick XID/hardware triage.