Skip to content

Observability

Grafana dashboards

Every cluster ships a Grafana instance on the service node, backed by a VictoriaMetrics stack — access and login details are described in Monitoring. On Kubernetes clusters, the most relevant folders are:

  • Kubernetes folder — cluster, node, pod and workload views (kube-state-metrics, kubelet/cAdvisor, control-plane metrics).
  • Health Checks folder — results of the automatic health checks (below). The Cluster Active Health Check Overview is the index: a registry table showing every check's last run, result and freshness, plus a per-node table — click a node's Instance cell to drill into per-node details. Inter-node NCCL results, the weekly training benchmark and cluster-wide trends are one click away via the header links.
  • The common dashboards: GPU Overview, GPUd Overview, NVIDIA DCGM Exporter, Node Exporter, and Cluster Log Explorer for centralized logs.

The metrics datasource is Prometheus-compatible — your own workloads can be scraped by adding a VMServiceScrape/VMPodScrape resource, and custom dashboards work as on any Grafana.

Kubernetes Dashboard

The Kubernetes Dashboard web UI is pre-installed and exposed as NodePort 32443 (HTTPS) on every node. Reach it through an SSH tunnel via the jumphost:

ssh -L 8443:<cluster-name>-service:32443 ubuntu@<jumphost-ip>

then browse to https://localhost:8443 (accept the self-signed certificate). Log in with the pre-created admin token:

kubectl -n kubernetes-dashboard get secret dashboard-admin-token \
  -o jsonpath='{.data.token}' | base64 -d

Automatic health checks

Recurring health checks watch the cluster continuously — see Ongoing health checks for the full list. On Kubernetes clusters:

  • The 6-hourly active suite runs as a CronJob in the monitoring namespace, one GPU-requesting pod per idle node (it queues behind and never preempts your workloads). The first sweep runs minutes after provisioning.
  • The inter-node NCCL AllReduce check runs 6-hourly as two-node MPIJobs across rotating node pairs, catching InfiniBand fabric degradation between full sweeps.
  • Passive monitoring (gpud, DCGM exporter, the passive suite) runs continuously on every node; hardware alerts are forwarded to Verda automatically.

All results land in the Grafana Health Checks folder.

# see the health-check workloads themselves
kubectl get cronjobs,jobs,mpijobs -n monitoring

Live resource usage

The pre-installed metrics-server backs the standard resource commands:

kubectl top nodes
kubectl top pods -A

Profiling

Profiling works out of the box for unprivileged users, in pods and on the nodes:

  • Nsight Compute (ncu) — GPU hardware counters are unlocked (NVreg_RestrictProfilingToAdminUsers=0), so kernels can be profiled without root or SYS_ADMIN.
  • perf — kernel.perf_event_paranoid=1 and kernel.kptr_restrict=0 are set on the workers, so perf stat / perf top work for CPU-side profiling (data loading, launch overhead) inside pods.
  • dmesg is readable by unprivileged users for quick XID/hardware triage.