---
description: Observability on a Kubernetes Instant Cluster — Grafana dashboards, automatic health checks, and GPU/CPU profiling.
revision_date: 27.07.2026
---

# Observability

## Grafana dashboards

Every cluster ships a Grafana instance on the service node, backed by a VictoriaMetrics stack — access and login details are described in [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/). On Kubernetes clusters, the most relevant folders are:

* **Kubernetes folder** — cluster, node, pod and workload views (kube-state-metrics, kubelet/cAdvisor, control-plane metrics).
* **Health Checks folder** — results of the automatic [health checks](https://docs.verda.com/clusters/instant-clusters/kubernetes/health-checks/). The *Cluster Active Health Check Overview* is the index: select a **Check** name to open its corresponding dashboard, or select a node's **Instance** to drill into per-node details.
* The common dashboards: *GPU Overview*, *GPUd Overview*, *NVIDIA DCGM Exporter*, *Node Exporter*, and *Cluster Log Explorer* for centralized logs.

The metrics datasource is Prometheus-compatible — your own workloads can be scraped by adding a `VMServiceScrape`/`VMPodScrape` resource, and custom dashboards work as on any Grafana.

## Health checks

Passive node agents run continuously, while active node, fabric, storage, training, and inference checks run as scheduled Kubernetes workloads in the `monitoring` namespace. GPU checks request resources through the normal scheduler and do not preempt customer workloads.

See [Health checks](https://docs.verda.com/clusters/instant-clusters/kubernetes/health-checks/) for the full check catalog, dashboard statuses, Kubernetes resources, manual runs, and troubleshooting commands.

## Live resource usage

The pre-installed [metrics-server](https://github.com/kubernetes-sigs/metrics-server) backs the standard resource commands:

```bash
kubectl top nodes
kubectl top pods -A
```

## Profiling

Profiling works out of the box for unprivileged users, in pods and on the nodes:

- **Nsight Compute (`ncu`)** — GPU hardware counters are unlocked (`NVreg_RestrictProfilingToAdminUsers=0`), so kernels can be profiled without root or `SYS_ADMIN`.
- **`perf`** — `kernel.perf_event_paranoid=1` and `kernel.kptr_restrict=0` are set on the workers, so `perf stat` / `perf top` work for CPU-side profiling (data loading, launch overhead) inside pods.
- **`dmesg`** is readable by unprivileged users for quick XID/hardware triage.