Skip to content

Getting started

Accessing the cluster

SSH into the jumphost and use kubectl. Admin credentials are pre-configured at:

/root/.kube/config
/home/ubuntu/.kube/config

k9s and helm are also available out of the box. Verify access with:

kubectl get nodes

You should see the service node (control plane) and your GPU workers in Ready status. Confirm the GPUs are schedulable:

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Each worker should report 8.

Working from your own machine

You can copy the kubeconfig to your workstation and replace the server address with an SSH tunnel through the jumphost:

scp ubuntu@<jumphost-ip>:.kube/config ~/.kube/verda-cluster.yaml
ssh -L 6443:<cluster-name>-service:6443 ubuntu@<jumphost-ip>   # keep open
# then edit ~/.kube/verda-cluster.yaml: server: https://127.0.0.1:6443
KUBECONFIG=~/.kube/verda-cluster.yaml kubectl get nodes

Running your first job: NCCL all_reduce test

A pre-configured example job is available on the jumphost. It runs an NCCL all_reduce_perf benchmark across 2 nodes — the standard way to verify that the cluster's InfiniBand networking is healthy. The example is rendered for your cluster's hardware (including the correct NCCL_IB_PKEY setting — see InfiniBand and NCCL configuration). A full-cluster variant that spans every worker node is provided alongside it.

Submit the job:

$ kubectl create -f /home/ubuntu/verda_k8s_all_reduce_perf_2_nodes.yml
mpijob.kubeflow.org/nccl-test-2n-wcq4s created

Check pod status:

$ kubectl get pods
NAME                                READY   STATUS      RESTARTS   AGE
nccl-test-2n-8cnbc-launcher-przrf   0/1     Completed   4          30m

Downloading the container image on all workers may take a few minutes on first run.

View the results:

$ kubectl logs -f nccl-test-2n-8cnbc-launcher-przrf | tail -10

  4294967296    1073741824     float     sum      -1  9230.25  465.31  872.46       0  9221.35  465.76  873.31       0
  8589934592    2147483648     float     sum      -1  18376.5  467.44  876.45       0  18337.6  468.43  878.31       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 827.441
#
# Collective test concluded: all_reduce_perf
#

=== NCCL test completed ===

Understanding the output: the key metric is Avg bus bandwidth — how fast GPUs collectively communicate across the InfiniBand fabric. A healthy cluster reaches 800+ GB/s for large message sizes. Significantly lower numbers may indicate a network issue (bad cable, misconfigured IB port, or a wrong P_Key). The #wrong columns should read 0 — anything else indicates data corruption and warrants a support ticket.

Monitoring jobs

Use standard kubectl commands to monitor your workloads:

# List all pods and their status
kubectl get pods

# Follow logs from a specific pod
kubectl logs -f <pod-name>

# Describe a pod for detailed status and events
kubectl describe pod <pod-name>

# List all MPIJobs
kubectl get mpijobs

# Live resource usage (served by the pre-installed metrics-server)
kubectl top nodes
kubectl top pods

For cluster-level dashboards, health checks and the Kubernetes Dashboard web UI, see Observability.

Container registry

Your container images are pulled by containerd on the worker nodes. It is recommended to use authenticated access when pulling images.

Verda provides a managed container registry — see Container Registries for setup instructions. To pull from a private registry, create a pull secret and reference it in your pod specs:

kubectl create secret docker-registry my-registry \
  --docker-server=vccr.io \
  --docker-username=<user> \
  --docker-password=<token>

spec:
  imagePullSecrets:
    - name: my-registry

For quick experimentation, public images from NVIDIA NGC (nvcr.io/nvidia/...) and Docker Hub can also be used.

Next steps