---
description: Node roles, naming, storage layout and preinstalled software of a Verda instant cluster.
revision_date: 12.11.2026
---

# Architecture

A standard cluster has three node roles: a single **login node** (jumphost / bastion), a single **service node**, and **N worker nodes** (up to 16, each with 8 GPUs). The login node is the only node reachable from the internet - it is the SSH entry point and the NAT gateway for everything behind it. Worker nodes are interconnected by a high-speed InfiniBand fabric (exact topology differs between locations) and all nodes share a `/home` filesystem.

```mermaid
graph TD
    Internet([Internet / User])
    Internet -->|"SSH :22 · HTTPS :443"| Login

    subgraph cluster [Cluster private network]
        Login["Login node<br/>jumphost / bastion<br/>NAT gateway · Nginx → Grafana"]
        Service["Service node<br/>auth.cluster.verda.internal<br/>Slurm controller·<br/>k8s control plane · monitoring"]
        W1["Worker 1<br/>8 GPUs · local NVMe"]
        W2["Worker 2<br/>8 GPUs · local NVMe"]
        Wn["Worker N<br/>(up to 16)<br/>8 GPUs · local NVMe"]
        IB["InfiniBand fabric<br/>leaf / spine switches<br/>(topology varies by location)"]

        Login --- Service
        Login --- W1
        Login --- W2
        Login --- Wn
        W1 -. InfiniBand .- IB
        W2 -. InfiniBand .- IB
        Wn -. InfiniBand .- IB
    end
```

The Slurm controller, the Kubernetes control plane and the monitoring/observability stack run on the service node, not on the login host. Worker nodes use the login node as their default gateway and NAT firewall. The Grafana UI is reachable at `https://<login-ip>:443` (see [Monitoring](https://docs.verda.com/clusters/cluster-basics/monitoring/)).

## Node naming

Cluster node names are based on the `Hostname` you specify when creating the cluster:

* Login / jump host: `hostname-login` (still labeled as **jumphost** in the Console and API)
* Service node: `hostname-service`, also reachable as `auth.cluster.verda.internal`
* Worker nodes: `hostname-1`, `hostname-2`, etc.

## Storage

* A shared network filesystem is mounted at **`/home`** on every node of the cluster. It is created fresh with the cluster; anything that must outlive the cluster belongs on a shared filesystem you keep and re-attach.
* Each worker node has a local NVMe drive for fast scratch I/O, mounted at **`/mnt/local_disk`** on the node. On Kubernetes clusters it backs the `local-disk` and `local-path` (default) StorageClasses, alongside the RWX-capable `shared-path` class on the shared filesystem - see [Storage](https://docs.verda.com/clusters/kubernetes/storage/). On Slinky clusters the same drive backs **`/local`** and **`/tmp`** inside jobs, see [Job environment](https://docs.verda.com/clusters/slinky/job-environment/#storage-inside-a-job).

!!! info "File locks on `/home` work across nodes"
    `flock()` and `fcntl()` locks on `/home`, and on any other cluster shared filesystem, exclude processes on every node of the cluster. Each lock call is a network round trip of about 0.25 ms. A blocking lock that is still waiting after 60 seconds fails with `ENOLCK` ("No locks available"), and the waiting process cannot be killed until then, so a cancelled Slurm job can stay in `COMPLETING` for up to a minute. For concurrent Hugging Face downloads from several nodes into one shared cache, use `huggingface_hub` 1.18.0 or newer: older versions take a lock for every file and queue behind each other.

!!! tip "Keep SQLite databases and caches on local disk"
    Every SQLite commit on `/home` waits for a network flush: in our tests about 10 ms per commit with SQLite's default settings (about 1 ms in WAL mode), against well under 1 ms on `/mnt/local_disk`. How much that slows an application depends on how often it commits. MLflow's `log_metric` took 9 ms instead of 1.4 ms, and a `diskcache` lookup 0.25 ms instead of 0.003 ms. This includes Optuna and MLflow with a `sqlite:///` backend, and caches built on `diskcache` such as DSPy's (`~/.dspy_cache`, moved with `DSPY_CACHEDIR`). Put them on `/mnt/local_disk` where you can. Never share a database in WAL mode between nodes: SQLite's WAL only works when every process using the database is on the same machine.

!!! tip "Retry `open()` on \"Stale file handle\""
    If one node replaces a file by renaming a new file over it while a process on another node opens it, that `open()` can fail with `ESTALE` ("Stale file handle"); in our tests, under 1% of opens under constant rewrites. Retry the open; the next attempt reads the new file.

## Preinstalled software

CUDA, `doca-ofed` and the NVIDIA drivers are installed on each server. [HPC-X](https://developer.nvidia.com/networking/hpc-x) lives in `/opt/hpcx` on the nodes and provides MPI (`/opt/hpcx/ompi/bin/mpirun`). A PyTorch environment helper (`/usr/local/bin/pytorch.setup.sh`) installs `uv` and PyTorch into a virtualenv.

On Slinky clusters, jobs run in the worker pods, which have their own software (no HPC-X or Lmod) and Python: see [Job environment](https://docs.verda.com/clusters/slinky/job-environment/). Run Python workloads in jobs in a container, not with `pytorch.setup.sh` on the login node; see [Python environments](https://docs.verda.com/clusters/slinky/job-environment/#python-environments).
