Architecture¶
A standard cluster has three node roles: a single login node (jumphost / bastion), a single service node, and N worker nodes (up to 16, each with 8 GPUs). The login node is the only node reachable from the internet - it is the SSH entry point and the NAT gateway for everything behind it. Worker nodes are interconnected by a high-speed InfiniBand fabric (exact topology differs between locations) and all nodes share a /home filesystem.
graph TD
Internet([Internet / User])
Internet -->|"SSH :22 · HTTPS :443"| Login
subgraph cluster [Cluster private network]
Login["Login node<br/>jumphost / bastion<br/>NAT gateway · Nginx → Grafana"]
Service["Service node<br/>auth.cluster.verda.internal<br/>Slurm controller·<br/>k8s control plane · monitoring"]
W1["Worker 1<br/>8 GPUs · local NVMe"]
W2["Worker 2<br/>8 GPUs · local NVMe"]
Wn["Worker N<br/>(up to 16)<br/>8 GPUs · local NVMe"]
IB["InfiniBand fabric<br/>leaf / spine switches<br/>(topology varies by location)"]
Login --- Service
Login --- W1
Login --- W2
Login --- Wn
W1 -. InfiniBand .- IB
W2 -. InfiniBand .- IB
Wn -. InfiniBand .- IB
end
The Slurm controller, the Kubernetes control plane and the monitoring/observability stack run on the service node, not on the login host. Worker nodes use the login node as their default gateway and NAT firewall. The Grafana UI is reachable at https://<login-ip>:443 (see Monitoring).
Node naming¶
Cluster node names are based on the Hostname you specify when creating the cluster:
- Login / jump host:
hostname-login(still labeled as jumphost in the Console and API) - Service node:
hostname-service, also reachable asauth.cluster.verda.internal - Worker nodes:
hostname-1,hostname-2, etc.
Storage¶
- A shared network filesystem is mounted at
/homeon every node of the cluster. It is created fresh with the cluster; anything that must outlive the cluster belongs on a shared filesystem you keep and re-attach. - Each worker node has a local NVMe drive for fast scratch I/O, mounted at
/mnt/local_diskon the node. On Kubernetes clusters it backs thelocal-diskandlocal-path(default) StorageClasses, alongside the RWX-capableshared-pathclass on the shared filesystem - see Storage. On Slinky clusters the same drive backs/localand/tmpinside jobs, see Job environment.
File locks on /home work across nodes
flock() and fcntl() locks on /home, and on any other cluster shared filesystem, exclude processes on every node of the cluster. Each lock call is a network round trip of about 0.25 ms. A blocking lock that is still waiting after 60 seconds fails with ENOLCK ("No locks available"), and the waiting process cannot be killed until then, so a cancelled Slurm job can stay in COMPLETING for up to a minute. For concurrent Hugging Face downloads from several nodes into one shared cache, use huggingface_hub 1.18.0 or newer: older versions take a lock for every file and queue behind each other.
Keep SQLite databases and caches on local disk
Every SQLite commit on /home waits for a network flush: in our tests about 10 ms per commit with SQLite's default settings (about 1 ms in WAL mode), against well under 1 ms on /mnt/local_disk. How much that slows an application depends on how often it commits. MLflow's log_metric took 9 ms instead of 1.4 ms, and a diskcache lookup 0.25 ms instead of 0.003 ms. This includes Optuna and MLflow with a sqlite:/// backend, and caches built on diskcache such as DSPy's (~/.dspy_cache, moved with DSPY_CACHEDIR). Put them on /mnt/local_disk where you can. Never share a database in WAL mode between nodes: SQLite's WAL only works when every process using the database is on the same machine.
Retry open() on \"Stale file handle\"
If one node replaces a file by renaming a new file over it while a process on another node opens it, that open() can fail with ESTALE ("Stale file handle"); in our tests, under 1% of opens under constant rewrites. Retry the open; the next attempt reads the new file.
Preinstalled software¶
CUDA, doca-ofed and the NVIDIA drivers are installed on each server. HPC-X lives in /opt/hpcx on the nodes and provides MPI (/opt/hpcx/ompi/bin/mpirun). A PyTorch environment helper (/usr/local/bin/pytorch.setup.sh) installs uv and PyTorch into a virtualenv.
On Slinky clusters, jobs run in the worker pods, which have their own software (no HPC-X or Lmod) and Python: see Job environment. Run Python workloads in jobs in a container, not with pytorch.setup.sh on the login node; see Python environments.