Vanilla / custom image¶

Instead of a managed orchestrator (Kubernetes or Slurm), you can deploy an Instant Cluster on a plain Ubuntu 24.04 image — or any other custom image. You get the same hardware — jump host, service node, and worker nodes wired together over InfiniBand — but the node operating system is exactly as the image ships. Nothing is installed or configured on top.

This is the right choice when you want full control of the software stack: your own drivers, scheduler, or container runtime.

What you set up yourself¶

On a managed orchestrator image these are done for you. On a vanilla image they are not, so the steps below are yours to run:

• Internet access for the worker and service nodes — only the jump host has a public IP. The other nodes route through it, so the jump host has to act as a NAT gateway.
• The shared filesystem — the SFS is offered to every node over virtiofs (tag home) but is not mounted until you add it to /etc/fstab.
• NVIDIA GPU drivers and NVIDIA networking (DOCA / OFED) drivers — needed before the GPUs and InfiniBand fabric are usable.

1. Give the worker and service nodes Internet access¶

The worker and service nodes have no public IP; their default route points at the jump host over the internal cluster network (eth1 on the jump host, eth0 on the other nodes). For them to reach the Internet — to install packages or pull drivers — the jump host has to forward and masquerade their traffic.

Run this on the jump host:

# Find your internal cluster subnet (the eth1 address, e.g. 10.13.118.100/24)
ip -4 addr show eth1

# Enable IPv4 forwarding (persist across reboots)
echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-cluster-nat.conf
sudo sysctl -w net.ipv4.ip_forward=1

# Masquerade cluster traffic out of the public interface.
# Replace 10.13.118.0/24 with your internal subnet from the command above.
sudo iptables -t nat -A POSTROUTING -s 10.13.118.0/24 -o eth0 -j MASQUERADE

# The internal network uses jumbo frames (MTU 9000) while the uplink is MTU 1500.
# Clamp the TCP MSS so large responses don't stall on PMTU discovery.
sudo iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN \
    -j TCPMSS --clamp-mss-to-pmtu

Verify from a worker node:

ssh WORKER_NAME curl -sI https://verda.com | head -n1

2. Mount the shared filesystem¶

Your shared filesystem is attached to every node over virtiofs with the tag home, but on a vanilla image it is not mounted. Add it to /etc/fstab on every node (jump host, service node, and each worker):

echo 'home  /home  virtiofs  defaults  0  0' | sudo tee -a /etc/fstab
sudo mount /home

Confirm it is mounted:

mount | grep /home

3. Install the NVIDIA GPU drivers¶

Install the data center GPU driver, CUDA toolkit, and Fabric Manager on every worker node. NVIDIA's network repository is the recommended source — follow the official guide:

• NVIDIA Driver Installation Guide for Ubuntu
• CUDA Downloads (Linux → x86_64 → Ubuntu → 24.04 → deb (network)) — we recommend CUDA 13.0.

Once installed and rebooted, verify with nvidia-smi.

4. Install the NVIDIA networking (DOCA / OFED) drivers¶

The InfiniBand fabric needs the NVIDIA networking drivers (the doca-ofed package, formerly MLNX_OFED), which include the drivers, tools, and libraries. The NVIDIA DOCA download page generates the exact repository commands for Ubuntu 24.04:

• NVIDIA DOCA Installation Guide for Linux

Install on every worker node, reboot, and verify the InfiniBand ports with ibstat. You should see eight ports, all Active at NDR (or faster).