Vanilla / custom image¶

If you prefer full control of the software stack (your own drivers, scheduler, and container runtime), you can deploy a vanilla Ubuntu 24.04 image (or other custom image) on an Instant Cluster. The cluster provides the same hardware that you receive from the managed Kubernetes or Slinky clusters: login node, service node, and worker nodes wired together over InfiniBand. You need to manually set up the operating environment: Internet access, shared filesystems, NVIDIA GPU drivers, CUDA, and NVIDIA networking drivers. This guide explains how to configure them.

What you set up yourself¶

• Internet access for the worker and service nodes, only the login node has a public IP. The other nodes route through it, so the login node has to act as a NAT gateway.
• The shared filesystem, the SFS is exposed to every node over virtiofs (tag home) but is not mounted until you add it to /etc/fstab.
• NVIDIA GPU drivers and NVIDIA networking (DOCA / OFED) drivers, needed for the GPUs and InfiniBand fabric to be usable.

1. Give the worker and service nodes Internet access¶

The worker and service nodes have no public IP; their default route points at the login node over the internal cluster network (eth1 on the login node, eth0 on the other nodes). Since the worker and service nodes need to install packages or download drivers from the Internet, you must configure traffic forwarding and masquerading on the login node before proceeding.

Run this on the login node:

# Find your internal cluster subnet (the eth1 address, e.g. 10.13.118.100/24)
ip -4 addr show eth1

# Enable IPv4 forwarding (persist across reboots)
echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-cluster-nat.conf
sudo sysctl -w net.ipv4.ip_forward=1

# Masquerade cluster traffic out of the public interface.
# Replace 10.13.118.0/24 with your internal subnet from the command above.
sudo iptables -t nat -A POSTROUTING -s 10.13.118.0/24 -o eth0 -j MASQUERADE

# The internal network uses jumbo frames (MTU 9000) while the uplink is MTU 1500.
# Clamp the TCP MSS so large responses don't stall on PMTU discovery.
sudo iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN \
    -j TCPMSS --clamp-mss-to-pmtu

Verify from a worker or service node:

On the login node, find the internal IP address of a service or worker node:

# IP of the first worker node typically ends with 101,
# IP of the last worker node typically ends with #TOTAL_WORKER_NODES + 100
# IP of a service node typically ends with 200
ip neighbor show dev eth1

From your workstation, connect to a worker or service node through the login node:

ssh -J root@CLUSTER_IP root@NODE_INTERNAL_IP 'curl -sI https://verda.com | head -n1'

2. Mount the shared filesystem¶

Your shared filesystem is attached to every node over virtiofs with the tag home, but it is not mounted on a vanilla image. Add it to /etc/fstab on every node (login node, service node, and each worker):

echo 'home  /home  virtiofs  defaults  0  0' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /home

Confirm it is mounted:

mount | grep /home

3. Install the NVIDIA GPU drivers¶

Install the data center GPU driver, CUDA toolkit, and Fabric Manager on every worker node. Please follow the official guide:

• NVIDIA Driver Installation Guide for Ubuntu
• CUDA Downloads (Linux → x86_64 → Ubuntu → 24.04 → deb (network)). We recommend CUDA 13.0.

Once installed and rebooted, verify with nvidia-smi.

4. Install the NVIDIA networking (DOCA / OFED) drivers¶

The InfiniBand fabric needs the NVIDIA networking drivers (the doca-ofed package, formerly MLNX_OFED), which include the drivers, tools, and libraries. The NVIDIA DOCA download page (DOCA-Host → Linux → x86_64 → doca-ofed → Ubuntu → 24.04 → deb (online)) generates the exact repository commands for Ubuntu 24.04:

• NVIDIA DOCA Installation Guide for Linux

Install on every worker node, reboot, and verify the InfiniBand ports with ibstat. You should see eight ports, all Active at NDR (or faster).

The following packages have unmet dependencies:
doca-ofed : Depends: mft (>= 4.36.0-147) but 4.35.0.159-1 is to be installed
            Depends: doca-sosreport (= 4.9.0-6) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

# get the versions of mft and doca-ofed
apt-cache policy mft doca-ofed
# install mft and doca-ofed with specific versions
apt-get -y install mft=MFT_VERSION doca-ofed=DOCA_OFED_VERSION