---
description: Deploying an Instant Cluster on a vanilla Ubuntu 24.04 (or other custom) image, and the manual setup steps you take care of yourself.
revision_date: 16.07.2026
---

# Vanilla / custom image

Instead of a managed orchestrator (Kubernetes or Slurm), you can deploy an
Instant Cluster on a plain **Ubuntu 24.04** image — or any other custom image.
You get the same hardware — jump host, service node, and worker nodes wired
together over InfiniBand — but the node operating system is exactly as the image
ships. Nothing is installed or configured on top.

This is the right choice when you want full control of the software stack: your
own drivers, scheduler, or container runtime.

!!! info
    At this time only the **Ubuntu 24.04 minimal** image is available. For other
    Ubuntu versions, [get in touch with our support](https://docs.verda.com/welcome-to-verda/contact-support/?h=support).

!!! info
    On a vanilla image the only account is `root` — there is no `ubuntu` user.
    Log in to the jump host with `ssh root@CLUSTER_IP` (using the command from
    the **Clusters** screen), and reach the worker nodes from the jump host with
    `ssh WORKER_NAME`.

## What you set up yourself

On a managed orchestrator image these are done for you. On a vanilla image they
are not, so the steps below are yours to run:

- **Internet access for the worker and service nodes** — only the jump host has
  a public IP. The other nodes route through it, so the jump host has to act as
  a NAT gateway.
- **The shared filesystem** — the SFS is offered to every node over virtiofs
  (tag `home`) but is not mounted until you add it to `/etc/fstab`.
- **NVIDIA GPU drivers** and **NVIDIA networking (DOCA / OFED) drivers** — needed
  before the GPUs and InfiniBand fabric are usable.

!!! info
    The cluster reaches **running** status in under ~7 minutes, at which point
    the jump host and service node are up. The worker nodes take a few minutes
    longer to finish booting — give them time before you log in to them.

---

## 1. Give the worker and service nodes Internet access

The worker and service nodes have no public IP; their default route points at
the jump host over the internal cluster network (`eth1` on the jump host,
`eth0` on the other nodes). For them to reach the Internet — to install packages
or pull drivers — the jump host has to forward and masquerade their traffic.

Run this **on the jump host**:

```bash
# Find your internal cluster subnet (the eth1 address, e.g. 10.13.118.100/24)
ip -4 addr show eth1

# Enable IPv4 forwarding (persist across reboots)
echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-cluster-nat.conf
sudo sysctl -w net.ipv4.ip_forward=1

# Masquerade cluster traffic out of the public interface.
# Replace 10.13.118.0/24 with your internal subnet from the command above.
sudo iptables -t nat -A POSTROUTING -s 10.13.118.0/24 -o eth0 -j MASQUERADE

# The internal network uses jumbo frames (MTU 9000) while the uplink is MTU 1500.
# Clamp the TCP MSS so large responses don't stall on PMTU discovery.
sudo iptables -t mangle -A FORWARD -o eth0 -p tcp --tcp-flags SYN,RST SYN \
    -j TCPMSS --clamp-mss-to-pmtu
```

Verify from a worker node:

```bash
ssh WORKER_NAME curl -sI https://verda.com | head -n1
```

!!! info
    `iptables` rules are not persistent by default. Re-run them after a reboot,
    or install `iptables-persistent` (`sudo apt-get install iptables-persistent`)
    to save and restore them automatically.

---

## 2. Mount the shared filesystem

Your [shared filesystem](https://docs.verda.com/storage/shared-filesystems-sfs/shared-filesystem-for-a-cluster/)
is attached to every node over virtiofs with the tag `home`, but on a vanilla
image it is not mounted. Add it to `/etc/fstab` **on every node** (jump host,
service node, and each worker):

```bash
echo 'home  /home  virtiofs  defaults  0  0' | sudo tee -a /etc/fstab
sudo mount /home
```

Confirm it is mounted:

```bash
mount | grep /home
```

---

## 3. Install the NVIDIA GPU drivers

Install the data center GPU driver, CUDA toolkit, and Fabric Manager on **every
worker node**. NVIDIA's network repository is the recommended source — follow
the official guide:

- [NVIDIA Driver Installation Guide for Ubuntu](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html)
- [CUDA Downloads](https://developer.nvidia.com/cuda-downloads) (Linux → x86_64 → Ubuntu → 24.04 → deb (network)) — we recommend CUDA 13.0.

Once installed and rebooted, verify with `nvidia-smi`.

---

## 4. Install the NVIDIA networking (DOCA / OFED) drivers

The InfiniBand fabric needs the NVIDIA networking drivers (the `doca-ofed`
package, formerly MLNX_OFED), which include the drivers, tools, and libraries.
The [NVIDIA DOCA download page](https://developer.nvidia.com/doca-downloads)
generates the exact repository commands for Ubuntu 24.04:

- [NVIDIA DOCA Installation Guide for Linux](https://docs.nvidia.com/doca/sdk/nvidia+doca+installation+guide+for+linux/index.html)

Install on every worker node, reboot, and verify the InfiniBand ports with
`ibstat`. You should see eight ports, all `Active` at NDR (or faster).

!!! tip
    Running the same setup on 16 worker nodes is tedious by hand. Once the jump
    host has Internet access (step 1), loop over the workers from it, e.g.
    `for n in $(seq 1 16); do ssh CLUSTER_NAME-$n 'sudo ...'; done`, or use a
    configuration tool such as Ansible.