Networking and ports¶
Default open ports (login node)¶
The login node firewall (configured from /etc/default/verda_iptables) drops all inbound traffic on the public interface by default, except for the ports below. Traffic from the internal cluster network and ICMP (ping) are always allowed.
| Port | Purpose |
|---|---|
22 (TCP) |
SSH - login node shell. On Slinky clusters, the Slurm login pod instead (see How users log in). |
80 (TCP) |
HTTP - Let's Encrypt ACME (HTTP-01) challenge and redirect to 443. |
443 (TCP) |
HTTPS - Grafana via Nginx. |
2222 (TCP) |
SSH - on Slinky clusters, the login node admin shell. Not used on other clusters. |
Worker and service nodes are not reachable from the internet by default; reach them by SSHing to the login node first (or see external SSH to worker nodes).
Limit access by source IP¶
On the login VM, edit /etc/default/verda_iptables. These settings need cluster-k8s 1.1.20 or later. Check that they exist in your configuration first.
For example, to allow SSH from one IP and an office subnet, and Grafana from the office subnet only:
Replace the example addresses with your public IP or VPN exit range. SSH_ALLOW covers ports 22, 2222, and enabled worker SSH ports. HTTPS_ALLOW covers Grafana on port 443. Use spaces or commas between IPv4 addresses or CIDR ranges. 0.0.0.0/0 allows any source; an empty value blocks new public connections.
Warning
Keep HTTP port 80 open for Let's Encrypt certificate renewal. Restricting HTTP_ALLOW can prevent renewal.
Apply allowlist changes with automatic rollback¶
A timer can restore the old configuration if you lose SSH access. Keep your current session open. Do not reboot during these steps: the timer and backup are temporary.
-
Connect to the login VM and start a root shell. On Slinky, use port
2222:On a Kubernetes-only cluster, omit
-p 2222. Run the admin commands below in this root shell. -
Back up the configuration and start a five-minute rollback timer:
backup_dir=$(mktemp -d /run/verda-firewall.XXXXXX) rollback_unit="verda-firewall-rollback-$(date +%s)" cp -a /etc/default/verda_iptables "$backup_dir/verda_iptables" && \ systemd-run --unit="$rollback_unit" --on-active=5m \ --timer-property=AccuracySec=1s \ /bin/sh -c 'cp -a "$1" /etc/default/verda_iptables && systemctl restart iptables-custom' \ sh "$backup_dir/verda_iptables" systemctl is-active "$rollback_unit.timer"The last command must print
active. If any command fails, stop before you edit the configuration. -
Edit the allowlists, check the shell syntax, and apply:
-
Before the timer runs, test a new SSH connection from another terminal. On Slinky, use
ssh -S none -p 2222 ubuntu@<cluster-ip>and also check port22. Check Grafana if you changed its allowlist.-S noneprevents reuse of an existing connection. An existing session can stay open even if the new rules block your IP.If a new connection fails, leave the timer running. Wait for it to restore the old configuration, then try again.
-
If the checks pass within five minutes, cancel the timer from the original root shell:
If five minutes have passed, check the configuration first. The rollback may already have restored the backup.
External SSH to worker nodes (optional)¶
Workers are not reachable from the internet by default. The usual path is to SSH to the login node and then to a worker by name (e.g. ssh hostname-1).
If you want to reach worker SSH directly from outside the cluster, enable DNAT on the login node:
- On the login node, edit
/etc/default/verda_iptablesand setWORKER_SSH_DNAT=1. - Apply:
systemctl restart iptables-custom
The login node will then forward <login-ip>:1000N to hostname-N:22. For example, <login-ip>:10001 → hostname-1, <login-ip>:10002 → hostname-2, and so on.
Warning
Enabling DNAT exposes worker SSH to the public internet. Make sure each worker's sshd only accepts key-based authentication.
Infiniband partitioning¶
Worker nodes are interconnected using a partitioned 400 Gb/s Infiniband fabric with M_KEY. For this reason commands like ibhosts will not work, while distributed workloads like MPI work correctly.
On B300 clusters the partition key is assigned at index 0, so Infiniband and NCCL work from inside a Docker container without any extra configuration.
On H200 and B200 clusters, to use Infiniband and NCCL from inside a Docker container make sure to set environment variable NCCL_IB_PKEY=1.
For example: