Skip to content

Networking and ports

Default open ports (login node)

The login node firewall (configured from /etc/default/verda_iptables) drops all inbound traffic on the public interface by default, except for the ports below. Traffic from the internal cluster network and ICMP (ping) are always allowed.

Port Purpose
22 (TCP) SSH - login node shell. On Slinky clusters, the Slurm login pod instead (see How users log in).
80 (TCP) HTTP - Let's Encrypt ACME (HTTP-01) challenge and redirect to 443.
443 (TCP) HTTPS - Grafana via Nginx.
2222 (TCP) SSH - on Slinky clusters, the login node admin shell. Not used on other clusters.

Worker and service nodes are not reachable from the internet by default; reach them by SSHing to the login node first (or see external SSH to worker nodes).

Limit access by source IP

On the login VM, edit /etc/default/verda_iptables. These settings need cluster-k8s 1.1.20 or later. Check that they exist in your configuration first.

For example, to allow SSH from one IP and an office subnet, and Grafana from the office subnet only:

SSH_ALLOW="203.0.113.10 198.51.100.0/24"
HTTPS_ALLOW="198.51.100.0/24"
HTTP_ALLOW="0.0.0.0/0"

Replace the example addresses with your public IP or VPN exit range. SSH_ALLOW covers ports 22, 2222, and enabled worker SSH ports. HTTPS_ALLOW covers Grafana on port 443. Use spaces or commas between IPv4 addresses or CIDR ranges. 0.0.0.0/0 allows any source; an empty value blocks new public connections.

Warning

Keep HTTP port 80 open for Let's Encrypt certificate renewal. Restricting HTTP_ALLOW can prevent renewal.

Apply allowlist changes with automatic rollback

A timer can restore the old configuration if you lose SSH access. Keep your current session open. Do not reboot during these steps: the timer and backup are temporary.

  1. Connect to the login VM and start a root shell. On Slinky, use port 2222:

    ssh -p 2222 ubuntu@<cluster-ip>
    sudo -i
    

    On a Kubernetes-only cluster, omit -p 2222. Run the admin commands below in this root shell.

  2. Back up the configuration and start a five-minute rollback timer:

    backup_dir=$(mktemp -d /run/verda-firewall.XXXXXX)
    rollback_unit="verda-firewall-rollback-$(date +%s)"
    cp -a /etc/default/verda_iptables "$backup_dir/verda_iptables" && \
    systemd-run --unit="$rollback_unit" --on-active=5m \
      --timer-property=AccuracySec=1s \
      /bin/sh -c 'cp -a "$1" /etc/default/verda_iptables && systemctl restart iptables-custom' \
      sh "$backup_dir/verda_iptables"
    
    systemctl is-active "$rollback_unit.timer"
    

    The last command must print active. If any command fails, stop before you edit the configuration.

  3. Edit the allowlists, check the shell syntax, and apply:

    nano /etc/default/verda_iptables
    bash -n /etc/default/verda_iptables && systemctl restart iptables-custom
    
  4. Before the timer runs, test a new SSH connection from another terminal. On Slinky, use ssh -S none -p 2222 ubuntu@<cluster-ip> and also check port 22. Check Grafana if you changed its allowlist. -S none prevents reuse of an existing connection. An existing session can stay open even if the new rules block your IP.

    If a new connection fails, leave the timer running. Wait for it to restore the old configuration, then try again.

  5. If the checks pass within five minutes, cancel the timer from the original root shell:

    systemctl stop "$rollback_unit.timer"
    

    If five minutes have passed, check the configuration first. The rollback may already have restored the backup.

External SSH to worker nodes (optional)

Workers are not reachable from the internet by default. The usual path is to SSH to the login node and then to a worker by name (e.g. ssh hostname-1).

If you want to reach worker SSH directly from outside the cluster, enable DNAT on the login node:

  1. On the login node, edit /etc/default/verda_iptables and set WORKER_SSH_DNAT=1.
  2. Apply: systemctl restart iptables-custom

The login node will then forward <login-ip>:1000N to hostname-N:22. For example, <login-ip>:10001 → hostname-1, <login-ip>:10002 → hostname-2, and so on.

Warning

Enabling DNAT exposes worker SSH to the public internet. Make sure each worker's sshd only accepts key-based authentication.

Infiniband partitioning

Worker nodes are interconnected using a partitioned 400 Gb/s Infiniband fabric with M_KEY. For this reason commands like ibhosts will not work, while distributed workloads like MPI work correctly.

On B300 clusters the partition key is assigned at index 0, so Infiniband and NCCL work from inside a Docker container without any extra configuration.

On H200 and B200 clusters, to use Infiniband and NCCL from inside a Docker container make sure to set environment variable NCCL_IB_PKEY=1.

For example:

docker run -e NCCL_IB_PKEY=1