Skip to content

Networking and ports

Default open ports (login node)

The login node firewall (configured from /etc/default/verda_iptables) drops all inbound traffic on the public interface by default, except for the ports below. Traffic from the internal cluster network and ICMP (ping) are always allowed.

Port Purpose
22 (TCP) SSH — bastion shell. Also the seamless-SSH redirect target if SEAMLESS_SSH_PORT=22.
80 (TCP) HTTP — Let's Encrypt ACME (HTTP-01) challenge and redirect to 443.
443 (TCP) HTTPS — Grafana via Nginx.
2222 (TCP) SSH — allowed by the firewall, but nothing listens unless seamless SSH is enabled (opt-in, Slinky clusters).

Worker and service nodes are not reachable from the internet by default; reach them by SSHing to the login node first (or see external SSH to worker nodes).

External SSH to worker nodes (optional)

By default the login node only NATs outbound traffic from workers — workers are not reachable from the internet. The usual path is to SSH to the login node and then to a worker by name (e.g. ssh hostname-1).

If you want to reach worker SSH directly from outside the cluster, enable DNAT on the login node:

  1. On the login node, edit /etc/default/verda_iptables and set WORKER_SSH_DNAT=1.
  2. Apply: systemctl restart iptables-custom

The login node will then forward <login-ip>:1000N to hostname-N:22. For example, <login-ip>:10001 → hostname-1, <login-ip>:10002 → hostname-2, and so on.

Warning

Enabling DNAT exposes worker SSH to the public internet. Make sure each worker's sshd only accepts key-based authentication.

Infiniband partitioning

Worker nodes are interconnected using a partitioned 400 Gb/s Infiniband fabric with M_KEY. For this reason commands like ibhosts will not work, while distributed workloads like MPI work correctly.

On B300 clusters the partition key is assigned at index 0, so Infiniband and NCCL work from inside a Docker container without any extra configuration.

On H200 and B200 clusters, to use Infiniband and NCCL from inside a Docker container make sure to set environment variable NCCL_IB_PKEY=1.

For example:

docker run -e NCCL_IB_PKEY=1