Skip to content

Instant Cluster Release Notes

Changes to the Instant GPU Cluster offering: new GPU accelerators, orchestrator images, software upgrades, and infrastructure improvements.

November 2026

File locks on shared filesystems work across nodes

flock() and fcntl() locks on /home and on every other cluster shared filesystem now exclude processes on all nodes of the cluster, not only on the node that took the lock. Tools that coordinate through lock files, such as concurrent huggingface_hub 1.17 downloads into one shared cache, no longer lose updates or corrupt files when they run on several nodes.

A lock call is now a network round trip of about 0.25 ms; reads, writes, metadata operations and git are as fast as before. A blocking lock fails with ENOLCK after waiting 60 seconds, and a process cannot be killed while it waits. See Architecture.

This applies to newly created clusters. An existing cluster gets it the next time its nodes are shut down and started again; a reboot from inside the node is not enough.

October 2026

Slinky: Pyxis containers, Slurm 25.11.8 and no shared jail

On newly created Slinky clusters, job steps run directly in the worker's slurmd pod as the submitting user; the shared execution jail is retired. With it gone, Pyxis works: srun --container-image=... runs a container with your $HOME, /shared, /local, the requested GPUs and InfiniBand, so multi-node NCCL runs over the fabric with no extra flags. Every Slurm component runs 25.11.8, alongside Enroot 4.2.1, Pyxis 0.24 and Apptainer 1.5.5. The Slurm commands on the login node now submit as the calling user, so sbatch as ubuntu runs as ubuntu instead of root. See Job environment for what a job sees.

Enroot now runs only in job steps: the login pod has no enroot, and running it there prints how to import an image once into /shared with srun enroot import and start jobs from the .sqsh with --container-name, which keeps the unpacked image on the node's NVMe for fast restarts. Large images such as NGC PyTorch now import in a default-sized job. See Containers.

Known issue: a job can hang when it first reads files just written to the shared /home from another node, which mostly hits Python virtual environments. Until it is fixed, run Python workloads in a Pyxis container; see Python environments.

RDMA: InfiniBand devices only, unlimited locked memory in pods

On newly created Kubernetes and Slinky clusters, Slurm jobs, their containers and pods that request RDMA devices see only the InfiniBand HCAs. Newer nodes also have an Ethernet RDMA device (mlx5_0), which Open MPI/UCX could pick, failing MPI_Init with uct_iface_open(rc_verbs/mlx5_0:1) failed: Address not valid. NCCL and MPI now find the fabric without NCCL_IB_HCA or UCX_NET_DEVICES.

On Kubernetes clusters, containers running as a non-root user now get unlimited locked memory (ulimit -l) instead of 8 MiB, so NCCL can register memory for InfiniBand without running as root.

The welcome message at SSH login now carries a link that opens Grafana signed in as a read-only viewer user, so the dashboards and logs are one click away without looking up the admin password. On Slinky clusters the Slurm login pod shows the link too. The link rotates weekly and the current one is always in the welcome message; see Monitoring. The admin account and /root/grafana_password.txt are unchanged.

Kubernetes and Slinky images on Ubuntu 26.04 and CUDA 13.4

Kubernetes and Slinky Instant Cluster images are now built on Ubuntu 26.04 with CUDA 13.4 and the NVIDIA 580 open driver. They are available for Kubernetes 1.35, 1.36 and 1.37. The cluster software stack was upgraded too: DOCA-OFED 3.5, DCGM 4.7, HPC-X 2.26, gpud 0.13.1 and Grafana 13.1.

Slinky: SSH on port 22 lands in the Slurm login pod

On newly created Slinky clusters, ssh <user>@<cluster-ip> on port 22 drops you straight into the Slurm login pod, so it works like a login node on a native Slurm cluster. The host admin shell (ubuntu, with kubectl) moved to port 2222: ssh -p 2222 ubuntu@<cluster-ip>. The login pod's welcome message explains what the pod is, where $HOME, shared and local scratch storage live, and how to reach the admin shell. cat /root/grafana_password.txt works in the login pod as well.

Worker hostnames resolved by cluster DNS

Worker hostnames are now served by a DNS server on the cluster's service node instead of being written into /etc/hosts on every node. Pods resolve them too, so ping <cluster>-1 works from inside the Slinky login pod and from your own pods.

July 2026

Kubernetes clusters: Kueue, GPU Operator and RWX storage preinstalled

Kubernetes Instant Clusters now ship with Kueue for job queueing and quota admission, pre-wired with a ready-to-use default queue - label a workload with kueue.x-k8s.io/queue-name: default and it queues; see Job queueing (Kueue). GPU scheduling moved from the standalone device plugin to the full NVIDIA GPU Operator (drivers and container toolkit remain preinstalled in the image). A new shared-path StorageClass provides dynamic ReadWriteMany volumes on the shared filesystem for multi-node datasets and model caches.

Profiling enabled for unprivileged users

On all cluster images, GPU hardware counters are unlocked for Nsight Compute (ncu) and the kernel perf interface is opened up (perf stat / perf top), so workloads can be profiled inside jobs and pods without root.

Inter-node NCCL health check and health-check overview

In addition to the 6-hourly per-node benchmark suite, clusters now run a 6-hourly inter-node NCCL AllReduce across rotating node pairs, catching InfiniBand fabric degradation between nodes within hours. All recurring checks report into a new health-check overview dashboard - one row per check with last run, result and freshness, with per-node drill-down. See Ongoing health checks. On Kubernetes clusters the first active sweep now runs minutes after provisioning instead of waiting for the next 6-hour slot.

Weekly automated training benchmark

GPU clusters now run a short training benchmark automatically once a week, training Llama-3.1-70B and Qwen3-30B-A3B for a limited number of steps across all of the cluster's GPU nodes. This exercises real multi-node NCCL/InfiniBand communication, GPU memory pressure, and end-to-end throughput, catching regressions that a synthetic benchmark can miss. Results are visible on the monitoring dashboards.

GPU memory fully scrubbed on reset

When a GPU is reset between allocations, its memory is now zeroed across the entire framebuffer rather than just the portion that was already free. Any processes still holding GPU memory are terminated first, so no data from a previous workload can resurface later.

June 2026

Kubernetes and Slinky are now separate orchestrator options

The default Kubernetes Instant Cluster image now provisions a Kubernetes-only cluster. It no longer installs Slinky or reserves GPUs with Slurm worker pods by default.

Slinky is available as a separate orchestrator/image option for customers who want Slurm running on top of Kubernetes. This keeps Kubernetes workloads such as MPIJob, Kueue, Ray, and custom GPU pods from competing with a bundled Slurm deployment unless Slinky was explicitly selected.

Slinky beta validation and worker health checks

Slinky beta clusters now run lightweight multi-node validation during provisioning, including Slurm worker readiness, srun smoke tests, shared-jail checks, and sbatch with nested srun.

The Slinky worker image also includes a Slurm HealthCheckProgram that checks GPU visibility, DCGM health, and recent fatal NVIDIA Xid events. Failed checks drain the affected Slurm node so new Slurm jobs avoid unhealthy workers.

Observability stack moved to VictoriaMetrics

New clusters deploy a VictoriaMetrics-based observability stack on the service node: VictoriaMetrics stores metrics (Prometheus-compatible), VictoriaLogs and Vector handle log collection, and Grafana sits on top. The dashboard set was expanded with GPUd health, a cluster log explorer, per-orchestrator Slurm dashboards, and a Kubernetes dashboard folder on Kubernetes/Slinky clusters.

Slurm management plane pinned to the service node on Slinky

On Slinky clusters the whole Slurm management plane (operator, controller, REST API, login pod and accounting database) now runs on the service node; workers run only slurmd pods. GPU workers no longer host control-plane pods.

Spack no longer preinstalled on Kubernetes and Slinky images

Kubernetes and Slinky images no longer ship the prebuilt Spack tree. Use uv, the HPC-X modules, or Apptainer containers there instead. Spack remains preinstalled on native Slurm clusters.

May 2026

NCCL_IB_PKEY=1 no longer required on B300 clusters

On B300 clusters, the InfiniBand partition key is now assigned at index 0. NCCL workloads running inside Docker containers no longer need the NCCL_IB_PKEY=1 environment variable to use InfiniBand. H200 and B200 clusters are unaffected and still need the variable set. See networking and ports for details.

April 2026

NVIDIA B300 GPU Accelerator support

Instant Clusters now support NVIDIA B300 GPUs, with InfiniBand XDR networking and NVMe passthrough. NCCL tests are compiled for the B300 compute capability.

SLURM upgraded to 25.11.4

The SLURM scheduler has been updated to version 25.11.4.

Jump host renamed to CLUSTER_NAME-login

The jump host's static hostname is now CLUSTER_NAME-login. It continues to act as the SSH entry point (and NAT gateway) into the cluster. The Console and API still label this node as the jumphost for now.

The login node no longer hosts the SLURM controller, the Kubernetes control plane or the monitoring/observability stack - these have moved off the login node to improve isolation and stability. The Grafana UI is still reachable at https://<login-ip>:443 as before.

Kubernetes orchestrator beta included bundled SLURM (Slinky)

During the Kubernetes orchestrator beta, selecting Kubernetes also installed a Slinky SLURM deployment on top of Kubernetes. You could drop into the SLURM login pod and submit jobs with srun / sbatch as you would on a SLURM-only cluster.

As of June 2026, Kubernetes and Slinky are separate orchestrator options. Select Kubernetes for a Kubernetes-only cluster, or select Slinky for Slurm running on top of Kubernetes.

Validation coverage on Kubernetes / Slinky beta clusters

The deployment-time validation and ongoing health checks were initially less thorough on Kubernetes / Slinky beta clusters than on SLURM-only clusters. Slinky validation and worker health checks were expanded in June 2026.

March 2026

Kubernetes cluster image

A Kubernetes orchestrator image is now available as an alternative to SLURM. Clusters can be deployed with Kubernetes, including GPU device plugins (NFD, GFD), local NVMe storage classes, and InfiniBand networking support.

Kubernetes MPI Operator

Kubernetes clusters now deploy the MPI Operator, enabling distributed multi-node training jobs using MPIJob resources.

Slurmrestd REST API

SLURM clusters now run slurmrestd, enabling programmatic job submission and cluster management via the SLURM REST API.

DOCA networking updated to 3.2.2

The NVIDIA DOCA SDK has been updated to 3.2.2, keeping InfiniBand and networking drivers current.

February 2026

Instant GPU Clusters Generally Available

The Beta label has been removed from clusters, marking their general availability.

Centralized log aggregation with VictoriaMetrics

Cluster logs can be viewed using Grafana

Logs are retained with a size-based limit of 50 GB.

Production-ready alerting

The observability stack now includes production-ready alert rules with tuned thresholds for GPU health, node availability, disk usage, and cluster state.

To configure destinations of alerts see monitoring section

January 2026

Reduced availability

We have moved most of our instant cluster availability away for a few dedicated customers. More availability expected later in spring.

kanidm for identity management

Cluster internal authentication now uses kanidm.

kanidm provides centralized POSIX identity management across all cluster nodes, including SSH key distribution and user/group synchronization.

SLURM upgraded to 25.11.1

The SLURM scheduler has been upgraded from 25.05 to the 25.11 series, bringing improved scheduling performance and new features.

gpud upgraded to 0.9.2

The gpud daemon has been upgraded to 0.9.2, improving GPU health monitoring.

October 2025

SLURM upgraded to 25.05.3

The SLURM scheduler has been upgraded to 25.05.3. This is the first major version upgrade from the initial SLURM build.

Ubuntu 24.04 cluster image

Cluster images are now built on Ubuntu 24.04 with the 6.14 HWE kernel, replacing the previous Ubuntu 22.04 base.

CUDA 12.9 removed

CUDA 12.9 has been removed from the cluster image. Clusters now use CUDA 13.0.

gpud integrated with Node Health Check

The gpud daemon is now used within SLURM Node Health Check (NHC) to continuously monitor GPU state, InfiniBand port health, and driver status. Nodes with degraded GPUs or network links are automatically drained and can be rebooted.

Chrony time synchronization

Cluster nodes now use chrony for NTP time synchronization, ensuring consistent timestamps across all nodes.

August 2025

NVIDIA B200 GPU Accelerator support

Instant Clusters now support NVIDIA H200 GPUs

HPC-X pre-installed

NVIDIA HPC-X is now pre-installed in /opt on all cluster nodes, providing optimized MPI, SHMEM, and PGAS libraries for multi-node communication over InfiniBand.

Grafana monitoring dashboards

Clusters now include pre-provisioned Grafana dashboards for GPU metrics (DCGM), node resource usage, SLURM job status, and IPMI sensor data.

Customer-facing alerts for common failure modes are included.

NCCL all_reduce_perf validation

Cluster deployment now includes automated NCCL all_reduce_perf benchmarks to validate GPU-to-GPU communication performance across nodes before the cluster is marked as ready.

Enroot and Pyxis container support

Enroot and Pyxis are pre-installed, allowing SLURM jobs to run inside unprivileged containers pulled directly from container registries.

September 2025

virtiofs replaces NFS for /home

The shared /home filesystem now uses virtiofs instead of NFS-backed SFS, improving file I/O performance for cluster workloads.

April 2025

NFS backed SFS

The /home filesystem is now backed by NFS instead of CephFS

March 2025

Observability stack

Prometheus and Grafana are deployed on the cluster jumphost, providing metrics collection and visualization from day one. Node Exporter and DCGM Exporter run on all compute nodes.

February 2025

NVIDIA H200 GPU Accelerator support

Instant Clusters Initially Released with support for NVIDIA H200 GPUs