Instant Cluster Release Notes¶
Changes to the Instant GPU Cluster offering -> new GPU accelerators, orchestrator images, software upgrades, and infrastructure improvements.
June 2026¶
Kubernetes and Slinky are now separate orchestrator options¶
The default Kubernetes Instant Cluster image now provisions a Kubernetes-only cluster. It no longer installs Slinky or reserves GPUs with Slurm worker pods by default.
Slinky is available as a separate orchestrator/image option for customers who want Slurm running on top of Kubernetes. This keeps Kubernetes workloads such as MPIJob, Kueue, Ray, and custom GPU pods from competing with a bundled Slurm deployment unless Slinky was explicitly selected.
Slinky beta validation and worker health checks¶
Slinky beta clusters now run lightweight multi-node validation during provisioning, including Slurm worker readiness, srun smoke tests, shared-jail checks, and sbatch with nested srun.
The Slinky worker image also includes a Slurm HealthCheckProgram that checks GPU visibility, DCGM health, and recent fatal NVIDIA Xid events. Failed checks drain the affected Slurm node so new Slurm jobs avoid unhealthy workers.
May 2026¶
NCCL_IB_PKEY=1 no longer required on B300 clusters¶
On B300 clusters, the InfiniBand partition key is now assigned at index 0. NCCL workloads running inside Docker containers no longer need the NCCL_IB_PKEY=1 environment variable to use InfiniBand. H200 and B200 clusters are unaffected and still need the variable set. See good to know for details.
April 2026¶
NVIDIA B300 GPU Accelerator support¶
Instant Clusters now support NVIDIA B300 GPUs, with InfiniBand XDR networking and NVMe passthrough. NCCL tests are compiled for the B300 compute capability.
SLURM upgraded to 25.11.4¶
The SLURM scheduler has been updated to version 25.11.4.
Jump host renamed to CLUSTER_NAME-login¶
The jump host's static hostname is now CLUSTER_NAME-login. It continues to act as the SSH entry point (and NAT gateway) into the cluster. The Console and API still label this node as the jumphost for now.
The login node no longer hosts the SLURM controller, the Kubernetes control plane or the monitoring/observability stack — these have moved off the login node to improve isolation and stability. The Grafana UI is still reachable at https://<login-ip>:443 as before.
Kubernetes orchestrator beta included bundled SLURM (Slinky)¶
During the Kubernetes orchestrator beta, selecting Kubernetes also installed a Slinky SLURM deployment on top of Kubernetes. You could drop into the SLURM login pod and submit jobs with srun / sbatch as you would on a SLURM-only cluster.
As of June 2026, Kubernetes and Slinky are separate orchestrator options. Select Kubernetes for a Kubernetes-only cluster, or select Slinky for Slurm running on top of Kubernetes.
Validation coverage on Kubernetes / Slinky beta clusters¶
The deployment-time validation and ongoing health checks were initially less thorough on Kubernetes / Slinky beta clusters than on SLURM-only clusters. Slinky validation and worker health checks were expanded in June 2026.
March 2026¶
Kubernetes cluster image¶
A Kubernetes orchestrator image is now available as an alternative to SLURM. Clusters can be deployed with Kubernetes, including GPU device plugins (NFD, GFD), local NVMe storage classes, and InfiniBand networking support.
Kubernetes MPI Operator¶
Kubernetes clusters now deploy the MPI Operator, enabling distributed multi-node training jobs using MPIJob resources.
Slurmrestd REST API¶
SLURM clusters now run slurmrestd, enabling programmatic job submission and cluster management via the SLURM REST API or the s9s tool.
DOCA networking updated to 3.2.2¶
The NVIDIA DOCA SDK has been updated to 3.2.2, keeping InfiniBand and networking drivers current.
February 2026¶
Instant GPU Clusters Generally Available¶
The Beta label has been removed from clusters, marking their general availability.
Centralized log aggregation with VictoriaMetrics¶
Cluster logs can be viewed using Grafana
Logs are retained with a size-based limit of 50 GB.
Production-ready alerting¶
The observability stack now includes production-ready alert rules with tuned thresholds for GPU health, node availability, disk usage, and cluster state.
To configure destinations of alerts see monitoring section
January 2026¶
Reduced availability¶
We have moved most of our instant cluster availability away for a few dedicated customers. More availability expected later in spring.
kanidm for identity management¶
Cluster internal authentication now uses kanidm.
kanidm provides centralized POSIX identity management across all cluster nodes, including SSH key distribution and user/group synchronization.
SLURM upgraded to 25.11.1¶
The SLURM scheduler has been upgraded from 25.05 to the 25.11 series, bringing improved scheduling performance and new features.
gpud upgraded to 0.9.2¶
The gpud daemon has been upgraded to 0.9.2, improving GPU health monitoring.
October 2025¶
SLURM upgraded to 25.05.3¶
The SLURM scheduler has been upgraded to 25.05.3. This is the first major version upgrade from the initial SLURM build.
Ubuntu 24.04 cluster image¶
Cluster images are now built on Ubuntu 24.04 with the 6.14 HWE kernel, replacing the previous Ubuntu 22.04 base.
CUDA 12.9 removed¶
CUDA 12.9 has been removed from the cluster image. Clusters now use CUDA 13.0.
gpud integrated with Node Health Check¶
The gpud daemon is now used within SLURM Node Health Check (NHC) to continuously monitor GPU state, InfiniBand port health, and driver status. Nodes with degraded GPUs or network links are automatically drained and can be rebooted.
Chrony time synchronization¶
Cluster nodes now use chrony for NTP time synchronization, ensuring consistent timestamps across all nodes.
August 2025¶
NVIDIA B200 GPU Accelerator support¶
Instant Clusters now support NVIDIA H200 GPUs
HPC-X pre-installed¶
NVIDIA HPC-X is now pre-installed in /opt on all cluster nodes, providing optimized MPI, SHMEM, and PGAS libraries for multi-node communication over InfiniBand.
Grafana monitoring dashboards¶
Clusters now include pre-provisioned Grafana dashboards for GPU metrics (DCGM), node resource usage, SLURM job status, and IPMI sensor data.
Customer-facing alerts for common failure modes are included.
NCCL all_reduce_perf validation¶
Cluster deployment now includes automated NCCL all_reduce_perf benchmarks to validate GPU-to-GPU communication performance across nodes before the cluster is marked as ready.
Enroot and Pyxis container support¶
Enroot and Pyxis are pre-installed, allowing SLURM jobs to run inside unprivileged containers pulled directly from container registries.
September 2025¶
virtiofs replaces NFS for /home¶
The shared /home filesystem now uses virtiofs instead of NFS-backed SFS, improving file I/O performance for cluster workloads.
April 2025¶
NFS backed SFS¶
The /home filesystem is now backed by NFS instead of CephFS
March 2025¶
Observability stack¶
Prometheus and Grafana are deployed on the cluster jumphost, providing metrics collection and visualization from day one. Node Exporter and DCGM Exporter run on all compute nodes.
February 2025¶
NVIDIA H200 GPU Accelerator support¶
Instant Clusters Initially Released with support for NVIDIA H200 GPUs