Slinky (Slurm on Kubernetes)¶

Slinky is SchedMD's way of running Slurm on top of Kubernetes: the Slurm control plane runs as Kubernetes pods, and each worker node runs a slurmd pod. You get the normal Slurm experience (sbatch, srun, squeue, sacct) on a Kubernetes-managed cluster.

Slinky is its own orchestrator option when you deploy an instant cluster, alongside plain Kubernetes. If you selected it, this is what you are running. It differs from a conventional Slurm installation in a few user-visible ways, documented in this section.

Architecture¶

graph TD
    User([User])
    User -->|"SSH :22"| Pod
    User -->|"SSH :2222 admin"| Login

    subgraph cluster [Cluster private network]
        Login["Login node<br/>admin shell · kubectl"]
        subgraph service [Service node]
            Op["Slinky operator"]
            Ctl["slurm-controller<br/>(slurmctld)"]
            Acct["accounting<br/>(slurmdbd + DB)"]
            Rest["slurm-restapi"]
            Pod["login pod"]
        end
        subgraph workers [GPU workers]
            W1["slurmd pod · 8 GPUs<br/>jobs in shared jail"]
            Wn["slurmd pod · 8 GPUs<br/>jobs in shared jail"]
        end
        Login --- service
        Ctl --- W1
        Ctl --- Wn
    end

SSH on port 22 lands in the login pod, where every user (including ubuntu) is themselves and runs the Slurm CLI directly. The login node behind it is the admin shell on port 2222 (sudo, kubectl, user management); see User management.
The whole Slurm management plane (operator, controller, accounting, REST API and login pod) runs on the service node, so workers do nothing but run jobs.
Each worker runs a slurmd pod that owns all 8 GPUs. Every job step executes inside a per-job shared jail.
/home is shared across the login node, the login pod and all workers: build environments once, use them from every job.
Storage inside jobs: /shared is a common dataset area on the shared filesystem, /local is the worker's NVMe (persists across jobs on that node), and /tmp is a private per-job scratch on the same NVMe, emptied at job end.
The operator owns slurm.conf. Configuration changes go through the controller resource, not through files; see Changing the Slurm configuration below.

What is different on Slinky¶

User management: add users with one command, slinky-user-add, instead of driving Kanidm by hand. Users SSH straight into the Slurm login pod on port 22, and identities can survive cluster re-provisioning via the identity ledger (replay).
The shared jail: every srun/sbatch step runs inside a per-job sandbox. Your /home, CUDA and HPC-X are available inside it; GPUs appear only when you request --gpus.
Containers: containers run via Apptainer from inside a step. Pyxis (srun --container-image) and an in-job Docker daemon are not available.
Observability: Slinky-specific Grafana dashboards, centralized metrics, and custom prolog/epilog hooks.
Health checks: recurring node, fabric, storage, training, and inference checks; dashboard results; manual runs; and troubleshooting.

What is the same¶

The Slurm CLI and job model, see the Slurm documentation and Getting started.
The Lmod/HPC-X modules work (/opt/hpcx is bind-mounted into the jail). For Python, create a virtualenv under /home with python3 -m venv, or run the /usr/local/bin/pytorch.setup.sh helper, which installs uv on first run.
Monitoring access and validation work as on any instant cluster; the dashboards and checks are Slinky-aware.

Changing the Slurm configuration¶

On Slinky clusters there is no /etc/slurm/slurm.conf to edit. The operator generates slurm.conf and distributes it to the controller and all workers, so a hand-edited file is overwritten on the next reconcile. Extra settings go into the controller's extraConf field, whose lines are appended to the generated slurm.conf.

To add or change a setting (example: enable accounting enforcement):

Open the controller resource in an editor:
```
kubectl -n slurm edit controller slurm
```
Find spec.extraConf (a multi-line text block) and append your line at the end, keeping the indentation:
```
extraConf: |
  # ...existing lines...
  AccountingStorageEnforce=associations,limits,qos
```
Save and exit. The operator regenerates slurm.conf and applies it live (equivalent to scontrol reconfigure), with no control-plane downtime and no pod restart.

Verify from the login node:

scontrol show config | grep AccountingStorageEnforce

Two footnotes:

A small set of Slurm parameters only take effect on a full slurmctld restart (noted in the slurm.conf reference). For those, delete the controller pod and let the operator recreate it: kubectl -n slurm delete pod slurm-controller-0.
If you manage the slurm Helm release yourself, set controller.extraConf in the release values (helm upgrade slurm <slurm-chart> -n slurm --reuse-values -f values.yaml) instead: a live kubectl edit is overwritten by the next helm upgrade of the release.

In this section¶

Getting started

User management

Shared jail

Containers (Apptainer)

Observability

Health checks

Tutorials