Skip to content

Slinky (Slurm on Kubernetes)

Slinky is SchedMD's way of running Slurm on top of Kubernetes: the Slurm control plane runs as Kubernetes pods, and each worker node runs a slurmd pod. You get the normal Slurm experience (sbatch, srun, squeue, sacct) on a Kubernetes-managed cluster.

Slinky is its own orchestrator option when you deploy an instant cluster, alongside plain Kubernetes. If you selected it, this is what you are running. It differs from a conventional Slurm installation in a few user-visible ways, documented in this section.

Architecture

graph TD
    User([User])
    User -->|"SSH :22"| Pod
    User -->|"SSH :2222 admin"| Login

    subgraph cluster [Cluster private network]
        Login["Login node<br/>admin shell · kubectl"]
        subgraph service [Service node]
            Op["Slinky operator"]
            Ctl["slurm-controller<br/>(slurmctld)"]
            Acct["accounting<br/>(slurmdbd + DB)"]
            Rest["slurm-restapi"]
            Pod["login pod"]
        end
        subgraph workers [GPU workers]
            W1["slurmd pod · 8 GPUs<br/>jobs run here"]
            Wn["slurmd pod · 8 GPUs<br/>jobs run here"]
        end
        Login --- service
        Ctl --- W1
        Ctl --- Wn
    end
  • SSH on port 22 lands in the login pod, where every user (including ubuntu) is themselves and runs the Slurm CLI directly. The login node behind it is the admin shell on port 2222 (sudo, kubectl, user management); see User management.
  • The whole Slurm management plane (operator, controller, accounting, REST API and login pod) runs on the service node, so workers do nothing but run jobs.
  • Each worker runs a slurmd pod that owns all 8 GPUs. Job steps run inside that pod as the submitting user; see Job environment.
  • /home is shared across the login node, the login pod and all workers: build environments once, use them from every job.
  • Storage inside jobs: /shared is a common dataset area on the shared filesystem, and /local and /tmp are on the worker's NVMe. /local persists across jobs on that node; /tmp is private to each job and deleted when it ends. Extra shared filesystems you attach are mounted in the login pod and the worker pods as well, at /<SFS_NAME>.
  • The operator owns slurm.conf. Configuration changes go through the controller resource, not through files; see Changing the Slurm configuration below.

What is different on Slinky

  • User management: add users with one command, slinky-user-add, instead of driving Kanidm by hand. Users SSH straight into the Slurm login pod on port 22, and identities can survive cluster re-provisioning via the identity ledger (replay).
  • Job environment: steps run in the worker's slurmd pod (Ubuntu 24.04), not on the host. GPUs appear only when you request --gpus, and HPC-X/Lmod modules are not available inside jobs.
  • Containers: Pyxis (srun --container-image) with Enroot, and Apptainer. There is no in-job Docker daemon.
  • Observability: Slinky-specific Grafana dashboards, centralized metrics, and custom prolog/epilog hooks.

What is the same

Changing the Slurm configuration

On Slinky clusters there is no /etc/slurm/slurm.conf to edit. The operator generates slurm.conf and distributes it to the controller and all workers, so a hand-edited file is overwritten on the next reconcile. Extra settings go into the controller's extraConf field, whose lines are appended to the generated slurm.conf.

To add or change a setting (example: enable accounting enforcement):

  1. Open the controller resource in an editor:

    kubectl -n slurm edit controller slurm
    
  2. Find spec.extraConf (a multi-line text block) and append your line at the end, keeping the indentation:

    extraConf: |
      # ...existing lines...
      AccountingStorageEnforce=associations,limits,qos
    
  3. Save and exit. The operator regenerates slurm.conf and applies it live (equivalent to scontrol reconfigure), with no control-plane downtime and no pod restart.

  4. Verify from the login node:

    scontrol show config | grep AccountingStorageEnforce
    

Two footnotes:

  • A small set of Slurm parameters only take effect on a full slurmctld restart (noted in the slurm.conf reference). For those, delete the controller pod and let the operator recreate it: kubectl -n slurm delete pod slurm-controller-0.
  • If you manage the slurm Helm release yourself, set controller.extraConf in the release values (helm upgrade slurm <slurm-chart> -n slurm --reuse-values -f values.yaml) instead: a live kubectl edit is overwritten by the next helm upgrade of the release.

In this section

Getting started

User management

Job environment

Containers (Pyxis, Apptainer)

Observability

Tutorials