---
description: What Slinky (Slurm on Kubernetes) is, how a Slinky instant cluster is put together, and what differs from native Slurm.
revision_date: 27.07.2026
---

# Slinky (Slurm on Kubernetes)

**Slinky** is [SchedMD's](https://github.com/SlinkyProject) way of running Slurm on top of Kubernetes: the Slurm control plane runs as Kubernetes pods, and each worker node runs a `slurmd` pod. You get the normal Slurm experience (`sbatch`, `srun`, `squeue`, `sacct`) on a Kubernetes-managed cluster.

Slinky is its own orchestrator option when you deploy an instant cluster, alongside plain Kubernetes. If you selected it, this is what you are running. It differs from a conventional Slurm installation in a few user-visible ways, documented in this section.

## Architecture

```mermaid
graph TD
    User([User])
    User -->|"SSH :22 admin · :2222 seamless (opt-in)"| Login

    subgraph cluster [Cluster private network]
        Login["Login node<br/>bastion · Slurm CLI wrappers"]
        subgraph service [Service node]
            Op["Slinky operator"]
            Ctl["slurm-controller<br/>(slurmctld)"]
            Acct["accounting<br/>(slurmdbd + DB)"]
            Rest["slurm-restapi"]
            Pod["login pod"]
        end
        subgraph workers [GPU workers]
            W1["slurmd pod · 8 GPUs<br/>jobs in shared jail"]
            Wn["slurmd pod · 8 GPUs<br/>jobs in shared jail"]
        end
        Login --- service
        Ctl --- W1
        Ctl --- Wn
    end
```

* The **login node** is the SSH entry point. The Slurm CLI there is a set of admin wrappers into the login pod; once an admin enables seamless SSH, ordinary users connect over it and land in the login pod as themselves (see [User management](https://docs.verda.com/clusters/instant-clusters/slinky/users/)).
* The whole **Slurm management plane** (operator, controller, accounting, REST API and login pod) runs on the **service node**, so workers do nothing but run jobs.
* Each **worker** runs a `slurmd` pod that owns all 8 GPUs. Every job step executes inside a per-job [shared jail](https://docs.verda.com/clusters/instant-clusters/slinky/shared-jail/).
* `/home` is shared across the login node, the login pod and all workers: build environments once, use them from every job.
* Storage inside jobs: `/shared` is a common dataset area on the shared filesystem, `/local` is the worker's NVMe (persists across jobs on that node), and `/tmp` is a private per-job scratch on the same NVMe, emptied at job end.
* The operator owns `slurm.conf`. Configuration changes go through the controller resource, not through files; see [Changing the Slurm configuration](#changing-the-slurm-configuration) below.

## What is different on Slinky

* **[User management](https://docs.verda.com/clusters/instant-clusters/slinky/users/)**: add users with one command, `slinky-user-add`, instead of driving Kanidm by hand. Users log in over seamless SSH once it is enabled (`--enable-seamless`; off by default), and identities can survive cluster re-provisioning via the identity ledger (replay).
* **[The shared jail](https://docs.verda.com/clusters/instant-clusters/slinky/shared-jail/)**: every `srun`/`sbatch` step runs inside a per-job sandbox. Your `/home`, CUDA and HPC-X are available inside it; GPUs appear only when you request `--gpus`.
* **[Containers](https://docs.verda.com/clusters/instant-clusters/slinky/containers/)**: containers run via Apptainer from inside a step. Pyxis (`srun --container-image`) and an in-job Docker daemon are not available.
* **[Observability](https://docs.verda.com/clusters/instant-clusters/slinky/observability/)**: Slinky-specific Grafana dashboards, centralized metrics, and custom prolog/epilog hooks.
* **[Health checks](https://docs.verda.com/clusters/instant-clusters/slinky/health-checks/)**: recurring node, fabric, storage, training, and inference checks; dashboard results; manual runs; and troubleshooting.

## What is the same

* The Slurm CLI and job model, see the [Slurm documentation](https://slurm.schedmd.com/man_index.html) and [Getting started](https://docs.verda.com/clusters/instant-clusters/slinky/getting-started/).
* The Lmod/HPC-X modules work (`/opt/hpcx` is bind-mounted into the jail). For Python, create a virtualenv under `/home` with `python3 -m venv`, or run the `/usr/local/bin/pytorch.setup.sh` helper, which installs `uv` on first run.
* [Monitoring](https://docs.verda.com/clusters/instant-clusters/monitoring/) access and [validation](https://docs.verda.com/clusters/instant-clusters/validation/) work as on any instant cluster; the dashboards and checks are Slinky-aware.

## Changing the Slurm configuration

On Slinky clusters there is **no `/etc/slurm/slurm.conf` to edit**. The operator generates `slurm.conf` and distributes it to the controller and all workers, so a hand-edited file is overwritten on the next reconcile. Extra settings go into the controller's `extraConf` field, whose lines are appended to the generated `slurm.conf`.

To add or change a setting (example: enable accounting enforcement):

1. Open the controller resource in an editor:

    ```bash
    kubectl -n slurm edit controller slurm
    ```

2. Find `spec.extraConf` (a multi-line text block) and append your line at the end, keeping the indentation:

    ```yaml
    extraConf: |
      # ...existing lines...
      AccountingStorageEnforce=associations,limits,qos
    ```

3. Save and exit. The operator regenerates `slurm.conf` and applies it live (equivalent to `scontrol reconfigure`), with no control-plane downtime and no pod restart.

4. Verify from the login node:

    ```bash
    scontrol show config | grep AccountingStorageEnforce
    ```

Two footnotes:

* A small set of Slurm parameters only take effect on a full `slurmctld` restart (noted in the [slurm.conf reference](https://slurm.schedmd.com/slurm.conf.html)). For those, delete the controller pod and let the operator recreate it: `kubectl -n slurm delete pod slurm-controller-0`.
* If you manage the `slurm` Helm release yourself, set `controller.extraConf` in the release values (`helm upgrade slurm <slurm-chart> -n slurm --reuse-values -f values.yaml`) instead: a live `kubectl edit` is overwritten by the next `helm upgrade` of the release.

## In this section

[Getting started](https://docs.verda.com/clusters/instant-clusters/slinky/getting-started/)

[User management](https://docs.verda.com/clusters/instant-clusters/slinky/users/)

[Shared jail](https://docs.verda.com/clusters/instant-clusters/slinky/shared-jail/)

[Containers (Apptainer)](https://docs.verda.com/clusters/instant-clusters/slinky/containers/)

[Observability](https://docs.verda.com/clusters/instant-clusters/slinky/observability/)

[Health checks](https://docs.verda.com/clusters/instant-clusters/slinky/health-checks/)

[Tutorials](https://docs.verda.com/clusters/instant-clusters/slinky/tutorials/)