---
description: Manage users on a Slinky cluster with slinky-user-add, SSH login to the Slurm login pod, and identity that survives cluster re-provisioning (replay and reconcile).
revision_date: 03.11.2026
---

# User management

On a Slinky cluster you do **not** drive Kanidm by hand. A single tool, `slinky-user-add`, provisions everything a user needs in one step:

* a **Kanidm** person (the identity + their SSH public key),
* a **POSIX** account (UID/GID) and `cluster_users` group membership,
* a **Slurm** account association (`default_acct` by default) so the user can submit jobs.

Run it as an admin (root) on the login node, which you reach with `ssh -p 2222 ubuntu@<cluster-ip>` (see [How users log in](#how-users-log-in)).

## Add a user

```bash
sudo slinky-user-add alice \
  --display-name "Alice Example" \
  --ssh-key "ssh-ed25519 AAAA... alice@laptop"
```

Useful options:

| Option | Purpose |
|--------|---------|
| `--ssh-key "ssh-... "` | Public key (repeatable for multiple keys). |
| `--ssh-key-file FILE` | Read one or more public keys from a file. |
| `--from-authorized-keys` | Pick keys interactively from the node's `authorized_keys`. |
| `--interactive` / `-i` | Prompt for missing values. |
| `--uid UID` | Fix the POSIX UID/GID (otherwise auto-allocated). |
| `--account ACCOUNT` | Slurm account to join (default `default_acct`). |
| `--enable-seamless` | Turn seamless SSH back on if it was disabled (see below). |
| `--path DIR` | Filesystem holding the identity ledger (see below). |
| `--test` | Run an in-cluster NSS/Slurm smoke test after provisioning. |

`--test` is worth using the first time: it confirms the new user resolves on the workers and can launch a Slurm step (it prints the step's `uid`/`user`).

## Remove a user

```bash
sudo slinky-user-remove alice --yes
```

* `--archive-home` moves `/home/alice` under `/home/.verda/removed-users/` instead of leaving it in place.
* `--keep-kanidm` keeps the Kanidm person but drops `cluster_users` membership (revokes cluster access without deleting the identity).

## How users log in

Port `22` on the cluster's public IP leads straight into the Slurm login pod (**seamless SSH**). Every user, including `ubuntu`, lands there as themselves (their own UID), with the full Slurm CLI, and their jobs run under their own identity:

```bash
ssh alice@<cluster-ip>
ssh ubuntu@<cluster-ip>
```

The login pod has no `sudo`, and anything installed outside `/home` is lost when the pod restarts. For admin work (`sudo`, `kubectl`, `slinky-user-add`), use the login node's own shell on port `2222`:

```bash
ssh -p 2222 ubuntu@<cluster-ip>
```

Members of the kanidm `cluster_users` group (everyone added with `slinky-user-add`) only get the login pod. SSH to the login node and to the worker nodes refuses them; they reach the workers through Slurm jobs. `ubuntu` can still SSH to every node.

!!! tip
    The first SSH attempt as a freshly created user can fail with `Permission denied` while the nodes' kanidm caches warm up. Retry after a few seconds.

The Slurm commands on the login node (`sbatch`, `srun`, `squeue`, ...) are wrappers that run inside the login pod through `kubectl`, as the **calling user**: `sbatch` as `ubuntu` submits as `ubuntu`, and `sudo sbatch` submits as root. They need the admin kubeconfig, so only `ubuntu` and root can use them; other users get a pointer to `ssh <user>@<cluster-ip>`. Prefer the login pod for regular work; `srun id` shows which identity a shell submits as.

### Changing the port layout

The layout is set in `/etc/default/verda_seamless_ssh` on the login node:

| Setting | Effect |
|---------|--------|
| `SEAMLESS_SSH_PORT=22` (default) | `:22` → login pod, `:2222` → login node shell. |
| `SEAMLESS_SSH_PORT=2222` | `:22` → login node shell, `:2222` → login pod (the layout used before October 2026). |
| `SEAMLESS_SSH_ENABLED=0` | No login pod redirect; `:22` → login node shell only. |

Apply a change with `sudo systemctl restart slinky-login-dnat.service`. Only `22` or `2222` are valid ports.

## Access control

When users submit from the login pod, each user is a distinct POSIX/Slurm identity and normal Slurm and filesystem boundaries apply: a user cannot `scancel` another user's job, and `/home/<user>` is private. Jobs submitted with `sudo` on the login node run as root and bypass these per-user boundaries. Per-user limits and QoS are set with `sacctmgr`.

## Identity across cluster re-provisioning (replay and reconcile)

File ownership on a shared filesystem is stored as numeric UIDs. If you delete a cluster and create a new one with the same filesystem attached, the new cluster starts with an empty identity store, and naively re-created users would get different UIDs and lose access to their own files.

Slinky clusters solve this with an **identity ledger**: a small record of every user (username, UID, display name, SSH keys) kept on the filesystem itself.

* `slinky-user-add` and `slinky-user-remove` maintain the ledger automatically. By default it lives at `/home/.verda/slinky-users/`; pass `--path <mount>` to keep it on a filesystem you retain between clusters instead.
* **`slinky-users-replay`** re-creates every user in the ledger, pinned to the original UID/GID and with the stored SSH keys. It runs automatically at boot and is idempotent, so it is safe to run by hand, for example after attaching a kept filesystem at `/main`:

    ```bash
    sudo slinky-users-replay --path /main
    ```

* If the attached filesystem holds user data but no ledger yet (a volume from before this feature), replay first **auto-seeds** the ledger from directory ownership. It only considers regular user UIDs and refuses to guess if two directories share a UID.
* **`slinky-users-reconcile`** audits the ledger against actual directory ownership and reports mismatches without changing anything; `--write-ledger` writes the inferred entries:

    ```bash
    sudo slinky-users-reconcile --path /main
    ```

!!! warning
    `/home` is created fresh with every new cluster, so a ledger kept there does not survive re-provisioning. To carry identities across clusters, keep them on the shared filesystem you retain and pass its mount point with `--path`.

!!! info
    `slinky-user-add` is the Slinky front-end to Kanidm.
