Skip to content

Getting started

Log in

As the ubuntu admin, SSH to the cluster's public IP:

ssh ubuntu@<cluster-ip>

The Slurm commands on the login node (sinfo, squeue, sbatch, srun, scancel, scontrol, sacct, salloc, sacctmgr) are thin wrappers that execute inside the Slurm login pod. They behave like the native commands, with two caveats:

  • they need kubectl credentials (a readable ~/.kube/config), which only the ubuntu admin and root have by default, and
  • they run inside the login pod as its root user, so jobs submitted this way are owned and accounted as root.

Additional users are provisioned with slinky-user-add and connect over seamless SSH (port 2222), where they are themselves and their jobs run under their own identity. srun id shows which identity a shell submits as.

Check the cluster

sinfo

Worker nodes appear as slinky-0, slinky-1, ... in the all partition and should be idle (or mix/alloc when busy).

Run your first job

Each worker has 8 GPUs, so a 16-GPU job spans two nodes:

srun --gpus=16 nvidia-smi -L

Warning

Steps only see the GPUs they request. With no --gpus (or --gres=gpu:N), nvidia-smi reports No devices found. inside the step. See Shared jail.

Batch jobs

Write the script under /home (it is shared with the workers) and submit from there:

cat > /home/ubuntu/hello.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --nodes=2
#SBATCH --gres=gpu:8
#SBATCH --time=00:05:00
srun hostname
srun nvidia-smi -L
EOF

sbatch /home/ubuntu/hello.sbatch

Output lands in slurm-<jobid>.out next to where you submitted. Use squeue for active jobs, sacct for finished ones. The full command reference is on the Slurm page, which applies to Slinky unchanged.

Interactive sessions

srun --nodes=1 --gres=gpu:1 --time=00:15:00 --pty bash

drops you into a shell on a worker with one GPU bound. For multi-node interactive work, hold an allocation with salloc and run srun inside it. See interactive sessions.

Where to put things

  • /home is shared everywhere: Python environments (uv venv), job scripts, container images (SIF files). Build once, use from every job.
  • /shared is a world-writable dataset area on the same shared filesystem: one stable path for team data, identical on the login pod and inside every job.
  • /local inside a step is the worker's local NVMe: a per-node cache for datasets and checkpoints. It persists across jobs and node restarts, so a re-run landing on the same node finds its data already there.
  • /tmp inside a step is private to the job, also on the NVMe: use it for heavy temporary I/O. It is emptied when the job ends; keep anything valuable on /local or /home.
  • HPC-X modules are available inside steps from /opt/hpcx: srun bash -lc 'module load hpcx && ...'. See Environments.

Next steps