Skip to content

Job queueing (Kueue)

Kueue is pre-installed on every Kubernetes Instant Cluster. It adds what vanilla Kubernetes scheduling lacks for batch GPU work: admission control — jobs wait in a queue before their pods exist, instead of flooding the scheduler with Pending pods — plus quota, priorities and fair sharing between teams.

The pre-wired default queue

The cluster ships with a ready-to-use queue chain:

LocalQueue "default" (namespace: default)  →  ClusterQueue "default"  →  ResourceFlavor "default-flavor"

To queue a workload, add one label to any Job, MPIJob or other supported workload:

metadata:
  labels:
    kueue.x-k8s.io/queue-name: default

What happens next:

  1. Kueue holds the workload suspended — no pods are created yet.
  2. When quota is available, the workload is admitted and its pods start.
  3. Workloads beyond quota wait in FIFO order (per priority).

Workloads without the label bypass Kueue entirely, so plain kubectl apply flows are unaffected.

Watch the queue:

kubectl get workloads -n default          # per-workload: admitted or pending
kubectl get clusterqueue default          # queue depth at a glance
kubectl get jobs                          # SUSPEND column = still queued

Enforcing real quota

The pre-configured quota is effectively unlimited, which makes Kueue admit everything immediately — fine for a single user, but no real gating. To turn it into an actual queue, set the quota to your cluster's capacity. For example on a 2-node × 8-GPU cluster:

kubectl patch clusterqueue default --type=json -p '[{"op":"replace",
  "path":"/spec/resourceGroups/0/flavors/0/resources",
  "value":[{"name":"cpu","nominalQuota":400},
           {"name":"memory","nominalQuota":"3000Gi"},
           {"name":"nvidia.com/gpu","nominalQuota":16},
           {"name":"rdma/rdma_shared_device_a","nominalQuota":2000}]}]'

With capacity-sized quota, a burst of jobs admits only what fits. As an illustration, submitting twelve 8-GPU jobs to that 16-GPU quota gives:

$ kubectl get clusterqueue default
NAME      ... PENDING WORKLOADS
default   ... 10

$ kubectl get jobs
NAME      SUSPENDED   ACTIVE
load-1    false       1        # admitted — running
load-2    false       1        # admitted — running
load-3    true                 # queued — no pods exist yet
...

As each job finishes, the next in line is admitted automatically.

Priorities

Create WorkloadPriorityClass objects and label workloads to let urgent work jump the queue:

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: high
value: 1000
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: low
value: 100

metadata:
  labels:
    kueue.x-k8s.io/queue-name: default
    kueue.x-k8s.io/priority-class: high

A high workload is admitted ahead of every waiting low workload (it does not evict already-running work by default — enable preemption on the ClusterQueue if you want that).

Per-team queues

The standard Kueue pattern applies unchanged: one namespace + LocalQueue per team, all feeding one ClusterQueue (shared quota, borrowing allowed) or several (hard splits). See the Kueue administration guide. The pre-installed default queue can serve as-is for the first team; add more LocalQueues in other namespaces pointing at the same ClusterQueue.

Gang scheduling

For multi-node training, partial scheduling wastes GPUs — you want all pods of a job to start together or not at all. Kueue admits a workload's pod sets as a unit, and pairs well with SkyPilot for a full launch pipeline: see the gang-scheduled training tutorial.