---
description: Queue, quota-gate and prioritize GPU workloads on a Kubernetes Instant Cluster with the pre-installed Kueue.
revision_date: 24.07.2026
---

# Job queueing (Kueue)

[Kueue](https://kueue.sigs.k8s.io/) is pre-installed on every Kubernetes Instant Cluster. It adds what vanilla Kubernetes scheduling lacks for batch GPU work: **admission control** — jobs wait in a queue *before* their pods exist, instead of flooding the scheduler with Pending pods — plus quota, priorities and fair sharing between teams.

## The pre-wired default queue

The cluster ships with a ready-to-use queue chain:

```
LocalQueue "default" (namespace: default)  →  ClusterQueue "default"  →  ResourceFlavor "default-flavor"
```

To queue a workload, add one label to any Job, MPIJob or other [supported workload](https://kueue.sigs.k8s.io/docs/tasks/run/):

```yaml
metadata:
  labels:
    kueue.x-k8s.io/queue-name: default
```

What happens next:

1. Kueue holds the workload **suspended** — no pods are created yet.
2. When quota is available, the workload is **admitted** and its pods start.
3. Workloads beyond quota wait in FIFO order (per priority).

Workloads *without* the label bypass Kueue entirely, so plain `kubectl apply` flows are unaffected.

Watch the queue:

```bash
kubectl get workloads -n default          # per-workload: admitted or pending
kubectl get clusterqueue default          # queue depth at a glance
kubectl get jobs                          # SUSPEND column = still queued
```

## Enforcing real quota

The pre-configured quota is effectively **unlimited**, which makes Kueue admit everything immediately — fine for a single user, but no real gating. To turn it into an actual queue, set the quota to your cluster's capacity. For example on a 2-node × 8-GPU cluster:

```bash
kubectl patch clusterqueue default --type=json -p '[{"op":"replace",
  "path":"/spec/resourceGroups/0/flavors/0/resources",
  "value":[{"name":"cpu","nominalQuota":400},
           {"name":"memory","nominalQuota":"3000Gi"},
           {"name":"nvidia.com/gpu","nominalQuota":16},
           {"name":"rdma/rdma_shared_device_a","nominalQuota":2000}]}]'
```

With capacity-sized quota, a burst of jobs admits only what fits. As an illustration, submitting twelve 8-GPU jobs to that 16-GPU quota gives:

```console
$ kubectl get clusterqueue default
NAME      ... PENDING WORKLOADS
default   ... 10

$ kubectl get jobs
NAME      SUSPENDED   ACTIVE
load-1    false       1        # admitted — running
load-2    false       1        # admitted — running
load-3    true                 # queued — no pods exist yet
...
```

As each job finishes, the next in line is admitted automatically.

## Priorities

Create [`WorkloadPriorityClass`](https://kueue.sigs.k8s.io/docs/concepts/workload_priority_class/) objects and label workloads to let urgent work jump the queue:

```yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: high
value: 1000
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: low
value: 100
```

```yaml
metadata:
  labels:
    kueue.x-k8s.io/queue-name: default
    kueue.x-k8s.io/priority-class: high
```

A `high` workload is admitted ahead of every waiting `low` workload (it does **not** evict already-running work by default — enable [preemption](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#preemption) on the ClusterQueue if you want that).

## Per-team queues

The standard Kueue pattern applies unchanged: one namespace + `LocalQueue` per team, all feeding one `ClusterQueue` (shared quota, borrowing allowed) or several (hard splits). See the [Kueue administration guide](https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/). The pre-installed `default` queue can serve as-is for the first team; add more LocalQueues in other namespaces pointing at the same ClusterQueue.

## Gang scheduling

For multi-node training, partial scheduling wastes GPUs — you want all pods of a job to start together or not at all. Kueue admits a workload's pod sets as a unit, and pairs well with SkyPilot for a full launch pipeline: see the [gang-scheduled training tutorial](https://docs.verda.com/clusters/instant-clusters/tutorials/kueue-gang-scheduling/).