Stop Waiting in Line: Scaling Faster with Kueue’s Concurrent Admission

If you run batch workloads in the cloud, you are likely dealing with heterogeneous compute resources. Your Kubernetes cluster might consist of a mix of reserved infrastructure for baseline capacity, on-demand nodes for burst scaling, and spot instances for cost-effective, interruptible workloads.

But orchestrating jobs across these different compute flavors has presented subtle, but frustrating scheduling bottlenecks. Until now, Kueue evaluated provisioning options serially, meaning you might wait for one specific node pool to spin up while another could have provisioned much faster. Furthermore, once a job started on a lower-tier instance, it was permanently locked there, even if premium capacity freed up moments later.

With Kueue 0.18, the team is introducing a powerful new feature to solve both halves of this problem: Concurrent Admission.

Workload variants

Concurrent Admission (as introduced in KEP-8691) addresses both the provisioning speed and the placement quality. Instead of picking one flavor, Kueue now creates workload variants — parallel representations of your job — for each acceptable resource flavor. If you’re relying on GKE and dynamically provisioned resources like DWS Flex-start, Kueue may also issue relevant Provisioning Requests for all these variants at the same time.

Kueue 0.18 introduces the new field in the Cluster Queue configuration called concurrentAdmissionPolicy that contains parallel admission-specific settings. Migration mode determines how Kueue handles workload placement and potential transitions between Resource Flavors and accepts the following options:

• TryPreferredFlavors: The workload is admitted to the first available flavor. If this is not the most preferred flavor (as defined by the order in the ClusterQueue), Kueue will continue to attempt to migrate the workload to a more preferred flavor as capacity becomes available.
• RetainFirstAdmission: Once the workload is admitted to a flavor, it will remain on that flavor for its entire duration. No migration to other flavors will be attempted, even if more preferred ones become available.

Additionally, the user can define the admission constraints, eg. define the last acceptable flavor a workload can migrate to.

Side note: The role of Provisioning Request API

Standard Kubernetes autoscaling can easily lead to partial capacity scenarios where half your job schedules and the rest hangs waiting for nodes. The Provisioning Request API acts as a formal interface between Kueue and the Cluster Autoscaler, allowing Kueue to request a specific set of resources as an atomic block. The workload is held in suspension by an AdmissionCheck until the autoscaler confirms the requested infrastructure is actually provisioned and ready. The example below relies on Provisioning Requests (specifically best-effort-atomic-scale-up.autoscaling.x-k8s.io class) to make sure that autoscaled (i.e. on-demand and spot) resources are provisioned properly.

Concurrent admission in action

The following comprehensive example demonstrates how to set up a Cluster Queue that utilizes concurrent admission across three distinct node categories: statically defined reserved nodes, alongside autoscaled on-demand and spot nodes. To execute this configuration, you must ensure the ConcurrentAdmission feature gate is active. While the resource flavor definitions provided here use GKE-specific node labels, they can be readily adapted for any Kubernetes cluster environment:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: "default"
spec:
  levels:
  - nodeLabel: "kubernetes.io/hostname"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ProvisioningRequestConfig
metadata:
  name: on-demand-config
spec:
  provisioningClassName: best-effort-atomic-scale-up.autoscaling.x-k8s.io
  managedResources:
  - cpu
  retryStrategy:
    backoffLimitCount: 2
    backoffBaseSeconds: 60
    backoffMaxSeconds: 1800
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: AdmissionCheck
metadata:
  name: capacity-check
spec:
  controllerName: kueue.x-k8s.io/provisioning-request
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: on-demand-config
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: flavor-reserved
spec:
  nodeLabels:
    cloud.google.com/gke-nodepool: "reserved-np"
  topologyName: "default"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: flavor-ondemand
spec:
  nodeLabels:
    cloud.google.com/gke-nodepool: "ondemand-np"
  topologyName: "default"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: flavor-spot
spec:
  nodeLabels:
    cloud.google.com/gke-nodepool: "spot-np"
  topologyName: "default"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: default-cq
spec:
  namespaceSelector: {}
  admissionChecksStrategy:
    admissionChecks:
    - name: capacity-check
      onFlavors:
      - flavor-ondemand
      - flavor-spot
  concurrentAdmissionPolicy:
    migration:
      mode: TryPreferredFlavors
      constraints:
        lastAcceptableFlavorName: flavor-reserved
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: flavor-reserved
      resources:
        - name: "cpu"
          nominalQuota: 3
    - name: flavor-ondemand
      resources:
      - name: "cpu"
        nominalQuota: 3
    - name: flavor-spot
      resources:
      - name: "cpu"
        nominalQuota: 3    
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: default
  name: default-lq
spec:
  clusterQueue: default-cq

We will now execute two example jobs, each containing one pod. The large-cpu-job is configured to run for 60 seconds with a 3 CPU requirement, whereas the small-cpu-job will run for 600 seconds and requires only 1 CPU:

apiVersion: batch/v1
kind: Job
metadata:
  name: large-cpu-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: default-lq
spec:
  parallelism: 1
  completions: 1
  template:
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: [ "/bin/sh" ]
        args: [ "-c", "sleep 60" ]
        resources:
          limits:
            cpu: "3"
      restartPolicy: Never
---
apiVersion: batch/v1
kind: Job
metadata:
  name: small-cpu-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: default-lq
spec:
  parallelism: 1
  completions: 1
  template:
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: [ "/bin/sh" ]
        args: [ "-c", "sleep 600" ]
        resources:
          limits:
            cpu: "1"
      restartPolicy: Never

Following submission, both jobs generate independent workload variants corresponding to the accessible resource flavors. As observed, the large job is allocated to the reserved infrastructure, while the small job is initially admitted into the on-demand resource pool:

> kubectl get kwl

NAME                                              QUEUE        RESERVED IN   ADMITTED   FINISHED   AGE
job-large-cpu-job-e9fdf                           default-lq   default-cq    True                  6s
job-large-cpu-job-variant-flavor-ondemand-3bfbb   default-lq                                       6s
job-large-cpu-job-variant-flavor-reserved-d1e7d   default-lq   default-cq    True                  6s
job-large-cpu-job-variant-flavor-spot-90a0d       default-lq                                       6s
job-small-cpu-job-22f2a                           default-lq   default-cq    True                  6s
job-small-cpu-job-variant-flavor-ondemand-f1a3d   default-lq   default-cq    True                  6s
job-small-cpu-job-variant-flavor-reserved-be9d9   default-lq                                       6s
job-small-cpu-job-variant-flavor-spot-70724       default-lq                                       5s

Since these operations occurred in parallel, the on-demand and spot variants for both jobs generated separate Provisioning Requests to secure the necessary infrastructure:

> kubectl get provreq
NAME                                                               ACCEPTED   PROVISIONED   FAILED   AGE
job-large-cpu-job-variant-flavor-ondemand-541c3-capacity-check-1   True       True                   20s
job-large-cpu-job-variant-flavor-spot-10a5e-capacity-check-1       True       True                   20s
job-small-cpu-job-variant-flavor-ondemand-e2335-capacity-check-1   True       True                   19s
job-small-cpu-job-variant-flavor-spot-00959-capacity-check-1       True       True                   19s

This action resulted in several new nodes being provisioned:

> kubectl get nodes
NAME                                                  STATUS   ROLES    AGE     VERSION
gke-concurrent-admission--ondemand-np-1b04a001-b98r   Ready    <none>   7m57s   v1.35.1-gke.1396002
gke-concurrent-admission--ondemand-np-1b04a001-kbmm   Ready    <none>   8m6s    v1.35.1-gke.1396002
gke-concurrent-admission--reserved-np-f398d4af-0l9c   Ready    <none>   166m    v1.35.1-gke.1396002
gke-concurrent-admission--reserved-np-f398d4af-8vl5   Ready    <none>   166m    v1.35.1-gke.1396002
gke-concurrent-admission-chec-spot-np-6e3d1c6f-9fmk   Ready    <none>   8m1s    v1.35.1-gke.1396002
gke-concurrent-admission-chec-spot-np-6e3d1c6f-f982   Ready    <none>   8m9s    v1.35.1-gke.1396002

Lastly, looking closely at the active workloads reveals that the large job is utilizing reserved capacity, while the small job is running on a spot node pool. This demonstrates that spot resources were provisioned the fastest:

> kubectl get pod -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE                                                  NOMINATED NODE   READINESS GATES
large-cpu-job-2n9fq   1/1     Running   0          58s   10.8.4.15   gke-concurrent-admission--reserved-np-f398d4af-0l9c   <none>           <none>
small-cpu-job-rlgw9   1/1     Running   0          23s   10.8.6.2    gke-concurrent-admission-chec-spot-np-6e3d1c6f-f982   <none>           <none>

Once the large job completes, the small job migrates to the reserved resources. You can observe this process by examining the status of the spot and reserved workload variants:

> kubectl describe kwl job-small-cpu-job-variant-flavor-spot-70724
...
Events:
  Type    Reason                      Age    From                                   Message
  ----    ------                      ----   ----                                   -------
...
  Normal  EvictedDueToDeactivated     5m53s  kueue-workload-controller              The workload is deactivated

> kubectl describe kwl job-small-cpu-job-variant-flavor-reserved-be9d9
...
Events:
  Type    Reason                      Age    From                                   Message
  ----    ------                      ----   ----                                   -------
...
  Normal   Admitted       8m44s                  kueue-admission  Admitted by ClusterQueue default-cq, wait time since reservation was 0s

The pod belonging to the small job has been recreated in the preferred, reserved node pool:

kubectl get pod -o wide
NAME                  READY   STATUS      RESTARTS   AGE     IP          NODE                                                  NOMINATED NODE   READINESS GATES
large-cpu-job-76mrw   0/1     Completed   0          3m34s   10.8.4.17   gke-concurrent-admission--reserved-np-f398d4af-0l9c   <none>           <none>
small-cpu-job-v2q6z   1/1     Running     0          2m41s   10.8.6.3    gke-concurrent-admission--reserved-np-f398d4af-0l9c   <none>           <none>

Once both jobs complete, you can resubmit them to validate the following outcomes:

• Even though new provisioning requests were generated, no new nodes were added because the capacity already existed.
• The pod placement logic followed the expected priorities, resulting in the small job running on the on-demand pool.

> kubectl get po -owide
NAME                  READY   STATUS      RESTARTS   AGE   IP          NODE                                                  NOMINATED NODE   READINESS GATES
large-cpu-job-kx2qs   0/1     Completed   0          73s   10.8.4.18   gke-concurrent-admission--reserved-np-f398d4af-0l9c   <none>           <none>
small-cpu-job-tmvm4   1/1     Running     0          62s   10.8.8.3    gke-concurrent-admission--ondemand-np-1b04a001-d9ds   <none>           <none>

Potential drawbacks and considerations

While concurrent admission successfully solves provisioning bottlenecks, you should be aware of the operational trade-offs this feature introduces:

• Control plane overhead: Creating workload variants for every single acceptable resource flavor multiplies the number of objects Kueue and the Kubernetes API server must track. In environments with rapid, high-volume job submissions, this operation can place noticeable strain on the control plane.
• Redundant costs: The parallel nature of this feature creates a race to provision infrastructure. For instance, if a spot node wins the race and the workload begins, an on-demand node may still have been temporarily spun up. Abandoning this newly provisioned infrastructure can incur minimum billing charges (e.g., 1-minute minimums) from your cloud provider.
• Penalty of workload Migration: The TryPreferredFlavors setting achieves optimal placement by evicting running workloads when better capacity becomes available. Unless your applications utilize checkpointing logic to save their state, evicting a job halfway through its execution results in lost compute time and delays the overall time-to-completion.

Summary

Concurrent Admission introduced in Kueue 0.18 aims to eliminate scheduling bottlenecks by evaluating multiple provisioning options simultaneously. By creating workload variants, Kueue can issue parallel provisioning operations, ensuring that jobs start on the first available infrastructure rather than waiting for a serial evaluation process. Combined with the Provisioning Request API for atomic resource allocation, concurrent admission significantly improves both scaling speed and overall cluster utilization. However, administrators must carefully balance these benefits against the risk of increased costs, control plane overhead and the operational nuances of automated workload migration.

If you found this article helpful, **follow me** for more deep dives into Kueue and the evolving landscape of Kubernetes workload scheduling.