How Volcano Adds Safe Per Queue Admission Limits for AI Workloads

Many teams run production and research AI workloads on the same Kubernetes cluster. Production training jobs need predictable admission. Research teams submit large hyperparameter sweeps, benchmark runs, and retry batches. Research jobs often wait longer, yet platform teams still need a limit on admitted work.

Volcano queues separate these groups. Before queue scoped overcommit, the Overcommit plugin used one cluster wide overcommit-factor when a job moved from Pending to Inqueue. Every Queue received the same admission policy.

A single factor creates a tradeoff.

  • A high factor accepts a large research backlog. Production Queues receive the same wide admission range.
  • A low factor keeps production admission tight. Volcano rejects research sweeps earlier.

Queue scoped overcommit adds an optional Queue level admission limit. The existing cluster wide limit stays in place.

Admission Flow

Volcano runs the global check first. It then applies the Queue limit only when the Queue has the opt in annotation and the feature gate is enabled.

What Inqueue Means

Inqueue means Volcano accepted a job for scheduling, but the job does not yet hold cluster resources.

Inqueue is not a GPU reservation. Inqueue does not promise immediate start time.

Overcommit controls how much allocated work and admitted waiting work Volcano accepts. Resorce Allocation still follows Queue policy, Capacity policy, priority, gang rules, and Node feasibility.

The feature controls admission pressure. Physical GPU capacity does not change.

A Practical AI Platform Example

Assume a platform owns 100 GPU units.

Cluster wide overcommit factor: 2.0
Maximum Queue factor: 1.5

Production training Queue:
  Deserved GPUs: 40
  Queue factor: 1.0

Research sweeps Queue:
  Deserved GPUs: 40
  Queue factor: 1.5

The production Queue receives a 40 GPU admission budget. The research Queue receives a 60 GPU admission budget. Each value includes allocated work and Inqueue work. The budget also respects Queue capability or derived real capability.

Suppose production jobs use 25 GPUs. Volcano admits up to 15 additional GPU units of waiting production work under this Queue policy. Suppose research jobs use 15 GPUs. Volcano admits up to 45 additional GPU units of waiting research work under its policy.

Each group receives a policy suited to its work.

  • Production work keeps a narrow admitted backlog.
  • Research users submit bursty sweep jobs without requiring a loose policy for the production Queue.
  • The global cluster check still runs for every job.

A Queue factor never bypasses the administrator’s cluster wide limit.

Budget Example:

This is the same admission behaviour verified on a local Kind cluster during the implementation review.

The Queue Configuration

The alpha implementation uses a Queue annotation.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research-sweeps
  annotations:
    volcano.sh/overcommit-factor: "1.5"
spec:
  deserved:
    nvidia.com/gpu: "40"

The factor works only when the QueueScopedOvercommit feature gate is enabled. Existing behaviour stays unchanged while the gate is disabled.

The Safety Model

Queue scoped overcommit adds an extra check. Volcano admits a job only when every relevant check passes.

Global cluster admission check passes
AND
Leaf Queue admission check passes
AND
Each annotated parent Queue check passes

The global check protects the full cluster. The Queue check protects the workload group. An annotated parent Queue check protects a department or tenant group in a hierarchy.

The Queue factor has three limits.

  1. The Queue annotation provides the requested factor.
  2. The scheduler argument max-queue-overcommit-factor limits every Queue request.
  3. The global overcommit-factor limits the Queue cap.

For a Queue with deserved resources, Volcano uses the smaller of these two values as the Queue admission budget.

Queue real capability

Effective deserved resources × effective Queue factor

Volcano uses the larger value from deserved and guarantee, following the existing Capacity plugin model.

This design blocks a Queue annotation from becoming an unlimited admission switch.

Hierarchical Queues

Many platform teams use a parent Queue for a department and child Queues for teams.

research
├── vision
├── language
└── simulation

The feature tracks Inqueue resources for a leaf Queue and each ancestor. Volcano checks every annotated ancestor during admission. A child Queue therefore does not escape the parent policy.

This helps a shared research division. The division administrator sets an upper admission boundary. Team administrators receive room for their own Queue policy inside that boundary.

The scheduler updates these counters during each scheduling session. The work depends on Queue depth and requested resource dimensions. The scheduler does not scan every Queue or every job for each admission attempt.

What the Feature Does Not Change

Queue scoped overcommit changes Inqueue admission only.

  • The feature does not change Queue deserved resources.
  • The feature does not change Queue capability.
  • The feature does not change allocation fairness.
  • The feature does not change reclaim policy.
  • The feature does not reserve GPUs or guarantee start time.

Existing Volcano components keep ownership of those decisions.

How I Found the Problem

This work began while I studied issue #4530, Overcommit factor for the root Queue.

In the issue, Hajnal Máté raised a root Queue problem for hierarchical GPU clusters. Static Queue capability becomes hard to maintain when cluster capacity changes or new GPU resource types appear. Jesse Stutler asked for a proposal describing how an overcommit model should work.

While tracing the Capacity and Overcommit paths, I found a related gap. Volcano supported one cluster wide admission factor, but no safe Queue specific admission limit.

My first proposal covered two directions.

  1. Queue capacity and reclaim ceilings.
  2. Inqueue admission.

Maintainer feedback raised an important risk. A Queue factor must not let one Queue overwhelm the cluster.

Following that feedback, I separated the admission work into issue #5687, Add opt in queue scoped overcommit admission controls.

I also discussed the scope with maintainers during Volcano community meetings. The final design keeps the global overcommit check as the cluster boundary and adds Queue limits as extra checks.

From Design to Implementation

I wrote the design first in PR #5746, Queue scoped overcommit design.

The design covers the Queue budget, factor caps, hierarchy behaviour, validation rules, accounting, rollout plan, and tests. Both maintainers approve the design.

The implementation follows the reviewed design in PR #5810, Queue scoped admission limits.

The pull request adds:

  • The alpha QueueScopedOvercommit feature gate.
  • The volcano.sh/overcommit-factor Queue annotation.
  • Queue webhook validation.
  • Global, Queue, and annotated ancestor admission checks.
  • Incremental Queue and ancestor resource accounting.
  • Unit tests and an end to end Queue overcommit test suite.
  • Operator documentation and the max-queue-overcommit-factor scheduler argument.

Using the Feature Safely

Start with one Queue and a small factor.

Keep Queue write access with cluster administrators. The annotation changes admission policy for a cluster scoped resource.

Set clear deserved resources for every Queue using the feature. Resource dimensions absent from deserved do not receive Queue scoped overcommit.

Set max-queue-overcommit-factor at or below the global overcommit-factor.

Monitor Pending and Inqueue jobs after rollout. Compare Queue admission behaviour before and after enabling the gate.

Production training and research sweeps form a useful first rollout pair.