Inside Volcano Scheduler: Plugins, DRF, and How Queue Fairness Actually Works (Part 2 of 2)

A code-first breakdown of how Volcano decides which job runs next and which node wins.

Where We Left Off

Part 1 covered the scheduling cycle: a fresh Session every second, five actions running in order, and gang scheduling through Pipeline and atomic Commit. The allocate action placed pods on nodes. But one question was left open.

How does the allocate action know which queue to process first? Which job inside a queue goes next? Which node is best for a given task?

The allocate action has no opinions. Plugins provide all of those answers.

What a Plugin Is

A plugin is a policy provider. It does not schedule anything. It registers decision functions into the Session during OnSessionOpen(). Actions call those functions when making decisions.

type Plugin interface {
  Name() string
  OnSessionOpen(ssn *Session)
  OnSessionClose(ssn *Session)
}

Every plugin runs OnSessionOpen() at the start of each cycle. This is where a plugin does its calculations and registers its callbacks into the Session’s function maps.

// A plugin registers a callback like this:
ssn.AddJobOrderFn(plugin.Name(), func(l, r interface{}) int {
  // return negative if l goes before r
  // return positive if r goes before l
  // return 0 if equal
})

The Session then holds that function. When the allocate action sorts jobs, it calls every registered JobOrderFn and combines the results. Multiple plugins register ordering functions. They are all consulted.

Big Word Alert — Callback: A callback is a function you hand to someone else to call later. The plugin hands its function to the Session. The Session stores it. The allocate action calls it when needed. The plugin never calls the allocate action directly. The allocate action never calls the plugin directly. They communicate through the Session’s function maps.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/framework/interface.go

The Gang Plugin

The gang plugin enforces the all-or-nothing rule. No other plugin does this. Without the gang plugin, the allocate action places tasks one at a time and partial placements accumulate — the exact deadlock problem Volcano exists to solve.

The gang plugin registers three callbacks.

JobValidFn — is this job worth considering at all?

validJobFn := func(job *api.JobInfo) *api.ValidateResult {
  vtn := job.ValidTaskNum()
  if vtn < job.MinAvailable {
    return &ValidateResult{Pass: false, Reason: "NotEnoughPods"}
  }
  return nil
}

ValidTaskNum counts tasks not in a permanently failed state. If a job needs 8 pods but one failed permanently and will never restart, the gang can never be assembled. The gang plugin marks the job as invalid. The allocate action skips it entirely and does not waste cycles trying.

JobReadyFn — is this job fully placed and ready to commit?

ssn.AddJobReadyFn(gp.Name(), func(job *api.JobInfo) bool {
  return job.CheckTaskReady() && job.IsReady()
})

The allocate action calls this after each Pipeline operation. Once all minAvailable tasks are Pipelined, IsReady() returns true and the Statement commits the full gang. This is the trigger for the atomic commit from Part 1.

PreemptableFn — which tasks from this job can be evicted?

preemptableFn := func(preemptor *api.TaskInfo, preemptees []*api.TaskInfo) ([]*api.TaskInfo, int) {
  for _, preemptee := range preemptees {
    job := ssn.Jobs[preemptee.Job]
    if job.ReadyTaskNum() > job.MinAvailable {
      victims = append(victims, preemptee)
    }
  }
  return victims
}

A task is evictable only if removing it still leaves the job above its MinAvailable threshold. The gang is never broken below its minimum. A job with minAvailable: 8 and 10 running tasks has 2 tasks available for eviction. A job with exactly 8 running tasks has zero.

Asides: The gang plugin is the reason ValidTaskNum() exists as a separate method from ReadyTaskNum(). ValidTaskNum counts tasks in any non-permanently-failed state — Pending, Pipelined, Allocated, Running. ReadyTaskNum counts only Allocated and Running tasks. JobValidFn uses ValidTaskNum to check whether the gang is still achievable. JobReadyFn uses ReadyTaskNum to check whether the gang is currently placed.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/plugins/gang/gang.go

The DRF Plugin

DRF stands for Dominant Resource Fairness. It decides which job within a queue goes next. The job with the lowest dominant resource share gets scheduled before the job with the higher share.

Your dominant resource is whichever resource type you are consuming the most of relative to total cluster capacity.

A concrete example:

Cluster: 10 CPUs, 20 GB memory

Job A: using 4 CPUs and 2 GB
  CPU share: 4/10 = 40%
  Memory share: 2/20 = 10%
  Dominant resource: CPU
  DRF share: 40%

Job B: using 1 CPU and 8 GB
  CPU share: 1/10 = 10%
  Memory share: 8/20 = 40%
  Dominant resource: Memory
  DRF share: 40%

Both have equal DRF shares. They are treated as equals. Now say Job A gets one more allocation — it reaches 80% CPU share. Job B is still at 40%. DRF schedules Job B next because it has consumed less of its dominant resource.

The plugin tracks this per job:

type drfAttr struct {
  share float64 // dominant resource share (0.0 to 1.0)
  dominantResource string // "cpu" or "memory" or "nvidia.com/gpu"
  allocated *api.Resource
}

DRF registers a JobOrderFn. The allocate action calls it when sorting jobs. The job with the lower share value goes first.

Big Word Alert — Dominant Resource Fairness: DRF is a scheduling policy from a 2011 research paper by Ghodsi et al. It extends fair sharing to environments with multiple resource types. Without it, a cluster with both CPU-heavy jobs and memory-heavy jobs would need to pick one resource type to optimize for. DRF avoids this by using each job’s dominant resource as its fairness metric, so CPU-heavy jobs compete on CPU share and memory-heavy jobs compete on memory share. Both get a fair deal without one crowding out the other.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/plugins/drf/drf.go

The Proportion Plugin

While DRF handles fairness between jobs, proportion handles fairness between queues. It calculates how much of the cluster each queue deserves based on its weight.

type queueAttr struct {
  deserved *api.Resource // fair share this cycle
  allocated *api.Resource // what the queue actually holds
  request *api.Resource // what its pending jobs are asking for
  guarantee *api.Resource // minimum floor - always protected
  capability *api.Resource // maximum ceiling - never exceeded
  reclaimable bool
}

The calculation runs inside OnSessionOpen() every cycle. It is iterative because of capability caps:

Start with remaining = all cluster resources

Repeat until no changes:
  For each unsatisfied queue:
    deserved = remaining × (queue.weight / total_remaining_weight)
    cap deserved at queue.capability
    floor deserved at queue.guarantee
    if queue.request <= deserved: mark queue satisfied
      reclaim its unused deserved back to the pool

A queue with capability: 30 CPUs gets capped there even if its proportional weight would give it 57. The unclaimed 27 CPUs flow to other queues in the next iteration. A queue whose jobs are only asking for 20 CPUs takes 20 even if deserved says 46. Its leftover goes back to the pool too.

The result is a deserved value for every queue. The allocate action uses this to order queues — queues using less than their deserved share go first. The reclaim action uses this to find queues using more than their deserved share and evict their jobs to return borrowed capacity.

Asides: deserved is not a reservation. No block of cluster resources is pre-assigned to a queue. The cluster is one shared pool. deserved is a benchmark computed each cycle. If a queue’s allocated is above deserved, it borrowed from idle queues. When those queues need their resources back, reclaim takes them. The borrowing and lending happen automatically based on reclaimable: true.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/plugins/proportion/proportion.go

NodeOrder and Predicates

These two plugins handle node selection. They answer the question the allocate action asks for every pending task: which node should this task run on?

Predicates Plugin

Predicates filter. Before scoring any node, the scheduler removes every node where the task cannot run. This is a hard yes/no check.

Checks include:

  • Does the node have enough CPU and memory for this task’s requests?
  • Does the node match the pod’s node selector labels?
  • Does the pod tolerate the node’s taints?
  • Are the required ports free on the node?
  • Does the node have the GPU model the pod requires?

A node failing any predicate is removed from consideration entirely. Scoring never sees it.

NodeOrder Plugin

NodeOrder scores the nodes surviving the predicate filter. Higher score means better fit. The allocate action picks the highest-scoring node.

Scoring factors with configurable weights:

  • LeastRequested: prefer nodes with the most free resources. Spreads load across the cluster.
  • MostRequested: prefer nodes with the least free resources. Packs tightly to keep some nodes empty for auto-scaling.
  • BalancedResource: prefer nodes where CPU and memory usage stay proportional.
  • NodeAffinity: prefer nodes matching the pod’s soft affinity rules.
  • ImageLocality: prefer nodes that already have the container image pulled. Avoids pull latency.

Big Word Alert — Node Affinity: Node affinity rules are labels on nodes combined with rules in the pod spec. A hard affinity rule says “I will not run on any node without this label.” A soft affinity rule says “I prefer nodes with this label but will accept others.” Predicates enforce hard affinity rules at the filter step. NodeOrder implements soft affinity as a scoring bonus at the scoring step.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/plugins/nodeorder/nodeorder.go and github.com/volcano-sh/volcano/blob/master/pkg/scheduler/plugins/predicates/predicates.go

How All Plugins Work Together in One Cycle

Every plugin runs at OpenSession. Here is the full picture of one scheduling cycle with all plugins active:

OPEN SESSION
  proportion.OnSessionOpen():
    calculates deserved for all queues
    registers QueueOrderFn: prefer queues where allocated < deserved

  drf.OnSessionOpen():
    calculates dominant share for all jobs
    registers JobOrderFn: prefer jobs with lower share

  gang.OnSessionOpen():
    registers JobValidFn: skip jobs where ValidTaskNum < MinAvailable
    registers JobReadyFn: commit gang when ReadyTaskNum >= MinAvailable
    registers PreemptableFn: only evict tasks that leave gang above MinAvailable

  nodeorder.OnSessionOpen():
    registers NodeOrderFn: score nodes by LeastRequested, affinity, etc.
  predicates.OnSessionOpen():
    registers PredicateFn: filter nodes by CPU, GPU, selectors, taints

ALLOCATE ACTION
  For each queue ordered by proportion's QueueOrderFn:
    For each job ordered by drf's JobOrderFn:
      gang plugin checks JobValidFn - skip invalid jobs
      For each pending task:
        predicates filter nodes
        nodeorder scores remaining nodes
        Pipeline task on best node
        gang plugin checks JobReadyFn - commit if gang complete

CLOSE SESSION
  Write pod bindings to Kubernetes
  Update PodGroup phases
  Update queue allocated counters

Each plugin contributes one piece of the answer. Proportion decides queue order. DRF decides job order within a queue. Gang decides when to commit and what to protect. Predicates decide which nodes are eligible. NodeOrder decides which eligible node wins.

No single plugin knows about the others. They all register into the Session and the actions call them through the same interface.

Design Trade-offs

Pros of the plugin design

  • Policies are swappable. Replace DRF with priority-based ordering by changing one plugin, no action code touched.
  • Multiple plugins compose. Proportion handles queue order. DRF handles job order. Gang handles gang safety. Each does one thing and they stack cleanly.
  • Adding a new policy means adding one new plugin file and registering it. No existing plugin changes.
  • Plugins are testable in isolation. A plugin that only registers callbacks has no dependencies on cluster state beyond what it reads from the Session.

Cons of the plugin design

  • Debugging scheduling decisions requires tracing through multiple plugins simultaneously. When a job gets stuck, the cause might be in gang, proportion, predicates, or any combination.
  • Plugin interaction is implicit. Proportion registers a QueueOrderFn and DRF registers a JobOrderFn. Neither knows the other exists. A misconfigured plugin silently affects decisions without any error.
  • OnSessionOpen() runs all plugin calculations every cycle even when nothing changed. In large clusters, the proportion calculation across hundreds of queues adds latency to every single cycle.

Asides: The plugin tier system in the configuration file controls how plugins compose for node scoring. If two plugins both register a NodeOrderFn, their scores are combined by default. Tiers let you separate them into primary and fallback groups. If a primary-tier plugin produces a decisive winner, lower-tier plugins are not consulted. This gives you control over which scoring factors dominate without writing new plugin code.

Where to Go Next

You now have the full scheduler picture: the session lifecycle, all five actions, and the five plugins that supply the policies.

The third pillar of Volcano is Webhooks. Webhooks run before any controller or scheduler sees your YAML. They validate incoming Job specs, fill in default values, and inject plugin declarations. Understanding webhooks closes the loop on how a Job goes from raw user input to a fully-specified object the controllers trust.

  • Gang plugin: pkg/scheduler/plugins/gang/gang.go
  • DRF plugin: pkg/scheduler/plugins/drf/drf.go
  • Proportion plugin: pkg/scheduler/plugins/proportion/proportion.go
  • NodeOrder plugin: pkg/scheduler/plugins/nodeorder/nodeorder.go
  • Predicates plugin: pkg/scheduler/plugins/predicates/predicates.go
  • Full codebase: https://github.com/volcano-sh/volcano