Inside Volcano Scheduler: Sessions, Actions, and the Scheduling Cycle That Prevents Deadlocks (Part 1 of 2)

Inside Volcano Scheduler: Sessions, Actions, and the Scheduling Cycle That Prevents Deadlocks (Part 1 of 2)

A code-first breakdown of how Volcano decides where distributed job pods land.

The controllers article explained how the Job controller creates PodGroups and waits for the Inqueue signal before creating pods. The Queue controller tracks resource consumption. The PodGroup controller extends scheduling to non-VCJob workloads.

But none of those controllers answer the central question.

Once 8 pods exist in the cluster, how does Volcano decide which nodes they land on? And how does it guarantee all 8 land simultaneously rather than one at a time?

The Kubernetes default scheduler places pods one at a time. It sees a pending pod, finds a node with free resources, and binds the pod. For distributed training jobs where all 8 pods must start together, this one-at-a-time approach causes the same deadlock you read about in Part 1 of this series — 6 pods running and holding resources, 2 pods stuck Pending, nothing making progress.

Volcano replaces this with a scheduling cycle built for batch workloads from the ground up.

What the Volcano Scheduler Is

The Volcano scheduler is a separate process running alongside the controller manager. It handles any pod with schedulerName: volcano set in its spec.

Every second it wakes up, looks at the entire cluster, makes placement decisions, and goes back to sleep. It does not create pods. It does not manage job lifecycles. It does one thing: decide which pod runs on which node and when.

Two components make it work: Sessions and Actions. This article covers both. Part 2 covers Plugins, the fairness algorithms and node selection policies.

Asides: The Volcano scheduler does not replace the default Kubernetes scheduler entirely. Both run simultaneously in the same cluster. Pods without schedulerName: volcano are still handled by the default scheduler. Teams migrate workloads to Volcano incrementally without disrupting existing deployments.

Further Reading: https://volcano.sh/en/docs/schduler/

Two Concepts You Must Understand First

1. Session: A Session is a frozen snapshot of the entire cluster state for one scheduling cycle.

At the start of each cycle, the scheduler copies all current state, every job, every node, every queue into a Session object. All scheduling decisions happen against this snapshot. The real Kubernetes cluster is not touched until the session closes.

type Session struct {
  Jobs map[api.JobID]*api.JobInfo // all jobs
  Nodes map[string]*api.NodeInfo // all nodes
  Queues map[api.QueueID]*api.QueueInfo // all queues
  jobOrderFns map[string]api.CompareFn // how to sort jobs
  queueOrderFns map[string]api.CompareFn // how to sort queues
  predicateFns map[string]api.PredicateFn // can this task run on this node?
  nodeOrderFns map[string]api.NodeOrderFn // score nodes for a task
  preemptableFns map[string]api.EvictableFn // which tasks can be evicted?
}

The function maps are empty when the session opens. Plugins fill them in during OnSessionOpen(). Actions then call those functions when making decisions.

Big Word Alert — Snapshot: A snapshot is a point-in-time copy of state. Working from a snapshot means all actions in one cycle see the same consistent view of the cluster. A pod starting on node-5 mid-cycle does not affect the current cycle’s decisions. The scheduler never reads partially-updated state during a cycle.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/framework/session.go

2. Action: An Action is one scheduling operation. The scheduler runs a fixed list of actions every cycle in order. Each action receives the same Session and modifies it.

type Action interface {
  Name() string
  Execute(ssn *Session)
}

The default action sequence:

enqueue → allocate → backfill → preempt → reclaim

Actions run sequentially. Allocate sees the results of Enqueue. Preempt sees the results of Allocate. Order matters.

Big Word Alert — Action: In Volcano’s scheduler, an action is not a reaction to a specific event. It is a standing operation running every cycle regardless of what changed. Enqueue runs every second, not only when a new job arrives. A job unable to be enqueued last cycle gets reconsidered automatically this cycle without any external trigger.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/actions/

The Scheduling Cycle

The entire scheduling cycle is three lines:

func (pc *Scheduler) runOnce() {
  ssn := framework.OpenSession(pc.cache, plugins, configurations)
  for _, action := range actions {
    action.Execute(ssn)
  }
  framework.CloseSession(ssn)
}

Open a session. Run every action. Close the session.

This runs every schedulePeriod (default: 1 second). Everything else in the scheduler is implementation detail around those three lines.

OpenSession does two things: copies cluster state from the SchedulerCache into a fresh Session, then calls OnSessionOpen() on every plugin. Each plugin registers its callbacks into the Session’s function maps.

CloseSession does two things: calls OnSessionClose() on every plugin, then writes results back to Kubernetes — pod bindings, PodGroup status updates, queue resource counter updates.

Asides: The schedulePeriod is configurable. Setting it lower than 1 second increases scheduling throughput but increases CPU usage and API server load. Setting it higher reduces load but increases the time a job waits before the next cycle sees it. The default of 1 second works well for most production deployments.

The Enqueue Action

Enqueue moves PodGroups from Pending to Inqueue.

func (enqueue *Action) Execute(ssn *framework.Session) {
  for _, job := range ssn.Jobs {
    if job.IsPending() {
      if ssn.JobEnqueueable(job) {
        job.PodGroup.Status.Phase = scheduling.PodGroupInqueue
      }
    }
  }
}

JobEnqueueable checks whether the job’s queue has capacity for the job’s minimum resources right now. If the queue is at its limit, the job stays Pending. If there is room, it moves to Inqueue.

This is the Inqueue gate from the controllers article. The Job controller was waiting for exactly this transition. Once the PodGroup moves to Inqueue, the Job controller creates all pods simultaneously on its next reconcile cycle.

Why run Enqueue every cycle? A job submitted yesterday when the cluster was full fits today because other jobs finished. Enqueue rechecks all Pending jobs every second and admits them the moment capacity opens up.

Asides: JobEnqueueable checks queue-level capacity, not node-level capacity. A job moves to Inqueue as long as the queue has room for its minimum resources. Node-level placement happens in the Allocate action after pods exist. In rare cases a job moves to Inqueue, pods get created, and Allocate finds no valid node for one of them. Those pods stay Pending until a node frees up.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/actions/enqueue/enqueue.go

The Allocate Action

Allocate is the main action. It assigns pending tasks to nodes.

The logic is three nested loops:

For each queue (ordered by queue priority):
  For each job in the queue (ordered by job priority):
    For each pending task in the job:
      Filter nodes using predicates (can this task run here?)
      Score the remaining nodes (which node is best?)
      Assign task to the highest-scoring node

Predicates filter out nodes where a task cannot run wrong CPU count, missing GPU, node selector mismatch. Scoring ranks the remaining nodes by fit quality. Both predicates and scoring come from plugins registered during OpenSession(). The allocate action itself has no opinion on what “best” means.

For gang jobs, allocate does not commit pods immediately. It uses a two-step approach: Pipeline first, then Allocate.

Pipeline marks a task as tentatively reserved on a node. The node’s resources move from Idle to Pipelined. The pod is not yet bound. Once all minAvailable tasks are Pipelined, all of them commit to Allocated at once. At session close, all pods get bound to their nodes simultaneously.

If the gang cannot be fully assembled — no valid node exists for task 7 out of 8 — all previous Pipeline operations roll back automatically. No tasks are committed. The next cycle tries again from scratch.

Big Word Alert — Atomic: An atomic operation either completes entirely or has no effect at all. Gang assembly in Volcano is atomic. Either all minAvailable tasks commit together or none of them do. The scheduler records every Pipeline operation internally and rolls them all back if the full gang cannot be assembled in one cycle. The cluster never ends up with a partial gang holding resources while waiting for the rest.

Asides: The allocate action processes queues in order of their deserved share — queues using fewer resources than they are entitled to go first. Within a queue, jobs are ordered by priority and resource share. These ordering decisions come from plugins registered during OpenSession(). Changing the scheduling order means changing a plugin, not touching allocate.go.

Further Reading: github.com/volcano-sh/volcano/blob/master/pkg/scheduler/actions/allocate/allocate.go

Gang Scheduling End to End

Now you can trace a gang job from YAML to running pods through the scheduler:

1. You submit a VCJob with minAvailable: 8
2. Job controller creates a PodGroup with minMember: 8, phase: Pending
3. Enqueue action sees the Pending PodGroup. JobEnqueueable checks queue capacity. Queue has room. PodGroup moves to Inqueue.
4. Job controller sees Inqueue. Creates all 8 pods. All 8 enter TaskStatus: Pending.
5. Next scheduling cycle: Allocate action sees 8 pending tasks for this job.
6. Allocate uses Statement.Pipeline() to tentatively reserve node-A for task-0, node-B for task-1, and so on through task-7.
7. After all 8 tasks are Pipelined: job.IsReady() returns true.
8. Statement.Commit() moves all 8 from Pipelined to Allocated at once.
9. Session closes. SchedulerCache sends Bind requests to Kubernetes for all 8 pods simultaneously.
10. Kubelet starts all 8 pods. PodGroup moves to Running.

If at step 6 no valid node exists for task-7: Statement.Discard() rolls back tasks 0 through 6. No pods are bound. The next cycle tries again.

This is the exact moment deadlock is prevented. Pods are never partially placed. Either all 8 commit together or none commit at all.

Design Trade-offs

Pros of the scheduling design

• One consistent snapshot per cycle means actions never see partially-updated state mid-cycle
• Actions are decoupled from policies. Changing a fairness algorithm means changing a plugin, not touching allocate.go
• Statement rollback means partial gang assemblies never leave corrupted resource accounting behind
• The SchedulerCache absorbs all API server reads. Scheduling throughput does not scale with API server capacity

Cons of the scheduling design

• The 1-second cycle introduces scheduling latency. A job submitted at cycle+0.9s waits up to 0.9 seconds before the next cycle sees it
• Snapshot cost grows with cluster size. A cluster with 10,000 nodes and 100,000 pods takes longer to snapshot than a small cluster
• Large gang jobs competing simultaneously cause repeated Pipeline rollbacks across many cycles before a gang fully assembles
• The session model means changes made by one action are only visible to later actions through the shared session state, not through real cluster state

Asides: The repeated gang assembly rollback problem becomes significant when many large gang jobs compete simultaneously for the same nodes. Volcano addresses this with a Pipelined state timeout — a task stays Pipelined across multiple cycles rather than rolling back every time. This is worth configuring for large jobs in high-contention clusters.

Where to Go Next

This article covered the scheduling cycle, session lifecycle, the Enqueue and Allocate actions, Statement-based atomic operations, and gang scheduling end to end.

Part 2 covers the plugin system: the Gang plugin enforcing all-or-nothing placement, DRF (Dominant Resource Fairness) deciding which job within a queue runs next, the Proportion plugin dividing cluster capacity across queues by weight, and NodeOrder with Predicates handling node selection.

• Scheduling cycle: pkg/scheduler/scheduler.go
• Session lifecycle: pkg/scheduler/framework/session.go
• Enqueue action: pkg/scheduler/actions/enqueue/enqueue.go
• Allocate action: pkg/scheduler/actions/allocate/allocate.go
• Full codebase: https://github.com/volcano-sh/volcano