Inside Volcano Webhooks: Admission, Validation, and the Gate That Rejects Bad Jobs Before They Run

A code-first breakdown of how Volcano checks and rewrites every workload before the API server stores it.

You run kubectl apply on a Volcano Job with no tasks. Where does Kubernetes stop the broken object, and how early?

The Kubernetes API server stores whatever passes authentication and authorization. The default path knows nothing about Volcano’s rules. The server does not know a Job needs at least one task. The server does not know an MPI job needs a master pod. Volcano teaches the server these rules through an admission webhook, an HTTPS endpoint the server calls before it writes your object to storage. The webhook says allow or deny. On deny, no pod ever starts.

This article traces one request through Volcano’s webhook layer, from kubectl to the verdict, using the code in pkg/webhooks.

Big Word Alert: admission controller. An admission controller is code the API server runs to inspect a request after auth, but before storage. Webhooks are the pluggable, out-of-process kind. Kubernetes also ships admission controllers compiled into the server, such as ResourceQuota.

What the Volcano Webhook Layer Is

The webhook layer is an HTTPS server running inside Volcano, separate from the scheduler and the controller manager. The server exposes one path per resource and operation, for example /jobs/validate, /pods/mutate, and /queues/validate.

Volcano registers each path with the API server through a standard Kubernetes object. Here is the Job validator registration, in pkg/webhooks/admission/jobs/validate/admit_job.go:

ValidatingConfig: &whv1.ValidatingWebhookConfiguration{
	Webhooks: []whv1.ValidatingWebhook{{
		Name: "validatejob.volcano.sh",
		Rules: []whv1.RuleWithOperations{{
			Operations: []whv1.OperationType{whv1.Create, whv1.Update},
			Rule: whv1.Rule{
				APIGroups:   []string{"batch.volcano.sh"},
				APIVersions: []string{"v1alpha1"},
				Resources:   []string{"jobs"},
			},
		}},
	}},
}

Read the registration as a filter. The API server calls this webhook only on Create and Update, only for jobs in the batch.volcano.sh group. Every other object skips this call.

The whv1 import resolves to k8s.io/api/admissionregistration/v1, a core Kubernetes API group. Volcano writes a native Kubernetes object to register. The webhook mechanism belongs to Kubernetes, not to Volcano.

Aside: the same layer serves native Pod objects through /pods/validate and /pods/mutate. So the webhook layer reaches past Volcano's own CRDs and reviews a built-in Kubernetes type. You review any resource you register, not only your own.

Further Reading: Kubernetes admission controllers, https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Two Concepts You Must Understand First

Two structures carry the whole flow. Learn both, and the rest reads cleanly.

The Admission Review

The AdmissionReview is the envelope the API server and your webhook both speak. The request body is not your raw Job. The body is an AdmissionReview wrapping your object plus metadata.

AdmissionReview
├── Request
│   ├── UID          match key for this call
│   ├── Operation    CREATE or UPDATE
│   ├── Object       your Job
│   └── OldObject    the previous Job, on update
└── Response
    ├── UID          copied from Request.UID
    ├── Allowed      true or false
    └── Result.Message   reason, on deny

The request carries your object. The response carries the verdict. Both travel inside the same envelope type.

Big Word Alert: UID. The UID is a unique id the API server stamps on each request. The server runs many admissions at the same time. The UID pairs each answer with its question.

The Admit Function

The admit function holds the rules for one resource. The type signature stays fixed across every resource:

type AdmitFunc func(admissionv1.AdmissionReview) *admissionv1.AdmissionResponse

Each path binds to one admit function. The /jobs/validate path binds to AdmitJobs. The /pods/mutate path binds to a pod mutation function. The transport code stays shared. The rules stay specific.

Aside: this split is the reason one server handles a dozen resources. Volcano writes the HTTP plumbing once, then plugs a different admit function into each path. Adding a new webhook means writing a new admit function and a new registration, nothing more.

Further Reading: AdmissionReview API types, https://kubernetes.io/docs/reference/config-api/apiserver-admission.v1/

The Request Cycle

One function drives every webhook call: serve() in pkg/webhooks/router/server.go. The function reads the request, decodes the envelope, calls the admit function, and writes the reply.

func serve(w http.ResponseWriter, r *http.Request, admit AdmitFunc) {
   r.Body = http.MaxBytesReader(w, r.Body, MaxRequestBody)
   body, err := io.ReadAll(r.Body)
   if err != nil {
      http.Error(w, "request body too large", http.StatusRequestEntityTooLarge)
      return
   }
  
   ar := admissionv1.AdmissionReview{}
   deserializer := schema.Codecs.UniversalDeserializer()
   if _, _, err := deserializer.Decode(body, nil, &ar); err != nil {
      reviewResponse = util.ToAdmissionResponse(err)
   } else {
      reviewResponse = admit(ar)
   }
  
   response := createResponse(reviewResponse, &ar)
   resp, _ := json.Marshal(response)
   w.Write(resp)
}

The cycle runs the same seven steps on every call:

kubectl apply
    |
    v
API server  --> matches a webhook rule?
    | yes
    v
HTTPS POST /jobs/validate
    |
    v
serve(): read body, cap 3 MB
    |
    v
decode into AdmissionReview
    |
    v
admit(ar)  -->  AdmitJobs runs the rules
    |
    v
AdmissionResponse { Allowed, Message, UID }
    |
    v
API server: allow -> store in etcd
            deny  -> return error to kubectl

The response copies the request UID before the reply goes out. createResponse performs the copy:

response.Response = reviewResponse
response.Response.UID = ar.Request.UID

Aside: the UID copy looks small, but skipping the line breaks the webhook. The API server rejects any response with no matching UID. createResponse also clears the object fields on the way out, since the server already holds the object and resending wastes bandwidth.

Big Word Alert: OOM, out of memory. When a process asks for more memory than the node holds, the Linux kernel kills the process. The MaxBytesReader call caps the body at 3 MB, so a crafted large body triggers no such kill. The 3 MB value matches the API server default, so both sides agree on the limit.

Further Reading: Dynamic Admission Control, https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

The Validate Path

A validating webhook reads the object and returns allow or deny. The webhook changes nothing. AdmitJobs shows the shape. Start optimistic, then look for a reason to reject.

func AdmitJobs(ar admissionv1.AdmissionReview) *admissionv1.AdmissionResponse {
   job, err := schema.DecodeJob(ar.Request.Object, ar.Request.Resource)
   if err != nil {
      return util.ToAdmissionResponse(err)
   }
  
   reviewResponse := admissionv1.AdmissionResponse{}
   reviewResponse.Allowed = true
  
   switch ar.Request.Operation {
     case admissionv1.Create:
      msg = validateJobCreate(job, &reviewResponse)
   }
  
   if !reviewResponse.Allowed {
      reviewResponse.Result = &metav1.Status{Message: strings.TrimSpace(msg)}
   }
   return &reviewResponse
}

The response opens with Allowed = true. The validation functions try to disprove the assumption. One rule inside validateJobCreate rejects a Job with no tasks:

if len(job.Spec.Tasks) == 0 {
	reviewResponse.Allowed = false
	return "No task specified in job spec"
}

Why check at admission and not later in the controller? Because a rejection here reaches the user at submit time, from the API server, before any pod, PodGroup, or scheduler work begins. The error stops the object at the front door.

Aside: a validating webhook does not change the object. The webhook only reads and votes. Any change to the object belongs to a mutating webhook, covered next. Mixing the two roles in one handler leads to surprises, since the API server treats the two kinds differently.

Further Reading: Volcano Job API, https://volcano.sh/en/docs/vcjob/

The Mutate Path

A mutating webhook returns a patch. The webhook rewrites the object before storage, for example to set a default or inject configuration. The API server runs mutating webhooks first, then validating webhooks last.

kubectl apply
    |
    v
[ Mutating webhooks ]   patch the object: defaults, injected config
    |
    v
[ Validating webhooks ] allow or deny the final object
    |
    v
store in etcd

Volcano’s MPI plugin shows a real mutation. An MPI job needs setup no user wants to write by hand. The master pod needs the list of worker hostnames. Every pod needs an open SSH port. The plugin performs the setup at pod creation, in pkg/controllers/job/plugins/distributed-framework/mpi/mpi.go:

if helpers.GetTaskKey(pod) == mp.masterName {
	workerHosts = mp.generateTaskHosts(job.Spec.Tasks[taskIndex], job.Name)
	env = v1.EnvVar{Name: MPIHost, Value: workerHosts}
	isMaster = true
}

The plugin builds a comma-separated list of worker hostnames and injects the list as the MPI_HOST environment variable on the master. The mpirun command inside the master reads MPI_HOST to find its workers. The plugin also opens SSH port 22 on every container, since MPI launches remote processes over SSH.

Big Word Alert: idempotent. A mutating webhook should produce the same object no matter how many times the server applies the webhook. The API server sometimes re-invokes mutating webhooks after other mutations run. A non-idempotent patch injects the same sidecar twice and corrupts the pod.

Aside: the MPI logic runs across two moments, and mixing the two confuses readers. An admission-time helper adds a master-depends-on-worker rule when you submit the Job. A runtime plugin injects the env var and ports when each pod gets created. Same MPI concept, two points in the Job lifecycle.

Further Reading: Mutating webhook reinvocation policy, https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#reinvocation-policy

A Denied Job End to End

Trace one broken Job through the whole path. The Job has zero tasks.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: empty-job
spec:
  minAvailable: 1
  tasks: []

Ten steps, with the state at each point:

  1. You run kubectl apply -f empty-job.yaml. State: request in flight.
  2. The API server authenticates and authorizes you. State: request accepted.
  3. The server matches empty-job against registered rules. The Job hits validatejob.volcano.sh. State: matched.
  4. The server wraps the Job in an AdmissionReview, stamps a UID, and sends an HTTPS POST to /jobs/validate. State: posted.
  5. serve() reads the body under the 3 MB cap. State: body read.
  6. serve() decodes the body into the AdmissionReview struct. State: decoded.
  7. serve() calls AdmitJobs. The function sets Allowed = true, then runs validateJobCreate. State: under review.
  8. validateJobCreate finds len(job.Spec.Tasks) == 0. The function flips Allowed = false and returns the message. State: denied.
  9. createResponse copies the request UID onto the response. serve() marshals and writes the reply. State: replied.
  10. The API server reads the denial, skips storage, and returns the message to kubectl. State: rejected.

The user sees:

Error from server: error when creating "empty-job.yaml":
admission webhook "validatejob.volcano.sh" denied the request: No task specified in job spec

No PodGroup gets created. No pod gets scheduled. The bad object never reaches etcd. The allow branch runs the same nine steps, then step ten stores the object instead of rejecting.

Further Reading: Volcano webhooks source, https://github.com/volcano-sh/volcano/tree/master/pkg/webhooks

Design Trade-offs

Pros of the webhook design

  • You enforce rules core Kubernetes does not know, at the moment of write
  • You reject bad objects before any pod or scheduler work starts, so failures land early and cheap.
  • You review any resource you register, including native Pods, not only Volcano CRDs.
  • Mutating webhooks set defaults for users, so users write less boilerplate.
  • Policy stays declarative and lives in one registered config, versioned with the cluster.

Cons of the webhook design

  • Your webhook sits on the write path. Every matching request waits on your reply, so a slow webhook slows the cluster.
  • A down webhook breaks writes, unless you set failurePolicy to Ignore, which then admits objects with no check.
  • TLS setup and certificate rotation add operational work. The API server talks to your webhook over HTTPS only.
  • Mutating webhooks need idempotency and careful ordering, or repeated patches corrupt objects.
  • Debugging runs one step removed. The failure shows at kubectl, and you trace back to a webhook pod log.

Big Word Alert: failurePolicy. This field tells the API server what to do when your webhook is unreachable. Fail blocks the write. Ignore admits the object with no check. You trade safety against availability, and you set the value on purpose.

Further Reading: failurePolicy and webhook availability, https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy

Where to Go Next

This article covered the webhook layer, the AdmissionReview envelope, the admit function, the shared serve() cycle, the validate and mutate paths, and one denied Job traced end to end. The webhook is the review step before storage. serve() handles transport. Your admit function holds the rules.

The next article covers Volcano’s other admission targets in depth: Queue hierarchy validation, PodGroup checks, and the pod mutation chain that wires jobs to the Volcano scheduler.

The flow you traced through Volcano is the Kubernetes admission contract. Volcano gives you a clean, readable copy to study.

Further Reading: