Rolling Updates for LLM Serving on Kubernetes: How Kthena Protects Availability

Rolling Updates for LLM Serving on Kubernetes: How Kthena Protects Availability

Changing an LLM serving image looks simple. You edit a container image and apply YAML.

For a live service, the change has a harder requirement. Requests must keep reaching ready model workers while new workers start, load weights, join distributed roles, and pass readiness checks.

The problem

An LLM serving Pod often needs time before serving traffic. The runtime pulls an image, starts a process, loads model weights, reserves accelerator memory, and passes readiness checks. A Pod exists long before a client request should reach that Pod.

Deleting every old Pod after an image change creates a capacity gap. For multi Pod serving, deletion also breaks a serving unit whose Roles depend on each other. A rollout needs a limit on unavailable serving capacity.

Kthena treats a ServingGroup as one independently working serving unit. A group includes a Prefill Role, a Decode Role, or another role layout. The controller updates whole groups under ServingGroupRollingUpdate.

Big word alert: availability Availability means ready serving capacity. A running container does not prove availability. Kthena waits for a ServingGroup readiness result before treating replacement capacity as available.

The three layer serving model

Kthena uses three layers to describe one serving workload.

ModelServing is the parent custom resource. You apply one ModelServing YAML file. Its specification declares the desired number of serving groups, Role templates, recovery rules, scaling rules, and rollout strategy.

ServingGroup is one replica of your serving service. A ServingGroup includes every Role required to process a request. Kthena names groups with an ordinal, such as qwen-0 and qwen-1.

Role is one functional part inside a ServingGroup. A simple deployment has one Decode Role. A Prefill Decode deployment has a Prefill Role and a Decode Role. Each Role has its own Pod template and replica count.

This hierarchy explains ServingGroup rolling update. Kthena replaces a complete group, rather than selecting one random Pod from the ModelServing. Role rolling update uses a smaller replacement unit.

The objects involved

Three Kubernetes objects carry most rollout state.

ModelServing

Holds your desired configuration and reports rollout status.

ControllerRevision

Stores a durable snapshot of Role templates for one revision.

Pods and Services: Carry labels for the ServingGroup, Role, revision, and role template hash.

The controller also keeps a datastore cache of ServingGroup and Role state. Kubernetes objects remain the durable source of truth. After a controller restart, informer events rebuild cache state.

A small ModelServing example

This example asks for four ServingGroups. Each group contains one Decode Role.

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: qwen
spec:
  replicas: 4
  template:
    roles:
      - name: decode
        replicas: 1
        entryTemplate:
          spec:
            containers:
              - name: engine
                image: registry.example/qwen:v1
  rolloutStrategy:
    type: ServingGroupRollingUpdate
    rollingUpdateConfiguration:
      maxUnavailable: 1
      partition: 0

The controller creates four logical groups:

qwen-0
qwen-1
qwen-2
qwen-3

Each number is a ServingGroup ordinal. The ordinal identifies a logical slot. Kthena uses this slot for naming and partition rules.

Two controls define rollout safety

maxUnavailable limits how many ServingGroups lack ready capacity during the update. partition protects lower ordinals from update.

With maxUnavailable: 1, Kthena waits for a replacement group to become ready before taking down another healthy group. This produces a sequential rollout for four groups.

With partition: 2, groups qwen-0 and qwen-1 stay on their current revision. Groups whose ordinal is 2 or higher are eligible for replacement.

Use a partition for a staged rollout. Start with a high partition, update a small set of groups, inspect service behavior, then lower the partition step by step.

Aside: maxUnavailable controls ready capacity. partition controls eligibility. One setting does not replace the other.

Further reading: https://github.com/volcano-sh/kthena/blob/main/pkg/apis/workload/v1alpha1/model_serving_types.go

Revision answers one critical question

After you change the image from qwen:v1 to qwen:v2, the controller calculates a new ModelServing revision. The revision identifies a set of Role templates.

The controller creates a ControllerRevision object for the new configuration. Generated Pods receive modelserving.volcano.sh/revision. This label lets the controller compare an existing group with the desired configuration.

qwen-3 revision: rev-a
desired revision: rev-b

Result: qwen-3 is outdated.

ModelServing.status records two revision fields during the rollout.

currentRevision

Stable revision still used by older groups.

updateRevision

Desired revision used for replacement groups.

When every required group uses updateRevision and reaches readiness, both fields converge on the same revision.

Further reading: https://github.com/volcano-sh/kthena/blob/main/pkg/model-serving-controller/utils/controller_revision.go

What happens after an image edit

Assume all four groups serve qwen:v1. You change the image to qwen:v2 and keep maxUnavailable: 1.

The controller runs this logic through syncModelServing.

syncServingGroupReplicas
syncRoleReplicas
manageRollingUpdate
syncHeadlessServices
UpdateModelServingStatus

manageRollingUpdate finds outdated groups outside the partition. For ServingGroup rolling update, the controller deletes eligible groups in descending ordinal order. With ordinals 0 through 3 and partition 0, qwen-3 receives replacement before qwen-2.

Deleting a group does not finish the update. Normal reconciliation creates replacement Pods using the desired revision. Readiness determines whether Kthena advances.

Why Kthena counts unavailable replacement groups

The controller separates outdated groups into two categories.

Outdated and not running Prefer removal first, since removal does not reduce ready capacity.

Outdated and running Remove only when the availability budget permits.

The controller also counts new revision groups whose status is not Running. This value appears as newServingGroupUnavailableCount.

Example:

Desired groups: 4
maxUnavailable: 1
New group qwen-3: starting, not ready

Unavailable replacement count: 1
Safe additional healthy group removals: 0

Without this check, Kthena might remove qwen-2 while qwen-3 still loads model weights. Two groups would then lack ready capacity, violating your maxUnavailable: 1 policy.

Readiness controls rollout progress

A replacement group must become ready before Kthena treats capacity as restored. A successful Pod creation is not enough. A container process runs while model loading, worker coordination, or readiness probes still fail.

UpdateModelServingStatus reads ServingGroup state and writes observed progress into ModelServing.status.

status:
  replicas: 4
  availableReplicas: 3
  updatedReplicas: 1
  currentReplicas: 3
  currentRevision: rev-a
  updateRevision: rev-b
  conditions:
    - type: UpdateInProgress
      status: "True"

Three condition types matter during operations.

Available=True

All tracked groups are ready.

Progressing=True

Groups are creating, deleting, or waiting for readiness.

UpdateInProgress=True

A rollout has started and eligible groups still need replacement.

Run these commands while testing a rollout:

kubectl get modelserving qwen -o yaml
kubectl get pods -l modelserving.volcano.sh/name=qwen
kubectl get controllerrevisions -l modelserving.volcano.sh/name=qwen
kubectl get events --sort-by=.lastTimestamp

Look for a growing updatedReplicas count, a stable availability count within your configured budget, and matching currentRevision and updateRevision after completion.

Ordinals are logical slots, not age

An ordinal identifies a ServingGroup slot such as qwen-2. A revision identifies configuration. These two values solve different problems.

Ordinal: which logical ServingGroup is this?
Revision: which template created this ServingGroup?

This distinction matters for partition behavior. A partition is documented as an ordinal boundary, so partition: 2 protects ordinals 0 and 1 even when group creation order differs.

Kthena has an open issue for noncontiguous ordinals, where list position and logical ordinal differ. For example, active groups 0, 2, 3, 4 need ordinal based partition handling. Treat this as an operational edge case when reviewing a rollout on a cluster with recreated groups.

Further reading: https://github.com/volcano-sh/kthena/issues/1483

ServingGroup rolling update versus Role rolling update

Kthena supports two rollout scopes.

ServingGroupRollingUpdate

Replacement unit: Whole ServingGroup.

Availability setting: Top level rollingUpdateConfiguration.maxUnavailable.

RoleRollingUpdate

Replacement unit: Individual outdated Role instance.

Availability setting: Per Role maxUnavailable.

Use ServingGroup rolling update when Roles form one serving unit and you need whole group replacement. This fits tightly coupled distributed model workers.

Use Role rolling update when one Role needs independent replacement. Check recoveryPolicy first. A ServingGroupRecreate recovery policy changes failure handling and removes the whole group after a Role failure.

Practical rollout checklist

Before applying an image change:

1. Confirm Pod readiness probes reflect usable inference capacity.
2. Set maxUnavailable from your real capacity requirement.
3. Start with a partition if you need staged rollout control.
4. Check serving group ordinals and current revision labels.
5. Watch availableReplicas, updatedReplicas, conditions, and events.
6. Keep old image references available until rollout completion.
7. Test the same configuration with a small replica count before production traffic.

Pros:

1. Whole serving groups protect role coordination during replacement.
2. Revisions give the controller durable rollout identity.
3. maxUnavailable gives a clear capacity limit.
4. Partitions support staged deployment.

Cons:

1. Large model loading time slows rollout progress.
2. A readiness probe with weak checks allows traffic before the engine is usable.
3. Whole group replacement costs more capacity than a single role replacement.
4. Gapped ordinals need careful partition handling.

What to remember

Kthena rolls a ServingGroup by replacing outdated groups, waiting for readiness, and respecting the configured availability budget. Revisions identify old and new templates. Partitions choose eligible ordinals. Status fields show progress.

Your rollout safety depends on three choices: a meaningful readiness probe, a realistic maxUnavailable value, and a partition plan suited to your traffic risk.

Further reading:

1. https://github.com/volcano-sh/kthena/blob/main/docs/kthena/docs/developer-guide/model-serving-rolling-update.md
2. https://github.com/volcano-sh/kthena/blob/main/docs/kthena/docs/architecture/model-serving-controller.mdx
3. https://github.com/volcano-sh/kthena/blob/main/pkg/model-serving-controller/controller/model_serving_controller.go