Prefill-Decode Disaggregation on Kubernetes: How Kthena Actually Implements It

Prefill-Decode Disaggregation on Kubernetes: How Kthena Actually Implements It

An LLM request looks like one API call. The inference engine sees two different workloads.

Prefill reads the input prompt and builds the KV-cache. Decode reads the KV-cache and generates output tokens one at a time. Both phases use the same model, but they place different pressure on the system.

Kthena separates these phases into Kubernetes roles. The controller creates the workloads. The router selects a compatible prefill and decode pair. A connector coordinates the KV-cache handoff.

This article follows one request from a Kthena custom resource to the final generated token.

The Problem: One Worker Serves Two Workloads

A traditional deployment runs prefill and decode inside the same inference worker.

Client -> Inference worker -> Prefill -> Decode -> Client

This model keeps deployment simple. One worker owns the request from start to finish. The same replica pool handles prompt processing and token generation.

Under load, the two phases compete for shared resources. A long prompt increases prefill work. A long response keeps decode busy for longer. One replica count must absorb both patterns, even when pressure grows on only one side.

This creates three concrete problems:

1. Prefill and decode need different resource profiles.
2. Prompt traffic and generation traffic do not grow at the same rate.
3. Scaling one phase also scales the other phase.

PD disaggregation separates the workers:

Client -> Router -> Prefill worker
                    |
                    | KV-cache transfer
                    v
                Decode worker -> Client

The split adds network traffic, connector configuration, and more failure paths. The design fits workloads where phase-specific capacity matters more than single-worker simplicity.

Prefill, Decode, and the KV-Cache

Prefill processes the input tokens. The model computes attention state for the prompt and stores intermediate data in the KV-cache. Long prompts usually increase prefill work and affect Time To First Token, or TTFT.

Decode generates the response one token at a time. Each new token depends on earlier output. Time Per Output Token, or TPOT, measures the delay between generated tokens. Decode performance depends on memory access, batching, and request scheduling.

Big word alert: KV-cache

The KV-cache stores intermediate attention data from the prompt. Decode reuses this data instead of processing the full prompt again.

The cache creates the connection between the two workers. Prefill produces the cache. Decode consumes the cache. A PD deployment needs a transfer path between both sides.

PD disaggregation does not guarantee better performance for every workload. Measure TTFT, TPOT, throughput, and transfer overhead before choosing the topology.

ModelServing Defines the Two Roles

Kthena uses the ModelServing custom resource to describe the serving workload.

The core structure looks like this:

spec:
  template:
    roles:
      - name: prefill
        replicas: 1
      - name: decode
        replicas: 1

The model-serving controller treats both entries as separate roles. Each role receives a pod template, a replica count, container arguments, resource requests, and readiness checks.

The two roles often load the same model weights. Their engine configuration differs. A vLLM deployment using NIXL typically marks the prefill side as the producer and the decode side as the consumer.

Prefill: kv_connector=NixlConnector, kv_role=kv_producer
Decode:  kv_connector=NixlConnector, kv_role=kv_consumer

Role replica counts stay independent. A deployment might use two prefill replicas and four decode replicas. The correct ratio depends on prompt length, output length, traffic volume, hardware, and the inference engine.

ASIDE: Separate roles do not require separate hardware types. Kthena supports different resource profiles and node placement, but both roles also fit on the same accelerator class.

The Controller Builds the Runtime Topology

The ModelServing controller reconciles the desired resource into Kubernetes workloads. The controller creates the serving groups, roles, pods, and related services. The controller also applies labels used later by the router.

The labels carry two pieces of information:

modelserving.volcano.sh/group-name: group-0
modelserving.volcano.sh/role: prefill

A matching decode pod uses the same group label and a different role label:

modelserving.volcano.sh/group-name: group-0
modelserving.volcano.sh/role: decode

The labels create a logical pair:

group-0:
  prefill-0
  decode-0

Kthena also supports role-level scheduling constraints. A PD serving group needs usable capacity on both sides. Starting decode capacity without matching prefill capacity creates an incomplete request path.

ModelServer Describes Pairing Rules

ModelServing describes how Kthena deploys the workers. ModelServer describes how the router finds and classifies them.

The PD section of a ModelServer looks like this:

spec:
  workloadSelector:
    matchLabels:
      modelserving.volcano.sh/name: vllm-qwen-06b
    pdGroup:
      groupKey: modelserving.volcano.sh/group-name
      prefillLabels:
        modelserving.volcano.sh/role: prefill
      decodeLabels:
        modelserving.volcano.sh/role: decode

The matchLabels field selects the pods belonging to the model server. The pdGroup field adds PD-specific classification.

The router uses three rules:

1. Read the group value from groupKey.
2. Classify prefill pods with prefillLabels.
3. Classify decode pods with decodeLabels.

Pods with the same group value form compatible candidates. A decode pod from group-0 pairs with a prefill pod from group-0. The router does not pair the decode pod with a prefill pod from group-1.

ModelRoute Defines the Client Entry Point

ModelRoute maps the client model name to a ModelServer.

spec:
  modelName: Qwen/Qwen3-0.6B
  rules:
    - name: default
      targetModels:
        - modelServerName: vllm-qwen-06b

Each resource answers a different question:

ModelServing answers: How should Kthena deploy the workers?

ModelServer answers: Which pods serve the model, and how do their roles pair?

ModelRoute answers: Which model should receive this client request?

The client sends a normal OpenAI-compatible request. The client does not need to know about prefill pods, decode pods, or KV-cache connectors.

One Request Through the Router

A client sends a request such as:

{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [
    {
      "role": "user",
      "content": "Explain Kubernetes scheduling"
    }
  ]
}

The router follows this path:

1. Match ModelRoute
2. Resolve ModelServer
3. Load model pods and PD groups
4. Filter unavailable pods
5. Score decode candidates
6. Find prefill pods in matching groups
7. Send the pair to a connector
8. Return the decode response

The current scheduler selects decode candidates first. The scheduler keeps up to five top-scoring decode pods. For each decode pod, the scheduler finds prefill pods in the same PD group and scores those candidates.

This order gives the router a complete pair before the request enters the connector layer. The scheduler does not select two unrelated pods and hope for compatible cache transfer.

KV-Cache Transfer

After prefill processes the prompt, decode needs access to the resulting KV state.

The router selects a connector from ModelServer configuration. Kthena includes connector paths for NIXL, MoonCake, HTTP, and SGLang-specific handling.

The router passes the connector:

1. The request body.
2. The prefill address.
3. The decode address.
4. In-flight request hooks.

The connector performs the engine-specific request flow. The exact protocol differs across connector types. NIXL, MoonCake, HTTP, and SGLang do not share identical cache transfer behavior.

The engine configuration must match the connector. A router setting alone does not create a working cache transfer path. The inference image, engine arguments, runtime settings, network interfaces, and connector type all belong to the same deployment contract.

This separation gives Kthena a stable routing interface while keeping engine-specific transfer logic inside connector implementations.

Failure and Retry Handling

A PD request depends on both workers. One unhealthy prefill pod or decode pod breaks a pair.

The router tries another available pair when a connector operation fails. The router stops after all candidate pairs fail and returns an error to the client.

The retry behavior limits the impact of one unhealthy pair. The retry behavior also creates operational requirements:

1. Pod readiness data must stay fresh.
2. Role labels must match ModelServer selectors.
3. Group labels must identify compatible pairs.
4. Connector configuration must match the inference engine.

Decode failure also needs cleanup. A prefill operation should not continue consuming resources after decode has failed. Connector implementations handle this lifecycle according to their transfer protocol.

Independent Scaling

Prefill and decode traffic often grow at different rates.

Long prompts increase prefill pressure. Long responses increase decode pressure. A single replica count does not represent both workloads equally.

Kthena supports role-level autoscaling through AutoscalingPolicy:

disaggregatedTarget:
  roles:
    prefill:
      minReplicas: 1
      maxReplicas: 8
    decode:
      minReplicas: 1
      maxReplicas: 16

A ratio constraint keeps the topology within defined bounds:

ratioConstraint:
  numeratorRole: prefill
  denominatorRole: decode
  minRatio: "0.25"
  maxRatio: "1"

The ratio protects the request path from an extreme imbalance. A large decode fleet with too few prefill workers still leaves prompt processing as the bottleneck.

Verifying a Deployment

Use the matching GPU PD example and guide. Apply ModelServing first, wait for both roles, then apply ModelServer and ModelRoute.

kubectl apply -f examples/model-serving/gpu-pd-disaggregation.yaml

Check both roles:

kubectl get pods \\
  -l modelserving.volcano.sh/name=vllm-qwen-06b \\
  -o wide

Check ModelServing status:

kubectl get modelserving vllm-qwen-06b \\
  -o jsonpath='{.status.conditions}' | jq

Send a request through the router after applying the matching network resources:

curl http://$ROUTER_IP:80/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Explain PD disaggregation"}]}'

Inspect router logs for the selected decode pod, selected prefill pod, connector errors, retry attempts, and request completion.

Measure prefill duration, decode duration, TTFT, TPOT, and request failures. Compare those values with a single-worker deployment under the same workload.

Trade-offs

PD disaggregation gives you:

1. Separate prefill and decode scaling.
2. Role-specific resource profiles.
3. PD-aware pod pairing.
4. Better visibility into TTFT and TPOT.
5. Connector support for several inference environments.

PD disaggregation also introduces:

1. More Kubernetes resources.
2. More network traffic.
3. KV-cache transfer dependencies.
4. More failure paths.
5. More configuration points.
6. More complex debugging.

ASIDE: PD disaggregation fits workloads with clear phase-specific pressure. A single inference worker fits smaller workloads with lower operational overhead.

The Implementation in One View

ModelServing defines the prefill and decode roles.

The controller creates and labels the pods.

ModelServer defines the PD group and connector details.

ModelRoute maps the client model name to the ModelServer.

The scheduler selects a decode pod and a compatible prefill pod.

The connector coordinates KV-cache transfer.

The decode worker generates the response.

Kthena turns one client request into a coordinated Kubernetes workflow. The client sees one API request. Kthena manages the split between prompt processing and token generation behind the API.