Inside Koordinator: Resource Policy, Scheduling, and Node-Level Enforcement

Kubernetes answers one important question: which node should run this Pod? Production clusters need a larger answer: how can latency-sensitive services, batch jobs, GPU workloads, and resource pressure share the same machines without creating unpredictable performance?

That is where Koordinator fits.

Koordinator is not only a custom scheduler. It is a group of control loops operating at different boundaries:

Pod intent and cluster policy
        ↓
Koordinator APIs, controllers, and webhooks
        ↓
Scheduler plugins choose a node
        ↓
Kubelet creates the Pod on that node
        ↓
Koordlet observes and enforces node-local policy
        ↓
Linux cgroups apply the actual resource controls
        ↓
Runtime proxy can participate in container lifecycle requests

These are not steps in one synchronous request. They are separate components with separate state, timing, and failure modes. That separation is the key to reading the project.

The Problem

Requests and limits are necessary, but they are not a complete resource-management model for a shared cluster.

Imagine one node running:

  • a user-facing API with a strict latency objective;
  • batch work that can tolerate slower progress;
  • a GPU job needing suitable NUMA locality; and
  • operating-system processes that still require reserved capacity.

The scheduler needs a cluster-level placement decision. The node needs local measurement and enforcement. Operators need policy and capacity objects. The runtime may need lifecycle-aware resource hooks. Putting all of that in one scheduler plugin would be the wrong design.

Koordinator splits the problem across four main components.

ComponentOwnsMain jobkoord-managerKubernetes control planeReconciles Koordinator resources and runs webhooks when enabled.koord-schedulerPlacementAdds Koordinator policies to the Kubernetes scheduling framework.koordletIndividual nodeCollects state and metrics, then applies local QoS/resource policy.koord-runtime-proxyRuntime lifecycleOptionally intercepts CRI or Docker lifecycle requests for resource hooks.

The Resource Contracts

The components need shared contracts. Koordinator expresses them through Pod labels and annotations plus custom resources.

QoS labels and Pod resource policy

Koordinator defines QoS classes including LSR, LS, BE, and SYSTEM. A Pod can explicitly carry koordinator.sh/qosClass; otherwise the project can derive a Koordinator QoS class from Kubernetes QoS. See apis/extension/qos.go.

The SLO API also defines Pod-level configuration for CPU burst, CPU QoS, memory QoS, and block-I/O QoS in apis/slo/v1alpha1/pod.go.

These values are intent, not enforcement. A label does not change CPU behavior by itself. Its value is that the scheduler, controllers, and node agent can reason about the same workload classification.

NodeSLO: policy for a node

NodeSLO is a cluster-scoped resource that holds node-level policy. Its spec can include resource-pressure thresholds, QoS policy for different classes, CPU-burst strategy, host-application policy, and system configuration.

The design matters: policy is represented as an API object rather than buried inside a node daemon flag. That makes it observable and reconcilable. The type is defined in apis/slo/v1alpha1/nodeslo_types.go.

NodeMetric: what the node is actually doing

NodeMetric carries observed node usage, aggregated usage, system usage, Pod usage, and reclaimable-resource information. Pod entries include QoS and priority. This is deliberately separate from requests and allocations: a scheduled CPU request is a planning value; measured CPU use is reality.

Source: apis/slo/v1alpha1/nodemetric_types.go.

Reservation, Device, and ElasticQuota

  • Reservation holds capacity for approved future owners. It has its own lifecycle, status, and allocation policy.
  • Device represents GPUs, FPGAs, and RDMA devices, including health, resources, topology, and allocations.
  • ElasticQuota provides tenant or namespace-level min, max, and observed used resource bounds.

Together, these APIs let a scheduling decision consider more than free CPU and memory.

Big word alert — control loop: A control loop repeatedly reads current state, compares it with policy, and makes a correction when needed. Controllers, scheduler plugins, and node agents are all control loops, but they operate at different boundaries.

koord-manager: Maintaining the API-Level View

koord-manager is the control-plane process. It creates a controller-runtime manager, registers indexes, applies controllers, and conditionally installs webhooks. Its entry point is cmd/koord-manager/main.go.

The configured controller set includes node metrics, node resources, NodeSLO, quota profiles, and colocation profiles. See cmd/koord-manager/options/controllers.go.

The manager does not bind Pods and it does not write cgroup files. Its job is to maintain valid API-level policy and derived state so that other components have reliable inputs.

Big word alert — reconciliation: Reconciliation means making actual state converge toward desired state. A controller expects events to be retried, delayed, or superseded; it does not treat one successful request as permanent truth.

koord-scheduler: Making the Placement Decision

Koordinator extends the Kubernetes scheduling framework. Its scheduler entry point registers plugin factories, then starts the scheduler application. The plugin list includes:

loadaware, nodenumaresource, reservation, coscheduling,
deviceshare, elasticquota, defaultprebind, noderesourcefitplus,
scarceresourceavoidance, schedulinghint

See cmd/koord-scheduler/main.go.

This tells us that “best node” is not a single score. It may combine capacity fit, current load, NUMA layout, devices, reservations, and quota policy.

NodeNUMAResource is a useful example. It participates in Kubernetes scheduling stages including pre-filter, filter, pre-score, score, reserve, and pre-bind. It also integrates with Koordinator reservation and topology interfaces. See pkg/scheduler/plugins/nodenumaresource/plugin.go.

ElasticQuota participates in enqueue, pre-filter, post-filter, and reserve stages while maintaining quota state from informers. This makes quota a scheduling concern instead of a post-scheduling report. See pkg/scheduler/plugins/elasticquota/plugin.go.

Big word alert — scheduler extension point: An extension point is a defined point where the scheduler asks a plugin to contribute. Filter answers whether a node is allowed. Score ranks allowed nodes. Reserve records a tentative allocation before binding. This gives policies clear responsibilities instead of one oversized scheduling function.

koordlet: Turning Policy into Node Behavior

The scheduler chooses a node; it cannot continuously observe kernel-level conditions or enforce local policy. Koordlet does that work.

At startup, pkg/koordlet/koordlet.go creates a metric cache, state informer, metric advisor, prediction service, QoS manager, runtime-hook service, and resource-update executor.

Its startup order is important:

  1. Start the resource executor, metric cache, and state informer.
  2. Wait for state synchronization.
  3. Start metric collection and wait for it to synchronize.
  4. Start prediction, QoS management, runtime hooks, and extensions.

The node agent should not enforce policy before the state it depends on has synchronized.

Koordlet also detects the cgroup driver and configures cgroup paths. Its resource updater maps policy to Linux interfaces such as CPU quota/period, CPU shares, CPU burst, cpuset.cpus, memory controls, and block-I/O controls. See pkg/koordlet/resourceexecutor/updater.go.

This is not simply “write a value to a file.” Some cgroup resources are hierarchical: a parent and child setting must remain valid together. The resource executor therefore contains ordering and merge behavior for selected interfaces.

Big word alert — cgroup: A cgroup is a Linux kernel mechanism for grouping processes and applying resource accounting or limits to that group. Container runtimes place container processes into cgroups; Koordlet updates relevant controls so policy has an operating-system-level effect.

koord-runtime-proxy: The Lifecycle Boundary

The runtime proxy is a separately configured component, but it covers a boundary the scheduler does not own. It can handle CRI lifecycle calls such as:

RunPodSandbox, StopPodSandbox, CreateContainer,
StartContainer, StopContainer, UpdateContainerResources

The CRI interception path maps requests to resource types and hook paths, optionally calls a pre-hook, forwards the request to the backend runtime, records successful lifecycle state, and dispatches a post-hook where applicable. See pkg/runtimeproxy/server/cri/criserver.go.

The scheduler sees a Pod before placement. The runtime proxy sees a container lifecycle request. Those are related concerns, but neither replaces the other.

One Resource Journey

Consider a latency-sensitive service sharing nodes with batch workloads.

  1. Declare intent. The Pod provides requests and limits, perhaps a QoS label and Koordinator annotations. Operators configure NodeSLO; platform policy may define quotas, reservations, and device information.
  2. Maintain policy state. koord-manager reconciles relevant API objects and runs admission/webhook behavior when enabled.
  3. Place the Pod. The Kubernetes scheduler invokes Koordinator plugins. They can reject unsuitable nodes, score viable ones, reserve capacity, and persist allocation information before binding.
  4. Create containers. The kubelet on the selected node creates the sandbox and containers. If enabled, the runtime proxy can participate in supported lifecycle requests.
  5. Enforce locally. Koordlet uses synchronized state and measured signals to apply local QoS policy through cgroup resource controls.

The division is deliberate:

Scheduler: “This node is the intended placement.”
Koordlet:  “This is the measured node state and policy to enforce.”
Kernel:    “These process groups receive these actual controls.”

Design Trade-offs

Precision versus complexity. Koordinator can represent topology, QoS, devices, quotas, reservations, and runtime-aware behavior. The cost is more API objects and more operational boundaries.

Planning versus reality. The scheduler acts on a delayed cluster view. Koordlet acts on current node state but cannot redo every placement decision. Both are necessary.

Extensibility versus interaction risk. Scheduler extension points make specialized policy possible, but plugins must compose correctly across filter, score, reserve, and bind stages.

Rich controls versus node variability. Linux cgroup behavior depends on cgroup version, cgroup driver, kernel features, and runtime support. Successful YAML submission is not proof that a node can enforce every requested behavior.

Big word alert — eventual consistency: Eventual consistency means components can temporarily see different versions of state but are designed to converge. Scheduler caches, controllers, metric collection, runtime calls, and cgroup writes are not one atomic transaction. Correctness depends on clear ownership, retries, and safe failure handling.

The main Lesson is:

Resource management in a production Kubernetes platform is not one scheduler feature. It is API contracts, placement decisions, node-local observation, kernel enforcement, and runtime lifecycle handling working together.

That is what Koordinator adds to Kubernetes.

Source Map