Model Router
The ModelRouter CRD exposes a single OpenAI-compatible HTTP endpoint that dispatches requests across multiple backends:
- Local
InferenceServiceinstances managed by LLMKube - External providers (Anthropic, OpenAI) called directly
- A LiteLLM proxy that aggregates many providers behind one URL
It is the cross-engine handoff layer for agentic chains: an agent running against a local model can transparently call out to a cloud model for specific steps, governed by declarative policy that enforces data classification, cost, and capability constraints.
Status
ModelRouter is alpha as of LLMKube 0.8. The CRD lives in inference.llmkube.dev/v1alpha1. The wire shape may evolve before v1beta1.
Why this exists
LLMs are increasingly composed: an agent does some work on a fast local model, hands off a hard step to a frontier cloud model, then comes back. Every team building this hits the same three problems:
- The agent code wants one endpoint, not a dispatch tree. Agent runtimes (LangGraph, CrewAI, OpenAI Agents SDK, Anthropic Agent SDK) all consume a single OpenAI-compatible URL.
- Routing policy belongs in the platform, not the agent. Compliance teams need to know that PII can’t egress; finance teams need cost caps; SRE teams need fallback on local outage. None of that should live in Python at the agent level.
- The choice between local and cloud changes. Today’s local model handles 80% of requests; tomorrow’s handles 95%. The agent shouldn’t have to change.
ModelRouter solves all three by sitting in the data path as a small managed HTTP proxy with declarative routing rules.
Architecture
+-------------+
| Agent / App |
+------+------+
| OpenAI-compatible API
v
+----------------------------------------------------+
| router-proxy Deployment (controller-managed) |
| - reads compiled config from a mounted ConfigMap |
| - matches each request against ordered rules |
| - enforces the fail-closed gate |
| - streams responses (SSE / chunked) with no buffer|
| - emits one audit-log line per request |
+-----+----------------------+-----------------------+
| |
v v
+-----------+ +-------------------------+
| local | | external provider |
| Inference | | (Anthropic / OpenAI / |
| Service | | LiteLLM passthrough) |
+-----------+ +-------------------------+ The controller compiles ModelRouter.spec into a JSON config, writes it to a ConfigMap, and reconciles a Deployment plus Service that mounts the ConfigMap and runs the router-proxy binary. The ConfigMap content is hashed and the hash lands on the pod template annotation, so any spec change triggers a clean rollout.
Since every spec change re-rolls the pods, set spec.proxy.revisionHistoryLimit to cap how many old ReplicaSets are kept (0 keeps none); unset keeps 3. InferenceService has the same knob at spec.revisionHistoryLimit.
Draining in-flight streams on a rollout
A proxy replica keeps serving the requests it already holds after SIGTERM, for its --shutdown-timeout: 30s by default. A streamed completion can run far longer, so a roll cuts any stream still open at that deadline. spec.proxy.drainTimeout raises it:
spec:
proxy:
replicas: 2
drainTimeout: 600s It sets the proxy’s --shutdown-timeout and the pod’s terminationGracePeriodSeconds, which is the drain plus a fixed 5s margin so the kubelet does not SIGKILL before the drain ends. The margin covers process exit only: this pod has no preStop hook, and the proxy bounds its own shutdown by the drain. Values run from 1s to 1h; a fractional value rounds the grace period up to the next whole second.
A replica that drains stays Terminating until its drain ends, so a roll waits up to drainTimeout per replica before that pod goes. replicas: 2 keeps new requests flowing on the other replica while one drains, but it does nothing for the requests already streaming on the replica that rolls, which is what this field bounds. A stream longer than drainTimeout still ends at the deadline.
Unset keeps the 30s default and renders nothing on the pod template, so upgrading the operator does not roll existing proxies. Setting or changing it does roll them once.
Exposing the proxy and sandboxing its pod
These spec.endpoint and spec.proxy options apply to dataPlane: Proxy, the default.
spec.endpoint shapes the data-plane Service in front of the proxy. spec.endpoint.port sets the Service port (default 8080); the proxy itself always listens on 8080, and the Service forwards to it. So port: 80 gives clients http://<router>-router-proxy.<namespace>.svc.cluster.local/v1 with no port in the URL, while the proxy still runs unprivileged. type: NodePort or LoadBalancer exposes it beyond the cluster. spec.endpoint.nodePort pins the node port and applies only to type: NodePort; it is ignored for LoadBalancer, as on InferenceService. When nodePort is unset, the port Kubernetes allocated stays the same across reconciles. Removing a previously pinned nodePort also keeps that port; to get a new one, change type or recreate the Service.
spec.proxy.serviceName names that Service, for a short in-cluster URL. With the router below in namespace llmkube, clients use http://llmkube.llmkube.svc.cluster.local/v1:
spec:
endpoint:
port: 80
path: /v1
proxy:
serviceName: llmkube Unset keeps <router>-router-proxy. The name must be a DNS-1035 label (lowercase letters, digits, and -, starting with a letter, at most 63 characters) and must not be the router’s admin Service, <router>-router-proxy-admin. Only the data-plane Service is renamed: the Deployment, ConfigMap, admin Service, pod labels, and selectors keep their generated names, so setting or changing it does not roll the proxy pods. status.endpoint follows the Service.
The operator never takes over a Service it did not create. If another object already holds the name, including a Service another ModelRouter serves on, the router reports RouterProxyNameTaken (below) and keeps serving through its previous Service, with status.endpoint unchanged. When you change the name, or remove it, the operator creates the new Service first and then deletes the previous one, only if this router created it. A Service you made yourself that selects the proxy pods is left alone. Clients of the old name must move to the new one. A NodePort the cluster allocated and a LoadBalancer address are reassigned with the new Service; a pinned spec.endpoint.nodePort is kept, but the node port stops answering between the old Service’s deletion and the new one taking the port, both in the same reconcile.
spec.endpoint.externalTrafficPolicy (Cluster or Local, only with type NodePort or LoadBalancer) is applied to that Service. Local preserves the client source IP, but traffic only reaches nodes that run a proxy replica, so pair it with spec.proxy.nodeSelector, or with one replica per node (below). spec.endpoint.apiKeySecretRef is not supported on a ModelRouter.
spec.proxy.runtimeClassName runs the proxy pod under a RuntimeClass, for example a sandboxed runtime such as gVisor or Kata. Unset leaves the pod spec unchanged.
spec:
endpoint:
type: NodePort
nodePort: 30801
externalTrafficPolicy: Local
proxy:
runtimeClassName: gvisor
nodeSelector:
kubernetes.io/hostname: edge-1 Labels and annotations on the proxy pods
spec.proxy.podLabels and spec.proxy.podAnnotations are merged into the proxy pod template, as spec.podLabels and spec.podAnnotations are on an InferenceService. Use them to match label-based NetworkPolicies. A proxy that needs the API server (for ModelPool activation or resolution: endpoint) may need such a policy to reach it.
spec:
proxy:
podLabels:
netpol.example.com/apiserver: allow
podAnnotations:
example.com/team: llm The operator’s own keys win on collision: the app and inference.llmkube.dev/model-router labels, the config-hash annotation, and the prometheus.io annotations when the operator emits them. The Deployment and Service selectors use only the operator’s labels, so changing these fields rolls the pods but never changes a selector. A key you remove from the spec is removed from the pods. The operator records the keys it applied in the inference.llmkube.dev/applied-pod-labels and inference.llmkube.dev/applied-pod-annotations annotations on the Deployment. Keys other tools put on the pod template, such as kubectl rollout restart’s, are left alone.
One proxy replica per node
spec.proxy.affinity and spec.proxy.topologySpreadConstraints are passed through to the proxy pod, as on InferenceService. The proxy pods carry two stable labels to select on: app: <router>-router-proxy and inference.llmkube.dev/model-router: <router>.
For a highly available edge on two GPU nodes, run one replica per node and send each node’s traffic to its own replica with externalTrafficPolicy: Local. An upstream that hashes the session header across the nodes’ addresses then keeps every session on one replica, and its session bindings stay on that replica:
metadata:
name: all-models
spec:
endpoint:
type: NodePort
nodePort: 30802
externalTrafficPolicy: Local
proxy:
replicas: 2
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: all-models-router-proxy With replicas above 1 and a hard placement rule (a constraint with whenUnsatisfiable: DoNotSchedule, or required pod anti-affinity), the proxy Deployment rolls out with maxSurge: 0 and maxUnavailable: 1: an extra replica could never be placed, so the default surge would hang the rollout. The proxy rolls on every LLMKube upgrade, and while one replica restarts, its node’s NodePort has no local endpoint, so the upstream should fail over to the other node’s address. Soft rules (ScheduleAnyway, preferred affinity) and single-replica routers keep the default rollout. A router with ModelPools always runs one replica.
Readiness that requires mesh listeners
By default the proxy’s readiness and liveness probes both call GET /health, which answers 200 while the process runs. In a service mesh that captures traffic inside the pod, a replica can be running while the mesh has not set it up, so requests sent to it fail. spec.proxy.readinessRequiredListeners lists TCP ports that must be listening in the pod before the replica is Ready. For Istio ambient mode, that is the HBONE inbound port, 15008:
spec:
proxy:
replicas: 2
readinessRequiredListeners: [15008] When the list is set, the readiness probe moves to GET /readyz (same port and timings), and the proxy gets one --readiness-required-listener flag per port. /readyz reads /proc/net/tcp and /proc/net/tcp6, which list the sockets in the pod’s network namespace, including ones the mesh opened. A port counts only when one of its entries is in the LISTEN state. While a port has no listener, /readyz answers 503 with the missing ports, for example {"status":"not ready","missingListeners":[15008]}, and the replica leaves the Service endpoints until the listener appears. Liveness stays on /health, so a replica the mesh hasn’t set up is taken out of rotation but not restarted.
Don’t combine the field with a spec.proxy.runtimeClassName whose runtime has its own network stack, such as gVisor: /proc/net/tcp* is then the sandbox’s table, so the mesh’s listener never appears and every replica stays unready.
If /proc/net/tcp or /proc/net/tcp6 can’t be read, or its header isn’t in the expected format, /readyz fails open: any port it didn’t find in a table it could read is treated as present, so the replica stays Ready. The proxy logs cannot read TCP socket table and required listeners not enforced; reporting ready once each, and logs again when the table can be read. A missing /proc/net/tcp6, as on a kernel booted with IPv6 disabled, counts as an empty table, not a read failure. A /proc problem is not a reason to take every replica out of service.
Fail-open covers unreadable tables only. A port that never listens in the pod’s network namespace (a wrong port, a mesh that hasn’t set the pod up, a namespace that isn’t enrolled) is not fail-open: every replica with the field reports unready at once, and a rollout that sets the field stalls on its first new pod.
The list takes up to 16 ports from 1 to 65535, with no duplicates. Unset or empty keeps readiness on /health and renders nothing new on the pod template, so upgrading the operator does not roll existing proxies. Setting or clearing it rolls them once. If spec.proxy.image pins a router-proxy image, it must be a release that has the --readiness-required-listener flag; an older image exits on the unknown flag.
/readyz is served on every proxy, on the same port as the inference API. Like /health, it needs no authentication and is reachable from any pod that can reach that port. It reports only whether the configured ports are listening. The proxy rescans /proc at most about once a second; callers in between, including concurrent ones, get the last answer.
The fail-closed gate
The headline differentiator. A rule that matches sensitive classifications (default: pii, phi) and is marked failClosed: true has two guarantees:
- Cannot reference cloud-tier backends. The controller rejects the manifest at
kubectl applyif it tries to. This is the static half of the gate. - Refuses rather than falls through. At request time, if every backend in the route is unhealthy, the proxy returns HTTP 503 and emits an audit-log denial. It does not fall through to other rules or to
defaultRoute. Sensitive data never leaves the cluster, even on outage.
This is the property regulated industries (healthcare, finance, defense, manufacturing) need to adopt local LLM inference at all.
Where the classification comes from
The gate acts on the classification a request carries, and the router reads it from a request header: policy.classification.headerKey, x-llmkube-classification by default. The router never classifies the request body. A request that holds regulated data but carries no header is unclassified: it matches no dataClassification rule and goes wherever an unclassified request goes, defaultRoute included. Have the caller, or a gateway in front of the router, set the header on every request that carries sensitive data.
policy.classification.mode accepts header-only (the default), detector and hybrid, but only header-only is implemented. detector and hybrid are reserved for an in-proxy content classifier (#441) and are refused until it ships, so a router never appears to inspect content it does not:
- On
dataPlane: Proxy, the webhook rejects either mode atkubectl apply. A router that already sets one, such as one created before this check, reportsValidated=Falsewith reasonClassifierNotImplemented, and the operator leaves its router-proxy as it was: a new router gets no proxy, and an existing one keeps serving the last config that compiled. The router-proxy itself refuses to start on a config that names either mode. - On
dataPlane: GatewayanddataPlane: AgentGateway, adataClassificationmatch under either mode is refused.
Minimal example
apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelRouter
metadata:
name: coding-router
spec:
backends:
- name: local-qwen
tier: local
inferenceServiceRef:
name: qwen3-coder
- name: cloud-opus
tier: cloud
external:
provider: anthropic
model: claude-opus-4-7
url: https://api.anthropic.com
credentialsSecretRef:
name: anthropic-key
rules:
- name: pii-stays-local
match:
dataClassification: ["pii", "phi"]
route:
backends: ["local-qwen"]
failClosed: true
- name: complex-to-cloud
match:
taskComplexity: complex
route:
backends: ["cloud-opus", "local-qwen"]
strategy: primary-fallback
defaultRoute: local-qwen After kubectl apply:
kubectl describe modelrouter coding-routershows the status conditions and per-backend health.kubectl get configmap coding-router-router-proxycontains the compiled JSON config.- The endpoint is
http://coding-router-router-proxy.<namespace>.svc.cluster.local:8080/v1/chat/completions.
Point any OpenAI-compatible client at that URL and it just works. The proxy serves the OpenAI-compatible routes, and the Anthropic Messages API, at their standard paths, and every POST route is matched on the request’s model field against the same rule set:
| Route | Serves |
|---|---|
POST /v1/chat/completions | Chat completions |
POST /v1/completions | Legacy completions: what fill-in-the-middle / IDE autocomplete clients and benchmark tools such as vllm bench serve --backend openai send |
POST /v1/embeddings | Embeddings |
POST /v1/rerank | Reranking |
POST /v1/messages | Anthropic Messages API: Claude Code and the Anthropic SDKs |
POST /v1/messages/count_tokens | Anthropic token counting |
GET /v1/models | Model discovery: one entry per backend, under its displayName, else its name, the id BackendNameMatch resolves. An external backend’s upstream model is not listed. |
The two Anthropic routes are placed, balanced and session-bound exactly like chat completions, and headers such as anthropic-version and anthropic-beta pass through. The backend must serve them itself: vLLM does natively; a backend that does not returns its own 404, which reaches the client unchanged and does not count against the backend. Token budgets charge Anthropic responses from their own usage fields (cached input included), and errors the router itself returns on these routes use the Anthropic error shape.
GET /v1/models lists each backend under its displayName, else its name. Coding clients, and services that generate client config, read this list to decide whether a model takes images and how much context it holds. A backend publishes both when it declares modelInfo:
backends:
- name: glm-5-3-flash
inferenceServiceRef: {name: glm-5-3-flash}
capabilities: [vision]
modelInfo: {} That backend’s entry then carries "context_length" and an OpenRouter-style architecture block, {"modality": "text+image->text", "input_modalities": ["text", "image"], "output_modalities": ["text"]}. Fields you leave unset are filled in:
| Field | When unset |
|---|---|
inputModalities | [text, image] if the backend’s capabilities include vision, else [text] |
outputModalities | [text] |
contextLength | vLLM: vllmConfig.maxModelLen, or --max-model-len in extraArgs. llama.cpp: contextSize, divided across parallelSlots when they are set, because llama-server then splits the context between slots. An unset runtime counts as llama.cpp except on a Metal Model, where the metal-agent picks the runtime. Other runtimes, and llama.cpp extraArgs that set --ctx-size, --parallel or --kv-unified, publish no context length. |
A value you set wins. If inputModalities leaves out image while capabilities lists vision, the controller records a ModelInfoMismatch warning event, and /v1/models publishes inputModalities. A backend without modelInfo publishes the bare entry, as before.
The router doesn’t enforce modelInfo, so keep it in step with what the backend serves: a vLLM backend started with --limit-mm-per-prompt '{"image":0}' rejects images whatever the list says. modelInfo applies to the Proxy data plane only. The Gateway and AgentGateway data planes ignore it, and AgentGateway answers /v1/models itself.
Headers that change routing:
| Header | Effect |
|---|---|
x-llmkube-classification: pii | Matches pii-stays-local rule; local-only, fail-closed. |
x-llmkube-task-complexity: complex | Matches complex-to-cloud rule; tries Opus first, falls back to Qwen on 5xx. |
| (no headers) | Falls through to defaultRoute: local-qwen. |
Rule strategies
route.strategy decides how a matched rule uses the backends it lists:
primary-fallback(default) triesroute.backendsin order. The first backend serves; the next one is tried only when it fails.weightedsplits the rule’s traffic across its backends in proportion to each backend’sweight(default1). Use it to keep two kinds of hardware warm behind one model name.shadowserves every request exactly asprimary-fallbackdoes and also sends a copy to the rule’s other backends, whose answers are thrown away. Use it to try a candidate model or new hardware on real traffic without clients seeing it. Proxy data plane only:dataPlane: GatewayanddataPlane: AgentGatewayreject it.
Whatever the strategy, a rule with match.requiredCapabilities sends its requests only to the backends that advertise every listed capability. The rule matches when at least one of its backends does; the others are never tried for it, not even as a fallback. A rule without requiredCapabilities uses all of its backends.
Whatever the strategy, a rule with match.requiredCapabilities sends its requests only to the backends that advertise every listed capability. The rule matches when at least one of its backends does; the others are never tried for it, not even as a fallback. A rule without requiredCapabilities uses all of its backends.
spec:
backends:
- name: mac-m5
inferenceServiceRef:
name: qwen38-metal
weight: 1
- name: nvidia-gb10
inferenceServiceRef:
name: qwen38-cuda
weight: 3
rules:
- name: qwen38
match:
models: ["qwen38-27b"]
route:
strategy: weighted
backends: ["mac-m5", "nvidia-gb10"] On the Proxy data plane, a weighted rule picks a backend for each request at random, weighted by weight, among the backends that can serve it right now. A backend that is quarantined after failures, that lacks a capability the rule’s requiredCapabilities lists, or (under poolActivation: IfIdle) whose ModelPool incumbent is busy is skipped, and the pick falls on the others in proportion to their weights. If the picked backend fails the way a primary fails over under primary-fallback (a 5xx, a connection error, a header timeout, a ModelPool activation that does not complete), the request moves to another backend of the rule, picked the same way, and never back to one it already tried. A weighted rule is therefore as available as a primary-fallback rule over the same backends. The audit line’s backend field and the llmkube_router_requests_total backend label show which backend served each request, and budgets charge the backend that served.
weight: 0 drains a backend from weighted rules: it gets none of their traffic, even while every other backend in the rule is down. This is the Gateway API meaning of a zero weight, which dataPlane: Gateway also follows (Envoy Gateway drops a zero-weight backendRef), so the same spec splits the same way on both data planes. To keep a backend as a standby for when the others fail, list it in a primary-fallback rule instead. A weighted rule needs at least one backend with a positive weight that has every required capability; the controller rejects one that has none.
A weighted rule on the Proxy data plane cannot list two different members of one ModelPool. The members share one GPU slot, so splitting traffic between them would drain one and cold-load the other on a large share of requests. The controller reports this as Available=False with reason CompileFailed and keeps serving the previous config. Splitting across a pooled backend and a backend outside the pool is fine: picking the pooled one activates it as primary-fallback would. dataPlane: AgentGateway compiles only primary-fallback.
Shadow traffic
rules:
- name: chat
match:
models: ["chat"]
route:
strategy: shadow
backends: ["primary", "candidate"] The client is served by the first backend that can serve the request, with the same fallback as primary-fallback: if primary fails, candidate answers the client instead. Once a backend has answered, the router-proxy sends a copy of the request to each backend listed after it that is healthy and has the rule’s required capabilities. A backend that already had the request (it failed before the fallback served) or was skipped as quarantined gets no copy. A shadow rule needs at least two backends.
- The client never waits for a copy. Copies start after the serving backend has answered, run in the background, and their responses (including streams, which are read to the end) are discarded. A copy that fails or hangs does not change the client’s response.
- Copies carry
x-llmkube-shadow: true, so the shadow backend can tell them from client traffic. They are otherwise the request the serving backend got: the same body (a streaming request stays streaming), the forwarded headers, the shadow backend’s own credentials and model name. - Copies change nothing shared. A failed copy never quarantines the backend (other rules may be serving from it), copies are not charged to budgets, and a ModelPool member cannot be a shadow target: a copy would swap it into the pool’s GPU slot. The controller rejects a shadow rule that lists a pooled backend after the first, with reason
CompileFailed. - Copies are best effort. At most 4 copies run at once per shadow backend; a copy that finds that many in flight is dropped. Each copy waits for headers as long as a normal request to that backend would, and the stream idle timeout ends one whose body goes silent. On shutdown the router-proxy cancels the copies still running.
- An external backend can be a shadow target, and then receives every request the rule serves, which its provider bills like any other.
Each copy is counted in llmkube_router_shadow_requests_total{router,rule,backend,outcome}, where outcome is ok (a status below 400, body read to the end), error or dropped, and timed in llmkube_router_shadow_request_duration_seconds{router,rule,backend}. The router-proxy logs one router.shadow line per copy, with the same fields as router.dispatch (model, rule, backend, status, outcome, latencyMs). The client’s request is logged by router.dispatch as usual.
Default-route strategy
defaultRouteStrategy controls what happens to a request that matches no rule, before the defaultRoute fallback applies:
Static(default) routes every unmatched request todefaultRoute.BackendNameMatchfirst checks whether the request’smodelfield exactly equals a backend’sdisplayName, else itsname; if so, it routes straight to that backend. Only when no backend matches does it fall back todefaultRoute. This lets clients address a backend directly ({"model": "cloud-opus"}) without writing a pass-through rule for each one.
The lookup is against backend name — the identifier you publish to clients — never the upstream provider model, so the proxied API surface stays on its own side of the boundary. Explicit rules always win: name matching only fires when no rule accepted the request, and it never bypasses a failClosed rule.
BackendNameMatch widens the trust boundary deliberately: every backend, including cloud-tier ones, becomes directly addressable by any client that names it. That is the point — it lets a single flag replace a per-backend passthrough rule for each model you want clients to reach by name. It does not weaken the fail-closed gate, because failClosed rules are ordinary rules and are evaluated first: a sensitive request that matches such a rule never reaches the name-match step. The same caveat as defaultRoute applies, though — a sensitive request that matches no rule will fall through to name matching (and then defaultRoute) exactly as it would fall through to defaultRoute under Static. Per-rule fail-closed protection is not a global PII-egress invariant; if you route sensitive classifications, gate them with an explicit failClosed rule.
A request whose model names a backend that is currently unhealthy fails fast (HTTP 502/503) rather than silently rerouting to defaultRoute — naming a backend is a request for that backend, and answering from a different model the client did not ask for would be worse than an honest error. This matches how an explicit single-backend rule behaves on backend outage.
Per-request balancing across replicas
A local backend resolves to its InferenceService’s ClusterIP Service by default. With several replicas that has a sharp edge: a keep-alive client pins to one pod through the Service’s conntrack entry, so one hot client saturates a single replica while the others sit idle. Embeddings and rerank workloads feel this first — they are typically called in tight loops over one connection.
spec.backends[].resolution: endpoint changes how the upstream address is compiled. Instead of the Service DNS name, the router-proxy addresses each ready pod directly (http://<pod-ip>:<port>, with the port taken from the referenced InferenceService’s spec.endpoint.port, default 8080), tracked out of the EndpointSlices behind the backend’s Service. The router-proxy then rotates each request across those addresses, skips pods quarantined by the per-pod circuit breaker, and considers the backend healthy as long as at least one pod is dispatchable. A pod is quarantined after two failures in a row (a 5xx or a network error), or at once when a stream goes silent past streamIdleTimeout; each quarantine is logged at Warn. On a LoadAware backend that reads engine metrics, five load polls in a row that cannot connect also quarantine a pod (see How load is read). A request that fails on a pooled keep-alive connection the pod had already closed, before any response byte, is retried once on a fresh connection to the same pod and does not count as a failure (llmkube_router_dispatch_retries_total{reason="stale_connection"}). The proxy closes idle upstream connections after 4 seconds, before vLLM’s 5-second keep-alive timeout. service remains the default, and external backends ignore the field.
The router-proxy tracks those pods itself: it watches the EndpointSlices behind each endpoint-resolution backend and updates its list as pods come and go, so a rollout or a scale change never restarts the proxy. The proxy runs under its own ServiceAccount with read access to EndpointSlices in the router’s namespace. When the controller sees no ready pods (a rollout in progress, or a backend with no EndpointSlices at all, such as a metal-agent member), the backend’s status reports no ready endpoints; dispatch falls back to the Service DNS name. Whether the proxy’s own watch has synced is a separate signal, shown only by llmkube_router_endpoint_source_ready, which is 0 until the watch syncs (for example when the proxy lacks RBAC to read EndpointSlices). Until then dispatch uses the Service address.
To run the proxy next to tainted GPU nodes, set spec.proxy.nodeSelector and spec.proxy.tolerations.
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
name: bge-m3
spec:
modelRef: bge-m3
mode: embedding
replicas: 3
endpoint:
port: 8080
---
apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelRouter
metadata:
name: embedding-router
spec:
backends:
- name: bge-embed
inferenceServiceRef:
name: bge-m3
resolution: endpoint
defaultRoute: bge-embed Use it on multi-replica local backends where one client should not pin one pod. A single replica gains nothing — there is only one endpoint to rotate to — and out-of-cluster backends (the metal-agent path, external: providers) have no EndpointSlices to compile, so they stay service-level.
Load-aware balancing
Round-robin spreads requests evenly, but agent sessions carry very different amounts of work: one copy can draw the long, uncached prompts and queue while others idle. balancing places each request by load instead, and keeps a conversation on its pod unless that pod is clearly the wrong place for it.
spec:
backends:
- name: qwen
inferenceServiceRef:
name: qwen3-8-27b
resolution: endpoint
balancing:
mode: LoadAware
sessionAffinity:
headers: [x-session-affinity, x-openwebui-chat-id, x-opencode-session, x-session-id]
idleTimeout: 30m sessionAffinity requires mode: LoadAware. The webhook rejects it with mode: RoundRobin when a router is created or its spec changes.
How load is read
- Load for a vLLM backend is the pod’s own running plus waiting requests, read from its
/metricsonce a second, plus the requests this proxy sent since. Other runtimes, and any pod whose metrics are more than 5 seconds old, use this proxy’s in-flight count. - A pod the proxy can’t reach is one whose last
/metricspoll got no HTTP response: the connection was refused, reset or timed out. Its in-flight count would read as idle, so it ranks below every reachable pod and is never a spill target; a session already on it stays there. After 5 such polls in a row (about 5 seconds, when its last sample would read stale anyway) the pod is quarantined, and its sessions move with reasonquarantined. The quarantine has no timer and no half-open probe: only a poll that connects ends it, so no client request is spent probing the pod. A backend whose every pod is quarantined this way answers502 all_backends_failedwithout contacting any of them. A poll that gets any HTTP response, even an error, counts as reached. The proxy logs the quarantine and the recovery once each, at Warn. A request quarantine is separate, and a successful poll does not clear it, since/metricsanswering doesn’t show that inference works. - Prefill tokens are this proxy’s estimate of the uncached prompt tokens it has sent to a pod that the pod has not started answering. A request’s estimate is its body size divided by 4, capped at 262,144. A known session’s request counts only how much its body grew since the session’s last request; a new or moved session counts its whole body. The tokens are released when the pod sends its first response byte, which for a streaming engine means prefill is done, or when the request fails or the client leaves.
- When this page says least-loaded, it means the lowest load counted in requests plus prefill tokens, with 16,384 tokens weighing as much as one request. So a new session, a request without a session header, a moved session and a spill target avoid a pod that is in the middle of a long cold prefill. Whether a known session moves is still decided by the pressure rules below, which use load in requests; prefill tokens only choose the target those rules are measured against. So a KV-full pod can occasionally shed a session it would otherwise have kept, always toward a pod with KV room.
- Like the in-flight count, prefill tokens are counted per proxy replica.
POST /v1/messages/count_tokenscarries the whole conversation but runs no prefill, so it counts no prefill tokens, is placed least-loaded without its session header, leaves the session’s binding and size record unchanged, and is not counted inllmkube_router_placements_total.- A pod whose KV cache is 90% full or more is KV-full. New sessions avoid KV-full pods while another pod has room.
Where a session goes
Without a session header, each request goes to the least-loaded pod. With one, the first configured header present names the session. Only a hash of its value is kept.
- A new session goes to the least-loaded pod and is remembered there.
- A known session stays on its pod unless one of three things moves it. With fresh engine metrics, it moves if its pod has requests waiting and carries at least 4 more requests than the least-loaded pod. Without fresh metrics, it moves on an in-flight gap of 8 or more alone, with no waiting requests needed. It also moves if its pod is KV-full and queuing and the target has a fresh sample showing KV room, as long as the target is no busier than the full pod; a KV-full pod sheds only to a pod no busier than itself, so a herd cannot pile onto one target. If the target has no such sample, the gap of 4 still applies. A pod whose fresh sample is KV-full is never a spill target. When every pod is KV-full, a session whose pod left can still land on a full pod.
- On a backend with fewer than 3 dispatchable pods, only a KV-full pod sheds sessions. A queue and a load gap alone never move a session there, because with one or two pods every move lands on a cold prefix cache. Moving off a pod that left the candidate set is unchanged.
- A session that moved stays on its new pod. It cannot move again for pressure for 60 seconds, and after that it can.
- A session whose pod left (not ready, or quarantined by the circuit breaker) moves at once.
- KV-full alone does not move a session. Only new sessions avoid KV-full pods.
- If the table is full (100,000 sessions per backend), a new session falls back to a consistent-hash home pod and is not remembered.
idleTimeout drops a binding that has gone unused that long. The default is 30m, and the allowed range is 1m to 24h. Common agent headers are x-session-affinity, x-openwebui-chat-id, x-opencode-session and x-session-id.
Prefix affinity
Session headers help only clients that send one. prefixAffinity places requests by what each pod already holds in its prefix cache, with or without a header: a follow-up turn returns to the pod that served the conversation, and clients that share a long system prompt or tool list share the pod that holds it.
balancing:
mode: LoadAware
prefixAffinity: {} # blockBytes: 1024, maxBlocksPerEndpoint: 16384 - What the proxy hashes. The request’s tool definitions, then its system prompt (Anthropic Messages), then each message in order, or a completion’s prompt. The model name and sampling fields are left out, and whitespace differences between clients don’t matter. The stream is cut into blocks of
blockBytes(default 1024, about 256 tokens), and each block is hashed together with the blocks before it, so a block matches only as part of a prefix. Only full blocks count. - What it remembers. For each pod, the blocks it sent there, up to
maxBlocksPerEndpoint(default 16384, about 4M tokens of prompt), dropping the least recently sent first. It never asks the engine, so it works the same for every runtime. Only block hashes are kept, never prompt text. - How it places. A pod is credited with the request’s leading blocks it holds, as estimated tokens taken off its least-loaded score. A pod holding a long shared prefix wins at equal load. One queued request weighs 16,384 tokens, so a busy pod still loses to an idle one unless it holds more prompt than that.
- With no match anywhere, equally loaded pods are tried starting from the prefix’s home on the consistent-hash ring, so every proxy replica sends the same new prefix to the same pod.
- Sessions keep their binding and pressure rules: a session under pressure still moves to the least-loaded pod, whatever another pod holds. Prefix affinity chooses a new session’s pod, and it sets each request’s prefill estimate from what the chosen pod holds, instead of from how much the body grew.
- A shared prompt concentrates traffic. Clients that share a long prompt queue on the pod that holds it until it is busy enough to lose, so the median wait can rise while the slow cold-prefill tail shrinks.
- Limits. A pod’s blocks are forgotten when it leaves the endpoint set. A pod that restarts in place keeps its address, so its old blocks count until they age out and its first requests may land cold. Blocks are recorded when the pod is chosen, so a request that fails there and moves to another backend still counts on that pod. A body the proxy can’t read as chat, Messages or a completion (an embeddings request, say) is placed by load alone.
prefixAffinity requires mode: LoadAware. The webhook rejects it with mode: RoundRobin, and a stored router that has it with RoundRobin gets a PrefixAffinityIgnored Warning event.
Running more than one proxy replica
Each proxy replica keeps its own binding table and prefix memory. A restart starts both empty, and they store only hashes. With more than one proxy replica, a session that lands on different replicas gets different bindings. Hash the session header to a proxy replica upstream so a session always reaches the same replica, for example with APISIX chash on the header. Prefix affinity needs no such pinning to agree on a new prefix’s pod, but a replica only knows the blocks it sent itself.
Reading the metrics
llmkube_router_placements_total{router,backend,result,reason} counts decisions. result is one of:
least_loaded: a request without a binding, or a new session, went to the least-loaded pod.prefix_match: likeleast_loaded, but the chosen pod already held part of the request’s prompt (prefix affinity).sticky: a known session stayed on its pod.spill: a session moved off a pod under pressure (waitingorkv_full) and is now bound to the new pod.migrated: a session moved because its pod left the ready endpoint set (not_ready) or was still in that set but quarantined by the proxy (quarantined).fallback: the table was full and the consistent-hash home was used, or no pod was dispatchable (all quarantined or circuit-open) and the request went round-robin over the live endpoints. A fallback count alone does not mean the table is full.
reason is set only for spill and migrated, and empty otherwise. llmkube_router_session_bindings{router,backend} is the current number of bindings. It counts every session used within idleTimeout, not only live ones, so a client that starts a new session whenever one ends can hold several times its concurrent session count. Per pod, llmkube_router_endpoint_load, llmkube_router_endpoint_metrics_fresh and llmkube_router_endpoint_prefill_tokens cover LoadAware backends with a metrics profile (vLLM today). llmkube_router_prefill_tokens_estimated_total counts the prefill tokens this proxy estimated it sent to each pod; compare its rate with rate(vllm:prompt_tokens_by_source_total{source="local_compute"}[1m]) on the same pod to check the estimate.
With prefix affinity, llmkube_router_prefix_prompt_tokens_total{router,backend} counts the estimated prompt tokens of each placed request and llmkube_router_prefix_cached_tokens_total{router,backend} the part the chosen pod already held. The ratio of their rates is the prefix hit rate this proxy expects; on vLLM, compare it with the engine’s own vllm:prefix_cache_hits_total over vllm:prefix_cache_queries_total.
That ratio weighs requests by their size. llmkube_router_prefix_match_ratio{router,backend} is a histogram of the same share per request (buckets 0, 0.25, 0.5, 0.75 and 0.9), so you can tell whether a 70% hit rate means every request is 70% cached or long conversations are fully cached while short prompts never match. A prompt shorter than one block never matches, which is what blockBytes trades. llmkube_router_prefix_changed_placements_total{router,backend} counts the prefix_match placements the match decided: load alone, without prefix credit, would have sent the request to a different pod. The rest of the prefix_match placements went where load would have put them anyway, so its rate over the prefix_match rate is how often prefix affinity overrides load.
Upgrading from maxLoadPercent
sessionAffinity.maxLoadPercent is deprecated and ignored. The webhook returns a warning when it is set, so remove it from your manifests. An existing RoundRobin router that has sessionAffinity keeps serving with its affinity ignored, and gets a SessionAffinityIgnored Warning event. Change it to mode: LoadAware to get sessions back.
Every LoadAware router created under 0.10.7 got maxLoadPercent: 125 from the old CRD default, so the warning can fire even if your manifest never set it, and removing the field from the manifest does not clear it. Remove the stored value with a JSON patch, using the backend’s index for <i>:
kubectl patch modelrouter <name> --type=json -p '[{"op":"remove","path":"/spec/backends/<i>/balancing/sessionAffinity/maxLoadPercent"}]' A ModelRouter whose spec.proxy.image is pinned to a 0.10.7 proxy keeps the old ring behavior, and under the new controller any custom maxLoadPercent falls back to 125, because the controller no longer compiles it. Unpin the image.
Several models on dataPlane: AgentGateway
With dataPlane: AgentGateway, each backend’s InferenceService becomes an InferencePool, and agentgateway asks that pool’s Endpoint Picker (EPP) which pod serves each request. A Gateway API Inference Extension EPP binds to exactly one pool, so a router that fronts several models gives each backend its own picker:
apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelRouter
metadata:
name: b200-models
spec:
dataPlane: AgentGateway
agentGatewayRef:
name: ai-gateway
backends:
- name: glm
inferenceServiceRef:
name: glm-53
endpointPicker:
service: {name: glm-epp, port: 9002}
- name: qwen
inferenceServiceRef:
name: qwen-38
endpointPicker:
service: {name: qwen-epp, port: 9002}
rules:
- name: glm
match: {models: [glm-5.3]}
route: {backends: [glm]}
- name: qwen
match: {models: [qwen-3.8]}
route: {backends: [qwen]} agentGatewayRef.endpointPicker and endpointPickerPort remain the router-wide default for any backend without its own picker. The router reports AgentGatewayReady=False when:
- two pools would share one picker (
MultipleInferencePoolsUnsupported), which is what the default does when backends reference more than oneInferenceService; the same reason covers twoInferenceServicenames that compile to one pool name, such asglm.53andglm-53; - two backends on the same
InferenceServicename different pickers (ConflictingEndpointPicker); - a pool has no picker at all (
EndpointPickerRequired).
The pickers are yours to run; LLMKube references them by Service name and port in the router’s namespace. A backend without its own picker uses the picker another backend on the same InferenceService names, and only then the router default.
Each InferencePool is named after its InferenceService, and a router never adopts a pool another router created. When you move from one router per model to one router for several, delete the per-model routers first: until their pools are garbage-collected, the combined router reports GeneratedResourceNameTaken.
primary-fallback routes each rule to its first healthy backend. When that backend’s InferenceService has no ready replica, the next reconcile moves the rule to the next backend. Failover follows replica readiness, not individual request errors: a primary that is up but failing requests keeps its traffic.
Model routing on dataPlane: AgentGateway
agentGatewayRef.routing: Model compiles the router onto agentgateway’s own model router instead of the Inference Extension. It needs agentgateway v1.6.0 or later, which ships the AgentgatewayModel CRD, and a Gateway. It needs no Gateway API Inference Extension CRDs, no Endpoint Picker, and no Gateway-wide policy, which suits single-replica InferenceServices reached through their Service.
apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelRouter
metadata:
name: llm-router
spec:
dataPlane: AgentGateway
agentGatewayRef:
name: ai-gateway
routing: Model
sectionName: llm
backends:
- name: chat
displayName: chat-large
inferenceServiceRef:
name: chat-vllm
- name: chat-small
inferenceServiceRef:
name: chat-llama
- name: reranker
inferenceServiceRef:
name: reranker chat-vllm must start vLLM with --served-model-name=chat-large in its extraArgs: without it, LLMKube starts vLLM serving the InferenceService’s spec.modelRef (and the model path), not chat-large.
The router generates, all in its own namespace and owned by it:
- one
HTTPRoutenamed after the router, attached to the Gateway, with a single rule namedmodelsat/whose only backend isAgentgatewayModel*. The rule sits at/because agentgateway strips the rule’s path prefix before it classifies the endpoint: under/v1,/v1/embeddingswould parse as a chat completion; - one
AgentgatewayModelper backend, named<router>-<backend>(cut and hashed past 63 characters), attached to that rule. Itsmatch.modelis the backend’sdisplayName, else itsname, so served names such asQwen3.5-2Bororg/modelwork. It reaches theInferenceService’s Service through a custom provider, and the request keeps its model name, so the served name must be one the runtime answers to: llama.cpp ignores it, and vLLM answers to its--served-model-namefromextraArgs, else tospec.modelRef.
The provider and API formats come from the InferenceService:
| Runtime | spec.mode (or inferred) | providerOverride | Formats |
|---|---|---|---|
llamacpp (or unset) | chat | llamacpp | Completions |
vllm | chat | vllm | Completions, Messages, AnthropicTokenCount, Responses |
llamacpp, vllm | embedding | as above | Embeddings |
llamacpp, vllm | rerank | as above | Rerank |
agentgateway then routes each request on the body’s model, answers GET /v1/models itself, and returns a JSON 404 model_not_found for a model no backend serves.
A rule at / takes every path on the listeners it attaches to, so this mode requires agentGatewayRef.sectionName (one listener), agentGatewayRef.hostnames, or both. Your own routes still share the listener: a rule with a longer match, such as a PathPrefix /v1 with a header match, outranks the router’s /. Use one routing: Model router per listener and hostname. Two routers’ / rules tie, Gateway API gives the tie to the older route, and the newer router’s models get no traffic.
The router reports AgentGatewayReady=False and generates nothing when:
- the
AgentgatewayModelCRD is not installed (AgentgatewayModelCRDNotInstalled); - the spec has
rules, adefaultRoute, anexternalbackend,resolution: endpoint, a backendendpointPicker, a served name containing*, or two backends serving one name (UnsupportedInModelRouting; the webhook rejects these at apply time). Fallback between backends and external backends need anAgentgatewayBackend, which this mode does not generate yet; - a backend’s
InferenceServiceruns a runtime other thanllamacpporvllm, or runs vLLM serving a name other than the backend’s served name (UnsupportedRuntimeInModelRouting); - a generated name is held by an object LLMKube did not create (
GeneratedResourceNameTaken); delete or rename that object.
When you replace hand-written AgentgatewayModels with a router, remove the hand-written ones for the same model names on that listener, so each name has one model.
Removing a backend deletes its model. Switching routing in either direction deletes the other mode’s objects, including the InferencePool mode’s extraction policy in the Gateway’s namespace.
Composition with LiteLLM
LLMKube does not replace LiteLLM. For organizations already running a LiteLLM proxy as their cloud-provider abstraction, point a ModelRouter external backend at it:
- name: anything-via-litellm
tier: cloud
external:
provider: litellm
url: http://foundation-router.gateway.svc.cluster.local:4000
model: openrouter/anthropic/claude-opus-4-7
credentialsSecretRef:
name: litellm-master-key LiteLLM handles provider auth, retries, and cost tracking. ModelRouter adds K8s-native policy, fail-closed enforcement, and audit logs.
Cluster-wide LiteLLM default
Platform teams can centralize the LiteLLM URL via a controller flag so application teams don’t have to repeat it on every ModelRouter:
# Helm values
controllerManager:
routerProxy:
defaultLiteLLMURL: http://litellm.litellm.svc.cluster.local:4000 With that set, application teams can declare LiteLLM-backed routers without url:
external:
provider: litellm
model: openrouter/anthropic/claude-opus-4-7 A per-backend url always wins over the cluster default.
Budgets
spec.policy.budgets caps tokens (maxTokens) or estimated dollars (maxUSD) over a rolling window (windowSeconds, default 3600). The Gateway and AgentGateway data planes accept only maxTokens and the router and team scopes. On the Proxy data plane, once a budget that applies to a request is used up, the request is refused with 429 and a Retry-After header before any backend is contacted. Each budget has a scope:
router: every request through the router.rule: requests that match the rule named inruleName.team: requests that carry a team header (headerKey, defaultx-llmkube-team). Each header value is charged separately.
Team budgets bind only callers that cannot choose the header. The router-proxy does not authenticate callers, so a caller that omits the header is held to no team budget, and one that sends a different value is charged to that value instead. Rule matching also reads the model name and headers the client sends. Only a router budget applies to every request. To enforce team budgets, put an authenticating gateway in front of the router-proxy that sets the team header from the caller’s identity and overwrites any value the client sent (the APISIX guide covers per-client keys), and use NetworkPolicy so callers reach the router-proxy only through that gateway and cannot call the backend Services directly. On dataPlane: Gateway, spec.policy.auth.jwt copies a verified JWT claim into the header instead; the router-proxy does not read spec.policy.auth. The multi-tenancy notes cover this in more detail.
How the proxy bounds team counters
Each team value gets its own counter, keyed team:<headerKey>:<value>. The value is whatever the request carries, so the router-proxy bounds what it keeps for it:
- Idle values are dropped. A value with no usage left inside its budget’s window is deleted from the proxy’s memory and from
/admin/budgets, and itsllmkube_router_token_budget_utilizationseries is removed the next time the gauge is updated. The proxy republishes the gauge off the request path, at most once a second and within a second of each budgeted request, so a charge shows on the gauge up to a second after the request that made it. The cleanup runs at most once a second, when a request is charged, when the budget endpoint is read, or when the cap below is reached. It changes no decision: a value with no usage in the window is treated exactly like a value never seen. A team budget’s entry instatus.budgetUtilizationtherefore aggregates only the values with usage in the current window. - At most 10,000 team values are tracked at once, across all team budgets of the router. While that many have usage in their window, a request with a new value does not get a counter of its own: it is charged to, and limited by, the budget’s shared overflow counter
team:<headerKey>:__overflow__, which carries the same cap as a single value. New values are never let through unlimited. Values that already have a counter keep it and its cap. Each folded request incrementsllmkube_router_budget_team_overflow_total{router, budget}, and the proxy logs a warning the first time it happens. A sustained non-zero rate means new teams are sharing one cap. - Long values are hashed. A value longer than 128 bytes, or one that is not valid UTF-8, is kept as
sha256:<hex>of the value, so a counter’s key has a fixed size however large the header is. Shorter values are kept as sent, so existing dashboards and status entries keep their keys.
Shape compatibility
The router-proxy forwards the inbound OpenAI chat-completion request body to upstream backends verbatim. For backends that already speak OpenAI (local InferenceService pods, LiteLLM, an in-cluster vLLM or oMLX), this is correct. For first-party providers whose native API does not match (Anthropic’s /v1/messages, Bedrock’s per-region shapes, Vertex AI’s structured Content payloads), put a LiteLLM proxy in front and reference it via provider: litellm — LiteLLM handles the per-provider translation.
When you specify provider: anthropic or provider: openai without url, the controller fills in the published default (https://api.anthropic.com, https://api.openai.com). This works as-is for OpenAI (which already speaks the OpenAI shape) and for any Anthropic-compatible endpoint that accepts OpenAI requests. Direct calls against api.anthropic.com itself require LiteLLM in front; native Anthropic-Messages translation is on the roadmap, not in Phase 1.
What’s in scope, what isn’t
In scope:
- Routing rules: data classification, task complexity, required capabilities, model glob, header match
- Strategies: primary-fallback, weighted and shadow
- Fail-closed gate, both static (apply-time) and runtime
- OpenAI-compatible request / response, including streaming SSE
- Structured audit log to stdout (other sinks roadmap)
- Token and dollar budgets per router, rule or team (see Budgets)
- MCP server endpoint (roadmap)
Out of scope:
- The agent runtime itself. ModelRouter is consumed by LangGraph, CrewAI, OpenAI Agents SDK, Anthropic Agent SDK, Cline, OpenCode, Aider, and any other framework that speaks the OpenAI API. We don’t reinvent that layer.
- Inference engines.
InferenceServicealready wraps llama.cpp, vLLM, TGI, oMLX. ModelRouter sits above them. - General-purpose K8s gateway. ModelRouter is scoped specifically to LLM traffic with policy.
Phase 1 limitations (v1alpha1)
The Phase 1 router-proxy ships with three concrete gaps that show up on agentic-coding workloads. They are intentional scope for v1alpha1 and tracked for Phase 2; calling them out so you can plan around them rather than discovering them mid-incident.
1. Timeouts cap TTFT, not stream duration
rule.timeout and backend.timeout apply to the time-to-first-byte (first response header from the upstream), not the total duration of
the stream. Once the upstream starts sending SSE chunks, the proxy
will pipe them to your client for as long as the upstream keeps
producing, with no aggregate cap.
For agentic-coding workloads this is usually what you want (large refactors generate 5-10 minute SSE streams that you don’t want the proxy interrupting), but it means a hung stream where the upstream goes silent mid-response is bounded only by kernel TCP keepalive or your client’s read deadline. Set a client-side timeout in your agent runtime as the safety net for hung streams. A stream-duration cap field is planned for Phase 2.
2. Audit log is coarse
The per-request audit-log line records latencyMs, status, outcome, and timeoutMs, but not streamed bytes, token count, or
stream-duration breakdown. “Why did this 8-minute agent loop run
slow?” needs upstream metrics, not the proxy log.
outcome says how a request ended:
okorok_stream: the response was relayed in full.client_cancelled: the client disconnected. If no backend had answered yet, the line records status499and no fallback is tried.upstream_error: the backend broke off mid-response.timeout: the backend missed the header deadline (on a rule that is not fail-closed, the client gets a504), or a stream went silent paststreamIdleTimeout.all_backends_failed: every backend errored, returned a 5xx, or was quarantined (502).no_route: no rule matched and there is nodefaultRoute(503).fail_closed: a fail-closed rule refused a sensitive request before dispatch (503).fail_closed_runtime: every backend of a fail-closed rule failed (503).pool_activation_timeoutorpool_incumbent_busy: a ModelPool member was not resident in time (503withRetry-After).budget_exceeded: a token budget was exhausted (429).bad_request: the request body could not be read or is not JSON (400). It is counted in the metrics below but writes no audit line.
The failure outcomes carry the underlying error. The same value labels llmkube_router_requests_total, which counts every request the proxy
answers on an inference endpoint, and llmkube_router_request_duration_seconds observes each of
them. The router label on these and the other llmkube_router_* series is the ModelRouter’s name.
For a request no backend served, the backend label is the last
backend the proxy tried, and empty when it tried none (bad_request, no_route, fail_closed, budget_exceeded). llmkube_router_backend_health is
refreshed for every backend a request tries, and llmkube_router_active_backends after every counted request, so a quarantined
backend reads 0 whether or not a request to it has since succeeded. Without
any request, both are also refreshed when a pod enters or leaves the
load-poll quarantine described under How load is read,
and when the proxy’s set of ready pods for an endpoint-resolution backend
changes. So a backend whose every pod is unreachable reads 0, and one that
regains a dispatchable pod reads 1, with no traffic.
The proxy cancels the upstream request when its own request ends, so a cancel task line in llama.cpp’s log usually pairs with a client_cancelled or timeout line here.
To see who sent each request, opt in under spec.policy.auditLog:
spec:
policy:
auditLog:
logClientAddress: true
trustedProxyCIDRs: ["10.0.0.0/8"] # only behind an ingress or gateway
callerHeader: X-LLMKube-Client logClientAddressaddsclient, the sender’s address.- In-cluster callers keep their pod IP through the ClusterIP.
- The backend only ever sees the proxy’s IP, so llama.cpp’s own log can’t tell you this.
- It’s off by default because client addresses are personal data in some deployments.
trustedProxyCIDRsnames the proxies in front of the router whoseX-Forwarded-Foris believed.- When the TCP peer is one of them,
clientis the first address inX-Forwarded-For, read from the right, that is not a trusted proxy. - Any client can send
X-Forwarded-For, so without this list the header is ignored. - Entries are CIDRs or single addresses, and the field requires
logClientAddress.
- When the TCP peer is one of them,
callerHeaderaddscaller, the value of a header each client sets.- The client chooses the value, so treat it as a label, not an identity.
- It’s cut to 128 bytes, and characters outside printable ASCII become
?. - Headers that carry a credential (
Authorization,Proxy-Authorization,Cookie,X-Api-Key,Api-Key,X-Goog-Api-Key) are refused.
The proxy doesn’t look up pod names: resolving an IP to a pod would need read access to every pod in the cluster. On Cilium, hubble observe --to-pod <namespace>/<router>-router-proxy --port 8080 shows the calling pods.
Per-request token/byte accounting and a proxy-emitted Prometheus histogram are planned for Phase 2 (issue #433).
3. Inbound request bodies are buffered
Outbound responses stream chunk-by-chunk; inbound requests are
fully buffered before routing (up to a 32 MiB cap, enough for
~128K-token prompts). A request’s body is released once it has been
sent upstream, so a reply that streams for minutes does not keep its
prompt in memory. While a request is being sent, the proxy holds
about two to three times its body size when it rewrites the model
name. A long-context agent prompt of 75K tokens is about 300 KB of
JSON, so a few hundred such requests arriving together need a few
hundred MiB. The default limit is 1 GiB (request 64 MiB); raise spec.proxy.resources for larger bursts. True request-side
streaming is planned for Phase 2.
Comparison to alternatives
| ModelRouter | LiteLLM proxy | KubeAI Model Proxy | llm-d Inference Gateway | |
|---|---|---|---|---|
| K8s-native CRD | ✓ | — (Helm chart, no CRD) | ✓ (limited) | ✓ (Gateway API extension) |
| Cross-engine handoff (local + cloud) | ✓ | ✓ (cloud only) | — (local intra-cluster only) | — (vLLM-focused) |
| Fail-closed for sensitive data | ✓ (static + runtime) | — | — | — |
| Audit log per request | ✓ | partial | — | partial |
| Composes with LiteLLM | ✓ (as a backend) | n/a | — | — |
The three peers all solve adjacent but different problems. ModelRouter’s specific niche is policy-aware hybrid routing for regulated-industry adoption: the place where “I run my own AI” meets “I sometimes need to call Opus” meets “compliance must be enforceable.”
Status surface
After a successful reconcile, status on a ModelRouter looks like:
status:
phase: Provisioning # or Ready / Degraded / Failed
endpoint: http://coding-router-router-proxy.default.svc.cluster.local:8080/v1/chat/completions
activeRules: 2
backends:
- name: local-qwen
tier: local
address: http://qwen3-coder.default.svc.cluster.local:8080
healthy: true
- name: cloud-opus
tier: cloud
address: https://api.anthropic.com
healthy: true
conditions:
- type: Validated
status: "True"
reason: SpecValid
- type: BackendsReady
status: "True"
reason: BackendsResolved
- type: Available
status: "True"
reason: DeploymentReady phase is the coarse summary; the conditions tell the full story.
When the spec itself is invalid, Validated is False, phase is Failed, and the operator changes nothing in the router-proxy: a new router gets no proxy, and an existing one keeps serving the last config that compiled. The message lists every problem, and the reason is one of these:
| Reason | Meaning |
|---|---|
SpecInvalid | The spec breaks a rule the schema cannot express, such as a sensitive rule routed to a cloud backend or a defaultRoute naming no backend. |
ClassifierNotImplemented | policy.classification.mode is detector or hybrid, which need the in-proxy classifier that is not built yet (#441), and nothing else in the spec is invalid. Set header-only. See Where the classification comes from. |
When the data plane cannot serve, Available flips to False with one of these reasons:
| Reason | Meaning |
|---|---|
CompileFailed | The routing spec did not compile into a proxy config. |
ReconcileFailed | Applying the ConfigMap, Deployment, Services, or activation RBAC to the API server failed. |
RouterProxyNameTaken | An object LLMKube did not create, or another router’s object, already holds a router-proxy name: one generated from the router’s name (the ConfigMap, Deployment, Services, ServiceAccount, Role, or RoleBinding) or the Service name set in spec.proxy.serviceName. The operator refuses to adopt or overwrite it; delete or rename the foreign object, or the router, and the reconcile recovers on its next pass, rechecked about every 2 minutes. |
DeploymentNotReady | The proxy Deployment exists but its pods are not Ready yet. |
Next steps
- Read the CRD reference for the full spec.
- See Multi-GPU sharding for backing the local InferenceService with sharded GPU pods.
- Read the comparison page to understand how ModelRouter fits relative to other K8s LLM operators.