My Kubernetes operator passed every health check while it ignored every change

A GitOps tool applied a one-line change to an InferenceService. The custom resource showed the new spec. The pods kept running the old one. The operator that should have rolled them was Running, passing its probes, and answering admission webhooks. Its controllers logged nothing at all. After seven minutes, someone restarted the operator by hand, and the Deployment picked up the change about ten seconds later. This was the third time I had seen a cluster look this healthy while ignoring changes. The first two times, I blamed my lab.

LLMKube is the Kubernetes operator I build to run LLM inference across NVIDIA, AMD and Apple Silicon. This post is about the operator itself, not the models. It is also not a new idea. In January, someone running their own operator on Oracle's managed Kubernetes reported the same silent stall in controller-runtime#3428. Their workaround was to restart the watch after three minutes without an event. The triage bot closed that issue in July without a fix, so there is no upstream change to wait for.

The first two times, the whole cluster had stopped listening

My lab cluster runs MicroK8s, which keeps its state in dqlite. On August 19 I deployed a new operator build and nothing happened. Helm reported success. The specs and CRDs were written. No ReplicaSet was ever created. The old operator pod, still running, then rejected the new CRD fields with strict decoding error: unknown field, which looked exactly like a bad deploy. It was not.

Every Deployment in the cluster had metadata.generation ahead of status.observedGeneration, and stayed that way for hours. Every API server returned the same, current data. The controller manager kept renewing its leader lease. Writes worked and reads worked. What had died was the watch path: controllers listed the current state when they started, then never heard about another change.

I restarted the Kubernetes daemon three times, and the fresh processes attached to the same dead watches each time. What fixed it was restarting the dqlite daemon underneath it on each node, one at a time, then Kubernetes. On September 9 I saw the same signature again: my operator's own Deployment sat at generation 162 with 161 observed, while all five control plane nodes showed Ready=Unknown and their leases kept renewing.

Both times I wrote it down as a problem in my lab's control plane, not in LLMKube. I never found what triggered either one.

Then it happened on a managed control plane

The third time was on a managed AKS cluster running LLMKube 0.10.3, with one operator replica and leader election on. There was no dqlite to blame, and the cluster's own controllers kept working: the restart was processed right away. Only LLMKube's controllers had gone quiet. The timeline from the operator log, in UTC:

There was not one error line in that window. The only evidence was the same one as in my lab: the InferenceService's conditions and its Deployment both stuck at generation 1 while the spec had moved on. I filed it as #2069. Whatever the trigger, the operator had to notice this itself, because the alternative is a person noticing that a change never happened.

The liveness probe only asked whether the process could answer HTTP

The operator's liveness check was the one the Kubebuilder scaffold generates:

mgr.AddHealthzCheck("healthz", healthz.Ping)

healthz.Ping always passes. It proves the health server can answer a request, and nothing about whether the controllers behind it are hearing about changes. The webhooks answering proved the same thing. A green pod told me the process was alive, which I already knew.

It was not one stuck reconcile, but the fix had to cover that too

My first suspicion was a single hung reconcile. The InferenceService controller ran with one worker, so one call that never returned would freeze every InferenceService behind it. But the Model controller had four workers, and Models were updated at 20:38 too. It also logged nothing. Two controllers going quiet together points at the events never arriving, not at one stuck worker.

I could not prove either one from the logs, so the fix in #2072 covers both. Every Model and InferenceService reconcile now has a 15-minute deadline, and the InferenceService controller runs four workers instead of one. The rest of the fix is for the watch path.

The fix is a write the operator has to see come back

Every minute, the leader renews a Lease named llmkube-controller-manager-watch-canary in its own namespace. Then it waits for its own informer to deliver that change back. If five minutes pass with a write it never saw, the liveness check fails and the kubelet restarts the pod. That restart is exactly what fixed the AKS stall by hand.

--watch-canary-interval=60s
--watch-canary-fail-after=5m
--reconcile-timeout=15m

A few choices in it took more thought than the idea itself:

• It reads the Lease straight from the API server, not from the cache. The cache is the thing being tested, so the canary cannot depend on it to decide what to write.
• It fails liveness, never readiness. Failing readiness would pull the pod out of the webhook Service. The InferenceService webhook fails closed, so every create and update would then be rejected, which turns a quiet stall into an outage. I have seen that happen in my lab when the webhook lost its endpoints for another reason.
• Only the leader writes. A standby replica has nothing to reconcile, so it passes.
• Two alerts back it up. One pages when the stall outlasts the restart, set to 15 minutes by default. The other warns when the canary has never completed a cycle at all, which often means its RBAC is missing.

It shipped in LLMKube 0.10.4 on October 16, on by default. The implementation, tests and chart changes were written by GLM 5.3 Flash, a local model that LLMKube serves on three NVIDIA GB10 machines. I reviewed the diff and ran the gates. The runbook covers the alerts and the manual check.

What it does not catch

I would rather list these myself than have someone find them in production.

• It watches Leases, not InferenceServices. Each resource type gets its own informer and its own watch. The canary proves that the operator's connection to the API server still delivers events. If the InferenceService informer died on its own, the way one did in the upstream report, the Lease would keep coming back and the canary would stay green. Today the only check for that is the manual one in the runbook. There is no alert for it yet.
• A restart only helps if the operator is the broken part. On AKS it was. In my lab, the whole cluster's watch path was dead, and restarted Kubernetes processes attached to the same dead layer three times. I expect a restarted operator would have done the same. There, the canary's job is to page someone, and the fix is still the datastore.
• It has not caught a real stall yet. My lab Prometheus has its readings since October 21. It reported down twice, each time in the first two minutes after an operator restart, before its first write came back. I do not have a second occurrence to show you.
• I still do not know the trigger. The canary detects the stall and gets the operator out of it. It does not explain it.

If you write an operator, check these

1. Compare generation and observedGeneration on something your operator owns. It was the one signal present all three times. For Deployments:

   kubectl get deploy -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,GEN:.metadata.generation,OBSERVED:.status.observedGeneration'

   A row where OBSERVED stays behind GEN for minutes means a controller is not processing changes.
2. Look at what your liveness check proves. If it is healthz.Ping, it proves the HTTP server is up and nothing else.
3. Add a write your operator has to observe. Pick a kind no controller watches, such as a Lease, so you can scope its informer to one namespace and label without narrowing anyone else's watch. Read it through the API reader, not the cache.
4. Fail liveness, not readiness, if you serve webhooks. Otherwise a stall becomes an admission outage.
5. Write only from the leader. Let standbys pass.
6. Put a deadline on every reconcile, and run more than one worker. One call that never returns should not freeze every object of that kind.
7. Alert above the restart threshold. The restart is the first response, and the page is for when restarts are not fixing it.
8. Know what your canary does not cover. Mine misses one informer dying alone.

If you run LLMKube 0.10.4 or later, the canary is already on. If your cluster manages RBAC outside the chart, the operator's ServiceAccount needs get, list, watch, create and update on Leases in its namespace. Without that, the canary fails every cycle and the pod restart-loops. The runbook shows how to turn it off until the rule is in place.

Where the receipts are

• #2069: the AKS stall, with the operator log and the timeline.
• #2072: the canary, the reconcile deadline, the worker count and the alerts, with the design notes.
• internal/watchcanary: the code, about 400 lines.
• The stalled-watch runbook: detection, mitigation and the manual check.
• controller-runtime#3428: the same silent stall in someone else's operator, and their workaround.

If you have hit this in your own operator and found the trigger, I would really like to hear it. The Discord is open.