Every release makes the harness harder to fool: LLMKube 0.9.19

LLMKube 0.9.19 shipped this morning, and it is the strangest release we have cut. About half of it exists because we caught our own agent pipeline lying to us, in four different ways at once. The other half was written by that same pipeline, after we fixed it. This post is the honest version of how that went, because the interesting product is not the features, it is the compounding: every release since 0.8.0 has made this system measurably harder to fool, and this is the release where that loop visibly closed on itself.

Nine files, full test coverage, zero callers

It started with a routine backlog triage. Twenty-five issues were sitting open with merged PRs attached, each honestly labeled "not wired into a running pipeline yet." When I checked what that actually meant, the answer was worse than a stale backlog: nine files on main, each fully unit-tested, each with zero production callers. Verdict rails, audit collectors, a staleness detector. Written, reviewed, merged, and completely inert.

The uncomfortable part is why nothing caught it. Go does not warn about unused package-level functions. Our linter's unused checker treats a function as used if tests reference it, and these files had excellent tests. A thoroughly-tested dead function is invisible to every standard gate. The tests are what hid the bodies.

The fix is almost embarrassingly small. The same linter, run a second time with test files removed from the analysis graph, reports every one of them:

# .golangci-deadcode.yml
run:
  tests: false   # test-only references must not count as uses
linters:
  default: none
  enable: [unused]

That runs as a blocking CI step now. We proved it against the exact bug shape before trusting it: a new function plus a passing test and no caller sails through the normal lint at zero issues and fails the guard immediately. The known dead files went into a suppression register where each entry names the issue tracking its wiring, and deleting the entry is the proof the code became reachable. The register started at nine. It is down to six, and every removal happened in a PR that made the code real.

How did it pass review? It didn't. There was no review.

Foreman's pipeline runs a reviewer agent against every coder branch. Those nine PRs all carried GO verdicts. Digging into how produced the second discovery: the reviewer agent had been running for three days with an empty system prompt. Its Agent resource had been hand-created on the cluster with no systemPrompt and a tool list missing fetch_issue, so the model was improvising a review process with no instructions and structurally could not read the issue it was reviewing against. The task prompt even said "follow Step 1 of your system prompt." There was no Step 1. There was no system prompt.

And the third discovery explained why the deterministic rails, which exist precisely to catch a bad reviewer, stayed silent: several of them quietly returned the model's verdict unchanged whenever their inputs went missing. No diff available? Skip, silently. No issue body in the transcript? Skip, silently. The model omitted the optional structured-output field? The entire rail layer received a nil map and every rail no-opped at once. A verdict nothing had checked was indistinguishable, in the record, from a verdict that had earned its pass.

One upstream failure disabling the exact safety net built to catch it: that is the failure class this release is obsessed with.

0.9.19: the receipts layer

The features in this release share one design rule: silence and success must never look the same.

• Rails record why they could not run. Every verdict rail that skips now writes the reason into the task record: railsSkipped: ["scope-overlap: no-issue-body"] instead of nothing. A review produced without its checks now says so in its own artifact.
• A rail that catches reviews that never read the diff. If a reviewer returns a verdict and the transcript contains no diff command, that is detected deterministically, because we watched it happen twice.
• Cross-stage contradiction detection. When one pipeline stage claims "already approved" and the recorded verdict says otherwise, the disagreement becomes data instead of being resolved by whichever stage spoke last.
• A config floor for agents. An Agent below the minimum its role needs, an empty system prompt, a reviewer without fetch_issue, now shows VALIDATED: False in kubectl get agents, and every task it completes anyway gets a warning stamped into its record. Warn, not refuse: a mis-configured agent still runs, it just cannot run invisibly.
• An execution-first reviewer rubric. The reviewer is now instructed to run the new tests and probe a near-miss case rather than only reading the diff, and to declare what it did not verify in an onTrust field. Its first live run honestly disclosed the steps it skipped, which told us exactly what to tighten next. Bounded verification that cannot name its bounds is claiming an exhaustiveness it does not have.

The other half: making "add tests for X" winnable

The release's second thread closes a bug class a contributor nailed with a beautiful report: the scope check that verifies a diff touches the files an issue names made an entire category of work unwinnable. An issue says "add tests for platform.py." The correct change creates tests/test_platform_gh_api_forgejo.py. The check reports the diff touches none of the named files, demotes the verdict, and the task burns its whole attempt budget with 1,100 lines of finished, correct tests stranded on a branch nobody looks at.

Three slices, each shipped with its boundary honestly named and the next slice filed with a live reproduction: GateProfile.testLayout declares a repository's parallel test tree, so src/test/java/FooTest.java folds to src/main/java/Foo.java. A content-based vouch reads the diff-added test file and matches its imports against the named module, so a test named for a feature still counts for the module it covers. And for tests appended to an existing file, the vouch matches on the diff's added lines only, because a file that merely already imported the module is not evidence of new coverage. That last distinction is the difference between a useful check and a loophole, and it came straight out of review pushback.

The loop closes: the fleet fixed its own pipeline

Here is the part that makes this release different from a changelog. After the guard landed and the reviewer got its instructions back, we fed the remaining work to the Foreman fleet itself. The content-vouch feature, the config-floor fix, the added-lines slice: written by a local coder model, gated by the CI guard that made its predecessors' failure impossible, reviewed under the rubric this release ships, and judged against the issue before any PR opened.

One of those runs produced a wrong-but-plausible implementation, a predicate that diverged from the executor's real routing logic. The pipeline's own verdicts said GO. Independent review caught it, fed the finding back as a revision cycle, and the corrected branch came back with a behavioral equivalence test proving the two predicates cannot drift apart again. The system did not just produce work, it produced work whose flaws the process found and fixed. That is a much stronger claim than "the model got it right," and it is the only claim we are actually interested in.

The gates bit me too, which is the point. While building the config floor I orphaned two constants into test-only use; the dead-code guard failed my own PR. My first wiring pushed a function over the complexity limit; the linter rejected it and forced a cleaner shape. A harness that only polices the agents is theater. This one polices whoever shows up.

What compounding actually looks like

0.8.0 shipped the first coder-verifier-reviewer pipeline. 0.9.x releases added the grounding rails, the honest-verdict contract, the gate profiles. Each of those made a class of failure louder. 0.9.19 is the first release where the accumulated loudness turned around and audited the auditors, and the result was nine dead files, one uninstructed reviewer, and a set of rails that failed open. Finding that is not an indictment of the approach. Finding it with the approach's own tools, then shipping the fixes through the pipeline being fixed, is the approach working.

Next up, in public as always: wiring the remaining registered rails, an operator-level model for multi-node serving that replaces a hand-built pattern we validated the hard way this week, and the first serious benchmark of a 397B model spanning two DGX Sparks. The register shrinks, the record gets more honest, and the next thing that tries to pass silently will have a harder time than the last one. That is the whole roadmap, really.

Full changelog: v0.9.19 on GitHub. Every issue and PR referenced in this post is public in the repo, task records and all. If you are building agent pipelines and want to compare notes on verdict integrity, the Discord is open.