My local coder wrote a test that passed by hiding its own bug

Foreman, the local agentic harness that maintains LLMKube, spent last night fixing a bug in its own coder loop. It cloned the repo on a Mac I own, ran a 27B model with no cloud inference in the loop, produced a diff, passed the gate suite, and reported the verdict the harness reserves for work it believes is done: GO. The tests were green. The change was also wrong. Not sloppy, not incomplete. Wrong in a way that is specific to code written by a model, and worth a whole post because most people shipping AI-written code have not yet been bitten by it.

The bug it was fixing

The coder loop has a failure mode where a verbose model runs its generation into the max_tokens ceiling mid-thought, before it emits a tool call. The harness catches that, nudges the model to stop deliberating and act, and continues. But the truncated, multi-thousand-token block of reasoning stayed in the transcript, so it re-entered the context window on every subsequent turn. In one evaluation run the context ballooned to about 82,000 tokens and a task burned through its entire turn budget, all from per-turn cost, not from doing more work. The bug is filed as #1328.

The fix is the obvious one: after the model resumes from a truncated turn, trim the wasted reasoning out of the stored transcript so it stops re-prefilling. The coder wrote exactly that, a small function that walks the transcript and clears the reasoning off the truncated message:

// clear reasoning from the truncated assistant message
for i := len(transcript) - 1; i >= 0; i-- {
    if transcript[i].Role == RoleAssistant &&
        transcript[i].ReasoningContent != "" {
        transcript[i].ReasoningContent = ""
        return
    }
}

It also wrote a test. The test built a transcript with a truncated reasoning message followed by the continuation, ran the function, and asserted the truncated reasoning was gone. It passed. So did the gate suite, the build, the linter. Green across the board.

The tell

I read the diff before I would let it merge, and the tell was in the test, not the code. The function trims "the most recent assistant message that has any reasoning." But think about what the continuation turn looks like. The model resumed, and the nudge that resumed it says, in so many words, keep your reasoning short. Short is not empty. A verbose model, told to be brief, still emits a little reasoning alongside its tool call. So the most-recent assistant message with reasoning is often the continuation, not the truncated turn. Clear that one and you have deleted the model's live reasoning and left the wasted block exactly where it was.

The test never saw this, because the test built its continuation message with a tool call and no reasoning. That is the one shape of continuation that cannot expose the bug. The fixture was constructed, precisely, around the input that keeps the function looking correct.

This is the part worth slowing down on.

When the same model writes the code and the test, the test is not an independent check. It is a second artifact shaped by the same understanding, including the same blind spot. My coder did not write a test that happened to miss the bug. It wrote a test whose fixture avoids the one input that triggers it, because the model that built the fixture had the same wrong mental model as the model that wrote the code. They agreed. The green check was not weak evidence of correctness. It was evidence of nothing, wearing the costume of a pass.

This is different from the failure I have written about before, where the harness catches the model overreaching. This is sharper and quieter: the model's own test actively vouched for the bug. Nothing looked wrong. That is exactly why nothing looking wrong cannot be the bar.

Proving it in ten lines

A suspicion is not a defect. Before touching the code I wrote the test the coder avoided writing: a continuation message that carries a short reasoning of its own, next to the wasted truncated one, and then asked the function which it would clear.

tx := []Message{
    {Role: RoleAssistant, ReasoningContent: "wasted truncated thought"},
    {Role: RoleUser, Content: nudge},
    {Role: RoleAssistant, ReasoningContent: "short continuation thought",
        ToolCalls: []ToolCall{{ID: "1"}}},
}
trimTruncatedReasoning(tx)
// want: tx[0] trimmed, tx[2] untouched
// got:  tx[0] UNTOUCHED, tx[2] wrongly cleared

The probe failed on both assertions. The function left the wasted reasoning in place and stripped the continuation's live reasoning instead. The fix was not merely incomplete. In the exact case it was built for, a verbose model, it did the opposite of its job, and shipped with a test that certified the reversal as correct.

The real fix keys off something the coder's version ignored. A turn cut off at the token limit is reasoning with no tool call, by construction. The continuation is the one carrying the tool call. So the message to trim is the most-recent reasoning-only assistant message, and the discriminator is a single extra condition:

if transcript[i].Role == RoleAssistant &&
    transcript[i].ReasoningContent != "" &&
    len(transcript[i].ToolCalls) == 0 {   // <- the truncated turn has no tool call
    transcript[i].ReasoningContent = ""
    return
}

Then I turned the probe into a permanent regression test and bite-checked it the way the harness bite-checks everything: revert the one-line guard, confirm the new test fails with the exact defect, restore. A test that still passes when you break the code it covers is not coverage, and this is how you find out which kind you have. The corrected fix and the real test shipped in #1352.

Why the harness never takes the model's word

Foreman's whole design is one idea: the model's own report is not evidence. A coder can say GO, and the harness still runs the real test suite in a clean-room job on the pushed branch, because a diff that passes inside the loop can fail on main. It can pass its own gate, and an independent reviewer on different hardware reads it anyway. And crucially, the review reads the test's fixtures, not just its pass or fail, because a green check is where AI-written tests hide. This one was caught by exactly that: a human review that looked at what the test was made of.

The deterministic backstop for the same failure is a mutation check: delete or break the implementation and require the test to fail. A test whose fixture dodges the bug also tends to survive mutations it should not, so the two catch the same lie from different directions. Neither one needs a smarter model. They need a system that assumes the model is confidently wrong until something that is not the model says otherwise.

None of this is a knock on local models, and it would be a mistake to read it as one. A frontier model writing its own tests has the same incentive and makes the same move, it just makes it in more places you are not looking. The lesson is not "use a bigger model." The lesson is that a passing test written by the same agent that wrote the code is a statement of intent, not a proof of correctness, and you need something outside that agent to tell the two apart.

LLMKube and Foreman are Apache-2.0 and built entirely on self-hosted inference. Every check in the harness looks like paranoia until you meet the run that needed it. This is the one that needed the fixture check. It will not be the last.

The full run, both diffs, and the bite-check are in #1328 and #1352. If you want to see the harness, or run it on your own fleet, the Foreman docs are the place to start, and the repo is open.