A flag I never questioned was costing 2.2x

One flag in my inference config had been costing me 2.2x for weeks. Not a bug, and not a bad model: a default I copied out of an upstream warning and never tested. Turning flash attention on took decode at 56k tokens of context from 9.33 to 20.73 tokens per second.

Finding it took three attempts, because the first two answers my own benchmark gave me were confidently wrong. One of them I had already written up as a result. Here is the whole thing, including the parts where I was the problem.

The benchmark gave me a clean, confident, wrong answer

The setup: poolside open-weighted Laguna-S-2.1, a 118B-total, roughly 8B-active coding model whose 48 layers mostly use a 512-token sliding window, which should make deep context cheap. I have a 128GB Strix Halo box that such a model ought to suit, and a dense 27B coder already running on it. So: does the new one win?

I ran a proper depth sweep against both: cold prefill on every request, three repetitions per depth with the spread reported, server-side timings, GPU power sampled during each request. Careful work. The decode variance came out at 0.02 tok/s.

And the result said Laguna degrades far worse with context than the dense 27B: it kept 40% of its shallow decode speed at 56k tokens, where the dense model kept 69%. The exact opposite of what the sliding-window design predicts. I wrote it up as an honest negative result, which felt appropriately humble and was completely wrong.

The tell was in the numbers I had already collected. Look at the run-to-run spread on each model:

Laguna decode stdev:  0.04  0.04  0.03  0.02  0.02
dense  decode stdev:  0.61  1.14  0.47  1.87

A 10x to 90x difference in variance is not noise, it is a fingerprint. Non-speculative decoding is deterministic. Speculative decoding wobbles, because throughput depends on how many drafted tokens got accepted that particular run. The dense model had speculative decoding on. Laguna did not. I was not comparing two models, I was comparing two configurations and calling it architecture.

Change one flag at a time

So I ran two isolating experiments, each changing exactly one thing.

Turning speculation off on the dense model showed it was worth 2.62x, 2.23x, and 2.08x decode at increasing depth. That was not a contribution to its lead. That was its lead. And the variance collapsed to 0.01, exactly as predicted.

Turning flash attention on for Laguna is the one that stung, because it had been off for a bad reason. The upstream Laguna work flagged flash attention as unstable on RDNA3.5, so I copied that caution into our serving config and never revisited it, despite the fact that the dense coder had been running with flash attention on the same GPU every day for weeks.

Decode retention at 56k went from 40% to 75.8%. The supposed architectural weakness was a flag.

I had also predicted the shape of this gain and got it backwards, which is the more interesting part. I expected flash attention to help prefill and barely touch decode, because its textbook win is avoiding materialisation of the N-by-N attention score matrix, and during decode there is no such matrix: you have one query token against N cached keys. That reasoning is correct and the conclusion drawn from it was not. Without flash attention, llama.cpp makes several passes over the KV cache per generated token. Flash attention fuses them into one. The KV read is not eliminated, but the number of passes over it collapses, and on a unified-memory APU where bandwidth is the scarce resource, that saving grows with the size of the cache. Hence a bigger win the deeper you go.

Flash attention is exact, but it reassociates the summation, so I did not take the speed on faith. Tool calls still parse, prose is still coherent, and a unique string planted in the middle of a long document was retrieved verbatim at 47,813 tokens of context. Fast and correct, verified separately.

The third answer

With both configurations honest, Laguna wins every cell I measured, on both axes, while still conceding the 2x that speculative decoding gives its opponent and that it does not yet have:

It is also 1.27x more efficient per watt at depth, which on a box that shares one power budget between CPU and GPU is the number that actually explains the rest.

The part before any of this worked

I skipped a step to get to the point. Before I could benchmark anything, the model would not call a tool at all.

Upstream llama.cpp loads Laguna fine but cannot parse its tool calls. Laguna emits poolside's native format, and the tool-call parser has no handler for it, so a perfectly good tool call arrives as raw text in the content field with finish_reason: stop:

<tool_call>read_file<arg_key>path</arg_key><arg_value>internal/controller/scheduling.go</arg_value></tool_call>

The model decided correctly and the plumbing threw the decision away. For an agent loop that is fatal, and it fails in the worst way: silently, looking like a model that is bad at tools.

The fix is not mine. TheTom's llama.cpp fork has a differential autoparser: rather than hand-writing a parser per model, it renders the model's chat template with varying inputs, diffs the outputs to work out where tool names and arguments begin and end, and generates a parser from that. It made Laguna's tool calls work without a single line of model-specific code on our side, and none of the rest of this post happens without it.

With one catch that cost me two attempts, and which rhymes with the rest of this post. The autoparser learns the format from the template, and the template poolside ships inside the released GGUF is missing the argument delimiters. So it learned the outer wrapper, never learned how arguments are encoded, and failed in a way indistinguishable from having no parser at all. The fork ships a corrected template; our runtime image now defaults to it so nobody else has to find that out.

The benchmark that cannot measure the thing

Laguna has its own speculative decoding path, DFlash, and poolside ships the drafter inside the same repo as the weights. I measured it and got a 2.1x slowdown. I nearly reported that as a bug.

Then I measured it again on a realistic request, real Go source in context with an instruction to modify it, and got 53.8% draft acceptance and a 1.15x speedup. Same model, same flags, same hardware. Two results, 2.4x apart, pointing in opposite directions.

My benchmark prompts are random text. That is deliberate: random prompts defeat llama.cpp's prompt cache, which is the only way to measure prefill honestly. But speculative decoding predicts the model's own output, and it only pays when that output is predictable. A coding model rewriting a file it was just shown is highly predictable. The same model responding to random letter strings is not, so every drafted token is rejected and you pay for the drafting anyway.

So the property that makes my harness good at measuring prefill makes it structurally incapable of measuring speculation. Both numbers are published, and the synthetic one is labelled as a negative control rather than a result. Had I stopped at the first measurement I would have filed a confident, well-evidenced, completely false bug report against someone else's project.

But can it do the work?

Tokens per second is not the question anyone actually cares about. So I pointed Foreman, LLMKube's agentic coding harness, at two real issues from our own backlog, with the model served on the Strix box.

The first was small and I specified it tightly: add a field, set it, test it. It converged in five and a half minutes and passed the gate. That is a fair test of following instructions and not much else, so the second one withheld the answer. The issue was a controller bug where a stale condition survived a state change. I told it the contract and told it that another code path already did this correctly, but deliberately did not say where.

It took 25 minutes. It found the fix site on its own, navigated into a different package, located the existing convention, and reproduced its exact reason string rather than inventing a second one, which was the entire point of the bug. It left the neighbouring code in the same switch alone. Two files, plus 200 lines of tests.

Both passed the full gate: format, vet, lint, the package test matrix including a 249-second envtest suite, codegen drift checks, and a bite check that reverts the production change and re-runs the new tests to confirm they fail without it. That last one matters most, because it is the difference between tests that pass and tests that mean something.

Both are open as pull requests, written entirely by a model running on hardware in my house: #1272 and #1273.

The limits are worth stating plainly. Two issues. Both scoped by a human who wrote the tickets. On a codebase whose conventions are documented explicitly for exactly this purpose. That is evidence about maintenance work on a well-documented repository. It is not a claim about designing something new.

Every verdict was wrong at least once

The thread running through the whole day is that nothing self-reported could be trusted, including me.

The coder returned GO on a run whose gate then failed. The gate returned FAIL on a change that was completely correct, because an unrelated test was broken. And when I investigated that broken test and filed a bug explaining the root cause, my explanation was wrong, which I only discovered because I tried to reproduce the failure and it refused to fail.

That test turned out to be environment-dependent in a nasty way. It reused one command object across two cases, so a flag stayed marked as set and the required-flag check never fired. Locally it passed anyway, because the code fell through to an operation that a non-root user is not allowed to perform, so an error came back and the assertion was satisfied for entirely the wrong reason. In CI, running as root, that operation succeeded and the latent bug surfaced. A test that passes on your laptop and fails in CI, for a reason that has nothing to do with the change in flight.

None of these were caught by being careful. They were caught by insisting on ground truth: reading the actual diff instead of the verdict, checking the reason string against the source instead of the comment claiming to match it, and trying to reproduce a bug before explaining it.

If there is one thing to take from this, it is that a benchmark you have not tried to break is a benchmark that will eventually tell you something confident and false. Mine told me a model was slow when it was fast, and told me a feature was broken when my own prompts were the problem. Both times the numbers were reproducible to two decimal places.

The raw results, the harness, and the writeup with every caveat live in the open. The runtime image that makes Laguna's tool calls work is in llmkube-runtimes, and it now recommends flash attention on, with the numbers attached.

Credit where it is due: to poolside for open-weighting Laguna, and to TheTom, whose fork is the only reason any of this ran at all.