Two machines, two runtimes, one model key: 1.85x throughput from a heterogeneous local fleet

I have two very different machines that can run the same model. An AMD Strix Halo mini-PC serving GGUF through llama.cpp on Vulkan, and an M5 Max serving MLX through Metal. The obvious question is whether pointing both at one model name buys anything real, or whether it just moves work around. I measured it, including the control that would have embarrassed me if I had skipped it. Eight concurrent requests: 55.4 seconds on one box, 30.0 seconds across both. The more interesting number is that median latency improved more than throughput did.

The setup

One ModelRouter rule, model key coder-fusion, weighted strategy, both backends at weight 50. A client sends one model name and never learns that two machines answered.

rules:
  - name: coder-fusion
    match:  { models: ["coder-fusion"] }
    route:
      strategy: weighted
      backends: ["fusion-strix-coder", "fusion-m5-mlx"]

The two backends have almost nothing in common except the weights they were trained from:

Different vendor, different accelerator, different inference engine, different quantization format. If heterogeneous fleets work at all, they should work here.

The numbers

Eight requests at concurrency eight, max_tokens: 200, identical prompt with a per-request variant suffix so nothing gets served from a prompt cache.

1.85x wall clock. 1.90x aggregate throughput. 2.66x better median latency. Zero failures in any arm.

Arm C is the whole post

Arm A ran the Strix with a single slot, which invites an obvious and fair objection: you did not beat one machine, you beat one badly configured machine. Add a second slot to the single box and maybe the gap disappears.

So I ran that. Arm C gives the lone Strix two slots, the same concurrency the two-machine setup has. It came out slightly worse: 57.9 seconds against 55.4, and 27.7 tokens per second against 28.0.

That is the result that makes the rest of it mean something. More concurrency on one box bought nothing, because the box was already saturated. Decode on a 27B model is memory-bandwidth-bound, not slot-bound. A second slot splits the same bandwidth two ways and adds scheduling overhead on top. It also costs you context: in llama.cpp --ctx-size is the total across slots, so two slots halved each one from 262144 to 131072 tokens.

The second machine brings its own memory bandwidth. The second slot does not. That distinction is the entire argument for a fleet over a bigger single box, and it is worth measuring rather than assuming.

The number people will under-report

Throughput makes the headline, but median latency fell from 30.1 seconds to 11.3, a 2.66x improvement. It improved more than throughput did, and that is not a rounding artifact. The dominant cost was queueing. With one box and eight concurrent requests, the median request waits behind three others before it starts. Splitting the queue shortens everyone's wait, not just the batch's.

Earlier the same day I sent a trivial "count from 1 to 25" request through the same endpoint while a long generation was in flight. First token arrived after 232 seconds. Every one of those seconds was queue.

What this does not do

This is request-level parallelism, and most writing about distributed inference is fuzzy on that point in a way that sets people up for disappointment. Concretely:

• Your single interactive coding session is exactly as fast as before. One request at a time means no queue to split.
• One long generation is not faster. A sixteen-minute single-file build takes sixteen minutes on either setup.
• Splitting one model across two machines would be slower, not faster. Token N+1 depends on token N, so the sequential chain cannot be parallelized. Layer-splitting across machines (pipeline parallelism) buys you capacity to run a model too big for one box, at the cost of a network hop per token. It is a different tool for a different problem.

Where it actually pays

Everything below has concurrent requests, which is the only thing that matters here:

• Agentic batch runs. My overnight harness works issues strictly serially against one endpoint, three to twelve minutes each. Two backends halve an eight-issue sweep.
• Subagent fan-out. When a coding agent dispatches parallel subagents, they all hit one endpoint and queue behind each other.
• Coder and reviewer at once. A review of branch N can run while the coder starts N+1, instead of contending for the same GPU.
• More than one person or laptop. Today the second person to press enter waits for the first to finish.
• Rolling updates. Drain one backend, swap its model or upgrade its runtime, keep serving from the other.
• Failover. Switch the strategy to primary-fallback and a dead box degrades throughput instead of ending service.
• Offline generation. Benchmark sweeps and synthetic dataset production are embarrassingly parallel and completely latency-insensitive.

The enterprise shape is the same problem with more zeros: route across tiers, keep sensitive traffic on local backends, fail over between nodes, and present one model name to every internal consumer.

The plumbing was harder than the measurement

Getting traffic to an out-of-cluster backend surfaced four separate defects, each one hidden behind the last, and the debugging lesson is more broadly useful than the benchmark:

1. External backends were rejected outright by the gateway data plane. Fixed.
2. A failed reconcile kept serving the previously compiled routes with no signal at the request path, so 12 of 12 requests returned HTTP 200 while an entire backend sat idle. Fixed: reconcile failures now raise a warning event that says explicitly that stale routes are still live.
3. The upstream model name was never rewritten, so every request that did arrive was rejected with a 404.
4. The gateway version in use applies neither of the two fields that are supposed to rewrite it, proven by capturing the request at the upstream: both the JSON body and the gateway's own model header still carried the router key.

Two of my four fixes were correct. The other two emitted perfectly reasonable-looking configuration that the data plane silently ignored, and I merged both before checking. I reverted them, because dead code that looks like it handles the problem is worse than no code at all: the next person to debug this would have seen the field, assumed it worked, and looked somewhere else.

The method that actually worked, every time: count requests at the upstream servers, not at the client and not at the router. A router that silently sends everything to one backend is indistinguishable from a working one if you only look at response codes. A CRD accepting a field does not mean the data plane applies it. The three-minute logging proxy I should have written first is what ended the guessing.

Caveats, stated plainly

• One run per arm, eight requests each. The deltas are large enough to survive that, but this is not a rigorous benchmark and I am not presenting it as one.
• The backends are not equivalent. Different quantizations on different runtimes. Earlier agentic testing found their behavior indistinguishable on tool calls, multi-turn loops and exact-anchor edits, so mixing them is defensible, but responses are not bit-identical across backends.
• 50/50 is almost certainly not optimal. The M5 prefills at roughly 773 tokens per second against the Strix's 294, while decode is a near tie at 29.7 versus 30.1. An even split ignores a 2.6x asymmetry. Weighting toward the M5 for prefill-heavy work is the obvious next experiment and it is untested.

The takeaway

A consumer AMD mini-PC and a Mac laptop, running different inference engines on different silicon with different quantizations, served the same model behind one name and nearly doubled concurrent throughput. No cloud, no matched hardware, no rewriting anything to speak the same dialect.

The honest version of the claim is narrower than the headline number: this buys concurrency and availability, not single-stream speed. But concurrency is what agentic workloads consume, and most people evaluating a local fleet are measuring the wrong thing when they benchmark one request at a time.