Skip to content

Blog

Insights, tutorials, and updates from the LLMKube team

42 posts

Engineering 12 min read

Trying a new engine on the Strix Halo: 41 minutes to first token became 3.5

My Strix Halo was serving Qwen3.8 Flash Next on llama.cpp at a quarter of its context window. I tried Gufo, a young MIT engine for AMD, and a 250K-token prompt went from 41 minutes to first token to 3.5. Then my agent sessions started re-reading their whole history every turn. The roadblocks in the order I hit them, the upstream fix that solved the second one, and a guide for the next person with a Strix Halo.

Christopher Maher
Christopher Maher
Read more
Engineering 12 min read

A weekend with TensorFold on a MacBook: the engine mattered, the quant did not

I wanted to know if a 27B model could be fast enough on a MacBook to do real agentic coding, served through Kubernetes. The answer was a new engine, TensorFold, at 227 tok/s behind a Kubernetes Service with byte-identical output. The weekend also took down a production model, turned up a regression hiding since June, and killed my plan to build custom quants. Everything in the order it happened.

Christopher Maher
Christopher Maher
Read more
Engineering 12 min read

Every release makes the harness harder to fool: LLMKube 0.9.19

We found nine fully-tested rails on main with zero production callers, reviews approved by an agent running on an empty system prompt, and safety rails that silently skipped when their inputs went missing. 0.9.19 is what shipped after we pointed the harness's own skepticism at itself: a CI guard that makes tested-but-dead code impossible, verdict records that name what was not checked, a config floor for agents, and an execution-first reviewer rubric. Much of it was written by the agent fleet through the pipeline it improves.

Christopher Maher
Christopher Maher
Read more
Engineering 11 min read

Day one with a new model: Meta's Muse Glimmer on an AMD iGPU, hours after release

Meta shipped Muse Glimmer 30B and llama.cpp added support the same day. This is the whole path from release to serving on hardware nobody upstream tested: a pinned runtime rebuild with an architecture guard, one preload into object storage, one Model CR carrying weights plus a vision projector plus a draft model. Inside: what the numbers are, the two serving traps that silently produce wrong ones, the benchmark design that gave me a fake 9% result before I caught it, and the two operator bugs a day-one bring-up shook out.

Christopher Maher
Christopher Maher
Read more
Engineering 13 min read

One GPU, four ways to share it: ten scenarios on a DGX Spark, and the headline finding I had to retract

When you have more models than GPUs, LLMKube gives you several answers: exclusive, time-slicing, ModelPool swapping, and an in-process router. I measured all of them across ten scenarios. Time-slicing is free while a co-tenant is idle and collapses prefill 16x when it is not. I also published a headline finding that duplicate models are nearly free on unified memory, then failed to replicate it and retracted it. Inside: what the controlled replication actually showed, the full decision table, the measurement traps that produced wrong answers first, and the three real bugs that only a live workload found.

Christopher Maher
Christopher Maher
Read more
Engineering 9 min read

Two machines, two runtimes, one model key: 1.85x throughput from a heterogeneous local fleet

An AMD Strix Halo running llama.cpp on Vulkan and an M5 Max running MLX on Metal, serving the same model behind a single model name. Eight concurrent requests finished in 30 seconds instead of 55, aggregate throughput went from 28 to 53 tokens per second, and median latency dropped 2.66x. Inside: the fairness control that proves it is real capacity and not a misconfigured first box, the four routing defects it took to get there, and an honest account of what this does not speed up.

Christopher Maher
Christopher Maher
Read more
Engineering 6 min read

My local coder wrote a test that passed by hiding its own bug

Foreman, the local agentic harness that maintains LLMKube, fixed a bug in its own coder loop overnight. The gates were green, the tests passed, the verdict was GO. It was also wrong, in a way that is specific to AI-written code: the model wrote both the fix and the test, and the test passed precisely because its fixture was built to avoid the one input that triggers the bug. A green check from a test the coder wrote is not evidence of correctness. Here is the catch, the ten-line probe that proved it, and why every check in the harness exists because a green light lied once.

Christopher Maher
Christopher Maher
Read more
Engineering 13 min read

Foreman 101: agentic coding as Kubernetes resources

Foreman is an agentic coder that runs as Kubernetes CRDs. You describe work as a Workload, it decomposes into tasks, agents on your nodes pick them up, and a reviewed branch comes out the other end. This is the full walkthrough: the four objects you need to understand, an install from the public Helm charts, an agent backed by a hosted API or by a model on your own GPU, and the deterministic verifier that decides whether the work is acceptable.

Christopher Maher
Christopher Maher
Read more
Engineering 11 min read

A flag I never questioned was costing 2.2x

I put a new open-weight coding model on a 128GB Strix Halo box and benchmarked it against the coder I already run. The first answer was that the new model degraded badly at long context. That answer was wrong, and so was the next one. The culprit was a flag I had copied from an upstream caution and never tested: flash attention was off, and turning it on was worth 2.2x decode at 56k tokens. Along the way the coder's own GO verdict was wrong, the gate's FAIL was wrong, and my bug report was wrong. Then the model opened two pull requests that passed the full gate.

Christopher Maher
Christopher Maher
Read more
Engineering 9 min read

The harness started filing its own bugs

Two Foreman runs failed on the same issue and the root cause was embarrassing: the field where operators write instructions to the coder was never read by anything. This is the week the harness crossed a line, from a thing we debug to a thing that participates in its own debugging. It found the scoping bug, then fixed its own research-grounding bug, verified a fact the cheap way so nobody had to build the expensive thing, and shipped three features into the same release week: tiered GPU sharing, VRAM-governed quotas, and admission-time enforcement, validated live on hardware we own.

Christopher Maher
Christopher Maher
Read more
Engineering 7 min read

The best part of LLMKube 0.9.0 is code I did not write

For two months this blog has argued that the hard part of a self-hosted coding agent is the harness, not the model. LLMKube 0.9.0 is where that harness, Foreman, stopped being something I build alone. The two changes I am proudest of were written by Jory, a contributor I have never met: he found our fleet scheduler funneling every task onto one node while the rest sat idle, and 23 orphaned nodes leaking on a live cluster, and fixed both correctly, including the concurrency subtleties my review caught. This is a post about the release, about why the partnership is the real story, and an open invitation to help build the parts you care about.

Christopher Maher
Christopher Maher
Read more
Engineering 12 min read

A local model opened 41 of our pull requests in five weeks. The model is the least interesting part.

Between May 21 and June 25, a fleet of local models running on a Mac and an AMD mini-PC opened 41 pull requests that we merged into LLMKube itself, at zero cloud-inference cost. The models are a coin flip on any given run. This is a case study in why that does not matter: the harness, not the model, is what makes a stochastic local coder trustworthy enough to leave running overnight. With the numbers, three times it went wrong, why Uber burned its whole AI budget in four months, and why the home lab crowd figured this out first.

Christopher Maher
Christopher Maher
Read more
Engineering 8 min read

A 27B model on an AMD mini-PC fixed a bug in our operator. Then it overreached.

A 27B local coder on a consumer AMD Strix box fixed a real bug in the LLMKube operator: a hardcoded 60-second gateway timeout that silently capped every request. The harness gate verified it compiled and passed, but the model also overreached and wrote two tests for unrelated code; review caught that and trimmed them. Verified end-to-end on hardware I own: a request that used to die at exactly 60 seconds now runs 128 seconds. The lesson is the same one this whole project is built on: trust the harness, not the model.

Christopher Maher
Christopher Maher
Read more
Engineering 10 min read

Trust the harness, not the model: a weekend of local agents building their own guardrails

A local 27B coder is a coin flip. The bet behind LLMKube's Foreman was never a better model, it was a better harness, and this weekend tested that bet harder than any benchmark could. The harness spent the weekend building its own guardrails (a scope guard, a reviewer rubric, a bite check), and the most instructive moment was when it produced the exact bug the bite check exists to catch, and the review caught it. Plus: the same model running on an AMD Strix and an Apple Silicon Mac, an Apple node that quietly out-prefilled the dedicated GPU box, and four clean PRs from three contributors who showed up while the machines worked.

Christopher Maher
Christopher Maher
Read more
Engineering 9 min read

Making a fleet of self-hosted LLM agents trustworthy

Running one local LLM node is easy. Running a fleet of them, off-cluster, and trusting it to stay current and stay honest, is the hard part. This is the work that got LLMKube there: declarative, health-gated self-update for off-cluster agents (helm and brew for the edge), liveness and admission validation so dead or malformed nodes cannot lie to the control plane, and a real end-to-end test. Plus the build-in-public part: the bugs our own dogfooding caught that the unit tests could not, including a self-update path that had quietly disabled itself in production.

Christopher Maher
Christopher Maher
Read more
Engineering 11 min read

My Mac kernel-panicked, so I taught my code reviewer to stop trusting itself: 48 hours of local-model review integrity

A wired-memory kernel panic, two enforcement rails, and the strangest model evaluation we have ever run. LLMKube 0.8.2 and 0.8.3 are the releases where we stopped trying to find a local model with frontier review judgment, and built a system that does not need one. Inside: why reviewer judgment is stochastic (our production reviewer caught our hardest scope-drift trap in May and missed it twice in June), the deterministic checks that out-reviewed every model, the first local model to ever catch that trap on its own, and the seven harness bugs it took to get there. Fully local, zero cloud inference.

Christopher Maher
Christopher Maher
Read more
Benchmarks 10 min read

Back to Shadowstack: a 35B at 256K context (and 512K with YaRN) on two consumer Blackwell cards

We've been deep on Apple Silicon for a couple of months; this weekend I turned back to Shadowstack (two consumer RTX 5060 Ti) to see what context thresholds we could push. With TheTom's TurboQuant KV cache built for CUDA Blackwell (sm_120), Qwen3.6-35B-A3B holds its full 256K native context, and 512K via YaRN, on the same 32 GB where standard f16 KV runs out of memory short of native. Inside: perplexity and KL divergence (the numbers the community asked for), throughput measured at depth (turbo isn't free, and I correct an earlier claim), a needle-in-a-haystack to ~409K, and the honest open items.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

Introducing Foreman: a Kubernetes-native orchestrator for your local LLM fleet (LLMKube 0.8.0)

LLMKube 0.8.0 ships Foreman, an opt-in add-on that dispatches agentic workloads (coder, verifier, reviewer) across a heterogeneous fleet of locally-hosted LLM nodes. Foreman authored its own debut PRs against this repository, surfaced real reviewer-quality bugs along the way that we shipped fixes for, and now slots into your existing K8s patterns. Plus Intel GPU support from a first-time contributor, and a few honest notes on what v0.1 doesn't yet do.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.9: a new mlx-server runtime for Apple Silicon, four bugs the autoscaling tutorial flushed out, and kubectl scale support

0.7.9 adds mlx-server as a first-class runtime on the metal-agent: an OpenAI-compatible MLX inference server you select with --runtime mlx-server. We dogfooded it serving Qwen3.6-35B-A3B-8bit to opencode on an M5 Max, fixed four real bugs that surfaced while building toward a metrics-driven autoscaling tutorial (a dead PodMonitor selector, the operator fighting the HPA, the Metal-path InferenceService never going Ready, and a skipped memory pre-flight), and landed a Kubernetes scale subresource so kubectl scale works on InferenceService. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.8: ModelRouter Phase 1, fail-closed PII routing, and a hybrid local + cloud agentic story

0.7.8 lands ModelRouter Phase 1: a single OpenAI-compatible endpoint that dispatches across local InferenceServices and external providers (Anthropic, OpenAI, LiteLLM, Bedrock, Vertex), with fail-closed semantics for regulated data, per-rule and per-backend timeouts, half-open circuit breaker, streaming SSE passthrough, and a structured audit log per request. Plus the supporting fixes that made this release ship-ready, three new docs guides, and an honest list of Phase 1 limitations. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Releases 7 min read

What we shipped in LLMKube 0.7.7: OpenShift first-class, vllm-swift + TurboQuant, and a community-shipped Longhorn fix

0.7.7 makes OpenShift a first-class deploy target, lands the vllm-swift runtime with TurboQuant KV cache passthrough on Apple Silicon, picks up two community-driven changes (vLLM tuning fields from an engineer in France, plus a Longhorn FSGroup fix from a user who filed the cleanest bug report of the year), and adds enough observability glue to make multi-runtime fleets legible. Here's what landed and the story behind it.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.6: memory-pressure protection, mutable modelRef, and a community PR worth celebrating

0.7.6 is the biggest LLMKube release since multi-GPU sharding landed. Memory-pressure protection on the metal-agent (priority-based eviction with a friendly-fire guard), modelRef finally mutable, ParallelSlots extended to vLLM thanks to a polished community PR from @Faylixe, three new K8s-native pod fields (runtimeClassName, podAnnotations, podLabels), a real CNCF-style docs site, plus a quickstart-killer caught and fixed Saturday night. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Benchmarks 16 min read

vllm-swift on M5 Max: A/B'ing TurboQuant+ against the llama.cpp data

TheTom asked us to run his vllm-swift TurboQuant+ work through the same kind of sweep we did on the llama.cpp fork. 36 cells, then a deep-context follow-up out to 192K. fp16 wins per-seq decode at every cell where it runs, but hits the memory ceiling at d=128K B=32 and d=192K B=32. turbo4v2 runs both: 1,360 tok/s and 1,024 tok/s aggregate. That is the value-prop confirmation: TurboQuant+ on this engine on this hardware is a memory-ceiling tool, not a throughput accelerator. Honest numbers below.

Christopher Maher
Christopher Maher
Read more
Benchmarks 11 min read

TurboQuant on a MacBook Pro, part 2: perplexity, KL divergence, and asymmetric K/V on M5 Max

Followup to the M5 Max long-context post. Comments asked for perplexity, KL divergence, asymmetric K/V combos, and a 64K data point. Overnight bench delivered all four. q8_0 KV is essentially free at 4k context (KL 0.0016, top-1 token agreement 98.6%). -ctk q8_0 -ctv turbo4 matches symmetric q8_0 throughput and fits 512K where symmetric q8_0 OOM'd. -ctk f16 -ctv turbo4 hits a Metal kernel fallback and craters 78x at 128K.

Christopher Maher
Christopher Maher
Read more
Benchmarks 10 min read

TurboQuant on a MacBook Pro: two findings the upstream discussion missed

Built TheTom's TurboQuant fork of llama.cpp for Metal, ran the bench overnight on M5 Max, and surfaced two findings the upstream community thread didn't have. First: at 128K+ context, turbo3 (3-bit KV) beats q8_0 (8-bit KV) on prompt processing. Second: turbo3 and turbo4 split by phase, turbo3 wins prefill, turbo4 wins decode at long context. Plus 1M context for batch coding workloads on a MacBook, and two PRs back to LLMKube to make TurboQuant first-class on the InferenceService CRD.

Christopher Maher
Christopher Maher
Read more
Benchmarks 12 min read

62.2% on Aider Polyglot from a MacBook Pro. Then the other model we tried scored 4%. Here's what actually happened, with a working cost loop attached.

Qwen3.6-35B-A3B Q8 on a MacBook Pro M5 Max scored 62.2% on Aider Polyglot (n=225/225), beating Claude Sonnet 4 with 32k thinking, o1-high, and DeepSeek R1 on the official leaderboard. Then Devstral 2 scored 4% on the same harness but 81.7% on HumanEval+: same model, 20× swing, benchmark numbers don't transfer. Plus the InferCost Apple Silicon collector that landed today, validating live cost-per-token attribution end to end with sub-watt agreement to the agent gauge.

Christopher Maher
Christopher Maher
Read more
Engineering 8 min read

Why Qwen 3.6 Doesn't Need --cpu-moe (and Why Qwen3-Coder Does) on Dual 16GB

The --cpu-moe flag trades VRAM savings for CPU compute cost per token. On dual RTX 5060 Ti cards that trade is required to run Qwen3-Coder-30B at all, but pure overhead for Qwen 3.6-35B-A3B, whose DeltaNet attention keeps the KV cache small enough that the model already fits in VRAM. Same hardware, same flag, opposite correct answers. Plus what shipped in LLMKube 0.7.0 because of the thread that surfaced this.

Christopher Maher
Christopher Maher
Read more
LLMKube LLMKube

Kubernetes for Local LLMs. Deploy, manage, and scale AI inference workloads with production-grade orchestration.

© 2026 Defilan Technologies LLC

Community

Built for the Kubernetes and AI communities

LLMKube is not affiliated with or endorsed by the Cloud Native Computing Foundation or the Kubernetes project. Kubernetes® is a registered trademark of The Linux Foundation.