Skip to content

Blog

Insights, tutorials, and updates from the LLMKube team

Engineering 7 min read

The best part of LLMKube 0.9.0 is code I did not write

For two months this blog has argued that the hard part of a self-hosted coding agent is the harness, not the model. LLMKube 0.9.0 is where that harness, Foreman, stopped being something I build alone. The two changes I am proudest of were written by Jory, a contributor I have never met: he found our fleet scheduler funneling every task onto one node while the rest sat idle, and 23 orphaned nodes leaking on a live cluster, and fixed both correctly, including the concurrency subtleties my review caught. This is a post about the release, about why the partnership is the real story, and an open invitation to help build the parts you care about.

Christopher Maher
Christopher Maher
Read more
Engineering 12 min read

A local model opened 41 of our pull requests in five weeks. The model is the least interesting part.

Between May 21 and June 25, a fleet of local models running on a Mac and an AMD mini-PC opened 41 pull requests that we merged into LLMKube itself, at zero cloud-inference cost. The models are a coin flip on any given run. This is a case study in why that does not matter: the harness, not the model, is what makes a stochastic local coder trustworthy enough to leave running overnight. With the numbers, three times it went wrong, why Uber burned its whole AI budget in four months, and why the home lab crowd figured this out first.

Christopher Maher
Christopher Maher
Read more
Engineering 8 min read

A 27B model on an AMD mini-PC fixed a bug in our operator. Then it overreached.

A 27B local coder on a consumer AMD Strix box fixed a real bug in the LLMKube operator: a hardcoded 60-second gateway timeout that silently capped every request. The harness gate verified it compiled and passed, but the model also overreached and wrote two tests for unrelated code; review caught that and trimmed them. Verified end-to-end on hardware I own: a request that used to die at exactly 60 seconds now runs 128 seconds. The lesson is the same one this whole project is built on: trust the harness, not the model.

Christopher Maher
Christopher Maher
Read more
Engineering 10 min read

Trust the harness, not the model: a weekend of local agents building their own guardrails

A local 27B coder is a coin flip. The bet behind LLMKube's Foreman was never a better model, it was a better harness, and this weekend tested that bet harder than any benchmark could. The harness spent the weekend building its own guardrails (a scope guard, a reviewer rubric, a bite check), and the most instructive moment was when it produced the exact bug the bite check exists to catch, and the review caught it. Plus: the same model running on an AMD Strix and an Apple Silicon Mac, an Apple node that quietly out-prefilled the dedicated GPU box, and four clean PRs from three contributors who showed up while the machines worked.

Christopher Maher
Christopher Maher
Read more
Engineering 9 min read

Making a fleet of self-hosted LLM agents trustworthy

Running one local LLM node is easy. Running a fleet of them, off-cluster, and trusting it to stay current and stay honest, is the hard part. This is the work that got LLMKube there: declarative, health-gated self-update for off-cluster agents (helm and brew for the edge), liveness and admission validation so dead or malformed nodes cannot lie to the control plane, and a real end-to-end test. Plus the build-in-public part: the bugs our own dogfooding caught that the unit tests could not, including a self-update path that had quietly disabled itself in production.

Christopher Maher
Christopher Maher
Read more
Engineering 11 min read

My Mac kernel-panicked, so I taught my code reviewer to stop trusting itself: 48 hours of local-model review integrity

A wired-memory kernel panic, two enforcement rails, and the strangest model evaluation we have ever run. LLMKube 0.8.2 and 0.8.3 are the releases where we stopped trying to find a local model with frontier review judgment, and built a system that does not need one. Inside: why reviewer judgment is stochastic (our production reviewer caught our hardest scope-drift trap in May and missed it twice in June), the deterministic checks that out-reviewed every model, the first local model to ever catch that trap on its own, and the seven harness bugs it took to get there. Fully local, zero cloud inference.

Christopher Maher
Christopher Maher
Read more
Benchmarks 10 min read

Back to Shadowstack: a 35B at 256K context (and 512K with YaRN) on two consumer Blackwell cards

We've been deep on Apple Silicon for a couple of months; this weekend I turned back to Shadowstack (two consumer RTX 5060 Ti) to see what context thresholds we could push. With TheTom's TurboQuant KV cache built for CUDA Blackwell (sm_120), Qwen3.6-35B-A3B holds its full 256K native context, and 512K via YaRN, on the same 32 GB where standard f16 KV runs out of memory short of native. Inside: perplexity and KL divergence (the numbers the community asked for), throughput measured at depth (turbo isn't free, and I correct an earlier claim), a needle-in-a-haystack to ~409K, and the honest open items.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

Introducing Foreman: a Kubernetes-native orchestrator for your local LLM fleet (LLMKube 0.8.0)

LLMKube 0.8.0 ships Foreman, an opt-in add-on that dispatches agentic workloads (coder, verifier, reviewer) across a heterogeneous fleet of locally-hosted LLM nodes. Foreman authored its own debut PRs against this repository, surfaced real reviewer-quality bugs along the way that we shipped fixes for, and now slots into your existing K8s patterns. Plus Intel GPU support from a first-time contributor, and a few honest notes on what v0.1 doesn't yet do.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.9: a new mlx-server runtime for Apple Silicon, four bugs the autoscaling tutorial flushed out, and kubectl scale support

0.7.9 adds mlx-server as a first-class runtime on the metal-agent: an OpenAI-compatible MLX inference server you select with --runtime mlx-server. We dogfooded it serving Qwen3.6-35B-A3B-8bit to opencode on an M5 Max, fixed four real bugs that surfaced while building toward a metrics-driven autoscaling tutorial (a dead PodMonitor selector, the operator fighting the HPA, the Metal-path InferenceService never going Ready, and a skipped memory pre-flight), and landed a Kubernetes scale subresource so kubectl scale works on InferenceService. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.8: ModelRouter Phase 1, fail-closed PII routing, and a hybrid local + cloud agentic story

0.7.8 lands ModelRouter Phase 1: a single OpenAI-compatible endpoint that dispatches across local InferenceServices and external providers (Anthropic, OpenAI, LiteLLM, Bedrock, Vertex), with fail-closed semantics for regulated data, per-rule and per-backend timeouts, half-open circuit breaker, streaming SSE passthrough, and a structured audit log per request. Plus the supporting fixes that made this release ship-ready, three new docs guides, and an honest list of Phase 1 limitations. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Releases 7 min read

What we shipped in LLMKube 0.7.7: OpenShift first-class, vllm-swift + TurboQuant, and a community-shipped Longhorn fix

0.7.7 makes OpenShift a first-class deploy target, lands the vllm-swift runtime with TurboQuant KV cache passthrough on Apple Silicon, picks up two community-driven changes (vLLM tuning fields from an engineer in France, plus a Longhorn FSGroup fix from a user who filed the cleanest bug report of the year), and adds enough observability glue to make multi-runtime fleets legible. Here's what landed and the story behind it.

Christopher Maher
Christopher Maher
Read more
Releases 8 min read

What we shipped in LLMKube 0.7.6: memory-pressure protection, mutable modelRef, and a community PR worth celebrating

0.7.6 is the biggest LLMKube release since multi-GPU sharding landed. Memory-pressure protection on the metal-agent (priority-based eviction with a friendly-fire guard), modelRef finally mutable, ParallelSlots extended to vLLM thanks to a polished community PR from @Faylixe, three new K8s-native pod fields (runtimeClassName, podAnnotations, podLabels), a real CNCF-style docs site, plus a quickstart-killer caught and fixed Saturday night. Here's what landed.

Christopher Maher
Christopher Maher
Read more
Benchmarks 16 min read

vllm-swift on M5 Max: A/B'ing TurboQuant+ against the llama.cpp data

TheTom asked us to run his vllm-swift TurboQuant+ work through the same kind of sweep we did on the llama.cpp fork. 36 cells, then a deep-context follow-up out to 192K. fp16 wins per-seq decode at every cell where it runs, but hits the memory ceiling at d=128K B=32 and d=192K B=32. turbo4v2 runs both: 1,360 tok/s and 1,024 tok/s aggregate. That is the value-prop confirmation: TurboQuant+ on this engine on this hardware is a memory-ceiling tool, not a throughput accelerator. Honest numbers below.

Christopher Maher
Christopher Maher
Read more
Benchmarks 11 min read

TurboQuant on a MacBook Pro, part 2: perplexity, KL divergence, and asymmetric K/V on M5 Max

Followup to the M5 Max long-context post. Comments asked for perplexity, KL divergence, asymmetric K/V combos, and a 64K data point. Overnight bench delivered all four. q8_0 KV is essentially free at 4k context (KL 0.0016, top-1 token agreement 98.6%). -ctk q8_0 -ctv turbo4 matches symmetric q8_0 throughput and fits 512K where symmetric q8_0 OOM'd. -ctk f16 -ctv turbo4 hits a Metal kernel fallback and craters 78x at 128K.

Christopher Maher
Christopher Maher
Read more
Benchmarks 10 min read

TurboQuant on a MacBook Pro: two findings the upstream discussion missed

Built TheTom's TurboQuant fork of llama.cpp for Metal, ran the bench overnight on M5 Max, and surfaced two findings the upstream community thread didn't have. First: at 128K+ context, turbo3 (3-bit KV) beats q8_0 (8-bit KV) on prompt processing. Second: turbo3 and turbo4 split by phase, turbo3 wins prefill, turbo4 wins decode at long context. Plus 1M context for batch coding workloads on a MacBook, and two PRs back to LLMKube to make TurboQuant first-class on the InferenceService CRD.

Christopher Maher
Christopher Maher
Read more
Benchmarks 12 min read

62.2% on Aider Polyglot from a MacBook Pro. Then the other model we tried scored 4%. Here's what actually happened, with a working cost loop attached.

Qwen3.6-35B-A3B Q8 on a MacBook Pro M5 Max scored 62.2% on Aider Polyglot (n=225/225), beating Claude Sonnet 4 with 32k thinking, o1-high, and DeepSeek R1 on the official leaderboard. Then Devstral 2 scored 4% on the same harness but 81.7% on HumanEval+: same model, 20× swing, benchmark numbers don't transfer. Plus the InferCost Apple Silicon collector that landed today, validating live cost-per-token attribution end to end with sub-watt agreement to the agent gauge.

Christopher Maher
Christopher Maher
Read more
Engineering 8 min read

Why Qwen 3.6 Doesn't Need --cpu-moe (and Why Qwen3-Coder Does) on Dual 16GB

The --cpu-moe flag trades VRAM savings for CPU compute cost per token. On dual RTX 5060 Ti cards that trade is required to run Qwen3-Coder-30B at all, but pure overhead for Qwen 3.6-35B-A3B, whose DeltaNet attention keeps the KV cache small enough that the model already fits in VRAM. Same hardware, same flag, opposite correct answers. Plus what shipped in LLMKube 0.7.0 because of the thread that surfaced this.

Christopher Maher
Christopher Maher
Read more
LLMKube LLMKube

Kubernetes for Local LLMs. Deploy, manage, and scale AI inference workloads with production-grade orchestration.

© 2026 Defilan Technologies LLC

Community

Built for the Kubernetes and AI communities

LLMKube is not affiliated with or endorsed by the Cloud Native Computing Foundation or the Kubernetes project. Kubernetes® is a registered trademark of The Linux Foundation.