KubeCon, a Raspberry Pi, and five first-time contributors

My talk at KubeCon in Salt Lake City was about turning Apple Silicon Macs into GPU nodes for Kubernetes. The moment I keep coming back to happened after it. Someone from the audience walked up and asked whether LLMKube could run on a Raspberry Pi. He was super excited. He wanted to build something he hadn't known he could do, and that excitement was the part that made me happiest.

The talk was "Turning Apple Silicon into Production GPU Nodes for LLM Inference on Kubernetes," and it was a joy to give. But the talk was the part I prepared for. The conversations around it are the part I'll remember.

Christian was already building it

Here's the best part. What he was asking for already existed as a work in progress, and someone else was building it.

On the Sunday before my talk, Christian Nuss filed an issue to run the LLMKube agent on a Raspberry Pi 5 with the AI HAT+ 2, so small models can run on the Pi's Hailo NPU right next to the Macs in a cluster. He had already tested it on a real Pi, with real answers coming back from a model on the HAT.

The agent was written for macOS, so Christian went looking for every place that assumption lived. Three of his Linux pull requests shipped in v0.10.18, four days after the talk. On Linux, the agent now reads host memory from /proc and cgroup v2. It keeps its model store and state under XDG_DATA_HOME instead of macOS paths. And it no longer requires llama-server when you pick a different engine. The last pieces, getting the agent past its platform check and making llmkube metal join work on Linux, are still in progress.

So one person had the idea, and another was already turning it into code. That's the thing about building in the open that I never get tired of.

The hallway was full of use cases I hadn't thought of

The Raspberry Pi wasn't the only surprise. People kept stopping me in the hallway with ways of using LLMKube that had never crossed my mind, and it was a blast to talk shop with every one of them. Some wanted to know how to get started. A few wanted to know how to contribute. Seeing that much excitement about something I started was a joy.

Containers are changing how I think about Macs

I also got to talk with some of the amazing folks at Apple who brought containers to the main stage. We talked about the future of LLMKube in Apple's ecosystem, and what containers make possible there. I came home with a lot to think about.

Some of it you can try today. The one-Mac quickstart now creates its Kubernetes cluster with Apple's container tool. A Mac on macOS 26 or later goes from nothing to serving a model with Homebrew, Apple's own tooling and the Metal agent, and no Docker runtime. Older Macs still use minikube.

Five people sent their first pull request in v0.10.18

The excitement wasn't only in the hallway. Seven contributors from outside the maintainer team wrote 17 of v0.10.18's 33 pull requests, not counting automated dependency bumps. For five of them, it was their first pull request to LLMKube. Here's who built it:

Christian Nuss: the Linux groundwork above, router readiness that can wait for the service mesh, client IPs that survive the Mac relay, and a kustomize fix for installing the FederatedCluster CRD.
Aditya Rai: running engines kept through an agent restart even when they miss their first health check, a quota check that no longer trips on a brand-new namespace, and a warning when a model file pattern matches nothing.
Cindy Singleton, first PR: memory prep on each multi-node member before it boots, and a corrected default context for Qwen 2.5 14B.
Gabriel Cosi, first PR: a model cache size per InferenceService, and Prometheus relabelings in the Helm chart.
Abhinav Dhawan, first PR: the right default context for Llama 3.3 70B.
Noah Khomer, first PR: the right default context for Qwen 2.5 Coder 7B.
Sai Ram, first PR: Mac endpoints that no longer point service meshes and observability tools at a Pod that doesn't exist.

Every one of those is someone who found something worth fixing and saw it through review. I don't take that for granted. The full list is in the v0.10.18 release notes.

Your first pull request could be in the next one

If you came up to me in Salt Lake City and said you'd like to contribute, this is your invitation. If you didn't, it's yours too. Here's how to start:

Read CONTRIBUTING.md. It covers the build, the tests and how pull requests get reviewed.
Look through the open issues, or open one for the thing that bugs you about running models on your own hardware.
Say hello in the Discord. If you want help getting your first pull request through review, ask me. I'll help.

Thank you to everyone who stopped me in a hallway, asked a question after the talk, or sent a pull request.