Documentation

Run any GGUF, vLLM-compatible, or MLX model on your own Kubernetes clusters, with the same operational primitives as the rest of your fleet.

start here · 5 minutes Install LLMKube and serve a model Install the operator with Helm, deploy a model from the catalog, and hit an OpenAI-compatible endpoint on a fresh kind, Minikube, or Docker Desktop cluster. No cloud account required. Read the quickstart

concepts Architecture Controller in-cluster, agent on the Mac. How the pieces split. operations Memory-pressure protection The watchdog only evicts when LLMKube is the cause.

What LLMKube is for

LLMKube is a Kubernetes operator for self-hosted inference. Install it in your cluster, declare the models you want to serve as Model custom resources, and declare how to serve them as InferenceService custom resources. The operator handles model download, pod scheduling, health checks, metrics, and, on Apple Silicon, native process supervision.

Where it sits relative to other projects in this space:

Multi-runtime by design. One CRD, many backends. The runtime field on InferenceService selects llama.cpp (including its router mode), vLLM, SGLang, TGI, PersonaPlex, or generic; the metal-agent path adds llama-server, mlx-server, oMLX, vllm-swift, Ollama, and TensorFold for Apple Silicon. Mix runtimes on the same cluster without standing up a different operator for each.
Native Apple Silicon, not just NVIDIA. The metal-agent runs as a native macOS process and manages its engines directly on the host. The engines listen on loopback behind the agent's authenticated TLS ingress, and the controller fronts each Metal InferenceService with an in-cluster relay, so pods reach the Mac-served model through an ordinary Service. M-series boxes participate in the same control plane as your GPU nodes.
Single-tenant boxes and multi-GPU clusters. The same operator deploys to a Mac mini, a Minikube cluster, or a multi-node GPU cluster with layer-sharded models. No split between "edge" and "datacenter" tooling.
Kubernetes-native, not Kubernetes-adjacent. Everything is a CRD with proper status conditions, events, owner references, and Prometheus metrics. kubectl get, HPA, NetworkPolicy, and PriorityClass behave the way you expect.

Recent additions

Full changelog

Signed and notarized macOS binaries: Starting with 0.10.14, the macOS release binaries (the llmkube CLI, the metal-agent, and the Foreman agent) are signed with the project's Developer ID and notarized by Apple, so Gatekeeper accepts them on first launch.
PR #2233
Metal engines survive agent restarts: Engines started by the metal-agent keep serving when the agent stops, restarts, or upgrades, and the next agent adopts them on startup. llmkube-metal-agent stop-engines stops them on purpose. New in 0.10.13.
PR #2116
Join a Mac with one command: llmkube metal token create prints a join string on the admin side; llmkube metal join on the Mac turns it into a running metal-agent whose credential renews itself, and llmkube metal leave undoes it. The new MetalNode resource reports whether each Mac's agent is connected and whether a verifier pod in the cluster reached its ingress. New in 0.10.12.
PR #2246, #2236, #2235

In the operator repo

These topics are documented in the operator repo and being migrated to this site. Until they land here, the canonical reference is the markdown alongside the code.