Browse documentation

Documentation

Run any GGUF, vLLM-compatible, or MLX model on your own Kubernetes clusters, with the same operational primitives as the rest of your fleet.

Start here · 5 minutes

Install LLMKube and serve a model

Install the operator with Helm, deploy a model from the catalog, and hit an OpenAI-compatible endpoint on a fresh kind, Minikube, or Docker Desktop cluster. No cloud account required.

Read the quickstart

Concepts

Architecture

Controller in-cluster, agent on the Mac. How the pieces split.

Memory-pressure protection

The watchdog only evicts when LLMKube is the cause.

What LLMKube is for

LLMKube is a Kubernetes operator for self-hosted inference. Install it in your cluster, declare the models you want to serve as Model custom resources, and declare how to serve them as InferenceService custom resources. The operator handles model download, pod scheduling, health checks, metrics, and, on Apple Silicon, native process supervision.

Where it sits relative to other projects in this space:

Multi-runtime by design. One CRD, many backends. The runtime field on InferenceService selects llama.cpp, vLLM, SGLang, TGI, PersonaPlex, or generic; the metal-agent path adds llama-server, mlx-server, and vllm-swift for Apple Silicon. Mix runtimes on the same cluster without standing up a different operator for each.
Native Apple Silicon, not just NVIDIA. The metal-agent runs as a native macOS process, manages llama-server / mlx-server / vllm-swift lifecycles directly on the host, and registers Endpoints back into Kubernetes. M-series boxes participate in the same control plane as your GPU nodes.
Single-tenant boxes and multi-GPU clusters. The same operator deploys to a Mac mini, a Minikube cluster, or a multi-node GPU cluster with layer-sharded models. No split between "edge" and "datacenter" tooling.
Kubernetes-native, not Kubernetes-adjacent. Everything is a CRD with proper status conditions, events, owner references, and Prometheus metrics. kubectl get, HPA, NetworkPolicy, and PriorityClass behave the way you expect.

Recent additions

Full changelog →

Draft-model speculative decoding: spec.speculativeDecoding now accepts a companion drafter via draftModel or draftModelRef, alongside the existing built-in MTP path. Faster decode on models that ship a draft head, without hand-rolling llama.cpp flags. PR #1533, #1564
Architecture-aware runtime placement: The controller now constrains inference pods to nodes whose CPU architecture the runtime image actually supports, so an arm64 node in a mixed fleet no longer draws an amd64-only image and crash-loops. PR #1544
Actual GPU offload in status: status.acceleration reports the device the model landed on and how many layers were offloaded out of the total, so "is this really running on the GPU?" is answerable with kubectl get instead of pod logs. PR #1566

In the operator repo

These topics are documented in the operator repo and being migrated to this site. Until they land here, the canonical reference is the markdown alongside the code.