llmkube · v0.10.14 · apache 2.0

Write a few lines of YAML. Talk to a model on your own hardware.

LLMKube is a Kubernetes operator for inference. The same spec runs on a Mac, an NVIDIA box or an AMD box, and you get an OpenAI-compatible endpoint when the model is ready.

Try it on your Mac Star on GitHub

one model, three kinds of hardware: only the hardware block changes

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: qwen38-27b
spec:
  source: hf://ggml-org/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q4_K_M.gguf
  format: gguf
  quantization: Q4_K_M
  hardware:
    accelerator: metal
---
kind: InferenceService
metadata:
  name: qwen38-27b
spec:
  modelRef: qwen38-27b

$ kubectl get inferenceservice qwen38-27b
NAME         PHASE
qwen38-27b   Ready
$ curl qwen38-27b:8080/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}'
"Hi! I'm running on the Mac in your cluster."

your mac is a node

Your Mac's GPU, in the same cluster as everything else.

Apple Silicon engines run as native macOS processes with full Metal GPU access, not in containers. LLMKube supervises them and puts them behind an ordinary Kubernetes Service, so kubectl sees them like any other workload.

llama-server · mlx-server · vllm-swift · tensorfold · omlx · ollama
agent binaries are signed and notarized from 0.10.14

Create a join string where you have cluster access
llmkube metal token create mac-studio
Paste the line it prints on the Mac
llmkube metal join llmkube://…
Deploy with accelerator: metal
The Mac never sees your cluster credentials. The join string is single use and expires in 30 minutes.

what the operator does for you

You write the spec. It handles the rest.

Downloads and caches the model

From Hugging Face, S3, OCI or a URL, cached so the next pod starts fast.

Schedules it on the right GPU

NVIDIA, AMD, Intel and Apple Silicon under one set of CRDs.

Serves an OpenAI-compatible API

Works with the OpenAI SDKs and the tools you already use.

Picks the engine you want

llama.cpp, vLLM, SGLang, TGI and the Mac engines. Twelve runtimes, one field.

Scales on inference metrics

Autoscaling on requests in flight, not CPU.

Routes between local and hosted

ModelRouter sends traffic to your models or to Anthropic and OpenAI, and fails closed for data that must stay local.

measured in the lab

Real machines, real numbers, written up in full.

227 tok/s

Qwen3.8-27B on a MacBook M5 Max, served through a Kubernetes Service.

The TensorFold write-up

3 × GB10

The 552B DeepSeek V4.1 Flash (1017) served by vLLM across three DGX Sparks, with a 512K context.

The three-Sparks write-up

41 → 3.5 min

Time to first token on a 250K-token prompt, Qwen3.8 Flash Next on a Strix Halo, after switching engines.

The Strix Halo write-up

try it

Four commands from nothing to a running model.

Any Kubernetes cluster works, including kind or Docker Desktop on your laptop. No cloud account.

Read the quickstart Join the Discord

# the CLI
brew install defilantech/tap/llmkube
# the operator
helm repo add llmkube https://defilantech.github.io/LLMKube
helm install llmkube llmkube/llmkube -n llmkube-system --create-namespace
# a model, with catalog defaults
llmkube deploy phi-4-mini

built in the open

Apache 2.0, with humans and agents in the commit log.

LLMKube is built by a community of contributors, and by Foreman, an agentic harness that runs coding agents on local models across the same fleet. Its pull requests go through the same review as everyone else's, and it has shipped real fixes to this repo.

every claim on this page links to a post, a PR or a doc

Built by 18 contributors and one agent.