v0.9.5 Open Source · Kubernetes Native · NVIDIA + Apple Silicon + AMD · vLLM + llama.cpp + mlx-server

Run production LLMs
on your own hardware

A Kubernetes operator for self-hosted LLM inference: vLLM, llama.cpp, and TGI on NVIDIA, Apple Silicon, and AMD. Driven by Foreman, our agentic harness, local models on your own fleet open their own pull requests, reviewed and merged alongside human contributors.

Get Started View on GitHub

Star Join the Discord The best part of LLMKube 0.9.0 is code I did not write →

See it in action

Deploy LLMs with any runtime in seconds using the llmkube CLI

terminal

$ llmkube deploy llama-3.1-8b --gpu --runtime vllm

🚀 Deploying LLM inference service ═══════════════════════════════════════════════ Name: llama-3.1-8b Runtime: vllm Accelerator: cuda GPU: 2 x nvidia 📦 Creating Model 'llama-3.1-8b'... ✅ Model created ⚙️ Creating InferenceService 'llama-3.1-8b'... ✅ InferenceService created (runtime: vllm)

Step 1/4: Deploy with vLLM runtime

What's happening here

The best part of LLMKube 0.9.0 is code I did not write

For two months this blog has argued that the hard part of a self-hosted coding agent is the harness, not the model. LLMKube 0.9.0 is where…

Read

Engineering 12 min read

A local model opened 41 of our pull requests in five weeks. The model is the least interesting part.

Between May 21 and June 25, a fleet of local models running on a Mac and an AMD mini-PC opened 41 pull requests that we merged into LLMKube…

Read

Engineering 8 min read

A 27B model on an AMD mini-PC fixed a bug in our operator. Then it overreached.

A 27B local coder on a consumer AMD Strix box fixed a real bug in the LLMKube operator: a hardcoded 60-second gateway timeout that silently…

Read

All posts

Why LLMKube?

Local LLMs are great for prototyping. Scaling them for a team is where it gets hard.

The scaling problem

× Silent failures with no alerts
× Multi-GPU memory math by trial and error
× Updates that break your setup
× Docker Compose that doesn't scale
× One person managing everything
× Every machine set up by hand

With LLMKube

Pluggable runtimes: vLLM, TGI, llama.cpp, or bring your own
HPA autoscaling that responds to real inference metrics
GPU layer offloading with custom sharding splits
Infrastructure as code, not scripts and duct tape
Grafana dashboards for inference metrics out of the box
CUDA 13 and NVIDIA Blackwell GPU support
Agentic coding pipelines that run on the same fleet (Foreman)

vLLM for speed. TGI for flexibility. llama.cpp for efficiency. LLMKube for all of them.

One operator, every runtime. The platform layer your inference stack is missing.

Deploy an LLM in seconds

Simple, declarative YAML that feels native to Kubernetes developers

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: phi-3-mini
spec:
  source: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf
  format: gguf
  quantization: Q4_K_M
  hardware:
    accelerator: cuda
    gpu:
      enabled: true
      count: 1
  resources:
    cpu: "2"
    memory: "4Gi"

Supports GGUF models from HuggingFace, with automatic download and caching

Foreman

Agents that build on your fleet

Foreman is a Kubernetes-native control plane that dispatches coder, verifier, and reviewer agents across your heterogeneous fleet of local models. They fix issues, open pull requests, and gate their own work, on hardware you own.