v0.6.0 Open Source · Kubernetes Native · vLLM + TGI + llama.cpp + Ollama

Run production LLMs
on your own hardware

We analyzed 200 GitHub issues with a 14B model on two $400 GPUs. Total cost: one cent. LLMKube makes self-hosted inference actually work.

Get Started View on GitHub Join Discord

See it in action

Deploy LLMs with any runtime in seconds using the llmkube CLI

terminal

$ llmkube deploy llama-3.1-8b --gpu --runtime vllm

🚀 Deploying LLM inference service ═══════════════════════════════════════════════ Name: llama-3.1-8b Runtime: vllm Accelerator: cuda GPU: 2 x nvidia 📦 Creating Model 'llama-3.1-8b'... ✅ Model created ⚙️ Creating InferenceService 'llama-3.1-8b'... ✅ InferenceService created (runtime: vllm)

Step 1/4: Deploy with vLLM runtime

Why LLMKube?

Local LLMs are great for prototyping. Scaling them for a team is where it gets hard.

The scaling problem

× Silent failures with no alerts
× Multi-GPU memory math by trial and error
× Updates that break your setup
× Docker Compose that doesn't scale
× One person managing everything
× Every machine set up by hand

With LLMKube

Pluggable runtimes: vLLM, TGI, llama.cpp, or bring your own
HPA autoscaling that responds to real inference metrics
GPU layer offloading with custom sharding splits
Infrastructure as code, not scripts and duct tape
Grafana dashboards for inference metrics out of the box
CUDA 13 and NVIDIA Blackwell GPU support

vLLM for speed. TGI for flexibility. llama.cpp for efficiency. LLMKube for all of them.

One operator, every runtime. The platform layer your inference stack is missing.

Deploy an LLM in seconds

Simple, declarative YAML that feels native to Kubernetes developers

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: phi-3-mini
spec:
  source: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf
  format: gguf
  quantization: Q4_K_M
  hardware:
    accelerator: cuda
    gpu:
      enabled: true
      count: 1
  resources:
    cpu: "2"
    memory: "4Gi"

Supports GGUF models from HuggingFace, with automatic download and caching

Limited to 10 Teams

Early Adopter Program

Help shape the future of LLMKube and get direct access to the maintainer.

What you get

Private Discord with other early adopters
Direct input on the roadmap
Your logo on our website (when ready)
Early access to new features

What we need

Real-world feedback on your use case
30 minutes monthly for a feedback call
Permission to share your story (anonymized if needed)

Apply to join

Ready to deploy your first LLM?

Join the community of developers deploying LLMs on Kubernetes.

Get Started on GitHub Read the Docs

Open source and free forever

Run production LLMs on your own hardware