Production-Ready LLM Infrastructure

Stop the metal-agent from killing your only inference workload when system memory spikes. Priority-based eviction, a friendly-fire guard for legitimate primary consumers, a per-service opt-out, and (new in 0.7.7) Kubernetes events on every pressure transition that kubectl describe picks up.

• Watchdog levels with MemoryPressure status condition
• Eviction-safety floor: never evicts the last managed process
• evictionProtection: true per InferenceService
• 50% RSS friendly-fire guard against external pressure spikes
• Kubernetes events on pressure / eviction / respawn-blocked transitions

First-class support for fork-specific KV cache types like TurboQuant turbo3, turbo4 on Metal and turbo2 on vLLM. Up to ~6.4× KV cache compression vs f16, which is what unlocks 256K–1M context for agentic coding on a single MacBook.

• cacheTypeCustomK / cacheTypeCustomV for llama.cpp forks
• kvCacheCustomDtype for vLLM v0.20+ turbo2
• Cache-type-aware memory pre-flight check (no false OOM on TurboQuant)
• Spec-drift respawn: kubectl patch isvc picks up KV cache changes
• Cross-runtime: same CRD shape on Metal and CUDA

Live SoC power gauges (CPU + GPU + ANE) sourced from macOS powermetrics. Pairs with InferCost for end-to-end $/MTok cost attribution on M-series Macs, where DCGM doesn't exist.

• Four Prometheus gauges: combined / GPU / CPU / ANE watts
• Opt-in via --apple-power-enabled flag
• One-command make install-powermetrics-sudo with pinned-argv NOPASSWD entry
• Security audit fixed three findings before merge
• Agrees with InferCost reading within ~1.6 W under sustained load

Run 30B+ MoE models on consumer GPUs. Expert weights in system RAM, active path on GPU, scheduler-aware host memory requests.

• moeCPUOffload, moeCPULayers, noKvOffload fields
• hostMemory request for scheduler placement
• Tensor overrides and batch-size controls
• Seven new runtime controls for llama.cpp and vLLM
• HuggingFace repo ID source for runtime-resolved models

Choose the best inference engine for your workload. One CRD, seven runtimes (including the new mlx-server MLX path for Apple Silicon), and FP8-friendly tuning via gpuMemoryUtilization + cpuOffloadGB on vLLM.

• vLLM with PagedAttention, tensor parallelism, and per-rank CPU offload (community PR)
• vllm-swift for native Apple Silicon with TurboQuant passthrough
• mlx-server for OpenAI-compatible MLX inference on Apple Silicon
• TGI (HuggingFace Text Generation Inference)
• llama.cpp with GGUF and low-memory efficiency
• Generic runtime for custom containers

Scale inference replicas automatically based on real metrics. Per-runtime metric defaults.

• Kubernetes HPA with custom inference metrics
• Configurable min/max replicas and target values
• Per-runtime default metrics via HPAMetricProvider
• Works with Prometheus Adapter

Pre-built Grafana dashboard plus shipped Prometheus recording rules for p95 latency and TTFT across runtimes. The new inference.llmkube.dev/runtime pod label is promoted onto every series so group-by-runtime queries are free.

• Recording rules: p95 request latency, p95 TTFT, llama.cpp request latency
• Per-model, per-runtime, per-namespace breakdowns out of the box
• Import-ready JSON in docs/grafana/llmkube-inference.json
• PodMonitor relabelings ship in the Helm chart

NVIDIA CUDA 13 support with Blackwell GPUs. Custom layer splits via GPUShardingSpec.

• CUDA 13 with Blackwell and Qwen3.5 support
• Custom layer splits from GPUShardingSpec
• 64 tok/s on Llama 3.2 3B (17x faster than CPU)
• GKE + NVIDIA GPU Operator ready

Metal Agent

Three pluggable runtime backends for Apple Silicon inference. Choose the best fit for your workflow.

• Ollama backend (200K+ users, auto model download)
• oMLX backend (MLX-native, 40% faster)
• llama-server backend (direct llama.cpp control)
• M1/M2/M3/M4/M5 with health checks and metrics

KV Cache & Advanced Config

Fine-tune llama.cpp and vLLM behavior through the CRD. Standard cache types are enum-validated; fork-specific types pass straight through.

• cacheTypeK / cacheTypeV (llama.cpp standard enum)
• kvCacheDtype (vLLM auto / fp8_e5m2 / fp8_e4m3)
• cacheTypeCustomK/V and kvCacheCustomDtype for fork values
• extraArgs escape hatch for any runtime flag

GPU Queue Visibility

See where workloads stand in the GPU queue. Priority classes control scheduling.

• Real-time queue position in status
• GPU contention visibility
• Priority classes for scheduling control

GPU Monitoring

Prometheus and Grafana integration with DCGM GPU metrics and pre-built dashboards.

• GPU utilization, temp, power monitoring
• Pre-built Grafana dashboards
• Prometheus metrics integration

Custom Resource Definitions for Model and InferenceService. Works with kubectl, GitOps, and the rest of your platform stack.

• Mutable modelRef: swap models without recreating the service
• runtimeClassName for gVisor, Kata, or custom runtimes
• podAnnotations + podLabels for service mesh and policy
• Declarative YAML, GitOps-ready, standard K8s patterns

Automatic Model Management

Download from HuggingFace, HTTP, or PVC sources. Automatic caching and SHA256 validation.

• HuggingFace + HTTP + PVC sources
• GGUF format support
• Persistent volume caching

OpenAI Compatible

Drop-in replacement for OpenAI API. Use existing tools without code changes.

• /v1/chat/completions endpoint
• Streaming responses
• Compatible with LangChain, etc.

CLI Tool

Simple command-line interface for deploying and managing LLM workloads.

• Deploy with --gpu flag
• List, status, delete commands
• macOS, Linux, Windows binaries

CLI Benchmark Suites

Five test suites for comprehensive validation with automated sweeps and markdown reports.

• llmkube benchmark --suite quick
• Concurrency, context, and token sweeps
• Markdown reports for sharing

Grafana Dashboard

Pre-built GPU observability dashboard for monitoring utilization, temperature, and memory.

• Multi-GPU monitoring with DCGM
• Import-ready JSON in config/grafana/

Persistent Model Cache

Download models once, deploy instantly across services. Reduce bandwidth and startup times.

• Per-namespace model cache PVC
• Instant model switching
• Configurable cache invalidation

Model Catalog

20+ pre-configured models. Deploy instantly with optimized settings.

• One-command deployments
• Llama, Mistral, Qwen, DeepSeek, Mixtral, Phi
• Smart defaults with override support

Multi-GPU Support

Deploy 13B-70B+ models across GPUs with automatic layer sharding.

• ~44 tok/s on Llama 13B with 2x RTX 5060 Ti
• Automatic tensor split calculation
• Layer-based sharding

Multi-Cloud Support

Deploy on any Kubernetes distribution. Standard K8s patterns.

• Tested on GKE, kind, Minikube
• Works on AKS, EKS, bare metal
• Custom tolerations and nodeSelector

Helm Chart

Production-ready chart with 50+ configurable parameters.

• helm install llmkube llmkube/llmkube
• ImagePullSecrets for private registries
• Namespace isolation, RBAC included