Frequently asked questions
Operational answers about LLMKube. Looking for how it stacks up against other projects? See how LLMKube compares.
Does it work with Argo CD / Flux?
Yes. LLMKube uses standard Kubernetes Custom Resource Definitions (CRDs). Your Model and InferenceService manifests are just YAML files that Argo CD and Flux handle like any other Kubernetes resource.
Store your model definitions in Git, let your GitOps tool sync them, and LLMKube's operator handles the rest. No special integration required.
What about AMD GPUs?
Yes. AMD GPUs are supported via the Vulkan (RADV) backend, shipped in v0.8.8.
Set vendor: amd and runtime: vulkan on the model's GPU spec, and the operator schedules LLMKube's own hardware-gated llama.cpp Vulkan image onto your AMD node. Pods are placed on /dev/dri through a device plugin (no nvidia.com/gpu), so no ROCm install is required. Validated on Strix Halo (gfx1151).
A ROCm/HIP compute tier also ships for cards that benefit from it: set runtime: rocm instead of vulkan on the same GPU spec. If you have a specific AMD card or workload in mind, open an issue on GitHub.
Does it support auto-scaling / HPA?
Yes. HPA autoscaling shipped in v0.6.0. InferenceService supports native Kubernetes Horizontal Pod Autoscaler with per-runtime metrics.
Set minReplicas, maxReplicas, and a target metric value in your InferenceService spec, and LLMKube handles the rest. Each pluggable runtime (llama.cpp, vLLM, TGI) provides its own default HPA metric through the HPAMetricProvider interface.
For example, the llama.cpp runtime scales on llamacpp:requests_processing, while vLLM scales on vllm:num_requests_running. These are Prometheus custom metrics, so the HPA needs Prometheus Adapter installed to read them; the operator writes the HPA object, the adapter serves the metric.
What models are supported?
LLMKube supports multiple model formats across its pluggable runtimes:
• GGUF via llama.cpp — most popular open models • SafeTensors and PyTorch via vLLM and TGI • MLX via the metal-agent's mlx-server runtime on Apple Silicon • Any format via the generic runtime with custom containers
We also ship a catalog of 17 pre-configured models that deploy with a single command (e.g., llmkube deploy llama-3.1-8b --gpu): Llama, Mistral, Qwen, DeepSeek, Mixtral, Phi, and more. Each catalog entry pins the source URL, recommended runtime, and resource defaults, so one command produces a reproducible deployment without you having to figure out where the weights live.
How does air-gapped deployment work?
For air-gapped environments, you download models once on a connected system, then transfer them to your air-gapped cluster via your approved data transfer process.
LLMKube reads models from an internal S3-compatible object store (s3:// with a credentials secret), a pre-populated PVC (pvc://), or a local path on the node (file://). Once the weights are reachable inside the cluster, deployments work identically to connected environments. No external network calls are required during inference.
The full procedure, including mirroring the operator and runtime container images into a private registry, is documented in the air-gapped install guide.