Consumer-hardware model matrix

Snapshot dated 2026-05-11. Not actively maintained.

The local LLM landscape changes by the week. New model releases routinely supersede the ones in this table within a month or two, file sizes shift as quanters re-upload with better imatrix calibration, and license terms occasionally tighten. Treat this document as a starting point, not a source of truth. Before you download anything large, click through to the upstream HuggingFace model card and verify the current size, context length, and license for yourself.

If you spot something out of date, a pull request is welcome. We do not promise to keep this current.

A practical guide to picking a GGUF model for LLMKube based on the hardware you actually have. Every file size in this doc was verified directly from HuggingFace on 2026-05-11. Capability claims come from the upstream model cards (linked throughout), not third-party quanters.

How to read the tables

File size is the raw GGUF size on disk. To actually run the model with a usable context you need additional headroom for the KV cache, runtime overhead, and OS. The rough rule:

required memory = gguf_file_size * 1.20  +  kv_cache(ctx_length)

For most chat workloads at 8K to 16K context, add 20 to 30 percent to the file size and you will be close. Long context windows (128K+) or fp16 KV cache push this much higher. The KV cache headroom section below covers the math, and the LLMKube KV cache types and Memory-pressure protection docs document the runtime knobs.

Quant choice. Q4_K_M and IQ4_XS are the sweet spot for most users: good quality, smallest size that still feels like the original. Q5_K_M and Q6_K cost more memory but get closer to fp16 quality. Q8_0 is near lossless at roughly 1 byte per parameter. The Unsloth UD-*_XL variants are dynamic quants that mix bit widths smartly; usually preferred when available.

MoE models (Qwen3 Coder, Qwen3.5/3.6 A3B, gpt-oss-20b, gpt-oss-120b) are bigger on disk than their throughput suggests, because only a small subset of experts is active per token. They still need the full file resident in memory, but they generate as fast as a much smaller dense model. A 35B MoE with 3B active params loads like a 22 GB model at Q4 and generates like a 3B dense.

Hardware tiers

In LLMKube specifically, multi-GPU sharding lets you span tiers by adding GPUs. A 70B at Q4_K_M (about 40 GB) does not fit a single 24 GB RTX 4090, but it shards cleanly across 2x RTX 4090 with gpuCount: 2. See Multi-GPU sharding.

Mac unified memory: usable vs advertised

Apple Silicon shares one memory pool between CPU and GPU. That is an advantage (no copy, no VRAM split) but means macOS and background processes are eating into the same pool you want to load a model into.

Inference speed on Apple Silicon is bandwidth-limited. Memory bandwidth, not core count, is the bottleneck:

Rough token/sec ballparks at Q4_K_M (interactive use):

These are ballparks. Real numbers depend on context length, llama.cpp build flags, and other apps competing for memory bandwidth. Treat them as “is this interactive” not as benchmarks.

KV cache headroom

The KV cache grows roughly linearly with context length. As a back-of-envelope at Q4_K_M weights:

Practical examples:

• A 17 GB model on a 24 GB GPU leaves about 7 GB for KV cache. Fine for 8K to 16K context. Not enough for 128K.
• gpt-oss-20b (10.8 GB) on a 16 GB card leaves about 5 GB. Comfortable at 16K. Tight at 32K.
• Llama 3.3 70B Q4_K_M (39.6 GB) sharded across 2x RTX 4090 (48 GB total) leaves about 8 GB for KV. Stay under 32K context unless you switch to a Q4 KV cache type.

LLMKube exposes the KV cache dtype as a Model CRD field; see KV cache types for the trade-off (Q4 cache halves memory at a small quality cost).

Multi-GPU notes

llama.cpp supports multi-GPU inference via layer splitting. LLMKube wraps that with gpuCount on the Model CRD.

• Two 12 GB cards can run a 20B to 24B model (10 to 12 GB of weights per GPU plus KV).
• Two 24 GB cards comfortably hold a 70B at Q4_K_M (about 40 GB of weights split across two devices).
• PCIe communication adds latency. For a given total VRAM budget, one bigger card beats two smaller cards in tokens/sec. Pick multi-GPU when you need the model to fit, not for speed.
• Sharding shape matters. Layer-split (the default in llama.cpp and LLMKube) is simpler than tensor-parallel and works well for inference. Tensor-parallel needs NVLink-class interconnect to actually help.

The matrix, by hardware tier

Tier 1: Edge devices (up to 8 GB)

Tier 2: Entry GPUs / laptops (12 to 16 GB)

Tier 3: Enthusiast (24 to 32 GB)

Tier 4: Pro single-node (48 to 64 GB)

Tier 5: Workstation (96 to 128 GB)

Tier 6: Multi-GPU / Ultra (192 GB and up)

The matrix, by task

Same models, sorted by what you want to do. Numbers cited come from the upstream model cards.

General assistant / chat / RAG

Coding / agentic SWE

Reasoning / math / STEM

Vision / multimodal

llama.cpp loads the vision projector from a separate mmproj-*.gguf file alongside the language GGUF. Account for both.

Function calling / tool use / agents

Long context (100K+ tokens)

Phi-4 14B is only 16K context; do not pick it for long-document workloads.

What this doc deliberately does not include

• Uncensored / abliterated / “heretic” variants. They show up high in HuggingFace trending but I want this list to be reproducible.
• Hobbyist frankenmerges (anything labeled “X-Distilled-GGUF” from non-vendor authors). The base models above are the inputs to most of those; pick the base unless you have a specific reason.
• Pure base models (no -Instruct / -it suffix). LLMKube users almost always want instruct variants.
• Embedding / reranker GGUFs. Those belong in a separate matrix for RAG infrastructure.
• GGUFs marked imatrix-only (calibration files, not runnable models).
• CPU-only inference benchmarks. Possible but typically 1 to 5 tok/s for a 7B; this doc assumes GPU or Apple Silicon.

Methodology and sources

Every file size in this doc was fetched from the HuggingFace tree API on 2026-05-11. Capability claims (context length, license, benchmarks) come from the upstream meta-llama/, Qwen/, mistralai/, google/, microsoft/, openai/, deepseek-ai/, and ibm-granite/ model cards, not from third-party quanters. The Unsloth and Bartowski GGUF repos are referenced because they are the most-downloaded community quants with reproducible naming, but the underlying weights and licenses follow the upstream publisher.

When in doubt, prefer Q4_K_M as the starting quant, measure quality on your own task, and step up to Q5_K_M or Q6_K only if you see issues. Going below Q4 (IQ3, Q2, IQ2, IQ1) trades real quality for memory; only do it if the model otherwise will not fit.

A reminder that this will go stale

The local LLM ecosystem is one of the fastest-moving spaces in software. Between when this was written and when you are reading it, expect that:

• New model generations (Llama 4.x, Qwen3.7, Gemma 5, etc.) have probably shifted the recommendations.
• Quanters have re-uploaded files with better imatrix calibration; sizes shift by a few percent.
• Tools like llama.cpp and llmkube have added new quant types, faster KV cache modes, or runtime backends.
• License terms occasionally change (especially Llama and Gemma).

Use this as a structured starting point. Always click through to the upstream HuggingFace model card before downloading a large file. If you find this doc is meaningfully wrong, a PR with the corrected row is appreciated, but please do not assume any single number here is current the day you read it.