One GPU, four ways to share it: ten scenarios on a DGX Spark, and the headline finding I had to retract

Everyone running local models eventually has more models than GPUs. LLMKube offers several answers to that, and until last week I was choosing between them on intuition. So I took two DGX Sparks and ran ten scenarios across every sharing mechanism the operator supports. I also got the headline finding wrong, published it, failed to replicate it, and retracted it. That is in here too, because how a measurement fools you is more useful than the measurement.

The four things you can actually do

Before numbers, the options. They operate at different layers, which is why comparing them needs care.

There is a fifth, partitioned (MIG), which I could not test. More on that honestly at the end.

The baseline

A 27B dense model at Q4 on one GB10, nothing else running: 768.85 tok/s prefill, 31.34 tok/s decode. Every later number is measured against that.

One immediate lesson before any sharing. My first baseline read 12.11 tok/s decode, which is 2.6x too slow, because I cloned an InferenceService spec and dropped speculativeDecoding. MTP is a first-class CRD field, not an extraArgs entry, so copying the args carried none of it. If you benchmark by cloning specs, diff the whole spec.

Time-slicing: free until it isn't

Two models co-resident on one physical GB10, with the second idle, cost the first nothing measurable. I checked this properly rather than assuming, by suspending the co-tenant and re-measuring:

Identical within noise. A model you are not using costs you nothing but memory.

Under real concurrency the picture changes, and it changes asymmetrically. With four tenants all active, decode fell 4.3x to 7.33 tok/s, which is roughly what you would expect from splitting a card four ways. Prefill collapsed 16x. A 10,050-token prompt took 209 seconds.

That asymmetry is the practical finding. Decode degrades gracefully and prefill falls off a cliff, so time-slicing suits chat-shaped traffic with short prompts far better than anything that reads large contexts. An agent that stuffs a repository into its window is the worst possible co-tenant.

The finding I retracted

Here is the number I led with, and why it was wrong. I am leaving the whole thing in, because how a measurement misleads you is more useful than the measurement.

I deliberately over-subscribed, running three copies of the same 27B on one card, expecting to document a failure mode. It never broke. Node memory went from 29.7 GiB to 33.5 GiB when the third instance loaded: about 4 GiB for a 27B model, not 17.

The obvious explanation is that llama.cpp mmaps weights, so the page cache holds one copy of the file and the kernel shares it across processes. On GB10 there is no separate VRAM, so those would be the same physical pages the GPU reads. That would make duplicate models nearly free on unified memory and expensive on a discrete card, which is a genuinely interesting claim about hardware.

It is also not true. Before publishing more of it I ran the controlled replication: same node, same cached 15.66 GiB GGUF, three instances with --n-gpu-layers 99, nothing else running.

About 1.03x model size each, perfectly linear. The third instance costs 16.1 GiB, not 4. The original observation does not reproduce.

What is actually going on

The variable was never the memory architecture. It is whether the weights are offloaded.

mmap page-cache sharing is real, and it only helps while the weights stay CPU-resident. Once llama.cpp offloads, it allocates private CUDA device buffers and copies the weights into them. That happens on unified memory too. The page cache still holds one copy of the file; it simply is not what the GPU reads from.

I verified the offload rather than assuming it, on the same box, warm: 99.00 tok/s decode with --n-gpu-layers 99 against 11.36 tok/s with 0. The GPU arm was genuinely using the GPU, and it still paid full price per copy.

How the original measurement fooled me

Three things, and they compounded.

Two different instruments. The GB10 figure was node memory during a noisy four-tenant run. The discrete figure was nvidia-smi. I compared them as though they measured the same thing. They do not, and worse, nvidia-smi reports [N/A] for memory on GB10, because there is no separate pool to report. The instrument I trusted on one machine does not exist on the other.

One observation, no replication. A single reading taken while four tenants competed, treated as a finding. Every other number in this post came from a repeated, isolated measurement. That one did not, and it is the one that was wrong.

Mismatched model sizes. A 3B on the discrete card against a 27B on the Spark. The fixed CUDA context cost is proportionally much larger on a small model, which is exactly why the 3B reads 1.38x and the 27B 1.03x. That gap is an artifact of model size, and I had attributed it to memory architecture.

The uncomfortable part is that the wrong version was the better story. It had a clean mechanism, a counterintuitive result, and practical advice that reversed on your hardware. Everything except being true. I caught it because I went looking for holes before promoting it further, which is not a habit I can claim to apply consistently.

What survives

The Spark's co-tenancy advantage is real. It is just capacity, not economics. 128 GB of unified memory holds roughly seven 16.8 GB models where a 16 GB card holds none. Every copy costs full price on both; there are simply far more places to put one.

So the sizing rule is duller than the one I published: count models against your memory pool at roughly full model size each, whatever the architecture. What unified memory buys you is the size of the pool.

ModelPool: correct, and slow enough to plan around

ModelPool takes the other approach: one member resident at a time, swapped on demand. It behaves well. Drain-before-unload preserved a full 1200-token generation while a swap was requested mid-flight, the anti-thrash coalescing turned two concurrent requests into one swap, and the fail-closed path kept the incumbent resident when the successor could not be reached.

The cost is the swap itself: 122.0s cold, 53.6s warm. That is pod teardown, scheduling, container start and model load. Fine if switches are rare relative to serving. Painful if your workload alternates.

llamacpp-router: the fastest swap you cannot use yet

Swapping models inside one server avoids all of that. Measured, an in-process switch is 4.5 to 4.7 seconds against ModelPool's 53.6s on identical hardware, roughly 11x cheaper.

I nearly published that as a recommendation. Then I put a real agent through the endpoint and compared it against the baseline:

22x slower prefill while holding a GPU the entire time. Router mode deliberately emits no --n-gpu-layers, because different models may want different offload settings, so llama.cpp defaults to zero GPU layers. Passing the flag through extraArgs does not help either: router mode spawns a child server per model and the parent's flags do not reach them.

So you can request a GPU, have it scheduled and consumed, and serve at a twenty-second of the speed with only a line in the operator log to tell you. That is tracked in issue 516. The swap latency is real and worth wanting; the runtime is not a serving option until offload reaches it.

So which should you use

And the sizing rule: count models against your memory pool at roughly full model size each, on either architecture. Unified memory buys you a bigger pool, not cheaper copies.

Two traps that produced wrong answers first

Both of these gave me confident, plausible, wrong numbers, and both are easy to repeat.

The first request after load is not representative. On GB10 a cold request measured 93 tok/s prefill where the warm figure was 743, an 8x difference from JIT alone. Any benchmark that fires one request at a freshly loaded model understates it by an order of magnitude.

Prefix caching will happily benchmark nothing. Reusing the same prompt for warm-up and measurement produced prompt eval time = 148 ms / 4 tokens. It was not fast, it was skipping the work. Distinct prompts per measurement.

A third one is operator-specific but worth knowing: kubectl scale on an operator-owned Deployment does nothing, because the controller reconciles the replica count straight back. My first two attempts at the idle-co-tenant comparison silently measured the same condition twice and "proved" the arms were identical for entirely the wrong reason. Use the InferenceService's own suspend field.

What benchmarking found that benchmarks do not usually find

The last scenario put a real coder-and-reviewer agent pipeline on one shared GPU rather than a synthetic load generator. It reported success end to end. It was also wrong in two ways nothing in the pipeline could see.

The coder wrote a correct two-line fix and pushed it to a completely different repository than the one it was working on, then reported success. And the reviewer approved a change it had never seen: its branch was force-reset to the base commit before it ran, so it reviewed an empty diff and returned a confident, well-written verdict describing an unrelated commit's contents.

That second one is the one that bothers me. Reviewers approving things they should not had been read as a model-quality problem. It was substantially a harness problem: the models were being handed the base branch. After the fix the reviewer reads the real diff and finishes in 100 seconds instead of 304, because it is no longer wandering a repository hunting for a change that was not there.

A third defect surfaced the same way and is my favourite, because the tool reported that nothing was wrong. An agent searching the repository for stats matched inside a vendored minified bundle, where the entire file is one line. The grep tool capped how many matches it returned but never how long one was, so a single result came back at 649,467 bytes, marked "truncated": false. That one tool call pushed the transcript past the stuck-loop detector's limit and the run was force-terminated as a loop. The detector was right. The tool had lied to it.

Both are fixed (1489, 1464). The general lesson is that a synthetic benchmark exercises the paths you thought to write down. A real workload exercises the ones you did not, and defects live in exactly the places the measurement was not looking.

Worth adding honestly: fixing the harness did not make the reviewer good. Given the real diff, a 12B model called a behaviour fix "improving readability without altering functionality", which is backwards. The approval was right by accident. Harness quality and model quality are separate problems and I had been conflating them.

What I did not test

partitioned, which is MIG. No GPU I have is MIG-capable: both GB10s and both RTX cards report nvidia.com/mig.capable=false. So of the three gpuSharing.mode values the CRD accepts, this post covers two, plus ModelPool and the router at other layers. If you have A100s or H100s, the mode I could not measure is very likely the one you should be looking at, and I would genuinely like to see those numbers.

I also measured throughput and memory, not accuracy. Nothing here says anything about whether sharing a card changes what a model produces. It should not, and I did not verify it.

The short version

Start with exclusive. Reach for shared when demand is genuinely concurrent and your prompts are short. Reach for ModelPool when roles alternate and you can absorb a minute per switch. Skip the router until it can reach the GPU. And treat every over-subscription number as costing full model size per copy, because that is what replication showed once I stopped trusting a single reading taken under load.

LLMKube is Apache 2.0 and the sharing modes are in the InferenceService CRD under spec.resources.gpuSharing. If you run this on hardware I do not have, I would like to hear what you get.