A weekend getting DeepSeek-V4.1-Flash onto three DGX Sparks in a ring, on Kubernetes

I went into the weekend wanting one thing: DeepSeek-V4.1-Flash running on my own hardware. I have three DGX Sparks cabled to each other in a ring, no switch in the room, and a 552B model that, quantized the right way, should just fit across them. By Sunday night it was serving behind one LLMKube InferenceService, at decode parity with the published reference numbers, and I had a list of everything that had tried to stop it. This post is that list, in the order it happened, so the next person with three Sparks can skip the parts that hurt.

Almost none of the hard parts are mine, and the credits matter here. tonyd2wild wrote the TP3 patch set and published the reference numbers on a shared RoCE subnet. bot-lab-21 produced the EXL3 3.5 bpw checkpoint and the exl3 config and MoE overlay that rides inside that patch set. Zeuss5 wrote cuda-exl3. Kai is credited upstream in the patch set for Engram-on-disk and the SM12x pages. LibertAI's TP2 REAP lane is where CUDA graphs produced NaN logprobs, which is why my boot checklist has a logprobs check in it. And zyang-dev wrote the subnet-aware IB routing commit that makes a cabled ring work at all. What I added is the Kubernetes shape, a pinned image, and ten boots' worth of finding out what actually breaks.

Why it takes three Sparks and not two

DeepSeek-V4.1-Flash is a 552B mixture of experts. bot-lab-21's checkpoint quantizes the routed experts to EXL3 at 3.5 bits per weight and leaves everything else at the official FP8 and MXFP4 release. Split three ways, that is 79.5 GiB of weights per Spark text-only, and 82.5 GiB with the DSpark drafter loaded. Three times 128 GB of unified memory holds the unpruned checkpoint with room left for a KV pool. Two Sparks do not. The two-Spark route runs a REAP-pruned expert set, which is a different model, and it is also the lane where CUDA graphs produced NaN logprobs. I wanted the whole model.

The whole model comes with a catch. 64 attention heads and 8 output groups do not divide by three. The patch set pads them to 72 and 9 at load time, and that is a config.json edit on the staged checkpoint, not a runtime flag. Remember that one, it comes back at the end as a Job. The Engram n-gram tables are another 189 GiB, and the patched engram.py reads them per forward straight out of the safetensors shards on NVMe instead of loading them whole, so they never have to fit in memory next to the weights.

NCCL refused the ring, and it was right to

The Sparks were already cabled when the weekend started. Each one has two ConnectX controllers, four RoCE ports, and the ring runs port 0 of each Spark to port 1 of the next, all the way around. No switch, no subnet manager, six point-to-point legs across three cables. The first thing I did, while 429 GiB of checkpoint copied onto the nodes, was run an NCCL test across it. It failed with ncclSystemError. It kept failing with every HCA ordering I tried, every NCCL_CROSS_NIC value, and any number of ports listed.

The reason is structural, and once I saw it I stopped trying knobs. NCCL pairs NICs by channel index on both ends of a link: channel 0 uses device 0 on the sender and device 0 on the receiver, channel 1 uses device 1 and device 1, and so on. On a switch that is always fine, because every device reaches every peer. On a ring it can never be fine. Walk it: the port that reaches your successor is port 0, the port that reaches your predecessor is port 1, and your successor sees that same cable arrive on its port 1. Index pairing is a two-colouring of the cycle. The cycle has three nodes. An odd cycle is not two-colourable. Every channel is wrong on exactly one end, and no variable that reorders devices can change the parity of a triangle.

One more thing about the shape, because it is the thing I would have got wrong later. The 2.5 GbE management NIC is a different plane entirely. It is the out-of-band rendezvous path and nothing else. If the data plane ever lands on it you will not see an error. You will see a slow model.

The fix was in the library the whole time

NCCL_IB_SUBNET_AWARE_ROUTING=1. It makes NCCL pick, per peer, the local IB device whose subnet matches that peer's, instead of pairing by index. On a ring that is exactly the right rule, and it is one environment variable.

I nearly wrote that it did not exist. It is not in NCCL's documented environment variables, and a GitHub code search for the symbol returned zero hits at the time. It started as zyang-dev's dgxspark-3node-ring fork, one 141-line commit in src/transport/net_ib/connect.cc dated 2026-03-13, and it has been upstream since NCCL 2.30.7. The official vLLM DSv4.1 arm64 base image ships exactly that version, so the fix was sitting in the library I was already running. NVIDIA's connect-three-sparks playbook runs its ring NCCL test with precisely this env. What found it for me was strings libnccl.so.2 | grep SUBNET. Grep the library before you conclude a feature does not exist.

The env block that boots the ring:

env:
  - {name: NCCL_NET,                      value: IB}
  - {name: NCCL_IB_DISABLE,               value: "0"}
  - {name: NCCL_IB_SUBNET_AWARE_ROUTING,  value: "1"}   # the one that matters
  - {name: NCCL_IB_MERGE_NICS,            value: "0"}
  - {name: NCCL_NET_PLUGIN,               value: none}
  - {name: NCCL_IB_ROCE_VERSION_NUM,      value: "2"}
  - {name: NCCL_IB_ADDR_FAMILY,           value: AF_INET}
  - {name: NCCL_IB_TC,                    value: "106"}
  - {name: NCCL_IB_TIMEOUT,               value: "22"}

A good boot says so twice. Every rank prints its four ports and its out-of-band interface: NET/IB : Using [0]rocep1s0f0:1/RoCE [1]rocep1s0f1:1/RoCE [2]roceP2p1s0f0:1/RoCE [3]roceP2p1s0f1:1/RoCE [RO]; OOB enP7s7:192.168.1.28<0>. Then, with NCCL_DEBUG_SUBSYS=INIT,NET, it prints every substitution it makes: NET/IB: Subnet-aware routing: overriding dev 1 with dev 0. The rank-0 log of the first built-image boot has 22,147 of those lines. NCCL_DEBUG=INFO alone prints the port listing and none of the substitutions, which is an uncomfortable way to be half-sure your ring works.

With the variable set, a three-rank all-reduce over 256 MB buffers measured 23.2 GB/s bus bandwidth with all four ports, against 11.3 GB/s on a single two-node leg with two ranks. The fabric was done. The weights finished copying. I booted the model.

Boot one took a node down

Everything about the first boot looked right until it did not. All three members came up. NCCL over RDMA connected first try, substitution lines scrolling past. Every rank loaded 79.47 GiB of weights. Then dgx1's kubelet stopped heartbeating while rank 0 was still loading. vLLM kept going, finished the load, and the other two ranks compiled their TileLang kernels for the KV-cache profile pass. Then nothing. Minutes later the node still answered ping, but SSH timed out at banner exchange and log fetches hit TLS handshake timeouts. Userland had starved. It needed a power cycle.

The previous boot's journal survived, 82,707 lines of it, and the striking thing is what is not in it. There is no OOM-killer line anywhere. The node never killed anything. With vm.overcommit_memory=1 and a 16 GB /swap.img, the kernel paged kubelet and sshd out to disk and kept the model. On GB10 the CUDA allocations come out of the same 121 GiB of unified memory the OS uses, and they are not charged to the pod cgroup, so the pod's 110Gi limit never engaged. A wedge, not a crash, which is the worse of the two because nothing in Kubernetes reports it.

It was not the KV cache. It was a compiler.

My working theory, written down before the node came back, was that the KV-cache profile pass had exhausted host memory on the rank-0 node. That is where the timing pointed, and it is the failure a 552B model invites you to expect. It was wrong, and I am glad I looked before I started turning memory knobs.

The culprit was in the JIT cache directories. Under .cache/flashinfer/0.7.0rc1/121a/cached_ops, dgx2 and dgx3 had a finished .so for FlashInfer's mxfp8_gemm_cutlass_sm120, six cutlass variants, built during that first boot. dgx1 had only the base .o. Its compile had been cut off by the wedge, which means it was running at the moment the node went. So at the profile pass dgx1 was compiling that module with MAX_JOBS=4, which is four cicc processes, the nvcc front end, at about 6 GB of host RAM each, on top of 79.5 GiB of pinned weights and the profile activations. Load average at the peak was 54. The spike was the compiler, not the KV budget.

Three fixes, then a clean failure

Three changes fixed it, and all three are now in the image or the sample.

Prewarm the modules in the image. sparse_mla_sm120 and mxfp8_gemm_cutlass_sm120 are JIT-compiled at build time under the exact runtime environment, because a JIT built with different nvcc flags is a different cache entry and gets rebuilt on the serving node. On the first built-image boot exactly one module still compiled at serve, trtllm_utils, pulled in by the mxfp8 autotuner.
MAX_JOBS=1 at serve. If anything does compile on a serving node, it compiles one variant at a time. The image prewarms at MAX_JOBS=1 too, so the serve-time environment matches the cache entry it is looking for.
swapoff -a on every member first. With swap off, an overrun becomes an OOM kill of the worker, and the operator restarts the worker. Boot two proved it: the kernel killed node_exporter, pipewire and dbus, and the node was back at 115 GiB available and Ready in 90 seconds with no power cycle. There is a Kubernetes wrinkle worth knowing. The operator sets requests equal to limits, so the pod is Guaranteed QoS with an oom_score_adj of -997, and the kernel will kill every daemon on the box before it touches the worker. tonyd2wild runs --oom-score-adj 500 on bare metal, the exact opposite. Swap off is my equivalent for now.

Boot three failed the way I wanted a boot to fail. I had pushed gpuMemoryUtilization down to 0.72 to buy the host some room, and the engine reported Available KV cache memory: -1.29 GiB and exited without touching the node. The arithmetic is simple once you see it: 0.72 of about 121 GiB is roughly 87 GiB, and 79.47 GiB of weights plus about 7 of activations plus 2 of CUDA graphs does not fit under that. The floor is around 88.5 GiB. At 0.78 it fits, with 10 to 14 GiB of host memory free in steady state. Boot four served.

k is the only knob, and it is a workload knob

Boots five through eight were about speculative decoding. DSpark is the drafter that rides inside the checkpoint, k=5 is the production setting in tonyd2wild's launcher, and before I made it the default for coding I wanted to know what it cost in quality. Three things came out of it.

First, adaptive verification is unavailable on GB10. Turn it on and the worker raises at memory determination with Adaptive verification trims verification requests on device, which the DeepseekV4IndexerSM12xBackend attention backend does not support. On the SM12x sparse indexer, k is the only lever you have.

Second, k=5 against k=3 is a workload choice, not a quality choice. Over the same six prompts they tie on aggregate, 1.44x against 1.45x, and split by prompt type. k=5 wins code, 64.9 against 51.0 tok/s, and arithmetic. k=3 wins prose and lists, and it leaves a much larger KV pool because the verification batch is smaller. For an agentic single-stream target, which is mostly code, k=5.

Third, the quality question, which was the one I actually cared about. DSpark uses block verification with a probabilistic draft, which is lossless in distribution, and my measurements agree with the theory without quite proving it byte for byte. Six prompts, temperature 0, seed 7, 200 tokens each: k=5 reproduced 2 of 6 outputs byte-identically against the no-speculation reference, and k=3 reproduced 1 of 6 against both that reference and the k=5 run. Every divergence starts at a near-tie token, a curly apostrophe against a straight one, "specifications" against "specs", and continues as equivalent text, and the flips go in both directions. That is float-order noise from verifying six tokens per forward instead of one, which changes the batch shape and therefore the reduction order. Temperature is not a speed lever either: three extra essays at 0.7, 0.7 and 0.0 decoded within a token per second of each other.

The ring is at parity, and the rest is speculation

Everything below is single stream, text only, on unique prompts, on the built image at a 131,072-token context, four sequences, gpuMemoryUtilization 0.78. The right column is tonyd2wild's TP3 on a shared RoCE subnet, from his per-try table.

Measurement	Ring, three Sparks, no switch	tonyd2wild, TP3 on a shared RoCE subnet
Weights per Spark	79.5 GiB text-only, 82.5 GiB with the DSpark drafter	84.2 GiB, vision tower and DSpark drafter included
Prefill, 3.4k prompt tokens	1,094 tok/s	712 to 846 tok/s at his nearest prompt size, 2,950 tokens
Prefill, 49.8k prompt tokens	1,334 tok/s	1,236 tok/s at 93k (his no-speculation try 9)
Decode, no speculative decoding	26.5 tok/s	25.4 to 26.8 tok/s
Decode, DSpark k=5	31 to 35 tok/s on prose, 65 on code, 1.5x on a mixed six-prompt set	51.5 tok/s

The row that matters is decode without speculation. Same configuration, CUDA graphs on: the ring decodes at 26.5 tok/s and the switch decodes at 25.4 to 26.8. The ring is at parity. The gap in the last row is speculation and prompt mix, not fabric. Prefill on the ring is at or above his across the range. The long prompt prefilling faster than the short one most likely reflects per-request overhead amortising over a longer prefill; I did not measure that separately.

Two numbers to keep and one to throw away. The KV pool at the shipped configuration is 355,955 tokens, 2.72x the context. Acceptance under DSpark k=5, across a probe, a quality set and a benchmark, came out at 30.0%, or 2.50 tokens per step. And a repeated prompt that hits the prefix cache reads 38,626 tok/s of prefill. That number is real and means nothing.

Boots nine and ten were the final image. The rc4 candidate reproduced the numbers within noise, decode 31.6 and 37.8 tok/s, prefill 1,100 and 1,341, and then a 524,288-token context at utilization 0.80 came up with a KV pool of 849,026 tokens and 9 to 11 GiB of host headroom on the rank-0 node. That is the one I am coding against now.

What it looks like in LLMKube

All of the above is one InferenceService. The part that encodes the ring is spec.multiNode, and it carries two planes that are easy to confuse.

multiNode:
  rdmaResource: rdma/rdma_shared_device_a
  ibGIDIndex: 3
  members:
    - node: ahazidgx1
      fabric:
        # bootstrap plane: rendezvous and the NCCL out-of-band handshake,
        # on the 2.5 GbE management LAN.
        address: 192.168.1.28
        socketInterface: enP7s7
        # data plane: all four RoCE ports. Subnet-aware routing picks the
        # right one per peer (NCCL_IB_SUBNET_AWARE_ROUTING=1 in spec.env).
        ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"
      modelCache: {claimName: dsv41-dgx1}
    - node: ahazidgx2
      fabric: {address: 192.168.1.86,  socketInterface: enP7s7, ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"}
      modelCache: {claimName: dsv41-dgx2}
    - node: ahazidgx3
      fabric: {address: 192.168.1.220, socketInterface: enP7s7, ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"}
      modelCache: {claimName: dsv41-dgx3}

address and socketInterface are the bootstrap plane. Give them the management IP and the management interface, not a fabric address, or rendezvous will try to ride the RoCE legs before NCCL has decided anything. ibHCA is the data plane: list every port and let subnet-aware routing sort them out per peer. That split is the whole trick, and it is the thing I would have got wrong if the NCCL test had not forced me to separate them first.

Around it sit a Model that stages the 54 checkpoint files onto every member from an s3:// prefix with an explicit file list, and two Jobs the image deliberately does not bake, because both edit the checkpoint or the host rather than the container. One writes the TP3 head and group counts into the staged config.json, the padding from earlier. The other drops the page cache before the group boots, because vLLM's worker refuses to start when CUDA-free memory is below the utilization target, and 429 GiB of freshly written files will hold it there. Then swapoff -a on each Spark, which is not yet something the operator does for you.

What LLMKube grows next from this is the obvious thing. The ring should stop being a set of per-member fabric strings a human keeps consistent by hand and become a first-class multi-node shape at the CRD level, so the operator knows the topology, derives the bootstrap and data planes from it, and can validate a member list before it schedules three pods that will fail at channel setup. Everything in this post is the hand-built version of that.

If you have three Sparks

This is the part I wish I could have read on Saturday morning.

Cable port 0 to port 1 around the ring, then run NVIDIA's ring NCCL test before you copy a single weight. List all four ports in NCCL_IB_HCA, set NCCL_IB_SUBNET_AWARE_ROUTING=1, and keep rendezvous on the management LAN. Do not start the model until the test passes.
Boot the first time with NCCL_DEBUG_SUBSYS=INIT,NET and look for the "overriding dev" lines. If you only see the port listing, you do not yet know whether routing is doing anything.
Use an image whose FlashInfer kernels are already compiled for this GPU, and set MAX_JOBS=1 anyway.
Turn swap off on every member before the group boots. A wedge is worse than an OOM kill, and Kubernetes only reports the second one.
Stage the checkpoint, apply the TP3 config edit, drop the page cache, then boot. gpuMemoryUtilization 0.78 works at this size and 0.72 does not, and that failure is at least a clean one.
Speculation: DSpark k=5 for code, k=3 if your prompts are mostly prose and you want the bigger KV pool. Adaptive verification does not work on GB10, so do not spend an evening on it.

A three-hour checkpoint stage followed by a fabric failure at channel setup is an expensive way to learn that one environment variable was missing. I ran the NCCL test while the weights were still copying, and that is why boot one got past channel setup at all.

What is pinned, and where the receipts are

Every layer is pinned, and the pin is a digest or a commit, never a tag. The base is the official vLLM DSv4.1 arm64 image by digest, Apache-2.0. FlashInfer 0.7.0rc1 is rebuilt from source in place at 07869c61, Apache-2.0, because the base's 0.6.18 lacks the SM12x sparse-MLA decode shapes, and its CUTLASS, CCCL and spdlog submodules are pinned by SHA too. cuda-exl3 is 6a1ffc34, MIT, compiled for sm_121a. tonyd2wild's TP3 patch set is fc725ecf, MIT. One vLLM file, the DeepSeek-V4.1 nvidia model, is restored at e47aa780b, Apache-2.0, because the base's copy imports symbols the patched engram.py does not define. The licenses across the tree are MIT, BSD-3-Clause and Apache-2.0. Nothing is AGPL.

The overlay replaces eleven files inside dist-packages, which is exactly the kind of thing that rots silently when the base moves. So the build carries a drift gate: MD5 sums for every file the overlay touches, checked before anything is copied. If a future base image moves one of them, the build stops and names the files. Two in-image test suites then prove the assembled tree imports and loads without a GPU, which is what lets the whole thing build on a GPU-less arm64 runner in about 30 minutes.

Where it all lives:

LLMKube pull request #1816, merged: the sample config/samples/inferenceservice_multinode_vllm_three_sparks_ring.yaml, the Model and both Jobs beside it, and a "Three-Spark ring" section in the multi-node guide.
llmkube-runtimes #41, merged: ghcr.io/defilantech/llmkube-vllm-cuda-gb10-dsv41-exl3. The table came off candidate digest sha256:8899a1285e156849f007848e78ef1cc0d31631664e780be0d53d998edeaa1fa3. The final candidate, sha256:8dab469dd37e28c26612648079eb075eb3152dc2dacd8347c55a6f2aba037b54, is the same vLLM tree plus three license texts, and it reproduced the table.

If you are running DeepSeek-V4.1-Flash on Sparks and your numbers disagree with mine, I would like to know. The Discord is open. Every figure in the left column came off a boot I can point at, and the right column is tonyd2wild's published table.