DeepSeek-V4.1-Flash on three DGX Sparks in a ring, on Kubernetes

Three DGX Sparks, cabled to each other in a ring with no switch in the room, serving a 552B DeepSeek-V4.1-Flash at tensor-parallel 3 behind one LLMKube InferenceService. Almost none of the hard parts are mine. tonyd2wild wrote the TP3 patch set and published the reference numbers on a shared RoCE subnet; Kai is credited upstream in that set for Engram-on-disk and the SM12x pages; bot-lab-21 produced the EXL3 3.5 bpw checkpoint and the exl3 config and MoE overlay that rides inside the patch set; Zeuss5 wrote cuda-exl3; LibertAI's TP2 REAP lane produced NaN logprobs with CUDA graphs on, which is why the boot checklist below has a logprobs check in it; and zyang-dev wrote the subnet-aware IB routing commit that makes a cabled ring work at all, which is upstream in NCCL from 2.30.7 and is the env NVIDIA's connect-three-sparks playbook runs its ring NCCL test with. What I added is the Kubernetes shape, a pinned image, and the ten boots it took to find out what actually breaks.

The fit

DeepSeek-V4.1-Flash is a 552B MoE. bot-lab-21's checkpoint quantizes the routed experts to EXL3 3.5 bpw and leaves everything else at the official FP8 and MXFP4 release. Split three ways, that loads as 79.5 GiB of weights per Spark text-only, and 82.5 GiB once the DSpark drafter is in. tonyd2wild reports 84.2 GiB per rank, which includes the vision tower he keeps and his DSpark drafter and I turn off with --language-model-only. The Engram n-gram tables are 189 GiB (203 GB) and are read per forward straight out of the safetensors shards on NVMe by the patched engram.py rather than loaded whole.

Three Sparks, not two, because 3 x 128 GB of unified memory is what holds the unpruned checkpoint at 3.5 bpw with room for a KV pool. The two-Spark route runs a REAP-pruned expert set instead, which is a different model, and it is also the lane where CUDA graphs produced NaN logprobs. Three ranks keep the model whole. They also mean the fabric has to be right, because 64 attention heads and 8 output groups do not divide by three: the patch set pads them to 72 and 9 at load time, which is a checkpoint-side config.json edit, not a runtime flag.

A ring is not a switch

Each Spark has two ConnectX controllers, four RoCE ports. The ring cables port 0 of each Spark to port 1 of the next one, all the way around. No switch, no subnet manager, six point-to-point legs across three cables.

NCCL will not bring that up by default, and the reason is structural. NCCL pairs NICs by channel index on both ends of a link: channel 0 uses local device 0 on the sender and device 0 on the receiver, channel 1 uses device 1 and device 1, and so on. On a switch that is always fine, because every device reaches every peer. On a P0 to P1 ring it can never be fine. Walk the ring: the local port that reaches your successor is port 0, the local port that reaches your predecessor is port 1, and your successor sees the same link arriving on its port 1. Index pairing is a two-colouring of the cycle, the cycle has three nodes, and an odd cycle is not two-colourable. Every channel is wrong on exactly one end. The result is ncclSystemError with any HCA ordering, any NCCL_CROSS_NIC value, and any number of ports listed. There is no combination of those knobs that works.

One more piece of the shape: the 2.5 GbE management NIC is on a different plane entirely. It is the out-of-band rendezvous path and nothing else. If the data plane ever lands on it you will not see an error, you will see a slow model.

The fix was already upstream

NCCL_IB_SUBNET_AWARE_ROUTING=1. It makes NCCL pick, per peer, the local IB device whose subnet matches that peer's, instead of pairing by index. On a ring that is exactly the right rule, and it is one environment variable.

It started as zyang-dev's dgxspark-3node-ring fork: one 141-line commit in src/transport/net_ib/connect.cc, dated 2026-03-13. It has been upstream since NCCL 2.30.7, and the official vLLM DSv4.1 arm64 base image ships exactly that, so it was in the library the whole time. NVIDIA's connect-three-sparks playbook runs its ring NCCL test with precisely this env. I could not find it in NCCL's documented environment variables, and a GitHub code search for the symbol returned zero hits at the time. What found it was strings libnccl.so.2 | grep SUBNET. Grep the library before you conclude a feature does not exist.

The env block that boots the ring:

env:
  - {name: NCCL_NET,                      value: IB}
  - {name: NCCL_IB_DISABLE,               value: "0"}
  - {name: NCCL_IB_SUBNET_AWARE_ROUTING,  value: "1"}   # the one that matters
  - {name: NCCL_IB_MERGE_NICS,            value: "0"}
  - {name: NCCL_NET_PLUGIN,               value: none}
  - {name: NCCL_IB_ROCE_VERSION_NUM,      value: "2"}
  - {name: NCCL_IB_ADDR_FAMILY,           value: AF_INET}
  - {name: NCCL_IB_TC,                    value: "106"}
  - {name: NCCL_IB_TIMEOUT,               value: "22"}

A good boot says so twice. Every rank prints its four ports and its out-of-band interface: NET/IB : Using [0]rocep1s0f0:1/RoCE [1]rocep1s0f1:1/RoCE [2]roceP2p1s0f0:1/RoCE [3]roceP2p1s0f1:1/RoCE [RO]; OOB enP7s7:192.168.1.28<0>. Then, with NCCL_DEBUG_SUBSYS=INIT,NET, it prints the substitutions: NET/IB: Subnet-aware routing: overriding dev 1 with dev 0. The rank-0 log of the first built-image boot has 22,147 of those lines. Without NCCL_DEBUG_SUBSYS set you get the first line but not the substitutions: NCCL_DEBUG=INFO alone does not print them.

The numbers

Fabric first, measured with a three-rank all-reduce over 256 MB buffers: 23.2 GB/s bus bandwidth on the ring with all four ports, against 11.3 GB/s on a single two-node leg with two ranks, and 7.9 GB/s for the Socket-over-nccl0 fallback that is kept documented and unused. The ring is doing its job.

Model numbers next. Everything below is single stream, text only, on unique prompts, on the built image at a 131,072-token context, four sequences, gpuMemoryUtilization 0.78. The right column is tonyd2wild's TP3 on a shared RoCE subnet, from his per-try table.

Measurement	Ring, three Sparks, no switch	tonyd2wild, TP3 on a shared RoCE subnet
Weights per Spark	79.5 GiB text-only, 82.5 GiB with the DSpark drafter	84.2 GiB, vision tower and DSpark drafter included
Prefill, 3.4k prompt tokens	1,094 tok/s	712 to 846 tok/s at his nearest prompt size, 2,950 tokens
Prefill, 49.8k prompt tokens	1,334 tok/s	1,236 tok/s at 93k (his no-speculation try 9)
Decode, no speculative decoding	26.5 tok/s	25.4 to 26.8 tok/s
Decode, DSpark k=5	31 to 35 tok/s on prose, 65 on code, 1.5x on a mixed six-prompt set	51.5 tok/s

The comparison that matters is row four. In the same configuration, no speculation and CUDA graphs on, the ring decodes at 26.5 tok/s and the switch decodes at 25.4 to 26.8. The ring is at parity. The gap in the last row is speculation and prompt mix, not fabric. Prefill on the ring is at or above his across the range, and the 49.8k figure is higher than the 3.4k one, which most likely reflects per-request overhead amortising over a longer prefill; it was not measured separately.

KV pool at the shipped configuration is 355,955 tokens, 2.72x the 131k context. Acceptance under DSpark k=5, measured across a probe, a quality set and a benchmark: 674 drafts, 3,370 draft tokens, 1,011 accepted, 30.0%, 2.50 tokens per step, with per-position acceptance 62 / 39 / 24 / 15 / 10%. One measurement to throw away: a repeated prompt that hits the prefix cache reads 38,626 tok/s of prefill. That number is real and means nothing.

Ten boots, and what each one cost

Attempt 1 took a node out. All three members came up, NCCL over RDMA connected first try, every rank loaded 79.47 GiB of weights. Then dgx1's kubelet stopped heartbeating while rank 0 was still loading. vLLM kept going, finished the load, and the other two ranks compiled their TileLang kernels for the KV-cache profile pass. Then silence. Minutes later the node still answered ping, but SSH timed out at banner exchange and log fetches hit TLS handshake timeouts: userland had starved. It needed a power cycle.

The previous boot's journal survived, 82,707 lines, and the striking thing in it is what is not there: no OOM-killer line anywhere. The node never killed anything. With vm.overcommit_memory=1 and a 16 GB /swap.img, the kernel paged kubelet and sshd out instead. On GB10 the CUDA allocations come out of the same 121 GiB of unified memory and are not charged to the pod cgroup, so the pod's 110Gi limit never engaged. A wedge, not a crash, which is the worse of the two because nothing in Kubernetes reports it.

The actual culprit was in the JIT cache directories. Under .cache/flashinfer/0.7.0rc1/121a/cached_ops, dgx2 and dgx3 had a finished .so for FlashInfer's mxfp8_gemm_cutlass_sm120, six cutlass variants, built during attempt 1. dgx1 had only the base .o: its compile had been cut off by the wedge. So dgx1 restarted it at the profile pass with MAX_JOBS=4, which is four cicc processes, the nvcc front end, at about 6 GB of host RAM each, on top of 79.5 GiB of pinned weights and the profile activations. Load average at the peak was 54. The spike was the compile, not the KV budget.

Three changes fixed it, and all three are now in the image or the sample:

• Prewarm the two modules in the image. sparse_mla_sm120 and mxfp8_gemm_cutlass_sm120 are JIT-compiled at build time under the exact runtime environment, because a JIT built with different nvcc flags is a different cache entry and will be rebuilt on the serving node. On the first built-image boot exactly one module still compiled at serve, trtllm_utils, pulled in by the mxfp8 autotuner.
• MAX_JOBS=1 at serve. If anything does compile on a serving node, it compiles one variant at a time. The image prewarms at MAX_JOBS=1 too, so the serve-time env matches the cache entry.
• swapoff -a on every member first. With swap off, an overrun becomes an OOM kill of the worker, which the operator restarts. Attempt 2 proved it: the kernel killed node_exporter, pipewire and dbus, and the node was back at 115 GiB available and Ready in 90 seconds with no power cycle. There is a Kubernetes wrinkle here worth naming: the operator sets requests equal to limits, so the pod is Guaranteed QoS with oom_score_adj of -997, and the kernel will kill every daemon on the box before it touches the worker. tonyd2wild runs --oom-score-adj 500 on bare metal, the exact opposite. Swap off is our equivalent for now.

Attempt 3 then failed cleanly, which was the useful kind of failure. At gpuMemoryUtilization 0.72 the engine reported Available KV cache memory: -1.29 GiB and exited without touching the node: 0.72 of about 121 GiB is roughly 87 GiB, and 79.47 of weights plus about 7 of activations plus 2 of graphs does not fit under it. The floor is around 88.5 GiB. 0.78 works and, on the 65,536-token no-speculation boot, leaves 7.4 GiB of KV (2.59 GiB and a 355,955-token pool at 131k with k=5), with 10 to 14 GiB of host memory available in steady state.

Attempts 5 through 8 were about speculation. DSpark k=5 is the production setting in tonyd2wild's launcher, and three things came out of testing it here. First, adaptive verification is unavailable on GB10: the worker raises at memory determination with Adaptive verification trims verification requests on device, which the DeepseekV4IndexerSM12xBackend attention backend does not support. On the SM12x sparse indexer, k is the only lever.

Second, k=5 versus k=3 is a workload choice, not a quality choice. On aggregate they tie: 1.44x for k=5 against 1.45x for k=3 over the same six prompts. They differ by prompt type. k=5 wins code, 64.9 against 51.0 tok/s, and arithmetic, 59.4 against 48.2. k=3 wins prose, 30.9 against 26.2, and lists, 35.4 against 31.7, and it leaves a much larger KV pool because the verification batch is smaller: 455,628 tokens against 337,677 at a 262k context. For an agentic single-stream target, which is mostly code, k=5.

Third, the quality question. DSpark uses block verification with a probabilistic draft, which is lossless in distribution, and the measurements agree with the theory without quite proving it byte-for-byte. Six prompts, temperature 0, seed 7, 200 tokens each: k=5 reproduced 2 of 6 outputs byte-identically against the no-speculation reference, k=3 reproduced 1 of 6 against both that reference and the k=5 run. Every divergence starts at a near-tie token, a curly apostrophe against a straight one, "specifications" against "specs", and continues as equivalent text, and the flips go in both directions. That is float-order noise from verifying six tokens per forward instead of one, which changes the batch shape and therefore the reduction order. Three extra essays at temperature 0.7, 0.7 and 0.0 decoded at 35.9, 34.9 and 35.4 tok/s, so temperature is not the lever either.

Boots nine and ten were the built image: the final rc4 candidate reproduced the numbers within noise (decode 31.6 and 37.8 tok/s, prefill 1,100 and 1,341), and then a 524,288-token context at utilization 0.80 came up with a KV pool of 849,026 tokens and 9 to 11 GiB of host headroom on the rank-0 node.

The declarative version

All of the above is one InferenceService. The part that encodes the ring is spec.multiNode, and it carries two planes that are easy to confuse.

multiNode:
  rdmaResource: rdma/rdma_shared_device_a
  ibGIDIndex: 3
  members:
    - node: ahazidgx1
      fabric:
        # bootstrap plane: rendezvous and the NCCL out-of-band handshake,
        # on the 2.5 GbE management LAN.
        address: 192.168.1.28
        socketInterface: enP7s7
        # data plane: all four RoCE ports. Subnet-aware routing picks the
        # right one per peer (NCCL_IB_SUBNET_AWARE_ROUTING=1 in spec.env).
        ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"
      modelCache: {claimName: dsv41-dgx1}
    - node: ahazidgx2
      fabric: {address: 192.168.1.86,  socketInterface: enP7s7, ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"}
      modelCache: {claimName: dsv41-dgx2}
    - node: ahazidgx3
      fabric: {address: 192.168.1.220, socketInterface: enP7s7, ibHCA: "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"}
      modelCache: {claimName: dsv41-dgx3}

address and socketInterface are the bootstrap plane. Give them the management IP and the management interface, not a fabric address, or rendezvous will try to ride the RoCE legs before NCCL has decided anything. ibHCA is the data plane: list every port and let subnet-aware routing sort them out per peer. That split is the whole trick, and it is the thing I would have got wrong if the NCCL test had not forced me to separate them first.

Around it: a Model that stages the 54 checkpoint files onto every member, either from an s3:// prefix with an explicit file list, and two Jobs the image deliberately does not bake, because both edit the checkpoint or the host rather than the container. One writes the TP3 head and group counts into the staged config.json, the other drops the page cache before the group boots, since vLLM's worker refuses to start when CUDA-free memory is below the utilization target and 429 GiB of freshly written files will hold it there. Then swapoff -a on each Spark, which is not yet something the operator does for you.

The sample is config/samples/inferenceservice_multinode_vllm_three_sparks_ring.yaml, with the Model and both Jobs beside it, plus a "Three-Spark ring" section in the multi-node guide: LLMKube pull request #1816, merged. The image is llmkube-runtimes #41, also merged: ghcr.io/defilantech/llmkube-vllm-cuda-gb10-dsv41-exl3, candidate digest sha256:8899a1285e156849f007848e78ef1cc0d31631664e780be0d53d998edeaa1fa3; the final candidate sha256:8dab469dd37e28c26612648079eb075eb3152dc2dacd8347c55a6f2aba037b54 (the same vLLM tree plus the cutlass, spdlog and cccl license texts) reproduced them: decode 31.6 and 37.8 tok/s, prefill 1,100 and 1,341. First green build on ubuntu-24.04-arm: 30 minutes; the FlashInfer source build and the cuda-exl3 compile account for about 4.5 minutes of it, the rest is the serial JIT prewarm and the image stages.

What LLMKube grows next from this is the obvious thing: the ring stops being a set of per-member fabric strings that a human has to keep consistent and becomes a first-class multi-node shape at the CRD level, so the operator knows the topology, derives the bootstrap and data planes from it, and can validate a member list before it schedules three pods that will fail at channel setup. Everything in this post is the hand-built version of that.

Supply chain

Every layer is pinned, and the pin is a digest or a commit, never a tag. The base is the official vLLM DSv4.1 arm64 image by digest, Apache-2.0. FlashInfer 0.7.0rc1 is rebuilt from source in place at 07869c61, Apache-2.0, because the base's 0.6.18 lacks the SM12x sparse-MLA decode shapes, and its CUTLASS, CCCL and spdlog submodules are pinned by SHA too. cuda-exl3 is 6a1ffc34, MIT, compiled for sm_121a. tonyd2wild's TP3 patch set is fc725ecf, MIT. One vLLM file, the DeepSeek-V4.1 nvidia model, is restored at e47aa780b, Apache-2.0, because the base's copy adds a DP-shared Engram path that imports symbols the patched engram.py does not define, and DP-shared Engram is irrelevant at TP3. The licenses across the tree are MIT, BSD-3-Clause and Apache-2.0. Nothing is AGPL.

The overlay replaces eleven files inside dist-packages, which is exactly the kind of thing that rots silently when the base moves. So the build carries a drift gate: a baseline of MD5 sums for every file the overlay touches, checked before anything is copied. If a future base image moves one of them, the build stops and names the files rather than quietly shipping an overlay written against a tree that no longer exists. Two in-image test suites then prove the assembled tree imports and loads without needing a GPU, which is what lets the whole thing build on a GPU-less arm64 runner.

Reproduce it

If you own three Sparks and cable them P0 to P1, the short version is: list all four ports in NCCL_IB_HCA, set NCCL_IB_SUBNET_AWARE_ROUTING=1, keep rendezvous on the management LAN, use an image whose FlashInfer kernels are already compiled, and turn swap off before you boot the group. Run NVIDIA's ring NCCL test first and do not start the model until it passes; on this ring a three-rank all-reduce measured 23.2 GB/s bus bandwidth. A three-hour checkpoint stage followed by a fabric failure at channel setup is an expensive way to learn that one environment variable was missing.

The image, the samples and the guide section are in the two pull requests linked above, both merged. If you are running DeepSeek-V4.1-Flash on Sparks and your numbers disagree with the table, I would like to know: the Discord is open, and every figure in the left column came off a boot I can point at; the right column is tonyd2wild's published table.