Repository navigation
GLM-5.3-Flash on CUDA: attention on one GPU, routed experts on the Sparks (proposal) #214
Description
On two DGX Sparks, each GLM-5.3-Flash decode step reads the routed experts and about 16.6 GB of BF16 non-expert weights out of LPDDR5x at roughly 230 GB/s. The non-expert weights are attention, the dense layers, the shared experts and the head, and each Spark reads half of them. The step also reads the KV cache. Plenty of Spark owners have a desktop GPU with about seven times that bandwidth on the same desk.
I'd like to add an attention/expert split to the GLM CUDA engine:
-
GPU node (
tensorfold serve … --experts remote): runs everything except the routed experts. That means the embeddings, the dense layers, KDA and DSA attention with the whole KV cache, the shared expert, the router, the head, DFlash2 and the server.- On a 32 GB card that is about 17.3 GiB of BF16 weights, 0.9 GiB for DFlash2 and about 10 GiB of state at a 262,144-token window.
-
Expert nodes (
tensorfold experts --rank 0|1): one process per Spark. Each holds a TP2 half of every routed expert (72.8 GiB each), cut with the existingsplit.pyrules and run through the grouped EXL3 kernel unchanged. -
Per MoE layer:
- the GPU sends the normed rows, with their top-k ids and weights, to both Sparks;
- each Spark returns its partial;
- the GPU sums the partials in rank order, the same rule
comm.pykeeps for bit-equal ranks.
The shared expert runs on the GPU while the exchange is in flight.
-
Transport: NCCL send/recv over RoCE, through the existing ctypes wrapper. GB10 and GeForce cards have no GPUDirect RDMA, so the traffic goes through host memory. Measured between an RTX 5090 and two Sparks with NCCL 2.30.7 defaults on one RoCE plane, eager (not graph-captured), one MoE layer's exchange takes 88 µs p50 at 1 row, 102 µs at 4 rows and 142 µs at 8 rows; a graph-captured exchange was slower (140 µs at 1 row).
The cut is small. moe_block in families/glm5_next/cuda/forward.py already routes, calls exl3_generic.routed and gathers. In remote mode that call becomes the exchange.
Why I think it's worth it. The split already runs on this hardware in a separate engine: hughmadden/glm53f-afd (Rust), on one RTX 5090 plus two Sparks over ConnectX-7.
| Setup (same harness) | Prose | Code | JSON | 4 streams, total |
|---|---|---|---|---|
| glm53f-afd, RTX 5090 + 2 Sparks | 73 | 84 | 76 | 117 |
| TensorFold 0.5 GLM recipe (MiaAI-Lab), another pair of Sparks | 58 | 66 | 79 | 91 |
| Stock 0.6.1 TP2, our two Sparks, same checkpoint (MTP off, 98K window: 262K does not fit) | 34.7 | 39.4 | 38.7 | 38.4 (one request at a time) |
Figures are tok/s, single stream unless noted.
I'd rather run this inside TensorFold, with its kernels, drafter and server, than keep a separate engine going.
Plan.
- First PR scope: GLM-5.3-Flash, EXL3 experts, CUDA, one stream at a time. A startup check refuses an expert node whose checkpoint doesn't match the GPU node's.
- Gates:
- identical replies run to run;
- KL against your TP2 path on the same checkpoint;
- decode and prefill numbers against TP2 on the same two Sparks.
- Batching, and overlapping two lanes so the GPU and the Sparks are busy at once, would come after.
If you'd rather build it your own way, the branch and the numbers are yours to use. Would you take a PR along these lines?
Credits.
- The attention/expert split on this hardware is Hugh Madden's (@Dangerm00se): glm53f-afd and mimo26f-afd, built on tpurtell's (@wrldsuksgo2mars) ds41rt and glmrt.
- MiaAI-Lab's GLM work on TensorFold showed what two Sparks can do.
- Written with the Hermes agent: Claude Opus 5.5 for the design, GLM-5.3-Flash for the builds and benchmarks.
Activity
ashhart commentedon Oct 2, 2026
Thanks for the detailed proposal. Splitting one model across machines is the job of mcdma (mcdma.dev), which handles cross-machine prefill and decode; TensorFold stays on one machine or two ranks of the same kind. We'd rather point this there than build a second split path. Closing here.
ashhart commentedon Oct 2, 2026
Reopening this while we decide where cross-machine splits should live. Sorry for the noise.
squarrier commentedon Oct 2, 2026
No worries about the noise. Since you pointed at mcdma, I ran the split over it end to end, so you have numbers for that route while you decide.
One RTX 5090 runs attention, the shared experts, KV and the drafter, and two Sparks each hold half of the routed experts. Each MoE layer sends its routed rows to both Sparks and adds up the two partials. That exchange now runs on mcdma's link daemon (mcdma-rpcd at e672c14, unmodified), with the connect side on the 5090 box and a listen side on each Spark. The checkpoint, DFlash2 drafter and harness are the same as the issue's table; figures are medians of 3 runs:
Stock TP2 comes from an earlier session on the same harness. It ran at a 98K window because 262K doesn't fit on two Sparks; the split runs at 262K.
Greedy replies over mcdma are byte-identical to the NCCL route's (5 of 5 prompts, same session), seeded replies match too, and drafted output matches undrafted. Against stock TP2 the split stays within bf16 noise. Teacher-forced over 1,866 positions, it gives mean KL 0.0026 nats, p99 0.043 and top-1 98.98%, and every disagreement is a near-tie. I read that as rounding from running attention and the shared experts whole on one GPU instead of in two TP halves.
The 5090 side is GPU-driven. Stream-ordered copies fill the registered mailbox, cuStreamWriteValue64 rings it, and a cuStreamWaitValue64 on each link's done word holds the stream until both partials land, so the CPU never waits on the GPU per layer. There's no NCCL communicator in this mode. A CPU-driven version that synced every layer was 3.5–3.9% slower than NCCL, because the sync stops the CPU queueing ahead.
The gain is small because the wire isn't where the time goes. At one row, a layer's round trip is 300 µs over mcdma against 320 µs over NCCL (the NCCL figure is from an earlier profile). About 258 µs of either one is expert compute on the Sparks, which leaves 44 µs against 60 µs for transport and queueing. The daemon on its own does a one-row round trip to both Sparks in about 10 µs, or 26 µs including the 5090's host copies, with 21,000 calls per link and no failures.
Limits: the mcdma path is eager only (CUDA graphs run on the NCCL path but bought just 0.4%), it serves one stream at a time, and it covers GLM-5.3 on CUDA only.
The engine side is about 930 lines across 14 files on 0.6.1, and it rebases cleanly onto 0.6.2. It has three parts: a world-1 loader that skips the routed experts, an experts-only node running the existing grouped EXL3 kernel on a
split.pyhalf, and one call inmoe_blockbehindbegin/exchange/collect. The mcdma backend adds about 1,200 lines of Python (ctypes on libmcdma-rpc and the CUDA driver), plus tests.Build note:
rpc/fails with-Werror=format-truncationatrpcd_connect.c:42on Linux GCC 11.4 and 13.3. It's a false positive, sincenameischar[32]. @ThinkOffApp's #5 fixes it, and that patch built and passedmake -C rpc testfor me on x86_64 Ubuntu 24.04. The run above used e672c14 with only that warning made non-fatal.#213 cuts by layer range, while this cuts inside each MoE layer, so the two seem to share the transport question more than the engine hook.
Any of these works for me:
integrations/vllm;Same setup as the issue: written with Hermes, Claude Opus 5.5 for the design and GLM-5.3-Flash for the builds and benchmarks.
ashhart commentedon Oct 6, 2026
Thank you, @squarrier. Running the split over mcdma and posting both routes side by side made the trade clear. Here's where cross-machine splits now live. The Python engine is frozen (#286), and the Zig engine has a cluster layer that places one model over several machines through mcdma links (
zig/src/cluster). Its planner splits by tensor, pipeline and expert degrees, with every sum in one fixed order. An attention-on-one-GPU, experts-on-the-Sparks layout would be a new placement in it. GLM-5.3-Flash isn't on the Zig engine yet, and Zig CUDA doesn't serve yet, so this waits for both. Your finding that expert compute, not the wire, sets the round time is the reference for that design, with your credit.ashhart commentedon Oct 7, 2026
Thank you, @squarrier. Attention on one GPU and the routed experts on the Sparks is a real split for GLM-5.3-Flash, and it wants two commands. One serve process for attention, one expert process for the routed experts, and a transport between them.
The Zig tree has no GLM engine and no remote-expert transport. The single-box GLM port comes first. This split follows it, and it uses the same expert weights, not a second quantisation. A failed expert process has to fail the request, not hang the attention rank. Thank you for writing it as a proposal with the two roles named. We are not starting the transport ahead of the model.
ashhart commentedon Oct 10, 2026
Thank you, @squarrier. Attention on a desktop GPU with the routed experts on the Sparks is listed in #593 as a later piece, once the family and tensor parallel land, so it builds on their exchanges. Closing in favour of #593.