Repository navigation
GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310
Description
What
I have a branch that serves GLM-5.3-Flash with --tp 2 or --tp 3, for three DGX Sparks. Before opening anything, I
would like to agree with you and the authors of #159, #219 and #243 on how it should fit, since all four touch the
same places.
What the branch has
- One shard plan decides every rank's share. Shares are whole units (heads, blocks of a width) and can be unequal:
the firstunits % worldranks take one more. A rank folder records the rank count it was split for, and a
mismatch is refused. - Vocabulary gathers over shards of unequal width.
- A rank loads only the layers it needs.
- Rank exchanges and
--tpfor two or three ranks. A prompt chunk's exchanges can use an exact reduce-scatter: each
rank glues only its own rows, and the sums keep rank order, so all ranks keep equal bits. - Tests run three ranks on one GPU (a thread-backed communicator) with the tiny model cut as the real one, and three
ranks of the real checkpoint against the uncut layers.
GLM-5.3-Flash's routed experts have moe_intermediate_size 2048. Over three ranks that is not a whole number of
32-wide tiles, so the unequal shares above are needed.
Where it meets other work
- feat(cuda): one communicator interface for two-rank engines (transports over NCCL) #219 (in 0.6.4): engines open
open_comm(rank, 2, …), andCommpromisesall_gatherandbarrier, with
exchangeandfast_gatheroptional. My branch adds a reduce-scatter and peer exchanges for three ranks. Should
those be optionalCommmethods with a fallback, asexchangehas, and shouldopen_commtake the world from
--tp? - Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159 (drowzeys, full GLM-5.3 on four Sparks): a separate
glm_moe_dsafamily with its own TP=4 engine. Is the
plan to keep GLM-5.3-Flash's engine at its own rank counts, or to share one N-rank layer (shard plan, exchanges)
between the two families? - feat(glm cuda): --parallel N on two ranks: one cache pool, batched verify rounds (each reply its solo bits) #243 (MiaAI-Lab,
--parallel Non two ranks): rank 0 sends rank 1 one message a round, and the cache pool is
laid out for two ranks. Should three ranks wait until feat(glm cuda): --parallel N on two ranks: one cache pool, batched verify rounds (each reply its solo bits) #243 lands and then build on it, or land first with
--parallelrefused at--tp 3?
What I can measure
Three DGX Sparks (GB10), the MLX 4-bit checkpoint and nvidia/GLM-5.3-Flash-NVFP4: drafted against serial by token
hash, --tp 3 against --tp 2 on the same prompts, decode, and cold prompts.
Written with Weave (My Symbiotic AI)
Activity
MiaAI-Lab commentedon Oct 4, 2026
@Bizuayeu, for reference while this is decided: the GLM-5.3-Flash recipe has carried
--tp 3since v1.4 (2026-10-03), as patches on v0.6.0 under Apache-2.0:0066-glm-tp-n,0067-glm-tp3-split-padand0068-glm-tpn-split-buffer-rows.They handle the points you list this way:
moe_intermediate_size2048 over three ranks);Measured on three Sparks: concurrent == serial 22/22, drafted == serial, the 195k needle. Happy for any of it to be used or compared against your branch. If code from them goes in, please credit the recipe, as #309 and #342 do.
Bizuayeu commentedon Oct 4, 2026
@MiaAI-Lab, I'm a fan of your recipes, so it's a real pleasure to talk with you directly!
Thanks, that is useful to have side by side. My branch (on 0.6.4) already follows three points of these patches and
credits them in its
THIRD_PARTY_NOTICES.md, with no code taken: the doorbell key per follower is 0066's fix for thesame wait, and the uneven row split at three ranks follows 0067 and 0068. The share rule is 0066's too: whole units,
the remainder on the lowest ranks (heads 22/21/21; the MoE width and the vocabulary in 64-wide units, 704/704/640 and
51,648/51,648/51,584).
Where it differs: the exchanges go over NCCL with no proxy, and a prompt chunk's partials can take an exact
reduce-scatter after #159's, so each rank glues only its own rows and all ranks keep equal bits. On three Sparks with
nvidia/GLM-5.3-Flash-NVFP4it repeats its decode-check hashes from launch to launch, prefills 38,960 tokens at about1,670 tok/s, and answers at three positions of a 1,036,859-token prompt.
If code from your patches goes in, it will be credited the way #309 credits the recipe. The question here still stands for the
maintainer: how three ranks should sit beside #219's
Comm, #159 and #243.Written with Weave (My Symbiotic AI)
MiaAI-Lab commentedon Oct 4, 2026
Proposal: CUDA: GLM over three or four ranks (
--tp 3/--tp 4): one share rule, N-peer exchanges and a doorbell per followerWhat
GLM on more than two CUDA ranks, one rank a machine. Today
--tptakes 1 or 2, and the GLM, Flash Next andNemotron engines all open
open_comm(rank, 2, master, port). Our GLM-5.3-Flash recipe has served--tp 3on threeDGX Sparks since its v1.4 (2026-10-03), with patches
0066-glm-tp-n,0067-glm-tp3-split-padand
0068-glm-tpn-split-buffer-rowson v0.6.0. Full GLM-5.3 needs three ranks (about 86 GiB of weights a rank; see #402).
#310 asks how three ranks should sit beside #219's
Comm, #159 and #243. This issue proposes a concrete answer fromwhat we run, so that it can be settled once for everyone working on it:
--tp 3): how it should sit beside #159, the communicator interface and #243 #310: @Bizuayeu's three-rank GLM-5.3-Flash branch, which already follows 0066-0068's share rule;--tp 4for the 27B and Flash Next, with rank sums of two to eight ranks;How it would fit main
--tpfrom the family. A family declares the rank counts it serves (for exampleCUDA_TP = (2, 3)forGLM-5.3-Flash). The CLI refuses any other count when it parses the arguments, as you did for Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159's
--tp 4.--rankruns from 0 to N-1, and the engines callopen_comm(rank, args.tp, ...). That answers GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310's question:yes, the world size comes from
--tp.tensorfold/cuda/shares.py. A dimension splits in whole units (a head, a 128-columnHadamard block, 64 vocabulary rows), as evenly as it goes, with the remainder on the lowest ranks. Over three
ranks, 64 heads split 22/21/21, and 2,048 expert columns in 128-wide blocks split 768/640/640. Nothing is padded:
the ranks exchange full-width fp32 partials and top-k candidates, never per-head tensors, so ranks may hold
different shapes. Two ranks of an even count get exact halves, so today's two-rank shards do not change. GLM-5.3-Flash on CUDA over three ranks (
--tp 3): how it should sit beside #159, the communicator interface and #243 #310'sshard plan and feat(qwen4_exp cuda): Flash Next on four ranks (--tp 4), each rank's sums over the shared rank-sum transports #379's
ranks.shareuse the same rule, so one module could serve every family.The glue kernels already take
WORLD. Every rank keeps the same bits at any N, and decode needs no newcollective.
exchange_all(sends, recvs)as an optionalCommmethod. It runs one NCCL group with several peers. Aprompt chunk split by rows uses it as an exact reduce-scatter. Each rank receives every peer's partial of its own
rows and adds them in rank order. It then glues those rows, runs the next norm on them, and sends the normed bf16
rows to every peer. That is the same idea as the reduce-scatter in Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159 and GLM-5.3-Flash on CUDA over three ranks (
--tp 3): how it should sit beside #159, the communicator interface and #243 #310. One group with every peer needsevery pair of ranks connected: three Sparks in a triangle of direct cables are.
comm.pywould get a fallbackthrough
all_gather, asexchangehas (today that fallback covers two ranks only), so aTransportdoes not haveto implement it. perf(cuda): rank sums over an engine's communicator; prompt-sized partials by copies through shared host memory #378's rank-sum transports fit beside it.
engine.py, one store key that rank 1 waits on) becomesone key for each follower, and rank 0 rings each of them. Bizuayeu's branch carries the same fix.
split.pynow checks only
1 - rank). DFlash2's KV groups and the vocabulary split by the same rule.memory estimate counts it (0067, 0068).
rocebackend throughregister_backend, as feat(cuda): one communicator interface for two-rank engines (transports over NCCL) #219 expected. It is the"RoCEnante" transport of b12x by local-inference-lab (Apache 2.0):
its C proxy, extended here to per-peer routes over the network cards that share a subnet with each peer (three
Sparks in a triangle of direct cables), and its all-gather kernel, rewritten in CUDA C++. NCCL stays the default.
--parallelabove two ranks. Our recipe decodes up to 8 streams at three ranks, but upstream's GLM--parallelis feat(glm cuda): --parallel N on two ranks: one cache pool, batched verify rounds (each reply its solo bits) #243 (ours, two ranks, in review). We would land N ranks with--parallelrefused above tworanks first, then lift that after feat(glm cuda): --parallel N on two ranks: one cache pool, batched verify rounds (each reply its solo bits) #243.
Evidence
Our GLM-5.3-Flash recipe on three DGX Sparks (v1.4 patches, NCCL, 2026-10-03, two boots):
Measured with sparkDash on the same recipe:
At v1.5's eight streams on three Sparks, the totals were 166.0 tok/s prose and 211.5 code, and replies were identical
to one at a time (11/11). Two Sparks run as before: their buffers, memory estimate and connections are unchanged.
Full GLM-5.3 runs only at
--tp 3. Every result in #402 is three-rank, withtools/exact.pyat 12/12 on everyboot. There, RoCE all-gathers decode about 10% faster than NCCL with the same replies (boot p15 against NCCL boots,
MTP drafts).
What stays exact
At any N, the ranks add their fp32 partials in rank order, so every rank holds the same bits. At a given N, a drafted
reply equals
"draft": falseand concurrent replies equal solo ones. Replies may differ between rank counts, as theREADME already says. At two ranks, the shards and sums are today's; the receipt would show the same
token_shaat--tp 2before and after.Platforms
CUDA only: DGX Sparks (GB10), at two and three ranks. The layout takes four (
tp.SIZES = (2, 3, 4)), but we have notrun four. Not run on a box with several GPUs over PCIe.
What we would submit, in order
shares.py,--tpfrom the family,open_commwith the world size from--tp, a doorbell per follower, andstartup settings compared across all ranks. Nothing changes at two ranks.
--tp 3: the layout, loader, row split withexchange_alland its fallback, and the pad row.rocebackend (b12x's transport), if you want it upstream.glm_moe_dsa) on three DGX Sparks, as layers of the GLM-5.3-Flash CUDA engine: how should it sit beside #159? #402).If you prefer Bizuayeu's branch for GLM-5.3-Flash at
--tp 3, we would review it and test it on our three Sparks,and bring only what it lacks.
Questions
shares.py, theCommadditions) serve GLM-5.3-Flash, full GLM-5.3, and feat(qwen3.8 cuda): --tp 4 for Qwen3.8-27B; rank sums of two to eight ranks #377 / feat(qwen4_exp cuda): Flash Next on four ranks (--tp 4), each rank's sums over the shared rank-sum transports #379'sQwen engines?
exchange_all, and a reduce-scatter built on it, as optionalCommmethods with fallbacks?--parallelrefused above two ranks), or after it?checkpoint on three gloo ranks against one, or three ranks as threads on one GPU, as in Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159?
Credit
The one-shot RoCE all-gather is b12x's "RoCEnante" transport by local-inference-lab
(local-inference-lab/b12x, commit 8a99d63, Apache 2.0). Its proxy
(
roce_proxy.c) is carried with its header, modified for more than two Sparks; its CuTe all-gather kernel isrewritten in CUDA C++. The rest of 0066-0068 is Mia's AI Lab's (Apache 2.0).
Bizuayeu commentedon Oct 5, 2026
@MiaAI-Lab, thanks for writing this up. It answers the three questions #310 asked, and from my side I'd be glad to see
one N-rank layer serve GLM-5.3-Flash, full GLM-5.3 and the Qwen engines (your question 1), with
exchange_allas anoptional
Commmethod that falls back toall_gather(2), landing before #243 with--parallelrefused above tworanks (3).
One correction to my last comment: my branch follows 0066's rule, but not its unit. It takes an MLP width and the
vocabulary in 64-wide units (NVFP4's 16-input groups, MLX's 64), so 2,048 expert columns split 704/704/640, where 0066
takes 128-column Hadamard blocks and splits them 768/640/640. A shared
shares.pycould take the unit from thecheckpoint's format, or use 128 everywhere, since it is a multiple of what NVFP4 and MLX need, at the cost of a less
even split. My branch has not been run on EXL3 at three ranks.
To avoid two implementations of the same thing: if your step 1 lands, I'll rebase GLM-5.3-Flash at
--tp 3onto it,or review your step 2 instead if that is preferred. Either way I can post receipts from three Sparks with
nvidia/GLM-5.3-Flash-NVFP4and the MLX 4-bit checkpoint: drafted against serial by token hash,--tp 2unchangedbefore and after, decode and cold prompts. The NVFP4 receipts need the loader from #308.
Whether a RoCE transport goes upstream is the maintainer's call. My branch uses NCCL only, so I have no measurements
to add there.
ashhart commentedon Oct 6, 2026
Thank you, @Bizuayeu. Thanks also to @MiaAI-Lab for the detailed proposal in this thread. The Python engine is frozen (#286), so
--tp 3won't land on 0.6.x. The Zig engine already has the N-rank layer you both asked for, shared by every family. Its placement planner (zig/src/cluster/plan.zig) splits shares by largest remainder with ties to the lower rank, which is your whole-units rule. Every sum follows one fixed tree of slices however the slices fall over ranks, so all ranks hold the same bits (zig/src/cluster/canon.zig). GLM-5.3-Flash isn't on the Zig engine yet, and Zig CUDA doesn't serve yet. When both are there, your branch and the recipe's patches are the reference for three ranks, with your credit.ashhart commentedon Oct 7, 2026
Thank you, @Bizuayeu. Three ranks for GLM-5.3-Flash, on the communicator the other multi-rank work already uses, is the layout we want. A private third transport would be the wrong outcome.
The Zig tree has no GLM engine and no three-rank serve. The Flash engine is the port, and --tp 3 is part of that port rather than a flag we turn on ahead of the model. A request for three ranks on a build that has no such engine has to be refused. Thank you for tying this to the existing communicator work so it does not become a third stack.
ashhart commentedon Oct 10, 2026
Thank you, @Bizuayeu. Your shard plan with unequal shares and the exact reduce-scatter in rank order are steps 2 and 3 of #593, which now tracks all GLM work on native CUDA. Your TP=2 port over NCCL is one of the candidates for the first family PR there. Closing in favour of #593.
app: declare the engine hook before App's bootstrap early return
app: declare the engine hook before App's bootstrap early return
app: declare the engine hook before App's bootstrap early return
app: declare the engine hook before App's bootstrap early return
Maintainer fix-up for #1122: hook order, the port documents, cancelle…