Full GLM-5.3 (glm_moe_dsa) on three DGX Sparks, as layers of the GLM-5.3-Flash CUDA engine: how should it sit beside #159? #402

What

We want to serve zai-org's full GLM-5.3 on three DGX Sparks with --tp 3,
one rank on each. The model (model_type glm_moe_dsa) has 78 layers of MLA with DeepSeek sparse attention, 256
routed experts and an MTP layer. Our recipe
MiaAI-Lab/GLM-5.3-EXL3-3x-DGX-Sparks-TensorFold
has served it since 2026-10-03, as patches on v0.6.0.

#159 (@drowzeys) serves the same model on four Sparks, with its own glm_moe_dsa family and TP=4 engine, and you have
said yes to it in principle. Two engines registering one model_type cannot both land, so we are asking how ours
should relate to #159 before we open a pull request. CONTRIBUTING asks for an issue first for a model family.

How it differs from #159

• Three Sparks, not four. Our checkpoint,
  Mia-AiLab/GLM-5.3-EXL3-2.75bpw-TensorFold,
  stores the routed experts in EXL3 at 2.75 bits a weight on average (each expert at its own width, 2 to 4 bits) and
  everything else in BF16. That is about 273 GiB, or about 86 GiB a rank.
• No new engine. Full GLM-5.3's layers run inside the GLM-5.3-Flash CUDA engine (families/glm5_next/cuda).
  They reuse its lane rounds and MTP verification, kept prompt states, server app, prompt-chunk row split and startup
  admission. A glm_moe_dsa family module (CUDA only) starts GlmEngine with a full-model config.

What patches 0100-0104 change

Each patch is one commit on v0.6.0, on top of the GLM-5.3-Flash recipe's 0001-0068. Together they add about 1,240
lines.

Measured

All on three DGX Sparks (GB10, 128 GB each, a triangle of direct ConnectX-7 cables), in NVIDIA's PyTorch container
26.07, with our checkpoint.

First boots (2026-10-03): 0100-0104, then 0105's prompt path. Settings: MTP drafts, NCCL, an FP8 latent
cache.

• Decode (boot p3, 163,840-token window): 28.3 tok/s prose, 33.0 code.
• Prefill: about 140 tok/s before 0105, about 290 with it (boot p3).
• tools/exact.py: 12/12 at 32k and 64k windows. Needles correct up to 58.2k tokens.

The recipe's release build (boot hcp, 2026-10-04, patches through 0141). This adds our prompt kernels, DSpark
and copy drafts, RoCE, context parallelism and an fp4 cache, at a 499,712-token window.

• Greedy decode: 31.3 tok/s prose, 41.7 code.
• Prefill: 601.5 / 606.5 / 598.4 / 591.2 tok/s at 4k / 8k / 16k / 32k.
• Needles correct at 94k and 314k tokens. tools/exact.py: 12/12.

Without context parallelism (boot dflt, FP8 cache). This boot ran at a 155,648-token window, the fallback
start.sh takes when free memory is short of the full window.

• Greedy decode: 30.6-32.6 tok/s prose, 41.7-43.2 code. Sampled: 26.5-27.8 and 31.2-33.1.
• Prefill: 630 / 679 / 660 tok/s at 8k / 16k / 32k.

Memory a rank (TensorFold's own estimate, which start.sh checks before every start):

• weights: about 85.3 GiB on rank 0 and 84.8 on ranks 1 and 2 (4-bit dense weights, MTP layer included);
• buffers: about 2.3 GiB;
• caches: 56.1 KiB a token with the FP8 latent cache, 94.4 KiB in bf16.

These are not like-for-like with #159's numbers: different Sparks count, checkpoint and drafter. tools/exact.py
sends each request drafted and with "draft": false, and several together and alone. It covers greedy and seeded
sampling, thinking on and off. A pull request would carry upstream's bench_concurrent.py --alone --serial,
bench_openai.py and prefill_cold.py.

What stays exact

• A drafted reply equals "draft": false (MTP verification in the shared lane rounds).
• A resumed prompt equals a fresh one.
• Every rank holds the same bits, because the ranks add fp32 partials in rank order.
• The prompt-expert kernels we would bring later store each pair's row and add them in slot order, with no atomics,
  so the same prompt gives the same bits every run. This is the issue @BHCC2025's run found on Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159.
• Replies at --tp 3 may differ from other rank counts, as the README says.

Two of the recipe's defaults are lossy and are not part of this proposal:

• 4-bit dense weights, quantized at load (the GLM-5.3-Flash recipe's patch 0005);
• the FP8 latent cache (GLM on CUDA: an opt-in FP8 or 4-bit latent cache (--kv-dtype fp8 / fp4), lossy, with paired quality results on full GLM-5.3 #401).

The base proposal serves BF16 dense weights and a bf16 cache. See question 2.

Depends on

• Mixed-width EXL3 experts. glm5_next/cuda/weights.py refuses mixed-bit encodes by name, although
  cuda/exl3/experts.py already takes a width per expert. A separate pull request for that is in preparation.
• More than two ranks (GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310, our proposal in this comment).

Possible follow-ups (not part of this proposal)

Each would get its own issue or pull request once the base has landed:

• context parallelism for a window of about 500k tokens (0112-0113, 0118, 0120, 0129, 0131, 0135, 0141);
• our own prompt kernels (0108 routed experts, 0110 sparse attention, 0111 tiles), and drowzeys' prompt-expert kernels
  with a deterministic mode added (0106);
• kept prompt states on NVMe (0132; compare CUDA glm5_next (TP / --parallel): kept prompt states never use the conversation spill tier (--spill-gib / --snapshot-dir are MLX-only and silently ignored) #373);
• --parallel 2 to 4 (0130, 0134, 0138, 0139; after feat(glm cuda): --parallel N on two ranks: one cache pool, batched verify rounds (each reply its solo bits) #243);
• drafts from Red Hat AI's DSpark speculator (0114; compare feat(qwen3.6 cuda): DSpark drafter (vllm-project/speculators dspark) on the DFlash v1 backbone #348).

Platforms

CUDA only: three DGX Sparks. There is no Mac engine for this model, and it does not fit on an RTX PRO 6000. The
layout takes four ranks, but we have not run four.

What we would submit, in order

1. Mixed-width EXL3 experts for GLM (the separate pull request).
2. The N-rank pieces (GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310).
3. The glm_moe_dsa base: 0100-0104 ported to main, as one pull request or a stack of five, with tests:
   • CPU: a tiny random GLM-5.3 checkpoint in the real layout (mixed-width EXL3 experts, full and shared indexer
     layers, MTP) on one rank and on three gloo ranks, against a reference forward written from transformers' code;
     drafted windows against serial steps, bit for bit.
   • GPU: the kernels, graphs against eager, and one layer of the real checkpoint's experts, on one Spark.
4. The prompt path (0105, 0107, then our kernels).
5. The cache formats (GLM on CUDA: an opt-in FP8 or 4-bit latent cache (--kv-dtype fp8 / fp4), lossy, with paired quality results on full GLM-5.3 #401), then the follow-ups above.

If you would rather build on #159's family, step 3 becomes three-rank support inside it (the rotating layout and
mixed widths), and we would bring our other pieces where they fit.

Questions

1. One glm_moe_dsa. Should it be Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159's engine, ours on glm5_next, or both (Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159's at four ranks, ours at three)?
2. Dense weights on three Sparks. BF16 dense weights cost about 9 GiB more a rank, and on our Sparks that left no
   useful window. Should that come from a checkpoint that stores those layers quantized, read as stored (like feat(glm5_next cuda): dense projections from an EXL3 pack (TF_GLM_DENSE_EXL3) #205's
   EXL3 dense projections), or from a load-time mode? The latter would be its own precision conversation.
3. Receipts. You have two Sparks. Would a full receipt from our three, plus the three-rank CPU tests and the one-GPU
   tests, be enough? These are the checks you asked of Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159.
4. Drafts. Is the checkpoint's MTP head enough for the first version?
5. Checkpoints. Ours and drowzeys' are both 2.75 bpw with a width per expert. We have not tried theirs on this
   engine. Should one family serve both?

Credit

• The model arithmetic follows the modelling code in Hugging Face
  transformers (Apache 2.0): GlmMoeDsa, DeepseekV32Indexer and
  DeepseekV3Attention. Our CPU tests' reference forward re-implements it. No code is copied from it.
• The sparse-attention design is DeepSeek-V3.2's, by DeepSeek-AI. That is the design only: no DeepSeek code is used.
• The top-k selection and score tiles adapt TensorFold's GLM-5.3-Flash pooled-key kernels (the radix select and its
  tie rule) to per-token keys.
• Routing GLM's EXL3 experts through the grouped kernels follows @akol1's glm5_next: route EXL3 experts through the universal path for any bit width #152.
• The context-parallel scheme in our follow-ups is drowzeys' design, from their TensorFold fork
  (drowzeys/TensorFold, branch glm-moe-dsa-tp4, which is Full GLM-5.3 on four DGX Sparks: glm_moe_dsa family, TP=4 CUDA engine #159's branch,
  commit befd47d, Apache 2.0). It works like this: tokens
  are interleaved over the ranks, local candidates are merged into the global top-k, and attention partials are
  merged by log-sum-exp in rank order. The code and kernels are ours.
• Patch 0106 carries drowzeys' EXL3 prompt-expert kernels from the same commit.