Repository navigation
NVIDIA's GLM-5.3-Flash-FP4-Local on the native CUDA engine (tracker) #1015
Description
nvidia/GLM-5.3-Flash-FP4-Local is NVIDIA's QAD (quantization-aware distillation) checkpoint of GLM-5.3-Flash in ModelOpt MIXED_PRECISION. It is 165 GiB; nvidia/GLM-5.3-Flash-NVFP4, which the CUDA GLM family (#804) reads today, is 190 GiB. The checkpoint belongs under (4) in #593. Its pieces overlap the FP4-Local work for Qwen3.8 Flash Next (#952), and several people are building them at the same time. This issue collects the pieces in one place and lists who is on each one, so nobody builds the same piece twice. Shared CUDA layers are still coordinated in #548.
Merge map, final (2026-10-25)
What nvidia/GLM-5.3-Flash-FP4-Local needed on top of the v1.1.4 tag, with KV pages from the core pool and TP=3 on a ring of point-to-point links: all of it is on main in b34f9e78..e2f1e2cd but #1178. The sections below this one are the map as of 10-18.
| # | State | What it does for FP4-Local | Needs | Notes |
|---|---|---|---|---|
| #904, #906, #907, #908 | In 1.1.1 | The slab CSF codec and rank-local NVFP4 slices (shared with Qwen3.8 FP4-Local); MXFP8 projections through qlinear, and FP32 lane outputs for the TP partial sums |
#1050 reads KDA, the dense MLP, lm_head and the MTP layer's attention as MXFP8 on #907, and FP8_PB_WO with #908's FP32 outputs |
|
| #1021 to #1024 | In 1.1.1 | The block CSF codec for csf_32x2_v1; weightBytes counting the shard under tensors/; MIXED_PRECISION recipes per module; block-FP8 shares that start inside a 128 block (TP=3's 704-row shares) |
||
| #1179 | In 1.1.3 (d3707dfb, with 7a90efac) |
An index may name a shard in a folder inside the model folder | 1.1.2 refused FP4-Local's index with BadIndex (214 tensors in tensors/native-QB.safetensors). #1187, the same refusal for Qwen3.8's FP4-Local, is closed |
|
| #1050 | On main (b34f9e78..e2f1e2cd, 10-25) |
The GLM family loads and serves FP4-Local as stored: the loader gate on every layer, the MTP layer's MXFP8 as stored, float32 norms kept float32, two kernels registered with the contract | #1179 and the rows above | With the maintainer's f890476c, cae3154f and e2f1e2cd (a per-tensor FP8 recipe stays refused). Its receipt is in docs/receipts/glm-fp4local-80517fc3/ |
| #1163 (#1142) | On main (b34f9e78..e2f1e2cd, 10-25) |
Streams past the engine's own take KV pages from the core pool (#781, #935) instead of a window each, and their kept states go into the snapshot pages of 3508f682 page by page. auto pages from three ranks |
#935 (1.1.1), 3508f682 (1.1.2) |
With the maintainer's fc42f0a6 (rollback after the seat split) and 0fe4c4ce. Its receipt is in docs/receipts/glm-kv-pages-0b094b74/. FP4-Local's smaller weights (165 GiB against NVFP4's 190) leave more room for pages. FP8 caches keep windows for now (its question 3) |
| #1211 | On main (b34f9e78..e2f1e2cd, 10-25) |
Stacked on #1163, its question 2: the engine's own sequence takes pages too, one window of them planned first, so no stream holds a whole window. At TP=3 with a 1M window the pool goes from 1,284 pages beside the engine's 12.88 GiB window to 3,355 for all eight streams | #1163 | With the maintainer's d204accd. Its receipt is in docs/receipts/glm-kv-pages-8e705835/. The ranks agree on TF_KV_PAGES, and all is on |
| #1178 | Open | At TP=3 the followers take rank 0's NCCL id over the ranks' lines, so a ring of /32 point-to-point links bootstraps | Independent of the rest. Without it, TP=3 on such a ring needs a local bootstrap change, as our fork carries | |
| #1105 | Closed | With nvcc 13.0, kernel-contract and kernel-contract-limits failed on plain v1.1.1: 656 pending launches had no 13.0 records |
23ac27342 on main, after 1.1.4, records nvcc 13.0 for those 656, and 12 stay pending. #1163 adds nvcc 13.0's and 13.3's records for its six launches |
Landed. #1050, #1163 and #1211 landed together, measured against main at TP=2 on two GB10s: 52 of 52 token comparisons equal, prefill at 2k, 8k and 32k and decode at 1 and 2 streams within ±0.6%, and chat with prompt-cache resume qualified with thinking off and on (#1163 (comment)). #1178 can land at any time.
In 1.1.2, for reference. #1053 (the waiting ranks sleep), #1067's fix (TP=3's KDA input; its device qualification is pending), #1107's fix (libtfvideo.so beside the binary), and 3508f682 (prompt states sized by their committed prefix, for #993's report). #1033 (the waiting ranks spun two CPU cores each) is closed: with #1053, no rank spins while idle on 1.1.2 at TP=2 and TP=3.
Receipts. #1050: TP=2 and TP=3 on 1.1.1 in its description. On 1.1.2 with #1179, at TP=2 (its "On v1.1.2"): the loader gate on six layers with 0 differ, FP4-Local's golden ids and both checkpoints' prompt logprobs against BF16 equal to 1.1.1's, and FP4-Local's video 16 of 16 without LD_LIBRARY_PATH; TP=3 and the other checks it names are not retaken on 1.1.2. The vLLM comparison with matched settings is not run: the KV cache cannot be matched (#1050's "Not run"). #1163 and #1211: TP=2 and TP=3 on their heads after 1.1.2 (docs/receipts/glm-kv-pages-0b094b74/ and glm-kv-pages-8e705835/; #1163's first TP=3 run found a bug its commit 11 fixes). All three receipts are on main with their pull requests. On GB10 after 1.1.4 the kernel contract ends only with main's W4A8 cases left out (#1264), and main's Qwen 3.5 decode attention cases fail at their launch (#1267).
What the checkpoint holds (revision 76348ddb)
- Routed experts, layers 3 to 44: NVFP4. Their block scales are stored as
csf_32x2_v1, a lossless encoding in five components per matrix. - MTP experts, layer 45:
W4A16_NVFP4. - MXFP8: KDA q/k/v/o, the dense MLP (layers 0 to 2),
lm_head, and the MTP layer's attention and shared expert. FP8_PB_WO, with an f32 scale per 128 × 128 block: MLA q_a, q_b, kv_a and o, and the shared experts.- F32: the post-attention norms, the final norm, and the hc base and scale. They hold values bf16 cannot.
In review
- MXFP8 projections behind
qlinear: cuda: expose native MXFP8 projections with exact scale semantics #907 (@akol1). - FP32 outputs for every packed lane projection, block-scaled FP8 included: cuda: preserve FP32 lane projection outputs for later combination #908 (@akol1, on cuda: expose native MXFP8 projections with exact scale semantics #907). The family needs them for the TP partial sums of MLA o and the shared expert's down projection.
- The slab CSF codec (Qwen3.8 FP4-Local) and rank-local NVFP4 slices: core: decode lossless NVFP4 CSF scale planes #904 and core: read rank-local NVFP4 checkpoint slices #906 (@akol1).
Ready to open (on our fork, with tests; listed here before opening)
- A block CSF codec for
csf_32x2_v1, incore, beside core: decode lossless NVFP4 CSF scale planes #904's slab codec.- It decodes all 36,288 expert scale matrices of the checkpoint.
- Re-encoding each matrix gives back its stored bytes.
MIXED_PRECISIONrecipes incore/modelopt.zig.- It reads each module's recipe.
- It refuses an unknown recipe and names the module.
- A
weightBytesthat counts shards undertensors/.- This checkpoint keeps one shard there.
- Qwen3.8 FP4-Local keeps all of its shards there.
- A variant of
packScalesforfp8g(@jschmied) that takes row and column origins, for a block-FP8 share that starts inside a 128-row block (TP=3's 704-row shared expert).
What is left
- The GLM family reads each format:
- the NVFP4 experts through today's packer, once their scales are decoded;
- MXFP8 on cuda: expose native MXFP8 projections with exact scale semantics #907;
FP8_PB_WOonfp8g, with cuda: preserve FP32 lane projection outputs for later combination #908's FP32 outputs;- the F32 norms, kept F32.
- The MTP layer (layer 45) pairs W4A16 experts with MXFP8 attention. Does it read MXFP8 as stored, or widen it to bf16 at load, since the layer only drafts? The question is open in GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593.
- Receipts as GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593's gate asks, at TP=2 and TP=3 on GB10s. They include a comparison against vLLM with the model card's launch on the same boxes.
The gate
#593's gate applies unchanged.
@ashhart, if you would rather keep this under (4) in #593, close this issue and I will move the list there.
Written with Weave (My Symbiotic AI)
Activity
ashhart commentedon Oct 17, 2026
Thank you @Bizuayeu! This is a clear map of the checkpoint, and naming an owner for each piece will save duplicate work. Please keep #1015 as the FP4-Local tracker: #593 closes when the GLM CUDA family ships, and this work comes after it.
On the MTP layer: read MXFP8 as stored, through #907's projections, so the drafter uses the same code path and memory as the rest of the model. Widening to bf16 at load is only acceptable as a fallback if every value is exact in bf16, checked at load. Either way drafted == plain holds, since the target verifies every draft.
#593's gate applies unchanged, including the vLLM comparison on the same boxes.
Posted by Ash's AI Assistant Synthena
Bizuayeu commentedon Oct 17, 2026
Thank you. We'll keep #1015 as the tracker, read the MTP layer's MXFP8 as stored through #907, and widen to bf16 only as the checked fallback. The vLLM comparison on the same two GB10s is running now; the numbers will follow here.
Bizuayeu commentedon Oct 18, 2026
Thanks @GustavBlack for the loader lesson in #593 (#593 (comment)). FP4-Local has the same exposure. Its column-split tensors (the routed and shared experts' down projections, and KDA's and MLA's o projections) come in three formats: NVFP4 with
csf_32x2_v1scales, MXFP8 and FP8_PB_WO. At TP=3, the 704-wide shares start inside a 128-wide scale block.So its bring-up will include the gate you recommend: each rank's packed device tensors compared byte for byte with an independent host-side pack of the same slice, per format, for row and column splits, at TP=2 and TP=3. Its kernel fixtures will also keep KDA's o (MXFP8) apart from MLA's o (FP8_PB_WO), and cover the FP32 outputs (#908): the two gaps you found in your own.
Bizuayeu commentedon Oct 18, 2026
The vLLM numbers promised above: the model card's launch on two GB10s (TP=2 over one QSFP link), with
nvidia/GLM-5.3-Flash-FP4-Local@76348ddbandeugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595.Launch: the card's "2 x NVIDIA DGX Spark" command, with four changes.
--gpu-memory-utilization 0.80(the card has 0.87) and--max-model-len 131072(500000), to start low.--kv-cache-memory-bytes 7650410496, from the card's RTX PRO command.PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.The last one was needed. Without it, both ranks lost 17 to 19 GiB of free memory in the ten seconds after "Loading weights took 134 s" and went under the 8 GiB we keep free on these boxes. A fixed KV size did not move that. With it, the lowest free memory was 13.7 and 14.5 GiB. The host-side peak (the container's cgroup) stayed between 4 and 12 GiB, so the load does not read whole shards into anonymous memory.
Speed: one request at a time, temperature 0. Prefill is a fresh prompt with
max_tokens1. Decode is 512 streamed tokens after a short prompt. Two runs each.B12X_W4A16_A4_PREFILL=0These runs differ from the native engine's contract in three ways:
B12X_W4A16_A4_PREFILL=1). C keeps bf16 activations: it is 13 to 15% slower in prefill and the same in decode.fp8_ds_mla), and CUDA graphs are on.num_speculative_tokens > 1can lower acceptance.Quality: tool-eval-bench
--short(15 scenarios) scores 93. 13 scenarios pass and 2 are partial, for 28 of 30 points, with no safety warning.In this image, the routed experts keep their
csf_32x2_v1scales compressed.Nvfp4CsfMoEMethod(nvfp4_csf.py) expands one layer at a time into a shared scratch buffer.ashhart commentedon Oct 18, 2026
Thanks @Bizuayeu for the TP=2 measurements and the TP=2/TP=3 loader gate. The 13.7/14.5 GiB minimum free-memory readings and no-MTP 25.7 tok/s runs give us a useful reference. The MTP results of 33.9 and 25.3 tok/s need their prefix-cache state kept visible; A4 prefill and FP8 KV also need matching settings before we compare engines. We'll retain byte-for-byte independent host packing against each rank's device tensors, for row and column splits in every format, plus separate MXFP8 KDA-o, block-FP8 MLA-o and FP32-output checks. #1021 through #1024 are planned as foundations for 1.1.1, not complete FP4-Local support. Stored MXFP8 MTP, F32 norms and #593's model/vLLM gate remain required. Please keep the launch logs and cache-state details with these results.
Posted by Ash's AI Assistant Synthena
Bizuayeu commentedon Oct 18, 2026
Noted, and kept with the receipt. vLLM ran with
--enable-prefix-caching, and bench_http's/reset_prefix_cachecall got a 404 on this build, so the second run of each pair repeated the same prompt with a warm cache; the logged prefix-cache hit rate rose from 0% to about 30% over the runs. The MTP-5 pair (33.9 and 25.3 tok/s) is one such pair, and we have not separated the cache from MTP's acceptance there. The launch commands, logs and hit rates stay with the results, and A4 prefill and FP8 KV will be matched before any engine comparison.Bizuayeu commentedon Oct 18, 2026
(#1015 (comment))
The family pull request is open as a draft, #1050, on top of #1021 to #1024: the MTP layer's MXFP8 is read as stored and the float32 norms stay float32. At TP=2 on the real checkpoint the loader gate and the three engine checks (
glm-lanes-cover,glm-lanes-concurrent,glm-shared-check) pass; the vLLM comparison with matched settings and TP=3 serving are still to do.Bizuayeu commentedon Oct 22, 2026
Added a merge map on 1.1.2 at the top of the description: what FP4-Local needs on the
v1.1.2tag, with KV pages (#1163) and TP=3 on a ring of /32 links (#1178), each item's state, purpose and dependencies, and an order for #1050 and #1163. #1179 is in main but not in the tag, and #1187 hits the sameBadIndexwith Qwen3.8's FP4-Local. The rows for #1050 and #1163 will be updated after tonight's pushes and receipts.Bizuayeu commentedon Oct 22, 2026
Updated the merge map after last night's pushes: #1050's row and receipts follow its rebase on 1.1.2 and its "On v1.1.2", and #1163 is out of draft, with TP=2 and TP=3 receipts on its head after 1.1.2, which also fixes a bug its first TP=3 run found.
Bizuayeu commentedon Oct 23, 2026
Added #1211, stacked on #1163, to the merge map, the order and the receipts.
Bizuayeu commentedon Oct 24, 2026
Updated the merge map for 1.1.4: #1179 is in 1.1.3, #1105 is closed with nvcc 13.0's records on main (
23ac27342), and #1050, #1163 and #1211 conflict with main after 1.1.4. We are rebasing #1163 and #1211 and retaking their receipts, then #1050.Bizuayeu commentedon Oct 25, 2026
Closing this tracker: #1050, #1163 and #1211 are on main in
b34f9e78..e2f1e2cd, and the merge map above is final. #1178 stays open for TP=3 on a ring of /32 links.