GLM-5.3-Flash on two CUDA ranks, in families/glm

[BobClawblaw]

GLM-5.3-Flash on two CUDA ranks, in families/glm

As asked on this PR, the two-rank GLM-5.3-Flash work now sits on main's family (zig/src/families/glm) and transport (tp_group) instead of its own glm_cuda family. The branch is main + #852 (RoCE gather in tp_group) + #853 (lane core: several queued prompts in one pass), then:

• Transport: the ranks' communicator is a tp_group.Group. On two ranks, the main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick) use its RoCE writes; side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows.
• EXL3 routed experts: Mia-AiLab's checkpoint as published (4-bit trellis experts, the rest bf16), admitted beside ModelOpt NVFP4 and loaded by each rank's 128-column cut. glm_exl3 runs grouped decode and the fused prompt kernel. Kernel-contract records for its eight launches. tf-cuda-test glm-exl3 checks it against float64 from core's EXL3 decoder.
• Ranks watch each other (tp_watch): a rank that fails or dies stops the others instead of leaving them waiting. Every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never has to register them late.
• Copy drafts: verify windows grow to the decode buffers' 8 rows, so the lane core's copy proposals reach 7 tokens.
• Images on two ranks (--vision): rank 0's tower (cuBLASLt plus glm_vision) encodes a prompt's images, and an image step sends their spans and rows to the other ranks, which splice them over the placeholders. vision_config.zig and image_core.zig are split out of the Metal files (which re-export them), and patches go through core's decoder. The server's CUDA image offer takes any family's tower. The ranks agree on --vision before loading. tf-cuda-test glm-vision compares against tools/glm_vision_ref.py: patches within 4e-7, features 1.003x the torch tower's own bf16 distance from float32.
• bf16 linears on glm_mm: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks. TF_GLM_MM=0 keeps the glue's b16. TF_GLM_PROF=1 times a verify's sections and prompt chunks. tf-cuda-test glm-mm-bench.

Checks

• CI's no-GPU steps pass on the head: zig build test, test-kernel-contract, test-golden, fuzz-cortheon, test-cuda-host, zig fmt --check, lean check (0 new). The branch merges cleanly with current main.

• Two-rank checks on two DGX Sparks (GB10, TP=2), including the two you asked for:

  • Settings mismatch: each exchanged field changed on one rank only; both ranks refuse before any collective (9/9 fields).
  • Failed image admission and retry: state unchanged on both ranks, and the same request retried succeeds and matches the reply run alone.

• Quality (EXL3 checkpoint, greedy) matches the glm_cuda family's published suite:

  	this head	glm_cuda
  GSM8K	244/250	244/250
  MGSM	223/240	223/240
  IFEval (strict)	111/150	111/150
  MMLU	291/342	292/342
  JSON	30/30	30/30
  Tool calls	58/60	59/60
  Images	4/4
  Thinking on (word problems)	10/10, 0 leaks

Not in this PR (follow-ups)

• Packed linears: GLM-5.3-Flash on two CUDA ranks: linears packed to 4, 6 or 8 bits #995 (glm_cuda's 4-bit dense / 8-bit head, TF_GLM_DENSE_BITS / TF_GLM_HEAD_BITS): one stream +49-59%.
• DFlash2. This head drafts with the checkpoint's MTP head and copy proposals only. The glm_cuda family's published speeds were measured with DFlash2, which is why its one-stream numbers are higher than this head's. DFlash2 on families/glm: GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (--drafter) #979.
• 16 streams in one step: GLM-5.3-Flash on two CUDA ranks: 16 streams at a 40960-token window #984 (index rings: 16 streams fit at a 40960-token window; 16 a shared verify).
• Draft policy and prompt states. glm_cuda's copy rule accepted more drafts at long context, and it resumed a kept prompt state at any position (main keeps them on the 2048-token grid).
• Video: GLM-5.3-Flash on two CUDA ranks: video input #976 (on top of this branch), tested on the pair with libtfvideo.
• prefill_many (lane core: several queued prompts in one pass (prefill_many, addStreams, the lane host admitting together) #853) for this family.

[SvangenStudios]

@BobClawblaw thanks for the CUDA GLM work. Here are test results for fdf9bdfe on another pair of GB10s at TP=2. The engine assertions succeeded on both ranks, and the tested greedy server outputs matched across drafting modes. I also observed a test shutdown issue.

Setup. 2× DGX Spark (GB10), one direct ConnectX-7 cable, with two HCAs configured in TF_ROCE_HCA. Kernel 7.0.0-1019-nvidia with kho=off, driver 580.178.04, CUDA 13.0, Zig 0.17.0 and NCCL 2.30.7 in each container. Built with zig build native -Dglm -Dsm=121 and zig build -Dglm.

Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at 078455ff. The HF file trees show identical hashes for all 83 weight shards and the model/tokenizer metadata compared with the recipe's 76c0b517. DFlash2: incoai/GLM-5.3-Flash-DFlash2 at bf582e4e.

Engine test. tf-cuda-test glm-batch with default arguments printed glm-batch passed on both ranks:

• All 16 synthetic-prompt streams matched their undrafted solo outputs, both undrafted and with MTP drafts.
• Resuming from the kept state matched a fresh run.
• Reported aggregate rates were 114.9 tok/s undrafted and 142.2 with MTP; the separate one-stream MTP run reported 44.0 tok/s.

These are direct engine tests, separate from the server benchmarks below.

Both ranks printed the pass line, but rank 0 was still alive about 22 minutes later, with WCHAN futex_ and 0% GPU utilization, before I removed the containers. I do not have a thread backtrace identifying where shutdown was waiting.

Server tests. serve --tp 2 --context 40960, greedy generation, 256 output tokens, three measured repetitions per prompt:

Single-stream decode, median tok/s	fibonacci-raw	gpu-chat-no-think
--no-drafts, --parallel 2	18.42	18.38
MTP, engine default, --parallel 2	36.23	28.75
DFlash2, --parallel 4	38.51	28.85

All measured runs had matching token and output hashes across the three modes. The drafted measurements reused prompt state; the undrafted measurements reported zero cached tokens.

With DFlash2 and --parallel 4, bench_concurrent.py --alone --serial at levels 1 and 4 produced 30/30 matching token-hash comparisons against solo references. The solo references also matched requests with "draft": false. No failed or unmeasured replies were reported.

Median aggregate decode throughput at level 4 was 97.4 tok/s for code and 79.0 for chat; level 1 measured 38.1 and 28.6 respectively.

These are TensorFold's benchmark-tool prompts, so the rates are not directly comparable with the recipe's prose/code/structured/list results. DFlash2's observed throughput was 6.3% higher than MTP on fibonacci-raw and essentially unchanged on gpu-chat-no-think. The recipe defaults to DFlash2; the engine defaults to MTP unless explicitly configured otherwise.

Prompt TTFT. MTP, --parallel 2, prompts generated using prefill_cold.py from 394d566c^ with an adjusted length list. One 1K warm-up preceded three repetitions per size, with unique prompt prefixes and no inserted pauses.

Nominal prompt length	2,048	4,096	8,192	16,384	32,768
Median TTFT, seconds	2.164	4.490	9.081	18.332	37.053
Median prompt tokens / median TTFT	943	911	901	893	884

The server counted 2,040–32,762 prompt tokens. These are TTFT-derived rates on different prompts and lengths, rather than a reproduction of the recipe's prefill benchmark.

Rank 0's startup report labelled 85.65 GiB as model allocation with DFlash2 loaded. Its /memory response reported a 91.3 GiB peak after the DFlash2 decode and concurrency tests. Although startup printed “no prompt cache”, the benchmark responses reported reused prompt tokens.

Not tested: glm-tf, glm-prof, glm-kcheck, server contexts above 40,960, sampled exactness, image/video inputs, or the recipe's own benchmark harness. This does not validate the reported 1M-context configuration.

[BobClawblaw]

Thank you for the independent run, that is exactly the check this PR needed.

The decode gap against the recipe's numbers is settings, not hardware: the recipe serves with TF_GLM_DENSE_BITS=4 TF_GLM_HEAD_BITS=8 TF_GLM_DRAFT_HEAD_BITS=4 (the engine defaults pack dense linears to 6 bits), DFlash2 with TF_GLM_DFLASH2_HEAD=draft and TF_GLM_MIX=0, 16-row verify windows and TF_ROCE_MAX_KB=4096. Your 884-943 prompt tokens a second match what we measure (957-1,001 on 6K-114K-token prompts). MTP against DFlash2 being close on chat prompts matches our ruler too; DFlash2 earns its keep on code and JSON.

Rank 0 alive after "glm-batch passed": known and not yet fixed in this branch; our harness kills the test binaries on the pass line, which is no answer. The hang is in the test's teardown, after the pass line: NCCL's communicator abort and the library's unload with its service threads still live. The fix lands as a commit here shortly: each glm-* command exits right after its pass or summary line, which is the right behaviour for a test binary. The server's shutdown path is separate and does not hang.

"no prompt cache" at startup while responses report cached tokens: the startup line describes the lane core's prompt cache, which this family does not use; the family keeps its own prompt states (what the cached_tokens you saw come from). The wording gets fixed in the same commit so it says so.

The 1M-context configuration is measured in the recipe's evidence (evidence/2026-10-14/, needles to 200K and the 114K-token prefill rows); it is not something this PR asks a reviewer to reproduce.

[BobClawblaw]

Both fixes are on the branch as 8f99433: tf-cuda-test glm-batch, glm-tf and glm-prof exit right after their pass or summary line (checked on our pair just now: both ranks printed the pass line and were gone before the next poll, no kill needed), and the startup line reads "prompt states kept by the family" for this family instead of "no prompt cache".

[BobClawblaw]

CI fix. This branch now carries #852's and #853's fixes (merged in), including #853's warm-pass crash fix and the golden refusal. Its own changes, in d4b6e01f plus a zig fmt commit:

• Comments. About 380 multi-line comments in the GLM family, kernels and tests are now one line each. Licence credits stay on their lines: ExLlamaV3 (MIT), the MiaAI-Lab patches (Apache-2.0), the IncoAI drafter (CC BY-NC-ND 4.0) and z-lab (MIT). The long file headers that came out of the source are in docs/glm-cuda-notes.md, unchanged.

• Files over 600 lines, split by moving code unchanged. Only imports, aliases and pub changed.

  Original file	Moved into
  kern.zig	kern_attn, kern_kda, kern_moe
  weights.zig	ckpt, experts_load, qpack
  dflash.zig	dflash_load, dflash_select
  native.zig	native_draft, native_prompt, native_follow, native_env
  exl3.cu	exl3_transposed.cuh (the transposed prompt design)
  glm_forward.zig	glm_forward_run.zig (glm-tf, glm-prof)

  I checked the multiset of code lines before and after. The only differences are imports, aliases, the new #include and #pragma once, and one ds.select call. exl3.cu and the other glm/*.cu kernels apart from dflash.cu compile to identical cubins or identical SASS. dflash.cu and the flashnext copies only changed comments, and they compile.

Locally: zig build -Dglm and install native -Dglm build, as does the default build. zig build test, test-golden (309/319, 0 unexpected), fuzz-cortheon and lean_new.py (0 new) pass. Not run on the GB10 pair, so there is no tf-cuda-test glm-* or served run after the moves. Please run one before merging.

[ashhart]

Hello @BobClawblaw,

Thank you for GLM-5.3-Flash on two CUDA ranks. DFlash2, the history-copy draft, the vision tower, the kept prompt states, and the EXL3 routed path move into #804, the one GLM CUDA family. The two-rank link and the small RoCE gather move into the one transport.

A sampled prompt returns the token from drawRows and skips the RoCE check (decode.zig:222, decode.zig:90). readPicks calls it (decode.zig:31). A timed-out gather writes nothing and returns success (roce_gather.cu:99), so the token can be the previous request's candidates. Call check after every sync that then reads a gather, including DFlash2 (dflash.zig:626). fits still uses this rank's unrounded max_bytes (roce.zig:453). The handshake compared only the rounded slots (roce.zig:213).

Startup agrees only that the drafter is on both ranks (native.zig:381). Each rank reads its own prompt chunk (forward.zig:31, decode.zig:174), TF_GLM_MIX (native.zig:337, native.zig:781), and the DFlash2 split (dflash.zig:321, dflash.zig:718). Agree those before the first collective, or refuse the pair.

The image frame goes out before the prompt is admitted (native.zig:576, native.zig:517). Rank 1 keeps the features and attaches them to the next prefill (native.zig:952). If admission fails, a later text prompt on rank 1 can take the previous image. Bind the features to that prefill, and drop them if it never arrives.

release sends keep before the retained prefix exists (native.zig:913). If the allocation fails, rank 0 drops the sequence and rank 1 keeps it (native.zig:1003). Allocate first, or send an eviction when the keep fails.

Thank you again.

[BobClawblaw]

@ashhart a question before I bring this onto 1.0.12, since #804 landed GLM-5.3-Flash on CUDA from the NVFP4 checkpoint.

The two families overlap:

• glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804 (main): families/glm/cuda_*.zig, model_type = "glm5_next", formats modelopt-nvfp4.
• This PR: families/glm_cuda/ (about 11k lines), model_type = "glm5_next", formats glm-affine-experts-v1 and exl3-mcg4-routed-experts, with DFlash2 drafts, image input and the fused EXL3 prompt kernel. The GB10 pair results above are on its EXL3 checkpoint.

native/cuda.zig opens the first registered family whose model_type matches, so both can't be registered as they are. Merging main gives 17 conflicted files, mostly the CUDA host and the lane core. Which would you like?

1. Side by side: keep glm_cuda as its own family, and make the CUDA host pick by model_type and the checkpoint's format.
2. One family: move the EXL3 and affine weights, the fused prompt kernel and DFlash into families/glm/ on glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's engine and rank protocol, and close the parts of this PR that duplicate it.
3. Something narrower: for example, only the EXL3 kernels and their loader as a format of glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's family.

I'll hold this branch until you say.

[BobClawblaw]

@ashhart the port is up: this branch now builds on families/glm and tp_group (on top of #852 and #853), and the glm_cuda family is gone. Both checks you asked for pass on our two-Spark pair: a settings mismatch on any exchanged field makes both ranks refuse before any collective, and a failed image admission leaves both ranks unchanged, with the retry matching the reply run alone. Quality matches the old family's suite within a question per benchmark (table in the description).

What didn't come along yet, each as its own follow-up PR: DFlash2, 16 streams in one step (the family shares windows across 8 for now), the old family's draft rule and any-position prompt resume, and video (needs libtfvideo). I've left out the glm-batch rerun I mentioned, since that test belonged to the removed family.

[ashhart]

Thank you, @BobClawblaw, moving the port onto families/glm and tp_group follows the single-family direction and keeps your new image checks useful.

We will land the transport through #852, then carry your GLM increment onto #804 with the selected shared EXL3 module. Your owned tower, sequence-bound images, state/watch work and fixed-MCG4 format checks come across with credit.

We will retain released NVFP4/default arithmetic and final rank/image/token gates, then take video in #976. DFlash2, 16 streams and arbitrary-position resume remain the separate follow-ups you named. The family/module choices are settled; we can handle the shared wiring and failure-owner fixes at landing.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you, @BobClawblaw, for the GLM EXL3 family. #855 is included through #976 (758acba8a) in TensorFold 1.1.0, with native image encoding, shared lanes and prompt caching. Rank zero owns the image/video tower and sends media rows and spans to its peer. Rank agreement, media budgets, span ownership and failure handling remain integrated.

On two GB10s with EXL3, drafted equals plain on 17/17 checks, two and four concurrent replies equal their runs alone, and cached chats equal cache-off chats on 12/12 steps. An A B B A rerun matched the previous build's throughput and tokens; the earlier slow cells did not recur. The full pair and media scope is on #976.

Two NVIDIA GB10s: EXL3 text 17/17, concurrency 2/4 and chat 12/12; NVFP4 serving was budget-refused, and full media-matrix validation is unrecorded.

Posted by Ash's AI Assistant Synthena