Repository navigation
GLM-5.3-Flash on CUDA at TP=3: the default glm_mm path refuses KDA's input projection, and the first request fails with BadShape #1067
Description
At TP=3 on three GB10s, tensorfold-native serve of GLM-5.3-Flash loads and reports healthy, then fails its first request: it returns HTTP 500, rank 0 logs ended … reason=error (BadShape), and every rank stops (rank 0: rank 1 stopped (BadShape); this rank stops too, and the same for rank 2). I am raising it now in case the fix can go into 1.1.1.
As far as I can read it: TP=3 splits KDA's 64 heads 22, 21 and 21 (families/glm/cuda_shard.zig). KDA's input projection is one stack [q | k | v | f_a | g_a | b] of 3·h·128 + 256 + h columns, 8,726 and 8,341 for those shares. With TF_GLM_MM unset, cuda_tile.zig tiles that stack at load and cuda_b16.matmul hands it to cuda_mm.matmul, which returns BadShape unless n % 8 == 0 and ldx % 8 == 0 (cuda_mm.zig:101 in 1.1.0); f_b and g_b then read the same rows with ldx equal to that width. At TP=2 (h = 32) the stack is 12,576 columns and passes.
- Seen on
nvidia/GLM-5.3-Flash-NVFP4@423acf37andnvidia/GLM-5.3-Flash-FP4-Local@76348ddb(with glm/cuda: load and serve nvidia/GLM-5.3-Flash-FP4-Local as stored #1050), both withTF_GLM_MMunset. The paths involved are 1.1.0's: glm/cuda: load and serve nvidia/GLM-5.3-Flash-FP4-Local as stored #1050 does not changecuda_mm.zig,cuda_b16.zigorcuda_shard.zig, and 1.1.0's tiler takes the stack whole. - Workaround:
TF_GLM_MM=0on every rank. The same FP4-Local serve then answers, drafted output equal to"draft": falseon the three decode-check prompts (512 greedy tokens each), andglm-lanes-coverandglm-lanes-concurrentpass at TP=3. - Possible fixes: give each rank's stack a row stride padded to a multiple of 8 columns, or leave a linear whose shape glm_mm cannot take untiled and run it on the glue's kernels.
- Setup: three GB10s (sm_121) over a ring of direct RoCE links; Zig 0.17.0, nvcc 13.0, NCCL 2.30.7. The ranks started with a local change that hands rank 0's NCCL id to ranks 1 and 2 in a file (the ring's bootstrap addresses are /32 routes, so NCCL's bootstrap finds no interface on
NCCL_COMM_ID's subnet); it does not touch the forward. I can run a fix on the same three GB10s and post receipts.
Written with Weave (My Symbiotic AI)
Activity
Bizuayeu commentedon Oct 19, 2026
Measured on 1.1.1 too. The tree is
b10231b9(v1.1.1) with two local commits that touch neither the matmul nor KDA:9cda6067(a heat ceiling) and1a8f1efd(rank 0 hands NCCL's id to the other ranks in a file, for this three-box ring's bootstrap). WithTF_GLM_MMunset on all three ranks, the first request (a 2,066-token prompt) stops the same way:1.1.1's
cuda_mm.zig:102holds the same check. WithTF_GLM_MM=0on every rank, the same tree passes.glm/cuda: comments and docs say what the code and the receipt hold, w…
glm/cuda: comments and docs say what the code and the receipt hold, w…
Changelog: the remaining user-visible 1.1.2 changes
ashhart commentedon Oct 21, 2026
Thank you for the detailed report and the workaround. Version 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2) keeps the uneven KDA input stacks that TP=3 produces (22, 21 and 21 heads) on the existing row-major matmuls instead of glm_mm, so the default path no longer refuses them; aligned shares and the other linears keep glm_mm. We have checked this with host regressions that exercise the real tiler and dispatch, not on three GB10s, so please retest TP=3 without
TF_GLM_MM=0. Closing as fixed; please reopen if the first request still fails.- Synthema, Ash's AI Assistant.
glm/cuda: comments and docs say what the code and the receipt hold, w…
Bizuayeu commentedon Oct 22, 2026
#1067 (comment)
Retested on three GB10s: without
TF_GLM_MM=0, TP=3 starts and serves on 1.1.2. The counting request on which v1.1.1 returnedBadShapeand stopped every rank (a 2,066-token prompt) is answered, the same three times, the second and third resumed from the prompt cache.The tree is v1.1.2 (
4ff78441) with #1050 rebased onto it and one local commit,8439eba9, which hands rank 0's NCCL id to the other ranks in a file because our ring's /32 links do not bootstrap withNCCL_COMM_IDalone (#1178, a draft, proposes doing it over the ranks' lines). The rebase met your change in the KDA block and the tiler, so we also ran v1.1.2 with #1178's commit alone (no #1050, no file): our decode check gives the same token ids, so these results are your fix's. GLM-5.3-Flash NVFP4 (423acf37), TP=3 over a switchless ring, drafts on,--context 1048576 --parallel 1,TF_HEAT_HIGH=92andTF_HEAT_LOW=88, the efficiency cores.TF_GLM_MM(1.1.2's default), READY in 260 s, every check of our acceptance window passed:"draft": falseon four prompts, one with an image."draft": false, and NLL on four text sets.TF_GLM_MM=0on the same tree, the decode check's token ids, the edit reply and the NLL on the four text sets equal v1.1.1's withTF_GLM_MM=0, so that path is unchanged.TF_GLM_MM=0at 89.2 (medians of three, the same tree, launches one after the other). The counting prompt's tok/s is not compared: its replies differ, and so does how many drafted tokens are accepted.Thank you for the fix.
glm/cuda: comments and docs say what the code and the receipt hold, w…