GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker)

@MiaAI-Lab

This issue collects all the GLM work on the native CUDA engine in one place. Eight threads asked overlapping questions about the same port: #509, #552, #402, #310, #308, #214, #339 and #566. Each is closed with a link here, and its proposals and receipts stay readable there. Please post claims, plans and receipts here, so two people don't build the same piece.

Where things stand

Native CUDA serves Nemotron 3.5 Lightning only. The native GLM family serves GLM-5.3-Flash on Metal.
Several working GLM-5.3-Flash CUDA ports exist outside the tree: @MiaAI-Lab's EXL3 port at TP=2 on two GB10s (their recipe, greedy replies with drafts off equal to the Python engine token for token), @Bizuayeu's TP=2 port over NCCL (GLM-5.3-Flash on CUDA: let a long prefill wait between chunks while any rank runs hot #339, Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548), and our own MLX affine 4-bit port, in progress.
Shared CUDA layers are coordinated in Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548. cuda: one quantized-weight interface (qlinear.zig); Nemotron's 4-bit projections route through it #587 (one quantized-weight interface, qlinear.zig) is in review, and the NVFP4 entry follows it.

The gate

Every PR here passes the same gate as the rest of the engine:

drafted equal to "draft": false, resumed equal to fresh, concurrent equal to solo, on the device;
an FP64 reference for every projection class whose reduction order changes, with worst-column errors;
prefill and decode measured against the current release and against vLLM on the same hardware;
bf16 KV, and no arithmetic below today's precision;
drafts and concurrency through the lane core, with no serial serving path;
files under 600 lines, one-line comments, and no Python in the engine or its tools.
Bit equality with the Python engine is not required.

The pieces, in order

Shared layers (Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548): the quantized-weight interface (cuda: one quantized-weight interface (qlinear.zig); Nemotron's 4-bit projections route through it #587), then NVFP4 behind it, including grouped NVFP4 experts on prompt rows.
Multi-rank communication: zig/src/cuda/nccl.zig has AllReduce, AllGather and Broadcast. Send/recv and an exact reduce-scatter in rank order (GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310's) come as one small PR.
The GLM-5.3-Flash family on one GPU, then TP=2 and TP=3 with GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310's unequal shares. Any of the ports above can be the first PR. Whichever passes the gate first lands, and the others build on it. Say here before you open one.
Checkpoint formats on that family: MLX affine 4-bit, EXL3 on the reference decoder from core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496, and NVIDIA's NVFP4 checkpoint read as stored, W4A16 (GLM-5.3-Flash on CUDA: read nvidia/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4) as stored, W4A16 #308).
Full GLM-5.3 (glm_moe_dsa) as its own family that reuses the Flash family's CUDA pieces rather than copying them: TP=3 (Full GLM-5.3 (glm_moe_dsa) on three DGX Sparks, as layers of the GLM-5.3-Flash CUDA engine: how should it sit beside #159? #402) and TP=4 (Zig CUDA family for full GLM-5.3 (glm_moe_dsa) at TP4: plan and questions #552).
Later, on top: the heat wait on GLM prefill under the engine's TF_HEAT_HIGH and TF_HEAT_LOW (GLM-5.3-Flash on CUDA: let a long prefill wait between chunks while any rank runs hot #339), DFlash2 drafts (Native CUDA: GLM-5.3-Flash EXL3 with DFlash2 and TP/EP/DCP #509), attention on a desktop GPU with routed experts on the Sparks (GLM-5.3-Flash on CUDA: attention on one GPU, routed experts on the Sparks (proposal) #214), and Living Weights for GLM-5.3-Flash on CUDA (Living Weights for GLM-5.3-Flash on NVIDIA GB10 (DGX Spark) CUDA — port request + test hardware offer #566, after Living Weights learns on CUDA (GB10): the training step, its kernels checked against float64 #591).

Answers to #552's questions

Yes, a glm_moe_dsa family is wanted, sharing the Flash family's pieces as in step 5. Who is doing what is listed above.
NCCL, which the CUDA engine already uses. The cluster layer's links are built for the Macs' Thunderbolt RDMA. A TF_NCCL_LIB override for hosts without a system NCCL is fine.
The gate above. Bit equality with the Python engine is not required.
Hand-written .cu. 1.0.3 moved Nemotron to our own CUDA kernels, and new families follow.
Yes, bf16 KV. An FP8 KV cache, such as Native CUDA: GLM-5.3-Flash EXL3 with DFlash2 and TP/EP/DCP #509's fp8_ds_mla layout, would trade precision, so it isn't on the default path.
Yes, a tiny synthetic checkpoint for host tests is welcome, made by a Zig tool in the tree.

docs: the CUDA recipe links the GLM-on-CUDA tracker (ashhart#593)

Claiming step 2: NCCL send/recv and an exact reduce-scatter in rank order, as one small PR.

It doesn't depend on the weight format. It sums the fp32 partial rows that each layer's attention and MLP (or experts) leave on every rank, so it serves any of the GLM-5.3-Flash ports above (MLX affine, EXL3, NVFP4) and the full GLM-5.3 family alike.

zig/src/cuda/nccl.zig: ncclSend / ncclRecv beside the AllReduce, AllGather and Broadcast already there.
A reduce-scatter that keeps rank order. Each rank gets its own slice of the sum, added rank 0 first, then rank 1, and so on. So every rank holds the same bits that an all-gather followed by a rank-order sum would give. Slices may be unequal (GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310's whole units: the first units % world ranks take one more).
Receipts: host tests on synthetic partials, then on device two GB10s (TP=2) and three (TP=3 with unequal slices). Each is byte-equal against an all-gather plus a rank-order sum on the same inputs.

Where we use it: our GLM-5.3-Flash port reads nvidia/GLM-5.3-Flash-NVFP4 as stored (the NVFP4 family claimed in #548). At TP=2, each prompt chunk's partial rows go through this exchange: each rank glues only its own rows, and the bits are equal across ranks. I'll say here before opening it.

Step 2 is ready: NCCL send/recv and a reduce-scatter in rank order, one commit on main, checked byte-equal on one GB10, two (TP=2) and three (TP=3, unequal slices). Opening the PR now.

@GustavBlack

Hello. Following the direction from #545/#546 (own kernels, MLX affine checkpoint, own reference),
I have been building the GLM-5.3-Flash CUDA family in a fork and want to line up with this tracker
before opening anything, as the issue asks.

What exists now (fork branch, rebased onto v1.0.5; the kernel byte-gates have run green on
our two GB10s — see below):

The family on the native engine: weights (MLX affine 4-bit g64, host-packed into the shared lane
matmul's tiled layout), per-sequence state, the forward (embed → per layer [hc_pre → kda | sparse
MLA → hc_post] → MoE → head), and the engine (init/bind/commit/prefill/graphs), following the
Nemotron CUDA files' structure.
Own kernels with host references, per the 1.0.3 playbook: glm_glue.cu / glm_hc.cu
(7+2 kernels, fp32 fixed-order reductions derived from a recorded kernel set, device-side
Sinkhorn at the captured order), glm_route.cu (sigmoid + bias top-k, raw-score weights), and
glm_kda.cu (the Python engine's kda.cu vendored verbatim per the qmm_group pattern).
tf-cuda-test glm-glue | glm-kda | glm-mla cells are staged and byte-compare each kernel
against its host reference (the glm-mla cell is the sparse.tokens step, which is host math
between async copies).
Serving wiring: a lanes backend, native open/admission, multi-family dispatch.
The first GPU window ran (2026-10-10, our two GB10s) and the kernel byte-gates are green:
glm-kda 66/66, glm-mla 4/4, and glm-glue 121/121 — the first run failed 36/121 and found two
real bugs (warp_combine gave the cross-warp total only to lanes < wpc, so rmsnorm/hc_finish
ran inv_rms(0) ≈ 316×; swiglu read GU at stride w where the fused rows are 2w), both
fixed byte-exactly against the host references. On the way the engine's CUDA admission proved
itself: the 169.2 GiB MLX checkpoint is refused cleanly against a 103.9 GiB single-card budget,
so TP2 is mandatory for this model and the two-rank path is the next piece here.
The submission's honest gaps, matching the tracker's ordering: the port's first boot is gated
on the two-rank path — no --tp/--rank/--master on the CUDA native side yet, the forward's
all-gather is still a stub, nothing sets Engine.setComm (the split rules, per-rank slicing and
cuda/nccl.zig are in-tree); sampled mode is refused by design (greedy first, as the gate
expects); MTP drafting and sparse-bucket graphs are later rungs; EXL3 is deliberately not loaded
(MLX affine is the direction, and core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496's trellis decoder is the later path we would build on).

What I can contribute that the tracker's gate currently lacks: a two-machine GB10 pair on a
direct CX7 link — the gate asks for prefill/decode and the exactness suite on the device, and as
far as I can tell no one has two-machine CUDA GLM receipts yet. Already run and postable from the
first window: tf-cuda-test grouped-plan PASS (all cases byte-equal to experts.route, #558)
and fp8-lane PASS (all row counts plus the qlinear decode/prompt check, #482/#587) on our
box; carveout (#487) refuses with DumbBufferRefused under privilege — the GB10's display
controller won't hand out a dumb buffer, so that experiment is not usable here as-is. The Python
engine's TP2 greedy reference on the MLX checkpoint (≈19.2 tok/s, ≈118.5 s startup) is captured and
will be the token target the moment our two-rank path lands. I can also run these gates on someone
else's port on request.

Two questions before I open anything:

Given cuda: one quantized-weight interface (qlinear.zig); Nemotron's 4-bit projections route through it #587's qlinear.zig (MLX affine 4-bit behind one Linear.decode/prompt interface) and
that "our own MLX affine 4-bit port" is in progress here — should my MLX-affine projections move
behind qlinear.zig before a PR, or is a parallel packing implementation (same tiled layout)
acceptable to review and align afterward? I would rather land receipts first and refactor against
your interface than race two implementations of the same layer.
One small generic fix was needed for a second family: native/cuda.zig's prompt-cache wiring
assumes the family provides loaded.target and uses Nemotron's snapshot vtable. I guarded it
per family at compile time (a family without kept prompt states gets no store, and its budget
gates nothing on a cache). Is a family-owned snaps declaration the shape you would want
instead? Happy to send that as its own small PR either way.

No PR yet, per the issue's "say here first". Thanks — the tracker's ordering (shared layers →
collectives → family → formats) matches what the port ended up needing, and the published gate is
exactly what our staged tests were written against.

— Gustav (@GustavBlack)

(The port is AI-assisted work, recorded honestly in the project notes; authorship is mine per the repo conventions.)

@GustavBlack

@GustavBlack welcome, and thanks for laying this out. A few facts from our side that might be useful, since we're working on the same model:

We're porting GLM-5.3-Flash to native CUDA from nvidia/GLM-5.3-Flash-NVFP4 (claimed in Zig CUDA ports: shared layers below lanes.Backend, and one place to coordinate them #548). It's a different checkpoint format from your MLX affine one.
- The full 45-layer forward runs at TP=2 on two GB10s over a direct RoCE link, one rank per process. It's served over HTTP.
- Greedy decode gives the same token ids as our Python engine for three 512-token prompts.
- The branch is local for now. It's on main a6aa4782, with the NVFP4 linears and experts behind qlinear and experts.Grouped.
Your question 2: we ran into the same assumption in native/cuda.zig. Our branch builds the store from the opened family's own snap_* declarations, and a family that declares none gets no prompt cache. That looks like the same shape as your guard.
The two-rank exchange: cuda: a reduce-scatter in rank order over NCCL send and receive #642 (step 2 of this tracker) adds NCCL send/recv and a reduce-scatter in rank order. It doesn't depend on the weight format, so it may cover part of what your two-rank path needs. It's checked byte-equal at TP=2 and TP=3, including unequal slices.

@GustavBlack

Thanks — yes, this contributes in three places, and thank you for the #642 pointer.

The snap_* family shape is better than what I had. My port carries a compile-time guard
(if (comptime F == nemotron.native)) so GLM gets no prompt cache and its budget gates nothing
on one; the family-declaring-its-own version is what I hoped for when I raised it, and I will
take your shape into the port.
cuda: a reduce-scatter in rank order over NCCL send and receive #642 goes straight at my forward's gather. The two-rank path I have is the Python donor's
forward.gather: an all-gather of each rank's fp32 partial, then a rank-order sum (rank 0
first). So cuda: a reduce-scatter in rank order over NCCL send and receive #642 is not a dependency — the collectives are in and host-gated — but a
rank-order reduce-scatter with the same bits and only each rank's own rows is exactly the
upgrade that step wants. I would like to take it when it lands, and if a second-machine
receipt is useful I will run it on our pair (TP=2) and post the byte-equality beside yours.
qlinear / experts.Grouped are the homes I was aiming at. The MLX-affine projections
pack into the shared lane-matmul layout already; the remaining slice is the dense and expert
wiring. If your NVFP4 extensions are landing behind those same two homes, I would rather rebase
onto your branch's shape than invent a parallel one — is that branch going to become a PR soon,
or should I keep targeting v1.0.5's shape and reconcile after?

The one answer that would change my immediate plans: is the MLX-affine CUDA path still wanted
now that NVFP4 is the line you are porting? The v1.0.5 GLM recipe documents
TensorFold/GLM-5.3-Flash-MLX-4bit-MTP with --tp 2 for the native binary, which is what I have
been building toward — weights loader done, the affine-4 kernels and the two-rank path in, kernel
byte-gates green on our pair (glm-glue 121/121, glm-kda 66/66, glm-mla 4/4), and a Python
engine reference on the same checkpoint captured (TP=2 greedy, ≈19.2 tok/s). If MLX stays a
documented CUDA checkpoint, I will finish that slice and boot it against the reference. If NVFP4
is the CUDA direction and MLX was ever only the Metal format, say so and I will redirect — the
format-independent parts (the engine, the TP2 comm, the kernel slices) carry either way, and our
pair's receipts are yours to use.

From our side, on the same two GB10s, already posted above: tf-cuda-test grouped-plan and
fp8-lane are byte-equal passes; carveout refuses with DumbBufferRefused under privilege (the
GB10's display controller will not hand out a dumb buffer, so that experiment is not usable on
this box as-is).

— Gustav (@GustavBlack)

(AI-assisted work, recorded honestly in the project notes; authorship is mine per the repo
conventions.)

@GustavBlack

@GustavBlack thanks. Yes, please run #642 on your pair when it's useful; a byte-equality receipt from a second pair next to ours would help.

On our branch: we're pushing to get it to a PR, but a few items of the gate above are still open on our side (the FP64 references, the release and vLLM comparisons, and tests that don't lean on fixtures from our Python engine), so we can't give a date yet. It runs at TP=2 on two GB10s with greedy decode equal to our Python engine. We'll say here before opening it.

If you build on its shape before then, a few things to know:

It sits on main a6aa4782 and will be rebased as main moves.
One commit is local only and won't be in the PR. It changes upstream's staged NVFP4 prompt kernel to sum per 32-input block, so the bits match our Python engine. Please don't build on those sums.
Three small changes are meant for upstream on their own and may change in review: fp32 partial outputs from qlinear and the prompt GEMM, a skip on the staged prompt kernel, and packGateUp packing gate and up in place.
The rank exchange is written inline over NCCL send/recv for now. It moves onto cuda: a reduce-scatter in rank order over NCCL send and receive #642's RankScatter once that lands.
The GLM tests compare against goldens captured from our Python engine on the NVFP4 checkpoint, so they won't carry over to an MLX checkpoint as they are.

On MLX affine: #545 (comment)

glm: the vocabulary's TP cut in the checkpoint format's unit (128 for…

Whole-checkpoint receipt on a second GB10 pair, on merged main.

Environment: upstream main at 6ee828c2 (1.0.12 plus its docs commits), built from source on both hosts (-Dsm=121, CUDA 13; the two hosts' binaries are sha-identical); nvidia/GLM-5.3-Flash-NVFP4 at 423acf37 on both ranks, the ranks' own checkpoint-bytes check passing before rank 0 served; the NVCC container image (pytorch 26.07, NCCL 2.30.7); driver 580.173.02 with the CUDA 13.3 forward-compatibility userspace; two GB10s over the direct RoCE link with NCCL_SOCKET_IFNAME and NCCL_IB_GID_INDEX=2 pinned (our worker exposes the link's GIDs at index 2, not 3); BF16 KV.

The whole checkpoint serves. The 1.0.12 notes say the family "passed its lane checks on two GB10s, and its first four layers served" on your hardware. On this pair the whole 45-layer checkpoint serves at TP=2: rank 0 serves HTTP with 93.72 GiB of model resident, a 40,960-token window, four streams admitted, loaded in about 5 minutes; /health stayed green through everything below.

Exactness (the PR's own checks):

tf-cuda-test glm-ref: 11/11 PASS, every projection within its float64 tolerance; tf-cuda-test quant-ref: 4/4 PASS.
On the checkpoint at TP=2: glm-shared-check — 108 rows of 5 streams over 12 rounds equal their lone twins' logits, tokens, drafts and state byte for byte; glm-lanes-concurrent — every stream against its solo run, 313 forwards held several streams' windows; glm-lanes-cover — cancels, discards, eos at every length and max_new, rank 1 followed to the end. The same three pass on glm-synth's synthetic checkpoint first (TF_GLM_LAYERS=2).
tools/bench_concurrent.py --alone --serial (levels 1 and 4, both prompts, 256 tokens, temperature 0, 3 repetitions): all 30 replies equal their requests alone, and the alone runs equal "draft": false.

Speed (same tools, median): serial decode 22.5–22.8 tok/s a stream; four concurrent streams 51.0 tok/s aggregate on the code prompt and 40.1 on chat.

One observation where the pairs differ, in case it is useful: drafting accepts but does not pay on this pair yet. With the served defaults (drafts: on, MTP head loaded) and also explicitly at a fixed @3, a 256-token greedy reply logs rounds=125 accepted=131/213 (about 2.05 tokens a round, 61% acceptance) but decodes at the serial speed — a drafted round costs about twice a serial round here, where the PR's pair made drafting about 1.9x. Our serial is also faster than the PR's (22.5 against 17.61). The differences between the pairs: the forward-compatibility userspace over an older kernel driver, and the pinned NCCL interface and GID index on a direct RoCE link. Happy to run anything that narrows it down.

@GustavBlack

@GustavBlack, the exact control behind “fixed @3” would help narrow this down. On 6ee828c2, the native CUDA GLM facts set a ceiling of three drafts and the shared rule still selects the depth. A CPU probe of the actual rule chooses 2/1/3 for synthetic first-depth acceptance estimates 0.85/0.60/0.95 with the same ceiling of three. Please include the command or patch that forced three in your comparison.

There is also a software-base difference: #804's 17.61 tok/s table is from 6380c8da on 1.0.10; this report is main 6ee828c2 on 1.0.12. The driver and GID settings are not isolated causes by that comparison.

Could you attach the saved benchmark JSON and server/gate logs, the binary SHA256 on each rank, and the full checkpoint revision? Existing files are enough to start. If a timeline was already captured, the useful split is target verification, MTP forwards and draws/collectives. In this source each greedy draw performs an NCCL all-gather, stream synchronization and host copy, so acceptance rate alone does not locate the extra round cost. No device run was performed in this source/CPU check.

@GustavBlack

Thanks @GustavBlack for the whole 45-layer checkpoint receipt and 30 matching served replies, and @cwschroeder for separating the controls. This extends the reported model coverage beyond the earlier four-layer run; we have not independently rerun it.

Please add the command or patch behind fixed @3: the normal ceiling still lets the shared rule choose depth. Keep the saved benchmark JSON, both binary hashes and rank logs with the checkpoint revision. A timeline separating target verification, MTP and draw/collective work would help explain the cost. Acceptance alone, or comparisons across different software bases, cannot isolate the driver or GID setting.

Posted by Ash's AI Assistant Synthena

Three stacked PRs for anyone serving GLM-5.3-Flash on the native CUDA engine across ranks. Today the server answers HTTP 400 on GLM to three things that work on Nemotron and worked on the Python engine:

GLM-5.3-Flash on CUDA: prompt_logprobs on /v1/completions #1016 prompt_logprobs on /v1/completions: scoring a fixed text, so quantizations, engines and kernel changes can be compared by NLL on the same text, and loglikelihood tasks such as lm-eval's can run against the server. This is GLM-5.3 on CUDA (two Sparks): logprobs and prompt_logprobs, so quantization variants can be compared by perplexity #298's request, now on the native engine.
GLM-5.3-Flash on CUDA: structured output (first_masked and held) #1017 structured output (response_format, guided_*, structured_outputs): replies that always parse against their schema, with drafts on and concurrent streams.
Required tool calls in GLM's markup: xgrammar's glm_xml style #1018 tool_choice: "required" and a named function in GLM's own call markup (<arg_key>/<arg_value>, xgrammar's glm_xml style): the reply is a call to an offered tool, with arguments inside its schema. "auto" stays unconstrained.

Requests that ask for none of these take the same path as before, with the same token_sha. With prompt rows, a 5,000-token prompt's prefill took 8 to 11% longer; a constrained draw adds one gather of its rows. Checked on DGX Spark (GB10) at TP=2 and TP=3, with the receipts in each PR. Teacher-forced NLL (ja, en, code, math) at TP=2 was 2.5293, 2.9109, 1.3138, 0.6272, against 2.5474, 2.9257, 1.3184, 0.6250 on the Python engine. Review order: #1016, #1017, #1018.

A status update on our MLX-affine port (following our answer above), since the exactness window ran this week.

The kernel gates are in place and biting. Our fixture gate (the donor's own kernels' inputs and outputs, dumped on the real layer-0 weights, our kernels byte-compared against them) passes byte-equal on the embedding, the affine grouped experts, and the prefill matmuls in both output precisions. Two gaps in our own fixture tooling surfaced and are fixed: the fp32 prefill variants had never been generated (a control-flow bug in our generator, so that path had never been gated at all), and the "o" fixture was the full-attention layer's o projection, not the KDA layers', which is a different tensor — the KDA o now has its own fixtures.

The boot's wrong tokens were a loader bug, not kernel math. Re-running the first boot after last window's fixes still gave a wrong first token and then a repeated one. A per-layer tap bisect against the donor's own forward (its set_taps, with matching dumps added to our boot tool) put the divergence between two byte-equal stages: the KDA chain's output was right to an ulp and the o projection's fp32 partial was wrong at many times its magnitude. The cause: our loader's gather-and-pack read a tensor's words, scales and biases through one shared staging buffer, and each split resized and overwrote it — so every column-split tensor's first rows (the KDA o projection, the dense and routed experts' down projections) were packed from the wrong bytes, while every row-split tensor was clean, which is why the kernel gates could not see it. Fixed in both pack paths; the port's first token now equals the Python engine's, and 24 of its first 64 greedy tokens match it.

The lesson we would flag for other ports: the kernel gates passed on fixture bytes while the engine's own device bytes were wrong. The check that caught it was a gate over the loader's packed output (our engine's tensors diffed against the donor's pack of the same slice), and we would recommend that check to anyone bringing a weight format up.

What remains: the two absorbed-attention kernels' reduction association. Ours summed ascending; the donor's tl.sum lowers to a specific tree, which we decoded from the captured TTGIR and PTX (even/odd half-sums, 32-lane butterflies, warp combines) and reimplemented — the byte gate is now 8 and 2 words of 65,536 and 32,768 from equal, every one a 1-ulp last-add rounding. When those match we expect the full token match, and then the format comes to the family as the loader branch #804's description points to.

@GustavBlack

@GustavBlack, a heads-up before we both change GLM's CUDA loader. For FP4-Local (#1015), I'm about to make families/glm/cuda_weights.zig stop reading every projection as bf16. Each projection's format will come from the checkpoint's recipe (admitRecipes, #1023), checked against the tensor's own dtype and shape, and a mismatch will be refused by the tensor's name. NVFP4, MXFP8 and block FP8 come first. I'd like MLX-affine to go through the same choice rather than a second path: where are you putting it? Your column-split lesson becomes a gate in the same change: each rank's device bytes against an independent host pack of the same slice, for every format, split and TP width.

@Bizuayeu

Thank you, @Bizuayeu, @MiaAI-Lab, @GustavBlack and @BHCC2025, for the GLM CUDA code and comparisons. TensorFold 1.1.0 ships GLM-5.3-Flash's EXL3 CUDA pair through #976 with #855, closing this Flash-family tracker. On two GB10s, drafted equals plain 17/17, two/four concurrent replies equal their runs alone, and cached chats equal cache-off chats 12/12; A B B A matched prior throughput and tokens. Full GLM-5.3 and FP4-Local in #1015 remain separate.

For 1.1.1's loader work, use one per-projection selector for NVFP4, MXFP8, block FP8 and MLX-affine, reading ModelOpt recipes through #1023. Check dtype/shape, refuse mismatches by name, and compare each rank's device bytes against an independent host pack for every format, row/column split and TP width.

Two NVIDIA GB10s: EXL3 text 17/17, concurrency 2/4 and chat 12/12; NVFP4 serving was budget-refused, and full media-matrix validation is unrecorded.

Posted by Ash's AI Assistant Synthena

Correction first, then the corrected receipt.

Retraction. In my 17 Oct comment I wrote "and also explicitly at a fixed @3". That is wrong. On 6ee828c2 the CUDA server exposes only "draft": true/false; the depth is chosen by the shared rule, with max_drafts = 3 as a ceiling, not a fixed depth. There is no flag, request field, or saved run that forced depth 3. The rounds=125 accepted=131/213 line is the adaptive path. I withdraw the fixed-@3 claim. The only fixed-depth control I found is GLM_DEPTHS in the Metal glm-run; it is a different engine and does not produce that server line.

I also found two packaging errors in the earlier receipt:

the saved glm-shared-check log was FAIL glm-shared-check: FileNotFound because the rerun directory was missing dc.json; with dc.json present the check passes (below).
the earlier artifacts did not record per-rank hashes at run time, and the worker binary was rebuilt afterwards. I could not substantiate the "sha-identical" wording from those files. This run copies one build to both ranks and hashes it on each.

Environment. 6ee828c2 (1.0.12); nvidia/GLM-5.3-Flash-NVFP4@423acf37583782c51c142d145aef733d72943d93; nvcr.io/nvidia/pytorch:26.07-py3; driver 580.173.02 with the CUDA 13.3 forward-compatibility userspace; two GB10s over the direct RoCE link with NCCL_SOCKET_IFNAME and NCCL_IB_GID_INDEX=2; BF16 KV; 40,960-token window; model 93.72 GiB a rank.

Per-rank SHA256, same binary copied to both:

tensorfold-native  c096efb88a05c70c292c0c352309810dc4c1e5c9607505d7a688f2cb96c9c641
tf-cuda-test       1b3727f7b9b1fa4e05b9834486abf24b0cbaaccf12f45fd93481ae2de8ce152e

Commands.

serve rank 1: tensorfold-native serve /modelroot --context 40960
serve rank 0: tensorfold-native serve /modelroot --name glm-nvfp4 --context 40960 --parallel 4 --host 0.0.0.0
benchmark:    python3 tools/bench_concurrent.py http://127.0.0.1:8080/v1 glm-nvfp4 \
                --levels 1,4 --tokens 256 --reps 3 --temperatures 0 --alone --serial --output bench-clean.json
shared check: tf-cuda-test glm-shared-check /modelroot /rec/dc.json 0 2 /rec/pair.id
lane checks:  tf-cuda-test glm-lanes-concurrent|glm-lanes-cover /modelroot /rec/dc.json 0 2 /rec/pair.id

Gates (whole checkpoint at TP=2; the two refs are synthetic):

glm-shared-check: PASS, 108 rows of 5 streams over 12 rounds (2 to 5 streams a forward, 1 to 4 rows each, greedy and sampled), byte-equal to lone twins.
glm-lanes-concurrent: PASS, 313 forwards held several streams' windows; every stream equal to its solo run.
glm-lanes-cover: PASS, cancels/discards/eos at every length; resumed at 2048 and 4096 equal their fresh passes (64 ids each); 512 ids with drafts equal.
tf-cuda-test glm-ref and quant-ref: PASS (projection and quantization references).

Benchmark (clean run, no profiler; --alone --serial, 256 tokens, temperature 0, 3 reps):

prompt	streams	tok/s	steady tok/s	alone equal	serial equal
code	1	22.2	—	3/3	1/1
code	4	49.8 aggregate (12.7–13.3 a stream)	51.8	12/12	1/1
chat	1	18.7	—	3/3	1/1
chat	4	39.3 aggregate (10.1–10.4 a stream)	40.7	12/12	1/1

The earlier "22.5–22.8 serial" did not distinguish the prompts; this run gives 22.2 for code and 18.7 for chat. The earlier 51.0/40.1 aggregate was a different run; this one is 49.8/39.3.

Round/acceptance (same run, from the server log):

prompt	level	rounds	accepted/drafted	tokens/round	approx. ms/round
code	1	96	160/231	2.67	120
code	4	87	169/257–258	2.94	228
chat	1	125	131/213	2.05	110
chat	4	107–108	149/317 or 148/321	2.37–2.39	235–237
either	plain	255	0/0	1.00	not saved

The depth rule selects different depths for the two prompts and for 1 versus 4 streams. That is exactly why the fixed-@3 claim needed a control I did not have.

Timeline. I do not have the target/MTP/draw split. The served lane path does not allocate the TF_GLM_PROF objects (Engine.verify does; verifyShared does not), and the nsys run produced a report with no CUDA kernel data. If you want that split, I can add a profiling command that runs the same round through Engine.profile; I did not want to present an inferred split as measured.

docs: the CUDA recipe links the GLM-on-CUDA tracker (ashhart#593)

glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP…

glm: the vocabulary's TP cut in the checkpoint format's unit (128 for…

glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP…

glm: the vocabulary's TP cut in the checkpoint format's unit (128 for…

Navigation Menu

Repository navigation

GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593

Where things stand

The gate

The pieces, in order

Answers to #552's questions

39 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593

Description

Where things stand

The gate

The pieces, in order

Answers to #552's questions

Activity

Bizuayeu commented on Oct 10, 2026

Bizuayeu commented on Oct 10, 2026

GustavBlack commented on Oct 11, 2026

Bizuayeu commented on Oct 11, 2026

GustavBlack commented on Oct 11, 2026

Bizuayeu commented on Oct 11, 2026

39 remaining items

GustavBlack commented on Oct 17, 2026

cwschroeder commented on Oct 17, 2026

ashhart commented on Oct 17, 2026

Bizuayeu commented on Oct 17, 2026

GustavBlack commented on Oct 18, 2026

Bizuayeu commented on Oct 18, 2026

ashhart commented on Oct 18, 2026

GustavBlack commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions