Repository navigation
GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (--drafter) - #979
Merged
Merged
Conversation
….cu) and the ranks' control link (tp_link.zig): small gathers as RDMA writes over both halves of the port, after b12x's exchange; NCCL carries the rest
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…glm_cuda), its kernels (zig/kernels/cuda/glm: the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with the fused prompt kernel, latent attention, the indexer, KDA, the hyper-connections, the vision tower's helpers, DFlash2), tf-cuda-test glm-batch/glm-prof/glm-tf, -Dglm (cherry picked from commit 611959f15964cda97ecd12d0e459ea4ac0e84f78)
…h), no include across families
…prefill, sample) and no Flash Next test commands
…meration, kernel parameter info, virtual memory)
…ardown hung in NCCL's service threads under the library's unload, reported on PR 855); the startup line says a family keeps its own prompt states instead of 'no prompt cache'
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…ders in docs/glm-cuda-notes.md Splits move code unchanged (only imports, aliases and pub change): kern.zig into kern_attn, kern_kda and kern_moe; weights.zig into ckpt, experts_load and qpack; dflash.zig into dflash_load and dflash_select; native.zig into native_draft, native_prompt, native_follow and native_env; exl3.cu's transposed prompt design into exl3_transposed.cuh; tf-cuda-test glm-tf and glm-prof into glm_forward_run.zig.
…cs/roce-gather.md, zig fmt
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
…; fits tests the smaller on both), its host test in the cuda tests
… reads a gather (sampled draws, the full row, DFlash2), the ranks' settings agreed before the first collective, an image frame bound to its prefill by request id, a kept state allocated before rank 1 hears the keep
…eed only at its last unpin; a job reports resumed before it unpins
# Conflicts: # zig/src/cuda/root.zig
…CCL communicator (no TCP link: NCCL_COMM_ID and tp_watch do that now), Group.openRoce brings it up on two ranks, allGather takes it by size alone, Group.check after a sync that reads a gather; tf-cuda-test roce-gather (every size against NCCL's bytes, graph replays, time a call)
…ill_many (Engine.addStreams; the lane host takes, sets up and launches each job as before and prefills the shared-pass ones together); a pass that errs or has no room leaves none prefilled and each stream goes alone; the CUDA and recorder wrappers forward it; the fake's batched backend and four host tests
…) into pr/glm-flash-cuda: the glm_cuda family leaves; DFlash2, history-copy drafts, the vision tower and the EXL3 routed experts move into families/glm commit by commit, its kept states are main's
…he main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick padded to 16 bytes) take its RoCE writes, side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows
…lis experts, the rest bf16): admitted beside ModelOpt NVFP4, loaded by the rank's 128-column cut, run by glm_exl3 (grouped decode, fused prompt) on an 8-slot plan into the MoE block's fp32 slot rows; bf16 dense layers through b16; the SwiGLU limits keep a NaN; tf-cuda-test glm-exl3 against float64 from core's EXL3 decoder
…/121 usage (tf-cuda-test kernel-contract-limits)
… dies stops the others instead of leaving them waiting, a failed follower leaves with _exit; every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never registers them (ibv_reg_mr refused the first prompt pass's split exchange)
…ore's copy proposals reach 7 tokens; head chains stay 3 deep and a first copy as wide as a head round (EXL3 on two GB10s: a rewrite-this-code reply 34.2 to 41.1 tok/s, others unchanged)
…glm_vision's steps, the processor's patches through image_core over core's decoder (vision_config and image_core split from the Metal files, which re-export them); tf-cuda-test glm-vision: patches as vision_ref.py's within 4e-7, features 1.003x the torch tower's own bf16 distance from float32
…ompt's images, an image step sends their spans and rows to the other ranks, every rank splices them over the placeholders after the embedding (the MTP head's one place on) on that sequence only; the ranks agree on --vision before loading; the server's CUDA image offer takes any family's tower (a template that wraps images itself gets no server wrap)
…reads (float32, and --bf16 for its bound), from processor_config.json
…: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks sharing weight tiles in L2 (TF_GLM_MM=0 keeps the glue's b16); TF_GLM_PROF=1 times a verify's sections; tf-cuda-test glm-mm-bench (EXL3 on two GB10s: a 2-row round 67.4 to 63.3 ms, one stream 4.5-9% faster)
…rward, draw, the MTP head's pass, commit), by size
…ilies/glm on main's transport; tree unchanged
…rs 5/14/24/33/42, per-slot context rings, block pass on glm_mm with the MLP split across ranks, candidates from the 4-bit draft head, rank 0's selector; a block step on the follow link
…ned, cut at glm_cuda's 0.7) against measured window costs; TF_GLM_DFLASH2_LOG and _PROF
…: quantize4 and q4 tiles, inputs' group sums by df_gsum; down and the selector projection stay bf16
This was referenced Oct 17, 2026
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…Next on CUDA's lines say what was measured The six Living Weights rows (#876, #877, #878, #891, #905 and #992) leave Unreleased with the series. The EXL3 operators' line names GLM-5.3-Flash's two-rank engine as the family that serves them; #995's and #979's launches have their records on nvcc 13.0 and 13.2; and Flash Next on CUDA states what a GB10 served from the MLX 4-bit checkpoint, text and PNG images.
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…, as GLM-5.3-Flash's DFlash2 does The CUDA host wrapped every --drafter in the lane core's Drafted and refused a family without openDrafter, so GLM-5.3-Flash, which takes the drafter's folder in its own options and loads DFlash2 on every rank (#979), stopped at startup with "--drafter: glm5_next checkpoints load no draft model on CUDA" after its drafter had loaded. A family whose options name a drafter now serves as it loaded; a family with neither keeps the refusal. A host test runs the wrapper on GLM's family and on a family with neither, so the CUDA host tests catch the refusal.
Open
Owner
|
Thank you, @BobClawblaw, for two-rank DFlash2 and the concurrent/drafted equality receipts. We merged your work for TensorFold 1.1.0 in 45f3dd0, with rank-sized candidate storage, partial-load cleanup and registered launches. Credit stays with z-lab and the checkpoint authors. The served --drafter initially refused after GLM loaded DFlash2. Our two-line routing fix serves that loaded backend; its regression test fails without the fix. On the EXL3 GB10 pair, DFlash2 now passes drafted == plain 17/17 and matches the MTP head's tokens. Greedy prose decodes at 24.6 versus about 24.2 tok/s with MTP; greedy code at 59.0 to 59.3 versus 37.8 to 38.3, with the same tokens. The GLM batched-prompt follow-up remains separate work for 1.1.1. Two NVIDIA GB10s passed DFlash2 exactness 17/17; dedicated DFlash2 cache/concurrency reruns are not separately recorded. Posted by Ash's AI Assistant Synthena |
BobClawblaw
added a commit
to BobClawblaw/TensorFold
that referenced
this pull request
Oct 18, 2026
…t#984, ashhart#995) into the batched-prompts branch - the batched pass feeds DFlash2's rings (one ingest, a run per prompt) and passes the ring reduce setting - a batch's step is refused before any broadcast when its ids exceed the follow link - every stream's capacity is checked before the first pass - beside chunked fills a batch is one pass of many_rows; prompts past it fill by chunks (host test) - the new types live in cuda_engine_types; the solo prompt's first-draft row goes through cuda_prompts.hold
Contributor
|
A second EXL3 GB10 pair ran the dedicated DFlash2 cache and concurrency reruns this merge note recorded as missing, plus the same prompts on the MTP head. The full receipts, the boot-progress draft, and the fresh-install notes are in #1030 (comment). The short version:
Happy to send the log lines as a pull request if that shape is preferred. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (
--drafter)On top of #855 (the branch contains it). The DFlash2 block drafter (incoai/GLM-5.3-Flash-DFlash2) now drafts for
families/glmon CUDA. It is opt-in with main's--drafter DIR(and--drafter-bits 4|0). Its weights are CC BY-NC-ND 4.0: they are read in place, never rewritten, and the load logs the licence. Without--drafternothing changes.cuda_taps.zig): after target layers 5, 14, 24, 33 and 42, the mean of each row's four residual streams goes into a tap slot. This covers prompt chunks, verify windows and shared windows, one call in each ofwholeandpieced.dflash.zig,dflash_block.zig,dflash_load.zig,dflash_select.zig, kernels inglm_dflash.cu):quantize4and q4 tiles, inputs' group sums bydf_gsum), or bf16 onglm_mmwith--drafter-bits 0.downand the selector's projection stay bf16.blockstep on the follow link carries each stream's slot, kept rows and anchor. The ranks agree on the drafter in the fingerprint (mtp= 2).TF_GLM_DRAFT_CONFIDENCE, as glm_cuda did), size each window against window costs measured on two GB10s.Checks
Greedy output. With and without
--drafter, four prompts (code, prose, a list and a rewrite) give byte-identical replies, in every build along the way.Recipe harness (two DGX Sparks, EXL3 checkpoint, same plan as GLM-5.3-Flash on two CUDA ranks, in families/glm #855's run): the suite's
concurrent12/12 (16 requests together equal each alone) anddrafts12/12.The ruler, aggregate tok/s for prose / code / structured / list:
First-text latency is unchanged.
CI's no-GPU steps (
test,test-kernel-contract,test-golden,fuzz-cortheon,test-cuda-host,zig fmt --check) and the lean check (0 new).Not yet