Repository navigation
Conversation
|
@jschmied, we ran GLM's checks on this branch at c48be49 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:
So the Ling commits leave GLM's checks and replies as they were. We have no Ling weights here, so this covers GLM only. |
|
Thank you, @Bizuayeu, that's the check we couldn't run. We'll push three more pieces to this branch shortly: serving at TP=1 (engine, lane backend, kept prompt states), the MTP drafter ( A rerun once it's up would be welcome, if you have the time. |
|
Pushed |
|
Thank you, @jschmied, for the GLM TP=2/3 checks and for identifying the upcoming serving, MTP and packer changes separately. The routing comparison needs care: At c48be49, the zero-limit GLM behavior and the 16-row helper's arbitrary-resume behavior remain open. Please send the pushed serving/MTP head and its delta from c48be49 so we can recheck those paths and the unchanged-shared-kernel claim. Posted by Ash's AI Assistant Synthena |
|
Thank you. Both open items are closed at
Delta from
Checked at |
|
Thank you, @jschmied, for restoring GLM's zero limit behavior and removing the 16 row WY path. We checked e29e0c2. Please send a focused revision covering these cases, with geometry rejection and allocation failure regressions, so we can recheck it. Posted by Ash's AI Assistant Synthena |
|
@jschmied, we ran GLM's checks again at e29e0c2 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:
We had also run ac58df6 the same way, with the same results. As before, this covers GLM only. |
|
@ashhart, thank you. @Bizuayeu, thanks for rerunning the GLM checks at e29e0c2. A focused revision on top of 1. Geometry rejected at config load (
Regression ( 2. Regression: 3. Capacity refusal before KDA commits ( Regression ( 4. Singleton routing groups ( Regression: the refusals and the four accepted edge cases in 5. Engine init cleanup ( Regression: 6. Undefined slot read in Regression: 7. Full checkpoint and HF source
Re-run at
|
|
Thank you, @jschmied, for the focused fixes and regression cases at 7c44606. The config refusals, Store ownership, normal sequence-capacity checks, slot initialization and engine cleanup now address the earlier cases in source. GLM's zero-limit behavior remains restored and WY remains removed. Two follow-ups remain. Please add the scratch preflight and drain-before-free error path, with a mismatched-capacity case and a test that keeps a transfer pending. We can fold these small fixes in at landing. We'll compare the newly pinned HF sources and run the host/fault checks, followed by full-model drafted/plain, resumed/fresh and concurrent/solo tests with arbitrary cuts and partial keeps. Posted by Ash's AI Assistant Synthena |
|
@ashhart, thank you. Both follow-ups are in, one commit each, on top of 1. Scratch preflight ( Regression ( 2. Drain before free (
If the drain itself fails, the memory stays allocated on purpose. Regression: Re-run at
The branch conflicts with main since 1.0.12 (#804 is in it). I can rebase it onto main if that helps the recheck. |
…layer shared SwiGLU limit, GLM's bytes unchanged tf_glm_topk takes n_group and topk_group: with more than one group it keeps the topk_group groups of the best top-2 biased sums (lower group on ties) before the top-k. GLM's one group skips that path. swiglu's limit 0 now means unclamped (as the experts' 0), and the MoE layer carries the shared expert's limit apart from the routed one's. Tests: glm-ref adds group routing vs float64 and the swiglu limit; glm-route-old compares GLM's select and swiglu with 6380c8d's fatbin; glm-route-ling checks captured Ling layer-4 routing against HF and exllamav3.
…SwiGLU limits as GLM's kernels take them config.zig parses config.json (BailingMoeV3ForCausalLM, bailing_hybrid) into a typed Config: widths, the MLA/KDA schedule ((i+1) % layer_group_size, tail MLA), dense layers, KDA safe gate, MLA head-wise gate and GPT-J rope, sigmoid routing with 8 groups keeping 4, per-layer expert and shared SwiGLU limits, one MLA MTP layer, FP8 128x128 blocks and modules_to_not_convert. It refuses what the engine does not run, each with a one-sentence reason. names.zig enumerates every tensor the config implies by role, with FP8 scale companions; against the checkpoint's index (fixtures/names.txt, folded, count and sha256 checked) it finds all 214,025 names, none unaccounted: 210,925 backbone and 3,100 of the MTP layer 72. cuda.zig gives select()'s Groups and each layer's limits for moe.Layer. Host tests: zig build test-ling (also under test-cuda-host).
…Opt NVFP4) at TP=1, synthetic checkpoints in each config.zig: Quant gains nvfp4. quantization_config fp8 (128x128) or modelopt NVFP4 (groups of 16), or hf_quant_config.json's block (parseWith); ignore / exclude_modules / modules_to_not_convert as names, dotted tails or * globs; format(module) gives each linear's format; exl3 refused naming ashhart#832. names.zig emits each format's side tensors and moduleName() gives the module a role reads. weights.zig / weights_proj.zig: one Layer per layer with a qlinear.Weight per projection (bf16, fp8g, nvfp4 through modelopt.Packer and fp8.packScales), routed experts as experts.Layer (Fp8Experts / Nvfp4Experts packers; bf16 as lane weights, no grouped bf16 kernel), the shared expert as projections, norms, fp32 q|k|v convs, A_log, dt_bias, bf16 router (a quantized one refused) and bias, the MTP layer's tensors. A file tensor whose dtype or shape disagrees with the config is refused. Tests: test-ling (21, with TF_LING_INDEX on the real index); ling-synth writes 4-layer + MTP checkpoints with the real names (bf16 282, FP8 517, NVFP4 978 tensors, NVFP4 mixing a bf16 dense layer); ling-weights: every projection decode and prompt vs float64 from the file, worst 0.456 of the bound, experts' gate-up and down, 37 small tensors byte-equal in each format; ling-weights-real: layers 0, 4, 7 with 4 experts from the FP8 release vs torch (oracle/ling_weights.py), 35 products, worst 0.414 of the bound, rel L2 <= 1.7e-3. Mutations caught: gate/up swapped, conv order, NVFP4 global dropped, exceptions ignored, expert down scales misplaced.
…thout the checkpoint, memory per format
…e heads, head-wise gate chunks_wide<HB, RW>: q and cache rows LW + RW wide, scores over all of them, values from the 512 latent; tf_glm_chunks_r16 is the 576-wide 16-head form (RW 0 builds GLM's kernels to the same SASS). absorb_q/expand_v take the head width as a template (GLM 256, Ling 128; the 128 absorb reads q_nope at a q stride, writes qa at a stride). New tf_glm_rope (GPT-J pairs, fp32 angles, sincosf) and tf_glm_head_gate (fp32 sigmoid). Host: absorbQStrided, expandV at 128, rope, ropeFreqs, headGate, attentionRoped. Checks: ling-mla (float64 bounds; decode rows, a verify window and a cut prompt across key 1536 equal the uncut call), ling-mla-real (layer 7 vs HF and float64; 400 + 32 decode rows, an 8-row window and a cut at 300 equal the prompt), glm-mla-same (GLM's MLA launches against a 6380c8d glm_glue fatbin, byte for byte).
…opt-in 16-row WY chunked prompt path - glm-kda-ling: glm_kda's fused chain, 16-row windows, wide prompt and chunked prompt on Ling-3.1 layer 0 (q/k/v convs as one q|k|v conv): rel L2 3.6e-3 vs fla, within float64's bound; replays == chain bits. - glm_kda_chunk.cu: new entry tf_glm_kda_chunk over prep's saves (kda.chunked, kda.chunkReplay). Opt-in only: GLM stays on the row path; glm_kda.cu and its fatbin are unchanged. Not the row path's bits; cuts at multiples of 16 == uncut, a prompt resumed elsewhere differs from a fresh one (measured in glm-kda-chunk). - glm-kda-chunk: float64 bounds end to end and from the saves, saves equal the wide chain's, resumed vs fresh. - glm-kda-speed: 64 heads x 128, 2048-row prompt row path vs chunked, decode row, 16-row window.
…dense and MoE MLPs, head, windows and commits Every projection runs as the qlinear.Weight and every routed-expert layer as the experts.Layer the checkpoint loaded (bf16, FP8 block, NVFP4); MLA's absorbed kv_b is derived through kv_b's own format on identity rows. ling_glue.cu rounds the residual, head gate and expert sum where the HF bf16 model does. Decode windows run KDA's three-kernel chain as GLM's forward does; commit(keep) replays the KDA states of a partial window. ling-forward-synth: synthetic bf16/FP8/NVFP4 logits vs a float64 reference routed on the same picks (2.1-2.3e-2, as far as HF's bf16 rounding points are), cut == uncut, windows == decode rows, a partial commit, batched == alone, fused == three-kernel chain, bit for bit. ling-layer-real: real FP8 layers 0, 4, 7, 71 vs HF 3.0-3.7e-3 without routing flips. GLM files untouched.
Ling's reference (inclusionAI's SwigluStepAndMul, SGLang's BailingMoeV3MLP) clamps silu's output; GLM's glue and the grouped expert kernels clamp the gate before silu. Ling now launches its own instances: tf_ling_swiglu_512/64 in glm_glue (glue.swigluWith) for the shared, dense and bf16 routed experts, and epilogue 4 of the FP8 and NVFP4 grouped kernels, decode and prompt, chosen by experts.Layer.after (set by the Ling loader, refused by affine4). Every pre-existing kernel keeps its SASS and resource usage (cuobjdump over all 40 fatbins vs a33dde0; 37 fatbins byte-identical, the other 3 only add the new instances). ling-clamp-old runs the kept FP8 (12) and NVFP4 (13) instance calls against a33dde0's fatbins byte for byte; glm-route-old and glm-mla-same still equal 6380c8d. At L = 10 both orders give the same bf16 bytes (silu is rounded to bf16 first and bf16(silu(g)) = 10 for g >= 10), so the real layers 65 and 71 and the synthetic forward measure the same digits; ling-clamp checks the order against float64 at L = 10 and L = 2 (where 16,255 bf16 gates differ) and catches 5 of 5 mutants. The float64 oracle clamps after silu too.
…ver hooks and kept prompt states Engine: a sequence per slot, prompt chunks cut at any row (marks included), verify windows of up to 16 rows, shared windows of up to 8 streams, greedy (tf_glm_top1) and sampled (sample.cu) draws, kept states as arena copies. Lanes: GLM's cuda_lanes.zig without the ranks. native.zig registers bailing_hybrid with the CUDA server. ling-lanes checks chunked == whole, drafted == plain, resumed == fresh (marks off multiples of 16), concurrent == solo, in tokens and state bytes, and the served logits against ling_forward.py's float64 on the same picks.
…h_proj, MLA + MoE on its own cache, chains of 3 mtp.Head runs the MTP layer over a view of the weights (final_layernorm as its norm, the shared head); each sequence keeps a second State for its MLA cache. Prompt passes feed every row whose next token is known, Engine.draft absorbs a verify's kept rows and chains drafts on the layer's own final-norm rows, kept states copy the MTP cache. ling-lanes --mtp: MTP-drafted == plain (greedy, sampled, resumed, concurrent), chained drafts by hand, MTP logits vs ling_forward.py --mtp float64; ling-mtp-real: the real layer vs HF's BailingMoeV3MTPLayer (oracle/ling_mtp.py).
… and checks its bytes against the host packers; ling-layer-real can dump its rows
…ed on the GPU (Packer.fp8Experts, fp4Experts), the file's pages dropped per batch
…hase; ling-layer-bytes writes every path's bytes for cmp
…P column copies -50 %), the same bytes
… no copies (4 KDA, 2 per MLP), the same bytes
…e expert kernels (same bits, -7 to -27 % expert time up to 768 rows)
…ne columns (Ling's 64), same picks and weights; ling-route-old
…dle GPU, the expert kernels and plan tiles side by side
…_spin), so other contexts' time slices miss the least
… to unclamped At 6380c8d GLM's glue swiglu (like its Metal glm_rows_glu) clamps at the limit it is given, 0 included. 51c0157 made the glue map 0 to +inf for every caller; that mapping now applies to Ling's kernels only. - glm-route-old: GLM's swiglu equals 6380c8d's kernels at limits 10 and 0 (0 was not compared before). - glm-ref: the limit-0 check runs on tf_ling_swiglu_*, the path Ling's unclamped layers take.
…ig is 6380c8d's again Nothing served called it: GLM and Ling run the row path, which resumes at any row. Removed with it: glm_kda_chunk.cu, glm-kda-chunk, glm-kda-speed (wide vs chunked timing), and the chunked path in glm-kda-ling (chain, 16-row windows and the wide prompt on Ling's layer 0 stay checked against fla and float64).
…with a named error parse ran KDA head widths, conv tap counts and the MLA widths straight into kernels built for one shape. It now refuses, before any allocation or launch: head_dim other than 128 and short_conv_kernel_size other than 4 (glm_kda.cu's DK and TAPS), key/value heads other than the query heads (KdaGeometry); kv_lora_rank other than 512, qk_nope/v_head_dim other than 128, qk_rope_head_dim other than 64, NeoX rope, a q_lora_rank without an rmsnorm instance (MlaGeometry); hidden_size other than 512/1536/4096, MLP and expert widths outside whole 128-column blocks (Width); tie_word_embeddings (TiedHead). - config_test: 26 refused values on the tiny config, each with its reason; the tiny config with the kernels' own values and the real config accepted. cuda.zig's test holds the fixed widths to glm.kda, glm.latent and fp8.block.
…oup and every group kept checked vs float64 HF's group_limited_topk scores a group by its top two (topk(2) raises on a group of one), and with one pick it skips the renormalization glue.select applies. Both now stop at config load (Routing), naming HF's rule; n_group 1, topk_group == n_group and groups of two stay accepted. - config_test: the refusals on the tiny config, and the four accepted edge cases. - glm-ref: glue.select against the float64 rule on five more shapes: one group, 8 of 8 groups kept, groups of two (64 experts in 32, 16 in 8) and the tiny config's 16 in 4; picks equal, weights within 2^-16.
… driver to fail every call in turn A failed name copy or named.append used to free the device buffer already in buffers (Store.deinit freed it again) and lose the copied name. alloc now reserves both lists and copies the name first, then allocates and appends without failing: on any error the store holds nothing new. - fake_driver.zig: the driver's entry points over host memory (kernels no-ops), counting allocations and handles, with call n failing on request; host tests and GPU-free checks build on it. - store_test: every host allocation failing in turn (checkAllAllocationFailures) and every device call failing in turn leave no leak, no buffer held and no buffer freed twice. The old alloc fails both.
…s.zig read the entry's unset slot) opening() did getOrPut and then freeSlot(), which walks every entry, the new one included, before its value was set. In a release build a reused bucket keeps its old slot, so a free slot could count as taken and the stream got a higher one. Lanes.claim picks the slot first, then puts the entry. - slotCheck: claims over buckets that still hold slot 0 from cleared entries (the same keys hash back into them), as a host test and as `ling-slots` in the release build, where the old order fails (SlotTaken).
…ns, not at the first MLA layer The capacity check sat in block_mla, after the KDA layers before it had run: a prompt chunk past the capacity had already committed those layers' states and conv windows (and a window had written its saves) when CacheFull came back. Forward.rowsOf now refuses every segment without room first, so a refused request changes nothing. The MTP layer, one MLA block, already refused before its cache write. - ling-lanes (engine checks): at 1012 of 1024, a 30-row prompt chunk, a 16-row window and (with --mtp) MTP rows past the end each come back CacheFull with the sequence's arenas byte for byte as they were; the next window equals a twin's (tokens, logits, states) and a new request on that slot equals one on a fresh slot. With the old check the arenas differ and the next commit fails (KdaLayersOutOfStep).
… sync fails; every init failure checked The one gap in init's errdefer chain: sequence 0 (its arena, MTP cache and hold row) and the slot list leaked when bind(0) or the closing synchronize failed. Both are freed now. - ling-init-faults DIR [--mtp]: native.open (Engine.init, sequence 0, the lane backend) on the fake driver, once cleanly, then with each driver call and each host allocation of the open failing in turn; each failed open must leave no device or pinned allocation, no module/stream handle, no host leak and nothing freed twice. bf16, FP8 and NVFP4, with and without --mtp: 714 to 1,832 calls and 251 to 426 allocations a clean open, 8,591 failure points, none left anything. Dropping this fix, or any of five errdefers in engine, graph, state, weights and mtp, fails it.
…s names, dtypes and shapes, from headers alone The loader's shapes now come from one table, names.dims (each role's [n, k], value count or conv shape); weights.zig reads every tensor through it instead of computing shapes inline (the same shapes, ling-weights unchanged). - ling-ckpt-check DIR [INDEX]: each shard's header only, every tensor against the weight map at the loader's dtype and shape (FP8's e4m3 weights and scale_inv blocks, ModelOpt NVFP4's codes and scales, bf16 exceptions, small tensors and convs), nothing missing, nothing extra, the bytes against the index's total_size. On the FP8 release (54 shards over sshfs, 3 s): 214,025 tensors, 3,100 of them the MTP layer's, 573,753,281,152 bytes, no mismatch. The three synthetic formats pass too; a wrong dims entry or scale shape fails it.
…ng-3.1-flash@a6943f4b by sha256 ling_hf.py pins the revision (a6943f4bf04f8041ad983d2b58dd43b5ba86ae4a: config.json as of ashhart#2; the modeling, configuration and index files are unchanged since the upload) and the sha256 of modeling_bailing_moe_v3.py, configuration_bailing_moe_v3.py, config.json and model.safetensors.index.json. ling_mtp.py checks the modeling package it imports and the checkpoint's config and index; ling_layer.py takes the modeling folder the chain was captured with and checks it; ling_weights.py checks the index. Any other copy stops the oracle naming both hashes. - test-ling holds fixtures/config.json to the pinned sha256. - The local copies match HF at that revision byte for byte; ling_layer.py rebuilds the layer-4 fixture identically.
…uns, not at the first MLA layer Forward.rowsOf checked the sequence's capacity only; the scratch's attention partials (Buffers.nch, sized from the capacity the Buffers were made for) were checked inside block_mla, after the KDA layers before it had committed and after the segment's latent row was written. With scratch smaller than the state a prompt chunk or window that fit the state still changed it before CacheFull came back. block_mla.room now checks both; rowsOf calls it for every segment when the graph has MLA layers, and block_mla calls it for every segment before its first cache write (the MTP layer, which runs the block without rowsOf). - ling-lanes (engine checks): Buffers made for 512 positions against sequences of 1024, twins at 500: a 30-row prompt chunk, a 16-row window and (with --mtp) 16 MTP rows each come back CacheFull with the arenas byte for byte as they were; the next window equals the twin's (tokens, logits, states) and a new request on that slot equals one on a fresh slot. With the preflight reverted the arenas differ and the next commit fails (KdaLayersOutOfStep); with only block_mla's order reverted the MTP case alone fails.
…a failed drain leaks them on purpose A failed packer launch, upload or copy returned through defers that freed what queued work could still read or write: the expert stage (pinned host and device, weights_proj Stage.deinit), the lane staging of bf16, FP8 and NVFP4 projections, the store of packed outputs (weights.load), absorbOf's identity rows and results, the engine's weights and graph when a later init step failed, and keepState's copy target. The Reader now tracks whether an upload or launch is queued (settled() drains it first); the other paths drain before freeing. A drain that fails leaves the memory allocated: Store.drop, Weights.drop and Graph.drop free the host lists and leak the device buffers. - fake_driver.zig: async copies, memsets and launches stay pending until a successful synchronize; frees with work pending are counted, failures can persist from call N on (drains included), and leaked memory can be swept. - ling-init-faults: every driver call and host allocation of the open and a kept state fails in turn with transfers pending, then every call from N on; no free may come while work is pending, and memory may stay held only after a failed drain. Over bf16, FP8 and NVFP4 with and without MTP: 15,180 points (1,804 failing with a transfer pending), all clean. Each fix reverted alone fails it (Stage.deinit, lane staging, NVFP4 lane staging, load's store, absorbOf, the engine's weights, keepState).
|
Rebased onto main |
…ent check counts, the recorder on main
|
We are pausing here. The branch stays as it is at Why: the whole model does not fit one GB10. FP8 is 574 GB and NVFP4 about 324 GB, so an end-to-end run, real MTP acceptance and TP all need several Sparks; this PR was checked on synthetic checkpoints and single real layers only. A one-Spark build would have to be around 1.25-1.7 bpw EXL3, which TensorFold reads only once #832 is in. We tried our own 1.25 bpw conversion and stopped it: with 1-bit experts, each layer's error rose about 1.5x over 1.5-bit experts and the hidden states drifted quickly with depth, so we didn't expect a usable model. Next steps: someone with two or more Sparks can take TP and the end-to-end checks ( |
|
Thank you, @jschmied, the scratch and failure controls leave useful work available even though the whole model exceeds one GB10. We checked e99e650 and will keep the branch parked, following your pause. It stays parked until a quality-qualified pack fits one GB10 or the TP fabric in #636 lands, with resumption left to you. The failed-drain teardown follow-up at engine.zig:149-163/199-204 can go in at landing. We will retain your synthetic, allocation and real-layer receipts with their scope; whole-model, real MTP and TP qualification still need their own runs when work resumes. Posted by Ash's AI Assistant Synthena |
3 participants
What this changes
Ling-3.1-flash (#850) on CUDA at TP=1, built on #804's GLM kernels as agreed there and on #850. #804 is in main since 1.0.12, so this PR is stacked on nothing: the branch is rebased onto main
46a08c18(TensorFold 1.0.12), and every commit on it is this PR's. The family is registered with the native CUDA server (model_typebailing_hybrid) and serves through the lane core at TP=1, with the prompt cache and MTP drafts. The whole model does not fit one Spark, so serving is checked on synthetic checkpoints and a 4-layer slice of the real one.docs/recipes/ling-3.1.mdlists what exists, what is open, and how to test without the 572 GB checkpoint.f9fc1a78); a division-free branch for groups that are whole lane columns, which GLM's single group never enters (e78d23bb);swiglukeeps its limit as given, 0 included, as at6380c8da; only Ling'sswigluWithmaps 0 to unclamped (32055e9e,991383f3);7d184cb7);ef8f12f3). The opt-in 16-row chunked prompt path that commit added is removed again in03171bf5: nothing served called it, and GLM'scuda_kda.zigis6380c8da's file again;min(silu(g), L) * clip(u, -L, L), as new kernel instances (bfb44ddc);tf_pack_fp8_expertsinmodelopt_pack.cu, an added kernel for Ling's loader; the existing packers are unchanged (bbe360f9).zig/src/families/ling/, format-agnostic:a6559571);qlinearandexperts. EXL3 is refused and points to EXL3 checkpoints on the native engine (tracker) #832 (d46e3723);ling-synthfor small synthetic checkpoints in each format;commit(keep)(6f583baf);a3f14d30):Enginewith a sequence per slot, prompt chunks cut at any row, verify windows of up to 16 rows, shared windows of up to 8 streams, greedy and sampled draws;Lanes, GLM'scuda_lanes.zigwithout the ranks;native.zigwith the server hooks and the prompt cache'ssnap_*hooks. A kept state is the sequence's arena copied whole, and a mark may fall on any row;model.layers.72) as the drafter:eh_proj(enorm(embed(t+1)) | hnorm(h_t)), MLA + MoE on its own cache, chains of up to 3 (577436b0). The server loads it unless--no-drafts;bbe360f9);99652bab,b3faea69,5b006370), and timing tests (7bd7a40a,412f66fc,0ffce4f7,bd7c23e3);faf7af1c).KdaGeometry,MlaGeometry,Width,TiedHead(66579cba);Store.allocowns a buffer only once it is in both lists;fake_driver.zigfails every driver call in turn (674163fa);82b3817b);group_limited_topkcan't run them (92afed61);Engine.initfrees sequence 0 and the slot list whenbind(0)or the last sync fails (0ba2adfb);0ba510ad);names.dims(5649c63b);inclusionAI/Ling-3.1-flash@a6943f4bby sha256 (e4c9bcc5);5e931c3a);1395e267).Receipt
Environment: TensorFold
46a08c18(1.0.12) plus this branch (e99e6505; its last commit only updates the recipe, so the checks below ran at1395e267), one GB10 (sm121), CUDA 13.0, Zig 0.17.0. Checkpoint:inclusionAI/Ling-3.1-flash, block FP8, pinned ata6943f4b; its real layers read one at a time.Serving (
ling-lanes, under upstream's round loop, prompts in 37-row chunks), bf16, FP8 and NVFP4, each with and without--mtp: 100 and 109 checks pass in each format. Tokens in every check, state bytes in the engine-level ones:Served logits against
ling_forward.py's float64 on the same picks: 2.09e-2 to 2.24e-2 rel L2, as far as the HF model's own bf16 rounding points are (2.10e-2 to 2.25e-2). On the real FP8 release,--layers 0,1,2,3(real embedding, layers 0-3, final norm and head) passes the same exactness checks; that run is not in the re-run at this head.Graph exactness, bit for bit, in bf16, FP8 and NVFP4 (
ling-forward-synth): a prompt cut at 48 equals the uncut prompt; an 8-row verify window equals 8 decode rows; keeping 3 of 8 rows and then running 5 more leaves every KDA state, conv window and MLA cache row equal to the serial run; two sequences batched equal each run alone. On real layers, a prompt cut at 300 equals the uncut prompt and a 16-row window equals 16 decode rows.Against the reference:
a78a7a49's;ling-mtp-real) against HF'sBailingMoeV3MTPLayer: 7.28e-3 on the 488 of 511 rows whose top-8 agrees;Full checkpoint (
ling-ckpt-check, shard headers only): 54 shards, 214,025 tensors (3,100 of the MTP layer), 573,753,281,152 bytes, no name, dtype or shape mismatch, nothing missing or extra. The shard data is not hashed.Fault paths (
ling-init-faults, onfake_driver.zig): every driver call and host allocation of the open and a kept state fails in turn with transfers pending, then every call from N on. Over bf16, FP8 and NVFP4, with and without MTP: 15,180 failure points (1,804 failing with a transfer pending), none leaves anything held, leaked, freed twice or freed before a drain.Loading one MoE layer's 512 FP8 experts (7.5 GiB): 6.1 s from local disk with the GPU packer, 14.3 s with the host packers before; host peak 1.3 GiB. Every packed byte equals the host packers' (
ling-pack).Decode and prompt speed: not measured. The whole model does not run on one Spark. The speed pass is measured per op only: prompt KDA and MLP column copies -50 % (
99652bab); FP8 expert time -7 to -27 % for prompt chunks up to 768 rows (5b006370).GLM unchanged:
glm-route-old(select, and swiglu at limits 10 and 0) andglm-mla-same(13 launches, 14.3 MB of outputs) run glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's6380c8dafatbins next to the new ones, byte for byte;ling-route-oldchecks the division-free branch against the older select's bytes;cuobjdump, all fatbins);ling-clamp-oldchecks the kept expert instances' bytes;c48be499,ac58df63ande29e0c20(now03171bf5): one host, TP=2 and TP=3, and the server at TP=2 against6380c8da's replies. All pass. GLM's and the shared code have not changed since03171bf5.Tests run at
1395e267:glm-ref,glm-route-old,ling-route-old,glm-mla-same,glm-kda-ling,ling-slots,ling-ckpt-check(three synthetic checkpoints and the full release),ling-init-faultsandling-lanes(three formats, with and without--mtp; ids and picks byte-equal toa78a7a49),ling-weights(three formats, worst 0.433 of the bound),ling-packlayer 4,ling-layer-reallayer 4;zig build test(1390/1398, 8 skipped),test-ling30/31 (1 skipped: it needsTF_LING_INDEX),test-cuda-host237/237,zig fmt --check,lean_check517, as on main. Deliberate mutations, all caught: routing 6, MLA 6, config 8, loader 5, graph 6, clamp 5, serving 5, MTP 5, and each review fix reverted alone.Not run, and why:
What is still open
glm/cuda_shard.zigand GLM-5.3-Flash on two CUDA ranks, in families/glm #855.--recordis refused forbailing_hybrid. The rows are the final-norm rowsEngine.verifyalready keeps.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (On synthetic checkpoints in three formats and the real model's first 4 layers; not on the full model.)Open question for @ashhart (from #850 and your reply of 2026-10-15): group-limited top-k is built into #804's
tf_glm_topk, since Ling's MoE runs through #804's routing. It ranks sigmoid(logit)+bias as HF does for Ling; #798'smoe_glue.curanks logit+bias, so the two can't simply replace each other. Where should the shared routing live?