[Mirrdhyn]

What this changes

Flash Next's prompt rows on CUDA run on new tiles that keep 1.1.0's arithmetic exactly: the factored per-group sum of #952 (d = mma(k0..15) + mma(k16..31) on the raw codes, then fma(xs, b, fma(d, s, acc))), in the same order. Decode keeps its bits and its speed, but it runs two of the new kernels too: the indexer scores tile on every sparse row, and the one-buffer attention tile on passes of 17 to 256 rows (decode's own fn_attn_chunks_ahead, up to 16 rows, is recompiled from the changed template with the same sums). The other tiles start at 64 rows (fused attention above 256), so they run on prompt rows only. Nothing changes in other families or on Metal. Follows up on #869 (closed into #952) as agreed: no change of precision, so the bits stay 1.1.0's.

16 commits, 31 files, all under zig/kernels/cuda/flashnext_*, zig/src/families/flashnext/, zig/src/cuda/route_rows.zig (used by Flash Next only; GLM's router is Metal), the kernel contract and the Flash Next CUDA tests:

• dense matmuls on a 64x128 factored tile (flashnext_fact.cu), K slices folded in the CTA, summed in slice order;
• hyper-connection read-out (down / up-mix) on factored tiles (flashnext_hcf.cu), bits of fn_hc_down / fn_hc_upmix;
• experts on factored 64-pair tiles (flashnext_fexp.cu), bits of expert_rows;
• attention tiles: one buffer for keys then values (17 to 256 rows: prompt chunks and wide decode rounds), and above 256 rows 1024 rows a launch with every chunk merged in the program;
• prompt chain on two threads a state row (from 64 rows), and indexer scores 8 rows a block (every sparse row, prompt and decode), both in cuda: add experimental native Flash Next family and local tensor parallelism #952's sum order;
• router logits 128 rows x 64 experts a block (same sums);
• TF_FLASHNEXT_CHUNK takes 1 to 16384 rows and refuses anything else at load, before the loader takes a stream, a module or a buffer;
• tile launches in cuda_tiles.zig / cuda_tile_kernels.zig (files under 600 lines);
• every new kernel registered in the kernel contract as a replacement of the 1.1.0 kernel it stands for, bit equality required, with nvcc 13.0's records in registry.cuda and 13.2's / 13.3's where they compile differently;
• flashnext-attn also checks the fused prompt attention launch against chunks then merge, byte for byte.

Receipt

• Environment: TensorFold v1.1.0 (76aafcf) + this branch; NVIDIA DGX Spark, GB10 sm_121, driver 595.84, nvcc 13.0.88 / 13.2.86 / 13.3, Ubuntu 24.04; checkpoint Flash Next MLX 4-bit with MTP (qwen4_exp). Server --context 131072 --parallel 2 --prompt-cache-gib 0, page cache dropped before each serve.

• Exactness:

  • output digests on 4 prompts (short, medium, 13k, 40k), chunks 16384 and 1536: 9b775054 800ae936 8c4ecf25 a9848d63 on 1.1.0 and on this branch;
  • bench_concurrent.py --alone --serial, T=0 levels 1/4 and T=1 levels 1/2, both chunks: 0 unequal, 0 failed, on both trees;
  • bench_cache.py rows below: the same token_sha on both trees.

• Decode (bench_openai.py, 256 tokens, 3 reps, T=0, tok/s): unchanged.

  	1.1.0	this branch
  code, 1 stream	81.7	81.3
  chat, 1 stream	68.7	68.8
  code, 4 streams (aggregate)	79.7	78.8
  chat, 4 streams (aggregate)	69.6	69.2

• Prompt speed, bench_cache.py --require-cold --repeats 1 --streams 1 --tokens 16, chunk 16384, server started fresh per tree, ttft_s in s (qualified on both; cached_tokens 0 on every row, the prompt cache being off):

  prompt	1.1.0 first	this branch first		1.1.0 repeated	this branch repeated
  32k	30.86	19.63	x1.57	26.63	15.30	x1.74
  64k	59.09	36.26	x1.63	55.77	31.60	x1.76
  128k	125.62	79.55	x1.58	123.84	66.22	x1.87

  And a size sweep, cold first request per prompt, median of 3 distinct prompts per size, s:

  prompt	1.1.0 chunk 16384	this branch		1.1.0 chunk 1536	this branch
  2k	1.64	1.17	x1.40	1.79	1.39	x1.29
  8k	6.63	4.12	x1.61	6.94	5.10	x1.36
  32k	27.83	16.71	x1.67	28.21	20.11	x1.40
  64k	56.66	32.81	x1.73	57.22	39.59	x1.45
  128k	121.71	68.81	x1.77	—	—

  Memory: model 89.27 GiB on 1.1.0, 89.28 GiB on this branch (chunk 16384).

• 1.0.12 on this checkpoint: refuses at load (the native CUDA engine has no backend for qwen4_exp checkpoints yet), so the "before" is 1.1.0 = cuda: add experimental native Flash Next family and local tensor parallelism #952.

• Tests run (exit 0 on both trees unless noted):

  • zig build native, install, test, test-kernel-contract, test-golden, fuzz-cortheon, test-cuda-host, zig fmt --check;
  • lean check: 503 problems, the 1.1.0 baseline;
  • kernel-contract built by nvcc 13.0 and by 13.2: SUMMARY 65 kernels: 65 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems (1.1.0: 55 kernels, same result);
  • kernel-contract-limits on sm_80/86/89/120/121 fatbins: PASS under 13.0 and 13.2;
  • tf-cuda-test smoke, symbols, route-rows, experts-rows, flashnext-hc, flashnext-attn, flashnext-dense-hc, flashnext-draft-affine, flashnext-shared, flashnext-nvfp4-tp, norms, norm-rows, sample, projection-f32, mxfp8, quant-ref, modelopt, glue, glm-ref: 19/19 pass;
  • on the final head (Ash's two review points): zig fmt --check, native, test, test-kernel-contract, test-cuda-host, lean 503, kernel-contract 65/65 with 0 limit problems, flashnext-attn;
  • community benchmark --backend cuda: one repetition of 5 (code, T=1) ends on an early EOS, on 1.1.0 as on this branch.

• New tests fail before the change: the new kernels do not exist before it, so I broke each one on purpose instead and checked the new tests go red. Eight mutations in one tree, one in each new kernel: the dense, expert and HC epilogues with the two fmas swapped, the indexer scores' four terms reordered, the prompt chain's in-thread tree made sequential, fn_hc_rinv summed in reverse, one ulp on one router output and one ulp on the fused attention's scale. The contract reported 57 pass gates 1-4, 8 fail, the 8 being exactly the registrations those mutations touch (dense, dense folded, expert gate-up, expert down, HC read-out with fn_hc_rinv, indexer scores, prompt chain, router wide); flashnext-attn caught the fused attention (attention dense fused: 132 of 61440 words differ). The clean tree stays green.

• TF_FLASHNEXT_CHUNK at load: a loader-level control (cuda_lanes.zig) opens with every driver entry point undefined and 0, 16385, 4k, and expects BadChunk with nothing allocated. With the parse moved back after the loads it fails (expected error.BadChunk, found error.FileNotFound).

• Not run, and why:

  • Metal: no Mac here; this branch touches no Metal file.
  • Goldens: not regenerated, since the bits do not change; test-golden passes as is.
  • nvcc 13.3: this branch's records are complete for it, but 1.1.0's own 13.3 set holds only tf_nemo_topk, so kernel-contract-limits under 13.3 reports 55 stale records on 1.1.0 and 53 here (none of them this branch's launches; fn_qsa_select, which this branch recompiles, is the two fewer).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. The bits do not change.
• A new or changed GPU kernel is registered with the kernel contract; the CUDA SUMMARY line is above.
• Prompt processing is no slower than the last release.
• CUDA GB10 run; Metal not run (no Mac, no Metal file touched).
• New tests fail before the change and pass after it (no new dependency to skip without).
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @Mirrdhyn. The factored dots, affine rounding and folded slice order look consistent with the old kernels, and your GB10 receipts help.

Please parse TF_FLASHNEXT_CHUNK before acquiring resources, or add cleanup after the kernel set loads: an invalid value currently returns BadChunk without unloading that set. Add a loader-level refusal/cleanup control.

The new indexer also runs during sparse decode, so please qualify the prompt-only wording. We'll carry it onto the 1.1.2 line ourselves, keeping the restored copy kernel and every capture buffer, including the folded-slice consumers, and land it in 1.1.2 after the integrated operator, attention-seam, drafting/cache/concurrency, media, supported-rank and speed gates pass, with the existing goldens preserved.

Posted by Ash's AI Assistant Synthena

[Mirrdhyn]

Thanks Ash. Both points are on the branch, two commits on top:

• 4dd3cbb0: TF_FLASHNEXT_CHUNK is now parsed first, in the loader (cuda_lanes_load.open) and in Engine.initRank, before the stream, the glue, a captured set or any weight loads. A new loader-level control in cuda_lanes.zig opens with every driver entry point undefined and 0, 16385, 4k: BadChunk, nothing allocated. With the parse moved back after the loads, it fails (expected error.BadChunk, found error.FileNotFound).
• 4637e1ba: the wording. It is a bit wider than the indexer: decode runs qsa_tile::scores on every sparse row and fn_attn_chunks_one on passes of 17 to 256 rows, and fn_attn_chunks_ahead (up to 16 rows) is recompiled from the changed template with the same sums. The comments and the PR description now say so; the tiles that start at 64 rows (dense, experts, HC read-out, chain, router) and fn_attn_fused (above 256 rows) are prompt-only.

On the new head: zig fmt --check, test, test-kernel-contract, test-cuda-host and native pass, lean 503, kernel-contract SUMMARY 65 kernels: 65 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems, flashnext-attn passes.

The zig-windows job fails the same way on the other open PRs, including a docs-only one; none of its errors are in files this PR touches.

Thanks for carrying it onto 1.1.2.

[ashhart]

The Flash Next CUDA prompt tiles are included in 1.1.2 (https://github.com/ashhart/TensorFold/releases/tag/v1.1.2), with your original authorship retained. We added an ownership check so tiled read-out and fused attention run only when native glue owns every operation they replace. Our single-GB10 measurements against 1.1.0 are in the release notes. Version 1.1.2 also fixes #1117: request text that spells a control token is encoded as ordinary text, so those requests can produce different replies from earlier releases. Closing this as carried, and thank you.

It landed as 16 commits, a929aac through 1be8530.

- Synthema, Ash's AI Assistant.