cuda: add experimental native Flash Next family and local tensor parallelism

RUNBOOK.md

@@ -123,6 +123,8 @@ The archives require glibc 2.28 or newer and an NVIDIA driver providing `libcuda
 CUDA serves Nemotron 3.5 Lightning, qualified on GB10 (`linux-aarch64`, greedy and sampled) and on Ampere cards with
 compute capability 8.6 (`linux-x86_64`, greedy: the RTX 30 series, RTX A6000, A10 and A40), and checked on an RTX PRO 6000 Blackwell (compute
 capability 12.0, `linux-x86_64`).
+Flash Next also serves on CUDA from its MLX 4-bit checkpoint, on the built-in kernels, run on a GB10: see
+[Flash Next on CUDA](docs/recipes/flash-next-cuda.md).
 
 On a GB10 arm64 host (an x86_64 host uses `linux-x86_64` in the same commands):
 

docs/kernel-contract.md

@@ -108,3 +108,7 @@ CUDA: the RMS norm rows (16-byte form and the fp32-out tree), Nemotron's router
 tile, lane_gemv and its split-K form (both bit-equal to it), prompt16's FP4 tile with bf16 and fp32 out, qmmf's FP4
 tiles with fp32 out (K slices met in a cluster, fused, or by `reduce_kernel<true>`), and the reduce-scatter's
 `tf_rank_sum`.
+
+## CUDA compiler resource baseline
+
+The registered usage arrays were re-derived with NVCC 13.3.73 (`cuda_13.3.r13.3/compiler.38244171_0`) for SM 80, 86, 89, 120 and 121. `kernel-contract-limits` reads each compiled image and reports its register count, static shared memory and thread bound. Source changes or a different compiler may change these values; re-derive the arrays and run the live audit rather than changing the audit tolerance. Launch dimensions, dynamic shared memory, device limits and planted controls remain independent checks. Only SM 120 was executed on hardware in this update; the other four records come from their compiled images, not hardware execution.

docs/recipes/README.md

@@ -9,7 +9,7 @@ the native engine, or the Python engine frozen at 0.6.6.
 | Nemotron 3.5 Super VL | [Native CUDA on one or two GB10s, images](nemotron-3.5-super-vl.md) |
 | Qwen3.8-27B | [MLX, quantization and CUDA](qwen3.8-27b.md) |
 | Qwen3.5-2B | [Native Metal preview, tied affine 4-bit](qwen3.5-2b-native.md) |
-| Qwen3.8 Flash Next | [MLX prefill and CUDA](qwen3.8-flash-next.md), [CUDA images](flash-next-vision.md) |
+| Qwen3.8 Flash Next | [MLX prefill and CUDA](qwen3.8-flash-next.md), [native CUDA](flash-next-cuda.md), [CUDA images](flash-next-vision.md) |
 | Ternary Bonsai 2 27B | [MLX](ternary-bonsai-2.md) |
 | GLM-5.3-Flash | [MLX on a 256 GB Mac, two-rank CUDA](glm-5.3-flash.md) |
 | Gemma 4 26B-A4B | [MLX, fused one-row decode](gemma-4.md) |

docs/recipes/adding-a-zig-family.md

@@ -38,7 +38,7 @@ Some of the host side already lives in core, but so far each piece serves one fa
 - staggered segments (`zig/src/core/segments.zig`, and `zig/src/cuda/segments.zig` on CUDA) serve Flash Next on Metal and Nemotron
   on CUDA;
 - every host returns the HTTP contract in `zig/src/core/engine_api.zig`, and `zig/src/core/lane_host.zig` serves Nemotron on Metal
-  and CUDA. Flash Next has its own host and serves one reply at a time.
+  and CUDA, and Flash Next on CUDA. Flash Next on Metal has its own host and serves one reply at a time.
 
 The GPU side is barely shared yet, and that costs time. In the GLM-5.3 port each extra row in a window cost about
 4.4 ms of GPU time, against about 2.3 ms for the row's own experts. Row kernels read every replicated weight again for
@@ -71,11 +71,11 @@ lane kernels go in, check against the engine's own plain output (step 4).
 
 | Piece | Shared now | Family copies or gaps |
 | --- | --- | --- |
-| Lane kernels | `zig/kernels/metal/ops/qmv.metal` (affine 4-, 6- and 8-bit row matmuls, MLX's arithmetic) and `zig/src/core/frags.zig` (the prompt layout on M1-M4) | Flash Next: `zig/kernels/metal/decode/fn_lane.metal` (6-bit, groups of 32, tensor ops, up to 16 rows). Kimi K3: MXFP4 experts and bf16 projections. Nemotron: generated per-shape kernels |
+| Lane kernels | `zig/kernels/metal/ops/qmv.metal` (affine 4-, 6- and 8-bit row matmuls, MLX's arithmetic) and `zig/src/core/frags.zig` (the prompt layout on M1-M4) | Flash Next: `zig/kernels/metal/decode/fn_lane.metal` (6-bit, groups of 32, tensor ops, up to 16 rows). Kimi K3: MXFP4 experts and bf16 projections. Nemotron: generated per-shape kernels. On CUDA, `zig/kernels/cuda/experts_rows.cu` runs affine 4-bit group-32 experts over a window's rows with the decode kernel's sums, and `route_rows.cu` the router logits and softmax top-k (Flash Next) |
 | GPU-side rounds | the host loop, depth rule and copy proposer in `zig/src/core/lanes`, used by Nemotron | Flash Next: its own host loop, plus `fz_accept`, the ring and event chaining (`zig/src/families/flashnext/replay.zig`). Nemotron on Metal: `zig/src/families/nemotron/gpu_round.zig` for a lone stream. Kimi K3: none |
 | Several Macs | the MCDMA fabric (`zig/src/fabric`) | Flash Next: `zig/src/families/flashnext/tp.zig` splits rows and holds the whole model on each Mac. Kimi K3: `zig/src/families/kimi_k3/parallel.zig` is a split plan with no link yet |
 | Prompt path | segments; the prompt cache, used by Flash Next, GLM, Nemotron and the Qwen3.8-27B | per-family prompt kernels and snapshots |
-| Serving | `engine_api` for every host; `lane_host` for Nemotron | Flash Next: its own host, one reply at a time |
+| Serving | `engine_api` for every host; `lane_host` for Nemotron, and for Flash Next on CUDA | Flash Next on Metal: its own host, one reply at a time |
 
 **The order.**
 1. GLM-5.3 builds the core pieces first:
@@ -194,6 +194,12 @@ On M5 tensor units. Chips before M5 need their own path.
     250-500 ms, on its own queue, inside the engine process. Without it macOS unwires the set after 1-2 s idle, and
     the next request pays a full re-wire: 2.7 s on K3.
 
+Native CUDA families may export `healthy(ctx: *anyopaque) bool`. The native host forwards this callback to
+lane health and admission. It must read thread-safe host state without binding a GPU context, waiting on device
+work or taking the inference thread's lock. Families without it retain the lane host's normal closing/stale checks.
+Unavailable or busy admission returns HTTP 503 before a streaming response opens. Context-window refusals retain
+their existing streamed error policy; failures after admission remain stream errors.
+
 ## Across several Macs
 
 From Kimi K3 on four Macs:

docs/recipes/cuda.md

@@ -8,7 +8,7 @@ They are not the native Zig release's support matrix. The kernel paths, the prom
 (`--prefill-fp8`, `--no-prefill-fp8`, `--precision`) and the request and memory notes below are that engine's too,
 and the native binary refuses those flags.
 
-The native binary registers only Nemotron on CUDA, qualified on NVIDIA GB10 and on Ampere cards with compute capability 8.6 (the RTX 30 series, RTX A6000, A10 and A40).
+The native binary registers Nemotron on CUDA, qualified on NVIDIA GB10 and on Ampere cards with compute capability 8.6 (the RTX 30 series, RTX A6000, A10 and A40), and Flash Next from its MLX 4-bit checkpoint, run on a GB10 ([Flash Next on CUDA](flash-next-cuda.md)).
 A user's receipt for 1.0.4 on a bare-metal RTX 3090, at 225 W and 350 W, is in [#592](https://github.com/ashhart/TensorFold/issues/592).
 GLM-5.3-Flash on CUDA, EXL3 loading and DFlash2 drafting remain Python-engine features, not native upgrade paths.
 Inspect the installed binary with `tensorfold-native --version` and `tensorfold-native capabilities --json`.

docs/recipes/flash-next-cuda.md

@@ -0,0 +1,159 @@
+# Qwen3.8 Flash Next on CUDA
+
+The native CUDA build serves Flash Next (`qwen4_exp`) on one NVIDIA GPU from an MLX affine 4-bit, group-32 checkpoint
+with its MTP head, on kernels built in this repository. A set of the Python 0.6.1 engine's Triton kernels, captured
+and packed, can stand in for its glue to compare against that engine's teacher path. @AdrianBinDC started the port in
+[#476](https://github.com/ashhart/TensorFold/pull/476); it is tracked in
+[#472](https://github.com/ashhart/TensorFold/issues/472). It has been run on a GB10 (sm_121) only.
+
+## Experimental ModelOpt and two-rank path
+
+The CUDA family also reads ModelOpt NVFP4 routed experts, including CSF-compressed scale planes, with separate
+MXFP8 shared experts and dense projections and the checkpoint's BF16 activations. Two local CUDA ranks partition
+expert intermediate channels and gather disjoint vocabulary ranges without reducing their BF16 bits. Each rank
+owns its recurrent state, and peer failure makes the serving rank unavailable rather than returning partial results.
+
+This path remains experimental pending independent full-model numerical qualification and complete GPU kernel
+contract receipts. Compile success and host tests do not establish checkpoint fidelity. Performance and context
+limits must be measured with the exact checkpoint, binary, device topology and request settings. Prefix retention
+with MTP is outside this path's qualified scope; evaluate with `--prompt-cache-gib 0`.
+
+## Run
+
+```bash
+tensorfold-native serve /path/to/checkpoint --temperature 0
+```
+
+- `--context N` takes up to the model's 262,144 tokens; the default is the whole window. Past 2,048 keys each attention
+  layer reads 512 blocks of 4 keys a row, chosen by the model's block indexer, as the Python engine does.
+- `--parallel N` admits N streams. Each stream's caches are its own: about 110 MiB of DeltaNet state and the n-gram
+  tail, plus 28 KiB a token of keys, values and indexer keys, and with drafts its MTP head's cache.
+- `--no-drafts` serves without the MTP head; `"draft": false` on a request does the same for that request.
+- `TF_FLASHNEXT_DRAFTS` (1 to 15, default 6) and `TF_FLASHNEXT_CONFIDENCE` (default 0.60) set the head's chain.
+- The engine uploads 73.3 GiB of the checkpoint: the n-gram tables stay mapped from the shards, and the vision tower is
+  not loaded.
+- It refuses another quantization, other shapes, and semantic settings its kernels fix (RMS epsilon, RoPE base, rotated
+  share, type and scaling, the gate and activation types, attention bias, the layer roles, the n-gram layer, kernel
+  and dilation): config.json is checked at load and the log names the field. It also refuses `--segments`, `--slide`
+  and a folder whose shared experts carry Living Weights neurons.
+- `TENSORFOLD_CUDA_KERNELS=DIR` swaps the glue for a captured set (see [A captured set](#a-captured-set)) when the set
+  holds every entry point the engine launches; a set without them (Nemotron's, say) is passed over, and the startup
+  line says which glue runs. With a set loaded, `TF_FLASHNEXT_OWNED` (`all`, or Triton names such as
+  `_chunks,_merge`) runs those entry points on our kernels and the rest on the set's, to put a difference down to one
+  kernel.
+
+## What the core runs and what the family runs
+
+The family plugs into the core's pieces and adds no scheduler:
+
+- The core's round loop and lane host drive every token. Drafted windows, shared rounds of streams without drafts, and
+  prompts a chunk at a time between other streams' rounds (`prefill_step`) all go through them.
+- The core's prompt cache keeps and resumes a stream's state at the marks it asks for. The family copies the state
+  (`cuda_snap.zig`), with the MTP head's cache rows for a drafting stream; the head reads one token past a row, so
+  states keep with a lookahead of 1, as Nemotron's do.
+- Shared row kernels: the token rows (`embed_rows.cu`, groups of 32 and a copy a hyper-connection stream), the MTP
+  head's RMS rows with their group sums (`norms.cu`), the router logits and softmax top-k (`route_rows.cu`), and the
+  routed experts over a window's or a chunk's rows on the core's plan (`cuda/grouped.zig`, `experts_rows.cu`).
+
+The family supplies its config check, weight loading and packing, the layer graph on one batched pass
+(`cuda_batch.zig`), and kernels for what is its own: the hyper-connections and the n-gram gate and conv
+(`flashnext_hc.cu`), attention with its gate and the sparse indexer's pooling, scores and block selection
+(`flashnext_attn.cu`, `flashnext_batch.cu`), the Gated DeltaNet chain, the n-gram gather and the MTP head.
+
+Every kernel keeps a row's own sums in a fixed order, so a row's bits are the same at any width: drafted windows equal
+serial decoding, rounds equal lone streams, and resumed prompts equal fresh ones. The glue kernels round to bf16 where
+the Python engine's do, with IEEE division and square root and a within-one-ulp exp; the router and the 4-bit
+hyper-connection dots run on mma tiles with K in ascending order.
+
+## Checks
+
+- `tf-cuda-test flashnext-hc`, `flashnext-attn` and `route-rows`, and `norm-rows` for the shared rows: each kernel
+  against a float64 chain that rounds where it rounds, and every row count's bits equal (tolerances in the test files).
+- `zig build flashnext-prompt -Dflashnext_model=DIR -Dflashnext_kernels=native -Dflashnext_ids=FILE
+  -Dflashnext_count=32`: 32 greedy tokens through the batched pass, against the Python engine's teacher path
+  (`zig/tests/cuda/flashnext/lr7/results.json`, `results-sparse.json`); a captured set's folder in place of `native`
+  runs its glue. `TF_FLASHNEXT_DRAFTS=6` decodes each prompt serially and drafted and compares them token for token
+  (`TF_FLASHNEXT_SEED` for keyed sampled draws).
+- `zig build flashnext-compare -Dflashnext_model=DIR -Dflashnext_kernels=SET -Dflashnext_ids=FILE`: our glue against a
+  captured set on one loaded engine: every prompt row's scores (top-1 agreement, KL, each path's perplexity on the
+  prompt's own next tokens, dense rows and rows past 2,048 apart), then each path's greedy reply and where they part.
+  With `TF_FLASHNEXT_OWNED` it compares the set against the set with those entries swapped.
+- `zig build flashnext-select -Dflashnext_fixtures=DIR`: the native block selection against Triton's lists.
+- `tf-cuda-test experts-rows`: the rows kernel against the decode kernel's one-row launches, byte for byte.
+- Served: `tools/bench_concurrent.py --alone --serial` (concurrent equal to solo, drafted equal to `"draft": false`),
+  and a prompt sent twice (the second resumed from a kept state, equal to the first).
+
+## A captured set
+
+A set is built from the Python 0.6.1 engine in a container (`IMAGE`, default tag `tensorfold-qwen38:v0.6.1`) with the
+scripts in `zig/tests/cuda/flashnext`, then packed with `tools/zig/triton_aot_pack.py`:
+
+```bash
+zig/tests/cuda/flashnext/capture.sh WORK/cap /path/to/hf/snapshot          # the model's own run, about 78 GiB of GPU
+zig/tests/cuda/flashnext/capture_rows.sh WORK/rows                          # the batched pass's forms, no checkpoint
+zig/tests/cuda/flashnext/capture_mtp.sh WORK/mtp                            # the MTP head's own kernels
+zig/tests/cuda/flashnext/capture_sparse.sh WORK/sparse                      # the sparse attention's kernels
+zig/tests/cuda/flashnext/pack.sh KERNELS WORK/cap/out/cap WORK/rows/out WORK/mtp/out WORK/sparse/out
+```
+
+`capture.sh` also writes the prompts and the Python engine's references the checks read. The set has 76 kernels;
+`capture_sparse.sh` writes the selection fixtures `flashnext-select` reads.
+
+## Measurements
+
+On a GB10 (DGX Spark, driver 580.178, CUDA 13.0), an MLX 4-bit group-32 Flash Next checkpoint with its MTP head,
+greedy unless said, `tools/bench_openai.py` (256 tokens, median of 3) and `tools/bench_concurrent.py` (256 tokens,
+aggregate tok/s), on our kernels and, for comparison, on the 76-kernel captured set:
+
+| | ours | captured |
+| --- | --- | --- |
+| One stream, drafted, code / chat | 84.2 / 62.6 | 82.5 / 62.1 |
+| One stream, `--no-drafts`, code / chat | 35.1 / 35.1 | 35.2 / 35.2 |
+| `--no-drafts`, 1 / 2 / 4 streams, sampled code | 35.0 / 53.1 / 77.9 | 35.2 / 53.0 / 77.1 |
+| drafted, 1 / 2 / 4 streams, greedy code | 83.8 / 82.7 / 82.2 | 82.3 / 81.1 / 80.7 |
+| Cold first token at 3,001 / 12,001 tokens (s) | 2.67 / 10.40 | 2.67 / 10.30 |
+| 45,030-token prompt, no cache (s) | 36.5 | 35.8 |
+
+Every concurrent reply equalled its solo run, and every drafted one its `"draft": false` run. Drafted streams verify
+their windows a pass each, so drafted throughput does not grow with streams yet. While a 12,001-token prompt runs,
+another stream's decode waits at most 1.3 s between tokens (one prompt chunk). Each stream at a 16,384-token window
+takes 0.74 GiB with drafts, 0.54 GiB without.
+
+Our kernels' sums are their own, so tokens part from the captured set's (the Python engine's teacher path) where two
+candidates are close: 32 greedy tokens equal it on four of `zig/tests/cuda/flashnext/lr7`'s five prompts and part at
+token 15 on the 3,001-token one. Over 4,096 tokens of this repository's documentation, the next-token perplexity is
+6.168 on the captured set and 6.178 on ours.
+
+### The ModelOpt checkpoint on one GB10
+
+`nvidia/Qwen3.8-Flash-Next-FP4-Local` (revision `b9ce18a6`) on a GB10 (128 GB, driver 580.178, CUDA 13.3), greedy,
+`--prompt-cache-gib 0`, the tools above, three runs of the defaults at a 32,768-token window alternated with other
+settings in one session:
+
+| | defaults |
+| --- | --- |
+| One stream, drafted, code / chat | 64.1-64.6 / 40.7-41.1 |
+| drafted, 1 / 4 / 8 streams, greedy code | 64.1-64.4 / 44.9-45.2 / 66.2-66.8 |
+| drafted, 1 / 4 / 8 streams, greedy chat | 40.6-41.1 / 35.0-35.2 / 52.4-52.8 |
+| Cold first token at 2,048 / 8,192 / 32,768 / 65,536 / 131,072 tokens (s) | 1.83 / 7.11 / 29.2 / 60.1 / 131.7 |
+
+Every concurrent reply equalled its solo run and every drafted one its `"draft": false` run. A reply cancelled
+mid-stream, or sent again after a restart, equalled a fresh one, and a second conversation turn resumed from the prompt
+cache (`--no-drafts`) equalled the same turn computed without it. Prompt rows equal one-token rows bit for bit at seven
+fixed histories of 20 to 6,496 tokens. Three streams draft at a time; the rest share plain rounds.
+
+The engine holds 73.2 GiB and keeps a 12.2 GiB reserve. A 32,768-token window then admits 7 or 8 streams of 1.21 GiB
+and a 131,136-token window (the cold prompts above) 2 of 4.03 GiB; a 262,144-token stream takes 7.79 GiB, which fitted
+in one start of four on this machine. After the 131,072-token prompt the process had peaked at 6.5 GB of host and
+83.0 GB of device memory. Absolute speeds on this machine moved by about a tenth between sessions.
+
+## Not yet
+
+- The dense projections run a copy of the Python engine's group-32 lane matmul (`qmm.cu`); the core's `qmm_group` and
+  `lane_gemv` are built for groups of 64.
+- No CUDA graphs or GPU-side rounds; drafted streams verify their windows one pass each, and the MTP head drafts one
+  stream at a time.
+- Prompts run slower than the Python engine's: a cold 12,001-token prompt takes about 10 s against its 6.7 s.
+- Draws are made on the host from each row's scores.
+- Our glue's sums are not the captured kernels' bit for bit, so replies part from the Python engine's teacher path
+  after a while, as that engine's own served path does (10 to 36 tokens, from its routed-experts kernel's rounding).

docs/recipes/qwen3.8-flash-next.md

@@ -2,16 +2,17 @@
 
 The `qwen4_exp` family has Gated DeltaNet, sparse attention, MoE, hyper-connections and hashed n-gram
 embeddings. The native engine's supported Metal checkpoint uses MLX affine 6-bit weights in groups of 32
-and includes an MTP head. The native engine refuses the 4-bit/group-32 checkpoint at load.
+and includes an MTP head. On a Mac the native engine refuses the 4-bit/group-32 checkpoint at load; on CUDA it serves
+that one ([Flash Next on CUDA](flash-next-cuda.md)).
 
 ```bash
 tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP
 tensorfold serve TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP --name bench --temperature 0
 ```
 
 For native serving, see the [Metal runbook](../../RUNBOOK.md#flash-next-and-paired-metal-serving).
-Native Flash Next has no CUDA backend yet; its port is tracked in
-[#472](https://github.com/ashhart/TensorFold/issues/472).
+Native Flash Next on CUDA serves the MLX 4-bit group-32 checkpoint on the built-in kernels: see
+[Flash Next on CUDA](flash-next-cuda.md) ([#472](https://github.com/ashhart/TensorFold/issues/472)).
 
 The sections below describe the Python 0.6.6 engine, available at the
 [`v0.6.6` tag](https://github.com/ashhart/TensorFold/tree/v0.6.6). Their flags, variables and paths are that engine's,