Living Weights on Qwen3.8 Flash Next (CUDA, GB10): the pocket in every pass, the trainer, and --slide

CHANGELOG.md

@@ -2,6 +2,17 @@
 
 Each release's page on GitHub has its notes and measurements. See [the 1.0.0 release notes](RELEASE-NOTES-1.0.0.md) for the native binary's supported models and migration details.
 
+## Unreleased
+
+- Living Weights learns into Qwen3.8 Flash Next on CUDA (`--slide`, GB10 only; Macs refuse it for this model). A folder it taught serves with its pocket, the learned SwiGLU neurons beside each layer's shared expert behind its gate, in every pass (prompt chunks, drafted windows, decode rounds): such a layer's MoE output is summed in fp32, the pocket added and written back rounded once, and every other layer writes back as before, so an untaught folder's replies and speed do not change. The trainer runs the learner's passes on the engine's own batched pass, sequences side by side, and the loss's gradient back through the 4-bit head, the hyper-connections, the MoE with its shared expert and pocket, gated attention and the Gated DeltaNet to the lowest nudged layer, on new CUDA backward kernels (`gdn_train.cu`, `train_q4.cu` and `train_attn.cu` in the core, `flashnext_train.cu` in the family), each checked against float64 by `tf-cuda-test` (`fn-back`, `fn-pocket`). `zig build flashnext-learn` checks the trainer's forward against the serving forward byte for byte and its gradient against finite differences. `qwen4_exp` joins the core's family list, and its first save widens every shared expert and records `tensorfold_learned_neurons`, which the CUDA config check now accepts. On a GB10 the trainer keeps 10.6 GiB beside the 75.8 GiB model; a fact took 8.7 to 10.5 minutes.
+
+- Qwen3.8 Flash Next serves on CUDA from its MLX 4-bit group-32 checkpoint with the MTP head, through the lane core: drafted windows, shared rounds of streams without drafts, kept prompt states (a drafting stream's too, with the head's cache rows and a lookahead of 1, as Nemotron's) and prompts a chunk at a time between other streams' rounds. Each prompt chunk, verify window and decode round is one batched pass whose kernels give every row the bits it gets alone, so drafted replies equal `"draft": false`, concurrent replies equal their solo runs and resumed prompts equal fresh ones; past 2,048 keys attention reads the 512 blocks the model's indexer picks, up to the 262,144-token window. Every kernel it launches is built from this repository (`flashnext_hc.cu`, `flashnext_attn.cu`, `flashnext_batch.cu`, and the core's token, RMS, routing and expert rows), so it needs no capture; `TENSORFOLD_CUDA_KERNELS` can name a captured set of the Python 0.6.1 engine's Triton kernels (scripts in `zig/tests/cuda/flashnext`), which then stands in for the glue so replies can be compared with that engine's teacher path. On a GB10 one stream decodes 84.2 tok/s drafted on code and 62.6 on chat, 35.1 without drafts, and 4 sampled streams without drafts 77.9 in all, within 2% of the captured set either way; a cold 12,001-token prompt's first token takes 10.4 s and a 45,030-token prompt 36.5 s, and another stream waits at most 1.3 s between tokens while it runs. 32 greedy tokens equal the Python engine's teacher path on four of five prompts of 19 to 12,001 tokens and part at token 15 on the 3,001-token one (the captured set gives all five); next-token perplexity over 4,096 tokens of this repository's documentation is 6.178 against the captured set's 6.168. A checkpoint of another width, shape or semantic setting (RMS epsilon, RoPE, gate and activation types, layer roles) is refused at load, by its config field. @AdrianBinDC started the port in #476, matching every layer of one decode token to host checks; see [Flash Next on CUDA](docs/recipes/flash-next-cuda.md) (#472). Thanks to @AdrianBinDC.
+- A Flash Next sequence's buffers are zeroed on the engine's compute stream. A fill with no stream returns before the memset lands and is not ordered against the non-blocking stream the passes run on, so a new sequence's first prompt chunk could read its DeltaNet state before it was zeroed; the drafted check, which zeroes the states between a prompt's serial and drafted runs, once left the serial tokens that way.
+- CUDA has a lane kernel for affine 4-bit experts over a window's or a prompt chunk's rows (`experts_rows.cu`, `cuda.experts.RowsExperts`): experts.cu's decode sums for each row, two row tiles of 16 a warp, so a warp reads an expert's weights once for 32 pairs. `tf-cuda-test experts-rows` checks it against the decode kernel's one-row launches byte for byte at 1 to 300 rows.
+- CUDA has routing rows for every family (`route_rows.cu`, `cuda.route_rows`): router logits on mma tiles, one chain over K from zero so a row's sums never depend on the row count, and softmax top-k with an optional sigmoid-gated shared expert; `tf-cuda-test route-rows` checks both against float64 (`core/route_ref.zig`). The token rows take groups of 32 and a copy count, and the RMS rows gain a form with the stored outputs' group sums (`Norms.rmsSums`); `norm-rows` covers both.
+- A Flash Next pass uploads its token ids, positions and n-gram rows from pinned staging the scratch owns until the copies land, instead of allocator memory freed when the call returned (an asynchronous copy from pageable memory is only documented as "might be synchronous"), and a failed zero fill frees the buffer it was filling.
+- `tools/zig/triton_aot_pack.py` leaves out a constexpr Triton folded from `None` (an unused pointer argument), where it stopped, and takes `--jit` once per capture, merged, so one call packs several captures; `test_aot_manifest.py` covers both.
+
 ## 1.0.11
 
 - `--cortheon`'s instruction filter keeps reading past a long word. A word longer than the filter's scan buffer ended the scan early, so an order written after one was never checked; each word now keeps at most 16 bytes, so a long word can no longer end the scan (#829). Thanks to @joeynyc.

RUNBOOK.md

@@ -123,6 +123,8 @@ The archives require glibc 2.28 or newer and an NVIDIA driver providing `libcuda
 CUDA serves Nemotron 3.5 Lightning, qualified on GB10 (`linux-aarch64`, greedy and sampled) and on Ampere cards with
 compute capability 8.6 (`linux-x86_64`, greedy: the RTX 30 series, RTX A6000, A10 and A40), and checked on an RTX PRO 6000 Blackwell (compute
 capability 12.0, `linux-x86_64`).
+Flash Next also serves on CUDA from its MLX 4-bit checkpoint, on the built-in kernels, run on a GB10: see
+[Flash Next on CUDA](docs/recipes/flash-next-cuda.md).
 
 On a GB10 arm64 host (an x86_64 host uses `linux-x86_64` in the same commands):
 

build.zig

@@ -1,4 +1,5 @@
 const std = @import("std");
+const glm_expiry_build = @import("zig/build/glm_expiry.zig");
 const builtin = @import("builtin");
 const cuda_build = @import("zig/build/cuda.zig");
 const dist_build = @import("zig/build/dist.zig");
@@ -178,6 +179,7 @@ pub fn build(b: *std.Build) void {
     const disk_run = b.addRunArtifact(disk_cpu);
     test_step.dependOn(&disk_run.step);
     b.step("test-learn-recovery", "CPU learned-state recovery and admission tests").dependOn(&disk_run.step);
+    glm_expiry_build.steps(b, test_step, lanes_cpu);
     const warning = b.addExecutable(.{ .name = "learned-warning-test", .root_module = b.createModule(.{
         .root_source_file = b.path("zig/src/learned_warning_test.zig"),
         .target = b.graph.host,
@@ -192,7 +194,7 @@ pub fn build(b: *std.Build) void {
 }
 
 /// `zig build native -Dcpu=apple_m1`: tensorfold-native with the Metal engines for the Python package's bundle (a native M5 build traps on M1-M4).
-fn nativeServer(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.builtin.OptimizeMode, metal: *std.Build.Module, engine: *std.Build.Module, api: *std.Build.Module, engines: *std.Build.Module, build_options: *std.Build.Step.Options, test_step: ?*std.Build.Step) *std.Build.Step.Compile {
+fn nativeServer(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.builtin.OptimizeMode, metal: *std.Build.Module, engine: *std.Build.Module, living: *std.Build.Module, api: *std.Build.Module, engines: *std.Build.Module, build_options: *std.Build.Step.Options, test_step: ?*std.Build.Step) *std.Build.Step.Compile {
     // the HTTP side keeps its safety checks; the engine below it runs at `optimize`
     const tokenizer = b.createModule(.{ .root_source_file = b.path("zig/src/core/tokenizer/tokenizer.zig"), .target = target, .optimize = .ReleaseSafe, .link_libc = true });
     const template = b.createModule(.{ .root_source_file = b.path("zig/src/core/template/template.zig"), .target = target, .optimize = .ReleaseSafe, .link_libc = true });
@@ -201,7 +203,7 @@ fn nativeServer(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.b
         .target = target,
         .optimize = .ReleaseSafe,
         .link_libc = true,
-        .imports = &.{ .{ .name = "engine_api", .module = api }, .{ .name = "tokenizer", .module = tokenizer }, .{ .name = "template", .module = template }, .{ .name = "native_engines", .module = engines }, .{ .name = "checkpoint_cli", .module = b.createModule(.{ .root_source_file = b.path("zig/src/cli/cli.zig"), .target = target, .optimize = .ReleaseSafe, .link_libc = true, .imports = &.{ .{ .name = "native_engines", .module = engines }, exlr8_build.cliImport(b, target, .ReleaseSafe, engine) } }) } },
+        .imports = &.{ .{ .name = "engine_api", .module = api }, .{ .name = "tokenizer", .module = tokenizer }, .{ .name = "template", .module = template }, .{ .name = "native_engines", .module = engines }, .{ .name = "living", .module = living }, .{ .name = "checkpoint_cli", .module = b.createModule(.{ .root_source_file = b.path("zig/src/cli/cli.zig"), .target = target, .optimize = .ReleaseSafe, .link_libc = true, .imports = &.{ .{ .name = "native_engines", .module = engines }, exlr8_build.cliImport(b, target, .ReleaseSafe, engine) } }) } },
     }) });
     exe.root_module.addOptions("build_options", build_options);
     if (test_step == null) {
@@ -219,7 +221,7 @@ fn nativeServer(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.b
         .target = target,
         .optimize = .Debug, // the server's unit tests run with safety checks, as zig_test.sh builds them
         .link_libc = true,
-        .imports = &.{ .{ .name = "engine_api", .module = api }, .{ .name = "tokenizer", .module = tokenizer }, .{ .name = "template", .module = template } },
+        .imports = &.{ .{ .name = "engine_api", .module = api }, .{ .name = "tokenizer", .module = tokenizer }, .{ .name = "template", .module = template }, .{ .name = "living", .module = living } },
     });
     test_step.?.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests })).step);
     @import("zig/build/cortheon_fuzz.zig").add(b, target, api, tokenizer, template);
@@ -466,7 +468,7 @@ fn metalTargets(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.b
     glm_checks.targets(b, target, optimize, metal, engine, sources);
     flashnext_checks.targets(b, target, optimize, metal, engine);
 
-    _ = nativeServer(b, target, optimize, metal, engine, api, engines, build_options, test_step);
+    _ = nativeServer(b, target, optimize, metal, engine, mods.living, api, engines, build_options, test_step);
 
     const cluster_metal = b.createModule(.{
         .root_source_file = b.path("zig/src/cluster/metal_sink.zig"),
@@ -539,7 +541,7 @@ fn metalTargets(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.b
 }
 
 /// The server and distribution use the same embedded-source Metal engine.
-fn metalEngineModules(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.builtin.OptimizeMode, draft_ids: *std.Build.Module) struct { metal: *std.Build.Module, lanes: *std.Build.Module, fabric: *std.Build.Module, sources: *std.Build.Module, engine: *std.Build.Module, qwen27: qwen27_build.Shared } {
+fn metalEngineModules(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.builtin.OptimizeMode, draft_ids: *std.Build.Module) struct { metal: *std.Build.Module, lanes: *std.Build.Module, fabric: *std.Build.Module, sources: *std.Build.Module, engine: *std.Build.Module, living: *std.Build.Module, qwen27: qwen27_build.Shared } {
     const metal = b.createModule(.{
         .root_source_file = b.path("zig/src/metal/metal.zig"),
         .target = target,
@@ -553,14 +555,16 @@ fn metalEngineModules(b: *std.Build, target: std.Build.ResolvedTarget, optimize:
     const lanes = b.createModule(.{ .root_source_file = b.path("zig/src/core/lanes/lanes.zig"), .target = target, .optimize = optimize, .link_libc = true });
     const fabric = b.createModule(.{ .root_source_file = b.path("zig/src/fabric/fabric.zig"), .target = target, .optimize = optimize, .link_libc = true });
     const sources = b.createModule(.{ .root_source_file = b.path("zig/kernels/metal/sources.zig") });
+    // Living Weights' family-neutral learner, which Nemotron's lessons and the server's view of them share
+    const living = b.createModule(.{ .root_source_file = b.path("zig/src/living/living.zig"), .target = target, .optimize = optimize, .link_libc = true });
     const engine = b.createModule(.{
         .root_source_file = b.path("zig/src/tensorfold.zig"),
         .target = target,
         .optimize = optimize,
         .link_libc = true,
-        .imports = &.{ .{ .name = "metal", .module = metal }, .{ .name = "kernel_sources", .module = sources }, .{ .name = "nemotron_draft_ids", .module = draft_ids }, .{ .name = "lanes", .module = lanes }, .{ .name = "fabric", .module = fabric } },
+        .imports = &.{ .{ .name = "metal", .module = metal }, .{ .name = "kernel_sources", .module = sources }, .{ .name = "nemotron_draft_ids", .module = draft_ids }, .{ .name = "lanes", .module = lanes }, .{ .name = "fabric", .module = fabric }, .{ .name = "living", .module = living } },
     });
-    return .{ .metal = metal, .lanes = lanes, .fabric = fabric, .sources = sources, .engine = engine, .qwen27 = qwen27_build.engineModules(b, target, metal, sources, engine) };
+    return .{ .metal = metal, .lanes = lanes, .fabric = fabric, .sources = sources, .engine = engine, .living = living, .qwen27 = qwen27_build.engineModules(b, target, metal, sources, engine) };
 }
 
 pub fn distMetalServer(b: *std.Build, target: std.Build.ResolvedTarget, draft_ids: *std.Build.Module, build_options: *std.Build.Step.Options, sdk: []const u8) *std.Build.Step.Compile {
@@ -574,7 +578,7 @@ pub fn distMetalServer(b: *std.Build, target: std.Build.ResolvedTarget, draft_id
     m.metal.addSystemIncludePath(b.graph.cwdRelativePath(b.pathJoin(&.{ sdk, "usr/include" })));
     const api = b.createModule(.{ .root_source_file = b.path("zig/src/core/engine_api.zig"), .target = target, .optimize = .fast, .link_libc = true, .imports = &.{.{ .name = "lanes", .module = m.lanes }} });
     const engines = b.createModule(.{ .root_source_file = b.path("zig/src/native/metal.zig"), .target = target, .optimize = .fast, .link_libc = true, .imports = &.{ .{ .name = "metal", .module = m.metal }, .{ .name = "engine_api", .module = api }, .{ .name = "tensorfold", .module = m.engine } } });
-    return nativeServer(b, target, .fast, m.metal, m.engine, api, engines, build_options, null);
+    return nativeServer(b, target, .fast, m.metal, m.engine, m.living, api, engines, build_options, null);
 }
 
 test {

docs/living-weights.md

@@ -11,14 +11,15 @@ as Sliding Weights (experimental), and its flag and API use the name `slide`.
 ## What you need
 
 - An Apple-silicon Mac (tested on an M5 Ultra and an M3 Ultra), or an NVIDIA GB10 (DGX Spark) with the CUDA build,
-  with room for the model and a copy of it. On a GB10, `--slide` also keeps about 26 GiB of GPU memory for its
-  trainer; on a Mac the trainer works on the model's own weights.
-- The Nemotron 3.5 Lightning MLX 4-bit checkpoint, the only model it learns into today.
+  with room for the model and a copy of it. On a GB10, `--slide` also keeps GPU memory for its trainer (about 26 GiB
+  for Nemotron, 10.6 GiB for Qwen3.8 Flash Next); on a Mac the trainer works on the model's own weights.
+- The Nemotron 3.5 Lightning MLX 4-bit checkpoint, or, on a GB10 only, the Qwen3.8 Flash Next MLX 4-bit group-32
+  checkpoint with its MTP head. Macs refuse `--slide` for Flash Next.
 - A TensorFold build with the pocket learner described here (the release after 1.0.6). A folder taught by 1.0.3 to
   1.0.6 still serves and answers from what it learned, but `--slide` refuses it: those releases saved lessons into
   the shared experts the pocket widens. Teach a fresh copy instead.
 
-A folder that learned on either backend serves on the other.
+A Nemotron folder that learned on either backend serves on the other.
 
 ## Quick start
 
@@ -171,6 +172,7 @@ data.
 | `--slide rewrites the model's own files, and … is a link to data another file shares` at startup | The folder holds links into the Hugging Face cache (or hard links); make the copy with `cp -cRL` (Linux: `cp -RL`) as in step 2 |
 | `--slide learns on a GB10 (sm_121) so far` at startup | CUDA learning is qualified on a GB10 only; serve this GPU without `--slide`, or teach on a Mac or a GB10 |
 | `--slide runs a prompt one chunk at a time` at startup | Serve it without `--segments` or `TF_CUDA_SEGMENTS` |
+| `--slide trains on Flash Next's own glue` at startup | Serve Flash Next without `TENSORFOLD_CUDA_KERNELS` |
 | `--slide adds learned neurons beside every shared expert, and … is unquantized` at startup | The folder was taught by TensorFold 1.0.3 to 1.0.6; serve it without `--slide`, or teach a fresh copy |
 | The answer has not changed yet | Wait for the `learned` event before asking |
 
@@ -199,6 +201,15 @@ Living Weights is experimental. Measured on Nemotron 3.5 Lightning (MLX 4-bit) o
   Living Weights. A folder holding many learned facts decodes slower on a Mac (about 16% with 21 facts in the pocket),
   and drafted replies still equal plain ones.
 
+On Qwen3.8 Flash Next (MLX 4-bit group 32 with its MTP head) on a GB10, five facts about the user were taught one
+after another to a fresh copy. Three were kept, in 8.7 to 10.5 minutes each (two and a half to four of them the model writing its
+lesson and three to four its trainer's first round, half a minute each round of checks), and each came back right after
+its lesson; one was refused at once, the model having written too few clean answers to learn from, and one was not
+kept after five rounds in 40 minutes. A later lesson can stop an earlier fact coming back (#870): after the five, two
+of the three kept facts still answered. Unrelated questions kept their answers, plain and `--slide` restarts gave the
+same replies, drafted replies equal `"draft": false` and four streams at once equal each alone. With three facts in
+the pocket, drafted decode was 5% slower (80 against 84 tokens a second on code) and `--no-drafts` 1% slower.
+
 While it learns, near misses and free writing about the fact's subject are trained to stay as they were. Before
 keeping a fact, it checks wordings it never trained on, with thinking off and on, near misses and unrelated prompts.
 If the fact does not come back, or another answer changes, the whole lesson is taken out before anything is written.

docs/recipes/README.md

@@ -8,7 +8,7 @@ the native engine, or the Python engine frozen at 0.6.6.
 | Nemotron 3.5 Lightning | [MLX](nemotron-3.5.md) |
 | Qwen3.8-27B | [MLX, quantization and CUDA](qwen3.8-27b.md) |
 | Qwen3.5-2B | [Native Metal preview, tied affine 4-bit](qwen3.5-2b-native.md) |
-| Qwen3.8 Flash Next | [MLX prefill and CUDA](qwen3.8-flash-next.md), [CUDA images](flash-next-vision.md) |
+| Qwen3.8 Flash Next | [MLX prefill and CUDA](qwen3.8-flash-next.md), [native CUDA](flash-next-cuda.md), [CUDA images](flash-next-vision.md) |
 | Ternary Bonsai 2 27B | [MLX](ternary-bonsai-2.md) |
 | GLM-5.3-Flash | [MLX on a 256 GB Mac, two-rank CUDA](glm-5.3-flash.md) |
 | Gemma 4 26B-A4B | [MLX, fused one-row decode](gemma-4.md) |