GLM CUDA のコピー窓を 16 行（15 トークン）まで広げる

RUNBOOK.md

@@ -212,6 +212,33 @@ path needs `LD_LIBRARY_PATH` set to that `lib` directory on both ranks.
 First launch. The first start can stay silent for several minutes while it reads the checkpoint digest. Wait for
 `checkpoint digest:` before treating it as stuck. A later start reports the cached files immediately.
 
+### GLM CUDA copy windows
+
+Rank 0 reads these. Give both ranks the same values. The decode buffers, sampler and KDA save always hold 16 rows.
+DFlash stays on its 8-row block (7 drafts). The MTP head still absorbs in 8-row chunks. Native verify launches
+kernels directly: there is no captured CUDA graph to widen, and `WIDE_GRAPHS` is not read. The KDA wide step already
+walks rows in tiles of 16.
+
+`TF_GLM_COPY_MAX` is the most tokens a copy may draft. The default is 15 (a 16-row window). `7` is the previous cap
+(an 8-row window). Values outside 1..15 clamp into that range; text that is not an integer keeps 15. The first copy
+is still 3 tokens and doubles plus one after a fully accepted copy, including in a shared or multi-stream round.
+That growth fits: each stream's KDA save holds 16 rows, a shared segment may be 16 rows, and one shared forward
+still holds at most 32 rows, so the allocator trims a round that would overflow.
+
+`TF_GLM_COPY_REPLY_MATCH` is the match a copy needs when its continuation starts in the reply. The default is 16.
+A copy taken from the prompt still needs 8. The recipe's 16-token reply bar moved code from 55.4 to 56.2 tok/s and
+edit from 120 to 125, with no prose regression called out for that bar. `TF_GLM_COPY_MAX=15` is what moved edit
+from 84 to 119 tok/s with prose and code unchanged. Those figures are the recipe's, not a measurement on this tree.
+
+`TF_GLM_COPY_EXTEND=1` is off by default. It turns on context copies and root copies, and a match of at least 16
+tokens may take the whole `TF_GLM_COPY_MAX` in that round instead of waiting for the ramp. Root copies then skip
+the head for an admitted copy. Use it for an A/B against the default. Unset or any value other than `1` leaves it off.
+
+A 16-row save is twice the KDA window of an 8-row save, and the shared scratch sizes its per-stream chain for 16
+rows. Watch free memory on both GPUs, especially at 4 streams. Drafted token ids must match `"draft": false`
+(`token_sha256`). Prose and prefill should not regress. The on-hardware plan is in the pull request that added
+these knobs.
+
 ### GB10 CPU pinning for long CUDA runs
 
 In [#1033](https://github.com/ashhart/TensorFold/issues/1033), @Bizuayeu reports two CPU threads spinning per CUDA rank,

docs/recipes/glm-5.3-flash-cuda.md

@@ -51,6 +51,22 @@ left after a 12.2 GiB reserve` and a 5.8 GiB prompt cache
 (`TF_GLM_PREFILL_REDUCE`) and the two-rank RoCE gathers (`TF_ROCE`) are in
 [placement](../multi-rank-receipts.md#the-settings-the-ranks-agreed-on).
 
+## Copy windows
+
+Rank 0 plans copies. The buffers below are allocated on every rank.
+
+| Variable | Default | Effect |
+| --- | --- | --- |
+| `TF_GLM_COPY_MAX` | 15 | Drafted tokens a copy may hold. `7` is the previous 8-row window. Clamped to 1..15. |
+| `TF_GLM_COPY_REPLY_MATCH` | 16 | Matching tokens required when the copied continuation starts in the reply. Prompt copies stay at 8. |
+| `TF_GLM_COPY_EXTEND` | off | `1` enables context copies, root copies, and a 16-token match that may use the whole copy cap at once. |
+
+DFlash remains a 7-draft block. The MTP head still chunks its absorb at 8 rows. A copy may grow past its first 3
+tokens after a fully accepted copy, in a lone round and in a shared round. The shared forward still admits at most
+32 rows. Native GLM verify does not capture CUDA graphs, so there is no 16-row graph set; the KDA wide kernel
+already accepts a 16-row window. Set the same values on every rank. Compare `token_sha256` with `"draft": false`,
+and watch memory: the KDA save and the shared per-stream chain are sized for 16 rows.
+
 ## Long runs on GB10
 
 Long prompts and long runs of replies heat a GB10. The runbook's [heat bands](../../RUNBOOK.md#linux-and-cuda) (with a

zig/src/core/lanes/config.zig

@@ -45,6 +45,9 @@ pub const Model = struct {
     lone_cost_tile: u32 = 0, // live lone-window pricing probes at these target row boundaries (0: off)
     context_copies: bool = false, // a lone head window shares its agreeing copy path and appends the rest
     copy_match: i64 = 8, // matching history tokens a copy needs
+    reply_match: i64 = 0, // extra match when the continuation starts in the reply (0: copy_match covers it)
+    strong_match: i64 = 0, // 0 off; a match at least this long may copy strong_copy tokens in one round
+    strong_copy: u32 = 0, // that longer cap, still within max_copy (0: the ramp only)
     lone_copies: bool = true, // a lone stream proposes copies (false: only streams in shared rounds do)
 };
 
@@ -90,6 +93,9 @@ pub const Config = struct {
     lone_cost_tile: u32 = 0, // Model.lone_cost_tile
     context_copies: bool = false, // Model.context_copies, within the existing verification width
     enter_match: i64 = 8, // matching tokens a copy needs (rejects coincidental indentation)
+    reply_match: i64 = 0, // Model.reply_match, stricter than enter_match for a copy out of the reply
+    strong_match: i64 = 0, // Model.strong_match
+    strong_copy: u32 = 0, // Model.strong_copy
     copy_rate: f64 = 0.94, // a copied token's chance of landing in allocation
     lone_copies: bool = true, // Model.lone_copies
     k: Constants = .{},
@@ -130,6 +136,9 @@ pub const Config = struct {
             .root_copies = m.root_copies and m.copy_drafts,
             .context_copies = m.context_copies and m.copy_drafts,
             .enter_match = @max(1, m.copy_match),
+            .reply_match = if (m.reply_match > m.copy_match) m.reply_match else 0,
+            .strong_match = if (m.strong_copy > 0) m.strong_match else 0,
+            .strong_copy = if (m.copy_drafts) @min(m.strong_copy, max_copy) else 0,
             .lone_copies = m.lone_copies,
             .lone_cost_tile = m.lone_cost_tile,
             .family_width = width,

zig/src/core/lanes/copy_exact_test.zig

@@ -0,0 +1,168 @@
+//! GLM-shaped copy windows: drafted tokens match a `"draft": false` run, greedy and sampled.
+const std = @import("std");
+const Config = @import("config.zig").Config;
+const Model = @import("config.zig").Model;
+const Engine = @import("engine.zig").Engine;
+const Stream = @import("stream.zig").Stream;
+const Sampling = @import("sampling.zig").Sampling;
+const SuffixLookup = @import("proposer.zig").SuffixLookup;
+const fake = @import("fake.zig");
+
+const gpa = std.testing.allocator;
+const eos = [_]u32{96};
+const sampled: Sampling = .{ .seed = 11, .temperature = 0.7, .top_k = 0, .top_p = 0.9 };
+
+fn glmModel() Model {
+    return .{
+        .exact_width = 16,
+        .first_copy_rows = 4,
+        .mtp = true,
+        .speculate = true,
+        .speculate_early = false,
+        .drafts = 7,
+        .hidden_rows = true,
+        .batch_rows = 32,
+        .max_streams = 16,
+        .copy_match = 8,
+        .reply_match = 16,
+        .streams_exact = true,
+    };
+}
+
+const Out = struct { tokens: []u32, drafted: u64 };
+
+fn run(prompts: []const []const u32, sampling: ?Sampling, drafts: bool) ![]Out {
+    var cfg = try Config.init(gpa, glmModel(), 16, 15);
+    defer cfg.deinit(gpa);
+    var target: fake.Fake = .{ .gpa = gpa };
+    defer target.deinit();
+    var clock: fake.FixedClock = .{};
+    var engine = Engine.init(gpa, &cfg, target.backend(), clock.clock());
+    defer engine.deinit();
+    const streams = try gpa.alloc(Stream, prompts.len);
+    defer gpa.free(streams);
+    const looks = try gpa.alloc(SuffixLookup, prompts.len);
+    defer gpa.free(looks);
+    for (prompts, streams, looks) |p, *s, *look| {
+        look.* = try SuffixLookup.init(gpa, .{ .ngram = 3, .min_match = 6 });
+        s.* = try Stream.init(gpa, .{
+            .id = "s",
+            .prompt = p,
+            .max_new = 24,
+            .eos = &eos,
+            .sampling = sampling,
+            .drafts = drafts,
+            .proposer = look.proposer(),
+        });
+    }
+    defer for (streams, looks) |*s, *look| {
+        s.deinit(gpa);
+        look.deinit();
+    };
+    for (streams) |*s| try engine.addStream(s);
+    while (engine.activeCount() > 0) try engine.step();
+    const out = try gpa.alloc(Out, prompts.len);
+    for (out, streams) |*o, *s| o.* = .{ .tokens = try gpa.dupe(u32, s.emitted()), .drafted = s.drafted };
+    return out;
+}
+
+fn freeOut(rows: []Out) void {
+    for (rows) |o| gpa.free(o.tokens);
+    gpa.free(rows);
+}
+
+/// Two copies of a 16-token block, then the block's first 8, so a prompt copy has a continuation.
+const repeat = blk: {
+    var p: [40]u32 = undefined;
+    for (&p, 0..) |*t, i| t.* = @as(u32, @intCast(i % 16));
+    break :blk p;
+};
+const other = [_]u32{ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 3, 5, 7, 9, 11 };
+
+test "drafted GLM copy windows match draft false, greedy and sampled" {
+    var cfg = try Config.init(gpa, glmModel(), 16, 15);
+    defer cfg.deinit(gpa);
+    try std.testing.expectEqual(@as(u32, 15), cfg.max_copy);
+    try std.testing.expectEqual(@as(u32, 3), cfg.first_copy);
+    try std.testing.expectEqual(@as(u32, 7), cfg.most_drafts);
+    for ([_]?Sampling{ null, sampled }) |sampling| {
+        const drafted = try run(&.{&repeat}, sampling, true);
+        defer freeOut(drafted);
+        const plain = try run(&.{&repeat}, sampling, false);
+        defer freeOut(plain);
+        try std.testing.expectEqualSlices(u32, plain[0].tokens, drafted[0].tokens);
+        try std.testing.expectEqual(@as(u64, 0), plain[0].drafted);
+        if (sampling == null) try std.testing.expect(drafted[0].drafted > 0);
+    }
+}
+
+fn propose(extend: bool, alone: bool, width: ?u32, prompt: []const u32, extra: []const u32) ![]const u32 {
+    var cfg = try Config.init(gpa, .{
+        .exact_width = 16,
+        .first_copy_rows = 4,
+        .mtp = true,
+        .speculate = true,
+        .drafts = 7,
+        .copy_match = 8,
+        .reply_match = 16,
+        .strong_match = if (extend) 16 else 0,
+        .strong_copy = if (extend) 15 else 0,
+    }, 16, 15);
+    var target: fake.Fake = .{ .gpa = gpa };
+    var clock: fake.FixedClock = .{};
+    var engine = Engine.init(gpa, &cfg, target.backend(), clock.clock());
+    engine.alone = alone;
+    var look = try SuffixLookup.init(gpa, .{ .ngram = 3, .min_match = 6 });
+    var s = try Stream.init(gpa, .{ .id = "p", .prompt = prompt, .max_new = 64, .proposer = look.proposer() });
+    if (extra.len > 0) try s.context.appendSlice(gpa, extra);
+    s.copy_width = width;
+    const got = try @import("windows.zig").copyProposal(&engine, &s);
+    const kept = try gpa.dupe(u32, got.tokens);
+    s.deinit(gpa);
+    look.deinit();
+    engine.deinit();
+    target.deinit();
+    cfg.deinit(gpa);
+    return kept;
+}
+
+test "reply copies need 16 tokens, and a shared ramp is the grown width" {
+    var block: [24]u32 = undefined;
+    for (&block, 0..) |*t, i| t.* = @intCast(i + 1);
+    const prompt = [_]u32{ 90, 91, 92, 93 };
+    var reply: [34]u32 = undefined;
+    @memcpy(reply[0..24], &block);
+    @memcpy(reply[24..], block[0..10]);
+    const short = try propose(false, true, null, &prompt, &reply);
+    defer gpa.free(short);
+    try std.testing.expectEqual(@as(usize, 0), short.len);
+    @memcpy(reply[24..], block[0..10]);
+    var long_reply: [40]u32 = undefined;
+    @memcpy(long_reply[0..24], &block);
+    @memcpy(long_reply[24..], block[0..16]);
+    const allowed = try propose(false, false, 7, &prompt, &long_reply);
+    defer gpa.free(allowed);
+    try std.testing.expect(allowed.len >= 2 and allowed.len <= 7);
+    var prompt_hist: [40]u32 = undefined;
+    @memcpy(prompt_hist[0..24], &block);
+    @memcpy(prompt_hist[24..], block[0..16]);
+    const capped = try propose(false, true, null, &prompt_hist, &.{});
+    defer gpa.free(capped);
+    try std.testing.expect(capped.len >= 2 and capped.len <= 3);
+    const extended = try propose(true, true, null, &prompt_hist, &.{});
+    defer gpa.free(extended);
+    try std.testing.expect(extended.len > 3 and extended.len <= 15);
+}
+
+test "two streams' drafted tokens match each stream's draft false run" {
+    for ([_]?Sampling{ null, sampled }) |sampling| {
+        const drafted = try run(&.{ &repeat, &other }, sampling, true);
+        defer freeOut(drafted);
+        const a = try run(&.{&repeat}, sampling, false);
+        defer freeOut(a);
+        const b = try run(&.{&other}, sampling, false);
+        defer freeOut(b);
+        try std.testing.expectEqualSlices(u32, a[0].tokens, drafted[0].tokens);
+        try std.testing.expectEqualSlices(u32, b[0].tokens, drafted[1].tokens);
+    }
+}

zig/src/core/lanes/copy_limit.zig

@@ -0,0 +1,43 @@
+//! How many tokens a copy may draft, including a reply-only match and a strong-match extension.
+const Engine = @import("engine.zig").Engine;
+const Stream = @import("stream.zig").Stream;
+
+/// The ramp this round would use before a strong-match extension (shared rounds included).
+pub fn ramp(e: *const Engine, s: *const Stream) u32 {
+    const width = s.copy_width orelse e.cfg.first_copy;
+    return width;
+}
+
+/// Tokens to ask the proposer for. A configured strong cap asks for that, and `take` shortens a weak match.
+pub fn ask(e: *const Engine, s: *const Stream) u32 {
+    const base = ramp(e, s);
+    return if (e.cfg.strong_copy > base) e.cfg.strong_copy else base;
+}
+
+/// The ramp after this copy: a full accept doubles plus one, up to max_copy; a miss returns to the first width.
+pub fn nextWidth(current: u32, first: u32, max_copy: u32, full: bool) u32 {
+    return if (full) @min(max_copy, 2 * current + 1) else first;
+}
+
+/// The copy to keep. A continuation that starts in the reply needs `reply_match`. A short match stays on the ramp.
+pub fn take(e: *const Engine, s: *const Stream, match: i64, end: ?usize, copied: []const u32) []const u32 {
+    const least: usize = if (e.alone and e.cfg.context_copies) 1 else 2;
+    if (copied.len < least or match < e.cfg.enter_match) return &.{};
+    if (e.cfg.reply_match > e.cfg.enter_match) {
+        if (end) |at| if (at >= s.prompt_len and match < e.cfg.reply_match) return &.{};
+    }
+    if (e.cfg.strong_match > 0 and match >= e.cfg.strong_match) return copied;
+    if (e.cfg.strong_copy == 0) return copied;
+    const n = @min(copied.len, @as(usize, ramp(e, s)));
+    return if (n >= least) copied[0..n] else &.{};
+}
+
+const std = @import("std");
+
+test "a full accept grows the ramp, and a miss returns to the first width" {
+    try std.testing.expectEqual(@as(u32, 7), nextWidth(3, 3, 15, true));
+    try std.testing.expectEqual(@as(u32, 15), nextWidth(7, 3, 15, true));
+    try std.testing.expectEqual(@as(u32, 15), nextWidth(15, 3, 15, true));
+    try std.testing.expectEqual(@as(u32, 7), nextWidth(7, 3, 7, true));
+    try std.testing.expectEqual(@as(u32, 3), nextWidth(7, 3, 15, false));
+}

zig/src/core/lanes/engine.zig

@@ -8,6 +8,7 @@ const be = @import("backend.zig");
 const sm = @import("stream.zig");
 const ev = @import("events.zig");
 const win = @import("windows.zig");
+const copy_limit = @import("copy_limit.zig");
 const shape = @import("shape.zig");
 const plan_lanes = @import("plan_lanes.zig");
 const fill = @import("fill.zig");
@@ -46,7 +47,7 @@ pub const Engine = struct {
     drafted: u64 = 0,
     accepted: u64 = 0,
     shared_rounds: u64 = 0,
-    alone: bool = true, // this step has one live stream: its copies may take their ramp
+    alone: bool = true, // this step has one live stream
     lone_kept: [2][lone_costs.bands]?lone_costs.Kept = @splat(@splat(null)), // the newest lone calibrations, greedy then sampled, a band each
     steps: u64 = 0,
     arena: std.heap.ArenaAllocator, // a round's temporaries
@@ -347,10 +348,9 @@ pub const Engine = struct {
                     if (p.recalled) {
                         if (s.recall) |r| r.observe(@intCast(own), @intCast(own_kept), s.rounds);
                     } else if (s.proposer) |pr| pr.observe(@intCast(own), @intCast(own_kept));
-                    if (e.alone) {
-                        const width = s.copy_width orelse e.cfg.first_copy;
-                        s.copy_width = if (own_kept == own) @min(e.cfg.max_copy, 2 * width + 1) else e.cfg.first_copy;
-                    }
+                    // a shared round grows the same ramp: each stream's save holds the copy window
+                    const width = s.copy_width orelse e.cfg.first_copy;
+                    s.copy_width = copy_limit.nextWidth(width, e.cfg.first_copy, e.cfg.max_copy, own_kept == own);
                 } else if (p.kind == .head) {
                     if (p.lanes) |l| if (s.odds != null) s.observeLanes(l, path);
                     const ds = try accept.depths(a, parents);

zig/src/core/lanes/lanes.zig

@@ -65,6 +65,8 @@ test {
     _ = @import("fill_test.zig");
     _ = @import("fill_trim_test.zig");
     _ = @import("fair_turns_test.zig");
+    _ = @import("copy_limit.zig");
+    _ = @import("copy_exact_test.zig");
 }
 
 pub const learned_disk = @import("learned_disk.zig");

zig/src/core/lanes/root_copy.zig

@@ -2,6 +2,7 @@
 const Engine = @import("engine.zig").Engine;
 const be = @import("backend.zig");
 const win = @import("windows.zig");
+const limit = @import("copy_limit.zig");
 
 pub fn prepare(e: *Engine, requests: []const be.DraftRequest) ![]const be.DraftRequest {
     if (!e.cfg.root_copies) return requests;
@@ -15,8 +16,7 @@ pub fn prepare(e: *Engine, requests: []const be.DraftRequest) ![]const be.DraftR
             const pending = try e.readFeed(r.first orelse continue);
             const most = @min(@as(i64, e.cfg.max_copy), s.draftRoom() - 1);
             const got = p.proposeAfter(s.context.items, &.{pending}, most) catch continue;
-            const least: usize = if (e.alone and e.cfg.context_copies) 1 else 2;
-            break :blk got.len >= least and p.lastMatch() >= e.cfg.enter_match;
+            break :blk limit.take(e, s, p.lastMatch(), p.lastEnd(), got).len > 0;
         } else (try win.copyProposal(e, s)).tokens.len > 0;
         if (!copies) continue;
         if (out == null) out = e.arena.allocator().dupe(be.DraftRequest, requests) catch return requests;

zig/src/core/lanes/windows.zig

@@ -12,6 +12,7 @@ const shape = @import("shape.zig");
 const ev = @import("events.zig");
 const trail = @import("trail.zig");
 const Engine = @import("engine.zig").Engine;
+const limit = @import("copy_limit.zig");
 const Stream = @import("stream.zig").Stream;
 const Value = ev.Value;
 const f = ev.f;
@@ -195,17 +196,16 @@ fn headTrunk(e: *Engine, s: *Stream, p: Plan, levels: usize) !?fill.Trunk {
     return .{ .tokens = chain, .others = others };
 }
 
-/// A copy from the context backed by `enter_match` ids (Python `_copy_proposal`), else Recall's when it should pay.
+/// A context copy backed by enter_match (Python `_copy_proposal`), else Recall; shared rounds use this ramp too.
 pub fn copyProposal(e: *Engine, s: *Stream) !Copy {
     if (e.cfg.max_copy == 0 or s.force.items.len > 0 or (e.cfg.root_copies and s.root_copy_miss)) return .{};
-    const width: u32 = if (e.alone) (s.copy_width orelse e.cfg.first_copy) else e.cfg.first_copy;
-    const most = @min(@as(i64, width), s.draftRoom() - 1);
+    const most = @min(@as(i64, limit.ask(e, s)), s.draftRoom() - 1);
     if (s.proposer) |p| if (!e.alone or e.cfg.lone_copies) {
         // a priced proposer's drafts are as it gives them, its none too: no width, length or match gate, no Recall
         if (p.vtable.priced) return .{ .tokens = p.propose(s.context.items, @min(@as(i64, e.cfg.max_copy), s.draftRoom() - 1)) catch &.{} };
         const copied = p.proposeAfter(s.context.items, &.{}, most) catch &.{};
-        const least: usize = if (e.alone and e.cfg.context_copies) 1 else 2;
-        if (copied.len >= least and p.lastMatch() >= e.cfg.enter_match) return .{ .tokens = copied };
+        const kept = limit.take(e, s, p.lastMatch(), p.lastEnd(), copied);
+        if (kept.len > 0) return .{ .tokens = kept };
     };
     const r = s.recall orelse return .{};
     if (most < 2) return .{};

zig/src/families/glm/cuda.zig

@@ -67,6 +67,7 @@ test {
     _ = engine;
     _ = follow;
     _ = lanes;
+    _ = @import("cuda_copy.zig");
     _ = native;
     _ = @import("cuda_qpack.zig");
     _ = @import("cuda_tile.zig");