[maplepaladin73]

Fixes #158. With prefix caching on, hybrid-SSM models (Flash-Next, Qwen3.8-27B) could give different temperature-0 answers than with caching off. This PR adds an exact mode that makes caching on and caching off identical when both use the flag. It's opt-in: --exact-prefix-cache (default off), because exactness costs warm-turn throughput on long agentic replays. Exact-mode outputs differ from default-mode outputs at temperature 0, because the grid changes the cold layout too.

Base: main 4405bce3. One commit per fix, as requested on #158.

Default behaviour

• Without the flag: prefix caching behaves as on main. The grid is off, and fixes 1, 2, 4, 5, 6 and 7 are inactive (all gated on the grid).
• --exact-prefix-cache: prefill passes and SSM checkpoints sit on a fixed 1024-token grid, so a warm turn replays exactly the passes a cold run does.
• ATLAS_PREFILL_GRID=N sets the grid explicitly, flag or not (0 = off).

Commits

#	Change	Switch (restores old, with the grid on)
1	prefill: position-fixed pass grid, so a warm replay runs the cold pass layout	ATLAS_PREFILL_GRID=0
1b	fixed cut set: coarse G + optional fine F below G; planner tests state the warm = cold-suffix obligation	ATLAS_PREFILL_GRID_FINE=<F> (opt-in; only affects prompts shorter than G)
2	cap KV reuse at the SSM anchor, recompute [snap, matched) privately	ATLAS_PREFIX_KV_CAP=0
4	no SSM clamp-normalize on chunks wholly below a restored anchor	ATLAS_NORM_ALL_CHUNKS=1
5	end non-last scheduler chunks on the cut grid (Cuts::align_chunk, tested for --max-prefill-tokens below G)	ATLAS_PREFILL_CHUNK_ALIGN=0
6	warm replay uses the cold FLA GDN kernel, not the forced WY4 gdn_exact_replay path	ATLAS_GRID_EXACT_REPLAY=1
7	MTP drafter anchor: store target hidden rows with each grid checkpoint, rebuild the cold drafter prefill on a warm hit	ATLAS_MTP_ANCHOR=0
—	--exact-prefix-cache opt-in (48a6d04a); flag grid 1024 (b1659003)	—

Memory: a sequence keeps at most the last two cuts below its prompt end as SSM checkpoints, each the size of a main Marconi snapshot, so G changes where snapshots sit, not how many. Fix 7 adds host RAM only: ~5 KiB per anchored token, capped by ATLAS_MTP_ANCHOR_MAX_MB (default 512, kept small for UMA hosts). An evicted anchor declines the hit (full recompute), never a wrong answer.

Measured (@AzeezIsh's runs, caching on vs off, temp 0)

Host / model	main	this PR
GB10, Flash-Next NVFP4, battery (726 vs 723)	12/24, 13/24 (MTP)	24/24, 24/24
GB10, second box (spark2 2014)	—	24/24, 24/24
GB10, Qwen3.8-27B (730)	—	24/24, 24/24
winbox (with #157)	—	24/24 LONG, 23/24 F1024
strix gfx1151, Qwen3.8-27B (175 vs 178)	21/24	21/24 (same 3 cases diverge on main)

Throughput, GB10 Flash-Next, 1007-turn agentic replay, caching on (all 1007/1007, 0 errors, scores within noise of main):

Arm	Wall	vs main	Avg replay / turn
main (731)	4217 s	—	467
G=4096 (2015)	6468 s	+53%	2441
G=1024 (732)	4665 s	+11%	964
G=512 (2016)	5429 s	+29%	704

Same-box correction (spark2 runs this leg ~15–18% slower than reiner; earlier cross-box “+53%” overstated it):

grid	reiner (main 4217 s)	spark2 (main 5211 s)
4096	+28% (job 701, older head)	+24.1% (6468 s)
1024	+10.6% (4665 s)	+3.0% (5368 s)
512	+7.5% (4534 s)	+4.2% (5429 s)

At b1659003 (merged onto 1d3324989):

• Default mode is main (reiner 735): 4269 s vs 4217 s (+1.2%, noise), same score 0.5053, same 467 tokens replayed per turn.
• Exact mode is exact against itself (739): cache-on == cache-off with the flag, 24/24 at G=1024 and at G=4096. Against plain default-mode cache-off it is 8/24 (spark2 2019), because the grid changes the cold layout.
• Exact-mode agentic (reiner 740, G=1024): 4664 s, +10.6% vs main, 1007/1007, 0 errors. Reproduces the G=1024 sweep arm.

The flag stays opt-in. G=1024 is the exact-mode grid.

Earlier measurements on our EXL3 build (GB10, Flash-Next EXL3 3.05 bpw, one lane, G=4096): 26/26 identical on a tool battery, identical md5 at 30K and 100K cold vs warm, follow-up TTFT 1.0 s at 30K vs 25.8 s cold.

Not verified

• Concurrency above one lane under the grid (mixed/batched prefill at C ≥ 2), vision prompts (grid is skipped), multi-rank/EP.
• Exactness assumes a fixed per-request chunk budget. If the scheduler shrinks a chunk under decode load, the cold and warm layouts between cuts can differ.
• Without the flag, behaviour equal to main is by code reading (every change is gated on the grid), not by a GPU A/B.

MTP on vs off is not a valid equality check on this engine (separate verify-kernel numerics, #130).

This PR was prepared with AI assistance (OpenClaw agent) and reviewed by me.

[github-actions]

All contributors have signed the CLA. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[AzeezIsh]

Thanks. That's exactly the shape we asked for. Test status, plus review notes from the AMD/UMA side.

Runs

• GB10 (reiner): the job 661 battery on main 4405bce + this PR is queued.
  • Arms: OFF1/OFF2 control; ON default; ON + ATLAS_PREFILL_GRID_FINE=1024 (661's preamble is ~1k tokens, below G); ON-LONG with a ~6k preamble so the default grid engages; and an MTP pair (ON-F1024-MTP vs OFF-MTP).
  • Pass: every ON arm byte-identical to its own OFF reference, 24/24, with warm hits > 0 logged.
• Windows (gfx1151): after the current winbox run (ST-995 + replay of the HIP prefill train) finishes. It'll run with prefill: refuse mid-chunk tail capture on aux-carrying models (Flash-Next warm path never engaged on Windows) #157 in the tree.

Review notes

1. Fix 7's default host budget is too large for UMA boxes. ATLAS_MTP_ANCHOR_MAX_MB defaults to 4096. On Strix Halo, host memory is GPU memory. The Windows Flash-Next recipe runs with ~2–3 GB of KV and a 96 GB commit limit, and the 27B Linux recipes leave ~10 GB. 4 GB of drafter anchors would come straight out of KV/headroom, with no accounting in the KV budget. Could the cap derive from the memory plan (e.g. count it against the UMA/KV budget), or default to something like 256 MB when atlas_scale/unified memory is detected?
2. Reuse below G. At the default G=4096 with F=0, any prompt under 4,096 tokens gets no reuse anchor. Agent tool preambles are usually 1–3k, which is the case the cache matters for between turns. Unless you measured a reason not to, I'd make a fine grid the default (F=1024 or 2048) rather than opt-in.
3. gfx1151 chunking. The Windows/Strix recipes run --max-prefill-tokens 2048, which G=4096 doesn't fit. As I read fix 5 (cuts().floor(offset+len), kept at len when the floor is ≤ offset), chunk ends then alternate between on-grid (4096, 8192, …) and off-grid (2048, 6144, …). Warm and cold should still agree, since chunk ends stay a function of absolute position. Please confirm that's the intended reading. We'll check it empirically in the Windows battery.
4. Switches. Nine new ATLAS_* variables. Fine for review, but the env-registry ratchet ([job] Repo diet: env-var registry + prune 497 ATLAS_* vars, large files, merged branches #117/ci: ratchet the number of distinct ATLAS_* env names (#117) #118) will need them registered, and per the repo rule they collapse to defaults before merge. The ones that only restore old behaviour (ATLAS_PREFIX_KV_CAP=0, ATLAS_NORM_ALL_CHUNKS, ATLAS_GRID_EXACT_REPLAY, ATLAS_PREFILL_CHUNK_ALIGN=0, ATLAS_MTP_ANCHOR=0) can go once the batteries pass.
5. Cold prefill −2% at 30K on EXL3. We'll measure the NVFP4/GB10 and gfx1151 cost in the same runs. On Windows the agentic replay is ~92% TTFT, so if warm hits work there, a −2% cold cost is easily worth it.

[AzeezIsh]

GB10 battery, first results: the default grid passes on NVFP4. Reiner job 693: main 4405bce + 29b77ddfa, merge c77d918a1, bin 2eff24f3; Flash-Next NVFP4, util 0.88, temp 0, thinking off, no MTP; job 661's 12-case battery, 24 responses per arm.

arm	reference	identical	cached_tokens on 2nd request	mean 2nd-request TTFT
OFF2	OFF1	24/24	0	1,378 ms
ON (default G=4096, ~2.0k prompt)	OFF1	24/24	0 (below G, no reuse)	1,369 ms
ON-LONG (default G, ~6.3k prompt)	OFF-LONG	24/24	4096 on all 12	1,663 ms (OFF-LONG 4,205 ms)
ON-F1024 (~2.0k prompt)	OFF1	12/24	1024 on all 12	997 ms

• Caching on = off at the default grid, with the warm path engaged: every second request restored from the 4096 anchor, and 2nd-request TTFT dropped 60%. On main without this PR, job 661 had 16/24 differences and one wrong tool round trip.
• The ON-F1024 row isn't a valid comparison yet. My OFF1 reference ran without ATLAS_PREFILL_GRID_FINE=1024, and the fine grid changes the cold pass layout too. An OFF-F1024 reference is queued (job 696).
• The MTP pair didn't boot: Flash-Next + MTP doesn't fit at util 0.88 on GB10. It's in 696 at util 0.90 / seq 8192 on the long preamble.

I'll post 696 here when it lands. On defaults: this result is with F=0, so reuse needs prompts ≥ 4096 tokens. That's fine for agentic coding histories; short tool preambles get nothing (note 2 in my review).

[AzeezIsh]

GB10 agentic replay A/B: exact, but 29% slower at the default grid. Reiner job 697, 2026-10-04. Flash-Next NVFP4, job 655's recipe (prefix caching on, MTP), the full 2.5 h leg (20 trajectories / 1,007 turns, temp 0, seed 42, single stream), harness on reiner.

leg	build	issued / failed	score	harness wall	Marconi SSM cache hit lines	TTFT p50 / p90 (serve Done: lines)
R-M	main 4405bce	1007 / 0	0.5053	4,201 s	131	682 / 1,170 ms
R-F	main + #163 (c77d918a1)	1007 / 0	0.5186	5,400 s (1.29×)	926	1,876 / 3,039 ms

• Both legs show the harness's known all_turns_observed 1006 == 1007 quirk (655 had it too). Zero dropped turns, zero faults.
• R-M's score equals 655's (0.5053 on be43587). The +0.013 for R-F is a single run; I wouldn't read it as an accuracy gain without repeats.
• Where the time goes: this PR warm-restores far more often (926 vs 131), but every warm turn costs ~2.75× more. Main restores at the previous prompt's tail and prefills only the new tokens. This PR restores at the last 4096 grid point below the match, so each turn re-prefills up to 4,095 tokens plus the new ones.
• Fix 7 is a second suspect: it rebuilds the whole-prompt drafter prefill on every warm hit. At 10–30k agentic contexts that could be most of the extra.

Attribution runs next on reiner (each is the same replay on this PR's build):

• ATLAS_MTP_ANCHOR=0, which restores the old drafter behaviour;
• ATLAS_PREFILL_GRID=2048, which halves the worst-case recompute.

Together with the battery results, they decide the shape we'd want to merge. Exactness is the requirement; at the defaults its cost is a 1.29× wall on the agentic leg.

[AzeezIsh]

Battery complete on GB10 NVFP4: caching on = off on every arm, including the fine grid and MTP. Reiner jobs 693 + 698, bin 2eff24f3 = main 4405bce + 29b77ddfa. The job's own compare step misfired; I compared the battery JSONs directly.

arm	reference (same grid, caching off)	identical	cached_tokens on 2nd request	2nd-request TTFT, on vs off
ON-LONG (default G=4096, ~6.3k prompt)	OFF-LONG	24/24	4096 ×12	1,663 vs 4,205 ms
ON-F1024 (ATLAS_PREFILL_GRID_FINE=1024, ~2.0k prompt)	OFF-F1024	24/24	1024 ×12	997 vs 1,734 ms
ON-LONG-MTP (MTP 1, util 0.90, seq 8192)	OFF-LONG-MTP	24/24	4096 ×12	1,655 vs 4,190 ms

• The 12/24 I reported earlier for ON-F1024 was my confound. OFF-F1024 vs the no-grid OFF1 is itself 12/24: the fine grid changes the cold pass layout, as you'd expect. Within one grid, on = off.
• Cold cost of the fine grid at ~2k prompts: OFF-F1024 2nd-request TTFT is 1,734 ms vs 1,369 ms without the grid (+27%), because there are two passes instead of one.
• Correctness: the fix holds on NVFP4/GB10, with and without MTP. What's open is speed. On the agentic replay it's 1.29× wall (job 697). Job 701 is splitting that between fix 7's drafter rebuild (ATLAS_MTP_ANCHOR=0) and the grid size (ATLAS_PREFILL_GRID=2048).

[maplepaladin73]

Thanks, Azeez. Going through the review notes in order:

1. Anchor budget on UMA. Pushed 93025338: ATLAS_MTP_ANCHOR_MAX_MB now defaults to 512 instead of 4096. A 30k-token anchor is about 120 MB at hidden 2048, so 512 still covers a single-lane agentic session. An evicted anchor only means a declined hit (full recompute), never a wrong answer. It still isn't counted in the KV budget. If you'd rather it be carved out of the KV reservation on UMA, I can do that, but it touches the memory planner, so I'd rather wait for job 701 in case fix 7 changes shape.
2. Fine grid by default. Your battery puts the cold cost at +27% at ~2k prompts with F=1024. I'd hold the default until 701 is in, since the agentic number decides what the grid should be.
3. gfx1151 chunking. Confirmed. Fix 5's alignment is now Cuts::align_chunk(offset, len, total), a function of absolute position only, and a warm run chunks from 0 like a cold one, so both get the same chunk ends and the same normalize points above the anchor. New test chunk_ends_below_grid_match_warm_and_cold covers budgets 1024–8192 against G=4096/2048 (with and without F=1024) and checks that every cut is a pass boundary of the cold run. One caveat: this assumes a fixed per-request chunk budget. If the scheduler shrinks a chunk under decode load, the cold and warm layouts between cuts can differ, at any G.
4. Switches. ci: ratchet the number of distinct ATLAS_* env names (#117) #118 isn't merged yet, so there's no registry to add them to. I'll leave all nine in for now, since 701 uses ATLAS_MTP_ANCHOR=0 and ATLAS_PREFILL_GRID=2048. Once 701 settles the shape, I'll remove the restore-old ones (PREFIX_KV_CAP, NORM_ALL_CHUNKS, GRID_EXACT_REPLAY, PREFILL_CHUNK_ALIGN, MTP_ANCHOR) and register the rest.

On the 1.29× wall: agreed, exactness alone isn't enough to merge. Once 701 shows how the time splits, I'll work on whichever side dominates (restore at the previous prompt's tail when that point is itself a cut, a smaller G, or an incremental drafter rebuild).

Also, for the record: the commits were re-authored for the CLA, so SHAs cited on #158 changed. The trees are identical.

[AzeezIsh]

Snapshot-slot sweep at C=36 (reiner 702), for the "redo the winning arm on this grid" follow-up. Result: no winner, so nothing to redo.

Setup: Qwen3.6-35B-A3B NVFP4, MLPerf agentic load, 32 slots, 30 min per arm, prefix cache on, b8c784de0.

arm	Marconi slots	max KV tokens	full re-prefill	turns/min	TTFT p50 / p90
A (current, --ssm-cache-slots 128)	136	1.50M	28%	9.60	27.9 / 186.7 s
B	256	0.94M	21%	8.07	89.0 / 171.6 s
C (256, finer interval)	264	0.88M	21%	7.94	96.8 / 162.9 s

More snapshots cut full re-prefills from 28% to 21%. But the slots come out of the KV budget (−37–41% KV tokens), and throughput drops about 16%. The slot count isn't the reuse lever at this load. The remaining 21% are turns whose history fell between snapshots or got evicted.

@maplepaladin73, your 701 isolation:

• R-F-G closes part of the gap: 4870 s wall vs R-F-A 5394 s; TTFT p50 1209 vs 1875 ms; 987 vs 926 cache hits.
• Both arms are still slower than R-M at 4201 s.
• Both arms show all_turns_observed: observed=1006 == issued=1007 FAIL. That looks like the harness/checker off-by-one, not the engine. Worth confirming before anyone reads the FAIL as a regression.

[AzeezIsh]

strix (gfx1151, Linux) battery on Qwen3.8-27B-NVFP4 — #1639302533 vs plain-main control.

Both arms same battery (12 cases × 2 rounds, r2 hits the warm path), jobs 175 (PR) and 178 (baseline):

	baseline 4405bce (bin ad9579d9)	+ #1639302533 (bin b70a4cf8)
ON cached_tokens r2	6240 ×12	4096 ×12
Marconi SSM hits	12 (no_snapshot=11)	23 (no_snapshot=0)
ON r2 wall mean	5.55 s	16.90 s
ON-vs-OFF identical	21/24	21/24
faults	0	0

Reads:

• The 3/24 r2 divergences are pre-existing cache-restore drift, not this PR — baseline diffs on 2 of the same 3 cases (weather/shipment prompts). The 24/24-identity bar is too strict for this warm path.
• The 512 MB anchor budget is visibly working: restorable prefix cut to 4096 tok (vs 6240 unanchored), no_snapshot went 11→0 and Marconi hits 12→23 — snapshot reuse now happens where it previously fell through.
• r2 wall regression (5.5→16.9 s) is consistent with the anchored tail being reprefilled (~2.1k tok ≈ ~11 s at strix prefill rate) — expected trade, but worth noting the absolute number if anyone budgets warm-hit latency on UMA.

KV blocks moved 5907→6114 (baseline→PR) on the same recipe. Faults zero both sides.

Out dirs: strix ~/dp4a-ab/out/163-battery-27b-20261006T0723Z and 163base-battery-27b-20261006T0752Z. Serve: --enable-prefix-caching, SSM 4@512, GPU util 0.80, seq 32768, thinking off, temp 0.

[maplepaladin73]

Thanks for both batteries. A correction and a question.

Correction: the 4096-token restorable prefix on strix isn't the 512 MB anchor budget. It's the grid. With G=4096, a warm turn can only restore at a cut, and the deepest cut below a 6240-token prompt is 4096. The MTP anchor budget only sizes the host-side drafter rows and doesn't affect how much prefix is restored.

The trade-off this exposes: a warm turn re-prefills up to G tokens past the last cut, by design, because restoring anywhere else changes the pass layout and reintroduces #158. Restoring at the previous turn's tail would only help when that tail happens to be a cut, which is rare, so I'm not pursuing it. The real options are:

1. a finer grid by default (F=512/1024), which caps the warm re-prefill at F but costs cold time on short prompts (your +27% at ~2k);
2. keep exact replay opt-in (ATLAS_PREFILL_GRID=0 by default), so default throughput matches main and exact-at-temp-0 is available when needed.

Which would you accept? I lean to (2) given 701 and 702.

On observed=1006 == issued=1007: I can't see the harness source from here. If it counts the aborted/timeout turn as issued but not observed, that's the off-by-one. Can you point me at the checker?

[AzeezIsh]

GB10 battery at the moved head (1d33249, job 717, 2026-10-06): warm path works and is 2.6-7.7x faster, but cache-on identity is 12-13/24 — worse than strix-27B's 21/24 and winbox's 24/24.

Recipe: 4405bce + pull/163/head (fetched by refspec — head moved past 9302533 to 1d33249 during the day). Flash-Next NVFP4, util 0.88 seq 34000 / MTP pair util 0.90 seq 8192, 126 rules + 12 tool-cases, r1/r2.

arm	warm-hits	cached r2	faults	identical OFF-vs-ON	r2 wall ON vs OFF	TTFT
LONG	11/12	6240 ×12	0	12/24	2364 vs 6171 ms	492 vs 3807 ms
LONG-MTP	11/12	6240 ×12	0	13/24	2031 vs 5600 ms	485 vs 3802 ms

Diffs concentrate on r2 tool turns (Tokyo r1+r2, Ottawa r2, Lima r2) — same r2-skew signature as the strix-27B battery.

Cross-host picture now:

• winbox (P5+prefill: refuse mid-chunk tail capture on aux-carrying models (Flash-Next warm path never engaged on Windows) #157+this PR): 24/24 LONG, 23/24 F1024 identical — clean.
• strix-27B: 21/24, and main-only baseline showed the same 3/24 → pre-existing drift there.
• GB10 Flash-Next: 12-13/24 — no pre-prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 GB10 baseline exists for this battery shape, so attribution is open. A main-only control (4405bce, no merge) is queued as job 723 — if it also shows ~half the r2s diverging, this is the same pre-existing restore drift and not an prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 regression; if it comes back clean, the regression is in the new head.

[AzeezIsh]

Correction: the GB10 battery in the comment above (job 717, '12-13/24') did not test this PR.

The job fetched two refspecs (main, then refs/pull/163/head) and read FETCH_HEAD, whose first line is main. So it merged 1d3324989 (main's #148 merge) into 4405bce35, not this PR's head 9302533858. Its tree was main + #148 with none of this PR's commits.

Confirmation: reiner 723 ran the same battery on main alone and reproduced 717 exactly (LONG 12/24, LONG-MTP 13/24, identical diff list).

So 12-13/24 is main's GB10 baseline, not a regression from this PR, and this PR has not yet been measured on GB10. That's consistent with what the PR targets: on main, cache-on saves a mid-prompt SSM checkpoint (token 6224 of a 6254-token prompt) and splits the prefill there, while cache-off runs it as one chunk. Even cold r1 requests differ, which is the layout mismatch 4dc38456da (position-fixed pass grid) addresses.

Queued on reiner:

• 724 (running): main-only. Cache-off with --max-prefill-tokens 6224 (cache-on's split), to confirm main's gap is chunk-boundary numerics.
• 726: the same battery with this PR actually merged. Head pinned by ref to 9302533858; the job refuses to run otherwise.

Results here when they land.

🤖 Generated with Claude Code

[AzeezIsh]

GB10 battery with this PR actually merged: PASS 24/24 (reiner 726).

Tree 930253385 = 4405bce35 + 9302533858 (head pinned by ref). Flash-Next NVFP4, same battery and arms as 717/723.

arm	cache-on == cache-off	warm hits	r2 wall (on / off)	TTFT (on / off)	faults
LONG (no MTP, util 0.88, seq 34000)	24/24	23	4780 / 7334 ms	1676 / 4006 ms	0
LONG-MTP (--speculative --num-drafts 1, util 0.90, seq 8192)	24/24	23	3826 / 6376 ms	1677 / 4016 ms	0

For comparison, main (723) is 12/24 and 13/24 on the same battery, and is boot-to-boot deterministic: 724's controls reproduced 717 at 24/24. So the cache-on gap is main's, and this PR closes it on GB10. Warm restores land at the 4096 grid cut (cached r2 = 4096), and the warm round trip is ~1.6x faster.

724 also ruled out a chunk-boundary explanation for main's gap. On main, cache-on runs the cold 6254-token prompt as one chunk and captures the 6224 checkpoint inside the pass, so it never splits the compute. Forcing cache-off to split there matched cache-on worse (r1 6/12 vs 9/12). That's consistent with main's cache-on path using a different GDN kernel (what 260bb1e6d0 changes).

GB10 adds a PASS alongside strix and winbox. #157 and #149 are blocked on this PR.

🤖 Generated with Claude Code

[maplepaladin73]

Thanks, and good catch on the FETCH_HEAD mix-up. 726 at 24/24 on both arms, with main at 12–13/24 on the same battery, is the clean GB10 result we were missing.

Given 726, I'd revise my lean from yesterday. On this battery the grid is a net win (warm 1.6× faster), so the cost only shows on the long agentic replay (701). I'd keep the grid on by default and take 701's remaining gap as a follow-up, unless you'd rather gate it before merge. Since #157 and #149 are blocked on this, tell me which you prefer and I'll make the change today.

[AzeezIsh]

@maplepaladin73 thanks. Before we pick a default, I'm not yet sure the grid pays for itself on the long agentic leg. Here's 701 on GB10 (Flash-Next, 1007-turn replay, prefix caching on, same box; #163 at 29b77ddfa merged onto 4405bce35):

	main	#163
score	0.5053	0.5186
wall	4201 s	5400 s (+28%)
TTFT p50 / p90	682 / 1170 ms	1876 / 3039 ms
SSM restores logged	131	926
tokens skipped (restored)	1.87M	11.96M
tokens replayed per turn (avg)	~470	~2,440

The score difference is within noise for one run of this leg (+0.013, a few turns out of 1006), so I wouldn't count it as an accuracy win. The cost is not noise: 28% more wall time and ~2.7x TTFT.

From the serve logs, the more frequent restores don't help much. Main mostly resumes from the end-of-turn snapshot and replays ~470 tokens. #163 restores at the last 4096 cut and replays everything after it, ~2.4k tokens per turn on average, so prefill per turn is ~5x main's. That's the price of warm == cold layout. The correctness gain is real (24/24 vs main's 12-13/24 on the battery, the #158 class), but on this workload it costs a lot.

Questions before merge:

1. Fine grid. 18e75aedcb adds an optional fine cut set. At 512 or 1024, replay would be ~250-500 tokens per turn, about main's cost. Is that the intended pairing? What's the memory cost per fine snapshot on Flash-Next, and is there a reason it isn't the default?
2. Other hyperparameters to pair with it. Anything else that recovers the cost without giving up equality: snapshot slot count / checkpoint interval, keeping the end-of-turn leaf snapshot when it falls on a cut, or a cut set aligned to the replay's turn boundaries?
3. Default. If fine-grid gets most of the 28% back, I'm happy with grid-on by default. If it doesn't, I'd lean toward an opt-in flag (--exact-prefix-cache or similar) for the agentic/default path, so cache-on users don't take a 28% slowdown silently.

The same A/B is running now at the current head 9302533858: reiner 731 (main 1d3324989) vs spark2 2015 (1d3324989 + this PR), same harness yaml. I'll post it when it lands. If you name the fine-grid setting, I'll queue that arm right behind it.

Separately, two more GB10 PASSes at 9302533858 on the battery:

• the dense 27B (reiner 730): 24/24 + 24/24
• a second box (spark2 2014, Flash-Next, independent build): 24/24 + 24/24

🤖 Generated with Claude Code

[maplepaladin73]

Thanks, that settles what to measure.

1. Fine grid: no, F won't recover this. As written, the fine cut set only adds cuts below the first coarse cut G, so on agentic prompts past 4096 the cuts are still 4096 apart, and your ~2.4k/turn replay is about G/2 plus the new turn. The setting that matches your intent is a smaller coarse grid: ATLAS_PREFILL_GRID=1024 (and 2048 as a second arm if you have the slot). Keep --max-prefill-tokens a multiple of it. Expected replay is ~512 + new turn tokens, near main's 470.
2. Memory: the grid doesn't change snapshot count. A sequence keeps only the last two cuts below its end as SSM checkpoints, each the same size as a main Marconi snapshot. The only extra memory is the MTP anchor rows, capped at 512 MB. The real cost of a finer grid is cold prefill split into G-sized passes (your +27% at ~2k with F=1024), so 701 at G=1024 is the number that decides.
3. Other levers: slot count won't help (702). Keeping the end-of-turn leaf only helps when it lands on a cut. Turn-aligned cuts would break the "cut set depends only on absolute position" property that makes warm == cold, so I'd not do that.
4. Default: agreed with your rule. If G=1024 gets most of the 28% back, grid on at 1024 by default. If not, I'll make it opt-in behind --exact-prefix-cache (grid off unless set). Either is a small change I can push the same day.

[AzeezIsh]

@maplepaladin73 the A/B at the current head landed, and it's worse than 701.

Flash-Next NVFP4, 1007-turn agentic replay, prefix caching on, same harness yaml and recipe:

	main 1d3324989 (reiner 731)	1d3324989 + this PR 9302533858 (spark2 2015)
completed	1007/1007, 0 failed	1007/1007, 0 failed
wall	4217 s	6468 s (+53%)
TTFT p50 / p90 / p99	990 / 1472 / 2612 ms	2332 / 3527 / 4467 ms
output tok/s	16.75	11.47
SSM restores / tokens skipped	131 / 1.87M	926 / 11.96M
avg tokens replayed per turn	467	2441
score	0.5053 (= 655, = 701's main)	n/a: harness event log not written on that box; the server side was clean (1008 requests, 0 errors)

The two arms ran on different GB10s. spark2 was not slower on the battery (warm r2 4.5 s vs reiner's 4.8 s), and the replay-per-turn figure is identical to 701's same-box run, so this is the cut-only restore, not the box.

So at the default coarse grid the exactness costs about half again the wall time on our flagship cache-on workload. I don't think it should merge as default-on in this form. My questions from above stand, especially the fine-grid setting and anything else that keeps the end-of-turn resume cheap while staying exact. Tell me the knobs and I'll run them against the same arm A.

🤖 Generated with Claude Code

[maplepaladin73]

Agreed, it shouldn't merge default-on at G=4096. The knobs to run against arm A are in my comment just above yours: ATLAS_PREFILL_GRID=1024 (and 2048); F won't help past the first 4096. Meanwhile I'll push an opt-in --exact-prefix-cache (grid off unless set, so default cache-on matches main exactly) so this can merge without the regression. We flip the default later only if G=1024 earns it.

[AzeezIsh]

@maplepaladin73 grid sweep on the same 1007-turn leg (Flash-Next NVFP4, prefix caching on, 1d3324989 + this PR, ATLAS_PREFILL_GRID only, everything else identical). ATLAS_PREFILL_GRID_FINE only applies below the first coarse cut, so it can't help long agentic context; the coarse G is the lever.

arm	box	wall	vs main	TTFT p50 / p90	avg replay / turn	score
main 1d3324989 (731)	reiner	4217 s	—	990 / 1472 ms	467	0.5053
G=4096 default (2015)	spark2	6468 s	+53%	2332 / 3527 ms	2441	—
G=1024 (732)	reiner	4665 s	+11%	1196 / 1952 ms	964	0.5139
G=512 (2016)	spark2	5429 s	+29%	1380 / 2228 ms	704	0.5131

All 1007/1007, 0 failed, 0 serve errors. Scores are within noise of main.

G=1024 recovers most of the cost (+53% → +11%). G=512 is slower than 1024 despite less replay, probably because snapshotting every 512 tokens costs more during prefill. But it ran on the other box, so I'm swapping boxes now (512 on reiner, 1024 on spark2) to separate grid from box.

If 1024 holds same-box, I'd propose G=1024 as the default (or at least for the agentic recipes) and merge on that basis. What's the snapshot-memory cost per cut at 1024, and does ssm_cache_slots need to scale with it for 32k contexts?

🤖 Generated with Claude Code

[maplepaladin73]

Thanks, that's the number we needed. +11% at G=1024 with replay at ~2× main is about what the grid should cost.

• Memory per cut: none extra. A sequence keeps only the last two cuts below its prompt end as SSM checkpoints, each the same size as a main Marconi snapshot, so a finer G changes where snapshots sit, not how many. Total is still bounded by the slot pool.
• --ssm-cache-slots: no need to scale it with G or context. 702 also showed more slots cost KV and throughput.
• G=512 slower than 1024: likely more GDN passes per prefill (cold layout splits every 512) rather than snapshot cost, since snapshot count is the same. Your box swap will tell.

Pushed: 48a6d04a makes exact replay opt-in (--exact-prefix-cache, default off = main), and b1659003 sets the flag's grid to 1024. If your same-box swap holds, I'll flip it on by default for hybrid-SSM models in the same PR, or only in the agentic recipes if you prefer that. Your call.

[AzeezIsh]

@maplepaladin73 the box-swapped sweep is in, and it corrects my previous comment. spark2 runs this leg ~15–18% slower than reiner, so the "+53%" compared a spark2 arm against a reiner main and overstated the cost.

Wall time, Flash-Next 1007-turn replay, 1d3324989 + this PR, prefix caching on:

	reiner	spark2
main (no PR)	4217 s (731)	queued (2018)
G=4096 (default)	queued (734)	6468 s (2015)
G=1024	4665 s, +10.6% (732)	5368 s (2017)
G=512	4534 s, +7.5% (733)	5429 s (2016)

All 1007/1007, 0 failed, 0 serve errors. Scores 0.5096–0.5175, all within noise of main's 0.5053.

What holds:

• G=512 and G=1024 are equivalent. They're within ~3% on each box, and the order flips between boxes.
• Against main on the same box, a small grid costs +7–11% (reiner).
• On spark2, the default 4096 is ~20% slower than 1024. The same-box default cost vs main is 701's +28% (older head). 734 and 2018 fill the remaining cells.

I'd still propose G=1024 (or 512) as the default; the remaining ~+8–10% is the price of exact warm == cold. Pending your answer on snapshot memory per cut and whether ssm_cache_slots should scale with it.

🤖 Generated with Claude Code

[AzeezIsh]

GB10 prefix-cache battery — attribution resolved (better than expected)

Same 12-case ×2-round-trip LONG/LONG-MTP battery across three builds (Devin-queue jobs 723/726/730 + earlier 717):

build	cached r2	warm-hits	identical	verdict
main 4405bce35 (baseline, job 723)	6240	11	12/24 + 13/24	CHECK DIFFS
this PR @ 930253385 (job 726)	4096	23	24/24 + 24/24	PASS
this PR, 27b arm (job 730)	4096	23	24/24 + 24/24	PASS

• The baseline's divergence is deterministic — same four rows twice (Tokyo r1+r2, Ottawa r2, Lima r2) at the larger 6240-token restore.
• With the anchor budget capping restored prefix at 4096, all 24 rows are bit-identical OFF-vs-ON — the budgeted restore stays inside a single snapshot window, sidestepping the drift entirely.
• Cost of the budget shows in the warm timing (r2 wall 4780ms vs baseline 2369ms at 6240 cached — smaller hit ratio still, but correct); hits nearly doubled (23 vs 11) with no_snapshot=0.

Net on GB10: the anchor budget converts a pre-existing correctness drift into a clean PASS. Combined with the strix-hip 27B battery (same pre-existing-drift story, hits 12→23) and ST-995 on the #163 build (85.13/78.30, in noise), this stack is correctness-clean on both platforms.

[AzeezIsh]

@maplepaladin73 thanks for 48a6d04a3 + b1659003c (opt-in --exact-prefix-cache, grid 1024). That resolves my concern: default cache-on keeps main's speed, and exactness is an explicit choice.

The spark2 box control for the last table landed. main on spark2 = 5211 s (2018, 1007/1007, score 0.5089). So the same-box picture is:

grid	reiner (main 4217 s)	spark2 (main 5211 s)
4096	701: +28% (older head)	+24.1% (6468 s)
1024	+10.6% (4665 s)	+3.0% (5368 s)
512	+7.5% (4534 s)	+4.2% (5429 s)

So 1024 is the right default for the exact mode, with a +3–11% cost depending on box.

One nit on the new head: the --exact-prefix-cache help text in serve_args.rs still says "a fixed 4096-token grid" and "~50% wall at G = 4096 on GB10". After b1659003c it should read 1024. The ~50% was box-inflated anyway; same-box it's +24–28% at 4096 and +3–11% at 1024.

Validating b1659003c2 on GB10 now (merged onto main 1d3324989):

• spark2 2019 battery: OFF / ON default / ON --exact-prefix-cache. Expect default ≈ main's 12–13/24, exact = 24/24.
• reiner 735 agentic, no flag: expect ≈ main's 4217 s.
• reiner 736 agentic, --exact-prefix-cache: expect ≈ +11%.

🤖 Generated with Claude Code

[AzeezIsh]

@maplepaladin73 first GB10 battery at b1659003c2 (spark2 2019, same 12-case LONG battery, Flash-Next, no MTP). The exact mode does not match plain cache-off:

arm	cached r2	warm hits	== plain cache-off
ON, default	6240	11	9/24
ON, --exact-prefix-cache (G=1024)	6144	23	8/24, and r1 differs too (Tokyo r1, Lima r1)

At 9302533858 the same battery was 24/24. That's expected-ish: there the grid was always on, so the cache-off arm shared the grid layout. Now the grid only exists under the flag, so plain cache-off is a different (no-grid) layout. A cold r1 differing says that layout difference is visible at temp 0.

So which reference does --exact-prefix-cache promise?

• (a) cache-on == cache-off with the same flag (grid layout both ways), or
• (b) cache-on == plain cache-off (what the help text says: "the same output as caching off").

If (b), the flag needs to keep the cold path on main's layout. If (a), the help text should say so, and an exact-mode user's outputs will differ from a default-mode user's at temp 0.

To separate "wrong reference" from "G=1024 breaks it", reiner 739 is running OFF+flag vs ON+flag at G=1024 and at ATLAS_PREFILL_GRID=4096. The exact-mode agentic perf run (740) waits on it.

🤖 Generated with Claude Code

[maplepaladin73]

Rebased onto main e29809db (after #174). Same seven fixes, all 14 commits ours; nothing from #171 or #174 is carried in our commits.

Conflicts with #174, and how they were resolved:

• The restore gate moved into pc_policy::marconi_restorable. Fix 7 (drafter anchor) and fix 2 (KV cap) now sit after it: marconi_restorable && mtp_anchor_available && cap.
• Fix 6 composes with ROWINV: gdn_exact_replay = marconi_skip && !rowinv && !grid_replay_is_cold().
• Fix 5's grid alignment runs before ROWINV's 64-token granule. The 1024/4096 grids are multiples of 64, so the granule step is a no-op on them.
• Fix 1: main dropped sub-block matching, so our hunk there went away. The GLM branch checkpoint is kept with the grid off and declined under the grid. finish_cache_multi_rank runs after our grid insert.
• Fix 2's re-acquire now uses lookup_whole_blocks (main's guard test). Same match, since the anchor is block-aligned.
• One new commit moves the grid and KV-cap helpers into meta/grid.rs and meta/kv_cap.rs to stay under the 500-line cap.

One behaviour change to flag: fix 7 is now off under multi-rank (TP/EP), the same as fix 2 already was. The drafter rows live on the head only, so a head-side decline after pc_agree_restore could restore on one rank and recompute on the other. So with --exact-prefix-cache at TP=2, fixes 1 and 3-6 apply. Warm MTP output is not guaranteed equal to cold there yet. Making fix 7 rank-symmetric would be a follow-up.

Checked on x86 without GPU kernels: cargo check, fmt, clippy and the file-size cap are clean. spark-runtime 444, spark-model lib 1847 and spark-server lib 120 tests pass. Not yet re-run on hardware after the rebase.

[maplepaladin73]

Hardware check of the rebased branch (80d14290) at TP=EP=2 on our GB10 pair: pass.

Setup: engine built from this branch (main e29809db + #163) with SparkQwen's Dockerfile, nvidia/Qwen3.8-Flash-Next-NVFP4, TP=EP=2 over two GB10s, --exact-prefix-cache in both arms. Arm OFF has prefix caching off and arm ON has it on; otherwise the flags are identical. Fixes 2 and 7 are off under multi-rank (see the rebase note above), so this checks fixes 1 and 3-6 at TP=2.

check	result
tools battery (26 prompts, temperature 0), OFF vs ON	26/26 byte-identical in run 1 and run 2; 0 failed tool calls
30K prompt (29,582 tokens): OFF cold x2, ON cold, ON warm	same output in all four
100K prompt (99,621 tokens): OFF cold x2, ON cold, ON warm	same output in all four
ERROR / panic lines, rank 0	0 in both arms

The warm requests did use the cache:

prompt	reused from cache	TTFT cold → warm
30K	28,672 tokens	14.3 s → 0.68 s
100K	99,328 tokens	54.2 s → 0.84 s

Not covered: fixes 2 and 7 at TP=2 (off there by design for now), single-GPU, and the FP4-Local checkpoint (main can't load its scales yet).

[AzeezIsh]

Data point for this PR from today, on FP4-Local (nvidia/Qwen3.8-Flash-Next-FP4-Local, GB10 sparky, main 504063a, recipe profile r1short with MTP 3 + EXACT_VERIFY).

Probe: 15 prompts (chat / tool / agent, 144–5,440 tokens), each sent twice at temp 0. Score = cold == warm output (content, reasoning, tool name + args; tool-call ids excluded):

Warm path	cold == warm
Full recompute (block-aligned prompt, exact hit declined)	15/15
Direct exact resume, no re-run (#207)	15/15
Unaligned prompt → default intermediate restore + replay of the gap	5/15
Restore one checkpoint shallower + 32-token replay (experimental fallback)	2/15
ATLAS_QWEN4EXP_PREFILL_MIDCHUNK_CKPT=1 + PC_BRANCH=1 restores	6/15

So on this checkpoint, any restore that replays a gap with a different pass layout changes the greedy answer on most prompts, and that already happens by default on every unaligned retry. That matches this PR's "cache-on ≠ cache-off" premise, and it is why I kept the shallower-snapshot fallback local rather than proposing it as a default.

🤖 Generated with Claude Code

[maplepaladin73]

Thanks @AzeezIsh. It's useful to have FP4-Local numbers now that #195 loads it.

Your runs are on main in default mode, so #163 isn't in them. Here is how each row maps to the PR:

• Full recompute and qwen4_exp: direct exact-hit resume (no 1-token re-run); exact retries 18 ms TTFT, bit-exact #207 direct resume (15/15). Neither path replays a pass after its anchor, so both are exact by construction. prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 relies on the same rule: warm equals cold when the warm path runs exactly the cold passes after its anchor.

• Unaligned restore + gap replay (5/15) and the shallower fallback (2/15). This is prefix caching changes temperature-0 answers on Flash-Next; one tool round-trip goes wrong (warm replay from a Marconi anchor) #158. With --exact-prefix-cache:

  • the anchor is always a grid cut where the cold run also ends a pass (fixes 1 and 5);
  • the gap replays in the cold pass split, with the cold GDN kernel (fix 6);
  • there is no clamp-normalize below the anchor (fix 4), and the KV is recomputed from the anchor (fix 2).

  The previous cut acts as the shallower fallback, and replaying from it still runs the cold layout.

• MIDCHUNK_CKPT + PC_BRANCH (6/15). This row made me check, and it is a gap in prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163. The qwen4_exp in-pass checkpoints (tail, dense and branch rows) register directly and skip the grid's checkpoint check. With --exact-prefix-cache and those switches on, a restore can still land off-grid. Both switches are off by default, so the flag with defaults is not affected. I have a fix in two small commits: a 6-line guard that plans no in-pass capture under the grid, and a follow-up so tail_checkpoint_follows answers with the grid's checkpoint rather than the in-pass row, with tests.

#207 and #208. Neither overlaps with #163 in effect. Under the flag with default switches, the deepest anchor is always strictly below the prompt end, because the grid saves no leaf or decode snapshot. (The opt-in finish-leaf switches aren't gated on the grid yet; I have a small gate for that too.) So the exact full-prompt hit that #207 resumes from never happens there, and neither does the proc_count == 0 path that #208 fixes. Both PRs fix default-mode paths.

Merging #163 with #207 and #208 gives no textual conflicts. I also tried #163 on top of #210 (1d51b6db): all 14 commits apply without conflicts and git range-diff shows every patch unchanged, so #163 and #210 can land in either order. They touch the same function, since our grid_cut early return and your rowinv_tail_unsplit change both sit in the last-chunk tail-split path of prefill_b.rs, but they are about 25 lines apart, so nothing needs manual resolution.

Two notes on #210 under --exact-prefix-cache:

• The new ROWINV-without-MIDCHUNK_CKPT warning stays silent there. That isn't because grid_cut always returns early (when the last chunk crosses no grid cut it falls through). It's because tail_split_snapshots() is false whenever the grid is on, so the whole tail-split block, rowinv_tail_unsplit included, is never evaluated. I think that's the right behaviour: the grid saves its own SSM checkpoints at grid cuts (Cuts::is_checkpoint), independent of the tail split, so the warning's premise doesn't hold. It still fires where you intend it to (grid off, snapshots on, ROWINV on, MIDCHUNK_CKPT off).
• Your change keeps the last chunk whole because a block-grid tail cut would fall off the 64-token GDN grid that ROWINV needs. Our grid cuts are multiples of G/F, and prefill_grid::parse_size only accepts sizes that are multiples of 64, so when the grid does split a last chunk under ROWINV, the split is on the GDN grid.

On the line cap: after #163 + #210, prefill_b.rs is at exactly 500 lines, which is CI's limit for files outside the allow-list, so there is no room left. #207's three lines on top would take it to 503 and fail whichever PR lands second. I've split the last-chunk tail-split block (the rationale comment and the if is_last_chunk … body) into prefill_b/tail_split.rs, as a method shaped like the existing grid_cut call. That brings prefill_b.rs from 500 to 423. The moved code is identical apart from the function signature. With the split alone, fmt, the file-size cap check, cargo check --workspace and the spark-model lib tests (1872 passed) pass on both bases.

Two ways to push. The moved block contains #210's three lines, so the split commit is slightly different depending on whether #210 is in. I have the split and the follow-ups (the two guard commits and the finish-leaf gate) ready in two forms:

• on prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 alone;
• on prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 on top of qwen4_exp: warn when ROWINV runs without the in-pass checkpoint #210.

The three follow-up commits are the same patches in both (git range-diff shows them unchanged); only the split commit differs. With the follow-ups, the spark-model lib tests pass in both forms (1874 passed; the two extra are the new tests). Which form fits depends on whether #210 lands first.

Rebased onto current main 504063ad (new head eedd503c):

• no conflicts, and git range-diff shows all 14 patches unchanged;
• against main, the only file both sides touch is serve_load.rs, in separate hunks (against qwen4_exp: warn when ROWINV runs without the in-pass checkpoint #210, see above);
• fmt is clean, and clippy --workspace --tests has 0 lint warnings at this head (built without GPU kernels), with prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 on top of qwen4_exp: warn when ROWINV runs without the in-pass checkpoint #210, and on the tail-split commit on either base (these built with the GPU kernels compiled, where the only warning: lines are atlas-kernels build-script info output);
• tests: spark-runtime lib 448, spark-model lib 1872 and spark-server lib 123 pass at this head, and the same three counts pass with prefix cache: caching on = caching off at temperature 0 on SSM-hybrid models (fixes #158) #163 on top of qwen4_exp: warn when ROWINV runs without the in-pass checkpoint #210, built with the GPU kernels compiled (on an x86 box, so built but not run on a GPU); the file-size cap check passes.

Not yet run on GB10 hardware at this head.

Next test. Your 15-prompt probe on FP4-Local with main 504063ad + #163, --exact-prefix-cache in both arms (caching off vs on), and the r1short recipe (MTP 3 + EXACT_VERIFY, single GB10, so fix 7 is active). With G=1024, prompts of 1024 tokens or fewer never restore, so ATLAS_PREFILL_GRID=512 as a second arm would put more of the 144–5,440 range through a restore. The expectation is 15/15 in both arms. A third arm with MIDCHUNK_CKPT + PC_BRANCH on would show the gap above, and should reach 15/15 once the guard is in.

If it helps, I'm happy to push the guards and the prefill_b.rs split into #163 or open them as separate PRs, whichever you prefer, and to run the probe on our side if you don't have a GB10 free.

This comment was prepared with AI assistance (OpenClaw agent) and reviewed by me.

[maplepaladin73]

@AzeezIsh a question on order. This PR is still mergeable on 504063ad but isn't in the #220 stack. Would you rather it go in on its own against main, or wait until #198 lands and then rebase onto it? #198 touches the same prefix-cache/restore paths, and #218's fix changes what "exact on restore" means for FP4-Local, so landing it after #198 could make for a cleaner gate. Whichever you pick, I'll rebase and re-run the warm==cold checks on our dual-GB10 pair.

[AzeezIsh]

@maplepaladin73 #198 is merged (8bfe8507a), so please rebase onto main and re-run warm==cold there. That gives the cleaner gate you describe. Once it's green it can go in on its own against main; it doesn't need to wait for the SparkQwen ports, which are being restacked onto main now (#220 has the order). If the ports and #163 touch the same restore paths, whichever lands second rebases.

[maplepaladin73]

Rebased onto main 26e0b20bc (14 commits, no conflicts; new head c17135008), and re-ran warm==cold on our dual-GB10 pair (FP4-Local, TP=EP=2, shipped 16K chunks, ~19K-token padded turns, temperature 0):

• cache off: 3/3 rounds identical;
• cache on: replays identical to the cold round 12/12.
  cargo check --workspace --tests and fmt are clean. Pushed; ready for review against main.