feat: raise the Gemma 4 server context ceiling to 256K, with measured admission limits

[ulises-c]

Summary

• Raises the TurboFieldfare OpenAI server's --max-context allowlist from a 64K maximum to the Gemma 4 26B-A4B native 256K.
• Rejects --prefill off above 64K, which this widening would otherwise have made an unguarded out-of-memory combination.
• Adds prompt/decode timing diagnostics, a portable admission-ladder harness, a retrieval harness, and focused tests.
• Records what happens as context grows from 64K through 256K, and what those measurements do not establish.

What is and is not established

Established on the test machine: prompts up to 253,952 tokens are admitted, prefilled to completion, and memory stable, with no swap, OOM, rejection, or cancellation at any rung.

Not established: that the model still attends across that span. The ladder sends repeated filler and takes a one-token completion, so a level passes when the server accepts and prefills the prompt. Gemma 4 sets slidingWindow: 1024 and marks only 5 of 30 layers as full attention, so all long-range signal rests on those 5 layers, and a degraded long context would return HTTP 200 here exactly like a healthy one.

Retrieval is verified at 1,543 and 14,043 prompt tokens (needle recalled at depths 0.1/0.5/0.9). 14,043 clears the 1024-token window by more than 13x, so the full-attention path demonstrably carries information well past the local window — but nothing is measured between 57K and 254K, which is the range this cap adds. See blocking question 2.

Blocking questions

These need a decision or a measurement before merge. They are recorded here rather than resolved silently in code.

1. The timing columns predate the code this branch now ships. Every rung was measured before this branch merged main, which brought in #159 ("Speed up prefill on pre-Apple10 Macs"). That change rewrote prefill attention pipeline selection and the prefill command-buffer structure — precisely the path the timing columns measure. The rungs were also captured across more than one commit with the 64K baseline recorded last, so they are not a matched set even among themselves. Re-run the full ladder on a single post-merge binary, baseline first? The memory columns are structural and are expected to survive (see the KV cross-check below); the timing columns are not.

2. Retrieval is unmeasured across the range this PR adds. Scripts/context_retrieval.py is committed and validated, but the model was unavailable when this branch was finished. Run --max-context 262144 --filler-words 250000 --depths 0.1,0.5,0.9 before merge, or merge with the scope explicitly limited to admission and open a follow-up? A miss would be a more valuable finding than the timing table.

3. Should the ladder rungs be exposed to clients at all? The client examples now advertise 32K rather than 262144, because agent clients size their history to the advertised window and one 254K request cost ~32 minutes of prefill. The 96K/128K/192K/256K caps remain available deliberately, as diagnostics. Confirm that split is what you want.

Measured findings

Apple M5 Max MacBook Pro, 18 CPU cores, 32-core GPU, 36 GB unified memory, macOS 26.5.2 (25F84), Apple Swift 6.3.3. Uniform 8,192-token reserve at every rung. Timing columns are pre-#159 — see blocking question 1.

Context cap	Actual prompt	E2E server time	Prefill	Prefill tok/s	Peak footprint	Peak Metal	Minimum free memory	Swap	Result
64K baseline	57,344	236.158s	236.000s	242.983	3,441 MB	1,537 MB	76%	0 MB	completed
96K	90,112	464.218s	463.937s	194.233	4,082 MB	2,177 MB	76%	0 MB	completed
128K	122,880	701.019s	700.724s	175.361	4,714 MB	2,817 MB	75%	0 MB	completed
192K	188,416	1,291.620s	1,290.896s	145.958	6,032 MB	4,097 MB	68%	0 MB	completed
256K	253,952	1,920.565s	1,919.868s	132.276	7,320 MB	5,377 MB	62%	0 MB	completed

Prompt length grows 4.43x while end-to-end time grows 8.13x. Time per prompt token rises from 4.118 ms to 7.563 ms and prefill throughput falls from 242.983 to 132.276 tok/s, so cost is superlinear in length even though a least-squares fit over these five points is near-linear (R² = 0.996).

Retrieval

Context cap	Actual prompt	Depths	Recall
4K	1,543	0.1 / 0.5 / 0.9	3/3
16K	14,043	0.1 / 0.5 / 0.9	3/3
57K–254K	—	—	not measured

KV memory cross-check

FP16 KV allocation follows from KVCacheManager: with the ring enabled each of the 25 sliding-window layers holds slidingWindow + prefillChunkTokens = 1,152 tokens, while each of the 5 full-attention layers holds maxContext, doubled for K and V.

KV(ctx) = 25 * 1152 * 4096 * 2  +  5 * ctx * 2048 * 2

Context	Predicted KV	Observed peak Metal	Difference
64K	1,505 MB	1,537 MB	+32 MB
96K	2,145 MB	2,177 MB	+32 MB
128K	2,785 MB	2,817 MB	+32 MB
192K	4,065 MB	4,097 MB	+32 MB
256K	5,345 MB	5,377 MB	+32 MB

A constant 32 MB offset across a 4.43x span means the model accounts for the scaling term and the residual is fixed overhead. This is the structural reason the memory columns should survive re-measurement, and it is where the --prefill off guard comes from: without the ring every layer allocates at maxContext, giving ~55 GiB at 256K on a 36 GB machine.

Implementation

• ServerArguments accepts the 96K/128K/192K/256K caps, and rejects --prefill off above 64K with a message naming the resulting KV requirement. The 64K bound is the largest previously reachable context, so every previously valid invocation still resolves.
• ServerInference exposes prefill and decode durations; ServerLog reports pp, pp_tok_s, tg, tg_tok_s alongside prompt/cached/completion counts.
• Scripts/context_ladder.py takes --model/--server/--port/--levels/--out, defaults to this checkout, validates inputs up front, records the source commit in every result, and merges a single rung into the existing aggregate instead of overwriting it.
• Scripts/context_retrieval.py runs needle-in-a-haystack with a real multi-token completion, sweeping depth because sliding-window degradation is position dependent. Exits non-zero on a miss.
• docs/experiments/summaries/10-long-context.md holds the measurements and their caveats, since benchmark-results/ is gitignored and the raw artifacts cannot travel with the PR.

Testing

• swift build -c release: clean, only pre-existing warnings.
• Full Scripts/test.sh: 1,262 tests in 195 suites passed (152s). The 149 tokenizer .authorizationRequired failures reported earlier were environmental and do not reproduce.
• ServerArgumentTests and ServerLogTests: 19 passed, including 13 new argument cases and 3 new log cases.
• New argument coverage asserts rejection at each ladder rung under --prefill off, plus matched controls proving pre-ladder --prefill off and ladder --prefill on still resolve.
• New log coverage exercises the pp_tok_s cached-token subtraction, which every ladder rung left untested at cached=0 despite prompt reuse being the server default.
• Guard verified end to end: --max-context 262144 --prefill off is refused with the 55 GiB message; --prefill on passes validation.
• git diff --check: clean.

Scope and limitations

This validates native 256K admission, full prefill completion, timing behavior, and memory stability on the stated machine. It does not establish long-context answer quality, semantic robustness, or useful multi-token decode throughput — the one-token completion makes tg/tg_tok_s unusable as general decode measurements. Repetitive filler is also the easy case for retrieval, so recall figures are an upper bound relative to real agent history.

Related issue

• Upstream tracking issue: [Feature]: Extend gemma-4-26b-a4b-it from 64K context length to 256K (model native max) #157

[ulises-c]

Verdict: Changes Requested — 3 blocking, 5 warnings.
(Posted as a COMMENT review: GitHub does not allow requesting changes on one's own PR.)

Adversarial review — not ready to merge

Reviewed at merged HEAD 40428a5. Rebuilt release (Build complete!), reproduced both cited suites (ServerArgumentTests 11 passed, ServerLogTests 2 passed), independently re-derived the KV memory model from KVCacheManager.swift, and ran a needle-in-a-haystack retrieval probe.

The engineering is clean and Scope and limitations is honestly written. But the title claims a capability the evidence does not establish, and the headline numbers now predate the code this PR ships.

Blocking

1. The ladder proves admission, not comprehension.
ArchConfig.gemma4_26B_A4B sets slidingWindow: 1024 with fullAttentionLayerMask = layers 5/11/17/23/29. 25 of 30 layers see only 1024 tokens; 5 full-attention layers carry all long-range signal. A "word " * N prompt with max_completion_tokens=1 cannot distinguish a working 256K context from one where those 5 layers degrade into noise — both return HTTP 200 finish=length.

I verified retrieval genuinely works below the ladder with a needle probe:

Context	Prompt tokens	Depths	Result
4K	1,543	0.1 / 0.5 / 0.9	3/3 recalled
16K	14,043	0.1 / 0.5 / 0.9	3/3 recalled
57K–254K	—	—	untested

14K already clears the window by 13x, so the full-attention path works. The untested range is exactly the range this PR adds.

2. Every timing number predates the merged code.
Rungs were measured 08-24/08-25 from c4f9442/4b3bd87. 40428a5 then merged #159 "Speed up prefill on pre-Apple10 Macs", which rewrote PrefillAttention.swift pipeline selection (new layerKind == .full + windowNeverClips guards) and RealForwardRunner prefill command buffers. The table's pp, pp_tok_s, the 242.983 -> 132.276 decline and the R^2=0.996 fit all describe a different prefill implementation. #159's own message states the prior TensorOps path "produced wrong output" for clipping windows.

3. The 64K baseline is not a matched control.
Run order: 96K/128K/192K (08-24 16:00-16:33) -> 256K (08-25 13:18) -> 64K last (08-25 14:55), spanning at least two source commits. A baseline run after the experiment on a different build is not a baseline.

Warnings

4. --prefill off --max-context 262144 is an unguarded OOM. Derived from KVCacheManager.swift:79-101, KV = 25 x 1152 x 4096 x 2 + 5 x ctx x 2048 x 2. This predicts every observed Metal peak to a constant +32 MB across a 4.4x span:

Context	Predicted	Observed	delta
64K	1,505 MB	1,537 MB	+32
96K	2,145 MB	2,177 MB	+32
128K	2,785 MB	2,817 MB	+32
192K	4,065 MB	4,097 MB	+32
256K	5,345 MB	5,377 MB	+32

256K only fits because the FP16 ring caps SWA layers at 1,152 tokens each. --prefill off disables that ring -> all 30 layers allocate at maxContext -> ~55 GB at 256K on a 36 GB machine, accepted with zero validation.

5. The harness cannot be run by any reviewer. MODEL is hardcoded to a different checkout, PORT is fixed, and benchmark-results/ is gitignored — so none of the cited raw artifacts are in the PR.

6. "Server-side context-capability validation" is overstated. ServerArguments only widens an integer allowlist; nothing validates the model supports 256K (no RoPE-scaling or manifest check).

7. Prompt cache untested at long context. Every rung reports cached=0, yet single-prefix is the default and is the real agent workload.

8. Docs recommend 262144 as the routine client budget — measured cost ~32 min of prefill per request. A routine budget should sit materially below the measured edge.

Good

• Memory instrumentation is well designed — it is what let me validate the KV model independently.
• Zero swap and no OOM across five rungs is a real, credible result.
• completedMessage extracted as a pure function for testability is the right call.
• Uniform 8,192-token reserve across rungs is a correct control.

To make this mergeable

1. Run needle retrieval at 64K/128K/256K, depths 0.1/0.5/0.9 — this is the missing evidence.
2. Re-run the ladder on merged HEAD, all rungs same binary, baseline first.
3. Guard --prefill off above ~64K, with a test.
4. Retitle toward what is proven, or hold the title until (1) lands.
5. Make the harness portable; commit or un-ignore the artifacts.
6. Walk doc examples back to a conservative budget; fix the CLI validation claim; drop unused os/select imports.

---

Adversarial review by Hermes Agent. Build/test/memory-model claims above were reproduced locally; the 57K+ retrieval gap is stated as untested rather than assumed.

[ulises-c]

Blocking (harness portability). MODEL is hardcoded to /Users/ulises/github/turbo-fieldfare/scratch/gemma4.gturbo — a different checkout from the one this PR lives in. No reviewer can run this, and it silently couples the harness to one machine's layout.

Combined with benchmark-results/ being gitignored (.gitignore:43), a reviewer gets a script that cannot run and cannot see the data it produced.

Suggest argparse with --model / --port, defaulting to ROOT / "scratch/gemma4.gturbo" and falling back to an env var.

[ulises-c]

Blocking (this is the core issue). max_completion_tokens=1 combined with the "word " * words prompt at line 134 means a level passes if the server admits and prefills the prompt — never that the model can use it.

Gemma 4 runs slidingWindow: 1024 on 25 of 30 layers; only layers 5/11/17/23/29 are full-attention. So a 254K context where those 5 layers degrade into noise returns HTTP 200 finish=length — identical to a healthy run.

I confirmed retrieval works at 1,543 and 14,043 prompt tokens (needle recalled at depths 0.1/0.5/0.9), so the mechanism is sound below the ladder. The untested band is 57K-254K, which is precisely what this PR adds.

Suggest a second pass: plant a fact at fractional depth in the filler and ask for it back with a real multi-token completion.

[ulises-c]

Warning (unguarded OOM introduced by this change). This widens the allowlist to 262144, but --prefill off remains independently settable.

From KVCacheManager.swift:79-101, the ring only caps SWA layers when fp16RingEnabled; with prefill off, all 30 layers allocate at maxContext:

• ring on @ 256K: ~5.2 GB (matches the observed 5,377 MB peak to +32 MB)
• ring off @ 256K: ~55 GB — unallocatable on the 36 GB test machine

Before this PR the cap was 65536, which bounded the damage. Now the combination is reachable and accepted with no validation. Suggest rejecting --prefill off above ~64K in validate(), with a test.

[ulises-c]

Fine as a formatting unit test, but worth noting it asserts values no real run produces: decodeSeconds: 0.250 -> tg_tok_s=4.000, whereas every actual ladder rung logs tg=0.001s with tg_tok_s of 785-1094.

The PR describes this as covering "prefill/decode timing" behavior; it covers string formatting of hypothetical values. The pp_tok_s cached-token subtraction path (ServerLog.swift:32-35) is never exercised with cached > 0 — and every ladder rung reports cached=0, so prompt-cache behavior at long context is untested in both the suite and the benchmark.

[ulises-c]

Warning. This flips the primary launch example from 16384 to 262144, and lines 115/153 do the same for the OpenCode and Pi client configs — making 256K the recommended default for every reader.

Measured cost of one such request on the test machine is 1,920s (~32 min) of prefill. Per the project's own methodology, a routine budget should sit materially below the measured edge, not at it.

Suggest keeping the example at a practical value and documenting 262144 as the tested maximum with its latency cost stated inline.

[ulises-c]

Inaccurate. This says CLI supports "4K through 256K, including 96K and 192K ladder points", implying enumerated CLI validation.

Sources/TurboFieldfareCLI/Args.swift:241-246 accepts any positive integer for --max-context — there is no allowlist on the CLI at all. Only the server enumerates. As written this documents validation that does not exist, in the same row that describes server behavior which does.

[ulises-c]

Review items addressed

Pushed five commits (80c1c82..4dd7dc1) responding to the adversarial review above. Summary of what changed and what deliberately did not.

Fixed

#	Item	Resolution
4	--prefill off unguarded OOM	80c1c82 — rejected above 64K during argument resolution, with the KV requirement named in the message. 13 test cases incl. matched controls.
5	Harness unrunnable	a1acfc5 — --model/--server/--port/--levels/--out, defaults to this checkout, validates binary and rung labels, records source commit, merges single rungs into the aggregate. Unused os/select dropped.
6	"Capability validation" overstated	f025a02 — docs now state plainly it is a supported-value list, not a probe of the weights.
7	Prompt cache untested	a74d421 — three cases covering partial hit, fully cached, and cached-exceeds-prompt clamp.
8	Doc budget recommendation	f025a02 — launch example back to 16K, client examples to 32K, with the ~32-minute prefill cost stated inline.
10	CLI validation claim	f025a02 — row now describes each binary's real behavior; the CLI accepts any positive integer.
—	Artifacts absent from PR	4dd7dc1 — measurements moved into docs/experiments/summaries/10-long-context.md, since benchmark-results/ is gitignored.

Partially addressed

#1 (admission vs comprehension) — Scripts/context_retrieval.py is committed, validated, and documented. It is not yet run across 57K–254K: the model directory became unavailable mid-session and I did not reinstall it. Now blocking question 2 in the description, with the untested range stated explicitly rather than implied.

#2 / #3 (stale timings, unmatched baseline) — cannot be fixed by editing; they need a re-run on one post-merge binary. Now blocking question 1, and the table and experiment note both carry a pre-#159 marker. The 64K baseline joined LEVEL_CONTEXTS so a matched re-run is one command.

Note on the KV model

The --prefill off guard derives its threshold from ArchConfig at runtime rather than a hardcoded constant. The predicted figure it reports (55 GiB at 256K) is the same formula that matches observed peak Metal allocation at all five rungs to a constant 32 MB — so the guard and the memory table are backed by the same verified model.

Verification

Full Scripts/test.sh: 1,262 tests in 195 suites passed (152s). The 149 tokenizer .authorizationRequired failures previously reported did not reproduce — that was environmental. swift build -c release clean; git diff --check clean.

---

The title has been changed from "extend Gemma 4 server context to native 256K" to one that claims the measured result rather than the aspirational one.