feat: reach native 256K on every supported model, with an architecture-aware KV model - #1
Merged
Merged
Conversation
ulises-c
force-pushed
the
docs/record-256k-context-ladder
branch
from
August 25, 2026 21:46
e4ae6df to
8dbbc60
Compare
ulises-c
force-pushed
the
docs/record-256k-context-ladder
branch
from
August 25, 2026 22:02
829d6bb to
508bf94
Compare
ulises-c
changed the title
ulises-c
marked this pull request as ready for review
August 25, 2026 22:09
Widening --max-context to the Gemma 4 ladder rungs made an unguarded out-of-memory combination reachable. Chunked prefill is what enables the FP16 sliding-window ring in KVCacheManager, and the ring is the only reason a long context fits: it caps each of the 25 sliding-window layers at slidingWindow + chunkTokens tokens instead of maxContext. With --prefill off the ring is disabled and every layer allocates KV at the full context. Measured against the five recorded ladder rungs, the ring-backed footprint matches observed peak Metal allocation to a constant 32 MB offset; without the ring the same 256K cap needs about 55 GiB of KV alone, which no supported machine can satisfy. Reject the combination during argument resolution, where it produces an actionable message, rather than at model load. The bound is 65536, the largest context reachable before the ladder rungs were added, so every previously valid invocation still resolves. Adds 13 cases: rejection at each ladder rung, and matched controls proving both pre-ladder --prefill off and ladder --prefill on still pass.
completedMessage divides by prompt tokens minus cached tokens, but every context-ladder rung ran against an empty cache and reported cached=0, so the subtraction was never exercised by either the suite or the benchmark. Prompt reuse is the server default, which makes a partial cache hit the common case rather than an edge case. Adds three cases: a partial hit where the rate must reflect only the computed tokens, a fully cached prompt that must read as zero rather than dividing by zero elapsed time, and a cached count exceeding the prompt, which the existing max(..., 0) clamps but nothing asserted.
The ladder hardcoded a model path pointing at a different checkout, a fixed port, and a fixed rung list, so no reviewer could run it and a single rung could not be re-measured without repeating the hours-long rungs beside it. It now takes --model/--server/--port/--levels/--out, defaults the model to this checkout's scratch directory, validates the binary and rung labels up front, records the source commit in every result, and merges a partial run into the existing aggregate instead of overwriting it. The 64K baseline joins LEVEL_CONTEXTS so it is produced by the same protocol as the rungs it is compared against. Drops the unused os and select imports. Adds Scripts/context_retrieval.py for the question the ladder cannot answer. A one-token completion on repeated filler proves admission, not comprehension: Gemma 4 runs sliding-window attention on 25 of 30 layers, so long-range signal rests on 5 full-attention layers and a degraded long context still returns HTTP 200. The new harness plants a distinctive fact at a fractional depth, asks for it back with a real multi-token completion, and sweeps depth because sliding-window degradation is position dependent. It exits non-zero on a miss so a sweep can be scripted, and its docstring states that repetitive filler makes recall an upper bound rather than a guarantee.
The launch example and both client configs advertised 262144, making a 256K window the recommended default for every reader. One such request measured about 32 minutes of prefill on the test machine, and agent clients size their history to the advertised window, so that setting commits every request to that cost. The examples move to practical caps and the reasoning is stated inline. Corrects three claims. Context prose now records the --prefill on requirement above 64K and why the sliding-window ring makes it necessary. It states plainly that the capability check is a supported-value list rather than a probe of the weights, and that the ladder measures admission rather than comprehension, pointing at the retrieval harness for the latter. The runtime-controls row claimed the CLI validates an enumerated set including the ladder points. Args.swift accepts any positive integer for --max-context; only the server enumerates. The row now describes each binary's real behavior.
The pull request cited raw artifacts under benchmark-results/, which is gitignored, so none of the data backing the table was actually in the change. This moves the measurements into a durable experiment note alongside the conditions needed to read them correctly. Records both questions separately. The admission ladder is reported with its recorded run, and the note states plainly that the timing columns predate the merge of drumih#159, which rewrote prefill attention selection and the prefill command buffers, so those columns describe a superseded implementation and are marked for re-measurement. It also notes the rungs were captured across more than one commit with the baseline last, so they are not a matched set. Adds the KV memory model derived from KVCacheManager and shows it predicts observed peak Metal allocation at all five rungs to a constant 32 MB offset. That is the structural reason the memory columns should survive re-measurement, and the same formula gives the ~55 GiB figure behind the new --prefill guard. Records the retrieval results that do exist, at 1,543 and 14,043 prompt tokens, and states that nothing is measured between 57K and 254K, which is the range the 256K cap adds.
The Gemma 4 KV formula was inlined at three sites: the server's --prefill off guard, the Mac app's context menu, and the long-context experiment notes. Each computed a Gemma-shaped number for any model, which is wrong for Qwen 3.6 -- its 30 gated-DeltaNet layers hold a fixed recurrent state instead of per-token K/V rows, so the estimate overstated its unringed 256K cost by 11x (55 GiB predicted vs 5 GiB actual). Replace all three with ArchConfig.kvFootprint, which mirrors KVCacheManager.init exactly and is cross-checked against the real allocator by test at 4K/16K/64K for both families. The --prefill off bound becomes a 16 GiB KV budget rather than a 65536 context constant. The budget reproduces the previous Gemma behavior exactly (64K = 13.75 GiB allowed, 96K = 20.62 GiB rejected) while letting Qwen reach every ladder rung, which it can serve unringed. The server reads the family from the manifest before resolving, and falls back to Gemma -- the stricter bound -- when it cannot. The app's context menu gains the 96K/128K/192K/256K options and computes its labels instead of carrying hand-rounded literals. 1,339 tests in 203 suites pass.
Scripts/memory_matrix.py projects resident memory from the same KV model the runtime uses, with the non-KV terms calibrated against the five measured Gemma rungs. --validate re-checks the projection and fails if it ever under-predicts. Two corrections that the obvious formulation gets wrong: The installed pack is not resident. Routed experts stream through a fixed slot cache, so Gemma holds 1,322 MiB + 51 MiB at 16 slots, not 13.31 GiB. Assuming otherwise overstates it by ~12 GiB and wrongly reports that nothing fits in 16 GB. KV is allocated at the context cap, not the prompt length. Measured on the M5 Max: --max-context 262144 with a 14-token prompt reached 7,268 MB, within 52 MB of the 253,952-token rung. --max-context is the memory dial. Result: every Gemma rung through 256K fits on both a 16 GB M4 and a 36 GB M5 Max. Memory is not the long-context limit; prefill time is. Qwen's KV column is exact but its total is left blank on purpose -- the pack is not installed here, so its resident split and runtime overhead are unmeasured.
The --prefill off rejection is a KV budget against the installed architecture, not a fixed context limit, and Qwen 3.6 is unaffected by the ring. Also state that KV is allocated at the cap rather than the prompt length, and link the per-device memory table.
ulises-c
changed the title
The CLI, the server, and the app each decided independently whether a --max-context was legal, and they disagreed. The CLI accepted any positive integer, so --max-context 1000000 was refused by the server and accepted by the CLI, failing later inside the allocator. The server's parser carried a fourth hardcoded copy of the ladder and rejected an out-of-range value with '--max-context is not supported', naming neither the value nor the legal set. ContextAdmission holds the rule once: the native maximum, the ladder, and the 16 GiB unchunked-KV budget, all derived from the loaded ArchConfig. Every surface calls it and renders the same rejection in its own vocabulary. A nil family means 'not yet identified', which admits whatever any supported model could run. Argument parsing happens before the manifest is read, so assuming Gemma there would reject Qwen-legal commands before looking at the model; the strict per-family check runs afterwards in the run path. The CLI now reads the family from the manifest the way the server already did. Tests pin the surfaces to each other through their real parsers, including a matched control at 256K prefill off where Qwen is admissible and Gemma is not. The app menu test now derives from ContextAdmission.ladder instead of repeating it, so a rung added to one and not the other fails.
The recorded ladder measured admission but probed recall only at 1,543 and 14,043 tokens, so 57K-254K was unmeasured -- and a degraded long context returns HTTP 200 exactly like a healthy one. The recorded prefill columns also predate the merge of drumih#159, which rewrote the path they measure, and were taken across several commits with the 64K baseline last. One sweep answers both: every probe runs on a single binary and records pp_seconds and pp_tokens_per_second alongside the needle result, so the matched timing set falls out of the recall run instead of costing a second full prefill of the ladder. Each rung starts and stops its own server, and results append to JSONL after every probe so an interrupted sweep keeps what it measured.
Retrieval is measured across the ladder for the first time: 15/15 at depths 0.1/0.5/0.9 from 57,043 to 253,143 tokens, every probe with cached_tokens 0. Depth 0.5 at 256K puts the needle ~126,000 tokens from either end, outside the 1,024-token window of all 25 sliding-window layers, so it is reachable only through the 5 full-attention layers -- and it was recovered verbatim. The cap extends what the model can retrieve, not just what it will accept. Filler is repetitive by construction, so this is an upper bound on recall and evidence against catastrophic sliding-window failure, not a guarantee for arbitrary content. Recorded as such. The prefill columns are re-measured on one binary, three probes per rung, replacing numbers taken across several commits before drumih#159. Every rung is slower, median -6.5%. That is NOT called a regression: the old column is not an internally matched set, drumih#159 targeted pre-Apple10 Macs while this host is an M5 Max, and no A/B against the pre-drumih#159 commit has been run. The new table is a matched baseline for future comparison; the old one is kept as a dated lab record. Raw sweep output is committed under docs/experiments/data/ since benchmark-results/ is gitignored.
GTurboLayoutValidator kept its own hardcoded 16 MiB cap after the verifier and the runtime both moved to 64 MiB for Qwen 3.6, whose 40 layers x 256 experts produce a 22,493,846-byte layout.json. Installing Qwen downloaded all 18.8 GiB, verified every weight file, and then failed at the last step: install failed: install state at .../packed_experts/layout.json is corrupt: size 22493846 exceeds 16777216-byte cap The validator now uses VerifiedInstallTool.layoutMaxBytes, the same constant the verifier applies. Nothing was re-downloaded; --resume finished from the preserved .partial. The verifier had cap tests and the validator had none, which is why the drift went unnoticed. LayoutBoundParityTests now asserts the installer and runtime bounds are the same number, that a Qwen-sized layout fits both, and that the layout bound stays above the generic metadata bound so the three cannot be collapsed into one.
Installed scratch/qwen36.gturbo and measured it at a 256K cap: a 17-token prompt reaches 6,641 MB phys_footprint against a projected 8,347 MB. The projection stays conservative, as required, but by 1,706 MB (20%) rather than Gemma's 0-46 MB -- the borrowed overhead constant and on-demand expert slots account for the slack. --validate now fails if it ever inverts. Also confirmed on the real binary that Qwen starts at a 256K cap with --prefill off, which Gemma is refused at identical arguments, and that Qwen allocates KV at the cap exactly as Gemma does. Records the measurement instrument. phys_footprint and RSS differ by 6x on this workload (6,641 MB vs 1,075 MB) because the Metal heaps backing KV are not counted in RSS; an RSS reading against these tables shows a large fake overestimate.
context_sweep.py hardcoded scratch/gemma4.gturbo, so the Qwen ladder could not run through it. Adds --model and writes per-model JSONL (sweep-<pack>.jsonl) so a Gemma sweep and a Qwen sweep cannot land in one file and be averaged across two architectures by a later aggregate.
context_retrieval.py hardcoded --model-id gemma-4-26b-a4b-it. The server does validate that field, so every Qwen probe returned HTTP 404 model_not_found in 8ms and was recorded as found=false. The sweep summary rendered that as 'RECALL 0/3' at every rung -- a total-retrieval-failure shape for what was really a wrong-name error, and the kind of result that would have been alarming to read as a finding. The id now defaults to whatever the server prints in its ready line, so it follows the pack. A 404 aborts the run with an explicit naming error instead of emitting rows that are indistinguishable from genuine recall misses. Verified both families at 16K: Qwen 1/1 (id qwen3.6-35b-a3b, auto-detected), Gemma 1/1 (id gemma-4-26b-a4b-it, same value that was previously hardcoded). Also stops context_sweep.py crashing in its own summary when a rung produced no usable rows -- it formatted None with ',' and lost the whole table.
Both families were fully wired for install -- descriptor, repo, revision, sizes, and a qwen36.gturbo directory -- but nothing in the app could select one. AppModelInstallDescriptor.selected read only TURBO_FIELDFARE_MODEL or a defaults key, so a GUI user could reach Gemma and nothing else. Adds a picker in two places: the Model menu, and the install screen itself, where the decision actually gets made. In the menu alone the choice would be invisible at the one moment it matters -- before committing to a 14.6 GB or 19.5 GB transfer. Selecting a family persists the choice, swaps the installer's descriptor, and moves the model directory to that family's pack, reusing setModelURL for the teardown (cancel loads and transfers, clear staged images, unload the runtime). Switching is refused while a transfer is in flight, which would otherwise strand a partial download owned by a descriptor no longer selected. The environment variable still wins on read, so a launch with TURBO_FIELDFARE_MODEL set is not silently overridden by a click. selectableFamilies sits on the descriptor rather than AppModel so it is reachable without main-actor isolation. Backfills the selection path, which had no coverage at all: nothing referenced TURBO_FIELDFARE_MODEL, .selected, or installDirectoryName. Eight tests cover the descriptor/family round-trip, per-family install directories, Gemma-only vision companions, the switch, the no-op re-select, and the in-flight refusal. 1,358 tests pass.
The picker went into the Model menu and the install screen, and a user with Gemma already installed saw neither: the menu bar is not where anyone looks for a model setting, and the install screen only renders when a model is missing. The reported symptom was exactly that -- no dropdown anywhere, so only Gemma is reachable. The inspector is a permanent 320pt pane with a Model section already at the top of it, next to Context and Slots. That is where a model setting belongs, so the picker goes there, styled to match the pickers beside it. The menu and install-screen entries stay; this adds the one placement that is visible during normal use.
The .railguard/ session-state files are Hermes agent tooling, not project content; they were committed by accident. Untrack them and add .railguard/ and .hermes/ to .gitignore so they stay out.
context_session.py grows context in --step-tokens increments to a target, replaying each turn's answer so the server KV cache keys on it -- this exposes the real per-turn prefill cost (new tokens only) that the cold ladder hides. Needle-in-haystack recall is checked at --checkpoints. The final step is clamped in token-space against --max-context (approx_depth is real tokens but step is a filler WORD count; filler_block + chat template emit more tokens than words) so the last turn can't overshoot the cap and draw an HTTP 400, as it did at the 256K rung on the first run. ladder_table.py renders the cold-ladder results.json as a markdown table. context_sweep.py takes the model id from the server, not a Gemma constant.
Growing context in ~16K steps with a warm KV cache reprefills only the new tokens, but the per-turn cost of that constant delta scales with session depth (13.6 -> 106.4 ms/new-token from 33K to 246K), so reaching 246K in steps cost 4.03 h -- essentially the cold 256K prefill. Incremental context amortizes the O(n^2) work into per-turn responses; it does not reduce it. Needle recall HIT at 64K/96K/128K/192K; memory flat from turn 1 because the full max-context KV ring is pre-allocated. Notes the missing 256K agentic checkpoint (a now-fixed harness clamp off-by-one) and why it is not re-run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises the server context ceiling to 262,144 — the native window of both supported model families — and makes the memory model that governs it architecture-aware instead of Gemma-hardcoded.
Retargeted at
fork-main. Upstream drumih#156 is the same branch; it is left open as a signal but is no longer the merge target.What changed since the adversarial review
The review at
40428a5raised 3 blocking items and 5 warnings. Commits80c1c82..4dd7dc1cleared all 5 warnings. This round addresses what merging into a fork with a second model family exposed.The Gemma formula was inlined in three places
ServerArguments.unchunkedKVGibibytes,AppContextLengthOption.fp16KVBytes, and the experiment notes each carried their own copy of the Gemma 4 KV arithmetic. Every one of them computed a Gemma-shaped number for any model. After PR #3 that is wrong for Qwen 3.6: its 30 gated-DeltaNet layers hold a fixed recurrent state rather than per-token K/V rows, so the old formula overstated its unringed 256K cost by 11× (55 GiB predicted vs 5 GiB actual).All three now call
ArchConfig.kvFootprint, which mirrorsKVCacheManager.initexactly and is cross-checked against the real allocator by test at 4K/16K/64K for both families. A change to the allocator that is not mirrored in the model now fails a test instead of silently producing wrong estimates.The
--prefill offguard is a KV budget, not a context constantThe old bound was
maxContext <= 65_536. That is a Gemma fact wearing a general name. It is now a 16 GiB unringed-KV budget evaluated against the installed architecture:--prefill offat 256KThe budget reproduces the previous Gemma behavior exactly — 64K allowed, 96K and above rejected — while letting Qwen reach every rung, which it can genuinely serve. The server reads the family from the manifest before resolving and falls back to Gemma, the stricter bound, when it cannot.
Verified end-to-end on the real binary:
Two measurement corrections
The installed pack is not resident. Routed experts stream through a fixed slot cache: Gemma holds 1,322 MiB + 51 MiB at 16 slots, not its 13.31 GiB installed size. The naive assumption overstates Gemma by ~12 GiB and wrongly concludes nothing fits in 16 GB.
KV is allocated at the context cap, not the prompt length. Measured on the M5 Max: a server at
--max-context 262144sent a 14-token prompt reached 7,268 MB — within 52 MB of the 253,952-token rung.--max-contextis the memory dial, and this is now stated in the docs.Memory expectations
Scripts/memory_matrix.pyprojects resident memory per device/model/context and self-validates against every measured point (--validatefails if it ever under-predicts). Current margin: 0 to +46 MB across the five Gemma rungs.Gemma 4, 16 expert slots, chunked prefill:
Every rung fits on both machines, including 16 GB at full 256K. Memory is not the long-context limit here — prefill time is (~32 min for 253,952 tokens on the M5 Max).
Qwen's KV column is exact and ~160 MB below Gemma's at every rung despite the larger model. Its total is deliberately left blank: the pack is not installed in this checkout, so its resident split and runtime overhead are unmeasured, and filling those cells with Gemma's constants would be a guess wearing a measurement's clothes.
Agentic growth measured (Qwen 3.6)
A real session grows context in small steps and reuses the KV cache across turns.
Scripts/context_session.pytests this directly on Qwen 3.6 35B-A3B: it grows a session in ~16K-token steps toward the 262,144 cap, replaying each turn's answer so the server cache keys on it, and re-asks a turn-1 needle at 64K/96K/128K/192K.The cache works and buys nothing on total cost. Each turn reprefills only its ~16.4K new tokens, but that constant delta gets ~8× more expensive with depth because attention is causal:
Summed over 15 turns, reaching 246K in steps cost 4.03 h — essentially the cold 256K prefill. Incremental context amortizes the O(n²) attention work into per-turn responses; it does not reduce it. Memory stayed dead-flat (61–63% free, ~5.2 GB Metal) from turn 1, because the full
--max-contextKV ring is pre-allocated up front.Recall held: needle HIT at 64K, 96K, 128K, and 192K (4/4). The 256K agentic checkpoint is absent — turn 16 projected 424 tokens over the cap and the server correctly returned HTTP 400; that harness off-by-one (a filler-word vs. token miscount) is fixed by a token-space clamp in
dbcc532. Cold 256K admission + recall already proves the capability, so the single agentic point is not re-run.Validation
swift build -c releaseclean.Scripts/test.sh: 1,339 tests in 203 suites passed (up from 1,262; 77 new).python3 Scripts/memory_matrix.py --validatepasses.git diff --checkclean.Known limitations
These are disclosed rather than resolved, and are why this is scoped to admission and memory.
c4f9442/4b3bd87before Speed up prefill on pre-Apple10 Macs drumih/turbo-fieldfare#159 rewrote prefill attention. Table carries a pre-Speed up prefill on pre-Apple10 Macs drumih/turbo-fieldfare#159 marker; a matched re-run is one command (64Kis now inLEVEL_CONTEXTS, baseline first).