Repository navigation
Conversation
A verify step's rows past its lane's verify rows (draft_verify_rows, which a MoE target's verify budget chooses) are never retained or read, and the state the step writes is the one before its rows. The scan now runs a lane's step rows up to its verify rows and leaves the others' recurrent output zero. State, tape and the verified rows' outputs are bitwise the every-row ones: gdn-decode compares partial and every-row steps over fp32 and bf16 states, with the out-projection table. Dense targets verify every row and keep today's work. Qwen3.6-35B-A3B, 1K prompt, verify rows pinned (same tokens both sides), ms per accepted token at B1 / B4: 4 rows: M5 Max 32c -1.2 / -1.3%, M5 Pro 16c -1.2 / -1.5%, M6 12c -0.8 / -0.9% 8 rows: within 0.2% on all three. Under the verify budget (1K and 32K, B1 and B4, 4 + 4 runs) the budget's rows do not move (mean verify rows within 0.15) and a cycle at its rows costs 0.05-1.2% less on the M5s; ms per accepted token changes within the runs' spread (-1.7 to +0.7% on the M5 Pro and M5 Max except +1.9% and -5.9% on noisy 4-lane windows). Qwen3.8-27B (every row) and Flash-Next (streamed) unchanged within noise.
One unsplit decode lane runs Apple10's 16-row staged tile below 4.5 bits per weight, as before, and from 4.5 bits where its grid runs in as many waves of the 16-row tile as of the 8-row tile, two at most. The 16-row tile computes the 8-row tile's bits, a wave of most formats in less time, but holds a threadgroup fewer per core (5120-input Q4_K and MLX 4-bit step past 9 threadgroups per core on it, past 10 on 8 rows, 16-core M5 Pro and 32-core M5 Max): a 10240 x 5120 Q4_K, 10 per core on 16 cores, takes 17.5-18.7% more on it. Two is the most waves measured end to end (the 27B's 4.5-bit gate/up passes, 17 per core on 16 cores: 7.1-8.2% less time). Tied grids of three to nine waves keep 8 rows: kernel sweeps alone measured one, 24 per core, where Q4_K and Q6_K took 1.4-2.8% less time on 16 rows on macOS 27 and Q4_K 1.0% more on a 20-core M5 Pro on macOS 26.5. The 27B's 4.5-bit gate/up tensors take it on 16, 20 and 32 cores, and a pair follows its tensors' own plans whatever its passes: two passes on 16 and 32 cores, one on 20, where the 16-row pair takes 4.1-4.5% less time. Its fused GDN inputs (16640 columns, two waves of either tile on 16 cores) take it too, and its heads, which run more waves of it, keep 8 rows. Of the measured shapes it moves, the Q5_K 17408 x 5120 passes run 2.6-4.4% slower on a 16-core M5 Pro. The M6's 16-row tile takes as long a wave as its 8-row tile from 4.5 bits, so family 11 and later become their own class, Apple11, whose row keeps the 4.5-bit threshold alone (kApple11Projection). Apple9 and Apple11 plans are unchanged; kernel-selection gains the 10- and 20-core Apple10. Decode cycle, one lane (decode-profile, ABBA, 6 runs a side): the 27B UD-Q4_K_M 87.97 -> 86.62 ms (-1.5%) on a 16-core M5 Pro and 76.12 -> 75.26 ms (-1.1%) on a 20-core M5 Pro; the same 27B with its Q4_K tensors stored as Q5_K blocks of the same values 91.53 -> 90.15 ms (-1.5%) on the 16-core M5 Pro, its Q5_K gate/up passes 0.38 ms slower and its fused inputs 1.12 ms faster. Two to four lanes and the 35B unchanged within noise. Based on incoai#653 by linson007. Co-authored-by: Linson <3253044+linson007@users.noreply.github.com>
On Apple10 and later the neural accelerator multiplies INT8 at twice its bf16 rate, so each dense split kernel over INT8 pages gains a q8q entry that quantizes the tile's queries once per threadgroup (absmax/127 per fused row, by the row's softmax lanes) and runs QK as INT8 x INT8 -> int32; the row's scale folds into the score scale. The codes live in the threadgroup's own partials slot, written only after its last page. That quantization is paid once per threadgroup, so it wins only where a split reads enough pages. A plan over INT8 pages takes the q8q entry where its longest tile's or lane's splits read at least AttentionCalibration::int8QueriesFromPages pages each, a law-table field: 8 on Apple10 and Apple11 (M5, M6), kNever on Apple9, whose matrix operations share the FP32 pipe (a prototype with INT8 queries everywhere took 8-12% longer on an M4); Apple9 GPUs of more than 10 cores, which shared Apple10's attention row, get one of their own. Verify below ~7K tokens, a 513-key window's verify and chunks of 64 rows or fewer over short or windowed histories keep bf16 queries; chunks of 225 rows or more, and any tile or lane past the threshold, take INT8. A verify's lanes share the choice: a short lane batched with a long one scores INT8 queries, though its split partition stays its own. PagedAttention::prefillPlan and verifyPlan take the device's AttentionCalibration before the variant, with no default. ABBA, 3 rounds. Against main (the commits before it change no attention plan), prefill attention takes 9-24% less time in full-attention layers on M5s and an M6; INT8 queries in splits of 1-5 pages ran up to 31% slower on the M6 and in splits of 9 faster, which sets the threshold. Against main with the commits before it at the time, a 35B prompt's first token comes 4.4% sooner at 32K and 6.6% at 64K on an M5 Max, whose verify split takes 3-5% less at B1-B4. Teacher-forced KL against bf16 KV, pooled over windows of two documents, is 1.01x today's INT8 KV's for the 35B and 1.37x for the 27B (six windows each, none above 1.96x) and 1.00x for Flint (16 windows, 95% interval 0.82-1.22x). paged-attention-plan asserts the rule: 7 and 8 pages a split either side of the threshold, a batch's longest lane choosing for every lane, never on BF16 pages or selected blocks. attention-variants runs every INT8 case with bf16 queries and with INT8 queries in every split, its split counts and windows included, against a reference that quantizes its queries as the kernel does; q8-flash-attention runs every dense geometry's q8q entries. The kernel-selection golden pins each device's choice.
The repack's cost is its per-element work (a byte load, mask and slot store per element, then a pack), not its access pattern: coalescing the plane stores or staging the reads in threadgroup memory moved it by 0.75x to 1.15x on the M4 and the M5 Pro. For the streamed models' expert formats the slot order maps native bytes to plane bytes in fixed groups, so a group's planes are a few shifts of its block's words: Q4_K and Q5_K pair words from two qs words each, Q5_K's fifth bits gathered by one multiply, Q2_0's (and PQ2_0's) bytes moved whole. These decoders replace the three formats' byte decoders; every other format keeps its own. Planes stay bytewise the same. The blocks are read as words (Q2_0, PQ2_0: halfwords) because their sizes are multiples of 4 (2) and rows are whole blocks after a tensor start on 8 bytes: the GGUF parser now requires general.alignment to be a power of two of 8 or more, as the GGUF spec does, and the staging starts on pages. gguf-moe-benchmark's repack mode times moe_expert_load at 1 to 32 jobs and gguf_repack on whole tensors, in Q4_K, Q5_K, Q6_K, Q8_0, Q2_0, IQ2_XXS and af4g64 at the 35B's expert shape, the 35B's Q4_K/Q5_K pair and Flash-Next's Q2_0 shape, and compares every plane with the reference repack. decode-profile takes --verify-rows all, as backend-benchmark runs, so two builds' cycles route the same experts, and reports each width's tokens per cycle and wall ms per token. This build against its parent's library, ABBA: moe_expert_load at 4 and 8 jobs, Q4_K/Q5_K 1.5-1.6x and Q2_0 1.9-2.0x on the M5 Pro 16c, 1.14-1.2x and 1.4-1.5x on the M6; other formats 1.00-1.01x on the M5 Pro, 0.96-1.02x on the M6. Qwen3.8-Flash-Next streamed at 16G on the M5 Pro 16c (decode-profile, every row verified, 4 runs a side): B1 22.93 -> 22.30 ms per accepted token (-2.8%, every run below every run of the parent, spreads 1%); B2 to B4 -1.1%, within noise. Qwen3.6-35B-A3B UD-Q2_K_XL at 4G on the M6, whose IQ experts keep their byte decoders: B1 to B4 within noise.
Apple9 GPUs of more than 10 cores now run GGUF MoE experts on the staged tile, in both phases and every format, as Apple10 and the 10-core Apple9 GPUs already did. Nothing selects the MoE register tile any more, so it goes: its expert passes (moe_expert_gguf_sg_*), the Table16 gather and prepare kernels, MoeGgufTile and MoeConfig::ggufTile, the law table's MoE rows (MoeCalibration), the experts' format unit (MoeFormats::experts), the grouped sums scratch and the room the register gate pass took in the expert output. The dense register tile and its Table16 inputs stay. The register tile kept those GPUs on a September ranking: on a 40-core M3 Max, Q4_K/Q5_K, Q2_K and IQ4_XS experts took 3-34% longer staged (1bf892b). Re-measured on a 10-core M4 (35B-shaped layers that stream their experts, ABBA): since 1bf892b the staged tile takes 11-45% less time at decode, mostly from the column tile, while the register tile moved by -7% to +2%. It now takes 16-42% less time than the register tile at one to four decode lanes with random routes, 3-17% less where each lane's rows share 24 experts, and 14-22% less on 2048-row chunks, for K-quant, IQ4_XS and Q2_K experts. On a 60-core M3 Ultra, one 35B layer (Q4_K gate/up, Q5_K down, Q8_0 shared) took 1-24% less time staged in all 8 measured decode cases, one to four lanes with random and pooled routes (one run beside live servers). Apple9 GPUs of 11 to 40 cores (the M3 and M4 Pro and Max) are unmeasured with today's tile. On those GPUs a MoE layer runs 7 dispatches instead of 9, and prefill chunks past one route per expert take 32-row tiles: the 35B's 2048-row prefill scratch grows by 39 MiB (20%), while a decode lane needs 390 KiB less. As on every other GPU, a row's decode result now depends on the rows batched beside it (the column tile). Every other device's plans are unchanged: against main with the commits before it at the time, kernel-selection --dump of this commit differs only in the deleted fields, on every device, and in the MoE prefill and workspace lines of the Apple9 GPUs above 10 cores, which now equal the 10-core Apple9's. The golden is regenerated (each device's MoE groups print one field fewer). On the 10-core M4 the 35B's eight greedy transcripts are bitwise those of main with the commits before it at the time. gguf-moe, expert-store, execution-plans and operator-workspace lose their register cases; gguf-moe's two-pass reference parks the gate in a buffer of its own, and its verified-rows case compares every row through its every-row run, as no row is sure to keep its tile kinds. gguf-moe-benchmark times the device's plans alone. The M3 Ultra measurements are janhilgard's, from incoai#367.
A prefill command's runtime reads one row of the last layer's output per sequence, its last, unless the draft taps that layer. The Qwen3.5 family's last layer (an attention layer in the 27B, the 35B and Flint) now writes its state on every row and runs its output work on those rows alone: the attention tile that holds each read row, as a chunk of its own on the chunk's kernels and splits (PagedAttention::prefillTail), its gated row, then the o-projection, the post-norm and the FFN over the compacted rows on the chunk's tile kinds and K splits (Linear::prefillPlan of a chunk's rows). A MoE block runs the plan of the read rows; the Neural Engine split, which takes whole chunks, leaves them to the GPU. A dense target's read rows keep their bits: gguf-projection checks a chunk's last rows on its tile and splits at their own height bitwise and paged-attention-plan the tail tile, and the 27B's logits (MLX 4-bit and NVFP4: 96 prompts one at a time, 24 of up to 17K tokens four at a time) are byte-identical to those of main with the commits before it at the time. A MoE block's routes see fewer rows of their experts and sum in another order, so a MoE target's read rows are not bitwise that build's: next-token KL mean 5e-10 to 2e-8 (p99 at most 2.5e-7, top-1 unchanged) for the 35B, GGUF and MLX 4-bit, and Flint, where that build's own logits of the same prefixes move by 6e-4 to 2e-2 between batch widths 1 and 4. Prefill time against main with the commits before it at the time, 2 ABBA rounds: on a 20-core M5 Pro the 27B 1.1% less at 2K tokens and 1.3% at 8K (MLX 4-bit and NVFP4 alike); on a 16-core M5 Pro the 35B 1.9% at both (GGUF), 1.5% and 2.1% (MLX 4-bit), Flint 1.7% and 2.3%. The 27B's prompts of 8, 16 and 32 tokens are within noise on both Macs (-0.3 to +0.1%). Ornith-1.5-122B and the rotated PQ2_0 27B take the rule too and were not run. incoai#691 by fishy-ops had the same idea, computing only the last layer's final rows of a prefill chunk.
Based on incoai#680 by zaiyang-zhang. Prism's rotated PQ2_0 (Ternary-Bonsai) stores ternary codes, exact in int8, for Hadamard-rotated inputs, whose values spread evenly over each 1024-input block, so one int8 scale per row and block quantizes them with little loss. On Apple10, whose neural accelerators multiply int8 operands, every staged decode plan of 16 or 32 rows of such a unit whose K splits keep whole PQ2_0 blocks now runs int8 tiles: gguf_rotate_int8 writes the rotated input as int8 rows and scales, and gguf_decode_int8_pq20_m{16,32}_* stage the codes as int8, accumulate in int32 over each 128-input block and fold acc * d * s into fp32. A block holding an infinity or a NaN, whose codes alone would round to finite values, takes a NaN scale (kernels/common/int8_activations.h), so that its row's outputs are not finite, as the bf16 tiles' are. The int8 tiles take the staged plan's rows, K split, spread walk and pass mode (the one-pass gate/up pair included), so the split law's choice stands and the plan says what runs; scratch bounds are unchanged. One lane runs them on its 16-row plan; one lane's split plans, of 8 rows, keep bf16: the 8-row int8 matmul takes about twice the 16-row one's time. ProjectionCalibration::decodeInt8FromRows (16 on Apple10 and Apple11; kNever on Apple9, unmeasured) holds the rule, LinearConfig::decodeInt8 marks the plan, UnitFormats::decodeInt8Takes the formats (quant_decode_int8_format: PQ2_0). A plan in int8 activations over weights the int8 tiles do not take (unrotated, or not PQ2_0) is refused. No option. The one-pass gate/up's split reduction moves into gguf_store_gate_up, which the bf16 and int8 pairs share. Apple11 takes the same row: on a 12-core M6, where gate/up runs in two passes and one lane's down projection unsplit on the 16-row tile, the policy's int8 plans take 10-25% less time than the same plans on the bf16 tiles for every Bonsai projection at one to four lanes (projection benchmark, two runs); end to end unmeasured there. Bonsai-27B PQ2_0 with the Qwen3.8-27B DFlash2 draft, same machine and toolchain: - M5 Pro 20c, against main with the commits before it at the time (A B B A): decode-profile cycle -12.8/-14.0/-15.9/-15.0% at B1-B4 - M5 Pro 20c, against main 9124ff2 (A B B A A B, 3 per side): cycle -13.9/-15.0/-16.2/-15.9% at B1-B4; backend-benchmark decode tok/s x1.16/1.19/1.20/1.19 - M5 Max 32c, against main 9124ff2: cycle -8.4/-10.8/-19.7/-19.0%; tok/s x1.09/1.13/1.25/1.24 Acceptance and output tokens unchanged on the backend-benchmark decode scenario (64 tokens). Teacher-forced next-token KL against the bf16 tiles (production verify, four 256-row windows after 1-4K tokens): - the production plans (one lane: int8 on gate/up, the input projections and the head; down and out bf16): 2.8e-4 nats mean, p99 1.6e-3, top-1 agreement 99.1%, NLL +0.01% +- 0.04% - every projection on the int8 tiles, as from two lanes (a measurement build running one lane unsplit): 3.2e-4 mean, p99 1.7e-3, top-1 99.1%, NLL +0.07% +- 0.04% (M5 Pro 20c, against main with the commits before it at the time, which matches itself exactly.) gguf-projection checks every int8 plan against fp64 plus the int8 term, lanes and pass modes bitwise, the fp32 destination bitwise against bf16's, a NaN and an infinity input leaving their rows alone not finite, and the production plans' choice on every GPU (the int8 plans themselves run where the device row takes int8 activations); gguf-rotation each block's stored scale, the NaN bits where it holds a NaN or an infinity; linear-plan the refusals. The kernel-selection golden gains a section, projection-decode-int8: the decode plans of every PQ2_0 projection case of whole rotation blocks with their int8 activations, 555 groups on the 15 devices; every other group is unchanged. The projection benchmark now runs PQ2_0 specs of whole rotation blocks rotated, as Prism stores them, rotation included in every case, and times the int8 tiles beside the bf16 ones: its PQ2_0 numbers are not comparable with earlier runs. Co-authored-by: Zaiyang Zhang <102530270+zaiyang-zhang@users.noreply.github.com>
The menu bar's panel could only stop the model that runs: starting one, or switching to another, meant opening the window's Server screen. The window's Server menu repeated the panel's Stop and Coding Agents, which the Server screen has too. The panel now lists the models on this Mac under Model. The one that starts or runs, or a switch goes to, is checked (AppState.checkedModel); none is while a model stops or once it failed, so that choosing it starts it again, as the Server's Try Again does. Choosing another starts it, or switches to it, through AppState.start with the launch options set now, as the Server's Start and Switch do; choosing the one checked leaves it as it is (AppState.choose). The panel keeps its look: Model opens its rows as Coding Agents does, scrolling past some ten of them so that a long list keeps the panel on the screen. The Server menu is gone: the Server screen does all it did in the window, and the panel out of it. Whether a model can start is one rule, AppState.canStart (the app ready, the Keychain giving the API key), which start(), the Server's Start, Switch and Restart buttons and the panel ask; the buttons each wrote their own part of it before. Co-authored-by: James Martin <1485492+jiiim@users.noreply.github.com>
Quitting asked only while a model downloaded, in a modal alert that held up logging out too, and cut off the requests the model was answering and the programs the drawer's terminals ran without a word. A quit of the user's, from ⌘Q, the menus, the menu bar's panel or the Dock, now asks where it would cut off work: requests the model is answering, as its last /status said, or the Chat's reply as it streams; a download, which starts over next time; and programs the terminals run, those whose foreground process group is not their shell's. The question says each. A model that idles, sleeps, starts or switches is no loss, so the quit goes ahead at once, as one does from a script, from the system as it logs out, from a signal, and from the installer of an update the user chose in Sparkle's window: none of them is the user's quit event. The question is a sheet of the main window where it shows, else an alert of its own: neither holds the app modal, which would refuse the system's quit as it logs out. A quit that goes ahead takes it away. The download's modal alert is gone. Co-authored-by: James Martin <1485492+jiiim@users.noreply.github.com>
A chat could be renamed or deleted only from a right click on its row, which nothing shows, and only one at a time. Under the pointer, a row now shows a button at its end that opens its menu, the one a right click opens too: Rename…, Delete…, and Select, which starts picking chats to delete. While the list picks, a check before each title says whether it is picked and a click picks it or not; under the search field, Select All picks every chat shown (or none again), Delete deletes those picked, and Done ends the picking. Either Delete asks one question, fixed as it was asked (ChatDeletion), which names the chat or says how many. ChatStore.delete takes a set, so one chat and several go the same way: a reply streaming in one stops, each file goes, and the open one deleted, a new chat opens. Only the conversations the store holds are deleted; a file in the folder that is no chat stays, even under the name of an id asked for. Co-authored-by: James Martin <1485492+jiiim@users.noreply.github.com>
New Chat was a button at the toolbar's end while the Chat showed, apart from the chats it adds to. It is now the first row of the chat list, above the search field, where the chats it joins are; ⌘N and File › New Chat do as before, and do so while the sidebar is hidden.
The chat list was one run of chats, the latest first, with no way to keep one at hand. A chat's menu now pins it, or unpins it (Conversation.pinned, which a file from before reads as nil); the list puts the pinned chats first, under Pinned, the last pinned at the top, and the others under Today, Yesterday, Previous 7 Days, Previous 30 Days and Older, by the day each changed last (ChatSection), grouped again as a new day starts. The questions to rename and to delete chats now show on the Chat itself rather than in its sidebar.
… and Apple11 Apple10/11 matmul2d multiplies int8 at twice its bf16 rate; Apple9 at 0.9x. Prefill chunks past the staged tile (65+ rows) of a target projection whose quantized segments are all af4g64 or af5g64 now run LinearTile:: GgufPrefillInt8 (kernels/prefill/gguf_prefill_int8.metal): gguf_prefill_int8_quantize writes each 64-input group of each row as int8 codes with its own scale, and the GEMM feeds the weights' 4/5-bit codes straight from the planes into int8 fragments, folding each group's affine correction (sa s) sum(qa q) + z (sa sum(qa)) once in fp32. Weights stay exact; a scale per group (not per row) keeps an outlier channel to its own 64 inputs. A group holding an infinity or a NaN takes NaN scales, so its row's outputs are not finite, as the bf16 tile's (the int8 decode's rule, kernels/common/int8_activations.h). One rule in the law table, ProjectionCalibration::prefillInt8 (Apple10/11 on, Apple9 off), read by selectProjection with UnitFormats::prefillInt8Takes (quant_prefill_int8_format). The prefill arena's LinearTable and LinearSums hold the codes and scales of a full chunk of the target's widest projection whose formats the tile takes (ProjectionShape::prefillInt8Formats) wherever the family runs the tile: +40.1 MB on Qwen3.8-27B MLX 4-bit, its 17408-wide down projection; a GGUF target reserves none. The ANE split's GPU part and its calibration take the tile through Linear; it reads the split's leading-input view through a plane stride. A prefill's last layer runs its read rows alone on the chunk's int8 tile, which computes each row from its own codes and scales: they keep the chunk's bits (gguf-projection, linear-plan). A DFlash2 draft, prepared af4g64 whatever the target's format, prefills on the bf16 tiles: its projections run on ExecutionPlans::draftLinear, the target's plans with no prefill chunk on the int8 tile whatever --prefill-int8 says, and the prefill arena sizes its context projections on those plans (execution-plans, model-execution-plans). Drafts stay on bf16 because their prefill gained nothing measurable from the tile (the 27B MLX's TTFT, GPU alone on a 32-core M5 Max, moves +0.15% at 2K and +0.05% at 10K when its draft leaves it, within A/A), while their context projection, the widest a prefill runs (25600 inputs on the 27B), would size the tile's scratch for every target and can tip a model at the memory edge into streaming: 36 MiB on Qwen3.6-35B-A3B GGUF at --max-memory 24G streamed the routed experts its plan otherwise holds whole. A tile is two simdgroups of 32 rows by 32 columns for both formats, and threadgroups walk 16 row tiles down each column tile before the next (GGUF_PREFILL_INT8_WALK): walking every column tile of a row tile first re-streamed the weights once per row tile, which left the M6's 2048-row down projections (K 17408) at 0.95x (af5g64) and 1.12x (af4g64) the bf16 tile; they now take 1.19x and 1.34x less time. af5g64 on 64-row tiles lost to the bf16 tile at 65-96 rows on a 16-core M5 Pro (0.95x); on 32-row tiles it takes 16-22% less time there, and 6% more at 2048 rows on the M6. Over the 27B's projection shapes in af4g64 and af5g64 at 65-2048 rows the tile takes 1.20-2.24x less time than the bf16 tile on a 32-core M5 Max, 1.13-1.77x on a 16-core M5 Pro and 1.19-2.39x on a 12-core M6. The destination's element coordinates follow matmul2d's int8 fragment layout, as the operands' do: taking them from get_multidimensional_index loads each element's scales alone, up to 25% more time. splash serve --prefill-int8 on|off (default on; serve-native, backend-benchmark and teacher-logits --prefill-int8 on|off) turns the tile off: ExecutionPlans and Linear take one prefillInt8 flag, the ANE split and its calibration take the plans' Linear rather than their own, and a remembered ANE calibration is keyed by the setting. Off, every prefill projection plan and the prefill arena are the bf16 ones, and the 27B MLX prefill reproduces its parent's logits bit for bit over 24 real 1K-8K prefixes. TTFT, default flags, ABBA x2 with a warm-up per binary: Qwen3.8-27B MLX 4-bit, 20-core M5 Pro, against this commit's parent, 4 samples a side: 2K 3095.4 -> 2258.7 ms (-27.0%), 10K 15764.3 -> 11696.1 (-25.8%), 176-288-row appends -36..-40%, 16- and 32-row prompts -2..-4% (the Neural Engine split takes 10 of 34 units instead of 15). Earlier builds of this change, whose plans and kernels for these targets' chunks are the final ones, against main with the commits before it at the time, before the last layer's row trim: Qwen3.6-35B-A3B MLX 4-bit, 16-core M5 Pro: 2K 650.7 -> 565.8 ms (-13.1%), 10K 3440.9 -> 3034.1 (-11.8%), 176-288-row appends -9..-10%. Next-token KL against its parent's bf16 GPU prefill over 24 real 1K-8K prefixes, one position each, with bootstrap 95% intervals (20-core M5 Pro): - GPU alone: Qwen3.8-27B MLX 2.15e-3 [4.8e-4, 4.8e-3] (median 3.6e-4; 2.8e-2 at one 8000-token prefix), top-1 24/24, NLL +6e-4; Qwen3.6-35B-A3B MLX 2.35e-3 (median 1.4e-3), top-1 24/24 (16-core M5 Pro, an earlier build whose logits the final kernel reproduces bit for bit, against main with the commits before it at the time). - default flags, the Neural Engine split at each side's own calibration: this change (10 of 34 units) 9.4e-4 [5.5e-4, 1.4e-3], NLL +2.3e-2; the parent (15 units) 1.11e-3 [3.8e-4, 2.3e-3], NLL -1.0e-2. Both change the most likely token at the same one prefix (3000 tokens) of 24, and every NLL interval holds 0. Based on incoai#445 by jma24 (fragment schedule ported from oMLX's oQ A8 kernel, Apache-2.0, THIRD_PARTY_NOTICES), reworked as a default family rule with per-group activation scales. Co-authored-by: John Appleby <51924052+jma24@users.noreply.github.com>
… step A streamed MoE layer's load (ExpertStore's host step: route, SSD reads, jobs) left the GPU idle. The passes over the experts already resident and the shared expert read nothing the step writes, so they now run while it works; the passes over the experts it brings and the combine run after the repack. - MetalBackend: a host step's beside dispatches (EventStep::beside, CommandGraph::host's `beside`) run in a buffer of their own after the step's signal; the work after them waits for the step. They read no 128-byte line the step's work writes. All of a step's buffers are made before any is committed; a step after a buffer of them that failed by its turn is skipped. - An awake step's spin waits for the step's signal. A spin that ran ahead of the work before its step, while that work waited for the beside dispatches before it, at times held that work till its bound (M5 Pro, Flash-Next decode: 1 step in about 1000 held 100 ms; the new awakeHostStepsBesideLongWorkResumePromptly saw it in 4 of 5 runs). The runner raises that signal itself, before its own, for an awake step it gives up or that follows a failed buffer, so the spin still ends: without it a stopped runner's step whose buffer then failed drained only after Metal's 5 s limit (blockedHostStepRunnerStops). On plain awake steps the wait costs nothing measurable (M5 Pro, Flash-Next with every expert resident and the PLE table streamed, whose one awake step a cycle is plain, B1: 16.19 against 16.22 ms per token, A/A spread 1.0%, outputs identical). - The slot table marks an expert no slot holds MOE_SLOT_ABSENT. MoE::add's grouping of a streamed block orders its tiles resident | shared | absent, writes grids for all, resident and loaded tiles (MoeTileCount) and copies the slot table into scratch (MoeScratch::expertSlots), which the passes beside the load read: no GPU work reads a line of the tables the step writes while it runs. Each expert's tiles, kinds and the combine order are unchanged, so outputs are bitwise the same. Other groupings read no slot. - The expert passes take a tile range. The passes after a load read a second table of the slots it routed, which its host step writes and no GPU work reads before them. - kernel-selection --dump differs only in the MoE workspaces (the tile count's 80 bytes and the expert slots); the golden is regenerated. Measured on 90287a28. Decode, verify rows pinned, B1-B4: M5 Pro Flash-Next Q2_0 at 8G (17-22% of routed experts read) -10.9/-16.6/-14.4/-13.9%; M6 35B Q2_K_XL at 4G -7.5/-15.6/-14.0/-15.4%; M4 35B Q4_K_M at 3G (53-70% read, steps of 8-20 ms) -0.8/-0.4/-0.4/+0.1% over two windows. Ms per accepted token, Flash-Next at 16G: M5 Pro -10.0/-13.9/-13.1/-8.2%. Resident 35B within noise (M5 Pro +0.2/-0.2/-0.1/-0.4%). Flash-Next transcripts' output tokens identical to main's.
A start reads the volume the SSD cache's files will take through
RuntimeResourcesConfig::diskVolume: it decides whether they open, and the
room a block-sparse target's plan spills into. Once they opened, DiskSpace
read the volume again with fstatfs on the first file's descriptor and set
the tier's limits and that room from it. In production both name the same
volume, but a test that gives the start a volume gave it only the first
reading, so the tier's limits came from the test Mac's own disk:
- runtime-resources, on the M6 (9.8 GiB free on a 460 GiB volume, below
its 10 GiB floor): the first Flash-Next start on a 500 GiB test volume
had 0 GiB of room for active KV, served only what memory holds, and the
relaunch test failed ("a block-sparse target did not spill past what its
memory holds ..."), at every head run on it since 9124ff2. On the
Studio, with 37 GiB free, the room was 26.7 GiB instead of the volume's
32 GiB.
- disk-space, on the M6: each DiskSpace measured the real volume as it was
made, its root's limit was 0, and it removed the other caches the test
then counts ("the other caches were not counted").
DiskSpace now reads its volume through a reader it is given. The start
builds it from diskVolume for the place the files opened in, a persistent
tier's root, else the temporary files' directory, as it read them before
they opened, so the two readings agree. Production reads the same volume
through volumeOf; SlotFile::volume, which only DiskSpace used, goes. The
disk tests give each DiskSpace a volume of their own, and the relaunch test
checks that the first start's cache measured the 500 GiB its test volume
has available, not the test Mac's free disk.
--prefill-int8 (adcac68) is a serve option engine_argv reads for every engine command, and main_args, the one serve namespace the tests build by hand, never gained it: every main() test of test_server_main, test_server_recovery, test_stop_with_app, test_chat_formats and test_served_model_names stopped with AttributeError before it reached what it tests (1 failure and 29 errors in make test-python). main_args gives it the parser's default, on, written as its neighbours' are, so the engine's command line it makes is unchanged.
After a failure every restart would meet again, or the sixth short-lived failure in a row, the backend stopped relaunching the engine for good and the server stayed up: /health answered 200 and every request got 500 engine_failed until someone restarted the server. launchd's KeepAlive and other supervisors act on an exit, so a Mac serving unattended stayed down after a burst of crashes long after its cause had passed. One rule: once the backend stops restarting, it says so (NativeBackend.on_stopped, after its Engine stopped line), and the server stops as SIGTERM stops it, its engine closed and its requests answered, records why in its serve record (kind stopped) and exits with status 69, EX_UNAVAILABLE, apart from a stop's 0 and a failed start's 1. What runs the server decides whether to start it again, and a new server counts failures from none. No decay is added: 60 s of service already starts the count anew, and only memory-refused restarts, which by design neither add nor clear, sit between two bursts that count together. The reason, which said to restart the server, now says to start it again, as it has exited. The app words every exit it did not ask for from its pid's serve record, a ready server's too (Server.exitProblem), so it shows the stop's reason as it did from /status; an internal failure, which a ready server records as well, is worded without a phase. What it kept for a failed server that still ran goes: reading transport.stopped, Not responding, Switch to and the stop on dismissing the failure. Tests: a server whose relaunch is refused exits 69 once its engine is closed, its record as serve_record_stopped_golden.json, which the app shows word for word; the sixth short-lived failure and each permanent relaunch failure call on_stopped once with the reason; the app's fake server stops as a server does, and the app shows its record's words and runs nothing. Reported by @dmkjr in incoai#716.
One finalizer thread completes every request, and it printed each
request's Started and Done lines on the way (print_status). A write to a
stdout or stderr pipe whose reader stops draining blocks, so every
request then waited for its terminal. The engine writes to the same
stderr, from its loop, a line at each request's first token: there the
loop blocked, left its status probe unanswered and was failed.
Reproduced on the hub (M5 Pro 20-core, Qwen3.6-35B-A3B UD-Q4_K_M, the
console on a FIFO whose reader stops after Ready, four chat requests):
- stdout on the FIFO: the engine completed all four, and none was
answered until the reader resumed, 61 s later;
- stderr on the FIFO: the loop stopped at the next request's TTFT line,
was failed 30 s later ("native loop did not answer status within
30 s") and the four got 503 runtime_unavailable.
With this change all four are answered in 64-91 ms either way, and
every line reaches the FIFO once it drains.
- diagnostics: print_status, and print_line for an untimed line, hand
each line to one console thread, which writes both streams in the
order the lines come. One writer keeps one order across stdout and
stderr, which a log of both (2>&1, the app's server log) shows; so a
stdout that stops draining holds up stderr too. A line that finds
1024 waiting is dropped and counted, and /status reports the count
(console.dropped_lines). A wait for lines lasts 1 s at most
(LAST_LINES_SECONDS): the exit's for the last lines, an engine's that
stopped with it included, a failed engine's report's, and the
launcher's before an exec. The console thread keeps both stop signals
blocked, so that one sent to the launcher while it spawns or executes
a program stays pending for that program, as before.
- runtime: the engine's stderr is a pipe whose lines a thread hands to
the console as it does the server's, never waiting on it, so the
engine loop never waits either. A failed engine is asked to stop
before its failure is reported, not after, and the report waits for
its lines to end at its exit, 1 s at most, so that why it failed,
which it says there as it exits, or what it saved as it stopped comes
before the server's "Engine failed" or startup error. A failure the
server finds, as broken output, thus waits for the engine's stop: a
test engine that exits on SIGTERM is reported 14 ms after it, and
one still running a second later, as a hung loop, that second after
it.
- launcher: every line of its own goes through the console, as its
serve record's warnings already did, so a refusal keeps its place
after the warning that its record could not be written; before it
executes the server or a client, which would drop the lines still
waiting, it waits for them.
- server: a connection's unexpected error is one console line
(log_unexpected), not socketserver's traceback written straight to
stderr on the connection worker.
- backend: the reader thread prints the Started line itself again, and
the finalizer's queue holds requests' terminals alone, as before
a2d8f41, which made it carry Started lines so that a console that
stopped draining would not hold up the reader.
- StderrLine.hpp: its comment says that the server reads the engine's
stderr line by line, no longer that both write to one stderr.
- Tests: a console that stops draining holds up no caller, request,
engine start or HTTP answer, and its lines are written in order or
counted as dropped; the exit writes its engine's last lines; a failed
engine's last lines, its cause or what it said as it stopped, come
before its failure; a connection's error is one console line; a stop
sent to a launcher process while it spawns a program ends that
program; the launcher's lines come before the client it executes.
Tests that read console output wait for it to be written
(console_written).
Cancelled requests still decode their tail and count their reasoning
on the finalizer: with the Qwen3.6 tokenizer on an M5 Pro that takes
0.3 ms for a 2K-token output and 3.8 ms for a 32K-token one, too little
for a case of its own.
Reported by @dmkjr in incoai#719.
On 1.6.0 a long reply's decode logged, 1-2 times a second for as long as it ran (Qwen3.8-27B IQ3_S, --max-memory 28G, --state-format bf16, --persistent-cache): Memory: in-service claim of request 17 needs 130.0 MiB, 1 pages: met, took cache 0.0 (0.0 MiB now, 0.0 later) Each page the decode's KV grows into is an allocation of the request's one claim. At the engine's limit a new extent (130 MiB) is refused, and the ledger meets the need with the next page the cache's order gives up: a cached page in an extent that stays allocated, so the take frees 0 MiB and gives the page. The ledger logged each decision once for each kind of need since the claim's last grant, and every page is a grant, so every page logged again. incoai#706 read the same lines as a sign that the cache was lost. The rule: a request's claim met from cache logs nothing unless the claim logged since its last grant. "Met from cache" is the notion free() already had for the pools' regrowth: met, taking nothing that costs work later (Run::costly), that is by the room or by the take order's steps 1-4 alone (idle state buffers, cache that frees now, rebuilt caches, cache that needs a write). One predicate, metFromCache, now serves both. Once the claim logged since its grant, a met from cache logs as before, so a pending or short line the log shows ends in a met. Every other decision of a request's claim logs as before, once for each kind of need since its last grant: a wait (pending or short), a need met past the host, and a take of the pass command, a conversion, a pool unit above its floor (the expert store), a unit in use or the next lane's buffers. Only a decision that logs marks its outcome logged, so a silent met hides no costly met of the same need after it. A claim of no request (the host's pass, a pool's growth) still logs each take. Besides the cache's own units this leaves out the met line of a rebuilt cache's unit (the vision arena, images' rows, idle history staging, an unread hot block cache), which the regrowth rule counts as cache too, and of a need the room met taking nothing. /status shows the cache's pages (kv.pages_cache), its evicted states (state.evictions) and the pools' shrinks (memory_governor.accounts). Tests: - memory-ledger testARequestMetFromCacheLogsNothing: eight decode pages, each a cached page of a kept extent and each granted in turn, log nothing; the same need met next, with no grant between, by a store chunk logs its take. On the parent it fails with the reported line: "a request's claim met from cache logged 8 lines: Memory: in-service claim of request 17 needs 130.0 MiB, 1 pages: met, took cache 0.0 (0.0 MiB now, 0.0 later)". - The tests that asserted a met-from-cache line with nothing logged before it follow the rule: testEachDecisionLogsOneLine's met takes a store chunk; testAHostExemptionLogsMetPastTheHost drops the met the room gave before its retries and logs only the met past the host; and testABytesNeedBesideBuffersNoStateGivesFreesTheirBytes reads its one line from its retry, short once the state went (on the parent the met before it logs too, and it fails on that line). - testARetryBackToALoggedOutcomeLogsNothing keeps its last line, the met the room gives after the short logged past the grant. docs/internals.md describes the lines under "What a refused claim takes". Reported by @didikrm in incoai#726.
A client that vanishes without a FIN or RST, as a Mac asleep or off its network does, sends nothing a non-streaming request reads, so its request held a lane until the output ended. Accepted connections now enable TCP keepalive from one table, TCP_KEEPALIVE_OPTIONS (30 s idle, then 3 probes 10 s apart): once its peer stops answering, the connection drops 60 s after its last data, and the request's next disconnect check cancels the work. The options are set on the accepted socket before the handler makes its files. macOS refuses them (EINVAL) on a connection its client reset before a worker served it; setup raises that as ConnectionResetError, which handle_error treats as a client that left, so the connection ends silently, as any reset does, instead of logging an error. client_disconnected also reads a TimeoutError as a disconnect: a connection keepalive dropped fails its read with ETIMEDOUT. This changes how the end is reported, not what cancels the request: such a read already reached handle()'s TimeoutError branch, which cancelled the request and wrote a 408 to the dead socket; now the request ends as a disconnect does, without a response. A slow but live client cannot trip it, because the read runs only once poll reports the socket readable. docs/api.md says that a client that stops answering is dropped 60 seconds after its last data. Tests: an accepted connection has keepalive on and drops a peer that stops answering 60 s after its last data, TCP_KEEPALIVE plus TCP_KEEPINTVL times TCP_KEEPCNT; a connection its client reset before it was served ends its setup with ConnectionResetError; a read that fails with ETIMEDOUT reads as a disconnect. Fixes incoai#718.
Every request the engine had started at both failures is refused, not only the one that caused them, so with several lanes a refusal can fall on a request that did nothing, and a client that retried it, the same request in every field but its id, priority, deadlines, seed and pixels, was refused until the server restarted. Refuse it for an hour the Mac is awake instead (time.monotonic). Let back, a request that does fail the engine costs at most two failures an hour: one while it is still a suspect, and two once a later failure has replaced the suspects. By itself, a request let back cannot make the server exit, as it now does once it stops restarting its engine at the sixth failure in a row each within 60 s of starting: the engine it fails has served for nearly the hour since the refusal, which starts that count anew, and that failure or the next refuses it again. Tests: a refusal ends after QUARANTINE_SECONDS; let back, a request is refused at the next failure it is started at while still a suspect, and at the second once a failure it was not started at has replaced the suspects. Fixes incoai#717
…spilled A request that cannot fit even alone ended with capacity_exhausted naming only what it could not allocate and why, and on every run whose KV can spill a refused KV target added "the SSD cache took no more of its KV", whatever the lane's mode. A Resident lane's KV never goes to the SSD cache, so its failure read as a cache too small: incoai#713's lone 107K prompt, started Resident and failing 70 s into its prefill on the hub, said so. Nor did the message say what else held the memory the limit charges. One rule: the end names its lane's KV mode (Resident or Spill), its phase (in prefill, in decode, at its start or at its resume) and its tokens, and the Metal memory outside the backend's buffers as the device last counted it (device_untracked_bytes, the quantity memory_audit measures at startup, read now through MemoryGovernor::untrackedBytes), which the limit charges past the runtime reserves; it says the SSD cache took no more of the request's KV only for a Spill lane, whose KV alone goes there. A Resident lane whose conversion the quota refused so names no SSD cache either: none of its KV went there. On the hub serving Flash-Next that memory was 16,791,984 bytes at Ready and at the end of every incoai#713 hold run, so a value far above it names what took the room. Tests: a lone Resident prefill of 65,473 tokens at the engine's limit, Metal memory outside the engine's buffers taking the room (FakeMemory now counts such memory on the device: holdUntracked), fails naming a Resident lane of 65473 tokens in prefill and those bytes, without the SSD clause, where the parent named neither and added the clause; a short lane in decode, a prefilling one and one whose conversion the quota refused, all Resident, name their mode and phase and not the SSD cache; a lone Spill prefill names its lane and keeps the clause; a lone request that cannot get its state names its Resident lane of 97 tokens at its start, and a suspended lane retried with nothing completed names its lane at its resume, where the parent named no lane. Reported by @mperreir in incoai#713.
A start placed Spill whose SSD room is short of its projection, or whose tier is closed or out of write budget, started Resident once its projection fit one request's resident context (fitsResidentAlone), without the live test placement had just failed. It so started a lane into memory the engine already saw it could not have. incoai#713's lone 107K prompt on the hub, with 0.6 GiB of SSD room for its 3,350 projected pages and 512 MiB held outside the memory plan, was placed Spill by that test, started Resident and failed with capacity_exhausted 70 s into its prefill. One rule: such a start starts Resident only where the claim its admission makes holds its Resident KV now beside the lanes' projected growth, as placement prices it (Engine::residentRoom, now shared by both; Engine::inMemory). Alone that claim is InService, as its lane's growth is, which also takes the pooled buffers the next lane starts from. Beside other lanes it is a Start's, the test that placed the start Spill unless every request spills, and where that test fails the start waits for those lanes, taking memory from a lower priority's lane as a start refused memory does. Alone, where memory does not hold it, it takes the SSD cache's bound on its output or waits for its quota as before, and is otherwise refused at its start. The refusal keeps its code, kv_disk_quota (or kv_write_budget, or capacity_exhausted for a closed tier), since the SSD cache is where such a request's KV goes, and its message now names what memory has for it too: resident_bytes and memory_bytes beside needed_bytes and active_limit_bytes. Alone it so counts no chunk of the expert store, which a run that can spill keeps whole as its floor. admit() makes its claim before the SSD cache's test and hands it on, and a start refused memory, by that test or by its state's or KV's allocation, waits through one Engine::waitForMemory, which takes memory from a lower priority's lane only while none is on its way back. Tests in Flash-Next's proportions, each failing on the parent: a lone start of a 64K prompt whose Resident KV memory does not hold, its SSD room 1,000 of its pages or its tier closed, is refused at its start with kv_disk_quota or capacity_exhausted, naming resident_bytes and memory_bytes, and never prefills, where the parent started it Resident and it failed with capacity_exhausted in prefill; with room for 2,050 pages, a lone 64K start takes the quota's bound, 127 of its 300 output tokens, and decodes them Spill, where the parent started it Resident and it failed in prefill; beside a lane that keeps memory from holding it, the start waits and never prefills beside it, then decodes, where the parent prefilled it beside that lane; and beside a Background lane a Foreground start makes that lane yield before it prefills, and both decode, where the parent prefilled it beside the lane. The small-page test of a start that cannot spill, the tier closed or the quota short, is the parent's test rewritten for memory: alone and short of it, each is refused at its start, where the parent started it. Guards, which pass on the parent: the lone 64K start beside the next lane's pooled buffers, which cover what memory lacks, starts Resident, takes them and decodes, and so do the small-page starts beside such buffers; with their claim priced as a Start's, which keeps the buffers, none of them starts. Reported by @mperreir in incoai#713.
… pay for its refill Under macOS's critical pressure every claim's take rights had Reserves::All, so a walk took the Index runway too: the empty Index extent a Spill lane's landed pages move into without an allocation. The host's rule refuses every allocation at that pressure, exempt claims included, and a spill's moves refilled the runway before they released the Full extents they emptied. So the moves went as far as one runway, 256 pages on Qwen3.8-Flash-Next, and stopped: the lane's later landed pages stayed Full, and so did the extents the first moves emptied, for as long as the pressure lasted. The rule: the Index runway is never a unit any claim takes, and an Index extent a spill's moves allocate passes the host when the Full extents the same moves released cover its bytes. - TakeRights::Reserves loses All: under critical pressure a claim takes the empty extent the next request starts from (NextExtent), never the Index runway. Cache::Walk::keepIndexRunway goes; every walk keeps the Index runway (Walk::keeps). - Cache::spillWritten moves the pages a runway at a time: into free Index pages, then the Full extents those moves emptied leave as SpilledExtents says, and only then does the runway refill. The engine tells the governor what they released (MemoryGovernor::fund), and an allocation of no more than that in the same scope passes the host at every pressure level (MemoryGovernor::tryReserve): the host gets back at least what it gives, a Full extent 17 times an Index one's bytes (108.5 MB against 6.3 MB). The credit lasts the Scope, one attempt of one claim, so nothing the claim allocates later passes on it. A refused refill ends the moves with what they freed. The landed pages are found again for each runway (Cache::landedPages), since gathering the Full pages moves those that wait. - A pool's growth the cache funds whole keeps its rule, no critical pressure and no host deficit (MemoryGovernor::growthHost): it returns the host nothing, and a host short of memory takes those cache units itself, while the refill follows what the moves returned and is what lets them go on returning memory. - Under critical pressure the moves release every Full extent they empty, the Full pages left first gathered in the fewest extents (Engine::landedExtents): left where they are, the Full pages one runway's moves free rarely make a whole extent, and one that empties is kept as the runway, so nothing would pay for the next Index extent; the host's pass at that pressure takes a kept extent anyway. Tests, each failing on the parent: - engine-cache testCriticalKeepsTheIndexRunway: the host's pass under critical pressure takes the empty Full extent and keeps the Index runway. On the parent it found and took both, 440 of 440 bytes. - engine spill/testLandedPagesSpillAtCriticalPressure: a Spill lane with Flash-Next's pages and extents whose 768 pages land under critical pressure moves them all and gives back three Full extents, with no failure or yield. On the parent, after 2,000 ticks and 801 pages written, 256 had moved and the Full extents held 434,110,464 bytes, as before the pressure. - memory-ledger testTakeRightsTable: the two rows at critical pressure keep the Index runway. memory-governor testReleasedBytesPayOnlyForTheirScope is a guard, since fund is new: under critical pressure an allocation passes on what its scope released, as far as those bytes go, and a nested or later scope of the same claim starts with none. It fails if a nested scope keeps the outer scope's credit, if the credit is never used up, or if a nested scope's end drops the outer one's. Reported by @mperreir in incoai#713.
A Spill lane's pages moved to the index form as soon as their copies landed, whether memory was short or not, and the lane took its hot block floor (390 MiB) and its history staging from its first page past its dense history and window. So every Spill lane decoded through the hot block cache: at 100K tokens on Qwen3.8-Flash-Next 60.6 against 71.3 tok/s Resident on an M5 Pro (-15%) and 53.4 against 65.6 on an M5 Max (-18.6%), the hot cache's SSD reads (2.53 and 3.12 ms a token) the whole gap, while writing through cost its prefill +0.1% and +2.4% (timed, interleaved, on b71f2e5). And no claim could take a lane's pages: memory came back only as copies landed, or by the claimant's own conversion. The rule: a Spill lane writes each page through and keeps it Full; a claim the take order brings to the lanes releases the oldest pages whose copies landed of the lanes its rights reach, only what it lacks and the first reads of every lane the release gives Index pages, with no write and no wait, and the lanes' reads then cost only the released blocks. Requests on one prefix share its pages, so a page moves only where the claim may release every lane that holds it, and never one of the chain a Resident start restores until it holds it: a Resident lane reads its history Full, and the start's placement and its own claims count that chain as its own, in place. A count of the releases, a start's placement among them, adds up what their takes in turn free as the copies stand, and the first take that moves a page frees it: each page, and each lane's first reads, count once. A release keeps the Index runway it moves into at every pressure level: one in place that would leave fewer than an extent of free Index pages moves an extent more, as for bytes, whose emptied Full extent refills it, or none, since the room may hold no Index extent (6.3 MB) to refill it, and short of an extent of free Index pages no release for bytes can start. Under --kv-spill always pages still move as their copies land, for validation. The self-conversion (R8) stays; only its moves at once, and the extra attempt after them, go. - Cache: ReleaseRule, the requests a release takes pages from and the chain a Resident start restores (by position), which spillEligibility applies to every holder of a page. releasablePages counts a lane's clean pages and those whose copies land from its scan mark, and the holders of the oldest, among whom is every holder of a later one: the lanes a release gives Index pages. ReleaseCount: the pages the releases counted so far move and the requests they give Index pages; releasablePages leaves them out, and countRelease adds a release's, both walking the pages through forReleasable and finding a page's holders through holdersAt. releasePages moves the oldest clean ones, found again for each runway (landedPages): for a need of bytes only where the first runway's moves empty a Full extent (releaseFreesAnExtent, which the engine's count reads too), the Full extents they empty released, the Full pages left first gathered in the fewest extents, and their bytes paying for the runway's refill (MemoryGovernor::fund); for a need of pages its inPlaceMoves: into free Index pages where those keep an extent beside them; else an extent more, as for bytes, the Full extents it empties leaving only past the claim's pages (moveLandedPages' claimed); else none. A release counts (kv_spill.releases) where it freed what it was for. owesCopies: the copies a Spill request owes that no write started. restoresIndexPages: whether a candidate's restore brings Index pages (LaneShape::spill). extentPages. CacheSpillPolicy::always keeps the moves as copies land (spillWritten), the one case where a landing frees pages by itself (spillsOnLanding: pendingPages, returning). checkInvariants: a lane's scan mark passes no page a release takes. - MemoryLedger: VictimKind::Release in the stage PassOrConversion was, renamed PassOrRelease (after the cache's writes, before the store, costly). TakeRights::lanes: work in service and starts that wait, their own lane and lanes of their priority or a lower one; the host under critical pressure, every lane; a need the host refused, its own lane; every other claim none (VictimQuery::reaches). take() passes each source what the claim lacks still (VictimQuery::lackingBytes, lackingPages). MemoryGovernor::free and the host's pass take in the claim's scope, where a release's Index refill passes on what it freed. SpillGeometry::releasedPageBytes: a page's Full less its Index bytes. - Engine: EngineVictims::releaseRule is the one rule for the lanes a claim releases and every holder of a page it moves: between commands, a Spill lane its rights reach, neither restoring, ended nor between its passes; off the chain a Resident start restores, its admission probe's, in its count as Resident (VictimQuery::kept, residentRoom) and its own claims alike. releases offers, the lowest-ranked lane first, for a need of Full pages first the claim's pages in place (inPlaceMoves), counting the bytes of the extent more, only where the reads of the lanes they give Index pages lack nothing, since what it frees is not sized for them; then for a need of bytes whole extents of clean pages less those lanes' first reads (readsShortfall: in decode their hot block floors, counted once for them all, in prefill their staging through the next command, nextCommandPages, which copyDemandBytes now reads too), where they free some; outside --kv-spill always, for a claim that waits, those whose copies land (Pending until they have). The move in place comes first because it moves the fewest pages, and a release for bytes before it could take the pages it needs yet leave the claim short of its Full extent, by an Index extent's 6.3 MB where the room holds none. The units for bytes count what their takes in turn free as the copies stand, each moving its lane's oldest pages: a unit counts only the pages and first reads the units before it leave (Cache::ReleaseCount). Counted for each lane alone, two lanes on one conversation, of 3,000 pages and of those and 1,000 more, gave a start 307 MB more than their takes free. Copies that land before the takes can lower the count, a lane's landed pages then joining, in its whole extents, shared pages a later lane's unit counted: by an extent of pages (102 MB) for two lanes on one conversation. Counted in one pass over clean and landing pages alike, it would hold as copies land, yet the takes a claim makes before they land would free an extent less than it counts. A pass over the take order counts the units once, as it reaches them, not once a unit. So a growth at the wall releases 512 pages, two extents, for its 108.5 MB extent and 40 MB of staging, and the same prepare reserves the reads; a lone decoding lane with fewer than five extents of clean pages releases none and ends at its length. Placement and the in-memory test of a start that cannot spill (residentRoom, both) count the releases of the lanes at the start's priority or lower through its Start or InService rights, and the chain it restores once, held in place. laneDemand's readers are the lanes holding Index pages, in prefill with their staging; gainsIndexPages and LaneShape::spill follow Index pages, every Spill lane only under --kv-spill always. A lone Spill lane's capacity_exhausted says the SSD cache took no more of its KV only where it owes copies no write started, for the quota or the tier, and otherwise that it does not fit even with its KV released down to its floor, beside device_untracked_bytes; a lone Spill decoder ends at its length for either (refuse, endAtLength). - Status: kv_spill.releases and released_pages. - Docs: internals.md's Index runway (a release in place keeps it), take order (stage 5), take-rights table (a Lanes' landed pages column) and the rights paragraph after it (no claim releases a lane of a higher priority), regrowth (a release), Releases, Self-conversion, lane demand, take and Pending, runway refill, scheduling, hot block floor and capacity_exhausted; server.md's take order and regrowth under streamed experts, capacity_exhausted, and KV on the SSD (what a start may take, self-conversion and the host's share of the take order, releases, hot block cache, staging); api.md's kv_spill. - check_architecture: ENGINE_LINE_BUDGET from 29,500 to 29,566: the count of releases (ReleaseCount, countRelease, forReleasable, holdersAt) takes 66 lines; the engine has 29,523. Tests, each failing on the parent, which moves a Spill lane's pages as their copies land, unless marked a guard; "at its precondition" means there the parent's lane held Index pages before any release. The engine spill tests use Flash-Next's proportions: - engine spill/testASpillLaneKeepsLandedPagesWhileMemoryLasts: a 64K Spill lane with memory to spare writes every page through and holds no Index page, floor or staging. The parent moved 2,050 pages. - spill/testASpillLaneAtTheWallReleasesOnlyWhatItLacks: in prefill at the wall it releases 512 pages and holds its staging; no claim waits. The parent's lane held 927 Index pages there. - spill/testTheFirstReleaseFundsTheLanesReads: a decoding lane takes no floor before its first release and holds it from the release's prepare; a Background start at the same wall takes none of it. At its precondition. - spill/testAStartReleasesALaneOfItsPriority: a Normal start at the wall releases a Normal lane's pages and starts beside it, no yield. At its precondition. - spill/testAStartNeverReleasesAHigherPriorityLane: a Normal start waits for a Foreground lane to end, whose pages stay Full. The parent's left memory as their copies landed. - spill/testCriticalReleasesCleanPagesBeforeAnyLaneYields: under critical pressure a host pass of two extents' pages (204 MB) releases a lane's clean pages and takes none of the expert store's chunk above its floor, and the lane decodes on. The parent's pass took the chunk. - spill/testAHostRefusedGrowthReleasesItsOwnLane: a growth the host refuses under critical pressure, with no free Full page, releases its own lane's pages in place and decodes 64 tokens. The parent's lane did not decode them. - spill/testAReleaseInPlaceKeepsTheIndexRunway: after that growth, the host's pass of two extents' pages (204 MB) still releases 512 of the lane's pages, takes nothing of the expert store at its floor and leaves an extent of free Index pages. The parent's lane did not decode the growth's tokens; moved into the runway alone, the pass freed 64 B, released none and left 254 free Index pages. - spill/testAReleaseInPlaceKeepsTheRunwayTheRoomCannotRefill: at a wall with no free Full page and no room, a Normal lane's growth takes none of the 162 pages whose copies landed that an earlier release left a Background lane, too few for an extent more, but releases its own; then a host pass of two extents' pages under critical pressure releases 512 of them and leaves an extent of free Index pages. At its precondition; moved into the runway alone, the runway held 254 free Index pages and the pass freed 128 B and released none. - spill/testAGrowthTakesItsPageInPlaceBeforeAReleaseForBytes: at a wall with no free Full page and no room, a lone decoding lane whose first release left it fewer than two extents of pages whose copies landed moves its page and an extent more in place and decodes on, releasing fewer than 512 pages. At its precondition; with the release for bytes first, it ended with Length after 120 tokens. - spill/testNoReleaseOfALaneBetweenItsPasses: a start at the wall leaves a lane between its command's passes without Index pages. The parent's lane gained them there. - spill/testNoReleaseOfAPageALaneBetweenItsPassesHolds: a decoding lane's growth at the wall, and a start, move none of its conversation's pages while a longer prompt on it, which holds them, is between its command's passes; all three requests complete. At its precondition. - spill/testAReleaseLeavesNoLaneWithoutItsFloor (replaces testAConversionLeavesNoLaneWithoutItsFloor): a Background lane whose pages a Normal start released decodes its next token with its floor. At its precondition. - spill/testAResidentStartOnALanesPrefixRestoresItWhole: a start on the first 1,500 pages of a decoding Spill lane's conversation, placed Resident at the wall, releases the lane's other pages and restores that chain with no Index page. At its precondition; released by the start's own claims, it was restored with 1,499 Index pages. - spill/testAReleaseOfASharedPrefixFundsEveryLanesReads: a start on a decoding lane's conversation, 1,001 pages past it at 50 MB of room, releases their shared prefix and leaves the decoding lane its 390 MiB floor; counting the releasing lane's reads alone left it 68,161,728 B. At its precondition. - spill/testACountOfReleasesCountsASharedPageOnce: two Normal Spill lanes on one conversation, of 3,000 pages and of those and 1,000 more, give a start at the wall what the longer one's release frees (715,653,248 B, measured); a Normal start of 2,300 pages that cannot spill, the SSD cache closed, waits for the shorter lane to end, and nothing yields. At its precondition; with each lane's releases counted alone, the count was 1,022,361,728 B, and the start began beside both, released 3,994 of their pages and yielded. - spill/testACountOfReleasesFallsByAtMostAnExtentAsCopiesLand: two Normal Spill lanes on a 2,433-page prefix, of 769 and 897 pages more, count at the wall at most an extent of pages (102 MB) less once the shorter one's copies have landed, nothing taken: 817,889,536 B, then 715,653,376 B (measured). At its precondition. - spill/testTwoSpillLanesWithoutIndexPagesPrefillTogether: two Spill lanes without Index pages share a prefill command. The parent ran them apart. - spill/testALoneSpillWallNamesWhyItsKvGaveNoMoreMemory: a lone Spill prefill at the limit, Metal memory outside the engine's buffers taking the room and 1 GiB more, names that even a release down to its floor does not cover it where its copies landed, and the SSD cache where the tier took none of them. The parent failed its staging naming neither. - guard spill/testAStartOnALanesConversationCountsItsChainOnce: the start above with 200 MB of room starts Spill, and both requests complete with no yield; counting the chain also as the lane's to release, it was placed Resident, yielded and failed. - guard spill/testALoneLaneWhoseReleaseCannotPayItsReadsEndsAtItsLength: a lone decoding lane of 1,023 pages at the engine's limit releases none of them, four extents of which would free its floor alone, and ends with Length. - guard spill/testALoneLaneWaitsForItsLastWritesThenReleasesThem: a lone lane whose copies are in flight at the wall waits, releases them and completes. - guard cache-property: a release step in every run, half of them under --kv-spill always, by a random rule over the requests it takes pages from and, in half the steps, the chain a Resident start restores: it moves no more than were clean, for bytes only where free Index and Full pages can empty an extent, none of a holder's dense history or window, none a Resident request holds, none of a request the rule leaves out and none on that chain, and gives Index pages only to the holders of the oldest page it counted. - guard cache-property: a count of releases of Spill requests in turn, each of the whole extents of its clean pages, as a claim's units for bytes count them, counts for each no more than it holds clean when its take comes, and a take gives a request its first Index pages only where a count named it. With each request counted alone, plain seed 1 failed: "a count of releases counted 4 pages of request 18, which held 0 clean when its take came". - guard engine-sim testRandomSchedulesWithReleasesEndAsWithoutThem: four 80K-110K requests a seed at 1 GiB under 18G, one on another's conversation in odd seeds, with a critical episode, release 119 times over four seeds, keep the cache's invariants after every step and end with the output and finish reason they have at twice the limit. - Rewritten: memory-ledger testTheOrderTakesCacheThenRebuiltPoolsThenReleasesThenTheStore (renamed): cache, rebuild-only pools, a lane's release, then the store; memory-ledger testTakeRightsTable, which gains the lanes column; engine-cache testAConversionsHistoryWithCopiesIsReleasedAtOnce, where a conversion moves nothing and a release moves the landed history at once, with no allocation; spill/testALongDecodingLaneConvertsItselfAtTheWall, which meets the engine's limit, whose refusal carries the bytes a release nets the lane's floor from; spill/testLoneSpillLaneEndsAtTheWall, whose prefill names its floor: its first command's ten pages exceed the eight memory holds before it has published a page. Engine and cache tests of the moves as copies land run under --kv-spill always, and two engine tests whose lane needed Index pages get them from a release at the wall. Among the first, spill/testALaneTakesCachedPagesInPlace, whose lane's landed copies free the pages its prompt takes: outside it, its lone lane at room 0, with too few pages whose copies landed for an extent more and the runway an extent, frees none in place and fails with capacity_exhausted. Reported by @mperreir in incoai#713.
The open chat had no title anywhere but its row in the sidebar, so renaming, pinning or deleting it meant finding that row, or showing a hidden sidebar. Its title now stands in the window's top row, taking no room from the conversation, after a separator that lines up with the sidebar's divider while the sidebar shows and follows the switch while it hides, moving along as the sidebar slides. It is plain text the pointer highlights, which opens the chat's menu, Pin, Rename… and Delete…, asking the same questions as its row's menu, which also has Select. A new chat, with nothing in it, shows none. The toolbar lays its items out apart from the window's content, so the title measures where the toolbar puts it in the window (`ToolbarProbe`): as it is first laid out in a window, and as a window goes into full screen or out of it. Its item keeps whole points, and one size as the sidebar slides: the toolbar centers an item in a slot of whole points, and a title placed by where its item sits would otherwise chase itself. As the title's size changes, the probe lays the toolbar out, which takes an item's new size only as it lays out. The sidebar's width is a constant both read.
The model's name at the composer's start only opened the Server, where models start and switch. It now opens the models on this Mac, the one that runs checked: choosing one starts it, or switches to it, as the menu bar's panel does (AppState.choose), and Manage Models… opens the Server. With no model on this Mac, or none it can start now (the Keychain refusing the API key), it opens the Server as before. A popup now opens toward the window's middle, from its control's leading edge in the window's left half, and its items scroll where they take more room than it has.
A chat was titled by the first line of its first message, cut at 80 characters, so the list showed the start of long questions. Once its first exchange is answered in full, and while its title is still that first line, the app asks the model that answered for a title of a few words (ChatClient.title): in the background, where the user's requests come first, without thinking, in 32 tokens and not streamed. It takes the first line of the reply, tidied of Markdown's marks, a Title: label, quotes and a closing stop; a reply its tokens cut short gives none. Renaming or deleting the chat cancels the request, and a chat the model gives no title keeps its first line. The test servers answer the title request apart from the chat requests they count.
…r pins Two places where a Spill lane's release took other than its count of it. A release's count (EngineVictims::releases) is net of the first reads it gives the lanes: each page frees its Full bytes less its Index page's, less the hot block floor or staging of every lane that holds no Index page yet. Its take moved what the claim lacked and those reads, then reported every byte it freed, reads included, so free() stopped once the gross covered the claim: a start's claim of all it counted of two Spill lanes' releases on Qwen3.8-Flash-Next counted 1,840,251,008 B and kept 1,431,306,368 B beside the two 390 MiB floors (on one conversation, 715,653,248 B counted and 306,708,608 B kept). A release for bytes moved only where its first runway's moves empty a Full extent by the count of free pages, but compaction leaves a page in transfer where it is, and so its extent. Beside another Spill request's copies in flight in every Full extent, a release moved its pages into the Index runway, emptied nothing, and refilled the runway: memory grew, and the lane gained Index pages and reads for nothing. The rules: - A release's take gives the claim what it frees net of the first reads it gave the lanes (readsShortfall before and after its moves): the measure of its count. - A page in transfer pins its extent, and an empty extent needs no moves. A release for bytes moves whole extents of pages, as many as the Full extents that hold pages and none in transfer, beyond those the releases counted before it, and none unless its first runway's moves fill one (Cache::releaseMoves, which replaces releaseFreesAnExtent): its count and its moves agree with what compaction can empty, and a release that can empty none moves nothing and allocates no refill. A release of pages whose copies land counts the extents as they stand once the transfers land, which its claim waits for. - KvPool::unfixedExtents: a form's extents that hold pages, none of `fixed`: those compactExtent may empty. An empty one needs no moves: counted, it let a release beside writes in every other Full extent move pages that emptied nothing and refill the runway. - Cache: pagesInTransfer, the pages compaction keeps where they are, read by compactExtent and releaseMoves. releaseMoves serves the engine's count (with its ReleaseCount, whose pages give the extents counted before), releasePages and inPlaceMoves, whose extent more takes one too. - Engine: releases offers releaseMoves' pages; take reports its moves net of the reads they created, in place too, and leaves whole extents to releasePages. - MemoryLedger::CountChecks: for the tests that hold the sources to the peek contract, the runs whose count (bound) covered what the claim lacked; of those, the takes that covered less though no unit waited for a transfer and the host refused nothing in the claim's scope (overcounted), and the others (blocked); the releases taken first in a run, and of those the ones that freed less than the lesser of their count and what the claim lacked (short). The contract states that a source's units count disjoint memory. - Docs: internals.md's runway and releases (a take's net, the extents a release may empty). - Tests: FlashNextLanes' page ids from 8,192 to 65,536. Its 32 extents of ids, not memory, limited the pool, and the runway's refill failed for ids with memory to spare; production's ids hold every extent the limit could hold of Index pages. - check_architecture: ENGINE_LINE_BUDGET from 29,566 to 29,605: the CountChecks the release tests read take 40 lines, the rules 42; the engine has 29,605. Tests, failing on the parent unless marked a guard: - engine releases/testAClaimOfTwoLanesReleasesKeepsWhatItCounted and ...OnOneConversationKeepsWhatItCounted: a Normal start's claim of all it counts of two decoding Spill lanes' releases at the wall is met and keeps that count beside the floors the releases gave the lanes. The parent kept the bytes above. - engine releases/testAReleaseBesideWritesInFlightFreesWhatItCounts: two Spill requests took their 12 pages in turn in 4-page extents, and a third an extent of pages it left empty; while the second's copies are being written the first's release of its 8 landed pages moves none and allocates nothing, and once they land it moves the 8 and frees the 720 B its count gives them, and the empty extent's 400 B. The parent moved the 8 beside the writes, emptying none of the extents that hold pages, and took the pool from 2,840 B to 2,520 B: the empty extent left beside two refills of the Index runway. With the empty extent counted as one compaction may empty, the release moved 4. - engine releases/testRandomConversationTreesKeepTheReleaseCounts (guard): two to five requests as a conversation tree, of each priority, in Flash-Next's page bytes scaled to 16-page extents, at a random wall, the SSD cache closed for a third of the later starts, commands in one pass or four, the host normal, critical or refusing, and random claims between commands (a lane's growth of bytes or of pages, a start, the host's critical pass, a need the host refused). With no transfer in flight: a count that covered its need is met by its takes (CountChecks), the engine's own runs too; a met claim keeps its need beside the first reads its releases created; a release taken first frees what it counted; between commands at normal pressure the Index runway holds an extent where the room does; a start placed Resident beside a lane prefills with no yield and no short claim while the host and the wall stay. 20 seeds by default, SPLASH_RELEASE_SEEDS more, SPLASH_RELEASE_SEED one with its decisions printed; 300 seeds pass. Counted apart, as transfers in flight are: the engine's runs for a need of bytes and pages together that went short, where the cache's walk in place counts whole extents and pages in place while each take returns once it has a page more in place. That walk predates the releases and is not changed here (2 runs in 20 seeds, 14 in 300). - engine releases/testAStartPlacedResidentBesideTwoLanesKeepsTheRunway, ...OnTwoLanesReleasesPrefillsWithoutAYield and ...OnALanesConversationPrefillsWithoutAYield (guards): a start that cannot spill, placed Resident beside two lanes or on their conversation, prefills with no yield, and the runway holds an extent between commands. - cache-property: a release of whole extents frees what its count gives them (releaseMoves), beside transfers in flight too, wherever the runway held an extent and its refill passed (31 releases, 8 beside transfers). - engine-sim testRandomSchedulesWithReleasesEndAsWithoutThem (guard): its four requests as a conversation tree and a short turn on the first conversation's opening pages; no claim whose count covered its need takes less (CountChecks), and none yields at the tight limit outside the critical pressure where none yields at twice the limit. SPLASH_SIM_RELEASE_SEEDS runs more seeds. - memory_ledger testUnitsThatCountOnePageTwiceOvercount (guard): units that each count the pages two lanes share are taken on a count their takes do not meet and named overcounted; counted once, none is taken.
An empty chat list and a search that matched nothing both left the sidebar blank, and a search could be cleared only by deleting its text (incoai#685). The list now says No Chats Yet, or No Results for the search, and what to do about it; the search field has a button that clears it, as Esc does, which keeps the open chat and the draft. The search's rule, a chat whose title or a message holds the query, is now the store's (`ChatStore.matching`), which a test covers.
Past the memory wall a Spill lane keeps its landed pages Full until a claim releases them. The KV past a prefill command's checkpoint is an Optional claim that could release no lane's pages, so wherever it needed a new Full extent it was refused and cut the command there; the lane's next command then released the same pages. On Qwen3.8-Flash-Next at 13G a 237K prompt ran 14 of its 36 commands at 4,096 rows instead of 8,192, and each command streams the routed experts of every layer whatever its rows: 957 GB of expert reads against 799 GB, the first token at 198.7 s against 1.6.0's 182.7 s. That claim is memory its lane takes anyway (laneNeeds), so it now releases its own lane's landed pages, as a need the host refused already does. The same prompt runs 29 whole commands and no cut: first token at 179.3 and 179.4 s, 18 GB of KV read back against 1.6.0's 45 GB. Prefill commands past the wall run whole again, as on 1.6.0: the simulated churn's longest decode gap is 722 ms against 1.6.0's 702 ms, where the cuts left 827 ms.
A lane prefilling in commands of 6,144 rows meets the wall where its extent holds the next command's rows up to the checkpoint and no more, while it holds no Index page. The KV past the checkpoint then makes the lane's first release: the command is planned again to stage the lane's history, and runs whole. Without that release the claim cuts 3 commands.
A restore wrote each image on the engine's loop thread, one per tick, so the loop answered nothing while one was written. Flash-Next's PLE table loaded whole (--ple-memory off, or auto where memory holds every expert and the table) is one image of 28.8 GB; written from a disk slower than about 1 GB/s it outlasts the 30 s within which the server expects a status answer while a request is pending (supervision::kLoopStall), and the server stops the engine: the request that found the weights released fails with 503 runtime_unavailable, the engine starts over, and the same happens after every idle release. On the 48 GB M5 Pro with --ple-memory off, a 45 s debug delay in ple.bin's write failed the request after 32.4 s, and again after 35.2 s on the relaunched engine; with this change the request completes in 51.1 s and every status probe is answered meanwhile. WeightImages::restore admits an image's memory on the loop thread and writes it on a thread of its own, which calls the loop's completion notifier as it ends; the next call collects the write, its failure included, before it takes back the next image. The writer may allocate and run commands on the backend, which takes those from one thread at a time, and a KV extent's release refuses any command in flight, so while an image is written the engine leaves memory and the backend alone: a tick only ends requests at their deadlines, the control pass waits as for a command in flight, and the loop wakes for nothing else. Normal restores take as long as before on that machine: 4.93, 4.89 and 4.93 s against 4.91, 4.89 and 4.96 s.
The buffer after a host step waited on the GPU for an event the runner raised once the step returned. Metal fails a wait still unmet 5 s after its queue reached it, so a step that worked longer, such as an expert load or a spilled KV read from a slow external disk, failed its command with "Caused GPU Timeout Error" and the engine with it: every request in flight failed and the server relaunched the engine. The runner now commits the buffers after a step once the step ran or was skipped, in the order they were encoded; until then they are held. No GPU wait spans a host step, so only the command watchdog bounds one, and the second event that the runner raised goes away. An awake step's spin still keeps the GPU awake while the work after it is committed.
--ple-memory auto loaded Qwen3.8-Flash-Next's 28.8 GB PLE table whole wherever the budget held it beside the experts and the KV allowance, though a row cache of its share, the rows of every id a 128K-token prompt stages (237 MiB), hits as often as one of 2 GiB. Auto now streams a table larger than its share through it at most however much memory the Mac has, and the rest goes to the KV, the lanes' states and their caches. off still loads the table whole, and a table no larger than its share loads whole where the budget holds it.
The Context length menu offered 16K to 256K tokens, so a 1M context was reachable only by typing it under Custom…. The menu now offers the lengths the model picked serves: 1M tokens for a family whose window YaRN stretches that far (the Qwen families, up to 1,010,000 tokens), not for Flint, whose window is its most. A length past the model's own window says under the row that YaRN stretches it, which changes answers at every length, as the sheet for a typed length already did.
Two requests on one prompt prefix that then diverge, an agent and its subagent on one context, failed at the memory wall where each completes alone. On Qwen3.8-Flash-Next Q2_0 on a 48 GB M5 Pro, a 105,672-token request beside a 97,849-token one on its first 89,792 failed capacity_exhausted at --max-memory 9.38G (twice) and 9.43G: its hot KV cache could not be allocated. Alone it completes at both limits, and 1.6.0 served the pair at both. A claim counted a Spill lane's release in whole extents of the lane's own pages, yet compaction gathers the free Full pages of the extents that hold pages with the slots the moves free. The longer lane resumes beside the shorter one's cached tail with 511 releasable pages and 3 free Full pages: its release for the 390 MiB hot block floor counted one extent where compaction empties two, and the floor was refused. The lazy release leaves those pages Full for a claim to count; 1.6.0 moved them as their copies landed. A release now also counts the extent its last pages complete with those free pages (Cache::releaseBeside). Its take moves whole extents of its own pages first and the free pages complete one only where those run short; fewer than an extent of them lie beside, the count carries what is left of them from lane to lane (ReleaseCount), and each extent so emptied frees its Full bytes less an Index page's for each of its pages, no less than the unit counts. They count only where nothing else does: not where the pages only the cache holds would complete an extent with them, as its reclaims and returning demotions count them then, not for a need that takes the free Full pages itself, and not under --kv-spill always. On the same machine the pair now completes at 9.38G and 9.43G, twice each, the longer and the shorter request in 136.4/116.0 s and 136.7/116.8 s at 9.38G, 136.3/116.5 s and 135.7/116.1 s at 9.43G, where 1.6.0 took 137.8/117.9 s and 137.6/117.7 s. The longer request alone completes at both (125.6 and 125.3 s), and the pair at 9.5G runs as before, 136.4/116.4 s with the same 10 releases. In the engine tests the diverging pair failed from 650 to 725 MB, where lane 1 alone and two lanes of their own contexts complete from 650 MB and lane 2 alone from 625; it completes there now, and two lanes on one prompt still complete at the wall with the memory the longer completes with alone. A host's pass at critical pressure now takes a Background lane's 162 landed pages with the free pages beside them for one of its two extents, before the Normal lane's pages, so its test checks the extents emptied. The cache's release guard checks the free pages too: over 4,000 seeds 1,076 of 3,721 releases completed an extent with them, none freeing less than counted with the runway whole. The engine's line budget rises by the 53 lines this adds.
The engine could fail with "unknown KV cache block", thrown out of a tick: every request in flight fails and the server relaunches the engine. The release property test met it at seed 5219 of 60,000: a request on a lane's conversation, refused memory at the memory wall, backs off while the host's pass at critical pressure takes the cache. 1.6.0 places such a start the same way. A start refused memory backs off with the probe it was placed with (Engine::admitQueued ranks it by its last probe), and placement counts what a start may take beside the chain that probe restores (Engine::residentRoom), whose walk follows the chain up from its state's block (KvCache::countReach). A reclaim that takes certain reuse, the host's pass at critical pressure or a lane's claim in service, may take that chain meanwhile, leaf first: a state whose write the SSD cache does not take goes, and then its block where no copy of its KV was written. The control pass brings the probes up to date (Engine::collectLandings) before the host's pass, and the next placement, which any change of the ledger calls for, read a chain whose last block was gone. In seed 5219 the host's pass at tick 117 took the end of request 3's 310-block chain, its state's block among it, and the pass at tick 119 released a Spill lane, so the next tick placed request 3 again and KvCache::block threw. A probe's chain is now read as far as the cache still has it, up to its first block no longer matchable, in one place (Cache::matchableChain), and every reader of one reads it through that: residentRoom's walk and the pages it holds in place, which read it whole, and refresh, the quota's and the Index pages' reads (Cache::restoreTakesQuota, restoresIndexPages) and residentChainPages, which each cut it with a loop of their own, the last skipping missing blocks rather than stopping at the first. A chain the cache still has whole is read as before. Seed 5219 passes. testAPlacementCountsOnlyTheChainAStartKeeps reproduces it in a few ticks: an earlier turn's 16 pages cached, its resend refused memory, the control pass at critical pressure taking the chain, then the memory wall; the tick threw, and the start now waits on and completes once memory returns. Over seeds 1 to 60,000 the release trees fail at the same 14 seeds as before, with the same lines (invariants 2 and 3), and no longer at 5219; make test-engine-cpu passes. The engine's line budget rises by the 4 lines this adds.
…ages At the memory wall a lane's growth that lacks a Full page, with none free and no room, counts two units of the take order on the same free Index pages: the cache's reclaim of the empty Index extent past the runway, for bytes, and its lane's release of the page in place into those free pages (Cache::inPlaceMoves). Taken in turn, the reclaim released the extent first, the release found only the runway and moved nothing, and the run freed less than it counted: the release property test's invariant 2 failed at seeds 25932, 33424, 36841, 38424 and 54434 of 60,000. A lone Spill lane in that state ends at its length: in the new scenario it stopped after 24 of its 400 tokens. Beside other lanes the one that yields first is suspended instead. A release in place now moves the claim's pages beside the runway, into the free Index pages past an extent of them or into an extent allocated for them where those do not hold them, and refused that extent none moves (KvPool::movePages): the extent comes back out of the room the reclaim left. Where its count found the free pages it allocates nothing, as before, and an extent more still moves where the count found that. The path of moveLandedPages that moved only into free Index pages goes. The five seeds pass and the scenario's lane decodes its 400 tokens. The engine's line budget rises by the 10 lines this adds.
A start kept its placement for as long as the ledger's state stood (placeCandidates), and an attempt to start it in that state took it. The room under the limit moves with the memory outside the engine's buffers between the control passes that change that state, so a start backing off from a refusal, placed while the room held its Resident KV, started Resident on room that had gone: in the release property test seed 14594's start yielded in its prefill (invariant 3) after the room fell from 186 MB to 41 MB between its placement and its start, one tick apart. An attempt to start a request now places it in the memory as it is then; planning still reads the placement of the ledger's state. In the new scenario a start placed Resident backs off while the room falls to half its Resident KV: started on the old placement it went on Resident and converted itself in its prefill; it now starts Spill, and no lane converts or yields. Seed 14594 passes. The engine's line budget rises by the 4 lines this adds.
The release property test's invariant 3, that a start placed Resident on a Start claim's count prefills with no yield, no short claim and no conversion, failed at 8 of 60,000 seeds where the engine kept to its rules and the test was stricter. A start of a higher priority began after the placement. Its claims rank above the placed start's, which is the lane that yields first, or converts first (ClaimRank, Engine::laneToYield): seeds 12540, 17665, 17799, 36317 and 46851 yielded, 44298 converted. The test's comment excepted a yield to a start of a higher priority, but its check counted only a preemption at admission. A Spill lane's prefill command took pages after the placement. The lanes' projected growth prices a Spill lane at its floor (Engine::laneDemand), and its rows stay Full until claims release them once their copies land, so no count holds them, as none holds a resumed lane's replay. At 42028 an equal-priority lane's 96-page command right after the start took the room it counted, and the start converted in its prefill. At 16675 a resumed Background lane's replay took 191 pages; the start's claim went short once, and the Background lane yielded to it. The test also dated a placement by the start's event, which a start that restores its chain from the SSD sends only once the restore lands (Engine::completeAdmission): at 16675 two ticks after admission placed it, with the replay's command between them. The test now dates a placement at admission, and judges a placed start while its count holds, under one rule: a move of the host or the wall, or a resume, ends it for every placed start; a start of a higher priority for the starts below it; a Spill lane's prefill growth for the others. Over 3,000 seeds 254 of 449 placed starts are judged through their prefill; moves and resumes end 180, starts of a higher priority 8 and Spill prefills 7. With the two engine fixes before it, a sweep of seeds 1 to 60,000 fails only 5219, whose unknown KV block is fixed apart, where 15 failed before.
The release notes gain the engine's two slow-disk fixes (a host step past 5 s, a weight restore from a slow disk), the end of the "unknown KV cache block" failure, Flash-Next's PLE table streaming under auto on every Mac with --ple-memory off to keep it in memory, and the 1M context length in the app.
The automatic table's rule in one statement: its row cache takes what the allowance leaves, up to its share, and a table that cache would hold whole loads whole. The share's cap already applies in tableShare, so the test of a table larger than its share restated it.
The sentence on what YaRN costs past a model's window is said once (ContextLength.yarn), by the sheet for a typed length and the row's caution alike.
The design docs still described the removed wait. docs/internals.md had a host step hold the work after it through a shared event, and the work between two host steps end within Metal's 5 s; docs/models.md had the 5 s count against a prefill command's first host step. Both now follow the code: the backend commits the buffers after a step once it returned, so no wait spans a step and only the command watchdog bounds one, and Metal's limit holds the Neural Engine's waits, between two of which run a layer's GPU part and an evaluation the split bounds at 2 s, and an awake step's spin, which waits for the decode work before its step. Runtime::prefillAsync gave the Neural Engine's waits as the only reason a prefill command's image encodes run as a command of their own, yet every prefill with images splits. The split also times the encodes apart, so that the scheduler leaves them out of the time it sizes later prefill commands by (Scheduler::complete) while it counts them in what the prefill owes decode; the comment and both docs now give both reasons. The backend's comment on waits says that a command's first wait after a host step is reached once the step returned, not as the command is submitted.
The loop's completion notifier reached the cache once but the weights through a `written` parameter of every WeightMemory::restore call, which every implementer and test fake carried and ExpertStore ignored. The engine now sets it once, as for the cache: Engine::setCompletionNotifier hands it to WeightResidency, which hands it to the weights, ReleasableMemory to each of its parts, and WeightImages keeps it for the writes it starts. WeightMemory's default ignores it, since memory written before restore() returns never calls it, and restore() takes the admission alone again, in every implementer and test fake. Engine::nextWakeupMilliseconds scanned the requests' deadlines twice, once in a branch of its own while an image is written. One loop now takes every request's deadline and, while an image is written, skips the request's other wake times, as the engine's own (the health check, the drain's end, write-behind) are skipped: the times are those it returned before. ENGINE_LINE_BUDGET is the new count, 29,960.
The cut lived in Cache::matchableChain, which each reader of a probe's chain had to call: residentRoom cut the chain and residentChainPages cut it again, restoreTakesQuota and restoresIndexPages cut theirs, and the release rule (EngineVictims::releaseRule) read the chain raw. It moves into the probe's one accessor: CacheProbe::restoredChain gives the chain as far as the cache still has it, cut at its first block no longer matchable once the KV graph's generation has moved past the probe's (CacheProbe::matchableChain, with which refresh also cuts the probe's blocks). Every reader of a probe's chain reads it there, so none can read a raw stale chain: residentRoom, residentChainPages, restoreTakesQuota and restoresIndexPages no longer cut, and Cache::matchableChain is gone. The probe reads its cache's KV graph as a friend of Cache. While the generation stands every block a probe matched still matches (KvCache's generation), so the chain is cut where those readers cut it before. The release rule, which compared the raw chain by block id, keeps no block past the cut either: a block the cache took is on no page, and a poisoned one and those below it no start from the probe restores. Two cache tests read a chain the cache no longer has through those functions directly: testARestoreReadsAChainOnlyAsFarAsItMatches reads the stale probe's chain through restoredChain, and testAReleaseOfWhatALandingCompletesCountsOnce finds that the claim took the dead page's extent by the pool's free pages rather than by the page of the block it erased. Folded into its commit, make test-engine-cpu passes and the release sweep over seeds 1 to 60,000 fails at the same 14 seeds with the same lines as without this. The engine keeps its line count, so its line budget stands.
The fix placed a start afresh in admit while placeCandidates kept a placement per state of the ledger (ledgerChanges_), and admitQueued placed the same start that cached way just before its attempt: whether the start was held behind a quota refusal (competesForQuota) read a placement its attempt then discarded, and the comments still said the counter named what a placement reads. Each attempt now has one placement, made in admitQueued's loop over the starts the scheduler wants, in the memory as it is then, and both readers take it from there (Request::placement): competesForQuota, and admit for where the start's KV goes, which no longer places. Planning places every candidate in each pass (laneShapes). The cache goes rather than gain the room under the limit as a key: a placement reads more that no change of the ledger names, such as the staging a Spill lane's next prefill command takes (laneDemand), so no key short of the memory itself would keep it true. ledgerChanges_ and its seven increments, Request::placementState and placeCandidates are gone, and so is the check of a claim's wait in deferWait, which only guarded an increment (setWaiting(false) changes nothing on a claim that does not wait). admit set a start's output limit back to maxNewTokens before it placed the start, and the placement reads the start's projection: placed in the loop, before admit, it would read a bound an earlier attempt that did not start left. A placement now projects the whole output itself (projectedTokens with maxNewTokens), as admit's reset gave it. The cost: over seeds 1 to 6,000 of the release property test the engine places 64,097 times instead of 45,937, 1.99 placements in an admission pass with candidates instead of 1.42, and the placements take 0.41 s instead of 0.35 s in all, 12.8 us a pass with candidates instead of 10.8. In engine-sim it places 8,134 times instead of 6,649 over 109,475 passes, the same 5,505 of them reading the room (residentRoom), in 0.122 s instead of 0.120 s. Both runs make the same admission passes over the same candidates, and engine-sim reports the same numbers. testAStartIsPlacedInTheMemoryItStartsIn and seed 14594 pass, make test-engine-cpu passes, and the release sweep over seeds 1 to 60,000 has no failure. The engine's line budget falls to 29,925, the count with both fixups.
The release property test's invariant 3, that a start placed Resident on a Start claim's count prefills with no yield, no short claim and no conversion, held only because the test stopped judging the placed starts once a Spill lane's prefill took pages. Without that exception it fails at seeds 16675 and 42028 of 60,000. The lanes' projected growth (Engine::laneDemand), which placement leaves the lanes and a pool's growth leaves free, priced a Spill lane at its floor: past its dense history, tail window and two groups, an Index page for each page more. A prefill command writes its rows to Full pages, which stay Full until claims release them once their copies land, so the lane's next command takes Full pages no count held. At 42028 a resumed lane's 225-page replay command came right after a start beside it was placed Resident: priced at 5.5 MB, it took 95 MB of the room the start counted, and the start converted itself to Spill in its prefill. At 16675 a resumed Background lane's replay took 191 pages and the start's claim went short. laneDemand now counts the pages of a Spill lane's next prefill command, as many rows as one command holds within its prompt (nextCommandPages), Full, as a Resident lane's growth, and its floor grows from their end. A decode step keeps the floor's count: its page comes as the tail window lets go of one whose copy landed long since. The Full pages join the Resident lanes' in the start's need (Engine::residentRoom: the free pages first, then whole extents) and in what a pool's growth leaves free. The test's exception goes; a move of the host or the wall, a resume or a start of a higher priority still ends a placed start's judging. Seeds 16675 and 42028 pass, and the sweep of seeds 1 to 60,000 breaks no invariant. Over those seeds 8,894 starts are placed Resident beside lanes on the count, 67 fewer, and of their 210,365 requests 28,966 fail, 25 more (33 seeds fail more, 16 fewer): in those seeds 22 starts that cannot spill are refused at their start, where memory does not hold them beside a prefilling Spill lane, against 7, and 11 Spill lanes stop as the SSD cache, closed in the test until such a start starts, takes no more of their KV, against 4. engine-sim's churn keeps its longest decode gap, 721.8 ms of 837.7. On Qwen3.8-Flash-Next Q2_0 on a 48 GB M5 Pro the diverging pair took 136.4 and 136.2 s at --max-memory 9.38G and 9.43G (137.1 and 135.8 s before) and the 237K-token incoai#713 prompt at the wall 283.6 s (284.0 s), with the same releases and suspensions; every request completed and none was capacity_exhausted. The engine's line budget rises by the 14 lines this adds.
A release's count completes its extents with completedExtents, the rule the cache's reclaims and landings count extents by, instead of its own (pages + beside) / extent: ReleaseCount::beside now holds the free Full pages beside the next release's moves, those releaseBeside found and the pages each counted release frees (countRelease adds them, where it took off what completed the last extent), and completedExtents leaves out the extents those completed already. releaseBeside reads the free Full pages the claim's walk keeps for it (Walk::keptPages) instead of deriving them again from the need, so it takes the claim's query; the take asks it for every release, as releasePages reads the pages only for bytes. The release paragraph of docs/internals.md says what the pages beside do, and names ReleaseCount, which is no longer Cache's. The deferral stays the test it was, its reason now in its comment. The walk's reach of the Full form (Walk::reaches) is no test of whether the cache counts those pages: an empty Full extent alone puts the form in the reach of a claim that keeps no Full runway, and beside that extent nothing counts them. Along the release property test's seeds 1 to 60,000 the two tests disagree 94,540 times with pages beside: 92,407 times so, in 18,734 seeds, and 2,133 times, in 583 seeds, where pages of chains states in use keep, which the walk's class does not take, made the old test defer. Deferred by the walk's reach, 1,315 seeds ran otherwise, with 516 fewer releases (293,356 of 293,872) and 12 more failed requests (28,953), no invariant broken. As it stands the count is the same: over seeds 1 to 60,000 every seed's summary is the head's (293,872 releases, no failure), the cache property guard counts the same 32 releases of whole extents, 5 completed by free pages beside, and engine-sim the same 164 releases and longest churn decode gap (721.8 ms of 837.7). The engine's lines are unchanged.
…es into The claim's walk keeps the free Index pages a lane's release of its pages in place counts on, where the take moved them into an extent allocated back. The engine's release source names them (EngineVictims::movesInto): where the claim counts a lane's release of its Full pages alone, those pages and the runway past them (Cache::inPlaceIndexPages, the test inPlaceMoves reads too). The ledger hands what its sources name to every source with the claim's query (VictimQuery::movedInto, filled by claimQuery for free() and available() alike), and walkFor keeps them beside the free pages the need's attempt takes. The cache's reclaims then neither count nor release that Index extent, and the release's count and take read one rule. releasePages loses its second path in place (KvPool::movePages with an extent held back, which repeated moveLandedPages' move, follow and count lines and allocated an Index extent no count held). The claim's pages alone go through moveLandedPages again, into free Index pages only: its path without extents is back, since gathering would compact the Full pages the claim takes in place for no extent. inPlaceMoves no longer speaks of the room that path took, and docs/internals.md says the claim keeps those pages. The cache alone cannot tell whether a lane releases in place: keeping the pages for every claim that may release lanes changed 183 of 60,000 seeds, in claims with no Spill lane, with lanes short of the claim's pages or lacking their first reads. Named by the release source they are kept only in claims that count such a release, which releases() then computes once more for a need of Full pages. There the cache no longer counts the extent the release needs, and the release moves the claim's pages alone, where the head's reclaim released the extent and the release then moved an extent more or allocated one back. Seeds 25932, 33424, 36841, 38424 and 54434 and testAReleaseInPlaceAfterAReclaimOfItsIndexPagesFreesWhatItCounted pass. Over seeds 1 to 60,000 no invariant breaks and 42 seeds run otherwise than at the head: 293,924 releases against 293,872, the same 181,424 requests completed and 28,941 failed. engine-sim's churn decode gap stays 721.8 ms of 837.7 and its random schedules make the same 164 releases. The cache property guard, whose releases in place come with no count, counts 39 releases of whole extents, 6 completed by free pages beside, as it did before 8025098; with the allocating path it counted 32 and 5. The engine's line budget rises by the 41 lines this adds.
This reverts the pricing of a prefilling Spill lane's next command in the lanes' growth (Engine::laneDemand). On Qwen3.8-Flash-Next Q2_0 at --max-memory 9.4G on a 48 GB M5 Pro, two distinct streamed prompts of 87,682 and 79,475 tokens sent 0.5 s apart were both admitted at once with it and prefilled side by side; the shorter's 390 MiB hot block floor then came up short at its decode's start and it failed capacity_exhausted, twice, though it fits alone. Without it the second start waits until the first's prefill ends and both complete (105.7 s and 200.3 s to their first tokens), as 1.6.0 and the commits before it do. The release property test's exception for a Spill lane's prefill growth comes back with it: a start placed Resident beside a prefilling Spill lane can still convert in its prefill, or the lowest-ranked lane yields to it, which costs time rather than fails a request.
The pricing of a prefilling Spill lane's next command in the lanes' growth (Engine::laneDemand) was reverted for a failure on Qwen3.8-Flash-Next Q2_0 at --max-memory 9.4G on a 48 GB M5 Pro: of two distinct streamed prompts sent 0.5 s apart, the second started beside the first with it and failed capacity_exhausted at its decode, and waited for the first's prefill without it. Nothing that reads the lanes' growth held that start. The scheduler did: a prompt starts beside another's prefill only where it leads the next command, with fewer rows left than the other has past the command between its passes (Scheduler::prefillAdmissionOrder), and the 48G's timeline shows the waiting prompt queued, not waiting for memory. The long tests' driver draws its prompts from the clock, so each run had prompts of its own: on the 48G the three runs that failed had the second prompt 15, 60 and 180 rows shorter than what the first had left past its first 8,192 rows (87,682 and 79,475 tokens in the bisect's failing run), and the four that passed had it 283 to 1,189 rows longer, two of them on the revert. On a 36 GB M5 Max the revert started the second of two prompts drawn to lead (85,669 and 77,131 tokens, 346 rows) beside the first at once, where both completed, and with this commit the second of the driver's own pair, 9 rows behind (85,276 and 77,093 tokens), waited for the first's prefill. In the engine's test executor with Flash-Next's prefill groups (256 pages), a 2,740-page prompt and a 2,483-page one submitted while the first's first command runs fail so with either pricing: they prefill side by side, each command's claim releasing the other's landed pages down to its floor, and from 690 to 720 MB, where each completes alone, the shorter's 390 MiB hot block floor at its decode finds nothing to release; the prefilling lane yields, but its pages stay in the extents it shares with the decoding lane's, and the decoding lane, then alone, fails. One page longer, the second waits for the first's prefill and both complete. That failure stays: it is the engine's at the wall however the lanes' growth is priced. So laneDemand again counts the pages of a Spill lane's next prefill command, as many rows as one command holds within its prompt (nextCommandPages), Full, as a Resident lane's growth, and its floor grows from their end: a prefill command writes its rows to Full pages, which stay Full until claims release them once their copies land. Priced at the floor, a resumed lane's 225-page replay command took 95 MB of the room a start placed Resident beside it counted, and the start converted itself to Spill in its prefill (release property test seed 42028; at 16675 a Background lane's replay took 191 pages and the start's claim went short). The release property test's exception for a Spill lane's prefill growth goes again. Seeds 16675 and 42028 pass without the exception, the sweep of seeds 1 to 60,000 breaks no invariant, the engine's CPU suites pass, and engine-sim's churn keeps its longest decode gap, 721.8 ms of 837.7. On the M5 Max at --max-memory 9.4G it completed every request of the long tests' pair_distinct, pair_shared and long (147,245 tokens). The engine's line budget rises by the 14 lines this adds.
Metal allocates a buffer in whole host pages and the memory ledger charges what it allocates, but the decode arena's plan (decodeArenaBytes) counted its gate and linear scratch buffers at their fields' sizes. On Qwen3.8-Flash-Next Q2_0 the 122,880 bytes of linear sums take 131,072: the 8,192 bytes /status reports as memory_audit.backend_unclassified_bytes, which the plan left out and the governor charges beside the runtime reserves. The streamed stores take all that the KV allowance leaves, so at --max-memory 9.4G the plan held one request's KV in nine extents of 256 pages with one byte to spare, and those 8,192 bytes cost the ninth: with the SSD cache off the start advertised 73,721 tokens of context while one request could use eight extents, 65,529 tokens. On the Studio's M5 Max a lone 66,171-token prompt failed capacity_exhausted 47 s into its prefill (additional_pages=147, free_pages=128), as on the 48 GB M5 Pro, a 65,490 one completed and a 65,560 one failed at its 2,049th page. 1.6.0 advertises the same 73,721 tokens and fails a prompt of all of them the same way. The arena now allocates each scratch buffer in whole pages and the plan counts them so, which drops LinearScratchSize::bytes, the fields' sum. The plan then counts every byte the backend allocates: no unclassified bytes for Qwen3.8-Flash-Next, Qwen3.8-27B and Qwen3.6-35B-A3B, and the stores take the 22,512 bytes the arena gains. The same start advertises 73,721 tokens again, which one request uses whole: a 73,689-token prompt with 32 tokens of output, all of the context, and a 72,246-token one complete on fresh servers. model-execution-plans checks that the decode arena plans its buffers in whole pages, and runtime-resources that a start's backend holds no buffer the plan does not count; without the change the first fails and the second finds the same 8,192 bytes on the synthetic model. The engine tests, test-engine-cpu (engine-sim's churn decode gap 721.8 ms of 837.7) and the release property test over seeds 1 to 60,000 pass, as do runtime-resources, model-loading and pressure-serving under shader validation. A 32,768-token prompt with 256 tokens of output on a fresh 9.4G server runs as before over two ABBA rounds: 22.19 s to its first token and 14.69 tokens/s, against 22.09 s and 14.75.
On Qwen3.8-Flash-Next Q2_0 at --max-memory 9.4G with --max-cache-disk 600M, whose 600 MiB of room for active KV takes no 60K-token request, the start advertised 65,529 tokens of context and refused lone prompts of 60,000 and 65,497 tokens (kv_disk_quota): their Resident lane needed 878,444,544 bytes where memory had 790,124,059. A start the SSD cache cannot take goes on in memory where its claim holds its Resident KV (Engine::inMemory), and that claim (Engine::residentRoom) charged the start a captured state's buffers beside its lane's: 118,751,232 bytes that the memory plan never counts, as it holds one request's KV beside its lane's state, and that a captured state takes only where memory has them; refused, it is written from the lane to the SSD cache or not kept. Lone prompts of 57,337 tokens or more were refused. A start is now charged its lane's buffers alone, those the pool lacks. That left two gaps between the plan and the claim, both memory that a start that can spill holds whatever its requests do: the Index runway, the extent no claim takes (6,291,456 bytes), and the page tables the readers of spilled KV map their pages through, the hot block cache's location pages and one for each history staging buffer (589,824 bytes, the memory audit's backend_unclassified_bytes). The resident context counted neither, so wherever the room past one request's last extent was less than their 6,881,280 bytes it advertised an extent no lone start could take. The readers now allocate their tables in whole host pages and the plan counts them as they allocate them (KvBlockCache::tableBytes, KvHistoryStream::tableBytes): among the spill floors, where a second staging buffer the readers' memory holds already takes its table before the stores do, and as a memory audit category of their own. The resident context is the whole extents the budget leaves beside the lane's state, the runway and the tables (KvSpillPlan::heldBytes), what the claim of a lone start finds. At 9.4G the start still advertises 65,529 tokens, and lone prompts of 50,000, 60,000 and 65,497 tokens with 32 tokens of output complete on fresh servers, where only the first did. At --max-memory 10058861118, which leaves 3,000,000 bytes past eight extents beside the lane's state, the start advertised 65,529 tokens and refused lone prompts of 60,000 and 57,305 tokens, the second needing 769,916,928 bytes where memory had 755,812,032; it now advertises 57,337, all of which a 57,305-token prompt with 32 tokens of output takes. The audit finds no unclassified bytes at either budget. spill/testALoneStartWithinTheResidentContextStarts starts a lone prompt at the resident bound in the room the plan leaves it, and in up to an extent less a byte more, whichever way the SSD cache refuses it; charged a captured state, it is refused in the first. The memory plan's testResidentContextIsWhatALoneStartTakes advertises seven extents exactly where the room past eight beside the state is short of the runway and the tables, on the M5 Max and the M5 Pro; counted beside the state alone, it advertised eight there. runtime-resources' new test finds a spilling start's tables the plan's and nothing the plan does not count. Three placement tests and qwen-state-storage's testStartBytes no longer charge a start the state its prefill captures. The engine tests, test-engine-cpu (engine-sim's churn decode gap 721.8 ms of 837.7, as before) and the release property test over seeds 1 to 60,000 pass, as do runtime-resources and qwen-state-storage under shader validation. The engine grows by 62 lines, the readers' tables in the plan and the audit and the reasons beside them; ENGINE_LINE_BUDGET rises by as much.
A lane alone failed capacity_exhausted whenever its claim's count fell short of what it lacked, having taken nothing, though the count adds up each unit in whole extents of its own and their takes can free more together. On Qwen3.8-Flash-Next Q2_0 at --max-memory 9.4G on the 48 GB M5 Pro (context 170,425), two distinct prompts of 87,682 and 79,475 tokens prefilled side by side; at the shorter's decode the longer yielded, and the shorter, then alone, failed its 390 MiB hot block floor "short (0.0 MiB now, 0.0 later)" with nothing evicted, where 1.6.0 completes the pair. In the engine test that reproduces it, at 700 MB that count of the pages the yielded lane left cached and of the lane's own release came to 219 MiB of the 290 MiB it lacked, where their takes free 316.5 MiB. A claim with no other lane in service (ClaimSpec::lone), whose need no yield can meet, now takes what the order gives whatever the count where it lacks bytes, and waits while it takes something; Engine::judge fails its lane only once no write is in flight either. capacity_exhausted so comes only once every cached unit the claim may take is gone, the store is at its floor and no copy is on its way, as its message says. The wait for writes is a resource wait: copies that never land end it with resource_timeout, and a write that fails stops the SSD cache's file, after which the lane fails at once. Beside other lanes nothing changes: the lane that yields first yields, and counts still decide when a claim takes and retries. Without the change testTwoDistinctSpillPromptsAtTheWallCompleteWhereEachDoesAlone fails at 690 to 720 MB and testALaneAloneOnTheChainAnotherLeftCompletesWhereItDoesFromIdle at 670 MB; testALoneLaneFailsOnlyPastWhatItHoldsAlone holds a lane past what it holds alone to failing capacity_exhausted at once, and to resource_timeout where its copies never land. The engine tests, test-engine-cpu (engine-sim's churn decode gap 721.8 ms of 837.7) and the release property test over seeds 1 to 60,000 pass. Over those seeds 28,905 requests fail at the property test's walls, against 28,966 before, 172 of them for a hot block floor against 264, and none fails capacity_exhausted once the room is ample, before or after. On the 48 GB M5 Pro at 9.4G the pair above completes, the shorter's floor met by a 97.5 MiB release once the longer yielded, and so does a pair on one conversation. runtime/engine gains 19 lines, the field, the judge's wait for writes and the lone take, so ENGINE_LINE_BUDGET rises by 19.
The first start after an update spends about two minutes before Ready creating the GPU's pipelines in the warm-up (the 27B 118 s and Flint 134 s on a 48 GB M5 Pro, where later starts take about 14 s), and all that the server and the app could say of it was that the engine warmed up: Warming carried nothing. Warming now carries the steps of the warm-up done, of all (u32 stepsDone, u32 steps), so protocol 13 replaces 1.6.0's 12. ServingRuntime::requireWarmup sends it as the warm-up begins and as each step ends, through a callback that start() points at NativeRuntime::announceWarming. A step is a command: the prefill of the most rows, each row class's, each decode width a batch may take, and the state restore; a width the plan does not hold, or memory keeps from running, ends at once. The steps count commands, not time, so they are progress the engine knows rather than an estimate. Both sides refuse a Warming with no steps or more done than it has. The server prints each, `Warming up · 3 of 11 steps`, so a start from a terminal says how far it is too, and writes it to the serve record as `warmup` (`steps_done`, `steps`), which the app reads (serve_record_warming_golden.json). A relaunched engine's warm-up is a recovery and stays unreported, as before. On the Studio's M5 Max, a new checkout's first start of Qwen3.8-Flash-Next Q2_0 warmed up in 145 s, with tests building beside it, and the next start in 3.8 s, Ready 13.6 s after its launch. The console and the serve record counted the 11 steps as each ended; of the 145 s, the prefill of 2,048 rows took 58.6 s and one row class's 43.4 s, the other four classes 0.1 to 18.3 s, and each decode width and the restore under a second. The engine tests, test-engine-cpu (the protocol golden; the bootstrap test counts each step, a memory-limited one too, and no failed one), the Python tests and ruff pass, here and cherry-picked onto the engine branch. runtime/engine grows by 33 lines, for the two counts, their check and encoding, and the warm-up's count of its steps; the engine line budget takes them.
…ate is slow The first start after an update spends a couple of minutes warming up while the engine makes the GPU's programs (the 27B 118 s to Ready on a 48 GB M5 Pro, against 14 s later), and the app said only "Warming up", the reason in a tooltip that said a minute. The serve record now counts the warm-up's steps (`warmup`), and the app follows it. While a start warms up, the Server shows a row as it does a download: the steps done of all, as a bar and in words, and that the first start after an update prepares the GPU and can take a couple of minutes, which replaces the tooltip. What says what the server does says how far it is (`Warming up · 3 of 11`): the model's row, the menu bar's panel and the composer; the panel says why under it, and a new chat under its mark. A step is a command the engine runs, not a share of its time, so the bar moves as the engine counts and estimates nothing. MainViewTests follows a fake start's record to `Warming up · 3 of 10` and the note, and the app reads serve_record_warming_golden.json as the server writes it. Through the app's flow test, a start of Qwen3.8-Flash-Next Q2_0 on the Studio's M5 Max, with the engine change and its library built with line tables so that the shader cache had none of it, showed each step as the record counted it: 1 of 11 after 281 s, 3 after 326 s and 11 after 409 s, then Ready.
UserDefaults.kept(in:) named a test's defaults by its folder alone, and AppFlowTests keeps its defaults in one fixed folder, so each run opened the domain the run before had left. On macOS 27 a process that opens a domain whose file holds a key goes on reading the value it found after it writes the key, though the write reaches the file: the 1 GB memory limit one run's aStartThatFailsSaysWhy left came back in the next run on macOS 27.0, and 8 of AppFlowTests' 9 tests failed. What a run left can outlive the folder too: the domain of a folder that no longer existed still gave an earlier run's 32K context length and port. The name of each domain now carries a UUID made once per run, so a run's defaults start empty and are never another run's, while the calls of a run with one folder share them as before. With a 1 GB limit and another port planted in the old domain, the old helper failed all three AppFlowTests tried, on that limit or that port; with this change two runs in a row started every server on the run's own port and options. Each run leaves its own defaults file in AppFlowTests' folder, 165 bytes.
The release notes gain the three changes on top of the earlier heads: a lone request takes the whole context the server advertises where the SSD cache cannot take its KV, and Flash-Next's advertised context is one extent shorter where no lone request could take its last (Upgrade); two long distinct requests at the memory limit complete where each does alone; and the first start after an update counts its warm-up's steps, in the console and the app's Server screen, with the note on why it is slow. The pair's line says what 1.6.1 does rather than that it no longer fails: 1.6.0 completes that pair, and only 1.6.1's development failed it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splash for Mac 1.6.1, with the installer changes it needs. Stacked on #733: until that lands this shows both, and this compare shows only what is new here.
Why
What changes
Chat. Titles the model sums up. Pinned chats, and the others by day. A menu on each chat, and several chats deleted at once. The open chat's title in the toolbar. Models started and switched from the composer. PDFs attached in the app and the chat page, within the limits
/v1/modelsnow reports. A message the server refuses goes back to the composer. Attachments are files of their own; the app moves earlier images only once it is the one Splash that runs, from where it can.Models. A menu on each model: a projector and a draft from this Mac, Check for Updates, and Delete. Started and switched from the menu bar's panel.
Server screen. A row per request from its start to its end, kept start logs, and the start a failure says would fit. While a start warms up, a row shows its steps on a bar, the model's row, the composer and the menu bar show
Warming up · N of M, and a note says the first start after an update prepares the GPU and can take a couple of minutes. Scrolls at the display's rate. Context length offers 1M tokens for the models that serve it (the Qwen families, which YaRN stretches up to 1,010,000 tokens, not Flint), and says under the row that YaRN changes answers past the model's own window.Elsewhere. Coding agents open in the drawer's terminals, in their folder. A quit asks first only where it would cut off work. The programs the app runs get the login shell's Hugging Face settings, through a pipe, never a file. The API key reaches the Keychain once it is typed.
Installer.
install/models.py --model ID removetakes every selection of the model and the files of its Hub snapshots nothing else links, with their blobs and the references to them. An update frees the earlier commit's files, except a snapshot another tool has downloaded into, which stays whole. Neither touches anything else in the Hugging Face cache.Testing
swift test: 182 tests, and AppFlowTests on a runtime packaged from this branch with the 27B, 8 of 9 offline (the ninth expects the Hub's answer for a model it lacks, which an offline run cannot give). Each run of the tests now has defaults of its own: on macOS 27 a domain named by a fixed path returned an earlier run's--max-memory 1G. The installer's Python tests, which cover removing every selection, a server holding one of them, an update beside another tool's download, a reference to a commit removed, a repository folder's other files, and a record that cannot be read.Known limitations
--revision.Credits
@jiiim's pull requests are where these started, rewritten here on the app's own components, and those commits carry them as co-author: the menu bar (#699), the quit question (#698), chat deletion (#697), a model's projector and draft (#687), and PDFs (#686).