[ghisguth]

On Apple7/8, Flash-Next decodes about 10x slower when its routed experts don't all fit in memory. An awake host step commits a buffer that spins (host_step_wait) until the CPU releases it. On my M2 Max that buffer starts as soon as it is committed, before the work ahead of its step is done, so it spins up to its bound (2^18 loads, 141 ms here) beside the model's work. With streamed experts a decode cycle has 49 awake steps, one per expert load, and the cycle takes 0.5–1 s instead of ~95 ms. The spin check still passes, because the spin does see the release (3–8 µs after it), it just starts too early.

With all experts resident (--max-memory 44G on a 64 GB Mac) a cycle has one awake step, so it doesn't show. It does show on any Mac that has to stream experts, like a 32 GB M1/M2 or with --expert-memory set, and in decode-profile whenever host-step.metallib sits beside the metallib.

This makes Apple7/8 skip the spin check and run awake steps as plain ones, without loading host-step.metallib or a warning. Apple9/10 are unchanged. I also tried letting the spin's buffer wait for its step's ready event. That removes the slowdown too, but it gained nothing over plain steps here (decode-profile B4 +5.5%, 24G server −1.5%), so I kept the simpler one. As before, I used Claude Code to author it and tested it by hand.

What changed

• runtime/metal/MetalBackend.mm: checkHostStepSpin() returns early on family < 9 and never re-checks, so encodeHostStep runs awake steps as plain ones. The comment above it says why.
• dev/tests/engine/metal_backend_test.mm: the three awake-step tests assumed a spin on every GPU. On Apple7/8 they now check that the spin is decided off, and that a broken host-step.metallib beside the library is neither loaded nor warned about. Also fixed awake && spins, which was true for a decided "no" (an optional in a bool context); it is now spins.value_or(false).

Results (M2 Max, 38-core GPU, Flash-Next Q2_0, A B B A)

Server decode, tok/s, median of 5 greedy 400-token requests per run:

Config	Before	After
--expert-memory 24G (experts stream)	4.57, 4.54	44.56, 44.60
--max-memory 44G (experts resident), mean of 6 runs	55.50	55.29

The 44G difference (−0.4%) is inside the 1.3% spread between the before runs.

decode-profile, 2048 prompt tokens, --expert-memory 40G, fused GPU ms:

Width	Before	After
B1	513–530	92.8–93.3
B2	790–807	98.8–98.9
B3	821–843	117.9–119.9
B4	710–984	142.1–143.9

Testing

• metal-backend, expert-store, kv-block-cache and qwen4-kernels: PASS without validation, PASS with MTL_SHADER_VALIDATION=1. qwen4-kernels bitwise: PASS (without validation, as make runs it).
• make check: CPU tests and the Python suite (1320 tests) pass. In the Metal tests everything passes but gguf-moe, which times out under validation on apple7-m1-1.4 itself on my M2 Max (the 120 s command limit, as in perf: decode PQ2_0 & Q2_0 with its zero point as a min on Apple7/8 #31; with fix(tests): gguf-moe forces only GPU families the device supports #37 it passes). That stops test-engine-metal, so I ran the Metal tests after it separately with validation, and they all pass.
• Tested only on M2 Max (Apple8). M1 (Apple7) takes the same path but I couldn't check it; a decode-profile with --expert-memory on Flash-Next before and after would show it.

[ghisguth]

@abssoul if you have time, could you check this on your M1 Max? I could only test Apple8. Flash-Next Q2_0 with decode-profile (2048 prompt tokens, --expert-memory 24G or anything below the experts' 32 GB), before and after, with host-step.metallib next to the metallib. Thank you!

[abssoul]

Checked on my M1 Max — the fix works, and the bug it removes is worse here than your M2 numbers suggested.

Environment: MacBook Pro 14" (2021), Apple M1 Max — 32-core GPU, 64 GB unified memory; macOS 27.2 (26B123) (updated from 27.0.1 since my previous test); quiet machine, no background GPU/CPU load. Before side = the installed splash 1.4.0-m1 release; after side = your branch 5250df25, built after a fresh Metal Toolchain download (the macOS update had removed the old one — the rebuilt metallib differs from the release bytes, but the kernel sources are identical: git diff 1.4.0-m1..5250df25 -- runtime/metal/kernels is empty). Model: the ISTA-DASLab Q2_0 Flash-Next pack, decode-profile, 2048 prompt tokens, 20 cycles; the metallib copies sat in their own directories with host-step.metallib beside the metallib where noted.

One addition your recipe needs on a 64 GB machine: --ple-memory as well. Without it the PLE table loads whole and decode-profile dies with "existing Metal allocations exceed memory governor limit" before the first cycle. I used --expert-memory 24G --ple-memory 2G.

Median fused decode cycle (ms):

config	B1	B2	B3
stock, host-step.metallib beside	292.49	426.85	611.08
stock, host-step.metallib beside (2nd run)	564.48	545.04	581.18
stock, no file beside	129.55	171.36	212.35
stock, no file beside (2nd run)	128.99	172.40	203.48
your branch, host-step.metallib beside	129.48	144.88	194.40
your branch, host-step.metallib beside (2nd run)	115.40	152.30	193.65
stock, no expert store (resident reference)	61.63	134.55	166.86

With the file beside the metallib the streamed B1 cycle is 292–564 ms here — 2.5–4.4× the same stack with your fix (115–129 ms). The buggy rows also swing hard between runs while every other row is stable to a few percent, which matches the spin starting at a random point. With your fix the file's presence stops mattering, and B2/B3 come out slightly better than the stock no-file control (145–152 vs 171–172, ~194 vs 203–212).

Two side observations: decode-profile stops after B3 with decode-profile: completed request was decoded (exit 1) in every streaming run here — stock with and without the file, and on your branch — so it predates the fix. And a stock 1.4.0-m1 server in streaming mode hits this too: the release install ships host-step.metallib next to the engine's metallib (~/Library/Application Support/Splash-M1/1.4.0-m1/engine/). My own log has a 24→4 tok/s episode from before I pinned --expert-memory, and the buggy rows above are exactly that regime.

[ghisguth]

@abssoul thank you for checking it on M1 Max! Good to know Apple7 behaves the same, and even worse.

The exit 1 after B3 is decode-profile itself, not a bug. Its first lane stays in every width, so it decodes 8 x --cycles cycles, and its request is capped at 256 tokens. At ~2 tokens per cycle --cycles 20 runs out in B4. I hit the same with --cycles 30 (in B3), with --cycles 12 B4 completes.

[paperniuk]

Thanks, merged. On an M1 Max 64 GB it matters even with every expert resident. Flash-Next Q2_0 through the server, three runs each: before, a decode cycle took anywhere from 69 to 114 ms (1 lane 42.5, 50.9, 65.0 tok/s), with this branch it stays at 71 to 74 ms (62.7, 62.8, 65.5). So the spin seems to steal time on the 32-core M1 even with one awake step.