[jianc99]

On Apple9 GPUs that report 10 cores or fewer (the base M3 and M4), GGUF and MLX MoE checkpoints now run their experts on the staged tile in decode and prefill, instead of the register tile. moeGgufTile takes the planned core count for this, and ExecutionPlans passes it. Apple10, Apple9 GPUs of more than 10 cores, and GPUs that report no core count (planned as kAssumedGpuCores, 32) plan as before. No kernel changes.

Apple9 took the register tile for its experts because a 40-core M3 Max decoded Q4_K/Q5_K, Q2_K and IQ4_XS experts 3-34% faster there than staged. The staged tile has since gained a one-pass gate/up and the column tile for an expert's lone tile of one or two rows. On a 10-core Mac mini M4 it is now faster in nearly every case measured. Built at 1bf892b, the commit of the M3 Max's measurement, the same M4 ranked the tiles as the M3 Max did (staged taking 2-15% longer at B1-B4), so the ranking turned with the staged tile.

Measurements

One MoE layer on the Mac mini M4 (Apple9, 10 GPU cores, 16 GB), gguf-moe-benchmark, register → staged ms. Rows are routed at random, or each lane's rows among few experts (pooled). Medians of four runs (eight for the 122B's MLX 4-bit), main's build and this change's in ABBA order, which time the two tiles in opposite orders.

Layer: gate/up, down, shared	B1 random	B1 pooled	B4 random	B4 pooled	2048-row chunk
35B Q4_K/Q5_K/Q8_0	1.406 → 1.163	0.599 → 0.555	3.943 → 3.167	1.973 → 1.786	46.20 → 39.02
35B Q5_K/Q5_K/Q8_0	1.548 → 1.300	0.671 → 0.611	4.381 → 3.571	2.181 → 1.958	49.34 → 39.41
Flash-Next Q4_K/IQ4_NL/Q8_0	2.610 → 2.138	1.063 → 1.024	8.329 → 6.550	3.706 → 3.471	88.00 → 78.25
Flash-Next IQ4_XS/IQ4_NL/Q8_0	2.479 → 2.061	1.030 → 1.026	7.940 → 6.327	3.549 → 3.474	87.59 → 77.43
122B Q4_K/Q5_K/Q8_0	4.025 → 3.315	1.611 → 1.517	11.459 → 9.247	5.657 → 5.122	131.10 → 111.44
122B MLX 4-bit	3.680 → 3.042	1.475 → 1.412	10.665 → 8.524	5.355 → 4.884	125.55 → 110.39

• 35B shape, all 25 formats the register tile takes (the K-quants, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, IQ4_XS, IQ4_NL, IQ3_S, MXFP4, NVFP4, FP8, Q2_0, PQ2_0, BF16, and MLX 2-, 3-, 4- (groups of 32 and 64), 6- and 8-bit). Staged takes 12-45% less time at B1-B4 with random routes, up to 17% less with pooled routes, and 7-28% less on a 2048-row chunk. It is slower only with pooled routes in IQ4_NL at B1 and B2 (1.3%, 0.1%), and on some 128- to 512-row chunks in IQ4_NL, IQ4_XS, MXFP4 and MLX 2- and 4-bit (at most 3.8%).
• 122B rows. These used the benchmark with the 122B's shape (hidden 3072, top 8 of 256 experts, intermediate 1024) edited in. The shape is not part of this PR.
• Flash-Next's K-quant and Q2_0 checkpoints were already staged and do not change: their 640-input down projection keeps them off the register tile (quant_register_tensor).
• At this PR's head, two runs of gguf-moe-benchmark 35b on the M4 confirm that it now plans the 35B's layer staged. Staged takes 16-18% less time than the register tile at B1 and B4 with random routes, 8-10% less with pooled routes, and 15% less on a 2048-row chunk (1.169 vs 1.384 ms at B1 random, 39.03 vs 46.14 ms on the chunk), within 2% of the table's row.

End to end (an estimate). No MoE checkpoint fits the M4's 16 GB, so no model ran. A one-lane decode cycle of the 35B UD-Q4_K_M runs 40 of these layers, which would take 1.8 ms (pooled) to 9.7 ms (random) less, out of 24 and 56 ms. The rest of the cycle mostly reads weights: 1.2-1.4 GB of the 35B's other tensors and LM head (1.2 GB in its UD-Q2_K_XL GGUF), the LM head again for the draft, and the 1 GB DFlash2 draft. At the 90-100 GB/s these layers reach, that is 26-32 ms. So a one-lane cycle on a base M3 or M4 with the memory for the 35B would take roughly 3-12% less, depending on how widely its verify rows spread over the experts. At B4 the layers would take 7.5-31 ms less per cycle.

Costs

• Numerics. Outputs change by the two tiles' rounding. Before ed63ffb gave the staged tile its column tile, the 35B's teacher-forced logits on the two tiles agreed at top-1 95.5%. Decode results now depend on the rows decoded beside them through the column tile, as they already do on Apple10.
• Memory. The MoE prefill scratch takes the staged 32-row tiles' bound: 39 MiB more on the 35B, 111 MiB on Flash-Next and 62 MiB on the 122B. Four decode lanes take 1.5-2.4 MiB less.

Related PRs

• Decode zero-point formats with fewer chains on Apple9's register tile #569 changes Apple9's register tile for dense projections only. With its diff applied, moe_expert_gguf_sg_a and _g differ from main's AIR only in the names of lambda types. The M4 times the register expert tile as main does (Q4_0, Q6_K and Q4_K/Q5_K within 1%), and staged still takes 6-22% less time. Either PR can land first.
• The Flint stack (Compile the attention, MoE and stop-token variants Flint runs #573-Serve Flint (RecursiveSI/flint) from its GGUFs #576) also changes runtime/ops/MoE.hpp, runtime/ops/ExecutionPlans.cpp/.hpp and DEVELOPMENT.md. This branch merges into Compile the attention, MoE and stop-token variants Flint runs #573's and Serve Flint (RecursiveSI/flint) from its GGUFs #576's heads without conflicts, and the merged ExecutionPlans.cpp and execution_plans_test.cc compile. Whichever lands second should still rebase and re-run the MoE tests. With both, Flint's experts on a base M3 or M4 run staged as well.

Tests

• M5 Max (Apple10, 32 GPU cores, 36 GB), at the head. make check-native-build, check-source and test-engine-cpu (execution-plans among it) pass, and gguf-moe, gguf-moe-e512 and expert-store pass under MTL_SHADER_VALIDATION=1.
• Mac mini M4 (Apple9, 10 GPU cores, 16 GB), at the head. Built on the M4, execution-plans passes, and gguf-moe, gguf-moe-e512 and expert-store pass under shader validation. The two gguf-moe-benchmark 35b runs above also ran there.
• execution-plans covers Apple9 at 10 and 11 reported cores and with no core count (the assumed 32), and Apple10 and Apple11.
• The table's runs came earlier, on the M4 with the same plan code. 1bf892b's gguf-moe-benchmark gave the old ranking.

Not covered:

• Larger Apple9 GPUs (M3/M4 Pro and Max, 14 to 40 cores) keep the register tile because none could be measured. The M3 Max's ranking predates the staged tile's one gate/up pass and column tile, so they may now stage faster too.
• The base M3. It takes the rule by its core count, but only the M4 was measured.
• A model end to end on the M4 (see the estimate above). The 95.5% top-1 agreement is ed63ffb's comparison of the register tile with the staged tile before its column tile; the tiles this PR switches between were not compared, on the M4 or elsewhere.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}