TL;DR: llama.cpp's Metal backend has a fast small-batch matrix-vector kernel (mul_mv_ext)
— the only path that amortizes decode batches of 2-8 rows, which is exactly what speculative
decoding produces. It admits Q4_0-family and K-quants. It never admits i-quants (IQ1/IQ2/IQ3/IQ4_XS)
— the quants that memory-constrained Macs are forced to run. The dequant functions already exist
in-tree with the correct signature; upstream simply never instantiated them for this path.
This repo: the 88-insertion patch, the measurements, the refuted hypotheses, and the scripts to reproduce everything on your own machine.
UD-IQ2_XXS (same quant family mix as Unsloth's 27B low-RAM builds):
| batch | upstream | patched | delta |
|---|---|---|---|
| 2 | 124.0 | 140.1 | +13% |
| 3 | 132.7 | 194.6 | +47% |
| 4 | 148.0 | 209.9 | +42% |
| 5 | 146.0 | 240.1 | +64% |
| 8 | 137.7 | 237.7 | +73% |
| 9 | 180.1 | 219.2 | +22% |
| 12 | 229.2 | 246.4 | +7% |
| 16 | 307.3 | 244.6 | −20% ← mat-mat correctly keeps ≥13 |
IQ3_XXS: +34/+47/+69/+71% at BS 3/4/5/8. Correctness: test-backend-ops -o MUL_MAT all-pass,
n=1-8, all 8 added types.
Why it matters: batch amortization for i-quants goes from 1.08× (i.e. none — the basis for "speculative decoding doesn't work on Apple Silicon") to 1.86×. The published dead end was a missing template instantiation, not a hardware property. This flips the viability math for MTP speculative decoding (3-5-row verifies) on i-quant models — the head Qwen3.8 ships in every GGUF.
01-mv-ext-iquants.patch— the clean 48-line PR subset (= llama.cpp#27350)02-vectorized-bs1.patch— vectorized batch-1 kernels + BS2-3 admission (stacks on 01). End-to-end lossless: greedy outputs verified byte-identical scalar vs vector on a real model.research-knobs.patch— standalone research build (env knobs for the gate/mv-ext/nsg experiments); NOT stackable with 01/02
ggml-metal.metal: 32 instantiations —kernel_mul_mv_ext_{iq2_xxs,iq2_xs,iq2_s,iq3_xxs,iq3_s,iq1_s,iq1_m,iq4_xs}_f32_r1_{2..5}reusing the existingtype4x4dequant functionsggml-metal-ops.cpp: admission for those types; env-tunable bounds (GGML_METAL_MV_EXT_MIN/MAX, defaults match upstream); r1ptg cases for BS 9-16; diagnostic knobs (GGML_METAL_NO_MV_EXT,GGML_METAL_MM_MIN,GGML_METAL_NSG)
git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp # tested at 6d05498
git apply ../patches/01-mv-ext-iquants.patch # + 02-vectorized-bs1.patch for the full stack
cmake -B build -DGGML_METAL=ON && cmake --build build -j8 --target llama-batched-bench test-backend-ops
./build/bin/test-backend-ops -o MUL_MAT -b MTL0 # correctness
../scripts/probe.sh # the A/B tablesRefuted along the way (details in RESULTS.md, PLAN.md): forcing small batches onto M5 Neural Accelerators (3-4× worse) · crossing the mm gate with deep drafts (BS 9 is a dead zone) · padded-ext BS1 (plain mv wins) · an 800 tok/s "win" that was a degenerate repetition loop (benchmark now auto-flags those).
ggml-org/llama.cpp#27350 — the mergeable subset (instantiations + admission, +48 lines, defaults untouched). If it lands, every i-quant Mac user gets this without patching anything.
One command produces your row (needs the patch built per Reproduce above):
./scripts/sweep-my-mac.sh /path/to/any-iquant-model.gguf| machine | model | BS4 | BS5 | BS8 | who |
|---|---|---|---|---|---|
| M5 Pro 24GB | Qwen3-1.7B UD-IQ2_XXS | 148→210 (+42%) | 146→240 (+64%) | 138→238 (+73%) | @ColeLundstrom |
| M5 Pro 24GB | Qwen3-1.7B IQ3_XXS | 137→202 (+47%) | 135→228 (+69%) | 139→237 (+71%) | @ColeLundstrom |
| M1 16GB (MacBook Air) | Qwen3-1.7B UD-IQ2_XXS | 71.5→90.3 (+26%) | 73.3→97.4 (+33%) | 73.6→94.2 (+28%) | @ColeLundstrom |
M1 row: interleaved ABAB, 2 reps each, rep-to-rep spread <1%; BS1 parity by design; correctness 63/0 on M1. PRs with rows welcome — especially M2/M3/M4, where the mv-ext win should hold but the absolute numbers will differ. Negative results welcome too; they're how this repo got here.
patches/02-vectorized-bs1.patch (applies on top of 01) adds vectorized
float4/dot() batch-1 mul_mv variants for IQ2_S / IQ2_XS / IQ2_XXS / IQ3_XXS — 86% of the
bytes in a typical low-RAM 27B build. Measured on exact 27B shapes (M5 Pro, interleaved A/B):
+2.7-9.9% per-kernel. Also extends i-quant mv_ext admission to BS 2-3 (+13% / +19%).
Opt-in via GGML_METAL_MV_IQ2_S_VEC=1 (etc.); scalar kernels remain for same-binary A/B.
Kernels authored by an OpenAI Codex agent working from this repo's findings; correctness
independently re-verified here (full Metal MUL_MAT suite: 0 failures). Deliberately NOT
extended to IQ4_XS/IQ1_M — they are 2.7%/0.1% of model bytes; measured priorities, not vibes.
Pending: end-to-end 27B numbers on the 24 GB machine (baseline / draft-mtp / draft-mtp+ngram-mod).
PLAN.md — the full 24 GB laptop optimization plan · RESULTS.md — every measurement · RESEARCH.md — the mechanism (ne11 gate, Neural Accelerator batch cliff, BaseRT/MLX landscape) · scripts/ — probes and the end-to-end harness