Metal backend is currently slower than CPU-only on all tested models

[unamedkr]

Summary

While exploring P3 (Metal compute graph for KV attention), we discovered that the existing Metal backend (`TQ_BUILD_METAL=ON`) makes inference 13–40% slower than the CPU-only build on every model size we tested. This applies to both `fp32` and all `turbo_kv_*` paths.

Measurements (3 runs each, Llama 3.2 3B Instruct, PPL eval)

Build	KV type	tok/s
Metal ON	fp32	15.07
Metal OFF	fp32	17.87
Metal ON	turbo_kv_4b	14.17
Metal OFF	turbo_kv_4b	16.53
Metal ON	turbo_kv_5b	13.43
Metal OFF	turbo_kv_5b	15.33

Across model sizes:

Model	Metal-OFF win
SmolLM2 135M	neutral (within noise)
Llama 3.2 1B	+13–17%
Llama 3.2 3B	+14–22%
Gemma 4 26B	+40%

Even on the largest model we tested (Gemma 4 26B at 1.0 tok/s with Metal vs 1.4 tok/s without), Metal is net negative.

Why?

The current Metal path uses per-matmul dispatch with command buffer commit + waitUntilCompleted at flush points. At batch-1 inference, the per-op dispatch overhead exceeds the GPU compute benefit. This is the same dispatch-overhead issue documented in our earlier failed compute-graph experiments.

What's surprising is that even on the very large Gemma 4 26B, Metal still loses. The matmul ops are large enough that GPU compute should win, but the dispatch + sync still dominates.

Impact on past benchmarks

All quant.cpp benchmarks published before commit `` (2026-04-08) used `-DTQ_BUILD_METAL=ON` and were therefore 14-22% slower than what users actually get with the default CMake build. README and CHANGELOG numbers have been updated to reflect the honest CPU-only baseline.

The CMake default is and has been `TQ_BUILD_METAL=OFF`, so end users were always getting the fast path. Only our internal benchmarks were misled.

Action items

• Document the finding in README + CHANGELOG (this commit)
• Re-baseline all benchmarks with `TQ_BUILD_METAL=OFF`
• Investigate the dispatch overhead source — is it the gather/scatter, the wait sync, or the per-encoder begin/end cost?
• Either fix the Metal path (likely requires fewer dispatches per token, e.g., a single command buffer per layer instead of per matmul) or remove it
• If fixed, find the model size threshold above which Metal wins; auto-enable based on model size
• Profile with Xcode Metal Frame Debugger / Instruments

Out of scope (won't fix here)

• Adding new Metal kernels (e.g., for turbo_kv attention) — would compound the problem until the existing dispatch path is fixed
• Full GPU compute graph (already failed in previous attempts)

How to reproduce

```bash

CPU-only (fast, default)

cmake -B build_cpu -DTQ_BUILD_METAL=OFF
cmake --build build_cpu -j

Metal (currently slower)

cmake -B build_metal -DTQ_BUILD_METAL=ON
cmake --build build_metal -j

Compare

for k in fp32 turbo_kv_4b; do
for build in build_cpu build_metal; do
$build/quant models/Llama-3.2-3B-Instruct-Q8_0.gguf --ppl bench/data/ppl_1k.txt -j 8 -k $k
done
done
```

[unamedkr]

Investigation update (2026-04-09)

Added a small instrumentation tool to get empirical data on the dispatch path: tq_metal_diag_get() returns the number of tq_metal_batch_flush() syncs and total ops dispatched during a run. The --ppl tool now prints flushes/token and ops/flush at end-of-eval when built with Metal. (Commit: 34f5ef4)

Reproduction attempt — Llama 3.2 3B Q8_0, turbo_kv_4b, 957 tokens (M-series, j=8, 3-run average is single run here for time)

Build	tok/s	flushes	flushes/token
Metal OFF (build/)	15.1	n/a	n/a
Metal ON (build_metal/)	15.1	0	0.0

Finding: On the exact reproducer from this issue (Llama 3.2 3B Q8_0, turbo_kv_4b), Metal=ON and Metal=OFF produce identical throughput, and the diag counter reports zero flushes. The Metal batch path never fires.

Why?

The batch is gated at src/engine/tq_transformer.c:2303:

if (layer_has_gguf) tq_metal_batch_begin_if_available();

where layer_has_gguf = (layer->gguf_wq != NULL). For Q8_0 weights, the loader populates wq_q8 (not gguf_wq), so the gate is always false. The Q8 path then runs tq_matmul_q8 (src/engine/tq_ops.c:511), which is pure CPU — no Metal entry point at all.

The remaining suspects for the originally-reported slowdown are:

1. tq_metal_forward_layer (tq_transformer.c:2263) — fused-layer Metal kernel, but only triggers when wq_q4 != NULL (Q4 weights), so Q8_0 models can't hit it either.
2. GGUF on-the-fly path — layer->gguf_w* populated for Q4_K weights (e.g., Gemma 4 MoE). This is where the per-layer batch mode is designed to amortize overhead.

Hypothesis

The original measurement may have been on a different model variant (e.g., a Q4_K HuggingFace download rather than the Q8_0 produced by tools/tq_convert), or on a build before some CPU-path optimization landed. The measurable slowdown on Q8_0 is no longer reproducible on this machine.

Suggested next steps

• Re-run on Gemma 4 26B Q4_K (originally showed +40% Metal-OFF) with the diag counter, see actual flushes/token. If it's > ~5–10, that's the smoking gun.
• Re-run on a Llama 3.x Q4_K_M download (not Q8_0) to test the tq_metal_forward_layer path directly.
• If diag shows high flush count: the win is to coalesce the 8 tq_metal_batch_flush_if_available() callsites in tq_transformer.c / tq_moe.c into 1–2 per layer by using GPU-side intermediates instead of CPU result-copies between matmuls.
• If diag shows low flush count but still slow: profile per-dispatch overhead with Instruments — likely [encoder endEncoding] + commit + waitUntilCompleted minimum overhead (~50–100µs each).

The diag counter is now in main so anyone with the right model can get the measurement in one command.