debug(mac): root-cause MLX batched-recall break — upstream mlx_lm gemma-4 batch>1 decode bug (RotatingKVCache ruled out)

[FluffyAIcode]

What

Investigated the Mac batched multi-tenant recall break (batched recall 0.125–0.25 vs serialized 1.0) along the requested RotatingKVCache / batched-attention-mask line, and root-caused it.

Findings (Mac M4, via bridge)

diagnostic	batched per-session recall	isolates
short prompt (ctx 339 < sliding window → no rotation)	0.25	not rotation
concat-based Kakeya SinkWindowKVCache (no in-place buffer assign)	0.25	not the cache (in-place or concat)
per-row first decoded token vs serialized	matches all rows	batched prefill is correct

Conclusion: the break is cache-independent and prefill-correct — batched decode diverges only after the first token, with both mlx_lm's in-place cache and Kakeya's concat cache, even with no rotation. That localizes it to mlx_lm 0.31.3's gemma-4 batched (batch>1) decode forward (RoPE-offset / mask / shared-KV path) — an upstream MLX limitation, not a Kakeya cache or RotatingKVCache rotation issue. CUDA (HF transformers batched decode) is unaffected — §3.5/§3.7 stay 8.04×/8.45×, recall 1.0.

Status: recall-safe Mac multi-tenant = serialized (recall 1.0). Mac batched throughput needs an upstream mlx_lm gemma-4 batched-decode fix (or a custom batched gemma decode) — tracked as a follow-up.

Tools added (reusable for the follow-up)

• --kakeya-cache mode (concat-based S5 cache) + per-row first-token diagnostic in mlx_batched_multitenant_bench.py
• presets mlx-batched-diag-short, mlx-batched-kakeya-cache
• evidence: results/research/k3_mlx_batched_{diag_short,kakeya_cache}_mac.json; ADR 0014 §3.7 root-cause note

Testing

• ✅ pytest tests/inference_engine/bridge/test_manifest.py (24 passed)
• ✅ Three Mac runs (conclusion=success) producing the diagnostic table above; serialized recall 1.0 throughout.