debug(mlx): root-cause Mac batched-decode bug to MLX core quantized/rope kernel (B>1, L=1) — full ablation

[FluffyAIcode]

What

Followed the upstream investigation as requested (per-layer logits diff + patch-and-verify) and fully root-caused the Mac batched-decode recall break.

Per-layer localization (Mac M4)

mlx_batched_layer_diff_diag.py: at decode step 1, batched row 0 is bit-exact vs serialized; row 1+ diverge from layer 0, with layer-0 input identical (embedding correct).

Ablations — every Python-level cause ruled out

candidate	test	result
sliding-window rotation	short prompt (no rotation)	still diverges
in-place cache write	concat SinkWindowKVCache	still diverges
KV-shared layers	model has 0 (first_kv_shared_idx=30)	N/A
embedding / per-layer-input	layer-0 input diff = 0 all rows	ruled out
attention mask	None at this offset	ruled out
mx.fast.scaled_dot_product_attention	manual matmul-softmax SDPA	still diverges

Root cause

Identical layer-0 input + concat cache + manual SDPA, prefill (L=327) correct but decode (L=1, B>1) broken for rows>0 → an MLX core-kernel bug for 4-bit-quantized batched single-token decode (mx.quantized_matmul / mx.fast.rope at B>1, L=1). Below the Python layer — not patchable in this repo; it's an upstream MLX issue (now precisely characterized for a report). CUDA is unaffected (HF transformers, no MLX quantized kernels — §3.5/§3.7 keep recall 1.0).

Honest outcome

I could not fix it this round — the bug is in MLX's compiled kernels, not our Python. The --manual-sdpa patch I tried (a legitimate candidate) did not resolve it, which is itself the evidence that ruled SDPA out. Per the conditional request, I'm not opening a separate "fix" feature PR/ADR (no fix achieved); instead this PR records the complete root-cause + reusable diagnostic suite. Recall-safe Mac multi-tenant stays serialized.

Files

• scripts/research/mlx_batched_layer_diff_diag.py (+ --kakeya-cache), mlx_batched_multitenant_bench.py (+ --manual-sdpa)
• presets: mlx-batched-layer-diff[-concat], mlx-batched-manual-sdpa
• docs/adr/0014-…md §3.7 full root-cause chain; evidence JSONs

Testing

• ✅ pytest tests/inference_engine/bridge/test_manifest.py (24 passed)
• ✅ 4 Mac runs (success) producing the ablation table; serialized recall 1.0 throughout, batched recall 0.125 under every Python-level fix.