Skip to content

Commit 4d64f9b

Browse files
docs(adr-0014): L>=2 padded decode confirms B>1,L=1 kernel bug — recall 0.125->1.0, 0.67x tps
Mac evidence (preset mlx-batched-pad-decode, 8 sessions): batched per-session recall recovered to 1.0 (matches serialized bit-for-bit on tok0 across all rows) by forcing length-2 decode forwards (qmm, avoids the qmv L=1 B>1 quantized kernel). Throughput 0.67x serialized — the 2x padding tax exceeds the batching gain at this scale, so it is a correctness-recovery probe, not a shippable throughput path. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent d11020e commit 4d64f9b

2 files changed

Lines changed: 47 additions & 0 deletions

File tree

docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,32 @@ characterized for an upstream report. Recall-safe Mac multi-tenant remains
318318
projections) is a possible future probe. Evidence:
319319
`results/research/k3_mlx_batched_manual_sdpa_mac.json` + the layer-diff logs.
320320

321+
**L≥2 padded decode workaround — hypothesis CONFIRMED, recall recovered, but
322+
no throughput win (Mac, 8 sessions, modal prompt 1149).** The `--pad-decode`
323+
mode (preset `mlx-batched-pad-decode`) duplicates the new token each step so
324+
every decode forward is **length-2**, routing through mlx's matrix-matrix
325+
(`qmm`) quantized kernel instead of the single-token (`qmv`) kernel; query
326+
position 0 is the real prediction (attends to cache+self only, == the L=1
327+
result) and the position-1 duplicate is trimmed from the (Kakeya S5, trimmable)
328+
cache so RoPE offsets stay exact. The batch dimension is untouched (stays
329+
parallel over sessions, Python-only).
330+
331+
| metric | batched **L=1** (bug) | batched **L≥2 padded** | serialized (truth) |
332+
| --- | --- | --- | --- |
333+
| per-session recall | **0.125**| **1.0**| 1.0 |
334+
| per-row tok0 vs serialized | row 1+ diverge | **all 8 match** ||
335+
| aggregate decode tok/s | 57.1 (recall void) | 9.945 | 14.927 |
336+
| speedup vs serialized || **0.67×** | 1.0× |
337+
338+
This **confirms** the root cause: forcing `L≥2` (avoiding the `B>1, L=1` `qmv`
339+
path) restores batched per-session recall to **1.0**, matching serialized
340+
bit-for-bit on the first decoded token across all rows. But the 2× per-step
341+
padding tax exceeds the batching gain at this scale (**0.67×**), so it is a
342+
**correctness-recovery probe, not a shippable throughput path**: a Mac batched
343+
*win* still needs the upstream `L=1, B>1` quantized-kernel fix (no padding tax)
344+
or a much larger cohort / cheaper verify. Evidence:
345+
`results/research/k3_mac_bridge_mlx_batched_pad_decode.json`.
346+
321347
## 4. Case 2 — cross-host proposer/verifier (FEASIBILITY VERDICT)
322348

323349
### 4.1 Verdict: the requested topology is not implementable today, and is architecturally bounded out
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"kind": "mlx_batched_multitenant",
3+
"config": {
4+
"sessions": 8,
5+
"modal_prompt_len": 1149,
6+
"max_new_tokens": 24,
7+
"verifier_path": "/Users/fluffy314/kakeya-models/gemma-4-26B-A4B-it-mlx-4bit",
8+
"kakeya_cache": true,
9+
"manual_sdpa": false,
10+
"pad_decode": true
11+
},
12+
"serialized": {
13+
"aggregate_tps": 14.927,
14+
"recall": 1.0
15+
},
16+
"batched": {
17+
"aggregate_tps": 9.945,
18+
"recall": 1.0
19+
},
20+
"batched_speedup_vs_serialized": 0.67
21+
}

0 commit comments

Comments
 (0)