test(mac): MLX batched multi-tenant run + honest finding (batched breaks recall on Mac) - #132
Merged
Merged
Conversation
… of §3.7 batched scheduler) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ll is elicited (was 0 with plain chat template) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…X (batch>1) breaks recall (0.125 vs serialized 1.0) §3.7 batched scheduler is CUDA-only (torch); Mac served-restored blocked by MLX-gemma load gap. Mac analog bench run on M4: serialized recall 1.0; batched MLX decode breaks per-session recall (0.125) -> NOT recall-safe (recall is the bottom line). Recall-safe Mac multi-tenant = serialized; batched needs an MLX batch>1 cache/forward fix. ADR 0014 §3.7 platform note + evidence. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The §3.7
BatchedDecodeSchedulerran only on the H200 (CUDA) — it's torch/CUDA, and the Mac served-restored path is blocked by the MLX-gemma load gap (ADR 0014 §6). This adds the Mac analog and runs it on the Mac mini, with an honest finding.Result (Mac M4, 8 sessions,
mlx_batched_multitenant_bench.py)Finding: batched MLX decode over gemma-4 at batch > 1 breaks per-session recall (0.125 vs serialized 1.0). Since recall is the bottom line, the recall-safe Mac multi-tenant path is serialized; batched Mac throughput is not shippable until the MLX batch>1 forward/cache correctness is fixed (likely the gemma hybrid/sliding
RotatingKVCacheunder batching). The Mac nominal speedup ceiling is also low (M4 saturates at small batch — 2.7× vs CUDA's 8.45×). The validated batched scheduler stays CUDA-only.(Recall fix during the work: the first run had recall 0 even serialized — a plain chat template; matched the working NIAH harness's direct-answer prompt + content-channel marker, which fixed serialized to 1.0 and exposed the batched-only correctness gap.)
Files
scripts/research/mlx_batched_multitenant_bench.py+ presetmlx-batched-multitenantdocs/adr/0014-…md§3.7 platform note;results/research/k3_mlx_batched_multitenant_mac.jsonTesting
pytest tests/inference_engine/bridge/test_manifest.py(24 passed)