test: multi-tenant resident-window pressure A/B vs MLX-native (ADR 0014 §3.4) - #126
Merged
Merged
Conversation
… ADR 0014 + README Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
scripts/research/mlx_multitenant_pressure.py: per-agent KV + max concurrent agents in a memory budget, Kakeya S5 (sink+window) vs gemma native hybrid cache (sliding bounded to 1024). Ramps agents via cache replication (real N x memory), measures peak mem + per-agent decode tok/s. Addresses the single-tenant pressure gap (true multi-tenant served path = PR-A3c; this measures the model/cache-level capacity that is the real differentiator). New preset mlx-multitenant-pressure. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ecall-preserving config + sinkwin floor; A/B uses derived max-agents from measured per-agent KV Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…vs MLX-native Real per-agent prefills (not COW): Kakeya S5 fits ~4.2x more concurrent agents than gemma-native at ctx2048 (per-agent KV 61.1 vs 256.9 MB; budget hit 32 vs 15; derived 93 vs 22), recall-preserving. Pure sink+window floor 16.8x but sacrifices recall. Addresses the single-tenant pressure-test gap; true parallel served path needs PR-A3c. Evidence + Appendix A row added. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…nt-pressure-ab-2815 # Conflicts: # docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Addresses a methodology gap flagged on the Case-1 pressure test: it measured connection admission on the single-tenant served path, not multi-tenant parallel-inference capacity (max resident window/agent) with an A/B vs MLX-native.
Adds
scripts/research/mlx_multitenant_pressure.py(+ presetmlx-multitenant-pressure): builds one independent KV cache per agent, prefills each to a context length, and ramps the agent count with real per-agent prefills (real N× memory) until a memory budget is hit — A/B-ing Kakeya vs gemma's native cache.Result (Mac mini M4, gemma-4-26B-A4B 4-bit, ctx 2048, 21 GB budget)
Two bugs found & fixed during the work: the first attempt used
mx.array()cache replication, which MLX makes copy-on-write (peak memory stayed flat → invalid); replaced with real per-agent prefills. Pure-boundedmake_sink_window_cacheover-states the win by dropping full-attn recall; added the recall-preserving S5 config as the fair baseline.Files
scripts/research/mlx_multitenant_pressure.py(new)inference_engine/bridge/manifest.py+ test (mlx-multitenant-pressurepreset)docs/adr/0014-…md§3.4 + Appendix A row;results/research/k3_multitenant_pressure_mac.jsonTesting
pytest tests/inference_engine/bridge/test_manifest.py(24 passed)conclusion=success): native budget-hit N=15, S5 N=32; A/B 4.2× per-agent KV, 4.23× derived agents.