Skip to content

test: multi-tenant resident-window pressure A/B vs MLX-native (ADR 0014 §3.4) - #126

Merged
cursor[bot] merged 5 commits into
mainfrom
AgentMemory/multitenant-pressure-ab-2815
Jun 14, 2026
Merged

test: multi-tenant resident-window pressure A/B vs MLX-native (ADR 0014 §3.4)#126
cursor[bot] merged 5 commits into
mainfrom
AgentMemory/multitenant-pressure-ab-2815

Conversation

@FluffyAIcode

Copy link
Copy Markdown
Owner

What

Addresses a methodology gap flagged on the Case-1 pressure test: it measured connection admission on the single-tenant served path, not multi-tenant parallel-inference capacity (max resident window/agent) with an A/B vs MLX-native.

Adds scripts/research/mlx_multitenant_pressure.py (+ preset mlx-multitenant-pressure): builds one independent KV cache per agent, prefills each to a context length, and ramps the agent count with real per-agent prefills (real N× memory) until a memory budget is hit — A/B-ing Kakeya vs gemma's native cache.

Result (Mac mini M4, gemma-4-26B-A4B 4-bit, ctx 2048, 21 GB budget)

config per-agent KV budget hit at derived max agents
MLX-native (gemma hybrid cache) 256.9 MB N=15 ~22
Kakeya S5 (recall-preserving) 61.1 MB N=32 ~93
pure sink+window (no recall) 15.3 MB ~370
  • Kakeya S5 fits ~4.2× more concurrent agents than MLX-native at equal context, recall preserved (5 full-attention layers exact; 25 sliding layers drop from native's 1024-window to sink+window=68). Measured budget-hit points (15 vs 32) confirm the per-agent-KV ratio empirically.
  • Honest nuances recorded: gemma's native cache already bounds sliding layers to 1024, so the win vs native is 4.2×, not the 16.8× one gets vs a pure sink+window cache (which sacrifices long-context recall → not a fair comparison). The ratio shrinks at longer context (shared full-attn layers grow in both).
  • This is memory-fit capacity, not throughput — a single Mac GPU serializes compute; the value is fitting ~4× more bounded-window agents in the same RAM. A truly parallel served path still needs PR-A3c.

Two bugs found & fixed during the work: the first attempt used mx.array() cache replication, which MLX makes copy-on-write (peak memory stayed flat → invalid); replaced with real per-agent prefills. Pure-bounded make_sink_window_cache over-states the win by dropping full-attn recall; added the recall-preserving S5 config as the fair baseline.

Files

  • scripts/research/mlx_multitenant_pressure.py (new)
  • inference_engine/bridge/manifest.py + test (mlx-multitenant-pressure preset)
  • docs/adr/0014-…md §3.4 + Appendix A row; results/research/k3_multitenant_pressure_mac.json

Testing

  • pytest tests/inference_engine/bridge/test_manifest.py (24 passed)
  • ✅ Run on Mac M4 via bridge (conclusion=success): native budget-hit N=15, S5 N=32; A/B 4.2× per-agent KV, 4.23× derived agents.
  • ✅ ADR fences balanced.
Open in Web Open in Cursor 

cursoragent and others added 4 commits June 14, 2026 04:43
… ADR 0014 + README

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
scripts/research/mlx_multitenant_pressure.py: per-agent KV + max concurrent
agents in a memory budget, Kakeya S5 (sink+window) vs gemma native hybrid cache
(sliding bounded to 1024). Ramps agents via cache replication (real N x memory),
measures peak mem + per-agent decode tok/s. Addresses the single-tenant pressure
gap (true multi-tenant served path = PR-A3c; this measures the model/cache-level
capacity that is the real differentiator). New preset mlx-multitenant-pressure.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ecall-preserving config + sinkwin floor; A/B uses derived max-agents from measured per-agent KV

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…vs MLX-native

Real per-agent prefills (not COW): Kakeya S5 fits ~4.2x more concurrent agents
than gemma-native at ctx2048 (per-agent KV 61.1 vs 256.9 MB; budget hit 32 vs 15;
derived 93 vs 22), recall-preserving. Pure sink+window floor 16.8x but sacrifices
recall. Addresses the single-tenant pressure-test gap; true parallel served path
needs PR-A3c. Evidence + Appendix A row added.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…nt-pressure-ab-2815

# Conflicts:
#	docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
@cursor
cursor Bot merged commit c75a2ac into main Jun 14, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants