Skip to content

Commit cb29fbd

Browse files
docs: sync ADR 0014 (agent-connection capacity + cross-host topology) into README
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent fe1a55e commit cb29fbd

1 file changed

Lines changed: 35 additions & 0 deletions

File tree

README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,41 @@ engine lands at **≈ AR parity** — the memory + context wins are platform-ind
230230
Reproduce with `scripts/research/k3_e2e_gpu_bench.py` + `k3_specdecode_gpu_bench.py`
231231
(CUDA) and the `k3-beta-scorecard` / `k3-fused-allmlx-code-trim` Mac-bridge presets.
232232

233+
### Agent-connection capacity & cross-host topology ([ADR 0014](docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md))
234+
235+
**Agent connections (gRPC `RuntimeService`, Mac mini M4).** A connection load
236+
test (`scripts/research/grpc_agent_capacity_loadtest.py`, preset
237+
`agent-capacity-loadtest`) ramps N concurrent agents — an independent gRPC
238+
channel + session each — against one runtime:
239+
240+
| | result |
241+
| --- | --- |
242+
| Max concurrent agents | **256 / 256, zero errors** (the configured capacity — a clean floor, not a failure point) |
243+
| Per-session resident KV | **bounded** (sink+window; ~7.8 MB @ window 64, ~30 MB @ window 256) |
244+
| Node KV upper bound | **capacity × per-session bound** (≈2.0 GB @ cap 256) — independent of context length / churn |
245+
| Server RSS vs agents | **flat** (3825 → 3850 MB across 1 → 256) — adding agents costs ~0 memory |
246+
247+
Caveat: v0.3 is **single-tenant** (one shared verifier, RPCs serialized on one
248+
asyncio loop — per-session binding is a v0.4 / PR-A3c item), so create/generate
249+
latency is linear in N and "256" is the max concurrent connections *served*, not
250+
parallel inferences. Pushing further (preset `agent-capacity-stress`, FD raised
251+
to 100k / hard unlimited on the Mac) shows the true ceilings: **FD is not the
252+
limit**; **memory** scales with `capacity × window` (capacity 2048 @ window 256
253+
~11 GB RSS, theoretical node bound ~61 GB > 24 GB RAM, so capacity must be
254+
sized to RAM); and with a per-agent **context** prefill the binding constraint
255+
is **single-tenant serialization** (concurrent heavy-prefill agents serialize
256+
and time out well before any FD/connection limit). Bounded memory is structural:
257+
light-session agent count does **not** grow RSS; the memory lever is the
258+
resident **window**, not the number of agents.
259+
260+
**Cross-host proposer/verifier.** A GPU proposer ⇄ Mac verifier *token-level
261+
draft* data plane is **design-only** (no `CapabilityService` / `ProposeBlock` /
262+
gossip) **and** ruled out by the WAN latency budget (per-block RTT ≫ block
263+
compute). The realizable split is **WAN = control + tool plane** (the Mac
264+
bridge) and **LAN = co-located data plane** (spec-decode is a same-host win:
265+
GPU 1.79× AR, Mac ≈ parity). See ADR 0014 for the full plan, evidence, and the
266+
served-MLX-gemma gap found during testing.
267+
233268
## Kakeya Inference Engine for Mac — MLX speculative-decode port (K3 beta baseline)
234269

235270
After the **CUDA** beta (PR #107: f_θ + S5 K/V-restoration verifier, **fused DFlash

0 commit comments

Comments
 (0)