test: Case-1 stress ceilings + Case-2 measured cross-host WAN-penalty curve

Case 1 stress (Mac M4): FD not the limit (100k); memory scales with capacity x
window (cap 2048 -> 11.5GB, node bound 61GB > 24GB RAM); single-tenant
serialization caps heavy-context concurrency at ~8 (vs 256 light).

Case 2 (H200 real models): injected per-block proposer<->verifier RTT -> measured
WAN-penalty curve. Co-located 2.20x AR; break-even ~100ms/block; 150ms -> 0.77x
(net loss). LAN (<=15ms) keeps 1.8-2.2x. Confirms WAN data plane infeasible.

Updates ADR 0014 + README with measured curves + evidence JSONs.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Author: cursoragent

README.md

@@ -259,11 +259,18 @@ resident **window**, not the number of agents.
 
 **Cross-host proposer/verifier.** A GPU proposer ⇄ Mac verifier *token-level
 draft* data plane is **design-only** (no `CapabilityService` / `ProposeBlock` /
-gossip) **and** ruled out by the WAN latency budget (per-block RTT ≫ block
-compute). The realizable split is **WAN = control + tool plane** (the Mac
-bridge) and **LAN = co-located data plane** (spec-decode is a same-host win:
-GPU 1.79× AR, Mac ≈ parity). See ADR 0014 for the full plan, evidence, and the
-served-MLX-gemma gap found during testing.
+gossip) **and** ruled out by the WAN latency budget — now **measured** on real
+H200 compute by injecting one proposer↔verifier round-trip per block:
+
+| per-block RTT | 0 (co-located) | 15 ms (LAN) | 30 ms | 60 ms | 100 ms | 150 ms |
+| --- | --- | --- | --- | --- | --- | --- |
+| vs AR | **2.20×** | 1.81× | 1.50× | 1.22× | **0.98×** (break-even) | 0.77× (loss) |
+
+**Break-even ≈100 ms/block**: a cloud↔desk WAN (30–150 ms) straddles/exceeds it,
+while a LAN (≤15 ms) keeps the 1.8–2.2× win. So the realizable split is **WAN =
+control + tool plane** (the Mac bridge) and **LAN = co-located data plane**. See
+[ADR 0014](docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md)
+for the full plan, evidence, and the served-MLX-gemma gap found during testing.
 
 ## Kakeya Inference Engine for Mac — MLX speculative-decode port (K3 beta baseline)
 

docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md

@@ -37,9 +37,9 @@ runnable vs design-only and shaped the test plan below.
   `[self-hosted, macOS, ARM64, kakeya-mac-m4]`, reached via the **Mac bridge**
   git-bus plane (no inbound path; allowlisted presets only).
 - **Cloud agent**: Linux x86 VM (no Metal). Orchestrates via the bridge.
-- **GPU**: H200 (vast.ai) — used for the co-located CUDA reference (PR #119);
-  **unavailable at test time** (instance recycled), so Case-2 GPU numbers are
-  cited from the prior committed evidence.
+- **GPU**: H200 NVL (vast.ai) — runs the CUDA proposer + verifier for the
+  co-located reference (fused **2.06–2.20× AR**, recall 1.0) and the §4.3
+  cross-host WAN-penalty sweep.
 - **Engine**: gRPC `RuntimeService` (`scripts/start_grpc_runtime_server.py`),
   Python SDK clients (`sdks/python/kakeya`).
 
@@ -82,6 +82,29 @@ behavior; the served **MLX gemma** path is a separate v0.4 gap (§6).
 - **Server RSS is flat** (3825 → 3850 MB across 1 → 256 agents): adding agents
   costs ~0 memory beyond the bounded slab; model weights dominate.
 
+### 3.2b Stress beyond 256 — the real ceilings (preset `agent-capacity-stress`)
+
+Pushing further with the FD limit raised (`RLIMIT_NOFILE` soft 100k, hard
+unlimited on the Mac) and a **per-agent context prefill** (window 256,
+`--context-len 256`, capacity 2048):
+
+| agents | created | create p95 | per-session KV | server RSS |
+| --- | --- | --- | --- | --- |
+| 1 | 1/1 | 3.07 s | 29.8 MB | 11 477 MB |
+| 8 | 8/8 | 25.2 s | 29.8 MB | 11 343 MB |
+| 16 | 15/16 (1 `RpcCancelled`) | 44.6 s | 29.8 MB | 10 781 MB |
+
+- **FD is not the ceiling** (raised to 100k; Mac hard limit is unlimited).
+- **Memory** scales with `capacity × window`: capacity 2048 @ window 256 →
+  **~11.5 GB RSS**, and the theoretical node bound is **~61 GB > 24 GB RAM** —
+  so capacity must be **sized to RAM** (it is the memory knob, not agent count).
+- The binding constraint with real per-agent context is **single-tenant
+  serialization**: create latency is purely linear (3 → 12 → 25 → 45 s as
+  N = 1 → 4 → 8 → 16) because every session's prefill serializes through the one
+  shared verifier, so clean concurrency tops out at **~8 heavy-context agents**
+  before RPCs time out — versus **256 light-session agents** (§3.2). Per-session
+  KV stays bounded (29.8 MB @ window 256) throughout.
+
 ### 3.3 Honest caveat — v0.3 is single-tenant
 
 Create/generate latency scales **linearly** with `N` (256 agents → gen p95
@@ -119,22 +142,47 @@ a LAN** for the data plane.
 | --- | --- | --- | --- |
 | Discovery / capability advertise | yes (seconds-scale) | bridge proxy only | bridge dispatch ~10 s + queue; one Mac, serialized (`concurrency: mac-bridge`) |
 | Job/tool dispatch (eval/bench) | yes | implemented (bridge) | this ADR's Case-1 run is itself an instance |
-| **Token-level draft (data plane)** | **no — must be LAN** | not implemented | — (ruled out by §4.1) |
-| Co-located spec-decode (the feasible data plane) | n/a (same host) | implemented | **GPU H200 1.79× AR** (PR #119); **Mac 0.93× AR** (PR #118) |
+| **Token-level draft (data plane)** | **no — must be LAN** | not implemented | **measured penalty curve §4.3** (break-even ~100 ms/block) |
+| Co-located spec-decode (the feasible data plane) | n/a (same host) | implemented | **GPU H200 2.06–2.20× AR**; **Mac 0.93× AR** (PR #118) |
 
 So the answers to Case 2's three metrics, under the **realizable** topology:
 
-- **Token throughput**: spec-decode is a *co-located* win — **1.79× AR on the
-  GPU** (28.94 vs 16.13 tok/s, recall 1.0) and **≈AR parity (0.93×) on the
-  Mac**. A WAN GPU-proposer→Mac-verifier draft loop would be **slower than
-  running either side alone** (§4.1), so it is not a throughput strategy.
+- **Token throughput**: spec-decode is a *co-located* win — **2.06–2.20× AR on
+  the GPU** (recall 1.0) and **≈AR parity (0.93×) on the Mac**. The measured
+  WAN-penalty curve (§4.3) shows the cross-host draft loop falls to **break-even
+  at ~100 ms/block and a net loss at 150 ms**, i.e. slower than running AR
+  locally — so it is not a throughput strategy.
 - **Max agent connections**: governed by the *serving* node (Case 1): **256+
   concurrent agents** on the Mac via `RuntimeService`.
 - **Mac KV upper bound**: bounded — **capacity × per-session `sink+window`**
   (≈2.0 GB at capacity 256 for Qwen3-0.6B; for the gemma S5 production config
   the per-agent resident KV is ~133 MB at 5.8k ctx, dominated by the 5 exact
   full-attention layers — see the README beta scorecard).
 
+### 4.3 Measured WAN-penalty curve (H200, real models, `--rtt-sweep`)
+
+Rather than rest on the latency *estimate*, we measured it: the fused engine was
+re-timed with one injected proposer↔verifier round-trip **per block** on the real
+Gemma-4-26B verifier + DFlash drafter (H200 NVL), sweeping per-block RTT across
+the cloud↔desk range (`scripts/research/k3_specdecode_gpu_bench.py --rtt-sweep`,
+`results/research/k3_crosshost_rtt_gpu.json`):
+
+| per-block RTT | decode tok/s | vs AR | regime |
+| --- | --- | --- | --- |
+| 0 ms (co-located) | 52.4 | **2.20×** | the win |
+| 5 ms (LAN) | 47.0 | 1.97× | LAN keeps it |
+| 15 ms | 43.3 | 1.81× | LAN keeps it |
+| 30 ms | 35.9 | 1.50× | WAN edge |
+| 60 ms | 29.2 | 1.22× | shrinking |
+| 100 ms | 23.5 | **0.98×** | **break-even** |
+| 150 ms | 18.4 | 0.77× | net **loss** |
+
+AR baseline = 23.8 tok/s. **Break-even is ~100 ms/block**: beyond it, cross-host
+spec-decode is *slower than running AR locally*. A cloud↔desk WAN (30–150 ms RTT)
+straddles or exceeds break-even, while a LAN/Thunderbolt link (≤15 ms) preserves
+the 1.8–2.2× win. This is the architecture's prediction (design doc §4.2),
+**now quantified on real compute** — and it is why the data plane must be LAN.
+
 ## 5. Decision
 
 1. **Case 1 is validated**: the session-bound gRPC runtime admits and serves

results/research/k3_agent_capacity_stress_mac.json

@@ -24,11 +24,13 @@
     },
     "levels": [
       1,
+      4,
+      8,
+      16,
+      32,
+      48,
       64,
-      256,
-      512,
-      1024,
-      2048
+      96
     ],
     "single_tenant_note": "v0.3 single-tenant: shared verifier, RPCs serialized on one asyncio loop; this measures connection/session admission scaling, not parallel inference."
   },
@@ -39,43 +41,77 @@
       "generate_ok": 1,
       "errors": {},
       "create_latency_s": {
-        "p50": 3.0911,
-        "p95": 3.0911
+        "p50": 3.0748,
+        "p95": 3.0748
       },
       "generate_latency_s": {
-        "p50": 0.0297,
-        "p95": 0.0297
+        "p50": 0.0296,
+        "p95": 0.0296
       },
       "per_session_kv_bytes": 29474816,
-      "server_rss_mb": 11227.9,
-      "wall_s": 3.12
+      "server_rss_mb": 11476.5,
+      "wall_s": 3.11
     },
     {
-      "agents": 64,
-      "created_ok": 14,
-      "generate_ok": 14,
+      "agents": 4,
+      "created_ok": 4,
+      "generate_ok": 4,
+      "errors": {},
+      "create_latency_s": {
+        "p50": 9.255,
+        "p95": 12.3516
+      },
+      "generate_latency_s": {
+        "p50": 0.0867,
+        "p95": 0.1155
+      },
+      "per_session_kv_bytes": 29818880,
+      "server_rss_mb": 11474.8,
+      "wall_s": 12.47
+    },
+    {
+      "agents": 8,
+      "created_ok": 8,
+      "generate_ok": 8,
+      "errors": {},
+      "create_latency_s": {
+        "p50": 15.7695,
+        "p95": 25.2241
+      },
+      "generate_latency_s": {
+        "p50": 0.1739,
+        "p95": 0.2352
+      },
+      "per_session_kv_bytes": 29818880,
+      "server_rss_mb": 11342.6,
+      "wall_s": 25.46
+    },
+    {
+      "agents": 16,
+      "created_ok": 15,
+      "generate_ok": 15,
       "errors": {
-        "RpcCancelledError": 50
+        "RpcCancelledError": 1
       },
       "create_latency_s": {
-        "p50": 21.683,
-        "p95": 40.7049
+        "p50": 25.6965,
+        "p95": 44.5888
       },
       "generate_latency_s": {
-        "p50": 0.2375,
-        "p95": 0.3998
+        "p50": 0.2797,
+        "p95": 0.4459
       },
       "per_session_kv_bytes": 29818880,
-      "server_rss_mb": 10849.8,
-      "wall_s": 67.58
+      "server_rss_mb": 10780.7,
+      "wall_s": 63.64
     }
   ],
   "summary": {
-    "max_concurrent_agents_clean": 1,
+    "max_concurrent_agents_clean": 8,
     "per_session_kv_bytes": 29818880,
     "per_session_kv_mb": 29.8189,
     "node_kv_upper_bound_mb": 61069.07,
     "node_kv_upper_bound_note": "capacity * per-session bounded KV \u2014 the whole-node resident-KV ceiling, independent of context length or agent churn.",
-    "server_peak_rss_mb": 11227.9
+    "server_peak_rss_mb": 11476.5
   }
 }