test(case2): real two-process socket + real-network confirmation

Added socket_echo_server.py (remote-host boundary) + --socket-echo-addr socket
round-trip of the REAL per-block aux payload (~156KB/block) in the fused engine.
tc netem is blocked in the vast container (no NET_ADMIN), so real latency came
from a real inter-host link (reverse SSH tunnel, ~102ms RTT):
  loopback (~0ms):  51.5 tok/s = 2.02x AR
  real net (~102ms): 14.1 tok/s = 0.56x AR  (net loss; network = 71% wall time)
End-to-end real-models + real-network proof that the cross-host token-level
draft data plane is WAN-infeasible. Updated ADR 0014 + README.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Author: cursoragent

README.md

@@ -267,7 +267,10 @@ H200 compute by injecting one proposer↔verifier round-trip per block:
 | vs AR | **2.20×** | 1.81× | 1.50× | 1.22× | **0.98×** (break-even) | 0.77× (loss) |
 
 **Break-even ≈100 ms/block**: a cloud↔desk WAN (30–150 ms) straddles/exceeds it,
-while a LAN (≤15 ms) keeps the 1.8–2.2× win. So the realizable split is **WAN =
+while a LAN (≤15 ms) keeps the 1.8–2.2× win. Confirmed end-to-end with a **real
+two-process socket over a real ~102 ms network** (reverse SSH tunnel, real
+156 KB/block aux payload): co-located **2.02×** → real-network **0.56× AR** (a
+net loss; network was 71 % of wall time). So the realizable split is **WAN =
 control + tool plane** (the Mac bridge) and **LAN = co-located data plane**. See
 [ADR 0014](docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md)
 for the full plan, evidence, and the served-MLX-gemma gap found during testing.

docs/adr/0014-agent-connection-capacity-and-cross-host-topology-tests.md

@@ -183,6 +183,33 @@ straddles or exceeds break-even, while a LAN/Thunderbolt link (≤15 ms) preserv
 the 1.8–2.2× win. This is the architecture's prediction (design doc §4.2),
 **now quantified on real compute** — and it is why the data plane must be LAN.
 
+### 4.4 Real two-process socket over a real network (no simulation)
+
+To remove the "injected sleep" caveat, the per-block exchange was run through a
+**real TCP socket to a second process** (`scripts/research/socket_echo_server.py`),
+serializing the **actual** per-block payload — the verifier→proposer aux hidden
+states + draft tokens, **≈156 KB/block** — once per block
+(`--socket-echo-addr`). Two transports were measured (H200 NVL,
+`results/research/k3_crosshost_{socket_loopback,realnet}_gpu.json`):
+
+| transport | RTT | decode tok/s | vs AR |
+| --- | --- | --- | --- |
+| loopback socket (same host) | ~0 ms | 51.5 | **2.02×** |
+| **real network** (GPU↔cloud-agent, reverse SSH tunnel) | **~102 ms** | **14.1** | **0.56×** |
+
+- **Loopback** matches the co-located number → the socket + serialization of the
+  156 KB payload are themselves cheap; the killer is purely the round-trip.
+- **Real network**: at a genuine ~102 ms RTT the cross-host loop collapses to
+  **0.56× AR — a net loss, worse than running AR alone**, and *worse* than the
+  latency-only model (0.98× at 100 ms) because the real transport also pays the
+  **156 KB/block bandwidth** (network was **71 %** of decode wall time). This is
+  an end-to-end real-models + real-network confirmation that the token-level
+  draft data plane is WAN-infeasible.
+- Note: `tc netem` artificial latency could **not** be applied inside the vast
+  container (`RTNETLINK: Operation not permitted` — no `NET_ADMIN` in the
+  restricted netns), so real latency was obtained from a real inter-host link
+  (the reverse-tunnel RTT) rather than synthetic netem.
+
 ## 5. Decision
 
 1. **Case 1 is validated**: the session-bound gRPC runtime admits and serves

results/research/k3_crosshost_realnet_gpu.json

@@ -0,0 +1,260 @@
+{
+  "kind": "k3_specdecode_gpu_bench",
+  "config": {
+    "verifier_id": "google/gemma-4-26B-A4B-it",
+    "drafter_id": "z-lab/gemma-4-26B-A4B-it-DFlash",
+    "f_theta_dir": "results/research/f_theta_v5_s5_sliding",
+    "haystack_lines": 160,
+    "n_samples": 2,
+    "max_new_tokens": 64,
+    "block_size": 16,
+    "sink": 4,
+    "window": 64,
+    "seed": 0,
+    "skip_unfused": true,
+    "rtt_sweep": null,
+    "socket_echo_addr": "127.0.0.1:53999",
+    "netem_sweep": "0",
+    "netem_dev": "lo",
+    "output": "results/research/k3_crosshost_realnet_gpu3.json"
+  },
+  "env": {
+    "gpu": "NVIDIA H200 NVL",
+    "torch": "2.12.0+cu130"
+  },
+  "prompt_tokens": {
+    "min": 3238,
+    "max": 3238
+  },
+  "ar_incremental": {
+    "decode_tokens_per_s_mean": 25.286,
+    "recall": 1.0
+  },
+  "restored_pertoken": {
+    "decode_tokens_per_s_mean": 25.677,
+    "recall": 1.0
+  },
+  "restored_specdecode": {
+    "skipped": true,
+    "decode_tokens_per_s_mean": null,
+    "mean_accept_len": 0.0,
+    "recall": 0.0,
+    "per_sample": [
+      {
+        "decode_tokens_per_s": null,
+        "mean_accept_len": 0.0,
+        "time_breakdown_s": {
+          "aux_clean_forward": 0.0,
+          "drafter": 0.0,
+          "incremental_verify": 0.0
+        },
+        "tokens": []
+      },
+      {
+        "decode_tokens_per_s": null,
+        "mean_accept_len": 0.0,
+        "time_breakdown_s": {
+          "aux_clean_forward": 0.0,
+          "drafter": 0.0,
+          "incremental_verify": 0.0
+        },
+        "tokens": []
+      }
+    ]
+  },
+  "restored_specdecode_fused": {
+    "decode_tokens_per_s_mean": 46.032,
+    "mean_accept_len": 3.49,
+    "time_breakdown_s_mean": {
+      "drafter_cached": 0.179,
+      "incremental_verify": 1.216,
+      "ctx_kv_extend": 0.026
+    },
+    "recall": 1.0,
+    "per_sample": [
+      {
+        "tokens": [
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          28487,
+          1618,
+          236772,
+          236832,
+          236828,
+          236819,
+          236771,
+          84750,
+          106,
+          106,
+          107,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          28487,
+          1618,
+          236772,
+          236832,
+          236828,
+          236819,
+          236771,
+          84750,
+          106,
+          106,
+          107,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          28487,
+          1618,
+          236772,
+          236832,
+          236828,
+          236819,
+          236771,
+          84750,
+          106,
+          106,
+          106,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          28487,
+          1618
+        ],
+        "decode_s": 1.2094762809574604,
+        "prefill_s": 0.811,
+        "decode_tokens_per_s": 52.915,
+        "time_breakdown_s": {
+          "drafter_cached": 0.052,
+          "incremental_verify": 1.134,
+          "ctx_kv_extend": 0.023,
+          "network_rtt": 0.0
+        },
+        "blocks": 14,
+        "mean_accept_len": 3.64,
+        "decode_tokens": 64,
+        "block_rtt_ms": 0.0,
+        "net_bytes_per_block": 0
+      },
+      {
+        "tokens": [
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          236777,
+          59790,
+          236772,
+          236828,
+          236819,
+          236825,
+          236770,
+          84750,
+          106,
+          106,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          236777,
+          59790,
+          236772,
+          236828,
+          236819,
+          236825,
+          236770,
+          84750,
+          106,
+          106,
+          106,
+          107,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          236777,
+          59790,
+          236772,
+          236828,
+          236819,
+          236825,
+          236770,
+          84750,
+          106,
+          106,
+          106,
+          107,
+          45518,
+          107,
+          101,
+          818,
+          6789,
+          3393,
+          563,
+          5213,
+          236777
+        ],
+        "decode_s": 1.6348003204911947,
+        "prefill_s": 0.808,
+        "decode_tokens_per_s": 39.149,
+        "time_breakdown_s": {
+          "drafter_cached": 0.307,
+          "incremental_verify": 1.298,
+          "ctx_kv_extend": 0.028,
+          "network_rtt": 0.0
+        },
+        "blocks": 15,
+        "mean_accept_len": 3.33,
+        "decode_tokens": 64,
+        "block_rtt_ms": 0.0,
+        "net_bytes_per_block": 0
+      }
+    ],
+    "speedup_over_ar_x": 1.82
+  },
+  "crosshost_socket_netem_sweep": {
+    "ar_baseline_tps": 25.286,
+    "transport": "real TCP socket round-trip + tc netem on lo",
+    "sweep": [
+      {
+        "netem_delay_ms": 0.0,
+        "tc_applied": false,
+        "decode_tokens_per_s": 14.075,
+        "vs_ar_x": 0.557,
+        "blocks": 14,
+        "mean_accept_len": 3.64,
+        "net_bytes_per_block": 159477,
+        "network_s": 3.25,
+        "decode_s": 4.546940378844738
+      }
+    ],
+    "max_netem_ms_at_or_above_ar": 0.0
+  }
+}