Fix bench_long_session metric names + expose scheduler_kv_live_bytes

Triage of the 2026-05-30 Mac M4 4h run (results/platform-tests/
bench_long_session_mac_1780130542.aborted.json) surfaced a bench-side
bug that made the entire run produce no KV-memory data.

Root cause
----------
scripts/bench_agentic/bench_long_session.py looked for these metric
names:

    inference_engine_scheduler_active_sessions
    inference_engine_scheduler_pool_in_use
    inference_engine_scheduler_pool_size
    inference_engine_scheduler_kv_live_bytes

but the server actually exposes:

    scheduler_active_sessions
    scheduler_pool_in_use
    scheduler_pool_total
    scheduler_pending
    (no scheduler_kv_live_bytes at all)

So all 58 turns recorded an empty 'metrics: {}' dict and the
KV-bounded check (the headline ADR 0006 §2.3 claim) was effectively
disabled.

Fixes in this commit
--------------------
1. inference_engine/memory/pool.py
     SlabPool.live_kv_bytes property — sums RolloutSlab.live_kv_bytes
     across the in-use slabs. Free slabs report 0 (logical_size is
     reset on release). Verifier-side bytes flow in via
     PooledVerifier's existing live_kv_bytes_override mechanism, so
     this aggregate matches the verifier's actual footprint.

2. inference_engine/server/metrics.py
     New 'scheduler_kv_live_bytes' Gauge with descriptive HELP text
     pointing at ADR 0006 §2.3. snapshot_scheduler() takes an extra
     kv_live_bytes kwarg (defaults to 0 so existing callers stay
     valid; calling without it now also explicitly resets the gauge
     so a stale prior value never bleeds into the next scrape).

3. inference_engine/server/app.py
     Bootstrap snapshot + the per-/metrics-scrape refresh both now
     pass pool.live_kv_bytes through.

4. scripts/bench_agentic/bench_long_session.py
     - Corrected metric names everywhere they appear (5 sites:
       _METRIC_NAMES, docstring, progress line, bucketize aggregate,
       global aggregate).
     - Added a 'note on what this bench measures and what it
       doesn't' to the module docstring documenting the prefill-
       grows-linearly-with-history finding from the same triage.
       That observation is a *protocol-level* limitation of stateless
       OpenAI chat-completions, NOT a sink+window failure: KV memory
       is bounded but prefill cost is O(history). The bench reports
       both metrics independently — KV bounded is a hard claim,
       latency drift is a measurement.

Tests
-----
  tests/inference_engine/memory/test_pool.py      +3 tests for
                                                  live_kv_bytes
                                                  aggregation
  tests/inference_engine/server/test_metrics.py   +1 test for
                                                  default-zero kwarg,
                                                  +scheduler_kv_live_bytes
                                                  in expected-name set,
                                                  +assertions in
                                                  existing snapshot
                                                  tests
  tests/.../test_app_metrics_and_auth.py          +1 integration test
                                                  asserting /metrics
                                                  exposes the new
                                                  gauge at idle

Verified:
  pytest tests/inference_engine/   →  389 passed
  python3 scripts/bench_agentic/bench_long_session.py --dry-run  →  OK
  parser smoke test on real metric-name sample  →  parses all 5

Out of scope
------------
  - Server-side fixes for the orphan-session-on-disconnect bug
    (already applied locally on the Mac per the aborted.json
    fixes_applied list; will be a separate PR from that work).
  - Architectural decision on cross-request KV reuse (the actual
    fix for prefill latency growth) — needs ADR 0006 amendment
    and likely a v0.4 implementation, separate work line.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Author: cursoragent

inference_engine/memory/pool.py

@@ -112,6 +112,27 @@ def total_kv_bytes(self) -> int:
         """Sum of physical KV bytes across all slabs (capacity, not live)."""
         return sum(s.kv_bytes for s in self._all_slabs)
 
+    @property
+    def live_kv_bytes(self) -> int:
+        """Sum of *live* KV bytes across slabs currently in use.
+
+        This is what we want to expose as a Prometheus gauge for the
+        long-session memory-stability claim (ADR 0006 §2.3): it's the
+        actual KV memory consumed by active sessions right now, not
+        the pool's pre-allocated capacity. Free slabs contribute 0
+        (their ``logical_size`` is reset on release).
+
+        :class:`~inference_engine.scheduler.pooled_verifier.PooledVerifier`
+        keeps each slab's ``live_kv_bytes_override`` synced with the
+        real verifier KV size, so this aggregate matches the verifier
+        backend's actual memory footprint.
+        """
+        with self._lock:
+            in_use_set = set(self._in_use)
+        return sum(
+            self._all_slabs[i].live_kv_bytes for i in sorted(in_use_set)
+        )
+
     def acquire_optional(self) -> Optional[KVSlab]:
         """Acquire a slab if available, else return ``None`` instead of raising.
 

inference_engine/server/app.py

@@ -152,6 +152,7 @@ def create_app(
     metrics = Metrics.build()
     metrics.snapshot_scheduler(
         active=0, pool_in_use=0, pool_total=pool.total_count, pending=0,
+        kv_live_bytes=0,
     )
 
     @asynccontextmanager
@@ -295,6 +296,7 @@ async def metrics_endpoint() -> Response:
             pool_in_use=pool.in_use_count,
             pool_total=pool.total_count,
             pending=scheduler.pending_count,
+            kv_live_bytes=pool.live_kv_bytes,
         )
         return PlainTextResponse(
             content=metrics.render(),

inference_engine/server/metrics.py

@@ -105,6 +105,7 @@ class Metrics:
     scheduler_pool_in_use: Gauge
     scheduler_pool_total: Gauge
     scheduler_pending: Gauge
+    scheduler_kv_live_bytes: Gauge
     scheduler_admission_total: Counter
 
     @classmethod
@@ -169,6 +170,14 @@ def build(cls) -> "Metrics":
                 "Submissions queued for admission under QUEUE policy.",
                 registry=registry,
             ),
+            scheduler_kv_live_bytes=Gauge(
+                "scheduler_kv_live_bytes",
+                "Bytes of KV cache currently live across all active "
+                "sessions. Bounded by the per-session sink+window "
+                "configuration; verifies the ADR 0006 §2.3 long-session "
+                "memory-stability claim.",
+                registry=registry,
+            ),
             scheduler_admission_total=Counter(
                 "scheduler_admission_total",
                 "Total admission attempts, by result.",
@@ -207,11 +216,13 @@ def record_completion(self, *, finish_reason: str, n_tokens: int,
             )
 
     def snapshot_scheduler(self, *, active: int, pool_in_use: int,
-                           pool_total: int, pending: int) -> None:
+                           pool_total: int, pending: int,
+                           kv_live_bytes: int = 0) -> None:
         self.scheduler_active_sessions.set(active)
         self.scheduler_pool_in_use.set(pool_in_use)
         self.scheduler_pool_total.set(pool_total)
         self.scheduler_pending.set(pending)
+        self.scheduler_kv_live_bytes.set(kv_live_bytes)
 
     # ------------------------------------------------------------------
     # Exposition

scripts/bench_agentic/bench_long_session.py

@@ -15,10 +15,11 @@
   * completion tokens (from ``usage.completion_tokens``)
   * tokens/s for that turn
   * server-side KV pool state, scraped from ``/metrics``::
-        inference_engine_scheduler_active_sessions
-        inference_engine_scheduler_pool_in_use
-        inference_engine_scheduler_pool_size
-        inference_engine_scheduler_kv_live_bytes
+        scheduler_active_sessions
+        scheduler_pool_in_use
+        scheduler_pool_total
+        scheduler_pending
+        scheduler_kv_live_bytes
   * client-side RSS (best-effort; uses /proc/self/status on Linux,
     psutil if installed, otherwise ``None``)
 
@@ -35,6 +36,28 @@
 session can run for hours without OOM. A single agent driven for
 4 hours is therefore the cleanest evidence.
 
+A note on what this bench measures and what it doesn't (per the
+analysis of the 2026-05-30 Mac M4 run, ``bench_long_session_mac_
+1780130542.aborted.json``):
+
+  * **KV memory** stays bounded across hours (the §2.3 claim). The
+    ``scheduler_kv_live_bytes`` gauge is what proves it.
+  * **Per-turn latency** does NOT stay bounded. The OpenAI
+    chat-completions protocol is stateless: every turn the client
+    sends the full history, the server tokenizes it from scratch
+    and the verifier prefills the entire prompt, so prefill cost
+    grows linearly with history length. Sink+window only bounds
+    *generation-phase* memory, not prefill cost. A 30-min run on
+    Mac M4 showed p50 turn latency growing from ~15 s to ~55 s as
+    history grew from ~50 to ~3700 tokens. This is a **protocol-
+    level limitation**, not a memory-stability failure. Cross-
+    request KV reuse (a v0.4 feature) is the eventual fix; until
+    then, agent applications should manage prompt length via
+    summarization or sliding windows.
+
+The bench reports both metrics independently — KV bounded check is
+a hard claim, latency drift is a measurement, not a gate.
+
 Usage
 -----
 
@@ -131,11 +154,16 @@ def _client_rss_bytes() -> Optional[int]:
         return None
 
 
+# Names match the ones registered by inference_engine.server.metrics
+# (Prometheus-client does NOT add a service prefix). Changing any of
+# these breaks the bench's KV-bounded check; if you rename a metric on
+# the server, update both ends in the same commit.
 _METRIC_NAMES = (
-    "inference_engine_scheduler_active_sessions",
-    "inference_engine_scheduler_pool_in_use",
-    "inference_engine_scheduler_pool_size",
-    "inference_engine_scheduler_kv_live_bytes",
+    "scheduler_active_sessions",
+    "scheduler_pool_in_use",
+    "scheduler_pool_total",
+    "scheduler_pending",
+    "scheduler_kv_live_bytes",
 )
 
 
@@ -345,7 +373,7 @@ def _print_progress(
 ) -> None:
     last = turns[-1] if turns else None
     last_lat = f"{last['latency_s']:.2f}s" if last else "-"
-    kv = (metrics or {}).get("inference_engine_scheduler_kv_live_bytes")
+    kv = (metrics or {}).get("scheduler_kv_live_bytes")
     kv_str = f"{kv / (1024 * 1024):.1f} MiB" if kv is not None else "?"
     print(
         f"[bench] t={elapsed/60:6.1f} min | turns={turn_idx:5d} "
@@ -384,7 +412,7 @@ def _bucketize(
         bucket_turns = buckets[idx]
         latencies = [b["latency_s"] for b in bucket_turns]
         kv_vals = [
-            (b["metrics"] or {}).get("inference_engine_scheduler_kv_live_bytes")
+            (b["metrics"] or {}).get("scheduler_kv_live_bytes")
             for b in bucket_turns
         ]
         kv_vals_clean = [v for v in kv_vals if v is not None]
@@ -416,12 +444,12 @@ def _aggregate(
         }
     latencies = [t["latency_s"] for t in turns]
     kv_series = [
-        (t["metrics"] or {}).get("inference_engine_scheduler_kv_live_bytes")
+        (t["metrics"] or {}).get("scheduler_kv_live_bytes")
         for t in turns
     ]
     kv_clean = [v for v in kv_series if v is not None]
     pool_in_use_series = [
-        (t["metrics"] or {}).get("inference_engine_scheduler_pool_in_use")
+        (t["metrics"] or {}).get("scheduler_pool_in_use")
         for t in turns
     ]
     pool_in_use_clean = [v for v in pool_in_use_series if v is not None]

tests/inference_engine/memory/test_pool.py

@@ -175,3 +175,33 @@ def test_total_kv_bytes_is_sum_across_slabs(pool, cfg):
     one = KVSlab(cfg)
     expected = 3 * one.kv_bytes
     assert pool.total_kv_bytes == expected
+
+
+# ---------------------------------------------------------------------------
+# live_kv_bytes
+# ---------------------------------------------------------------------------
+
+
+def test_live_kv_bytes_zero_when_empty(pool):
+    """Idle pool reports 0 live KV bytes (every free slab has
+    logical_size==0)."""
+    assert pool.live_kv_bytes == 0
+
+
+def test_live_kv_bytes_counts_only_in_use_slabs(pool):
+    """Only acquired slabs contribute to live_kv_bytes; free slabs
+    contribute 0 because their logical_size is reset on release."""
+    s = pool.acquire()
+    # Simulate the verifier reporting 1024 KV bytes for this session.
+    s.live_kv_bytes_override = 1024
+    assert pool.live_kv_bytes == 1024
+    pool.release(s)
+    assert pool.live_kv_bytes == 0
+
+
+def test_live_kv_bytes_aggregates_across_multiple_in_use(pool):
+    a = pool.acquire()
+    b = pool.acquire()
+    a.live_kv_bytes_override = 100
+    b.live_kv_bytes_override = 250
+    assert pool.live_kv_bytes == 350

tests/inference_engine/server/test_app_metrics_and_auth.py

@@ -88,6 +88,23 @@ async def test_metrics_pool_total_gauge_reflects_config(short_engine):
     assert "scheduler_pool_total 4.0" in r.text
 
 
+async def test_metrics_kv_live_bytes_gauge_present_and_zero_at_idle(
+    short_engine,
+):
+    """The KV-live-bytes gauge must be exposed and read 0 on an idle
+    pool (every slab has logical_size == 0). This is the gauge that
+    bench_long_session.py scrapes to verify the ADR 0006 §2.3
+    KV-bounded claim, so its presence is part of the public contract.
+    """
+    app = create_app(short_engine, ServerConfig(max_concurrent=2))
+    async with AsyncClient(transport=ASGITransport(app=app),
+                           base_url="http://t") as c:
+        r = await c.get("/metrics")
+    text = r.text
+    assert "# HELP scheduler_kv_live_bytes" in text
+    assert "scheduler_kv_live_bytes 0.0" in text
+
+
 # ---------------------------------------------------------------------------
 # OpenAI error envelope
 # ---------------------------------------------------------------------------

tests/inference_engine/server/test_metrics.py

@@ -39,6 +39,7 @@ def test_build_registers_all_documented_metrics(metrics):
         "scheduler_pool_in_use",
         "scheduler_pool_total",
         "scheduler_pending",
+        "scheduler_kv_live_bytes",
         "scheduler_admission_total",
     }
     found = set()
@@ -170,22 +171,38 @@ def test_record_completion_clamps_acceptance(metrics):
 def test_snapshot_scheduler_sets_gauges(metrics):
     metrics.snapshot_scheduler(
         active=2, pool_in_use=2, pool_total=4, pending=3,
+        kv_live_bytes=12345,
     )
     assert _get_metric_value(metrics, "scheduler_active_sessions") == 2.0
     assert _get_metric_value(metrics, "scheduler_pool_in_use") == 2.0
     assert _get_metric_value(metrics, "scheduler_pool_total") == 4.0
     assert _get_metric_value(metrics, "scheduler_pending") == 3.0
+    assert _get_metric_value(metrics, "scheduler_kv_live_bytes") == 12345.0
 
 
 def test_snapshot_scheduler_overwrites_previous(metrics):
     metrics.snapshot_scheduler(
         active=5, pool_in_use=5, pool_total=8, pending=10,
+        kv_live_bytes=99999,
     )
     metrics.snapshot_scheduler(
         active=0, pool_in_use=0, pool_total=8, pending=0,
+        kv_live_bytes=0,
     )
     assert _get_metric_value(metrics, "scheduler_active_sessions") == 0.0
     assert _get_metric_value(metrics, "scheduler_pending") == 0.0
+    assert _get_metric_value(metrics, "scheduler_kv_live_bytes") == 0.0
+
+
+def test_snapshot_scheduler_kv_live_bytes_default_zero(metrics):
+    """Calling snapshot_scheduler without kv_live_bytes still sets the
+    gauge — to 0 — so a /metrics scrape never sees a stale prior
+    value."""
+    metrics.scheduler_kv_live_bytes.set(123456)
+    metrics.snapshot_scheduler(
+        active=1, pool_in_use=1, pool_total=2, pending=0,
+    )
+    assert _get_metric_value(metrics, "scheduler_kv_live_bytes") == 0.0
 
 
 # ---------------------------------------------------------------------------