docs(adr-0015): item #1 done - SDPA+chunked-logits unblocks long context (recall 1.0)

Measured on H200: 16k N=1-only->N=4, 32k OOM->N=2, 62k OOM->N=1, all recall 1.0;
16k N=1 peak 138.8->74.5GB. Honest: per-session memory still grows ~linearly
(full-T K/V capture + f_theta intermediates), so the concurrency win vs vLLM's
15.5@62k is gated on item #2 (bounded native decode cache). Evidence log added.

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Author: cursoragent

docs/adr/0015-kakeya-attention-and-engine-substrate.md

@@ -49,11 +49,16 @@ long prefills (N=1-only at 16k, OOM at 32k) due to O(T²) eager scores +
 full-vocab logits + a redundant `capture_verifier_own_kv` forward.
 
 **Substrate work items (ordered):**
-1. **Memory-efficient restoration prefill** — replace the patched *eager*
-   attention with SDPA/FlashAttention, chunk the LM-head logits, drop the
-   redundant full forward for the exact layers. (Unblocks long context.)
+1. ✅ **Memory-efficient restoration prefill** — patched forward routes to
+   **SDPA** (`--attn-impl sdpa`), LM-head logits chunked (`logits_to_keep=1` on
+   the restored forward + `capture_verifier_own_kv`), restored K/V held in bf16.
+   **Unblocked long context** (recall 1.0): 16k went N=1-only→N=4, 32k OOM→N=2,
+   62k OOM→N=1; 16k N=1 peak 138.8→74.5 GB. (`docs/reports/kakeya-vs-vllm-longcontext-h200.md`.)
 2. **Bounded decode cache as the native KV layout** — resident sink+window +
-   5 exact full-attention layers; no Python per-step `DynamicCache`.
+   5 exact full-attention layers; no Python per-step `DynamicCache`. **This is
+   the gating item for the long-context concurrency win**: today the bench still
+   captures full-T K/V at decode (~17 GB/session @16k), so the bounded advantage
+   is not yet realized — #1 only made the prefill *run*.
 3. **CUDA graphs** for the decode step; **fused-MoE** kernels for the verifier.
 4. (Optional) integrate Kakeya Attention as a **vLLM attention backend** so the
    bounded window rides vLLM's paged store + scheduler.

docs/reports/kakeya-vs-vllm-longcontext-h200.md

@@ -75,6 +75,34 @@ where bounded-KV produces a real long-context concurrency sweet spot.
   gemma-4** on this hardware (it runs 62k at ~15× concurrency; Kakeya eager does
   not run past 16k N=1).
 
+## Update — ADR 0015 item #1 (SDPA + chunked logits) landed
+
+Replacing the eager restoration prefill with **SDPA** (`--attn-impl sdpa`, the
+patched forward routes to `all_attention_functions["sdpa"]`) + **chunked logits**
+(`logits_to_keep=1` on the restored forward and `capture_verifier_own_kv`) +
+**bf16 restored K/V** **unblocked long context**, recall **1.0** throughout:
+
+| ctx | eager (before) | SDPA (after) | peak @ max N |
+| --- | --- | --- | --- |
+| 16k | N=1 only (138.8 GB) | **N=4** (recall 1.0) | 124.6 GB |
+| 32k | **OOM at N=1** | **N=2** (recall 1.0) | 133.8 GB |
+| 62k | OOM at N=1 | **N=1** (recall 1.0) | 120.8 GB |
+
+(16k N=1 peak dropped 138.8 → 74.5 GB.) This is the item-#1 win: the restoration
+prefill now executes at 32k/62k where eager could not.
+
+**Remaining bottleneck (→ items #2–3).** Per-session memory still grows
+~linearly with context (~17 GB/session @16k, ~37 GB @32k) because **this bench
+captures the full-T K/V into the decode cache** and holds the f_θ projection
+intermediates — i.e. it does **not** yet exercise the *bounded* resident cache
+(SinkWindowKVCache). So the restored path here is still effectively full-KV at
+decode; beating vLLM's **15.5-way** 62k concurrency requires item #2 (bounded
+native decode cache: store only sink+window + 5 exact layers, restore on demand)
++ freeing the f_θ intermediates. Item #1 was the prerequisite (it makes long
+prefills run at all); the concurrency-ceiling win is gated on #2.
+
+Evidence: `results/research/k3_cuda_mt_longctx_sdpa_ceiling_h200nvl.log`.
+
 ## Evidence
 
 - `results/research/vllm_multitenant_parallel_h200nvl_ctx62k.json`

results/research/k3_cuda_mt_longctx_sdpa_ceiling_h200nvl.log

@@ -0,0 +1,17 @@
+===== ctx16k restored-only sdpa N=1,2,4,8 =====
+[mt] modal prompt len=16038, 20 equal-length prompts (of 24)
+[mt] N=  1 | AR nan tok/s (xnan, recall nan) | restored 11.09 tok/s (x1.0, recall 1.0) | peak 74.46GB
+[mt] N=  2 | AR nan tok/s (xnan, recall nan) | restored 22.99 tok/s (x2.07, recall 1.0) | peak 92.39GB
+[mt] N=  4 | AR nan tok/s (xnan, recall nan) | restored 36.7 tok/s (x3.31, recall 1.0) | peak 124.61GB
+torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.38 GiB. GPU 0 has a total capacity of 139.80 GiB of which 5.00 GiB is free. Process 3233087 has 134.79 GiB memory in use. Of the allocated memory 133.74 GiB is allocated by PyTorch, and 380.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
+===== ctx32k restored-only sdpa N=1,2,4 =====
+[mt] modal prompt len=32038, 20 equal-length prompts (of 24)
+[mt] N=  1 | AR nan tok/s (xnan, recall nan) | restored 10.22 tok/s (x1.0, recall 1.0) | peak 97.02GB
+[mt] N=  2 | AR nan tok/s (xnan, recall nan) | restored 17.73 tok/s (x1.73, recall 1.0) | peak 133.84GB
+torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 502.00 MiB. GPU 0 has a total capacity of 139.80 GiB of which 389.06 MiB is free. Process 3235509 has 139.41 GiB memory in use. Of the allocated memory 138.48 GiB is allocated by PyTorch, and 262.44 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
+===== ctx62k restored-only sdpa N=1,2 =====
+[mt] modal prompt len=62038, 20 equal-length prompts (of 24)
+[mt] warmup note: CUDA out of memory. Tried to allocate 14.34 GiB. GPU 0 has a total capacity of 139.80 GiB of which 11.77 GiB is free. Process 3238344 has 128.02 GiB memory in use. Of the allocated memory 127.15 GiB is allocated by PyTorch, and 202.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
+[mt] N=  1 | AR nan tok/s (xnan, recall nan) | restored 7.55 tok/s (x1.0, recall 1.0) | peak 120.8GB
+torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.34 GiB. GPU 0 has a total capacity of 139.80 GiB of which 8.41 GiB is free. Process 3238344 has 131.39 GiB memory in use. Of the allocated memory 130.24 GiB is allocated by PyTorch, and 487.67 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
+CEILING_DONE