|
1 | 1 | # ADR 0007 — Cross-request KV cache reuse for long sessions |
2 | 2 |
|
3 | | -- **Status**: Proposed (in flight, blocking v0.3.0 GA) |
| 3 | +- **Status**: Accepted (2026-05-31, blocking v0.3.0 GA) |
4 | 4 | - **Date**: 2026-05-31 |
5 | 5 | - **Decision drivers**: ADR 0006 §2.3.b empirical evidence (4-hour Mac M4 |
6 | 6 | run); the gap between the "local agent infrastructure" framing and the |
@@ -649,36 +649,55 @@ This ADR is considered validated when: |
649 | 649 | Items 2–5 are GA gates; v0.3.0 cannot promote to GA without all of |
650 | 650 | them. |
651 | 651 |
|
652 | | -## 7. Open questions (require decision before implementation) |
653 | | - |
654 | | -These are decisions the ADR author is *not* taking unilaterally and |
655 | | -need explicit approval before PR 7-1 starts: |
656 | | - |
657 | | -- **OQ-1**: System-prompt change handling. What if the user's system |
658 | | - prompt rotates mid-session (an agent's "tool" message inserted |
659 | | - ahead of the conversation history)? This shifts every position by |
660 | | - N, breaking the prefix. Default proposal: treat as a full reset. |
661 | | - Alternative: detect the system-prompt shift specifically and |
662 | | - handle. **Recommend default until production data shows otherwise.** |
663 | | -- **OQ-2**: Numerical determinism strictness on Mac M4. Strict |
664 | | - bit-identical or relaxed-to-ULP-equivalent? **Recommend strict |
665 | | - first, relax only if tests fail.** |
666 | | -- **OQ-3**: Should the `--no-cross-request-reuse` server flag exist |
667 | | - for debugging? **Recommend no — keep the protocol surface |
668 | | - minimal. Operators debugging can compare old behavior by checking |
669 | | - out v0.3.0-rc1.** |
670 | | -- **OQ-4**: Should we also expose the matched-prefix length as a |
671 | | - per-response field (in the `usage` block)? **Recommend no for |
672 | | - v0.3** — it leaks server implementation detail. Add only if a |
673 | | - user need surfaces. |
674 | | -- **OQ-5**: Eviction in v0.3 is "implicit replacement" (next |
675 | | - non-matching request resets). Is that adequate, or do we need a |
676 | | - "stale cache idle timeout" even in single-tenant? **Recommend |
677 | | - no idle timeout for v0.3** — single-tenant servers are usually |
678 | | - long-running with one client. |
679 | | - |
680 | | -Each OQ has a recommended default. If you accept all defaults, this |
681 | | -ADR is ready to move from `Proposed` to `Accepted` and PR 7-1 starts. |
| 652 | +## 7. Open questions — resolved 2026-05-31 |
| 653 | + |
| 654 | +All five OQs were resolved with the recommended-default answers |
| 655 | +on 2026-05-31. The ADR moved from Proposed to Accepted on the |
| 656 | +same date. The original questions and resolutions are recorded |
| 657 | +below for traceability: |
| 658 | + |
| 659 | +- **OQ-1 (RESOLVED, default accepted)**: System-prompt change |
| 660 | + handling. What if the user's system prompt rotates mid-session |
| 661 | + (an agent's "tool" message inserted ahead of the conversation |
| 662 | + history)? This shifts every position by N, breaking the prefix. |
| 663 | + **Resolution**: take the new-session path uniformly. Do not |
| 664 | + attempt to detect "system prompt only changed" specifically. |
| 665 | + Revisit only if production data shows the simpler behavior is |
| 666 | + costly enough to justify the detection logic. |
| 667 | + |
| 668 | +- **OQ-2 (RESOLVED, default accepted, strengthened)**: Numerical |
| 669 | + determinism strictness on Mac M4. **Resolution**: bit-identical |
| 670 | + by default. If a future test cycle shows the strict gate is |
| 671 | + unreachable on Metal, the relaxation must be **written into this |
| 672 | + ADR explicitly** (as an amendment) before the gate is changed. |
| 673 | + Tests that auto-relax based on whether the strict path passes |
| 674 | + are explicitly forbidden — that pattern is a fallback in |
| 675 | + disguise. |
| 676 | + |
| 677 | +- **OQ-3 (RESOLVED, default accepted)**: Should the |
| 678 | + `--no-cross-request-reuse` server flag exist for debugging? |
| 679 | + **Resolution**: no. Path selection (§2.4) is total; there is |
| 680 | + no failure mode that requires a runtime toggle. Operators who |
| 681 | + want to compare against pre-cross-request-reuse behavior can |
| 682 | + check out the v0.3.0-rc1 tag. |
| 683 | + |
| 684 | +- **OQ-4 (RESOLVED, default accepted)**: Expose matched-prefix |
| 685 | + length in the `usage` block of the response? **Resolution**: no |
| 686 | + for v0.3. `usage` is a public OpenAI-compatible field; adding |
| 687 | + Kakeya-specific keys leaks implementation detail to clients |
| 688 | + that don't need it. The same information is available via |
| 689 | + Prometheus metrics (§2.10) for operators. Revisit only if a |
| 690 | + user-facing need surfaces. |
| 691 | + |
| 692 | +- **OQ-5 (RESOLVED, default accepted)**: Eviction in v0.3 is |
| 693 | + "implicit replacement" (next non-matching request takes the |
| 694 | + new-session path and overwrites the cache). Adequate, or need a |
| 695 | + "stale cache idle timeout" in single-tenant scope? |
| 696 | + **Resolution**: no idle timeout for v0.3. Single-tenant servers |
| 697 | + are typically long-running with one client; stale-cache-on-idle |
| 698 | + has no concrete operational benefit at this scope. v0.4 |
| 699 | + multi-tenant ADR will define LRU + idle-timeout policy at that |
| 700 | + scope. |
682 | 701 |
|
683 | 702 | ## 8. References |
684 | 703 |
|
|
0 commit comments