Prototype selective non-prefix KV reuse for RAG, tool docs, and code

[i386]

Context

Exact prefix caching cannot reuse tool documentation, RAG passages, or repository chunks when their position or surrounding context changes. Naively transplanting their KV state is not exact and can damage quality.

CacheBlend and Cache-Craft selectively recompute after combining cached chunks; Cache-Craft also demonstrates why naive chunk reuse is unsafe. For coding agents, CodeComp motivates preserving call sites, branches, returns, assignments, and def-use anchors rather than relying only on attention-based token pruning.

Coordinated in Buzz channel #skippy-kv. This is a separate experimental feature, not an extension of exact L3 lookup in #1632.

Proposed direction

• Create a distinct non-prefix/chunk-reuse namespace and API.
• Cache reusable chunk state plus enough provenance to decide what must be recomputed in the new context.
• Selectively recompute boundary/context-sensitive tokens or layers before commit.
• For code, evaluate structure-aware anchors and def-use preservation.
• Never report a chunk-composed result as an exact prefix-cache hit.

Checklist

• Define supported workloads, identity/provenance, privacy boundary, and invalidation semantics.
• Build an offline prototype for tool docs, RAG passages, and repository chunks.
• Implement selective recomputation and transactional commit into a fresh request state.
• Separate metrics, status, persistence, and cache keys from exact L1/L2/L3 prefix cache.
• Add adversarial ordering, changed-neighbor, duplicate-chunk, position/rope, multimodal, and model-version tests.
• Evaluate code-aware structural anchors separately from general document reuse.
• Compare against full prefill and naive concatenated reuse on quality, TTFT, throughput, and memory.
• Keep the feature experimental until quality and speed gates pass per model family and workload.

Acceptance gate

Ship only where selective recomputation preserves the chosen task-quality thresholds and produces a measured end-to-end speedup over full prefill. Any family/workload without evidence remains unsupported. This mechanism must never weaken #1632 exact-cache guarantees.