feat(skippy): wire benefit admission into L3 serving

[i386]

Problem and resulting behavior

The benefit-per-exclusive-byte policy on #1838 was pure library code with no serving caller. Durable L3 writes and capacity eviction therefore remained unconditional reference-aware LRU.

This stack wires measured benefit admission into local OpenAI exact-state serving. Once a stage has a real L3 restore sample, new candidates use the generation service estimator's cold-prefill cost and the stage restore-cost EWMA. First-seen entries remain in L1 probation, two observed reuses promote them to L3, and disk pressure asks the shared node manager for the lowest-benefit inactive manifests before the store's hard-budget reservation. Segment references receive fractional shared-byte accounting, manifest bytes are charged exclusively, and active pins remain protected.

When timing data is unavailable, the writer deliberately keeps the existing LRU write-through behavior. This covers startup, binary serving, and any sparse or invalid telemetry rather than allowing the policy to suppress caching without comparable cost data.

The existing L3 status activity object now exposes probation, persistence, fallback, and policy-eviction counters.

Validation

Validated at exact head 589df86ef70b4aa67cee8376be5a833e64ceea54:

• cargo test -p skippy-cache --lib — 266 passed, 2 ignored
• cargo test -p skippy-server --lib — 745 passed, 3 ignored
• cargo clippy -p skippy-cache -p skippy-server --all-targets -- -D warnings
• cargo fmt --all -- --check
• just ci-validate — 1,453 passed, 9 skipped

This remains a draft until the required Thoughtworks and c64/c128/c256 serving traffic compares benefit admission against reference-aware LRU.

Summary by CodeRabbit

• New Features

  • Added adaptive KV-cache storage that uses observed restore and prefill costs to decide which entries should be persisted.
  • New entries can remain in memory during probation and be promoted to durable storage after reuse.
  • Cache restores can rewarm frequently used entries when beneficial.
  • Disk cleanup now prioritizes removing inactive, low-benefit entries while preserving pinned data.
  • Existing LRU write-through behavior remains available when timing data is unavailable.

• Documentation

  • Expanded cache monitoring documentation with benefit-admission and eviction metrics.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 22516f63-a375-4472-9eef-ca048610ba4c

📥 Commits

Reviewing files that changed from the base of the PR and between ce0741e and 589df86.

📒 Files selected for processing (14)

• crates/skippy-cache/src/l3.rs
• crates/skippy-cache/src/manager.rs
• crates/skippy-cache/src/policy/mod.rs
• crates/skippy-cache/src/tier.rs
• crates/skippy-server/src/frontend/generation/queue.rs
• crates/skippy-server/src/frontend/local_generation/token_generation.rs
• crates/skippy-server/src/frontend/local_generation/token_generation/exact_state_recording.rs
• crates/skippy-server/src/frontend/local_generation/token_generation/kv_restore.rs
• crates/skippy-server/src/kv_integration/config.rs
• crates/skippy-server/src/kv_integration/exact_state.rs
• crates/skippy-server/src/kv_integration/l2_serving.rs
• crates/skippy-server/src/kv_integration/mod.rs
• docs/skippy/KV_CACHE_DISK.md
• docs/skippy/PROMPT_CACHE.md

---

📝 Walkthrough

Walkthrough

The change adds benefit-based L3 admission. The server supplies cold-prefill and restore costs. The cache tracks reuse, delays measured candidates in probation, promotes recurring entries, evicts selected manifests, and reports new activity counters.

Changes

Benefit-based L3 admission

Layer / File(s)	Summary
Policy state and eviction crates/skippy-cache/src/manager.rs, crates/skippy-cache/src/l3.rs, crates/skippy-cache/src/policy/mod.rs	The manager adds benefit policy state, counters, cost observations, admission decisions, and budget-driven eviction. The segment store exposes is_pinned and removes eligible manifest keys.
L3 tier admission flow crates/skippy-cache/src/tier.rs	The tier derives benefit keys and costs, gates measured spills, preserves LRU fallback when cost data is absent, records admitted manifests, and tests probation followed by persistence.
Server cost propagation and restore integration crates/skippy-server/src/frontend/generation/queue.rs, crates/skippy-server/src/frontend/local_generation/..., crates/skippy-server/src/kv_integration/...	The server estimates cold-prefill cost, carries optional L3 cost through exact-state records, observes L3 restores, and can enqueue memory-hit rewarm records. Documentation describes the new admission counters and behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant GenerationServiceEstimator
  participant KvStageIntegration
  participant L3Tier
  participant L3CacheManager
  GenerationServiceEstimator->>KvStageIntegration: provide estimated prefill cost
  KvStageIntegration->>L3Tier: restore or spill with cost
  L3Tier->>L3CacheManager: observe restore or memory hit
  L3CacheManager-->>KvStageIntegration: admission or rewarm decision

Loading

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch scama/skippy-benefit-admission-serving

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.