feat(skippy): wire bounded host-RAM L2 serving

[i386]

Problem

The consolidated cache stack in #1838 contains a bounded host-RAM L2 store, but serving never creates, fills, consults, invalidates, or reports it. model_fit.cache_ram_mib therefore still fails at model load and every L1 miss goes directly to disk L3.

Result

This wires an opt-in, stage-scoped L2 mirror into ordinary exact-state serving:

• model_fit.cache_ram_mib reaches StageKvCacheConfig.l2_max_bytes; zero remains disabled;
• L3 remains authoritative: lookup locates and validates the current durable manifest before an L2 mirror can serve;
• restore order is L1 radix, L2 mirror, then L3 payload load;
• an L3 fill promotes on its second hit within ten minutes, or immediately for prefixes of at least 4,096 tokens; payloads over 64 MiB are refused;
• hashing and byte copies run on the existing bounded exact-state worker;
• L2 hits asynchronously rewarm L1 without rewriting L3;
• changed or corrupt durable payloads invalidate matching mirrors;
• the existing hard byte budget, shared-segment accounting, LRU shrink/clear operations, and status counters are exposed through stage telemetry.

L2 is created only when an active local L3 tier exists. A positive L2 budget never creates an authority-free cache.

This is stacked on #1838 and should be rebased onto main after that integration branch lands.

Validation

Exact commit: ce0741eb51913cc5b7af0bfdacc0c8a78fe66eac

• cargo test -p skippy-cache --lib — 265 passed, 2 ignored
• cargo test -p skippy-server — 744 passed, 3 ignored
• cargo test -p mesh-llm-config — 237 tests passed across the package
• cargo test -p mesh-llm-host-runtime — 3,553 tests passed, 11 ignored
• cargo test -p skippy-prompt — 24 passed
• cargo test -p skippy-protocol — 72 passed
• strict Clippy across all touched crates with warnings denied
• just ci-validate — 1,453 passed, 9 skipped, plus repository consistency gates
• cargo fmt --all -- --check, just no-console-print, and git diff --check
• release L2 microbenchmark at 19,000 tokens and 9,728,000 payload bytes: 33.00x p50 lookup-to-usable-bytes speedup over L3 (366,375 ns versus 12,090,875 ns), 50/50 L2 hits, zero evictions

Before production rollout, run the required same-model Thoughtworks and c64/c128/c256 comparisons on Metal and discrete CUDA. The feature remains opt-in while that qualification is outstanding.

Summary by CodeRabbit

• New Features

  • Added an optional host-RAM L2 exact-state cache, configurable through model_fit.cache_ram_mib.
  • L2 cache hits can restore state before disk access and rewarm the faster cache.
  • Frequently used or large-prefix entries can be promoted from the durable cache.
  • Added L2 cache usage, restore, promotion, and eviction telemetry.

• Bug Fixes

  • Cache payloads and descriptors are now validated during admission and restoration.
  • Invalid or mismatched entries are removed safely.

• Documentation

  • Updated configuration references with L2 requirements, limits, defaults, and lookup behavior.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: dad902b6-536d-4ec5-ae81-d39d0914326e

📥 Commits

Reviewing files that changed from the base of the PR and between a85f8c8 and ce0741e.

📒 Files selected for processing (31)

• crates/mesh-llm-config/src/model/built_in_schema/declarations.rs
• crates/mesh-llm-config/src/wiring_status.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/family_policy.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/resolution.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/support.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/tests.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/translation.rs
• crates/mesh-llm-host-runtime/src/inference/skippy/resolver/types.rs
• crates/mesh-llm-host-runtime/src/runtime/local_model_only.rs
• crates/mesh-llm-host-runtime/tests/fixtures/config_schema_defaults_ui_reference.json
• crates/skippy-cache/src/l2/mod.rs
• crates/skippy-prompt/src/prompt_cli/stage_config.rs
• crates/skippy-protocol/src/config.rs
• crates/skippy-server/src/binary_transport/stage_execution.rs
• crates/skippy-server/src/frontend/local_generation/tests.rs
• crates/skippy-server/src/frontend/local_generation/token_generation/kv_restore.rs
• crates/skippy-server/src/frontend/prefix_cache.rs
• crates/skippy-server/src/frontend/tests/support.rs
• crates/skippy-server/src/kv_integration/activation.rs
• crates/skippy-server/src/kv_integration/cache_affinity.rs
• crates/skippy-server/src/kv_integration/config.rs
• crates/skippy-server/src/kv_integration/exact_state.rs
• crates/skippy-server/src/kv_integration/identity.rs
• crates/skippy-server/src/kv_integration/l2_serving.rs
• crates/skippy-server/src/kv_integration/mod.rs
• crates/skippy-server/src/kv_integration/resident_prefix.rs
• docs/USAGE.md
• docs/skippy/CONFIGURATION.md
• docs/skippy/PROMPT_CACHE.md
• tools/xtask/data/console_print_allowlist.json
• website/src/docs/pages/config-reference.md

---

📝 Walkthrough

Walkthrough

The change wires model_fit.cache_ram_mib to a bounded host-RAM L2 exact-state cache. It adds payload admission, digest validation, promotion from L3, L2-first restoration, L1 rewarming, telemetry, configuration defaults, tests, and documentation.

Changes

Configuration and resolution

Layer / File(s)	Summary
Configuration and resolution crates/mesh-llm-config/..., crates/mesh-llm-host-runtime/..., crates/skippy-protocol/...	cache_ram_mib is now supported and resolves to StageKvCacheConfig.l2_max_bytes. Positive values require prefix caching. Zero disables L2. Existing test and helper configurations initialize the new field.
L2 storage primitives crates/skippy-cache/src/l2/mod.rs	The L2 tier admits exact-state payloads, preserves KV descriptors, verifies durable digests, removes entries by digest, and evicts entries to a byte target.
L2 serving and L3 integration crates/skippy-server/src/kv_integration/..., crates/skippy-server/src/frontend/...	The server creates L2 only with an active L3 tier, promotes selected L3 fills, restores verified L2 entries before disk, queues L1 rewarm records, invalidates stale mirrors, and reports L2 telemetry.
Defaults and documentation docs/..., website/src/docs/...	Configuration references describe the L2 budget, prerequisites, promotion rules, restore order, and telemetry.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant KvStageIntegration
  participant StageL2
  participant L2Tier
  participant L3
  Client->>KvStageIntegration: request prefix restore
  KvStageIntegration->>StageL2: check exact-state entry
  StageL2->>L2Tier: validate key, token count, and digest
  L2Tier-->>StageL2: L2 hit or miss
  alt L2 miss
    KvStageIntegration->>L3: load durable exact state
    L3-->>KvStageIntegration: payload and manifest
    KvStageIntegration->>StageL2: consider L3 fill for promotion
  else L2 hit
    StageL2-->>KvStageIntegration: restore runtime state
  end

Loading

Suggested reviewers: ndizazzo, michaelneale

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch scama/skippy-l2-serving-pr1838

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.