feat(skippy): wire next-generation KV cache and publisher defaults

[i386]

What this draft does

This is the single integration branch for the next-generation Skippy KV stack. It combines the durable local L3 substrate, host-RAM L2 serving, benefit-based admission, CacheGen serving selection, remote-handoff groundwork, user/operator controls, and publisher-metadata-driven live KV defaults.

The serving path now provides:

• existing resident L1 exact-prefix reuse;
• opt-in bounded L3 persistence with status, prune, clear, corruption recovery, and restart reuse;
• opt-in bounded L2 as a verified mirror of L3, with second-hit or large-prefix promotion and invalidation when L3 removes an entry;
• benefit-per-exclusive-byte L3 admission backed by measured prefill and restore cost, with reference-aware LRU fallback when measurements are unavailable;
• experimental CacheGen encode and native restore for the currently qualified Metal dtype layouts, with native-page fallback elsewhere;
• fail-open cache behavior: unsupported layouts, pressure, corrupt entries, and optional-tier failures become a shallower-tier hit or cold prefill rather than an inference failure.

Publisher metadata and live KV defaults

The previous auto heuristic selected Q8 below 50 GiB of model weights and Q4 at or above 50 GiB. Model weight bytes and GGUF weight quantization do not establish a safe live KV precision, so this branch removes that heuristic.

New layer packages copy supported Hugging Face publisher files under metadata/, bind them to the source repository and immutable revision, hash them in model-package.json, and compare overlapping model geometry with authoritative GGUF metadata. The runtime resolves live K/V in this order:

1. explicit user K/V override;
2. package-validated explicit publisher KV declaration;
3. package-validated publisher compute dtype mapped to a supported live KV type;
4. F16/F16 fallback.

BF16 currently maps to F16 until BF16 live KV is qualified across the supported backends. Unsupported FP8 declarations also fall back to F16. Q8/Q4 remain available when explicitly declared and supported, but are never inferred from model size or weight quantization. Existing packages without publisher metadata remain readable and use F16/F16.

The unshipped auto | quality | balanced | saver KV policy presets have been removed from config, resolution, status, UI, fixtures, and documentation. Operators retain direct K/V dtype and offload controls.

Product and release status

• L1 remains on by default.
• L2 and L3 remain opt-in for the first release; L2 requires an explicit fixed RAM budget, and user-selected L3 auto means automatic sizing after explicit enablement.
• Native L3 pages remain the default stored codec; CacheGen remains experimental.
• Remote handoff remains separately disabled and requires production transport/trust wiring before release.
• Existing huggingface.co/meshllm layer packages need a metadata backfill after this code merges.

The former component and wiring PRs have been closed or merged into this branch so review and CI continue on one head. Further KV work lands directly on kv-cache-ng-merged rather than forming another dependent PR stack.

Validation

Local validation includes:

• full affected package suites for mesh-llm-config, skippy-cache, mesh-llm-host-runtime, and the UI;
• strict Clippy for the runtime, config, cache, and shipped binary;
• Rust formatting and diff checks;
• UI typecheck, unit tests, and production build;
• package and website documentation builds;
• console-print consistency;
• split-certified roster regeneration and exact-recipe checks for the consolidated native patches.

GitHub CI is rerunning on the current head. Hardware release gates remain the documented 19K production serving runs and cross-backend traffic matrices. Unqualified optional paths retain native/local/cold fallback.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.