docs(skippy): add KV caching user and operator guides

[i386]

The KV-cache stack did not have one user-facing entry point: operators had to infer resident-cache behavior, exact K/V representation, durable disk controls, and the experimental CacheGen boundary from implementation notes. This PR adds that website guide and keeps the detailed disk operator runbook.

What users get

• The normal OpenAI-compatible request flow, exact-token prefix matching, cache namespaces, and cached_tokens reporting.
• Exact default resolution: Q8_0 K/V below 50 GiB, Q4_0 K/V at or above 50 GiB, metadata-based compatibility fallback to F16, automatic KV offload and unified allocation, family-aware prefix caching, and disk caching disabled until enabled.
• The loadable embedded-runtime controls for K/V dtype, offload, unified KV allocation, Flash Attention, retained sessions, prefix payload and capacity, and disk persistence.
• The user-facing precedence chain for technical fields: per-model value, global value, family default, then built-in size policy.
• CLI, environment, status, prune, and clear examples for the disk tier.
• CPU, Metal, CUDA, and ROCm backend scope, with ROCm qualification still in progress.
• CacheGen labeled experimental and unavailable through normal serving config or CLI.

Validation

• just website-build
• npm run build
• npm run check:cli
• npm run test:cli-explorer
• git diff --check

Base: #1816, the consolidated CacheGen stack. This supersedes the earlier disk-only review shape of #1748 without creating a duplicate documentation PR.

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 93b1d928-cdbb-4e0e-8797-5c4fd7c7ea96

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[danielwinterw]

Approving.

I spot-checked the operator guide's numbers against the code on this branch and they hold: DEFAULT_KV_DISK_MINIMUM_FREE_MIB = 16 * 1024 and MIN_KV_DISK_MINIMUM_FREE_MIB = 1024 in mesh-llm-config/src/model.rs, AUTO_MAX_BUDGET_BYTES = 64 GiB in kv_disk_config.rs, and EVICTION_LOW_WATER_PERCENT = 85 in skippy-cache/src/l3.rs. The fail-closed-config / fail-open-runtime split is the distinction operators actually get wrong, and putting it in bold above the fold is the right call. The legacy SKIPPY_L3_* section is the most useful part of the page — field-level-fallback-only, SKIPPY_L3_BUDGET_BYTES=0 no longer meaning unbounded, and the implied-fixed-mode behaviour are all things that would otherwise be discovered by surprise. Restart-vs-live-apply table and the troubleshooting matrix are concrete rather than generic. All checks pass.

One nice-to-have: this is based on scama/skippy-cachegen-rocm-mask (#1816), which puts it five deep in the stack, but the content documents the L3 tier from #1632/#1736 and I did not find anything in it that depends on CacheGen. Rebasing onto #1736 — or onto main once that lands — would let the guide ship without waiting on a 13k-line codec change. Worth doing given how much of this page is aimed at people about to turn the tier on for the first time.

[i386]

Website guide validation is green at exact head 3d382cc8e54733ce0539c28e38e6b52f5715d908.

Local checks:

• npm run build
• npm run check:cli
• npm run test:cli-explorer
• git diff --check

GitHub CI: 18 passed, 10 expected skips, 0 failures.

The guide documents the default automatic in-memory prefix cache, model-fit TOML controls, disk-cache config/environment/CLI overrides, lifecycle commands, OpenAI cache fields, CPU/Metal/CUDA/ROCm scope, and CacheGen's experimental status.

[i386]

Superseded by consolidated integration PR #1838. The focused branch and review history remain available; further production wiring continues from the consolidated head.