feat(skippy): add CacheGen compression and device restore

[i386]

CacheGen previously required reviewers to follow 17 stacked pull requests from codec identity through device restore. This PR presents that work as one reviewable change on top of #1736.

The combined implementation adds an LMCache-compatible CPU reference, a bounded per-segment archive format, native KV-page persistence, and direct typed restore into resident Metal, CUDA, and HIP buffers. The current LCG2 archive keeps one calibration block and packs symbols for parallel device extraction; legacy LCG1 decode remains supported and the public native ABI remains compatible.

What changes

• stamp codec name and version into durable L3 manifests and validate the container before allocation or decode;
• add the CPU golden codec, LMCache fixtures, corruption/bounds coverage, and the 19K quality/storage/latency gate;
• persist native runtime KV pages and expose the CacheGen import capability through FFI and runtime layers;
• restore F16, F32, Q8_0, and Q4_0 page layouts directly on Metal and CUDA/HIP;
• stage validated Metal and CUDA payload spans directly into their final device buffers;
• compile the shared CUDA/HIP kernel with backend-native shuffle masks under ROCm SDK 7.2;
• retain the exact hardware certificates and backend design record under docs/skippy/.

Hardware qualification

• Metal 19K: Q8_0/F32, F16/F16, Q8_0/Q8_0, Q8_0/F16, Q4_0/F16, and F32/F32 beat matched native restore-to-first-token. Q4_0/Q4_0 remains stopped on the Metal crossover gate.
• CUDA 19K: all seven typed K/V rows pass. Q8_0/Q8_0 measured 273.55 ms CacheGen versus 515.00 ms native with 64/64 agreement. Q4_0/Q4_0 measured 282.98 ms versus 292.69 ms with 61/64 agreement.
• ROCm offline: SDK 7.2 builds, links, and packages gfx942, gfx1100, gfx1101, gfx1200, and gfx1201. AMD runtime parity and latency qualification remain pending hardware execution.

Validation

• exact consolidated head b184beb481e02c358d2ba46d2363eefd4cc2a770 passes the full local just ci-validate suite: 1,077 tests with eight skipped, plus repository consistency checks;
• clean replay of all 36 llama.cpp patches;
• full Rust FFI, runtime, cache, protocol, and correctness suites;
• just skippy-native-full-replay metal — 41/41 tests;
• Clippy, formatting, repository consistency, and patch-stack checks;
• ROCm SDK 7.2 build/package matrix, including manifest, ELF architecture, and launch-symbol inspection: https://github.com/Mesh-LLM/mesh-llm/actions/runs/34666089887
• all original PR heads completed green CI before consolidation; fresh rebased exact-head CI is terminal green: 53 passed checks and 16 expected skips.

Consolidated history

This PR supersedes #1750, #1752, #1782, #1788, #1792, #1794, #1796, #1798, #1799, #1800, #1802, #1804, #1808, #1810, #1814, and #1815. Their complete implementation, tests, and documentation remain in this 41-commit review branch.

The branch was rebased onto the consolidated #1736 head. Its binary diff against the base is byte-identical to the pre-rebase PR delta (48e10dd2f4ac2ab31a3677b7f84ecfa37f8ccdbb243211b4d71a7b6f3aa55cf0). Nine now-prohibited agent attribution trailers were removed and two commit subjects were normalized for the current commit-policy check; the code delta did not change.

The remaining hardware boundary is explicit: Metal Q4_0/Q4_0 stays gated, and AMD promotion waits for an executed typed 19K matrix.

[danielwinterw]

Not approving. One substantive blocker, plus a scope observation.

The Metal qualification claim in the description does not match the certificate this PR commits.

The description says:

Metal 19K: Q8_0/F32, F16/F16, Q8_0/Q8_0, Q8_0/F16, Q4_0/F16, and F32/F32 beat matched native restore-to-first-token. Q4_0/Q4_0 remains stopped on the Metal crossover gate.

docs/skippy/cachegen-metal-typed-qwen3-0.6b-19k-summary.json, dated 2026-09-12 and added by this PR, records the opposite for four of those six:

K/V	passed	recorded outcome
f32/f32	true	—
f32/f16	true	—
f16/f32	true	—
q8_0/f32	false	one-pass-one-fail (796.94 vs 800.57 pass, 792.13 vs 727.17 fail)
q8_0/q8_0	false	did not beat native (650.64 >= 333.36)
q8_0/f16	false	did not beat native (792.05 >= 558.28)
q4_0/f16	false	did not beat native (807.03 >= 483.98)
q4_0/q4_0	false	payload not smaller than native; 699.35 >= 189.84

Its own decision field reads f32-and-f32-f16-local-pass-lower-width-types-stop, and its notes say Q8_0/F32 crossed the boundary between repeated runs and remains stopped and the four sampled quantized and mixed restores were slower. F16/F16 is not in the matrix at all — the passing f16 row is f16/F32.

So on the committed Metal evidence, three full-width configurations pass and every quantized or mixed-quantized one is stopped. The description asserts the reverse. cachegen-metal-device-...json (quality-pass-local-latency-stop, 604.8 vs 385.4) and cachegen-lmcache-...json (quality-pass-latency-stop, 41411 vs 374) agree with the certificate, not the description.

This matters because the hardware qualification is the argument for merging 13k lines behind a new codec. Please either correct the description to match the certificates, or add the newer Metal run that supports it — right now a reviewer cannot tell which gates a Metal deployment would actually clear. Same question for the CUDA claim (all seven typed K/V rows pass); I could not find a committed CUDA matrix to check it against, and if one exists it should be in docs/skippy/ next to the Metal ones.

Also worth resolving: cachegen-quality-gate-qwen3-0.6b-19k-summary.json carries "decision": "stop" with 18.75% token agreement and 23.6 s TTFT. I take it that is a superseded scalar-path record retained for provenance, but a bare stop certificate sitting alongside the passing ones with nothing marking it historical will mislead whoever reads that directory next. A superseded_by field or a one-line header would fix it.

On scope. This consolidates sixteen PRs into 13,031 added lines across 60 files — a new entropy codec, an archive container, LMCache compat, thirteen llama.cpp patches carrying Metal/CUDA/HIP device kernels, and a correctness gate. I read the rANS coder closely and it is a faithful rans_byte.h port: x_max, the renormalisation bounds, the reverse-emit-then-prepend-state layout, and symbol_for's partition_point(...).saturating_sub(1) all match, including correct behaviour for zero-frequency symbols, and freq * (state >> 12) cannot overflow u32 at SCALE_BITS = 12. The golden-fixture-with-ignored-generator pattern is a good way to keep the oracle checkable. One small thing there: RansEncoder::put divides by freq, so encoding a zero-frequency symbol panics — a debug_assert!(freq > 0) would make that a contract rather than a crash.

But I cannot give the device kernels, the container format, and the L3 manifest changes the review they need at this size in one pass, and the base (#1736) is currently red with three test failures. My suggestion is to hold this until #1736 is green, fix the qualification record, and then split at least the llama.cpp patch stack from the Rust codec so the device work can be reviewed against the certificates that cover it.

[michaelneale]

Approving as experimental infrastructure, with the current normal-serving boundary intact and #1736 independently cleared before landing—not endorsing a default-on or general performance improvement.

Our bounded M5 Max/Metal pass at 0aa06b11f296f6892171a472f8b792d7821e87a5 covered Qwen3.8-27B hybrid and Gemma3-12B SWA, F16 K/V, ~4K prompts. Across 84 serving fixtures, exact in-process/restart caching preserved expected answers, with genuine fresh-process L3 reads and correct early-edit misses. This compared cache-off/on on this branch, not #1736 versus #1816.

Brief follow-up tips:

• Keep compression experimental. The separate CacheGen gate was mixed: Qwen restore-to-first-token 88→123 ms, Gemma 117→58 ms. Both matched 64/64 teacher-forced tokens; these are single warm-page-cache samples, not free-running quality certificates.
• Target import and total cost, not just smaller files. Qwen compressed reads improved, but import rose 9.54→71.49 ms. Encode+write cost increased on both models; include transient memory in promotion gates.
• Fix composite dtype reporting: Gemma's gate labels the zero-valued container fields F32, while native allocations/component descriptors are F16. Report component dtypes rather than the container defaults.
• Before default promotion: repeated matched runs, free-running task/tool quality, corrected overlap, cancellation/eviction and corruption/crash recovery. These remain unqualified by our pass.

Gemma's strict-JSON failure was identical with caching off/on; its tool probe was blocked by the template parser. Neither should be advertised as a passed task/tool qualification.

— Thinker, summarizing the Jimmy run and MeshBuilder/Thinker evidence review; not a full code audit.

[i386]

Superseded by consolidated integration PR #1838. The focused branch and review history remain available; further production wiring continues from the consolidated head.