Replace file-per-segment L3 KV storage with packed append-only storage

[i386]

Context

PR #1632 deliberately cuts growing prefixes on cache geometry so later turns reuse prior state. A realistic 19K-token prefix can therefore contain roughly 9,500 segment references. Today each content-addressed segment is a separate file, eviction scans/parses manifests to reconstruct reference counts, and reads pay filesystem metadata and locality costs.

SGLANG-LSM identifies file-per-object metadata and locality as disk-KV bottlenecks and reports up to 143% more cache hits and 24% lower TTFT in its evaluated workloads.

Coordinated in Buzz channel #skippy-kv and tracked as an expanded optimization candidate for #1632.

Proposed direction

Retain the existing logical format and exactness contract while replacing file-per-segment physical storage with append-only packfiles or an LSM-style substrate:

• digest to pack/offset/length/checksum index;
• batched sequential writes and reads;
• tombstones for logical deletion;
• background compaction with bounded foreground impact;
• atomic manifest publication only after all referenced segments are durable;
• reference-aware accounting of shared segments;
• crash recovery, quarantine, and index rebuild;
• hard-budget, minimum-free, pin/hold, prune, and clear semantics equivalent to feat(skippy): add configurable node-local L3 KV cache #1632.

Checklist

• Define the versioned packed-store format, index, and migration/compatibility behavior.
• Preserve digest verification and exact numerical/model identity.
• Make append, index commit, manifest commit, tombstone, and compaction crash-safe.
• Batch segment reads by physical locality without changing logical segment boundaries.
• Bound compaction memory, write amplification, and foreground latency.
• Preserve startup cleanup and safe fallback to cold prefill on any cache failure.
• Add corruption, torn-write, stale-index, duplicate-segment, concurrent writer, eviction, prune, and clear tests.
• Benchmark against the current file-per-segment feat(skippy): add configurable node-local L3 KV cache #1632 implementation using the shared instrumentation gate.

Acceptance gate

Promote only if a release-build A/B on the same hardware and workload improves end-to-end cache-hit TTFT and/or sustained cache throughput, does not regress p99 decode latency beyond the existing 5% limit, preserves exact output, and keeps write amplification within an explicitly reported bound.