feat(distributed): add P2P prefill KV cache network

[FluffyAIcode]

Summary

• add a distributed, immutable Prefill KV cache that reuses the longest compatible token prefix while keeping autoregressive decode local
• extend Kakeya capability gossip with cache compatibility and interface-aware endpoints, plus chunked gRPC publish/lookup/fetch with leases and SHA-256 validation
• wire MLX snapshots into cold AppendTokens, publish telemetry, and ship a public inference-network dashboard, launchd assets, and a Cloudflare Worker for kakeya.ai
• document the decision, operations, rollback, security boundaries, and measured two-Mac evidence in ADR 0016 and the release report

Why

Bounded KV controls resident memory but does not eliminate expensive repeated prompt prefill. Compatible Mac minis have idle unified memory that can serve as an immutable cache tier over Thunderbolt. This change turns that memory into reusable prefill state without adding any network dependency to the per-token decode path.

Live evidence

Gemma 26B MLX 4-bit, 93-token prompt, two M4 Mac minis over Thunderbolt:

• cold local prefill: 5.926 s
• remote snapshot import: 0.061 s
• observed acceleration: ~97×
• transferred cache: 2 snapshots / 36.4 MB, checksum validated
• deployed dashboard: https://kakeya.ai

Test plan

• 234 related Python regression tests passed, 1 skipped
• new platform-neutral modules: 478 statements, 100% coverage
• TypeScript SDK: 54 tests, 100% coverage, typecheck passed
• OpenMontage gateway proxy: 15 tests passed
• Cloudflare Worker dry-run passed; production dependency audit reports 0 vulnerabilities
• real head→peer publish, longest-prefix lookup, streaming fetch and SHA-256 verification over Thunderbolt
• runtime restart forced a true remote hit and preserved session history/KV accounting
• public read APIs return live topology/tokens; protected writes reject invalid API keys

Architecture and safety

• cache keys lock model weights, tokenizer/template, quantization, RoPE, layer geometry, KV dtype and schema version
• only the longest contiguous prefix is reusable; arbitrary holes are rejected
• snapshots are immutable and memory-bounded by LRU; leases prevent eviction during transfer
• peer failure, timeout, stale epoch or checksum mismatch falls back to local prefill
• decode performs no remote reads
• MVP assumes a trusted private fleet; ADR 0016 requires mTLS/PSK and HMACed prompt hashes before third-party nodes

Deployment and rollback

• launchd assets supervise the head runtime, head dashboard/control node and peer cache node
• kakeya-inference-network Worker owns kakeya.ai/*; agent.kakeya.ai remains the direct rollback origin
• rollback is disabling the cache flags/services and restoring the prior Worker route; cache entries are volatile and require no migration

Documentation

• docs/adr/0016-distributed-prefill-kv-cache-network.md
• docs/ops/distributed-prefill-kv-network.md
• docs/reports/distributed-prefill-kv-mac-thunderbolt.md
• root README.md feature/quickstart/evidence section

Made with Cursor