MiniLMCache is a minimal Go project that demonstrates the core ideas behind LMCache-like KV cache sharing for LLM inference, including chunking, metadata lookup, remote storage, and cross-instance reuse.
It is not a production-ready cache system, and it does not run real model inference.
Its purpose is to help developers understand how KV cache can be identified, stored, discovered, transferred, and reused across different inference instances.
Modern LLM inference systems spend a large amount of time and compute on the prefill stage.
If two requests share the same prompt prefix, the KV cache generated by one instance can potentially be reused by another instance, avoiding repeated computation.
Projects such as LMCache explore this idea by introducing a cache layer between inference engines and storage backends.
MiniLMCache is a simplified Go implementation that focuses on the underlying mechanism:
- How KV cache is divided into chunks
- How chunks are identified and indexed
- How metadata is separated from actual cache data
- How one instance can discover cache produced by another instance
- How remote cache can be pulled back and reused
MiniLMCache is designed to be:
- Educational: readable and easy to trace
- Minimal: only the core mechanism is preserved
- Observable: cache lookup, miss, store, pull, and reuse should be visible
- Extensible: future versions can add compression, multi-level cache, eviction, or peer-to-peer transfer
This project intentionally does not aim to provide:
- Real tensor storage
- GPU memory management
- vLLM integration
- High-performance networking
- Production-grade consistency guarantees
- Full cache eviction and lifecycle policies
- Multi-tenant security isolation
If you are looking for a real LLM cache system, this project is only a conceptual and educational model.
MiniLMCache models KV cache sharing as a pipeline:
- A request arrives at an inference instance
- The instance computes a deterministic cache key / chunk key from the prompt or token blocks
- The instance asks a metadata service whether the required chunks already exist
- If cache exists, the instance pulls it from remote storage and reuses it
- If cache does not exist, the instance simulates prefill and generates new chunks
- The generated chunks are stored in a cache backend
- The metadata service is updated so that other instances can discover them later
This project separates the system into two layers:
- Metadata plane: where chunks are registered and discovered
- Data plane: where chunk bytes are actually stored and transferred
That separation is one of the most important design points.
The repository now includes a first LOOKUP implementation in Go.
lookup/: core LOOKUP types, chunking, deterministic keying, service logic, and trace eventslookup/memory/: in-memory metadata controller and reservation stubcmd/minilmcache-lookup-demo/: CLI demo forhit,partial_hit, andmiss
Current LOOKUP v1 behavior:
- Input is token IDs only
- Only full chunks participate in lookup
- Lookup returns the longest reusable prefix only
- Any chunk stored in a non-
locallocation setsNeedRetrieve=true - Reservation is modeled as a minimal observable stub for future retrieve work
This stage intentionally does not implement real KV bytes, GPU integration, retrieve, or store.
Run the demo:
go run ./cmd/minilmcache-lookup-demoRead the teaching guide:
docs/lookup-demo.md
Run the tests:
go test ./...+------------------+ +------------------+
| Engine A | | Engine B |
|------------------| |------------------|
| lookup chunks | | lookup chunks |
| generate chunks | | reuse chunks |
| push chunks | | pull chunks |
+--------+---------+ +---------+--------+
| |
| metadata lookup / admit |
v v
+------------------+
| Controller |
|------------------|
| chunk metadata |
| ownership info |
| location mapping |
+--------+---------+
|
| store / fetch
v
+------------------+
| Cache Store |
|------------------|
| chunk data |
| serialized bytes |
+------------------+