Website · Documentation · Quickstart · Compatibility · Contributing
OrbitKV extends the KV cache of vLLM and SGLang beyond GPU memory. Store reusable prefixes in DRAM and optional SSD, then restore them after GPU eviction or an engine restart. This serves repeated documents, shared system prompts and long conversation prefixes.
Run an independent Cache Manager on each inference host. Engines own GPU pages and scheduling; Rust owns external replicas, byte budgets and transfer completion. The pinned official engines are vLLM 0.31.0 and SGLang 0.5.21. Start with dense TP=1/PP=1 inference and check the support matrix. Version 0.1.0 is not yet published; interfaces may change before 1.0.
- DRAM and SSD reuse: keep the Manager alive across engine restarts.
- Native transfers: engine-local raw DRAM Restore; Manager-owned Publish, SSD and codec work, with ownership retained through completion.
- Bounded resources: shared cache capacity and per-instance query budgets.
- Peer cache sharing: local inventory indexes discover replicas; Mooncake TENT transfers bytes through TCP or RDMA in qualified profiles.
Optional storage codecs, reuse policies and CUDA copy kernels have separate qualification limits. Native P/D uses the engines' own transports and independent cache adapters; see P/D composition.
Follow the installation and quickstart to build a complete wheel for your Python/CUDA runtime. Use a separate environment for each engine and install the same OrbitKV build in the Manager environment. The Manager needs compatible PyTorch and does not load model weights.
Start a Manager:
orbitkv-cache-manager --addr 127.0.0.1:50055 --http-addr 127.0.0.1:9091 --pool-size 8gbStart vLLM in its official release environment:
VLLM_USE_V2_MODEL_RUNNER=0 vllm serve /path/to/immutable-model \
--enforce-eager --enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"OrbitKVConnector","kv_role":"kv_both","kv_connector_module_path":"orbitkv.vllm"}'For SGLang, follow its linker recipe. To verify an
external hit, keep the Manager alive, restart the engine and repeat a multi-block
prompt; check orbitkv_load_bytes_total. A repeated prompt in the same engine can
hit its native GPU cache instead.
| Task | Guide |
|---|---|
| Select a release, model and topology | Compatibility, models |
| Share capacity or add SSD/peers | Deployment, Manager configuration |
| Understand ownership and data paths | Architecture, transport |
| Measure latency and throughput | Performance, benchmark methodology |
| Develop and review changes | Contributing, completion plan |
Distributed HA, native GDS, broader parallelism and hybrid external recovery remain subject to separate gates. Current metadata support stays at four owners; the 16-owner visibility gate passed, but DRAM/SSD isolation failed. Component speedups do not establish a win over native caching or LMCache in serving.
Apache 2.0. Bundled dependencies retain their own licenses and notices.