A Kubernetes-native scalable KV-aware LLM serving system, built from the runtime up.
中文 | Documentation | Quick Start | Architecture | Kubernetes
KVTide is a scalable LLM serving system that makes scheduling and KV-cache ownership explicit.
The Engine owns tokenization, requests lifecycle, scheduling, KV metadata. Executor owns model execution, local KV tensors.
Cache-aware routing creates a tension: routing requests to the executor that already owns reusable KV state reduces recomputation, but it can also overload cache-hot executors and constrain scheduling decisions.
KVTide explores a different direction: move reusable KV state toward available compute, instead of always moving requests toward cached state.
| Capability | Status |
|---|---|
| Executor-aware block ownership | In progress |
| Executor-to-executor KV transfer | Planned |
| Proactive KV placement | Research hypothesis |
The communication between Engine and Executor is using proto, see protocol introduce
Download the model first:
cd executor
uv run hf download Qwen/Qwen3-0.6B --local-dir ./models/Qwen3-0.6B
cd ..
docker compose up --build -d| Service | Address |
|---|---|
| Dashboard runtime console | http://127.0.0.1:5173 |
| Inference API | http://127.0.0.1:8800 |
| Admin API and metrics | http://127.0.0.1:8801 |
docker compose ps
docker compose logs -f
curl http://127.0.0.1:8801/metrics
docker compose down- Topology discovers the connected executor and shows its status.
- Metrics reads Prometheus metrics for the whole system.
The Kubernetes manifests preserve the same one-to-one runtime topology:
make docker-build
make kube-start
make kube-forwardSee k8s/README.md for manifests, probes, rollout behavior, inspection commands, and cleanup.
The executor Deployment intentionally has one replica. Adding replicas behind a Kubernetes Service would load-balance batches without preserving executor-local KV ownership.
The next architectural boundary is executor-aware block ownership. Each block table must be scoped by executor and runtime epoch before the control plane can recover safely from restarts or place work across replicas.
From there, KVTide can evaluate its central hypothesis: when compatible executors run the same model weights, dtype, and tensor-parallel configuration, an overloaded executor should be able to push selected KV blocks to an available peer. The control plane should observe the new placement, update metadata, and measure whether reuse saved more work than transfer consumed.
That path requires evidence, not only functionality. Future evaluations should compare recomputation, local reuse, and remote transfer across TTFT, TBT, throughput, transfer bandwidth, cache pressure, and tail latency.
- vLLM: paged KV-cache management and continuous batching.
- SGLang: prefix-aware scheduling and RadixAttention.
- LMCache: reusable KV storage and movement across serving instances.
- Mooncake — KV-centric disaggregated serving and direct transfer.
- NVIDIA Dynamo — distributed inference orchestration and KV-aware routing.
- llama.cpp: lightweight local inference and CPU execution.