perf(storage): restore l1 prefix save admission performance - #90
Open
GentleCold wants to merge 8 commits into
Open
GentleCold wants to merge 8 commits into
GentleCold wants to merge 8 commits into
Conversation
- emit store staging spans from request suffix to prefix - preserve io_uring store span order across IPC - update connector and IPC tests for suffix-first admission
- evict normal io_uring stores at an aligned 80 percent save target - let L2 miss promotion use full L1 capacity and evict by LRU when full - cover save headroom and promotion eviction behavior
- restore physical store staging to prefix order - keep one store span per slot for L1 replacement metadata - update connector staging expectations
- add prefix-aware LRU ordering for request slot groups - trim save admission after insertion so suffix slots can be evicted first - mark L2 promotions as headroom entries and evict promoted headroom first - cover suffix-first trim and promotion headroom behavior
- reserve about five percent of L1 for promotion instead of twenty percent - trim grouped store admission once after the group is inserted - update transfer tests for the smaller promotion headroom
- Batch adjacent grouped-store L2 writes behind one pending task while preserving per-slot L1 entries. - Keep per-slot pending buffers for load correctness and eviction lifetime tracking. - Add focused coverage for grouped-store L2 write batching.
- Release finished requests once KV has been copied into store staging buffers. - Keep store and commit futures tracked in the existing background save queue. - Update worker completion tests for background commit reporting.
- Restore bounded contiguous store staging spans for cold save performance. - Re-enable io_uring store coalescing so IPC keeps large physical writes. - Remove stale slot-level L1 admission plumbing and update focused tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
The previous prefix-aware L1 attempt split each prompt save into per-slot physical L1/store admission. In evict-mode benchmarks this pushed DaseR cold TTFT from the expected ~2.8s range to ~3.7s. The root cause was the cold save path, not the warm load path.
This PR restores the cold path to contiguous physical spans while keeping the replacement policy pluggable for future logical prefix-aware work.
Test plan
PYTHONHASHSEED=0 pytest -q -m "not integration" --ignore=tests/integration --ignore=tests/connector/test_daser_connector.py --ignore=tests/connector/test_gds_transfer.py tests/passespre-commit run --all-filespassesRan:
/data/zwt/vllm/bin/pytest -q tests/transfer /data/zwt/vllm/bin/pytest -q tests/server/test_ipc_server.py tests/connector/test_daser_connector.py /data/zwt/vllm/bin/pytest -q tests/transfer tests/server/test_ipc_server.py tests/connector/test_daser_connector.py -k "iouring or replacement or build_staging_store_batches or transfer_store_preserves_span_order or get_finished" /data/zwt/vllm/bin/ruff check git diff --checkResults:
tests/transfer: 40 passedtests/server/test_ipc_server.py tests/connector/test_daser_connector.py: 124 passedruff check: passedgit diff --check: passedruff checkandruff formatpassedEvict-mode benchmark, Qwen3-8B, 200 prompts, input 8192, output 1, max concurrency 8:
DaseR metrics from the same run:
Large benchmark store files were removed after the run.
Checklist
pre-commit run --all-filespasses