Repository navigation
perf(vllm): skip lookups covered by the local cache - #297
ArnavBalyan merged 2 commits into
Conversation
| max_external_tokens = ( | ||
| (request.num_tokens - 1) // self._sched_bs * self._sched_bs | ||
| ) |
There was a problem hiding this comment.
🔵 Low · Reusable-token cap is now computed twice and the two copies can drift
max_external_tokens is the same expression as the clamp a few lines down (ext = max(0, (request.num_tokens - 1) // self._sched_bs * self._sched_bs)). The early return only matches the old behavior because the two expressions agree. If one is edited later without the other, the shortcut will start dropping loads that a lookup would have returned.
Reuse the variable in the clamp. The max(0, …) is not needed, because token_len >= self._sched_bs already guarantees num_tokens >= 1:
if ext == request.num_tokens:
ext = max_external_tokensThe new check also covers the token_len < self._sched_bs early return: in that case max_external_tokens == 0 <= num_computed_tokens. So that branch can go too.
ArnavBalyan
left a comment
There was a problem hiding this comment.
LGTM Thanks for the changes!
The vLLM connector queries OpenLake even when the local cache already covers every reusable full block. For example, a 1,024-token request with 16-token blocks and 1,008 locally cached tokens still performs a lookup before returning zero additional tokens.
Compute the maximum reusable block boundary once. Use it both to skip unnecessary lookups and to clamp full external hits. Preserve the original lookup bound when external tokens can help, so the coordinator can still apply its EAGLE adjustment.
Add CPU regression tests for exact and partial block boundaries, several block sizes, empty and sub-block prompts, single-block prompts, useful external hits, and the bound passed to the coordinator.
Validation:
uv run --no-project --python 3.12 python external/connectors/vllm/tests/test_openlake_adapter.py -v.git diff --checkpasses.