Skip to content

perf(vllm): skip lookups covered by the local cache - #297

Merged
ArnavBalyan merged 2 commits into
openlake-project:mainfrom
anandheritage:perf/skip-redundant-kv-lookups
Oct 9, 2026
Merged

ArnavBalyan merged 2 commits into
openlake-project:mainfrom
anandheritage:perf/skip-redundant-kv-lookups

Conversation

@anandheritage

@anandheritage anandheritage commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

The vLLM connector queries OpenLake even when the local cache already covers every reusable full block. For example, a 1,024-token request with 16-token blocks and 1,008 locally cached tokens still performs a lookup before returning zero additional tokens.

Compute the maximum reusable block boundary once. Use it both to skip unnecessary lookups and to clamp full external hits. Preserve the original lookup bound when external tokens can help, so the coordinator can still apply its EAGLE adjustment.

Add CPU regression tests for exact and partial block boundaries, several block sizes, empty and sub-block prompts, single-block prompts, useful external hits, and the bound passed to the coordinator.

Validation:

  • All 24 connector test methods pass with Python 3.12.13: uv run --no-project --python 3.12 python external/connectors/vllm/tests/test_openlake_adapter.py -v.
  • git diff --check passes.
  • The tests use import stubs and a mock coordinator. Live vLLM/GPU integration and latency measurements have not run.
  • Rust formatting, Clippy, and workspace tests have not run locally because Cargo is unavailable on this macOS host. These remain for project CI.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 1 low-severity finding

Reviewed commit c2ed66a.

Comment on lines +1548 to +1550
max_external_tokens = (
(request.num_tokens - 1) // self._sched_bs * self._sched_bs
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low · Reusable-token cap is now computed twice and the two copies can drift

max_external_tokens is the same expression as the clamp a few lines down (ext = max(0, (request.num_tokens - 1) // self._sched_bs * self._sched_bs)). The early return only matches the old behavior because the two expressions agree. If one is edited later without the other, the shortcut will start dropping loads that a lookup would have returned.

Reuse the variable in the clamp. The max(0, …) is not needed, because token_len >= self._sched_bs already guarantees num_tokens >= 1:

if ext == request.num_tokens:
    ext = max_external_tokens

The new check also covers the token_len < self._sched_bs early return: in that case max_external_tokens == 0 <= num_computed_tokens. So that branch can go too.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues found

Reviewed commit 05be271.

@ArnavBalyan ArnavBalyan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Thanks for the changes!

@ArnavBalyan
ArnavBalyan merged commit 0aceab0 into openlake-project:main Oct 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants