Skip to content

feat(frontend): expose cumulative prefix cache metrics through vllm - #1045

Draft
xiaguan wants to merge 1 commit into
mainfrom
feat/prefix-cache-metrics
Draft

xiaguan wants to merge 1 commit into
mainfrom
feat/prefix-cache-metrics

Conversation

@xiaguan

@xiaguan xiaguan commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

The frontend currently leaves vLLM prefix-cache queries and hits at zero even when a scheduler reuses cached tokens. Add cumulative request/query/hit counters to SchedulerMetrics and convert them to per-send deltas in both the watch and stepped bridges. Prefix-only steps send stats without waiting for output tokens; repeated snapshots contribute zero and coalesced snapshots retain every increment.

This provides the frontend contract for kern; existing model schedulers retain zero counters until they populate them.

Validation: 74 frontend unit tests pass, including all coalescings of four admissions, repeated snapshots, the watch publisher, and prefix-only stepped output. Frontend Clippy with all targets and warnings denied, plus workspace formatting, pass.

Signed-off-by: JinYan Su <751080330@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant