[Frontend] Add x-vllm-* response headers for per-request stats

[vrdn-23]

Summary

Adds an opt-in --enable-request-stats-headers flag that attaches per-request timing and token-count headers to non-streaming OpenAI responses (chat, completion, responses):

x-vllm-total-time, x-vllm-queue-time, x-vllm-prefill-time,
x-vllm-decode-time, x-vllm-inference-time,
x-vllm-prompt-tokens, x-vllm-completion-tokens,
x-vllm-cached-tokens, x-vllm-time-per-output-token

Single source of truth. The timing intervals are computed in exactly one place — IterationStats.update_from_finished_request — which now returns the FinishedRequestStats it builds. Both the existing Prometheus path and the new headers middleware consume the same object; the middleware does no arithmetic, only formatting. This is a deliberate refactor on top of my earlier prototype: the previous version recomputed intervals inside the middleware, creating a third copy of the same math.

Headers, not body. Putting metrics in HTTP headers (rather than extending the response body) keeps the OpenAI response schema unchanged, so strict client validators (OpenAI SDK in some configurations, Pydantic-based proxies) don't need to be updated.

Streaming and error responses unchanged. The middleware is a no-op when request_metadata.finished_stats is None, which covers streaming (no terminal RequestOutput), errors, and non-OpenAI routes.

Bonus refactor: extract FinishReason to a leaf module

The first commit needed RequestResponseMetadata to carry a FinishedRequestStats field. That ran into a Pydantic forward-reference resolution failure: FinishedRequestStats lives in vllm.v1.metrics.stats and annotates finish_reason: "FinishReason", but FinishReason was defined in vllm.v1.engine.__init__ — which already imports from vllm.v1.metrics.stats at runtime. stats.py hid the FinishReason import under TYPE_CHECKING to break that cycle, which left the forward-ref string unresolvable when Pydantic introspected the dataclass.

The first commit worked around this with a model_rebuild(_types_namespace={"FinishReason": _FinishReason}) call at the bottom of protocol.py. The second commit fixes the structural issue properly:

• New leaf module vllm/v1/finish_reason.py holds FinishReason and FINISH_REASON_STRINGS. It imports nothing else from vllm.v1.*.
• vllm/v1/engine/__init__.py re-exports the symbols, so every existing from vllm.v1.engine import FinishReason keeps working unchanged. Class identity is preserved (re-export, not redefinition).
• stats.py imports FinishReason at module top level. The TYPE_CHECKING dodge is gone.
• The model_rebuild block in protocol.py is deleted; the forward reference resolves naturally.

Net: −1 line, 4 files touched, no behavior change. This makes future consumers of FinishedRequestStats (loggers, tracers, RPC schemas) "just work" without needing the rebuild trick.

Why this isn't a duplicate

• Supersedes [Feature]: Per-Request Timing Headers (--enable-request-stats-headers) #38572 (also mine). That PR is the same feature without the single-source refactor; it has been closed in favor of this one. The old branch's build_request_stats_headers recomputed the five timing intervals inside the middleware; this version reuses FinishedRequestStats directly.
• Different from feat(openai): add per-request timing metrics and completion_tokens_de… #36383 (DRAFT, last touched 2026-04-08). feat(openai): add per-request timing metrics and completion_tokens_de… #36383 adds a metrics field to ChatCompletionResponse body and bundles a completion_tokens_details.reasoning_tokens change. This PR is headers-only and makes no body-schema changes. The two could coexist; this one is targeted at the "don't change response schemas" use case.

Test Plan

# Unit + middleware tests (no GPU)
.venv/bin/python -m pytest \
    tests/entrypoints/openai/test_request_stats_headers.py \
    tests/v1/metrics/test_stats.py -v

# Live smoke test
vllm serve facebook/opt-125m --enable-request-stats-headers --port 8000
curl -is http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"facebook/opt-125m","prompt":"Hello","max_tokens":10}' \
    | grep -i '^x-vllm-'

Manual verification (Linux + CUDA):

• All 9 x-vllm-* headers present on non-streaming responses; values internally consistent (prefill + decode ≈ inference, queue + inference + overhead = total).
• Streaming requests ("stream": true) confirmed to NOT emit x-vllm-* headers.
• Server started without the flag confirmed to NOT emit headers.
• Prefix-cache hits (x-vllm-cached-tokens > 0) confirmed on repeated identical prompts past the block boundary.

Test Results

• 16 / 16 tests passed locally on Linux (5 new headers tests + 11 existing metrics tests, including a new assertion that update_from_finished_request returns the same object it appends).
• pre-commit run clean on all touched files (ruff check, ruff format, mypy-3.10, SPDX headers, all other hooks).
• Live smoke test on Linux + CUDA: facebook/opt-125m serving the /v1/completions endpoint with --enable-request-stats-headers returns all 9 headers with self-consistent values.

AI assistance

This change was developed with AI assistance (Claude Opus 4.7 via Claude Code). Every line was reviewed by me. I ran the test suite locally and verified the live behavior on my own hardware before opening this PR.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[gemini-code-assist]

Code Review

This pull request introduces per-request timing and compute statistics as x-vllm-* response headers for non-streaming OpenAI-compatible requests, controlled by a new --enable-request-stats-headers flag. The implementation includes a FastAPI middleware, updates to the V1 engine's output processing to capture FinishedRequestStats, and propagation of these stats through the OpenAI serving layer. Feedback indicates that finished_stats should also be integrated into PoolingRequestOutput to support embedding endpoints. Furthermore, it is recommended to calculate and attach these statistics before the RequestOutput is queued to avoid potential timing issues in asynchronous environments.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e725b27b42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[vrdn-23]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72507ce31d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Aggregate finished stats for parallel sampling

For non-streaming requests with n > 1, ParentRequest.get_outputs() only emits a combined RequestOutput when the last child finishes, but this line copies finished_stats from just that one child request state. The OpenAI serving layers then use final_res.finished_stats for x-vllm-* headers, so headers like x-vllm-completion-tokens and timing fields reflect only the last branch instead of the whole API request (while usage is aggregated across choices). This produces misleading per-request stats whenever parallel sampling is used.

Useful? React with 👍 / 👎.

[cjackal]

Just a question: what benefits of receiving the per-requests stats via custom response headers over collecting via distributed tracing? There already are so-called AI gateways (smg, envoy AI gateway, litellm proxy to name a few) whose whole purpose is to collect the per-request stats and transform into a business-standard trace protocols, and vllm integrates well with these. Is there a specific usecase where distributed tracing does not cover well?

[vrdn-23]

@cjackal Thanks for the interest in the PR!

So I am aware of a use-case where we have OTel data available as spans and they can be queried using Grafana. I think the value in adding headers is based on the fact that this gives us an easy in-line access path, which opens up much more configurability for proxies that are being run in front of vLLM deployments.

In our use-case, we have our own internal implementation of an AI Gateway, where we want to implement latency aware load balancing and also have cost-based rate limiting based on the inference time taken by the request. Since e2e wall time bundles queue waiting into the same number as compute, it's the wrong basis for cost attribution when multiple teams share a model — they'd be charged for each other's queueing. Exposing the headers gives us the opportunity to calculate cost and make routing decisions inline, without going through an OTel pipeline lookup that carries span-export lag (typically several seconds before a finished request's span is queryable) and a separate per-request query to the trace backend. It also helps us return values like per-request-cost in real time without having to thread that through the OTel pipeline.

For cross-service spans, audit, post-hoc analysis, OTel is clearly the right tool. I would think this to be a complementary source of the same data, without it being a replacement. Hope this makes sense and happy to answer any questions further!

[vrdn-23]

@chaunceyjiang @DarkLight1337 @russellb @robertgshaw2-redhat
Any chance I could get a preliminary review on this PR?