[Feature]: Per-request timing metrics in response body

[nv-nedelman-1]

🚀 The feature, motivation and pitch

Summary

Add an opt-in capability for vLLM to return per-request timing and compute metrics in the response body of OpenAI-compatible completion endpoints. The feature would be gated by a server-level flag (e.g. --enable-per-request-metrics) plus a per-request parameter (e.g. include_metrics: true), and would expose a structured metrics object alongside the normal response payload.

This issue is intentionally filed as a companion / counter-proposal to #36189, which proposes exposing the same information via HTTP response headers. A draft implementation of the body-based approach already exists in #36383.

Motivation

(The motivation here largely overlaps with #36189; only real difference is delivery mechanism)

vLLM already tracks detailed per-request timing internally (queue time, prefill time, decode time, inter-token latency, etc.) via RequestStateStats, and surfaces aggregated versions of this data through Prometheus metrics and OpenTelemetry traces. Those are backend-only, aggregate observability tools — they do not let an API consumer see where time was spent on their specific request.

Exposing this data to API consumers directly unlocks two use cases:

1. Per-user / per-tenant billing and cost attribution

Operators running multi-tenant deployments need to attribute GPU time and token counts back to individual requests for usage-based billing and chargeback. Prometheus gives aggregates per endpoint/model, not per request. Having generation_time_ms, queue_time_ms, prompt_tokens, and completion_tokens in the response body means the billing system that is already parsing the response JSON has everything it needs in one place, with no separate infrastructure.

2. Per-request SLA attribution and latency debugging

Application developers building on top of vLLM currently see only total latency. With a structured metrics field they can distinguish:

• Time waiting in the scheduler queue (capacity issue)
• Time in prefill / time-to-first-token (prompt cost)
• Time in decode / inter-token latency (generation cost)

This makes it trivial to add per-request SLA tracking to an application without running a Prometheus scraper or an OTEL collector.

Why the response body (and not only headers, as in #36189)

The headers-only approach proposed in #36189 is attractive for proxies and load balancers, but it has a hard limitation that a body-based approach does not:

• Streaming support - Headers are flushed before the first token. Metrics that are only known at end-of-generation (generation_time_ms, mean_itl_ms) cannot be carried in headers without HTTP trailers, which have very limited client/proxy support. A final SSE event (or final chunk) carries the completed metrics naturally and is trivial for clients to consume. |

The headers and body approaches are complementary, not mutually exclusive. Routers that want real-time hot-path signals benefit from headers; billing pipelines and SDK users need the body. The position of this RFE is that the body-based API covers cases that headers cannot.

Proposal

Opt-in flags (double gate)

vllm serve <model> --enable-per-request-metrics

Plus a per-request parameter:

{ "model": "...", "messages": [...], "include_metrics": true }

Both must be set for metrics to be computed and returned. Default: off (no behavior change for existing users, no CPU overhead for deployments that do not opt in).

Response body additions

A new optional metrics field on ChatCompletionResponse, ChatCompletionStreamResponse, CompletionResponse, and CompletionStreamResponse:

Field	Unit	Description
time_to_first_token_ms	ms	Time from scheduling to first output token
queue_time_ms	ms	Time spent waiting in the scheduler queue
generation_time_ms	ms	Total decode time (excludes queue wait)
mean_itl_ms	ms	Mean inter-token latency during decode
tokens_per_second	count / s	Output throughput for this request

(Prefill-time, cached-token, and per-phase GPU-time fields can be added incrementally as they become cleanly attributable to a single request.)

Streaming behavior

• For non-streaming responses: metrics is populated on the final response object.
• For streaming responses: metrics is emitted on the final SSE chunk — consistent with how OpenAI already emits final usage when stream_options.include_usage=true.

Alternatives

Alternative	Why Not
Headers only (#36189)	Cannot carry end-of-generation timing for streaming cases.
Prometheus only	Aggregate, not per-request; requires scraping infrastructure; invisible to the caller.
OpenTelemetry traces	Requires a tracing backend; not accessible over plain HTTP; high operational overhead.
Always-on body fields	Adds CPU cost and response size for deployments that don't care. Opt-in keeps zero overhead as the default.
Custom middleware	Can only observe wall-clock time; cannot reach engine-internal timings (queue / prefill / decode).

Additional context

Relationship to existing work

• Draft implementation: feat(openai): add per-request timing metrics and completion_tokens_de… #36383 (already implements the core shape proposed here, including PerRequestTimingMetrics and the double-gate flag).
• Companion RFE for headers: [Feature]: Per-Request Timing Headers (--enable-request-stats-headers) #36189. That issue remains a valid proposal for router-facing, hot-path metrics; this RFE is scoped to the body-side API that headers cannot replace.

Why opt-in?

• Zero overhead when disabled - no extra computation, no extra fields serialized.
• Response size - clients doing strict schema validation shouldn't see new fields unless they ask for them.
• Information disclosure - timing data can reveal server capacity characteristics; operators should choose to expose it.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

I don't think its appropriate to add this information into the response body of a well-defined API

Much of the functionality you are describing should be implemented via OTEL tracing. I agree that this requires implementing an OTEL stack, but the functionality you are describing benefitting from this is production:

1. Per-user / per-tenant billing and cost attribution
2. Per-request SLA attribution and latency debugging

So Im stuggling to think about the set of users who want to accomplish (1) and (2) but dont have the capability to setup an OTEL stack

[robertgshaw2-redhat]

others can feel free to opine if they have different thoughts though. opt-in does make this a bit more okay

[lawcontinue]

Great proposal. We implemented something similar in Hippo — a lightweight local LLM manager with built-in audit logging that records per-request timing, token counts, model, and status for every API call.

A few thoughts from our experience:

1. Double-gate is the right pattern

We use the same approach — server-level opt-in (--audit-log) plus no per-request flag needed (if audit is enabled, everything is logged). The double-gate in this proposal (server flag + per-request param) adds more granular control, which makes sense for vLLM's multi-tenant use case.

2. Response body vs headers

Strongly agree with the body-based approach for streaming. We log latency_ms in our JSONL audit trail and it works well. For streaming, we compute metrics at stream end and log them. This is straightforward in Python with FastAPI's streaming response.

3. Fields we found useful beyond the proposed set

Our audit log also captures:

• client_ip — for multi-tenant attribution
• api_key_hash — for usage tracking without exposing keys
• error — error message when status != 200

These might be worth considering for the metrics object, especially in multi-tenant deployments.

4. Performance overhead

In our testing, adding per-request metrics adds <0.1ms overhead (just timestamp diffs and dict construction). The opt-in design means zero overhead when disabled, which is the right default.

5. Question on streaming metrics delivery

The proposal says metrics are emitted on the final SSE chunk. Have you considered a separate /metrics endpoint where clients can query metrics by request_id? This would allow decoupling metrics collection from the inference response, which could be useful for billing pipelines that process metrics asynchronously.

[vrdn-23]

Just wanted to chime in here cause I'm actually the person who raised the other RFC for having the per-request timing in headers.
I am also personally against the idea of modifying response bodies when it is supposed to be a well-defined response API, but I am also not aware of a good solution to solve the problem with streaming.

Opt-in headers seems to me a reasonable choice for non-streaming responses, but I also think there is a larger discussion in #39979 that looks to solve this using OpenTelemetry. I do think there needs to exist a solution that lets us capture that information outside of scraping metrics, but I am curious to hear if @noooop or @markmc have some thoughts on which is a cleaner interface. This can help de-duplicate work across different PRs/implementations since we're all looking to solve the same problem, but are looking for guidance from the maintainer team and on what is the preferred solution. :)

cc @njhill @russellb @simon-mo

[vrdn-23]

Wanted to chime in with a slightly different framing that I think might bring the headers and body-based proposals closer together — happy to be wrong about any of this, but it seemed worth floating.

On scraping / OTEL as the full answer

Nothing here is meant to push back on OTEL — it's a great tool for cross-service tracing and after-the-fact analysis. A couple of gaps I'm not sure it covers cleanly even with a full stack deployed, though:

• Prometheus is aggregate-only — adding a request_id label isn't really viable (cardinality), so per-request billing and per-request SLA attribution end up outside what scraping can serve, regardless of how much infra an operator stands up.
• OTEL traces can do per-request, but there's usually seconds-to-minutes of lag before traces land in the backend, and the client holding the response has to make a separate correlated query to recover its own timing. Workable for offline analysis; probably too slow for hot-path LB routing or for anything that wants timing data on the same path as the response.
• For real-time load balancer routing specifically, scrape intervals (15–60s) seem structurally mismatched with decisions the LB needs to make on the next request. Envoy/NGINX-style routers already consume response headers for routing weight, which made me think that might be where this fits more naturally.

None of these feel like arguments against OTEL — more that response-attached metadata and OTEL might be complementary rather than substitutes.

On the "response-attached timing" pattern

Something that might be worth grounding the discussion in: the OpenAI-compatible ecosystem has already converged on a split pattern here, and I think that gives a useful frame for the headers-vs-body question.

• Timing → response headers. OpenAI emits openai-processing-ms on every response, alongside x-request-id. Anthropic emits request-id the same way. The reasoning is mechanical: headers flush before the body, so they carry what's knowable at flush time, and they're consumable by LBs and proxies without body parsing.
• End-of-generation stats → final SSE chunk. For anything not knowable until decode completes, the established shape is the final chunk — OpenAI's stream_options: {"include_usage": true} gates exactly this and appends a final chunk with the usage block.

HTTP trailers would have been the "clean" answer for streaming, but the WHATWG SSE spec doesn't expose trailers on EventSource, most fetch-based SSE readers don't surface them either, and intermediaries often strip them (Envoy discussion is a good illustration of how messy trailer handling gets once proxies are in the path). That's why trailers don't seem to have taken hold in this ecosystem.

Under that frame, the two approaches being discussed might be complementary halves of the same pattern rather than alternatives:

• Non-streaming → response headers (the shape in [Feature]: Per-Request Timing Headers (--enable-request-stats-headers) #38572), which has direct precedent in openai-processing-ms and is cheap enough that benchmarks so far haven't shown meaningful overhead.
• Streaming → extend the existing usage object in the final chunk (roughly what feat(openai): add per-request timing metrics and completion_tokens_de… #36383 is going at), rather than introducing a new top-level metrics field. include_usage already gates the final-chunk usage object, so adding fields like queue_time_ms, time_to_first_token_ms, generation_time_ms, mean_itl_ms, and tokens_per_second onto the existing usage shape would be a pretty light extension of something users are already opting into and parsing.

On SDK compatibility: the OpenAI Python SDK's response models are deliberately configured to allow extra fields for forward compatibility, and the Node/TypeScript SDK doesn't do runtime validation, so extending usage with additional fields shouldn't break callers using the OpenAI client pointed at a vLLM URL. Empirically, vLLM responses already carry a number of non-OpenAI fields today (prompt_token_ids, kv_transfer_params, stop_reason, prompt_logprobs, etc.) through those SDKs without issue, which I think is reasonable evidence that the extension is safe in practice.

That second piece is the part I'd genuinely like feedback on — extending usage rather than adding a new top-level body field feels (to me) like it addresses the "don't modify a well-defined API body" concern more directly, since usage is where this class of data already lives in the OpenAI schema. But there may be reasons that's a worse idea than I'm seeing.

One scope note

On the headers side, putting token counts in headers goes slightly beyond strict OpenAI precedent — OpenAI keeps those in the usage body field rather than in headers. The argument for surfacing them as headers is LB/proxy consumption without body parsing, but if that duplication feels off they're easy to drop; timing is the core of what this is trying to address.

Mostly wanted to see if there's appetite for unifying the two proposals around the "headers for non-streaming, extended-usage for streaming" shape, or if folks see tradeoffs with that I haven't thought through.

cc @robertgshaw2-redhat @nv-nedelman-1 @noooop @markmc @simon-mo

[vrdn-23]

Hey folks,
Just wanted to get some final opinions on this design decision and looking forward to any input specifically about what feels like the right path forward.
cc @robertgshaw2-redhat @markmc @simon-mo @DarkLight1337 @njhill @russellb