Skip to content

bugfix: defer streaming chat role chunk until generation starts - #1710

Merged
valarLip merged 1 commit into
ROCm:mainfrom
amd-youchen:bugfix-defer-streaming-chat-role-chunk
Jul 28, 2026
Merged

bugfix: defer streaming chat role chunk until generation starts#1710
valarLip merged 1 commit into
ROCm:mainfrom
amd-youchen:bugfix-defer-streaming-chat-role-chunk

Conversation

@amd-youchen

Copy link
Copy Markdown
Contributor

Motivation

The streaming chat handler sent the role: "assistant" chunk as soon as the
request was accepted, before the engine produced anything.
That is not what OpenAI does -- the first chunk is supposed to mean generation
has actually started (vLLM emits it on the first iteration of the result
generator). It also skews client-side metrics: a chat client that times the
first SSE chunk without checking for content records a near-zero TTFT and
counts the real first token as an inter-token latency. The /v1/completions
benchmark path has no such chunk, so existing numbers are unaffected, but the
chat path is a trap for external clients.
This PR defers the role chunk until the first engine chunk arrives, for both
the single-sequence and n>1 fanout paths. It still precedes any content or
tool_calls delta, so client-visible ordering is unchanged.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1710 --add-label <label>

@zufayu
zufayu requested a review from yitingw1 July 28, 2026 01:44
@valarLip
valarLip merged commit 181f529 into ROCm:main Jul 28, 2026
41 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants