[Bugfix] Replace assertion with guard in finished_sending KV transfer - #5
Open
Jackie2049 wants to merge 1 commit into
Open
[Bugfix] Replace assertion with guard in finished_sending KV transfer#5Jackie2049 wants to merge 1 commit into
Jackie2049 wants to merge 1 commit into
Conversation
In disaggregated prefill/decode deployments, a race condition exists where a request may be aborted (user cancellation, timeout, preemption) before the KV connector reports finished_sending. The previous code used `assert req_id in self.requests` which causes a hard crash instead of gracefully handling the race. Replace the assertion with a defensive guard check that logs a warning and skips the request. This is safe because aborted requests already have their blocks freed during the abort path. Fixes #1 Co-Authored-By: Claude
Jackie2049
force-pushed
the
fix/finished-sending-race-condition
branch
from
June 6, 2026 17:18
48f7882 to
449b6f5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1
In disaggregated prefill/decode (PD separation) deployments, a race condition exists in the scheduler's handling of
finished_sendingfrom the KV connector output.Problem
At
vllm/v1/core/sched/scheduler.py:2153:When a request is aborted (user cancellation, timeout, preemption, or KV load failure) while a KV transfer is still in flight, the request is removed from
self.requests. When the KV connector later reportsfinished_sendingfor that request, the assertion fires and crashes the entire serving process.Timeline of the race
Fix
Replace the assertion with a defensive guard:
This is safe because:
_free_request)continueensures remaining requests in the loop are still processedTesting