[Bug] Race condition in scheduler finished_sending KV transfer causes production crash

[Jackie2049]

Motivation

In disaggregated prefill/decode (PD separation) deployments, when KV cache blocks are transferred between prefill and decode instances, a race condition exists that can cause the scheduler to crash with an AssertionError in production. This makes the system fragile under real-world conditions where requests may be aborted, timed out, or cancelled while KV transfers are in flight.

Problem Identification

Location

vllm/v1/core/sched/scheduler.py, line 2151-2154:

for req_id in kv_connector_output.finished_sending or ():
    logger.debug("Finished sending KV transfer for request %s", req_id)
    assert req_id in self.requests  # ← PROBLEM: crashes on race condition
    self._free_blocks(self.requests[req_id])

Root Cause Analysis

In distributed KV transfer scenarios, the following race condition can occur:

1. Request A is submitted and begins KV cache transfer to a remote instance
2. During transfer, Request A is aborted (user cancellation, client disconnect, or timeout)
3. The scheduler processes the abort and removes Request A from self.requests
4. The KV connector completes the send operation and reports finished_sending = ["A"]
5. The scheduler hits assert req_id in self.requests → AssertionError → crash

This is not a theoretical concern — in production deployments with PD separation:

• Clients frequently disconnect (mobile users, flaky networks)
• Request timeouts are common (long reasoning tasks)
• Load balancers may retry and cancel original requests

Why this matters

The assert is a developer-level check that was likely intended for debugging during development, but in production it causes a hard crash of the entire serving process rather than gracefully handling the race. The correct behavior should be to skip cleanup for already-removed requests.

Proposal

Fix: Replace assertion with defensive guard check

for req_id in kv_connector_output.finished_sending or ():
    logger.debug("Finished sending KV transfer for request %s", req_id)
    if req_id not in self.requests:
        logger.warning(
            "Request %s not found when finishing KV send; "
            "may have been aborted during transfer", req_id
        )
        continue
    self._free_blocks(self.requests[req_id])

Why this is safe

• If the request was already removed, its blocks were already freed during the abort path (_free_request)
• Skipping the duplicate free is the correct behavior
• The continue ensures no other cleanup logic is skipped for remaining requests in the loop
• The warning log preserves observability for debugging

Testing plan

1. Unit test: simulate abort + finished_sending race condition
2. Verify no memory leak (blocks are not double-freed)
3. Verify normal KV transfer flow is unaffected

Scope

This is a minimal, focused fix — a single assert → if/continue change. No architectural changes needed.