[Feature] Request overload control with HTTP 429 response and queue timeout eviction

[Jackie2049]

Motivation

In production vLLM deployments, there is currently no mechanism to handle request queue overload. When the system receives more requests than it can process:

1. Requests sit in the waiting queue indefinitely with no timeout
2. Clients cannot distinguish between "waiting in queue" and "service broken"
3. Long-queued requests may already have timed out on the client side, but still consume scheduler resources on the server
4. There is no feedback mechanism to inform clients that the service is overloaded

This is a critical gap for production deployments where:

• Traffic is bursty and unpredictable
• Multiple clients share a single vLLM instance
• SLO compliance requires fast failure over slow hanging

Problem Identification

Current behavior

When vLLM's waiting queue fills up:

• New requests are accepted into the queue regardless of current load
• There is no upper bound on wait time for queued requests
• Clients receive no indication of queue length or estimated wait time
• Stale requests (whose clients have already disconnected) continue to occupy queue slots
• The scheduler processes them in FCFS order with no eviction mechanism

Impact

• Poor client experience: HTTP connections hang with no response
• Wasted compute: Abandoned requests are still processed
• No backpressure: Clients have no signal to throttle or retry
• Cascading failures: Queue grows unbounded → scheduler thrashing → all requests slow down

What exists upstream

• --max-num-seqs: limits concurrent running sequences, but does not limit queue wait time
• --max-model-len: limits context length, not queue size or age
• Prometheus metrics: expose queue length, but require external monitoring to act on

Missing: An in-band mechanism within vLLM to detect and reject stale/overloaded requests.

Proposal

Overview

Add a configurable maximum wait time for queued requests. When a request exceeds this threshold, it is:

1. Evicted from the waiting queue
2. Marked with a new status FINISHED_OVERLOAD
3. Returned to the client as HTTP 429 Too Many Requests with a Retry-After header

Design

1. New configuration parameter

--max-waiting-time SECONDS  # Maximum time a request can wait in queue (default: 0 = disabled)

When set to 0 (default), behavior is unchanged. When set to a positive value, requests waiting longer than this are evicted.

2. New request status

class RequestStatus(IntEnum):
    ...
    FINISHED_OVERLOAD = ...  # Request evicted due to excessive wait time

3. Scheduler change

In _schedule_waiting() (or _schedule_default()), before processing the waiting queue:

if self.max_waiting_time > 0:
    now = time.monotonic()
    evicted = []
    while self.waiting and (now - self.waiting[0].arrival_time) > self.max_waiting_time:
        req = self.waiting.popleft_left()  # FCFS: oldest first
        evicted.append(req.request_id)
    for req_id in evicted:
        self.finish_requests({req_id: RequestStatus.FINISHED_OVERLOAD})

4. HTTP response

In the OpenAI serving layer:

if finish_reason == FinishReason.OVERLOAD:
    raise HTTPException(
        status_code=429,
        detail="Request exceeded maximum waiting time. Please retry later.",
        headers={"Retry-After": str(recommended_retry_seconds)}
    )

5. Finish reason

class FinishReason(str, Enum):
    ...
    OVERLOAD = "overload"

Configuration considerations

• Default disabled (0) to maintain backward compatibility
• Could be combined with existing --max-num-seqs for dual protection (queue size + wait time)
• Future extension: per-priority-class different thresholds

Testing plan

1. Unit test: verify eviction of requests exceeding wait threshold
2. Integration test: send burst of requests, verify 429 responses for excess requests
3. Load test: measure impact on throughput and tail latency with overload control enabled
4. Edge case: verify requests in-flight are not incorrectly evicted

Open questions for community discussion

1. Should this be a CLI flag or purely an environment variable?
2. Should the default be changed from "disabled" to a reasonable value (e.g., 300s)?
3. Should we also expose current queue wait time in a response header for accepted requests?
4. Interaction with priority scheduling: should lower-priority requests be evicted first?