[Feature] Add request overload control with HTTP 429 queue timeout eviction

[Jackie2049]

Summary

Closes #3

Adds a configurable max_waiting_time that evicts requests waiting too long in the scheduler queue, returning HTTP 429 Too Many Requests to clients.

Problem

In production vLLM deployments, there is no mechanism to handle request queue overload:

• Requests sit in the waiting queue indefinitely with no timeout
• Clients cannot distinguish between "waiting in queue" and "service broken"
• Stale requests (whose clients have already disconnected) still consume scheduler resources
• No feedback mechanism to inform clients that the service is overloaded

Changes

Core (4 files)

1. vllm/v1/request.py: Add FINISHED_OVERLOAD to RequestStatus enum + mapping to FinishReason.OVERLOAD
2. vllm/v1/engine/__init__.py: Add OVERLOAD = 5 to FinishReason enum + "overload" to FINISH_REASON_STRINGS
3. vllm/config/scheduler.py: Add max_waiting_time: float = 0 field to SchedulerConfig
4. vllm/v1/core/sched/scheduler.py: Add eviction logic at the start of schedule() — scans waiting queues and finishes expired requests

HTTP Layer (6 files)

5. vllm/entrypoints/openai/engine/protocol.py: Add OverloadError exception with HTTP 429 status
6. vllm/entrypoints/openai/engine/serving.py: Extend _raise_if_error() to handle "overload" finish_reason + update streaming error handler
7. vllm/entrypoints/openai/api_server.py: Register OverloadError exception handler
8. vllm/entrypoints/openai/chat_completion/serving.py: Catch OverloadError in streaming path
9. vllm/entrypoints/openai/completion/serving.py: Catch OverloadError in streaming path
10. vllm/entrypoints/openai/responses/serving.py: Catch OverloadError in streaming path

Design

Request arrives → Scheduler queue → max_waiting_time exceeded?
                                         ↓ Yes
                                  FINISHED_OVERLOAD → HTTP 429
                                         ↓ No
                                  Normal scheduling

• Default disabled (max_waiting_time = 0): backward compatible
• Eviction happens before each scheduling cycle (not on a timer)
• FCFS ordering ensures oldest requests are evicted first

Testing plan

• Code review: all changes are additive, no existing behavior modified
• Unit test: verify eviction of requests exceeding wait threshold
  • Offline logic validation (4 scenarios, ALL PASS):
  • Backward compatibility: max_waiting_time=0 → zero evictions (100s wait still kept)
  • Eviction timing: only requests with wait > max_waiting_time evicted; exact-threshold kept
  • No eviction when disabled: max_waiting_time=0 with extreme wait times → no eviction
  • Status propagation: evicted requests → FINISHED_OVERLOAD, others → unchanged
  • Validation script: tools/test_overload_control_429.py in rollout-infra repo
• Integration test: send burst, verify 429 responses for excess requests
  • E2e test commands prepared (needs vLLM installed on GPU server)
  • Will validate: HTTP 429 response, streaming error handling, chat/completions/responses endpoints
• Load test: measure impact on throughput and tail latency