fix(hcu): prevent MRV2 async output stalls

[alexanderbin123]

Summary

• route HCU Model Runner V2 asynchronous output copies through the accelerator-generic event API
• bound output-copy event waits with VLLM_ENGINE_ITERATION_TIMEOUT_S so workers cannot silently block forever
• preserve the existing asynchronous copy stream and propagate timeouts through the worker FAILURE response path

Root cause of the MRV2 async-output stall

MRV2 copies sampled tokens and related output tensors to CPU on a dedicated output stream. Upstream AsyncOutput records a CUDA-specific blocking event for that copy and later waits with an unbounded call:

self.copy_event = torch.cuda.Event(blocking=True)
self.copy_event.synchronize()

On HCU, the event/copy completion wait was observed to occasionally stop returning. Because synchronize() has no deadline, the async-output thread could remain blocked forever. The worker would then enqueue neither a ModelRunnerOutput nor a failure response, so the engine and HTTP request appeared to hang silently.

This MR addresses the confirmed failure mechanism in two layers:

• use the accelerator-generic torch.Event API instead of the CUDA-only blocking event path
• replace the unbounded wait with event.query() polling bounded by VLLM_ENGINE_ITERATION_TIMEOUT_S

If the event does not complete before the deadline, get_output() raises TimeoutError; WorkerProc.enqueue_output() converts it into a FAILURE response, allowing the engine to fail explicitly instead of waiting forever.

The lower-level reason an individual HCU runtime event or asynchronous D2H copy can fail to complete is not established by this MR. The fix hardens the vLLM/HCU integration boundary so such a runtime condition cannot become an unbounded silent worker stall.

Scope

This MR targets v0.25.1 and contains only the MRV2 stability fix and its regression tests. It does not include model adaptation, MoE, attention, or other operator changes.

Validation

• portable contract suite: 953 passed, 37 deselected
• focused MRV2/worker lifecycle suite: 70 passed
• TP8 + MRV2 + async scheduling + FULL_AND_PIECEWISE CUDA Graph concurrent stress: all 8 requests completed
• long decode crossed the prior failure region and completed 7168 tokens with HTTP 200
• production-boundary, patch-coverage, compile, and diff checks passed

Review follow-up: test module-state isolation

Problem

test_hcu_model_runner_v2_is_thin_upstream_adapter temporarily replaces vllm.v1.worker.gpu.model_runner and re-imports the HCU MRV2 adapter. The test restored the sys.modules entry, but left the vllm_hcu.v1.hcu_model_runner_v2 attribute on the parent package pointing to the temporary adapter module.

When test_hcu_mrv2_runner_routes_async_output_before_upstream_init ran afterward, from vllm_hcu.v1 import hcu_model_runner_v2 returned that stale module. The runner then updated AsyncOutput on the temporary upstream module instead of the restored real module, making the test suite order-dependent.

Impact

• the new async-output tests passed in isolation but failed after the lifecycle adapter test
• the minimal ordered reproduction produced 1 passed, 1 failed
• this was a test-state isolation defect; no production failure was reproduced from this condition

Fix

Commit 8d8ce8a snapshots, removes, and restores both the adapter's sys.modules entry and its parent-package attribute in a finally block. This preserves the thin-adapter assertions while guaranteeing cleanup on success, import failure, or assertion failure.

Additional validation

• ordered regression reproducer: 2 passed
• lifecycle plus async-output suite: 40 passed
• MRV2, worker-framework, and shutdown suite: 45 passed
• compileall and git diff --check: passed