Skip to content

feat(rollout): migrate vLLM-Omni to stable 0.28.0 - #413

Merged
CjhHa1 merged 32 commits into
Tencent-Hunyuan:mainfrom
celve:LIN-981
Sep 24, 2026
Merged

CjhHa1 merged 32 commits into
Tencent-Hunyuan:mainfrom
celve:LIN-981

Conversation

@celve

@celve celve commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Migrates UniRL from vllm-omni==0.27.0rc1 to stable 0.28.0 (vllm==0.28.0, transformers==5.12.1, torch==2.13.0+cu130). This is a breaking runtime migration, not only a version pin.

  • Replaces all legacy stage_configs_path / stage_args assets with stable deploy_config YAMLs and registered PipelineConfig topologies.
  • Adapts the flat OmniRequestOutput.outputs contract and routes AR sleep/wake through EngineCore while retaining diffusion worker task ACKs.
  • Adapts the direct vLLM rollout engine added on latest main from its explicit vLLM 0.27 runtime contract to 0.28; native IPC, sleep/wake, collective RPC, and cache-reset APIs remain signature-compatible.
  • Registers UniRL patches through the stable general-plugin entry point so spawned stages receive them.
  • Fixes lane validation before MoE workspace indexing, preventing an invalid lane from aliasing another ubatch's workspace.
  • Keeps only compatibility required by real model runs: HI3 expert mapping, tensor LoRA loading, worker parameter discovery, and Qwen3-Omni thinker LoRA support.

Dependency overrides remain coherent across mutually exclusive engines: the global uv overrides pin diffusers==0.40.0 (exactly what vllm-omni 0.28.0 requires) and kernels>=0.14.1,<0.15 (satisfying vllm-omni's kernels==0.14.1 and sglang 0.5.19's range), so both engine stacks share one diffusers/kernels version; installation docs no longer enable prereleases globally.

Risk closure

  • Removed configuration API: all adapter-selected topologies resolve through stable StageConfigFactory; no production reference to stage_configs_path, stage_yaml, or stage_args remains.
  • HI3 AR mode split: comprehension adapters retain engine_output_type=text; terminal think/recaption uses latent, preserving the legacy generation sampler while still returning text.
  • HI3 t2t/i2t weight sync (behavior change): the vendored 0.27 comprehension YAMLs (hunyuan_image3_t2t.yaml, hunyuan_image3_i2t.yaml) had neither enable_lora nor worker_extension_cls, so ar_lora_passthrough LoRA pushes could not land. The replacement deploy configs (hunyuan_image3_ar_rl.yaml, hunyuan_image3_i2t_rl.yaml) enable LoRA and install HI3ARWeightSyncExtension. This path is not covered by the GPU runs below.
  • Output envelope change: AR token/text extraction reads the stable flat completion fields.
  • Lifecycle split: AR and diffusion stages use their owning control plane, and nested unsupported/error/ACK results fail closed.
  • Workspace corruption: invalid lanes delegate to vLLM before any flattened workspace index is touched; multi-ubatch/multi-lane H20 smoke covers slot isolation.
  • LoRA regressions: SD3 tensor push/checksum/remove/re-push and HI3/Qwen model-specific paths were exercised on GPU.

Test Plan

Exact stable environment on one 8×H20 node:

vllm 0.28.0 / vllm-omni 0.28.0 / transformers 5.12.1
torch 2.13.0+cu130 / torchvision 0.28.0+cu130
tokenizers 0.22.x / diffusers 0.40.0

Static validation:

  • Full repository pre-commit suite: pass.
  • The final diff intentionally contains no added test files or dedicated test workflow; behavioral evidence is recorded below from the completed H20 validation runs.

Real GPU validation:

  • SD3.5: two generations across sleep/wake; real tensor LoRA push, checksum, activation, removal, re-push.
  • Qwen3-Omni TP4: text generation plus audio-only, video-only, and video-with-audio processor/inference paths; sleep/wake.
  • BAGEL: T2I and conditional IT2I, each before and after sleep/wake.
  • HunyuanImage-3: full AR -> KV transfer -> DiT two-stage generation, expert mapping, sleep/wake, second generation.
  • HunyuanVideo-1.5: two 480p/5-frame generations across sleep/wake.
  • Real CUDA CuMem workspace allocation/release and multi-lane isolation smoke: pass.

Exact-stack 0.27.0rc1 vs 0.28.0 A/B on the same H20 node:

  • BAGEL and HunyuanVideo-1.5 remain byte-identical.
  • SD3.5 10-step 512x512x3: not byte-identical when 0.27 uses its required diffusers==0.38.0 and 0.28 uses 0.40.0; MAE 0.4738/255, max channel delta 28, 305115/786432 values changed. The earlier MAE-0 result had accidentally put both arms on diffusers 0.40 and is superseded.
  • Qwen3-Omni: a deprecated raw-string prompt diverged, but the isolation rerun used one production Qwen3OmniThinkerInputAdapter prompt-token hash across 0.27 sync, 0.28 async, and 0.28 with async execution disabled; all three produced identical 64 tokens, decoded text, and finish reason.
  • HI3 outputs are intentionally not parity-comparable: legacy 0.27 prefill_finished transfer stops AR after 2 tokens, while the fixed 0.28 topology decodes the requested 128 tokens before DiT handoff.

Performance follow-up (one warmup + five measured single-request runs):

  • SD3.5 median: 0.4057s -> 0.4071s (+0.35%).
  • BAGEL median: 0.4727s -> 0.4748s (+0.45%).
  • HunyuanVideo-1.5 median: 1.3088s -> 1.3148s (+0.46%).
  • Qwen3-Omni fixed-token median: 0.27 sync 0.4012s; 0.28 async 0.2406s (-40.0%); 0.28 with async_scheduling=false 0.3307s (-17.6%). Within 0.28, async execution is 27.2% faster and does not change output.
  • HI3 raw latency (1.0466s -> 8.7852s) is not a regression metric because output work changes from 2 to 128 AR tokens.
  • Cold-boot numbers are excluded from comparison because checkpoint page-cache/storage order dominates them.

Full public-engine weight/lifecycle E2E:

  • SD3 handle transport: tensor LoRA sync, checksum, generation, sleep, wake with automatic restore, checksum equality, and post-wake generation all pass.
  • Qwen3-Omni TP4 copy transport: initial sync 0.389s, sleep 15.603s, wake plus automatic LoRA restore 0.612s, checksum equality, post-wake generation 0.188s, and full GPU cleanup all pass.

Residual risk

  • Multi-node execution was not exercised.
  • HI3 t2t/i2t LoRA weight sync is newly enabled by the deploy configs but was not exercised on GPU.
  • AR lifecycle failures are now fatal. Before this PR, AR stages slept/woke through the worker RPCs handle_sleep_task/handle_wake_task, whose exceptions StagePool returns as {"supported": False, "error": ...}, leaving the engine alive for VLLMOmniEngine's partial-transition retry. This PR routes AR stages through EngineCore sleep/wake_up, which StagePool treats as control methods and re-raises, so a failure terminates the orchestrator and reaches the caller as a fatal RPC error (not a hang). The retry therefore cannot recover an AR-stage failure and only keeps it loud; diffusion stages still use worker task ACKs, where the retry remains effective. Upstream semantics; no UniRL change planned here.
  • HunyuanImage-3 passed full stable functional coverage; direct 0.27/0.28 output/latency parity is invalid because this PR deliberately fixes legacy prefill-time AR truncation.
  • vLLM-Omni shutdown can leave one EngineCore ignoring SIGTERM and emit a shared-memory tracker warning. UniRL's descendant reaper removed the process and all GPU memory in every observed run, but the upstream shutdown behavior remains visible.

… 5.12.1

vllm-omni 0.28.0rc1 requires transformers >=5.10.1,<5.15, above the repo's
5.6.x baseline. The base pin widens to admit both engine forks rather than
moving to the new floor, because sglang hard-pins 5.6.0 and raising the base
leaves that fork unsolvable; the vllm extra instead narrows itself to an exact
5.12.1. That version is deliberate — sglang 0.5.15+ pins it too, so unifying
the two forks later is a bump rather than a second zoo-wide migration. The
exact pin also matters because the repo carries no uv.lock, so a range here
would follow the next patch release silently. torch is unchanged: vllm 0.28.0
declares the same 2.13.0 as 0.27.0, so the CUDA-13 stack carries over.

Core vLLM 0.28 re-slots the reusable MoE workspace by lane within ubatch
(ubatch_id * num_lanes + lane) rather than by ubatch alone, so the workspace
pool patch now mirrors that arithmetic instead of indexing _current_workspaces
directly. Reading num_lanes off the manager and the lane off the ContextVar
keeps the old flat layout when either is absent. Nothing exercises lanes today
— the diffusion rollout never enables DSpark speculative decoding — but both
failure modes here are silent: a stale index writes None over the wrong slot
and orphans a tensor that still has live views.
@github-actions github-actions Bot added the wip Draft / work in progress label Sep 2, 2026
@CjhHa1
CjhHa1 marked this pull request as ready for review September 14, 2026 09:09
@github-actions github-actions Bot added need review Ready and waiting for review and removed wip Draft / work in progress labels Sep 14, 2026
Copilot AI lite review requested due to automatic review settings September 14, 2026 09:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

@CjhHa1 CjhHa1 changed the title feat(rollout): migrate the vllm-omni engine to 0.28.0rc1 on transformers 5.12.1 feat(rollout): migrate vLLM-Omni to stable 0.28.0 Sep 15, 2026

@Jayce-Ping Jayce-Ping left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most changes are config updates and variable migrations with no blocking risk. GPU test results look good, and there are no conflicts with main. Suggest merging with high priority.

@github-actions github-actions Bot added approved Approved by reviewer and removed need review Ready and waiting for review labels Sep 18, 2026
CjhHa1 and others added 2 commits September 23, 2026 17:15
- Describe the split AR/diffusion sleep-wake path in engine docstrings.
- Drop the stale sglang-unification rationale from the transformers pin.
- Access vLLM 0.28 workspace-lane state directly; negative lanes are
  rejected by use_workspace_lane itself.
- Drop a redundant str() on stage metadata and read stage_configs once.
- Rename stage YAML/config references in examples to deploy config.
- Hoist distributed_executor_backend to the top level in HI3 deploy
  configs; resolved stage configs are unchanged.
@CjhHa1
CjhHa1 requested a review from Ideny42 as a code owner September 24, 2026 01:47
@CjhHa1
CjhHa1 merged commit 53081c3 into Tencent-Hunyuan:main Sep 24, 2026
5 checks passed
@github-actions github-actions Bot removed the approved Approved by reviewer label Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants