feat(inf-train parity): add Qwen3-MoE train-inference parity plugin, part 1/2 — vLLM integration

[cboss6]

Summary

Part 1 of 2: vLLM integration for the Qwen3-MoE train–inference parity plugin.

This two-PR series splits #430 into reviewable components that together implement one end-to-end train–inference parity plugin feature, following the design in #431. Neither part alone provides the complete verified training-and-rollout workflow.

Part	Scope	Branch
Part 1/2 — this PR	vLLM plugin, shared numerical providers, model-specific inference patches, and reload lifecycle	feat/qwen3-moe-parity-vllm-plugin
Part 2/2 — companion PR	Training-side model adapter, core integration, exact parity gate, verification pipeline, and example	feat/qwen3-moe-parity-training

This PR packages the rollout-side component separately. Part 2 integrates it with differentiable actor replay and verifies the complete feature before and after an optimizer update.

Rollout and differentiable replay can produce different policy log-probabilities even for the same weights and sampled tokens. Such numerical differences affect importance ratios, clipping, and KL-related terms. This plugin supplies the inference-side numerical providers needed to align Qwen3-MoE rollout with the companion training-side implementation.

Changes

• Register through vllm.general_plugins, with no changes to the baseline runtime unless explicitly enabled through UNIRL_PARITY_ENABLE=1.
• Validate the selected profile, ordered patch set, pinned package versions, and required private-symbol signatures before installing patches.
• Provide shared reference operators for linear projections, normalization, reductions, precision settings, and attention.
• Add Qwen3-MoE-specific routing, RoPE, output projection, logits, and expert-combination implementations.
• Preserve parity across weight reload by invalidating derived MoE caches and loading the custom column-sharded o_proj without generic meta staging.
• Expose before/after provider evidence through a worker runtime manifest.
• Make successful repeated registration idempotent; reject in-process disabling and retries after a potentially partial installation.

Scope and merge order

All changes are contained in:

experimental/train_inference_parity/vllm_plugin/

This PR does not modify core UniRL, the training algorithm, the weight-sync transport, or training recipes.

Merge order: Part 1/2 → Part 2/2. Part 1 can be reviewed and merged independently, but the complete experimental recipe and end-to-end parity verification require both parts. The training-side adapter, parity gate, verification pipeline, and example belong to Part 2.

Related Issue

• Refs RFC: UniRL Experimental Plugin Framework for Train–Inference Parity #431.
• Part 1/2 supersedes the vLLM plugin portion of feat(experimental): add bitwise Qwen3-MoE train–inference parity case based on GRPO + vLLM(TP>1) + FSDP #430; Part 2/2 supersedes its training/core portion.

This is a split of the existing implementation, not an additional implementation intended to coexist with the original large PR.

Test Plan

No test; Reason: no need.

Plugin and local regression checks

Passed:

• Pre-commit checks over this PR’s complete diff:

  pre-commit run --hook-stage manual \
    --files $(git diff --name-only c8af5d6 HEAD)

• Plugin wheel build and entry-point metadata verification:

  python -m pip wheel --no-deps --no-build-isolation \
    experimental/train_inference_parity/vllm_plugin \
    --wheel-dir /tmp/parity-review-wheels

• Disabled registration under python -S, confirming that it imports neither Torch, vLLM, nor UniRL.
• Actual H20/vLLM registration, installed-provider manifest verification, repeated registration, and rejection of in-process disabling.
• Local checks for configuration rejection, manifest-copy isolation, failed-registration state, and column-sharded weight loading.

One-off verification harnesses and generated artifacts are not committed.

End-to-end integration validation

Passed on September 22, 2026, using Part 1/2 + Part 2/2 together. These results validate the complete train–inference parity feature, not this plugin-only branch in isolation.

The tested integration commit is b9e8ede407d3bb0c0dd47a773c1074527c988e04. The two split branches have disjoint diffs, and their merged source tree matches that tested commit exactly.

Configuration	Value
Model	Qwen3-30B-A3B
Model revision	ad44e777bcd18fa416d9da3bd8f70d33ebb85d39
Hardware / topology	4 × NVIDIA H20; FSDP4 actor + direct vLLM TP4
Software	Python 3.13.11, Torch 2.13.0+cu130, Transformers 5.6.0, vLLM 0.27.0
Precision	BF16 compute; FP32 selected-token log-probabilities
Data	Full GSM8K training JSONL, shuffled
Sampling	Temperature 1, top-p 1, top-k 0
Lengths / batch	256-token prompt limit; 1024 response tokens; 4 prompts × 8 samples
Engine settings	Eager execution; prefix caching and chunked prefill disabled

Command, with the companion training changes and documented environment configured:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m experimental.train_inference_parity.run \
  --config-name=qwen3_moe_30b_a3b_fsdp_tp4

Phase	Replay tokens	Exact FP32 equality	Mismatches	Max absolute difference	K3 mean / max
initial_checkpoint	32768	true	0	0	0 / 0
post_update_reload	32768	true	0	0	0 / 0

Both phases completed one optimizer update with finite gradient norms. The post-update phase confirmed changed actor parameters, receipts from all four TP workers, prefix-cache reset, and publication of model version 1 containing 18,867 tensors.

Final verification status: PASS; process exit code: 0; source tree: clean.

Local artifact:

outputs/train_inference_parity/review_20260922_214630/qwen3-moe-tp4.json

SHA-256:

a4230d10baf5c19d5cf12dc3ca21867b7732256a9be465520a8d93a43f6b7eba

Compatibility / Risk

• Enabled operation requires dedicated processes and the pinned software stack. Changing the enable flag does not uninstall process-global patches.
• UniRL must be installed for the worker capability bridge.
• Validation covers selected-token FP32 forward log-probability equality on the stated configuration—not bitwise gradients, optimizer state, or portability across arbitrary hardware and software.
• EP4/DeepEP, other model geometries, quantization, and vLLM 0.28 are outside this PR’s validated scope.
• The implementation prioritizes correctness over throughput; expert-loop and communication optimizations require separate parity validation.

Reviewer Notes

• Review this as the rollout-provider portion of RFC: UniRL Experimental Plugin Framework for Train–Inference Parity #431, not the complete parity framework.
• Related upstream work was checked during the split. In particular, the vLLM upgrade in feat(rollout): migrate vLLM-Omni to stable 0.28.0 #413 requires renewed compatibility and numerical validation.
• The verification artifact is currently local and should be attached or uploaded to a durable location for independent inspection.

Checklist

• I reviewed the full diff and accept ownership of the changes.
• Package documentation and installation metadata are included.
• Local checks and combined two-phase GPU validation passed.
• No generated artifacts or one-off test harnesses are committed.