Repository navigation
Refactor(moe): run DeepEP M2N through vLLM's MoE runner. - #619
Mohammad-nassar10 wants to merge 6 commits into
Conversation
On CUDA with P2pNcclAFDConnector, the Attention role now builds the MoE layers of the DeepSeek V2 family and Qwen3.5/3.6 through vLLM's own FusedMoEFactory, with an AFD runner (runner_cls) and parameter-free routed experts (routed_experts_cls), instead of the hand-made AFDAttentionFusedMoE stand-in. This is phase 1 of RFC vllm-project#225 for GPU P2P. It ports vllm-project#406 to vLLM v0.32.0 (5df63a72) and to the ffn/ layout of vllm-project#562. ffn/remote.py: - AFDRemoteMoEMethod allocates no expert weights and runs no kernel; AFDRemoteRoutedExperts selects it. - AFDRemoteMoERunner sends the hidden states through the existing RemoteFFNProxy transport (send, DBO yield, receive). AFDExternalRoutingMoERunner also applies the gate it holds, as native MoERunner does, and sends the router logits. Both reject input_ids and shared_experts_input. - build_attention_moe_runner() accepts only CUDA with P2pNcclAFDConnector and takes the model's gate exactly when compute_gate_on_attention is set. It rejects Attention-local EPLB, redundant experts, routed-expert capture (aux_output_config in vLLM 0.32) and a non-finite routed_scaling_factor, and fixes the arguments that describe local experts. - AFDAttentionFusedMoE is deleted. Model adapters: - AFDDeepseekV2RemoteExpertsMoE takes vllm_config, keeps its gate (assigned before the experts, so the mlp.gate.* checkpoint names hold) and passes it to the factory with the native 0.32 arguments. Its forward override is gone; the shell runs vLLM's native forward. - The decoder-layer EPLB check moves into the factory. The DeepEP M2N shell keeps a parameter-free RemoteFFNProxy transport and rejects EPLB itself. - AFDQwen3_5RemoteExpertsMoE matches the native constructor signature and builds the gate-on-FFN runner. Outputs are unchanged: a new CUDA test checks that the Attention MoE sends the payload its gate placement requires and returns the output of the unsplit native MoE bit for bit (rtol=0, atol=0), for both gate placements and for 1 and 7 tokens. Co-authored-by: lirx-pd <616517220@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
| ) | ||
| if shared_experts is not None: | ||
| assert shared_experts_input is not None | ||
| shared_experts( |
There was a problem hiding this comment.
[P1] With --ubatch-size 2 alone, AFD enables two microbatches, but vLLM passes parallel_config.enable_dbo (false here) to SharedExperts. Both microbatches therefore use output slot 0. The first stores its result here and yields on line 272 before the runner consumes it. When the second reaches this call for the same layer, the slot is still occupied and the empty-slot assertion fails. Could we select the shared-expert slot using use_ubatching and cover a two-microbatch forward with this flag combination?
| _current_afd_metadata(), | ||
| x, | ||
| layer_idx=self.layer_idx, | ||
| topk_ids=topk_ids, |
There was a problem hiding this comment.
[P2] Please mask the TP padding rows before dispatch. For example, with Attention TP=2 and three tokens, rank 1 gets one real row and one padding row. The router assigns valid expert IDs to both, so DeepEP sends the padding row to the FFN for expert computation at every MoE layer. The final gather trims that row, which hides the extra work from output-parity tests. The previous path set all top-k IDs for that row to -1. Could we preserve that behavior and assert the IDs received by the connector for this TP=2 case?
|
Could you fill in the Test Result section for the current PR head? Please include the commands you ran, the GPU and vLLM version you used, and which tests passed, failed, or were skipped. I'm particularly interested in the GPU routing parity test and any DeepSeek DeepEP M2N end-to-end runs. If you tested Attention TP=2 with an odd token count or two microbatches with |
Stacked on #593. Only the top commit is new: its diff. I'll rebase onto main once #593 merges.
Purpose
Issue
Scope
Implementation Notes
Test Plan
Test Result
Docs Impact
Essential PR Checklist