Skip to content

Refactor(moe): run DeepEP M2N through vLLM's MoE runner. - #619

Draft
Mohammad-nassar10 wants to merge 6 commits into
vllm-project:mainfrom
ms-llmd:refactor/m2n-routed-experts
Draft

Mohammad-nassar10 wants to merge 6 commits into
vllm-project:mainfrom
ms-llmd:refactor/m2n-routed-experts

Conversation

@Mohammad-nassar10

@Mohammad-nassar10 Mohammad-nassar10 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #593. Only the top commit is new: its diff. I'll rebase onto main once #593 merges.

Purpose

Issue

  • Related issue(s): #
  • Closing keyword, only if fully resolved: Closes #

Scope

  • In scope:
  • Out of scope:

Implementation Notes

Test Plan

Test Result

Docs Impact

  • Files updated:
  • If none, reason:

Essential PR Checklist
  • Purpose is clear and linked to public context when possible.
  • Scope is bounded.
  • Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable.
  • No changes are made to the vLLM source checkout.
  • Plugin-owned classes or explicit dotted class paths are preferred over monkey patches.
  • Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested.
  • Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated.
  • Validation evidence is included, including skipped GPU tests when applicable.
  • Documentation impact is stated.

Mohammad-nassar10 and others added 6 commits November 16, 2026 14:41
On CUDA with P2pNcclAFDConnector, the Attention role now builds the MoE
layers of the DeepSeek V2 family and Qwen3.5/3.6 through vLLM's own
FusedMoEFactory, with an AFD runner (runner_cls) and parameter-free
routed experts (routed_experts_cls), instead of the hand-made
AFDAttentionFusedMoE stand-in. This is phase 1 of RFC vllm-project#225 for GPU P2P.
It ports vllm-project#406 to vLLM v0.32.0 (5df63a72) and to the ffn/ layout of vllm-project#562.

ffn/remote.py:
- AFDRemoteMoEMethod allocates no expert weights and runs no kernel;
  AFDRemoteRoutedExperts selects it.
- AFDRemoteMoERunner sends the hidden states through the existing
  RemoteFFNProxy transport (send, DBO yield, receive).
  AFDExternalRoutingMoERunner also applies the gate it holds, as native
  MoERunner does, and sends the router logits. Both reject input_ids
  and shared_experts_input.
- build_attention_moe_runner() accepts only CUDA with
  P2pNcclAFDConnector and takes the model's gate exactly when
  compute_gate_on_attention is set. It rejects Attention-local EPLB,
  redundant experts, routed-expert capture (aux_output_config in vLLM
  0.32) and a non-finite routed_scaling_factor, and fixes the arguments
  that describe local experts.
- AFDAttentionFusedMoE is deleted.

Model adapters:
- AFDDeepseekV2RemoteExpertsMoE takes vllm_config, keeps its gate
  (assigned before the experts, so the mlp.gate.* checkpoint names hold)
  and passes it to the factory with the native 0.32 arguments. Its
  forward override is gone; the shell runs vLLM's native forward.
- The decoder-layer EPLB check moves into the factory. The DeepEP M2N
  shell keeps a parameter-free RemoteFFNProxy transport and rejects
  EPLB itself.
- AFDQwen3_5RemoteExpertsMoE matches the native constructor signature
  and builds the gate-on-FFN runner.

Outputs are unchanged: a new CUDA test checks that the Attention MoE
sends the payload its gate placement requires and returns the output of
the unsplit native MoE bit for bit (rtol=0, atol=0), for both gate
placements and for 1 and 7 tokens.

Co-authored-by: lirx-pd <616517220@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
Signed-off-by: Mohammad <mohammad.nassar@ibm.com>
@Mohammad-nassar10
Mohammad-nassar10 marked this pull request as draft November 19, 2026 23:29
@github-actions github-actions Bot added documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes labels Nov 19, 2026
@Mohammad-nassar10 Mohammad-nassar10 changed the title Refactor/m2n routed experts Refactor(moe): run DeepEP M2N through vLLM's MoE runner. Nov 19, 2026
)
if shared_experts is not None:
assert shared_experts_input is not None
shared_experts(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] With --ubatch-size 2 alone, AFD enables two microbatches, but vLLM passes parallel_config.enable_dbo (false here) to SharedExperts. Both microbatches therefore use output slot 0. The first stores its result here and yields on line 272 before the runner consumes it. When the second reaches this call for the same layer, the slot is still occupied and the empty-slot assertion fails. Could we select the shared-expert slot using use_ubatching and cover a two-microbatch forward with this flag combination?

_current_afd_metadata(),
x,
layer_idx=self.layer_idx,
topk_ids=topk_ids,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Please mask the TP padding rows before dispatch. For example, with Attention TP=2 and three tokens, rank 1 gets one real row and one padding row. The router assigns valid expert IDs to both, so DeepEP sends the padding row to the FFN for expert computation at every MoE layer. The final gather trims that row, which hides the extra work from output-parity tests. The previous path set all top-k IDs for that row to -1. Could we preserve that behavior and assert the IDs received by the connector for this TP=2 case?

@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

Could you fill in the Test Result section for the current PR head? Please include the commands you ran, the GPU and vLLM version you used, and which tests passed, failed, or were skipped.

I'm particularly interested in the GPU routing parity test and any DeepSeek DeepEP M2N end-to-end runs. If you tested Attention TP=2 with an odd token count or two microbatches with --ubatch-size 2 alone, please include those results as well. A link to the relevant CI run or logs would be helpful. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants