Skip to content

Add fixed-order MoE shared/residual merge reference - #404

Open
AsterWang wants to merge 2 commits into
RL-Align:dsv4-p6-devfrom
AsterWang:dsv4-p6-moe-merge
Open

AsterWang wants to merge 2 commits into
RL-Align:dsv4-p6-devfrom
AsterWang:dsv4-p6-moe-merge

Conversation

@AsterWang

@AsterWang AsterWang commented Sep 11, 2026

Copy link
Copy Markdown

Different addition orders and intermediate BF16 rounding can change the merged MoE token output. This adds shared_residual_merge_fwd(routed, shared, residual), an eager PyTorch reference that computes (routed + shared) + residual in FP32 and casts to BF16 once at the end.

The reference uses the existing MoeMergeOp module and semantic registry, exposes FP32 intermediate boundaries, and includes numerical fixtures, source-metadata validation helpers, sealed evidence tests, and owner-local EP replication tests.

Validation

  • 4 × NVIDIA H100 80GB, PyTorch 2.9.1 / CUDA 12.9: python3 -m pytest -q tests/test_moe_merge.py tests/test_kernel_registry.py142 passed, 1 skipped, including 55 CUDA test cases and real NCCL EP=1/2/4 validation.
  • Evidence: 13 artifact seals and the saved intermediate tensor checksums were reloaded and verified.
  • After the public function rename: CPU regression checks passed (84 passed, 59 GPU cases skipped). Compared with the H100-tested sources, changes are limited to the function name and its references; arithmetic, validation logic and fixtures are unchanged.
  • Black, isort, Ruff and diff whitespace checks passed for the relevant files.

The distributed tests use synthetic inputs and test-harness collectives; they do not establish model integration. The reference is forward-only and requires eager execution.

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 8628eb1f-4f66-472e-b1f3-7933f10ff7ca

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@AsterWang
AsterWang marked this pull request as ready for review September 11, 2026 07:16
@Flink-ddd Flink-ddd added DSv4 deepseek-P6 platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations) labels Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek-P6 DSv4 platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants