Repository navigation
Conversation
The synchronous gate shell sent and received on the AFD metadata stage and never yielded, so the validator rejected DBO/ubatching for it even though the connector already keeps one AFD group per ubatch and the FFN runner already receives on the stage it is told. The shell now resolves the stage once, before the routing send, prefers the forward context's `ubatch_idx` and falls back to the metadata stage (the rule the shared sync proxy already uses), writes the result back to the metadata, yields between the send and the receive, and receives on the resolved stage so a peer ubatch cannot switch it mid-transfer. DBO is accepted for this shell when it has exactly two microbatches; the pre-quantized A2E transport stays excluded from DBO until its yield interaction is verified. Coverage: the config test accepts DBO with two ubatches for both roles, rejects four ubatches, and keeps the quantized-transport rejection; the forward test runs with metadata stage 7 against context ubatch 1, switches the context inside the yield stub, and asserts the send carries stage 1 and the receive still uses it. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The registered DBO yield dispatches through CUDA, so a CPU tensor is rejected before the DBO check runs; the two protocol tests reached it once the shell started yielding. An autouse fixture replaces the helper with a pass-through, and the stage test overrides it with one that also moves the shared metadata stage. That override closes the second gap: the forward writes the resolved stage into `afd_metadata.stage_idx` before the send, so a receive that re-read the metadata (or the peer context) after the yield still passed. Both sources now change inside the yield stub, and only a receive that kept the resolved stage passes. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
4 tasks
Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The guide's configuration note still rejected DBO with the Attention-side gate, and the model boundary still required DBO off. Both now state the actual contract: DeepSeek-V4 at TP 1, and with DBO/ubatching Attention DP > 1 plus exactly two microbatches on the BF16 transport. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Pingzii
marked this pull request as ready for review
November 18, 2026 06:38
Pingzii
requested review from
hsliuustc0106 and
jiangkuaixue123
as code owners
November 18, 2026 06:38
Contributor
Author
|
Superseded by #506, which now carries the DBO scheduling change as two commits on top of the MXFP8 transport work: the gate shell resolves the stage before the routing send, yields and receives on the resolved stage, the validator accepts DBO with exactly two microbatches on either transport, and tests cover the config boundary, the stage resolution, two-ubatch routing/activation correspondence and the mode-4 per-stage scale rows. Keeping this thread as the discussion record. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables native DBO on synchronous CAMP2P when the router and shared experts run on the Attention role.
Scope
forward_context.ubatch_idxwhen present, otherwise the metadata stage), writes it back, yields between the send and the receive, and receives on the resolved stage so a peer ubatch cannot switch it mid-transfer;Verification (CPU)
vllm/vllm_ascendare absent. Fulltests/uniton this branch: 1093 passed, 89 skipped, 0 failed; ruff, format and mypy clean;Measured (8k input / 32 output, 128 concurrent, ACL graph, DBO configured)
Measured on the combined verification build
74d929c(d211266plus the #444 operator changes), which includes this PR's DBO scheduling changes and the MXFP8 transport implementation. The installed plugin version was0.32.0rc2.dev46+g74d929c49.quant_mode=0)quant_mode=4)With DBO configured, MXFP8 delivered 3.7% higher median throughput than BF16 (3.4% on the mean) and 4.3% lower median E2EL.
How the numbers were produced
Single host, A5 4A4F. Both roles use the same model and DBO settings below, with separate device assignments and HTTP ports: Attention on devices 0-3 served on port 18000, FFN on devices 4-7 served on port 18010. FFN starts first, then Attention, and the run begins once
/v1/modelsanswers.The FFN role is the same with
ASCEND_RT_VISIBLE_DEVICES="4,5,6,7",--port 18010and"role":"ffn". Port6239in the AFD config is the AFD communication port, not an HTTP port.quant_modeis0for the BF16 row and4for the MXFP8 row; nothing else changes between the arms.The NPU worker selected Model Runner V2.
Build and versions: the plugin ran at
74d929c, which isd211266plus the #444 A2E operator changes cherry-picked locally, built withSOC_VERSION=ascend950dt_9574 pip install -e . --no-build-isolation;a2e_accepts_mxfp8_arguments()returned True before the run. The environment manifest for these runs pins vLLM0.30.0(ced6857af) and vLLM-Ascend0.19.1rc1-2448-g8d4409d62(8d4409d).Workload:
Each arm was measured three times; the table reports the medians.