Skip to content

feat(npu): enable native DBO for synchronous Attention-side gating - #607

Closed
Pingzii wants to merge 4 commits into
vllm-project:mainfrom
Pingzii:feat/sync-dbo-gate-on-a
Closed

Pingzii wants to merge 4 commits into
vllm-project:mainfrom
Pingzii:feat/sync-dbo-gate-on-a

Conversation

@Pingzii

@Pingzii Pingzii commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Enables native DBO on synchronous CAMP2P when the router and shared experts run on the Attention role.

Scope

  • the gate shell resolves the stage once, before the routing send (forward_context.ubatch_idx when present, otherwise the metadata stage), writes it back, yields between the send and the receive, and receives on the resolved stage so a peer ubatch cannot switch it mid-transfer;
  • the validator accepts DBO/ubatching for this shell with exactly two microbatches and keeps the pre-quantized transport out of DBO;
  • the guide and the model boundary now state one contract: DeepSeek-V4 at TP 1, and with DBO/ubatching Attention DP > 1 plus exactly two microbatches on the BF16 transport.

Verification (CPU)

  • targeted suites: 57 passed; two modules skip because vllm/vllm_ascend are absent. Full tests/unit on this branch: 1093 passed, 89 skipped, 0 failed; ruff, format and mypy clean;
  • the config test accepts DBO with two ubatches for both roles, rejects four, and keeps the quantized-transport rejection;
  • an isolated harness executed the actual forward and test method bodies: four protocol cases passed, and two mutation checks detected re-reading either stage source after the yield.

Measured (8k input / 32 output, 128 concurrent, ACL graph, DBO configured)

Measured on the combined verification build 74d929c (d211266 plus the #444 operator changes), which includes this PR's DBO scheduling changes and the MXFP8 transport implementation. The installed plugin version was 0.32.0rc2.dev46+g74d929c49.

Arm Runs (total token/s) Median E2EL median
BF16 (quant_mode=0) 328,106 / 325,526 / 331,469 328,106 2,761.8 ms
MXFP8 (quant_mode=4) 340,369 / 341,375 / 336,775 340,369 2,643.2 ms

With DBO configured, MXFP8 delivered 3.7% higher median throughput than BF16 (3.4% on the mean) and 4.3% lower median E2EL.

How the numbers were produced

Single host, A5 4A4F. Both roles use the same model and DBO settings below, with separate device assignments and HTTP ports: Attention on devices 0-3 served on port 18000, FFN on devices 4-7 served on port 18010. FFN starts first, then Attention, and the run begins once /v1/models answers.

# Attention role
export ASCEND_RT_VISIBLE_DEVICES="0,1,2,3"
export VLLM_PLUGINS="ascend,afd" VLLM_USE_V1=1 HCCL_BUFFSIZE=2048 \
       PYTORCH_NPU_ALLOC_CONF=expandable_segments:False

vllm serve "$MODEL_PATH" --served-model-name DeepSeek-V4-Flash \
  --host 127.0.0.1 --port 18000 \
  --data-parallel-size 4 --tensor-parallel-size 1 --enable-expert-parallel \
  --max-model-len 9216 --max-num-seqs 128 --max-num-batched-tokens 3072 \
  --block-size 32 --kv-cache-dtype fp8 \
  --attention-config '{"indexer_kv_dtype":"fp8"}' \
  --enable-dbo --dbo-decode-token-threshold 2 --dbo-prefill-token-threshold 12 --ubatch-size 2 \
  --compilation-config '{"cudagraph_capture_sizes":[8,16,32,64,128],"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --additional-config '{"afd":{"role":"attention","connector":"CAMP2pAFDConnector",
      "host":"127.0.0.1","port":6239,"num_attention_ranks":4,"num_ffn_ranks":4,
      "compute_gate_on_attention":true,"connector_extra_config":{"quant_mode":0}}}'

The FFN role is the same with ASCEND_RT_VISIBLE_DEVICES="4,5,6,7", --port 18010 and "role":"ffn". Port 6239 in the AFD config is the AFD communication port, not an HTTP port. quant_mode is 0 for the BF16 row and 4 for the MXFP8 row; nothing else changes between the arms.

The NPU worker selected Model Runner V2.

Build and versions: the plugin ran at 74d929c, which is d211266 plus the #444 A2E operator changes cherry-picked locally, built with SOC_VERSION=ascend950dt_9574 pip install -e . --no-build-isolation; a2e_accepts_mxfp8_arguments() returned True before the run. The environment manifest for these runs pins vLLM 0.30.0 (ced6857af) and vLLM-Ascend 0.19.1rc1-2448-g8d4409d62 (8d4409d).

Workload:

vllm bench serve --backend openai-chat --model DeepSeek-V4-Flash \
  --host 127.0.0.1 --port 18000 --dataset-name random \
  --random-input-len 8192 --random-output-len 32 \
  --num-prompts 128 --max-concurrency 128

Each arm was measured three times; the table reports the medians.

The synchronous gate shell sent and received on the AFD metadata stage and never
yielded, so the validator rejected DBO/ubatching for it even though the connector
already keeps one AFD group per ubatch and the FFN runner already receives on the
stage it is told.

The shell now resolves the stage once, before the routing send, prefers the
forward context's `ubatch_idx` and falls back to the metadata stage (the rule the
shared sync proxy already uses), writes the result back to the metadata, yields
between the send and the receive, and receives on the resolved stage so a peer
ubatch cannot switch it mid-transfer. DBO is accepted for this shell when it has
exactly two microbatches; the pre-quantized A2E transport stays excluded from DBO
until its yield interaction is verified.

Coverage: the config test accepts DBO with two ubatches for both roles, rejects
four ubatches, and keeps the quantized-transport rejection; the forward test runs
with metadata stage 7 against context ubatch 1, switches the context inside the
yield stub, and asserts the send carries stage 1 and the receive still uses it.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes labels Nov 17, 2026
The registered DBO yield dispatches through CUDA, so a CPU tensor is rejected
before the DBO check runs; the two protocol tests reached it once the shell
started yielding. An autouse fixture replaces the helper with a pass-through, and
the stage test overrides it with one that also moves the shared metadata stage.

That override closes the second gap: the forward writes the resolved stage into
`afd_metadata.stage_idx` before the send, so a receive that re-read the metadata
(or the peer context) after the yield still passed. Both sources now change
inside the yield stub, and only a receive that kept the resolved stage passes.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii Pingzii changed the title feat(npu): allow native DBO with the Attention-side gate feat(npu): enable native DBO for synchronous Attention-side gating Nov 18, 2026
The guide's configuration note still rejected DBO with the Attention-side gate, and
the model boundary still required DBO off. Both now state the actual contract:
DeepSeek-V4 at TP 1, and with DBO/ubatching Attention DP > 1 plus exactly two
microbatches on the BF16 transport.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii
Pingzii marked this pull request as ready for review November 18, 2026 06:38
@Pingzii

Pingzii commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #506, which now carries the DBO scheduling change as two commits on top of the MXFP8 transport work: the gate shell resolves the stage before the routing send, yields and receives on the resolved stage, the validator accepts DBO with exactly two microbatches on either transport, and tests cover the config boundary, the stage resolution, two-ubatch routing/activation correspondence and the mode-4 per-stage scale rows. Keeping this thread as the discussion record.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant