Skip to content

feat(npu): support MXFP8 transport and native DBO in sync CAMP2P - #506

Open
Pingzii wants to merge 18 commits into
vllm-project:mainfrom
Pingzii:pr3/mxfp8-passthrough
Open

Pingzii wants to merge 18 commits into
vllm-project:mainfrom
Pingzii:pr3/mxfp8-passthrough

Conversation

@Pingzii

@Pingzii Pingzii commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

## What changed

Adds opt-in MXFP8 activation transport and native DBO support to synchronous CAMP2P with Attention-side routing.

BF16  (quant_mode=0): Attention → BF16 activation       → FFN quantizes locally
MXFP8 (quant_mode=4): Attention → E4M3 activation + E8M0 scale → FFN consumes both

Attention quantizes activations with torch_npu.npu_dynamic_mx_quant inside CAMP2P's opaque send operation, immediately before calling the native a2e operator. FFN consumes the transported activation and scale through ALLGATHER or MC2 without dequantizing and requantizing.

  • Transport and consumption: the MX scale follows the activation through padding, gathering and sharding. AFD-owned prepare and MC2 dispatcher subclasses handle the scale layout and pre-quantized dispatch.
  • Expert type restriction: when a scale is transported, the MoE entry requires an FP8 expert activation (W8A8MXFP or W4A8MXFP) and rejects W4A4MXFP (FP4 activation) and W4A16MXFP (BF16 activation), because prepare skips requantization for a pre-quantized input.
  • Startup checks: both roles validate their transport mode and gate placement through the startup handshake. The Attention token bound is used to check the routing-id budget and supported FFN communication paths.
  • Native DBO: the Attention shell resolves the microbatch stage before sending, runs shared experts, yields, and receives on the saved stage. Each microbatch uses its corresponding CAMP2P communication group. The validator also rejects Attention DP = 1, because the runtime builds no ubatch runner there and the target would silently run without overlap.
  • BF16 compatibility: quant_mode=0 retains the original twelve-argument A2E call, including compatibility with operator builds that predate the MXFP8 arguments.

Code volume

Relative to the merge base (cb2eec8), the branch changes 22 files, +2,571 / -95:

Area Files Added Removed What it holds
afd_plugin/ 8 +862 -50 MXFP8 transport, startup handshake and the DBO scheduling change
tests/ 10 +1,643 -34 Transport, handshake, config and dual-ubatch coverage
docs/ 4 +66 -11 Connector guide, connector contract, support matrix, model boundary

The DBO change is isolated in its own commits; the MXFP8 wire format and its FFN consumer stay together in this PR because either half alone is unusable.

Deployment

  • Use synchronous CAMP2pAFDConnector, DeepSeek-V4, TP 1, and afd.compute_gate_on_attention=true on both roles.
  • MXFP8 requires an A5 device, equal Attention and FFN rank counts between 1 and 32 per side, at least three AIV cores, and the MXFP8-capable A2E operator from [WIP] feat(npu): support MXFP8 datatype for A2E operator #444 with E4M3/E5M2/E8M0 dtype support.
  • The startup probe checks the operator's x_scale and quant_mode input schema. The [WIP] feat(npu): support MXFP8 datatype for A2E operator #444 operator implementation remains a separate change.
  • Upgrade Attention and FFN together and configure matching transport and gate modes on both roles.
  • DBO supports both BF16 and MXFP8 transport, with Attention DP > 1 and exactly two microbatches.
  • All Attention and FFN ranks validate the shared transport and routing configuration through the primary AFD group. Both microbatch groups use this validated configuration.

Build the compatible operator with:

SOC_VERSION=ascend950dt_9574 pip install -e . --no-build-isolation

See the CAMP2P guide for the configuration contract.

Verification

  • CPU suite: 982 passed, 90 skipped, 0 failed at this head.
  • Full-environment tests at 082b2b8: test_sync_attention_gate_proxy.py and test_camp2p_connector.py completed with 21 passed, no skips.
  • Static checks: Ruff check/format and mypy passed on the changed files; GitHub pre-commit, DCO and label checks passed.

Tests cover byte-level scale transforms, runner-to-expert forwarding, MC2 dispatch arguments, startup mismatch rejection, and the two-microbatch contract. DBO cases cover both transport modes and roles, preserve the stage across a yield that changes both stage sources, and verify that mode-4 sends use the correct stage's rows and communication group.

Accuracy (GSM8K, DBO disabled)

Full 1,319-question GSM8K with AIS Bench (gsm8k, gen), Attention-side routing, max-num-batched-tokens 3072, max-num-seqs 128; 2026-11-16/17, builds f862904 (BF16) and 857d8e1 (MXFP8, with the #444 operator). Two to four runs per cell, DBO disabled:

Execution BF16 (%) MXFP8 (%)
Graph 89.84 (2 runs) 90.03 (4 runs)
Eager 90.18 (2 runs) 90.11 (2 runs)

Run start times (host local): BF16 Graph 13:51 / 13:57 (11-16); MXFP8 Graph 16:48 / 16:53 (11-16) and 11:31 / 11:37 (11-17); BF16 eager 11:39 / 11:51 (11-16); MXFP8 eager 11:01 / 11:14 (11-17).

In these follow-up runs, MXFP8 averaged +0.19 pp relative to BF16 in Graph mode and -0.07 pp in eager mode. Thread: PR comment.

Performance

Single-node A5 4A4F, ACL Graph (FULL_DECODE_ONLY), 8,192 input / 32 output tokens, 128 concurrent requests, two runs per arm unless noted. All rows come from one build: 0433169 = 3cbdcb0 plus the #444 A2E operator commit b5c64100. Everything is measured on 2026-11-19 except row 1, which pools the 2026-11-19 run with two 2026-11-20 repeats; only the listed flags differ. Throughput is total token throughput (AIS Bench).

# Routing Transport DBO Start (host local) Runs (total token/s) Mean Mean E2EL
1 FFN BF16 off 11:18 (11-19) / 16:59, 17:05 (11-20) 360,366 / 363,712 / 362,695 362,258 2,454.8 ms
2 FFN BF16 on, 32/512 10:56 / 11:00 359,619 / 350,480 355,050 2,532 ms
3 Attention BF16 off 09:28 / 09:33 349,792 / 351,597 350,695 2,543.5 ms
4 Attention BF16 on, 32/512 10:03 / 10:27 335,261 / 336,217 335,739 2,672.7 ms
5 Attention MXFP8 off 09:12 / 09:20 371,399 / 368,222 369,810 2,406.2 ms
6 Attention MXFP8 on, 32/512 09:43 / 09:54 389,504 / 389,118 389,311 2,279.0 ms

Throughput comparisons using the means above:

Comparison Configuration change Throughput change
6 vs 1 Combined configuration vs the same-build default configuration (FFN-side BF16 routing, DBO off) +7.5%
6 vs 2 Attention-side MXFP8 vs FFN-side BF16, both with DBO on +9.6%
5 vs 3 MXFP8 vs BF16 with Attention-side routing, DBO off +5.5%
6 vs 4 MXFP8 vs BF16 with Attention-side routing, DBO on +16.0%
6 vs 5 Enable DBO with Attention-side routing and MXFP8 transport +5.3%

The combined configuration also reduces mean E2EL by 7.2% relative to the same-build default-configuration baseline (6 vs 1; baseline n=3, measured on 2026-11-19 and 2026-11-20).

With BF16 transport, enabling DBO reduces throughput by 2.0% with FFN-side routing (2 vs 1) and 4.3% with Attention-side routing (4 vs 3). With DBO disabled, Attention-side BF16 routing is 3.2% lower in throughput than FFN-side routing (3 vs 1). FFN-side BF16 routing also sees a 3.1% increase in mean E2EL with DBO enabled (2 vs 1).

DBO thresholds determine which model-execution steps are eligible for splitting, based on the number of tokens in the batch:

Setting Decode-only batch (--dbo-decode-token-threshold) Batch containing prefill (--dbo-prefill-token-threshold)
32/512 (used above) 32 tokens 512 tokens
2/12 2 tokens 12 tokens

Both settings use two microbatches (--ubatch-size 2). The thresholds are batch token counts, not request counts or microbatch sizes. Lowering them to 2/12 makes smaller steps eligible for DBO. A separate MXFP8 run with 2/12 on the same build measured 329,021 total token/s (n=1).

Pre-DBO transport measurements (2026-10, no DBO)

Arm Build Runs (total token/s) Mean
BF16, gate-on-FFN f683eb1 341,995 / 341,185 341,590
BF16, gate-on-Attention 52a292d 336,305 / 334,131 335,218
MXFP8, gate-on-Attention f683eb1 347,730 / 351,374 / 367,132 / 356,085 355,580

These runs predate the DBO work, so no DBO flags were set; they are history only. Against the upstream BASE/main run on bcc0126 (340,844, n=3), MXFP8 measured +4.3% (n=4); against gate-on-Attention BF16 on the earlier commit, the cross-commit gap was +6.1%. The historical DBO-off results show the same overall trend as the current measurements: MXFP8 improves throughput with Attention-side routing, while moving routing to Attention slightly reduces BF16 throughput. The respective changes were +6.1% and -1.9% historically, versus +5.5% and -3.2% in the current build.

Earlier data (2026-11-18)

Build DBO thresholds Transport Start (host local) Runs (total token/s) Mean
74d929c (d211266 + #444 operator) 2/12 BF16 11:55 / 11:59 / 12:11 328,106 / 325,526 / 331,469 328,367
74d929c 2/12 MXFP8 11:33 / 11:38 / 11:46 340,369 / 341,375 / 336,775 339,506

These rows are history only: both the build and the DBO configuration differ from the table above. The DBO configuration has a large effect on the result: with 2/12, the MXFP8 arm measured about 15% lower than with 32/512 on the measured build (329,021 versus 389,311 total token/s). Rows from different DBO configurations are therefore not directly comparable. Within these rows, however, MXFP8 is still 3.4% above BF16 (339,506 versus 328,367 total token/s), so the transport gain holds even under the aggressive configuration.

Reproduction: Attention on devices 0-3 (port 18000), FFN on 4-7 (port 18010), FFN started first. Both roles share:

--data-parallel-size 4 --tensor-parallel-size 1 --enable-expert-parallel
--max-model-len 9216 --max-num-seqs 128 --max-num-batched-tokens 3072
--block-size 32 --kv-cache-dtype fp8
--attention-config '{"indexer_kv_dtype":"fp8"}'
--compilation-config '{"cudagraph_capture_sizes":[8,16,32,64,128],"cudagraph_mode":"FULL_DECODE_ONLY"}'

DBO arms add --enable-dbo --dbo-decode-token-threshold 32 --dbo-prefill-token-threshold 512 --ubatch-size 2; routing moves with the common compute_gate_on_attention field. The numbers come from AIS Bench with the chat model config (DeepSeek-V4-Flash at http://127.0.0.1:18000/v1/chat/completions), synthetic_gen, 8,192 input / 32 output tokens, 128 requests, concurrency 128:

ais_bench --models vllm_api_general_chat.py --datasets synthetic_gen --dump-eval-details --debug -m perf

@jiangkuaixue123 jiangkuaixue123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject unsupported MXFP8 rank counts at connector construction (afd_plugin/connectors/npu/camp2p.py:554). With quant_mode=4, a 4A2F deployment can pass the AIV and schema checks and proceed to the first A2E call. The operator supplied by #444 explicitly requires equal positive Attention and expert rank counts, at most 32 each. The asymmetric padding path at line 1405 cannot satisfy that contract. Please enforce the rank constraint in the mode-4 validator, document it beside the 4A2F example, and cover asymmetric rejection and symmetric acceptance.

The GSM8K scores are from the same 1,319 questions. The two aggregate percentages alone cannot establish accuracy neutrality. Please report the paired counts (BF16-only correct and MXFP8-only correct) and a paired interval for the difference, or describe the observed 1.14-point drop without claiming equivalence. Static review of the incremental change and #444 operator contract. I did not reproduce device measurements.

jiangkuaixue123 pushed a commit that referenced this pull request Nov 10, 2026
## Background

The synchronous CAMP2P path normally runs attention on the Attention
role and the router, shared experts, and routed experts on the FFN role.

This PR makes router/shared placement an explicit deployment option.
With `compute_gate_on_attention=true`, Attention computes top-k and
shared experts, while FFN consumes the external routing and runs the
routed experts.

## Motivation

**Improve eager execution through a different A/F work split.** The
measured Attention-side placement delivers **+17.6% throughput** versus
the same-commit FFN-side placement in the evaluated eager workload.

**Establish the routing contract for quantized activation transport.**
Once FFN receives the routing alongside the activation, it can consume
an activation quantized on Attention. The MXFP8 pass-through builds on
this contract in #506.

The option is **off by default** and belongs in the common
`additional_config["afd"]` block.

## Implementation

```text
Default
  Attention : attention
  FFN       : router + shared experts + routed experts

compute_gate_on_attention=true
  Attention : attention + router + shared experts
  FFN       : transported top-k + routed experts
```

- **Reuse the A2E routing slots.** Send expert IDs as `int32` and top-k
weights as `float32` through the existing ids channel.
- **Owned prepare and comm subclasses.**
`AFDPrepareAndFinalizeWithAllGather` and `AFDPrepareAndFinalizeWithMC2`
provide `prepare_external`, which takes the routing for the local input
rows and returns the complete
`MoEPrepareOutput` together with the aligned routing. AFD comm
subclasses override only
`_get_prepare_finalize()`, so the native selector still decides which
path runs and ordinary execution keeps
  the upstream classes. There are no global prepare patches left.
- **Explicit routing contract.** The routing is an argument and a return
value; nothing is stashed on the
upstream class, and the module no longer reads or clears an
`afd_external_routing` attribute.
- **One MoE component.** `AFDExternalRoutingMoE` builds its comm
candidates during model initialization, where
the config context exists, and each forward only looks up the candidate
matching the native selection. It runs
prepare, expert execution and finalize, switches the forward-context
comm for the duration of the call and
restores it on every exit path, and rejects unsupported types such as
fused-MC2.
- **Consume the external top-k on FFN.** The component hands the aligned
routing straight to expert execution.
- **Place shared experts on Attention.** Run them between activation
send and routed-output receive, then combine.
- **Keep protocol selection explicit.** The model configuration selects
token-ID or external-routing semantics; the
model itself only builds the component for the synchronous FFN role and
delegates.

The supported opt-in configuration is synchronous CAMP2P, DeepSeek-V4,
TP=1, without DBO or ubatching. Both roles must use matching routing
settings.

## Performance and accuracy

Measured at `828582a` on one A5 4A4F node during the same evening
session, with BF16 A2E transport and automatic communication selection:

| Configuration | Eager throughput (token/s) | Graph throughput
(token/s) |
| --- | ---: | ---: |
| Attention-side gate (`828582a`) | **192,887** | **326,400** |
| FFN-side gate (`828582a`) | 163,997 | 331,455.5 |
| Clean baseline (`bcc0126`) | 164,543 | 333,407 |

| Comparison | Eager | Graph |
| --- | ---: | ---: |
| Attention-side vs FFN-side placement | **+17.6%** | **-1.5%** |
| FFN-side placement vs clean baseline | -0.3% | -0.6% |

The result supports Attention-side placement as an eager optimization
and a foundation for MXFP8 pass-through. Placement remains opt-in,
allowing deployments to select the appropriate A/F split for their
execution mode.

Graph-mode Attention-side routing scored **90.60% on the full
1,319-question GSM8K evaluation**, with normal generated output.

Measurement setup: concurrency **128**, reported input **8,275** /
output **32** tokens per request, `max-num-batched-tokens=3072`,
`max-num-seqs=128`, and zero failed requests. Graph uses
`FULL_DECODE_ONLY` with capture sizes `[8,16,32,64,128]`; eager uses
`--enforce-eager`. Each cell has two runs from the same
firmware/session. These are workload-specific measurements with small
run counts.

### Re-measured on the target stack at `e2c7075`

The refactored implementation was re-measured on the stack this PR
declares: vLLM `0.32.0`
@ `5df63a7`, vLLM-Ascend `bd42eea0`, BF16 A2E transport, Attention-side
gate, automatic
communication selection, TP=1, eager (no CUDA/ACL graph), 4A4F on one A5
node.

| Run | E2E (ms) | Prefill (token/s) | Output (token/s) | Total
(token/s) | Failed |
| --- | ---: | ---: | ---: | ---: | ---: |
| 13:28 | 5,126.2 | 1,614.2 | 728.59 | 189,136 | 0 / 128 |
| 13:34 | 5,146.5 | 1,607.9 | 726.69 | 188,643 | 0 / 128 |

The two runs agree within 0.4%, so they are recorded as a repeat
baseline rather than a single
sample. GSM8K on the same configuration scored **89.92%** on the full
1,319-question evaluation.

These numbers describe this configuration only: there is no same-stack
reference implementation to
compare against, so they are not evidence of a performance change from
the refactor. The pre-refactor
table above was measured on the previous stack and stays attributed to
`828582a`.

### Placement re-measured on the target stack (`e2c7075`)

Same node, same stack (vLLM `0.32.0` @ `5df63a7`, vLLM-Ascend
`bd42eea0`), same workload
(128 concurrency, 8,275 input / 32 output tokens per request), zero
failed requests. The clean
baseline is upstream main at `b093863`, which keeps the gate on the FFN
side.

| Mode | Clean baseline `b093863` (total token/s) | Attention-side gate
`e2c7075` (total token/s) | Delta |
| --- | ---: | ---: | ---: |
| Eager | 165,345 | 189,136 / 188,643 | **+14.2%** (mean) |
| Graph (`FULL_DECODE_ONLY`) | 364,627 / 362,563 | 354,731 / 353,972 |
**-2.5%** (mean) |

On this workload, Attention-side placement averages about **+14.2%
higher total throughput in eager**
and about **2.5% lower in graph**. That is a placement comparison across
two commits on one stack; it
is not a pure refactor cost, and it does not mean Attention-side
placement is unsuitable for graph
mode.

Graph mode also raises this configuration's own throughput by **87.6%**
over eager
(189,136 -> 354,731 total token/s). GSM8K on the target stack scored
**89.92%** in eager and
**89.99%** in graph mode, both on the full 1,319-question evaluation.

## Baseline and scope

Rebased onto main at `e58fdec`, which includes the merged #500 opaque
routing-op fix. The retired
`profile_run_eager` implementation and its wiring have been removed.
#506 builds MXFP8 pass-through
on this routing contract.

[Incremental diff for this PR:
`e58fdec...2c17cce`](Pingzii/afd-plugin@e58fdec...2c17cce)

## Verification

CPU, measured at `2c17cce`: **59 passed, 2 skipped**. The focused set
covers the routing-id op, the runtime installer
wiring, the CAMP2P token-ID path, the routing payload transforms, and
the new MoE component. The maintained
CPU suite also passes.

Two mutation checks were run on the refactored component before the
rebase, and both produced the expected failures:

- disabling the per-token state cleanup makes the MC2 finalize case fail
(it observes state that should already
  be cleared);
- disabling the comm restore makes the prepare-failure case fail (it
observes the AFD comm left in place).

Device evidence: the head `7430a13` starts on one A5 4A4F node with BF16
transport,
`compute_gate_on_attention=true`, the CAMP2P connector, TP=1 and eager
mode, and completes a generation
request (`POST /v1/completions` returned 200). Accuracy and throughput
were measured at `e2c7075`.
The pre-refactor table above stays attributed to `828582a`.

---------

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Co-authored-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch from 4d2d890 to 86495ce Compare November 11, 2026 01:56
@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes labels Nov 11, 2026
@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch 3 times, most recently from c0c42e3 to 5041bd5 Compare November 11, 2026 02:14
@Pingzii

Pingzii commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Ready for re-review at 5041bd5. All three points from the last review are addressed; dev re-ran the probe, CAMP2P transport, payload and FFN wiring cases (84 passing).

  1. Runner-level wiring coverage. tests/unit/model_executor/models/test_deepseek_v4_external_routing_moe_cpu.py now starts from the real _ffn_forward in v1/worker/npu/ffn_model_runner.py — the method body runs from the source file against stand-ins for the helpers it calls — with a payload handed over by a connector stand-in. From there the chain is real: compute_ffn_output → AFDExternalRoutingMoE → the prepare row transforms → the expert entry point, asserted as (4, 1, 2) on ALLGATHER and (4, 2) on MC2 with distinct per-row E8M0 bytes. Mutation check: deleting ffn_kwargs["activation_scale"] in the runner fails both cases on the pair contract.

  2. Mode 4 send, receive and construction. tests/unit/connectors/test_camp2p_token_ids.py gains 11 cases: the fourteen-argument send carries the quantized activation together with its scale, the receive reads the sixth value, a build that returns five values now raises RuntimeError: ... returned 5 instead of IndexError (_A2E_MX_RETURN_VALUES guards the read of outputs[5], while outputs[3]/outputs[4] stay readable on a pre-MXFP8 build), the token-ids slot is refused, and construction accepts 4A4F while rejecting 4A2F, an FFN-side gate, more than 32 ranks per side, and a probe that answers False or cannot answer.

  3. Accuracy wording. The body now reports GSM8K 89.76% for quant_mode=4 against 90.90% for quant_mode=0, −1.14 pp, and states that equivalence is not claimed: it would need per-question paired evidence, which has not been run. The device section also separates the schema gate (startup) from the substantive check (one real generation proving the sixth value reaches FFN), so a successful construction is not presented as mode-4 validation.

CPU verification on this head: full tests/unit 863 passed / 82 skipped / 8 failed, with the same eight environmental failures on the base commit; mypy-3.11, ruff check, ruff format and typos clean. There is still no device run: the operator that accepts x_scale and quant_mode is #444.

@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch 4 times, most recently from ad9044a to 44394d1 Compare November 11, 2026 06:23
@Pingzii

Pingzii commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Two follow-ups are in at ee59d9d; all three checks are green on that head.

The handshake has a test that fails if it is deleted. Rather than adapting the torch_npu-gated init test -- which is skipped in exactly the environments where the check would otherwise run -- I put a locally runnable one next to the other CAMP2P tests: it drives the real init_afd_connector against recorded stub groups and mocked torch/all_gather_into_tensor, and asserts that the collective runs once, on the AFD main group, before afd_moe and p2p are created. Removing the handshake call no longer publishes the handshake event, so the test fails on the order contract rather than on an incidental attribute; the mixed set then fails with the mismatch error on the same path. A follow-up will run the two mixed cases through the real init once per role, so Attention and FFN each prove they refuse before afd_moe and p2p exist.

The guide states the startup behavior. The two bullets that had been joined are split, the quant_mode bullet describes the AFD-group collection, and the section says the handshake runs for quant_mode=0 too: a role running a version without the handshake never joins the collective, so the newer side fails at startup with a communication timeout even when both are configured for BF16. Upgrading the Attention and FFN roles together is what avoids that.

Two carry-overs from the earlier, now superseded check. The per-payload mode comparison is still in the code and will go away in the follow-up that makes the startup handshake the single protocol check. What stays regardless: construction requires an Ascend A5 device (_require_a5_device, because the operator registers the same fourteen-argument schema on every build and rejects mode 4 only at runtime), and the guide states the A5 plus E4M3/E5M2/E8M0 build requirements. The dtype half cannot be decided from Python without a marker from the operator; that request is #444 (comment), and until it exists this stays a documented requirement rather than a check.

Testing is down to the seams that own each behaviour. Five cases are gone at ee59d9d: the payload padding case (the ALLGATHER component case walks the same padded() and checks the original rows and the appended 127 bytes), the zero-block scale case (the connector already validates the column count before a routing payload exists), the MoE row-count row (the payload-level check rejects it first), and the MC2 scale-dtype/activation-dtype rows (the normal A2E to MoE path validates both before a dispatcher is reached). The gather case with distinct per-rank bytes stays, since that is what proves the order, and so does no-scale, which catches a runner that stops forwarding.

The convergence follow-up will also keep the JSON round-trip test (dropping only its quant_mode assertion), delete the field plus its standalone missing-field case, and remove the per-packet comparison sentence from the guide.

CPU verification at this head: connectors and model_executor 323 passed / 28 skipped; ruff check/format, typos and the changed-file mypy clean. Still unrun: the real HCCL startup handshake, mode 4 end-to-end on ALLGATHER and MC2, and the operator capability marker.

@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch 7 times, most recently from 8b87fe4 to a33f413 Compare November 11, 2026 07:26
jiangkuaixue123 pushed a commit that referenced this pull request Nov 13, 2026
Three unit tests fail on Windows for reasons unrelated to what they
check:

- `tests/unit/compat/npu/test_ascend_config_lifecycle.py` reads a source
file with `Path.read_text()` and no encoding, so the GBK locale decodes
UTF-8 bytes and raises `UnicodeDecodeError`; both reads now state UTF-8.
- `tests/unit/package/test_cam_async_build.py` builds its manifest file
list with native separators while setuptools works on posix manifest
paths, so every required entry misses; the list now uses `as_posix()`.
- `tests/unit/package/test_package.py` needs installation metadata to
find the `vllm.general_plugins` entry point, which a source tree does
not have; it now skips with that reason, and still checks the entry
point when the distribution is installed.

No production code changes. With these, full `tests/unit` on a Windows
CPU host with `torch` and no `vLLM` is 900 passed / 89 skipped / 0
failed (previously 8 failed).
Where these come from, for review context:

- the six GBK failures:
`tests/unit/compat/npu/test_ascend_config_lifecycle.py` arrived in #425
(`b7561d7`) with `ast.parse(path.read_text())` and no encoding, so it is
latent everywhere except a locale that cannot decode UTF-8;
- the manifest failure: #539 (`f2c79d8`) replaced the posix-normalised
`required` set with `set(files.allfiles)`, whose entries carry native
separators, so the comparison only holds where `/` is the separator. The
#361 version of the test compared normalised paths and was correct;
- the entry-point failure: the test first appeared as
`tests/test_package.py` in #1 (`dffdb14`, "Add Phase 1 plugin skeleton")
and was later moved to `tests/unit/package/test_package.py` by
`d1fd84f`; it always needed installation metadata.

None of the three involves the transports in #506; the tests are older
than that branch.
On verification scope: this PR's checks are label, pre-commit and DCO --
none of them installs the plugin or runs pytest. The numbers above come
from a local run on this branch; the entry-point test exercising an
installed build is therefore *not* covered by CI, before or after this
change.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch from a7ab213 to 0f059d9 Compare November 13, 2026 02:43
@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch from 0f059d9 to aaa4dc1 Compare November 14, 2026 16:51

Copy link
Copy Markdown
Collaborator

Could you check whether the GSM8K accuracy drop from 90.90% to 89.76% is expected with MXFP8 transport? It would help to confirm whether this comes from quantization or run-to-run variance, and whether it falls within the acceptable accuracy tolerance.

The sync transport can only carry BF16 activations today. This adds the payload
half of an opt-in MXFP8 mode: the routing carries the activation MX scale, one
E8M0 exponent per 32 values and one row per token.

The scale follows the same row transforms as the routing -- pad, gather and
split -- because a row whose scale stays behind describes another token.
Padding writes the byte 127 (2**0, the neutral exponent) through the byte view,
because E8M0 stores an exponent and the collectives are not required to accept
the float8 dtype. That view is what the payload fakes reject, so a transform
that hands a collective the float tensor instead of its bytes fails the tests
rather than passing silently.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
An operator build that predates the MXFP8 arguments would either reject the
fourteen-argument call or accept it and ignore the scale. The probe answers
three ways -- the schema takes x_scale and quant_mode, it does not, or it cannot
be read -- so the caller decides what an unreadable build means instead of
treating it as support.

The check is structural, not textual. str(torch.Type) renders Optional[Tensor]
in a form that does not parse back, and a bare Tensor[] would satisfy a name
test while the operator expects the optional wrapper. Inspecting the types keeps
both faults visible.

The probe tests build their schemas with torch._C.parse_schema, so they exercise
the same rendering the production path reads.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
connector_extra_config.quant_mode=4 quantizes the Attention activation to E4M3
inside the opaque A2E call, ships it with its E8M0 MX scale, and hands the pair
to the experts on the FFN side without dequantizing. Mode 0 keeps the
twelve-argument BF16 call, so an operator build that predates the MXFP8
arguments still runs.

Construction refuses mode 4 unless the deployment can carry it: the common gate
must be on Attention, both sides must have the same rank count between 1 and 32,
three AIV cores are needed, and the operator probe must have said yes. The FFN
receive refuses the token-ids slot in this mode, because the router and the
shared experts need the model dtype. Quantizing the final wire tensor keeps the
scale covering exactly the rows the operator sends; mode 4 cannot reach the
padding branch because it requires equal rank counts.

The scale travels with the routing, so prepare pads, gathers and shards it by the
same rule as the activation rows. What consumes it fixes the layout: ALLGATHER
re-lays it as (rows, cols // 2, 2) for the fused GMM, while MC2 hands the
two-dimensional scale to its dispatch operator. MC2 gets a dispatcher subclass
that forwards the scale and selects quant_mode=0 instead of letting upstream
quantize an activation that already is; an explicit comm_quant_mode that
disagrees is rejected rather than overruled.

The tests drive the real prepare transforms and the real FFN entry point, with
only the connector payload supplied, and pin the bytes the experts read: every
scale row carries different values, so a transform that reorders, pads or
re-lays it cannot keep them in place.

This branch has no NPU run. The operator that accepts x_scale and quant_mode
lives in vllm-project#444, so the mode is wired and reviewed but unverified on hardware.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
quant_mode now has two legal values, so the connector tables and the support
matrix state what mode 4 carries and which deployment shapes it requires. The
payload contract gains activation_scale, including the row-per-token and
ceil(hidden / 32) shape the transport validates.

The support matrix records that mode 4 has no device run in this repository yet,
rather than implying the mode is verified because it is wired.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Both roles read their own `connector_extra_config.quant_mode`, and each value is
individually legal, so an Attention role on 4 with an FFN role on 0 passed both
local checks while the two A2E packet layouts differ in the activation offset and
the element width. The FFN rank would read the peer's bytes as its own mode and
hand the experts a silently wrong activation.

Two checks, in the order they bite:

- `init_afd_connector` gathers every rank's mode over the AFD group and fails
  them all when the set is not a singleton. Every rank joins that group, so the
  4/0 and 0/4 pairs both fail here -- before profile, capture and the first A2E
  call -- rather than leaving the Attention side waiting for an E2A the FFN side
  will never send.
- The control payload also carries the sender's mode, and the FFN role compares
  it in `recv_dp_metadata_list`. That path covers a peer that started before the
  group handshake existed; a payload without the field reads as 0, which is the
  right answer only for a peer that never had mode 4. A `5041bd5` peer does have
  it, so the handshake above is the one that keeps that combination from being
  silently accepted.

`quant_mode=4` also needed a way to see a device whose operator build lacks the
path: the operator registers the same fourteen-argument schema on every build and
rejects mode 4 only at runtime, so the construction gate now checks for an Ascend
A5 device. The dtype half of that build check needs a capability marker from the
operator side (vllm-project#444); until it exists this stays a documented requirement rather
than a check.

Tests: ranks must agree on the mode (4/0 and 0/4 both raise), the mixed pair
fails at the handshake, a matching pair passes, the send side declares its own
mode, the codec round-trips the field and defaults a missing one to 0, and the A5
check rejects another device family.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Three cases restated behaviour another seam already pins: the two
`compute_ffn_output` cases re-bound the same extracted methods that the runner
case drives through the real `_ffn_forward`, and the component-level row-order
case asserted the same bytes and layout as that runner case. They are gone.

Three pairs with identical setup became parametrize tables, keeping every
scenario: the pair contract now covers E4M3 without a scale, a scale next to
BF16, E5M2, and a scale with the wrong row count in one table; MC2 dispatch
covers an unset and an explicit pre-quantized comm mode; the control-payload
check covers 4/0, 0/4 and 4/4.

The BF16 no-scale assertion moved onto the vllm-project#503 routing case, which now states
`torch.bfloat16` explicitly instead of relying on the float32 default, and the
startup handshake case now also pins that `p2p` is built after the collective.

Mutation re-checked after the change: deleting the runner's
`activation_scale` forwarding still fails both the ALLGATHER and the MC2
implementation of `test_the_ffn_runner_carries_the_received_scale_to_the_experts`.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Round two of the same review: keep the seam that owns each behaviour and drop
the rows that only restate it.

- Schema probe: the float and int `x_scale` cases both hit the same
  OptionalType rejection, so they are gone. `Tensor[]` stays -- it is the one
  that catches an element-type-only check -- and it now parses directly, so the
  hand-built ListType scaffolding is gone with it.
- The FFN runner case keeps MC2 only: the DP-gather case above it already pins
  the ALLGATHER three-dimensional layout, its row order and the padding byte.
- The MC2 rejection table keeps `scale-dim`, `not-mxfp` and the
  `comm_quant_mode` conflict. The two dtype rows restated what the normal A2E to
  MoE path validates before the dispatcher is reached.
- The control payload's `quant_mode` assertion folds into the existing JSON
  round-trip; the missing-field-reads-as-0 case stays on its own.
- The startup check now runs through the real `init_afd_connector` for all four
  outcomes (0/0, 4/4, 4/0, 0/4): the fake collective asserts the mode that was
  actually published, and the refused cases assert `afd_moe` and `p2p` are never
  built. The direct helper call went away, so the test fails if the entry point
  stops calling the check.
- Sending is one test now: the public `send_attn_output()` drives the registered
  op, and the operator boundary asserts one fourteen-argument call, the
  quantized activation, and the scale with its mode.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
- The payload padding case restates `padded()`: the ALLGATHER component case walks
  the same call and checks the original rows and the appended 127 bytes. The
  gather case with distinct per-rank bytes stays, since it proves the order.
- The zero-block scale case can only be reached by hand: the connector already
  validates the column count as `ceil(hidden / 32)` before a routing payload
  exists.
- In the MoE pair table, `row-count` is rejected by
  `AFDExternalRoutingInputs` even if the component's own check is bypassed; the
  payload-level row test stays, and so does `no-scale`, which is what catches a
  runner that stops forwarding the scale.
- In the MC2 dispatch table, `scale-dtype` and `activation-dtype` restate what the
  normal A2E to MoE path validates before a dispatcher is reached. `scale-dim`,
  `not-mxfp` and the `comm_quant_mode` conflict stay.

No production change in this commit; the Gloo payload field is untouched.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The operator moves a call routing ids as one `[rows, topk]` int32 block into a
single UB region, so `batch_size x topk x 4` has to fit 195,584 bytes. Nothing in
the connector checked it, so `--max-num-batched-tokens 8192` with `topk=6`
started cleanly and the real operator would then reject the full batch.

Construction now computes the largest batch the mode can send (`8,149` tokens at
`topk=6`), refuses a configuration above it, and the guide states the bound next
to the other mode-4 requirements. The tests cover the boundary (8,149 accepted,
8,150 refused), a configured maximum that exceeds it, and a smaller `topk` that
fits under the same batch.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
A mode-4 pair can pass every construction check and still have the FFN resolve
to a MoE path the transport cannot feed: the pinned selector picks ALLTOALL or
FUSED_MC2 depending on batch size, EP, LoRA and MegaMoE state, and the
external-routing consumer supports only ALLGATHER and MC2. The control plane is
one-way, so refusing after the first control payload would leave the Attention
side inside an A2E waiting for an E2A that never comes.

The startup handshake grows a second step. The first collective now exchanges
`[quant_mode, attention batch bound]` per rank -- only the Attention role
publishes a bound, and the aggregate stays FFN first, so the FFN reads its
candidate from the Attention rows alone. A mixed set fails there, both roles at
once. An all-BF16 set continues without the second collective, so the ordinary
deployment keeps exactly one startup collective. An all-mode-4 set asks the
pinned selector, with its import, capacity lookup and call all inside the guard:
a selector that raises still publishes its verdict, because a rank that exits
early leaves the peer waiting. The verdict travels on the second collective and
every rank acts on it -- the FFN re-raises with the original exception as its
cause, the Attention side reports that the peer refused.

Tests drive the real `init_afd_connector` for both roles with 1A1F and record
each collective separately (`mode_gather`, `verdict_gather`), so the stop point
is proved rather than inferred: mixed pairs stop at the first, an unsupported
path or a raising selector stop at the second, and a mode-0 pair never reaches
the second. The FFN publishes no bound, the selector is asked about Attention
1024 rather than the FFN own 512, and the failing role keeps its original cause.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Two ways a configured pair could still reach a working transport and then read
or route wrongly:

- The roles read `afd.compute_gate_on_attention` independently. With the gate on
  the A2E ids slot carries routing ids, with it off the same slot carries token
  ids, and both configurations pass construction, so a mixed pair exchanges
  payloads successfully: the FFN reads expert ids as token ids in one direction
  and token ids as routing ids (with zero weights) in the other. The startup
  handshake now carries the flag and fails both roles when they disagree, before
  any A2E call.
- The selector pre-check only ran for mode 4. BF16 routing has the same
  ALLGATHER/MC2 contract, so a pair such as 8A8F with a bound beyond the MC2
  capacity started, served small batches, and only failed at prefill when the
  pinned selector chose ALLTOALL. The verdict now runs whenever routing is
  carried -- mode 4 always, mode 0 when the gate is on -- so the pair is refused
  at startup. A gate-off BF16 pair still exchanges exactly one collective.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The regression worth pinning is not that a mismatch is detected but that neither
side continues: if only the FFN rejects the gate disagreement, the Attention role
walks on to the verdict collective and waits for a peer that has already left.

The case now runs once per role and asserts the same stop point on both -- the
group exists, the first gather happened, nothing else was created -- so a
one-sided rejection fails the test instead of passing it.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii
Pingzii force-pushed the pr3/mxfp8-passthrough branch from 0546493 to f862904 Compare November 16, 2026 01:19
@Pingzii

Pingzii commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

We followed up on the earlier 1.14 pp drop with repeated runs per mode on the full 1,319-question GSM8K set, evaluated with AIS Bench (gsm8k, gen, no limit). All runs used Attention-side routing on a single A5 host with a 4A4F configuration, --max-num-batched-tokens 3072 and --max-num-seqs 128. Graph runs used --compilation-config '{"cudagraph_capture_sizes":[8,16,32,64,128],"cudagraph_mode":"FULL_DECODE_ONLY"}'; eager runs used --enforce-eager.

A2E mode / execution Runs (start time, host local) Mean
BF16, Graph (quant_mode=0, f862904, without #444) Nov 16 13:51 -> 89.46%, 13:57 -> 90.22% 89.84%
MXFP8, Graph (quant_mode=4, 857d8e1, with #444) Nov 16 16:48 -> 89.84%, 16:53 -> 90.52%; Nov 17 11:31 -> 89.84%, 11:37 -> 89.92% 90.03%
BF16, eager (quant_mode=0, f862904, without #444) Nov 16 11:39 -> 90.14%, 11:51 -> 90.22% 90.18%
MXFP8, eager (quant_mode=4, 857d8e1, with #444) Nov 17 11:01 -> 90.30%, 11:14 -> 89.92% 90.11%

The earlier drop did not recur in this follow-up. In Graph the MXFP8 mean is 0.19 pp higher; in eager it is 0.07 pp lower -- both inside the spread of their own runs, and the two directions disagree. The two BF16 Graph runs differed by 0.76 pp, the four MXFP8 Graph runs spanned 0.68 pp, the two BF16 eager runs by 0.08 pp, and the two MXFP8 eager runs by 0.38 pp. The modes ran in separate time blocks and used different A2E builds, so this comparison leaves the cause of the earlier drop and the acceptable tolerance open. The remaining check is to alternate modes on the same #444 build -- the BF16 runs above predate it -- keep the evaluation settings fixed, and retain per-question results.

For context, the measurement that raised the question: 2026-10-30/31, same tree (4d2d890 plus the #444 build), Graph, Attention-side routing, GSM8K 1,319 -- MXFP8 89.76% against BF16 90.90%, one run per arm on consecutive days, i.e. -1.14 pp. That single pair is what this follow-up retests.

The comparison for adopting this mode is separate: clean upstream/main BF16 remains the deployment baseline, and it is stated in the PR description rather than here.

The synchronous gate shell sent and received on the AFD metadata stage and never
yielded, so DBO/ubatching was rejected for it even though the connector already
keeps one AFD group per ubatch and the FFN runner receives on the stage it is told.

The shell now resolves the stage once, before the routing send, prefers the
forward context's ubatch_idx and falls back to the metadata stage, writes the
result back to the metadata, yields between the send and the receive, and
receives on the resolved stage so a peer ubatch cannot switch it mid-transfer.
The validator accepts DBO/ubatching for this shell with exactly two microbatches
on either transport; the guide and the model boundary state the same contract.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The sync gate tests now stub the DBO custom op, resolve the stage against a
forward context whose ubatch_idx differs from the metadata stage, and move both
stage sources inside the yield so a receive that re-reads either fails. A second
case runs two ubatches through the shell and checks that each send carries its own
routing and activation and that each receive stays on its own stage. The connector
case drives mode 4 for two stages and checks that the E4M3 activation and its E8M0
scale cover exactly the rows of the stage being sent.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Merge the nested DBO timing check into one condition for pre-commit, parametrize
the acceptance test over both transports and both roles, and let the mode-4 send
case assert the per-stage group name, an E8M0 scale and legal routing inputs. The
guide, the support matrix and the connector contract drop the remaining statements
that the Attention-side gate cannot run under DBO.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
quant_mode=4 always transports an E4M3 activation with its E8M0 scale, but the
MoE entry only checked that the pair matched. Prepare skips requantization for a
pre-quantized input, so an FP4 or BF16 expert type consumed the E4M3 bytes as its
own activation format and reached the fused GMM mismatched -- the MC2 path in
particular accepts W4A4MXFP through its is_mxfp check.

The entry now requires an expert type whose activation is FP8 (W8A8MXFP, W4A8MXFP)
whenever a scale is present, and rejects the rest with the reason. Tests cover the
two rejected types and an accepted W4A8MXFP case; the stub expert type defaults to
W8A8MXFP so the existing pair cases keep their intent.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>

@jiangkuaixue123 jiangkuaixue123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted by an agent on behalf of @jiangkuaixue123.

Thanks for putting the transport and DSV4 integration together. I think the CAMP2P changes are worth separating conceptually, but the MXFP8 wire format and its FFN consumer must remain one compatible feature: connector.py sends E4M3 plus E8M0 scale, while the DSV4 MoE path forwards that pair into ALLGATHER/MC2. Landing only one end would leave the mode unusable or misinterpreted.

Suggested split:

  1. Keep the already self-contained native DBO shell/validation work (deepseek_v4.py, feature_validation.py, about 25 changed lines) as its own change. It is already isolated in commits 81f1c28 and e038326; this also avoids mixing scheduler behavior with transport quantization.
  2. Keep the MXFP8 end-to-end contract together for correctness: A2E quantization, schema/device/shape checks and startup handshake (connector.py, compat/npu/ops.py, about 570 added lines); scale in external-routing metadata and row alignment (external_routing.py, metadata.py, about 90); and the DSV4 consumer/model seam (deepseek_v4_external_routing_moe.py, deepseek_v4.py, FFN runner, about 190). Tests and docs are another roughly 1,300 changed lines. These are reasonable review-sized commits within one feature PR, but extracting a “CAMP2P-only” PR from the FFN changes would break the contract.
  3. For future reuse, the A2E MX quantization/pair validation currently contributes roughly 100 lines in connector.py:1475-1548; consider moving that transport-specific code to a small CAMP2P transport module after this feature lands. Keep connector lifecycle and handshake ownership in the connector. This narrows future wire-format work without introducing a cross-PR protocol dependency.

Two concrete follow-ups:

  • The docs require Attention DP > 1 for DBO/ubatching (docs/npu/CAM_P2P_CONNECTOR_USER_GUIDE.md:137-138), but the validator accepts DP=1 whenever num_ubatches == 2. At DP=1 the runtime does not construct the ubatch runner, so this configuration can pass validation and silently run without DBO. Please reject data_parallel_size <= 1 here or make the documented/runtime behavior explicit. This is also the remaining finding from the now-closed #607.
  • Please add a same-configuration DBO-off vs DBO-on performance comparison (same model, rank placement, input mix, graph mode and repetitions). The current reported MXFP8 vs BF16 result with DBO enabled does not isolate DBO's effect.

Other code-grounded note: the connector currently owns transport-mode setup, startup capability agreement, A2E calls, and pair validation in one file (connector.py:654-800, :1100-1200, :1475-1548). The module extraction above would make these responsibilities easier to review while retaining the startup agreement where it can fail both roles before data transfer.

f"TP={int(parallel.tensor_parallel_size)}",
)
if bool(parallel.enable_dbo) or bool(parallel.use_ubatching):
if (bool(parallel.enable_dbo) or bool(parallel.use_ubatching)) and int(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs require Attention DP > 1 for DBO/ubatching (docs/npu/CAM_P2P_CONNECTOR_USER_GUIDE.md:137-138), but this new validation only checks that num_ubatches == 2. At DP=1 the runtime does not construct the ubatch runner, so this configuration passes validation and silently runs without DBO. Could this also reject data_parallel_size <= 1 (or could the docs describe the actual behavior)? This is the remaining finding from the now-closed #607.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at 3cbdcb0. When DBO/ubatching is requested, the gate validator now also rejects Attention DP = 1:

if getattr(afd_config, "role", None) == "attention" and int(
    parallel.data_parallel_size
) <= 1:
    raise RuntimeError(
        "Attention-side gate with the synchronous connector requires "
        "Attention DP > 1 for DBO/ubatching; at DP=1 the runtime does not "
        "build the ubatch runner and DBO would not run. Got "
        f"DP={int(parallel.data_parallel_size)}",
    )

So a DP=1 deployment now fails at validation instead of running without overlap, which matches what the guide already stated. Coverage: test_dbo_is_rejected_without_attention_data_parallelism asserts the rejection on the Attention role, and the accepted two-ubatch case still runs at DP=4. CI (DCO, label, pre-commit) is green on 3cbdcb0.

At DP=1 the runtime returns no ubatch runner, so the configuration passed
validation and then ran without overlap. The gate validator now rejects
Attention DP <= 1 when DBO/ubatching is requested, which is what the guide
already stated. Adds the rejection test and keeps the accepted two-ubatch case
at DP=4.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii

Pingzii commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the two concrete follow-ups from the latest review.

DP=1 validation gap (inline finding). Fixed in 3cbdcb0: when DBO/ubatching is requested, the gate validator now also rejects Attention DP = 1, because the runtime builds no ubatch runner there and the target would silently run without overlap. This matches what the guide already stated. New rejection test, and the accepted two-ubatch case stays at DP=4; the config file is 14 passed, full tests/unit 982 passed / 90 skipped.

DBO off vs on comparison. Not measured yet, so the Performance table stays an MXFP8-vs-BF16 result with DBO configured rather than a DBO claim. It needs a device run on the same build (same model, 4A4F placement, input mix, graph mode and repetitions) with the four DBO flags removed for the off arm; I will add it as measured data when it exists rather than extrapolate.

Split guidance. Taken: the native DBO shell/validation is already isolated in 81f1c28 and e038326, while the MXFP8 wire format and its FFN consumer stay in one PR because either half alone is unusable. The suggestion to extract the A2E MX quantization/pair validation into a transport module is noted as a follow-up after this lands, not a change in this PR.

@Pingzii

Pingzii commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Updated measurements addressing the DBO off/on comparison are below.

DP=1 validation gap. Fixed at 3cbdcb0: with DBO/ubatching requested, the gate validator rejects Attention DP = 1, because the runtime builds no ubatch runner there. A rejection test was added; the accepted two-ubatch case stays at DP=4. CI is green on that head.

Same-configuration DBO off vs on. One build (0433169 = 3cbdcb0 plus the #444 operator commit b5c64100), single A5 host, 4A4F, ACL graph FULL_DECODE_ONLY, 8,192 input / 32 output tokens, 128 concurrent requests. Everything is measured on 2026-11-19 with two runs per arm, except the baseline row, which pools that run with two 2026-11-20 repeats (n=3). Throughput is total token throughput (AIS Bench). DBO-enabled rows use decode/prefill thresholds of 32/512; the earlier 2026-11-18 dataset used 2/12 and is not comparable.

Routing Transport DBO off DBO on DBO effect
Attention BF16 350,695 335,739 -4.3%
Attention MXFP8 369,810 389,311 +5.3%
FFN BF16 362,258 (n=3) 355,050 (n=2) -2.0%

With DBO on and Attention-side routing, MXFP8 is 16.0% above BF16; with DBO off the gap is 5.5%.

Combined configuration against the same-build default baseline. Baseline (gate-on-FFN, BF16, DBO off, n=3): 362,258 total token/s, mean E2EL 2,454.8 ms. The combined configuration (Attention-side routing, MXFP8, DBO on) measured 389,311 total token/s and mean E2EL 2,279.0 ms, i.e. +7.5% throughput and -7.2% mean E2EL.

Not measured: per-build split evidence, accuracy on the DBO-enabled arms, and any mechanism-level attribution of the DBO difference.

Two constants had no callers: A2E_MXFP8_ARGUMENT_NAMES duplicates the two names
the a2e schema probe compares inline, and MX_BLOCK_SIZE duplicates the A2E MX
group size while the actual arithmetic uses A2E_MX_GROUP_SIZE. Deleting them keeps
the probe and the scale layout with a single source each.

Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
@Pingzii Pingzii changed the title feat(npu): carry a pre-quantized activation over the sync A2E transport feat(npu): support MXFP8 transport and native DBO in sync CAMP2P Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants