Repository navigation
Conversation
191c4f2 to
4d2d890
Compare
jiangkuaixue123
left a comment
There was a problem hiding this comment.
[P2] Reject unsupported MXFP8 rank counts at connector construction (afd_plugin/connectors/npu/camp2p.py:554). With quant_mode=4, a 4A2F deployment can pass the AIV and schema checks and proceed to the first A2E call. The operator supplied by #444 explicitly requires equal positive Attention and expert rank counts, at most 32 each. The asymmetric padding path at line 1405 cannot satisfy that contract. Please enforce the rank constraint in the mode-4 validator, document it beside the 4A2F example, and cover asymmetric rejection and symmetric acceptance.
The GSM8K scores are from the same 1,319 questions. The two aggregate percentages alone cannot establish accuracy neutrality. Please report the paired counts (BF16-only correct and MXFP8-only correct) and a paired interval for the difference, or describe the observed 1.14-point drop without claiming equivalence. Static review of the incremental change and #444 operator contract. I did not reproduce device measurements.
## Background The synchronous CAMP2P path normally runs attention on the Attention role and the router, shared experts, and routed experts on the FFN role. This PR makes router/shared placement an explicit deployment option. With `compute_gate_on_attention=true`, Attention computes top-k and shared experts, while FFN consumes the external routing and runs the routed experts. ## Motivation **Improve eager execution through a different A/F work split.** The measured Attention-side placement delivers **+17.6% throughput** versus the same-commit FFN-side placement in the evaluated eager workload. **Establish the routing contract for quantized activation transport.** Once FFN receives the routing alongside the activation, it can consume an activation quantized on Attention. The MXFP8 pass-through builds on this contract in #506. The option is **off by default** and belongs in the common `additional_config["afd"]` block. ## Implementation ```text Default Attention : attention FFN : router + shared experts + routed experts compute_gate_on_attention=true Attention : attention + router + shared experts FFN : transported top-k + routed experts ``` - **Reuse the A2E routing slots.** Send expert IDs as `int32` and top-k weights as `float32` through the existing ids channel. - **Owned prepare and comm subclasses.** `AFDPrepareAndFinalizeWithAllGather` and `AFDPrepareAndFinalizeWithMC2` provide `prepare_external`, which takes the routing for the local input rows and returns the complete `MoEPrepareOutput` together with the aligned routing. AFD comm subclasses override only `_get_prepare_finalize()`, so the native selector still decides which path runs and ordinary execution keeps the upstream classes. There are no global prepare patches left. - **Explicit routing contract.** The routing is an argument and a return value; nothing is stashed on the upstream class, and the module no longer reads or clears an `afd_external_routing` attribute. - **One MoE component.** `AFDExternalRoutingMoE` builds its comm candidates during model initialization, where the config context exists, and each forward only looks up the candidate matching the native selection. It runs prepare, expert execution and finalize, switches the forward-context comm for the duration of the call and restores it on every exit path, and rejects unsupported types such as fused-MC2. - **Consume the external top-k on FFN.** The component hands the aligned routing straight to expert execution. - **Place shared experts on Attention.** Run them between activation send and routed-output receive, then combine. - **Keep protocol selection explicit.** The model configuration selects token-ID or external-routing semantics; the model itself only builds the component for the synchronous FFN role and delegates. The supported opt-in configuration is synchronous CAMP2P, DeepSeek-V4, TP=1, without DBO or ubatching. Both roles must use matching routing settings. ## Performance and accuracy Measured at `828582a` on one A5 4A4F node during the same evening session, with BF16 A2E transport and automatic communication selection: | Configuration | Eager throughput (token/s) | Graph throughput (token/s) | | --- | ---: | ---: | | Attention-side gate (`828582a`) | **192,887** | **326,400** | | FFN-side gate (`828582a`) | 163,997 | 331,455.5 | | Clean baseline (`bcc0126`) | 164,543 | 333,407 | | Comparison | Eager | Graph | | --- | ---: | ---: | | Attention-side vs FFN-side placement | **+17.6%** | **-1.5%** | | FFN-side placement vs clean baseline | -0.3% | -0.6% | The result supports Attention-side placement as an eager optimization and a foundation for MXFP8 pass-through. Placement remains opt-in, allowing deployments to select the appropriate A/F split for their execution mode. Graph-mode Attention-side routing scored **90.60% on the full 1,319-question GSM8K evaluation**, with normal generated output. Measurement setup: concurrency **128**, reported input **8,275** / output **32** tokens per request, `max-num-batched-tokens=3072`, `max-num-seqs=128`, and zero failed requests. Graph uses `FULL_DECODE_ONLY` with capture sizes `[8,16,32,64,128]`; eager uses `--enforce-eager`. Each cell has two runs from the same firmware/session. These are workload-specific measurements with small run counts. ### Re-measured on the target stack at `e2c7075` The refactored implementation was re-measured on the stack this PR declares: vLLM `0.32.0` @ `5df63a7`, vLLM-Ascend `bd42eea0`, BF16 A2E transport, Attention-side gate, automatic communication selection, TP=1, eager (no CUDA/ACL graph), 4A4F on one A5 node. | Run | E2E (ms) | Prefill (token/s) | Output (token/s) | Total (token/s) | Failed | | --- | ---: | ---: | ---: | ---: | ---: | | 13:28 | 5,126.2 | 1,614.2 | 728.59 | 189,136 | 0 / 128 | | 13:34 | 5,146.5 | 1,607.9 | 726.69 | 188,643 | 0 / 128 | The two runs agree within 0.4%, so they are recorded as a repeat baseline rather than a single sample. GSM8K on the same configuration scored **89.92%** on the full 1,319-question evaluation. These numbers describe this configuration only: there is no same-stack reference implementation to compare against, so they are not evidence of a performance change from the refactor. The pre-refactor table above was measured on the previous stack and stays attributed to `828582a`. ### Placement re-measured on the target stack (`e2c7075`) Same node, same stack (vLLM `0.32.0` @ `5df63a7`, vLLM-Ascend `bd42eea0`), same workload (128 concurrency, 8,275 input / 32 output tokens per request), zero failed requests. The clean baseline is upstream main at `b093863`, which keeps the gate on the FFN side. | Mode | Clean baseline `b093863` (total token/s) | Attention-side gate `e2c7075` (total token/s) | Delta | | --- | ---: | ---: | ---: | | Eager | 165,345 | 189,136 / 188,643 | **+14.2%** (mean) | | Graph (`FULL_DECODE_ONLY`) | 364,627 / 362,563 | 354,731 / 353,972 | **-2.5%** (mean) | On this workload, Attention-side placement averages about **+14.2% higher total throughput in eager** and about **2.5% lower in graph**. That is a placement comparison across two commits on one stack; it is not a pure refactor cost, and it does not mean Attention-side placement is unsuitable for graph mode. Graph mode also raises this configuration's own throughput by **87.6%** over eager (189,136 -> 354,731 total token/s). GSM8K on the target stack scored **89.92%** in eager and **89.99%** in graph mode, both on the full 1,319-question evaluation. ## Baseline and scope Rebased onto main at `e58fdec`, which includes the merged #500 opaque routing-op fix. The retired `profile_run_eager` implementation and its wiring have been removed. #506 builds MXFP8 pass-through on this routing contract. [Incremental diff for this PR: `e58fdec...2c17cce`](Pingzii/afd-plugin@e58fdec...2c17cce) ## Verification CPU, measured at `2c17cce`: **59 passed, 2 skipped**. The focused set covers the routing-id op, the runtime installer wiring, the CAMP2P token-ID path, the routing payload transforms, and the new MoE component. The maintained CPU suite also passes. Two mutation checks were run on the refactored component before the rebase, and both produced the expected failures: - disabling the per-token state cleanup makes the MC2 finalize case fail (it observes state that should already be cleared); - disabling the comm restore makes the prepare-failure case fail (it observes the AFD comm left in place). Device evidence: the head `7430a13` starts on one A5 4A4F node with BF16 transport, `compute_gate_on_attention=true`, the CAMP2P connector, TP=1 and eager mode, and completes a generation request (`POST /v1/completions` returned 200). Accuracy and throughput were measured at `e2c7075`. The pre-refactor table above stays attributed to `828582a`. --------- Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com> Co-authored-by: 宋元盟 <songyuanmeng2@huawei.com>
4d2d890 to
86495ce
Compare
c0c42e3 to
5041bd5
Compare
|
Ready for re-review at
CPU verification on this head: full |
ad9044a to
44394d1
Compare
|
Two follow-ups are in at The handshake has a test that fails if it is deleted. Rather than adapting the The guide states the startup behavior. The two bullets that had been joined are split, the Two carry-overs from the earlier, now superseded check. The per-payload mode comparison is still in the code and will go away in the follow-up that makes the startup handshake the single protocol check. What stays regardless: construction requires an Ascend A5 device ( Testing is down to the seams that own each behaviour. Five cases are gone at The convergence follow-up will also keep the JSON round-trip test (dropping only its CPU verification at this head: connectors and model_executor 323 passed / 28 skipped; ruff check/format, typos and the changed-file mypy clean. Still unrun: the real HCCL startup handshake, mode 4 end-to-end on ALLGATHER and MC2, and the operator capability marker. |
8b87fe4 to
a33f413
Compare
Three unit tests fail on Windows for reasons unrelated to what they check: - `tests/unit/compat/npu/test_ascend_config_lifecycle.py` reads a source file with `Path.read_text()` and no encoding, so the GBK locale decodes UTF-8 bytes and raises `UnicodeDecodeError`; both reads now state UTF-8. - `tests/unit/package/test_cam_async_build.py` builds its manifest file list with native separators while setuptools works on posix manifest paths, so every required entry misses; the list now uses `as_posix()`. - `tests/unit/package/test_package.py` needs installation metadata to find the `vllm.general_plugins` entry point, which a source tree does not have; it now skips with that reason, and still checks the entry point when the distribution is installed. No production code changes. With these, full `tests/unit` on a Windows CPU host with `torch` and no `vLLM` is 900 passed / 89 skipped / 0 failed (previously 8 failed). Where these come from, for review context: - the six GBK failures: `tests/unit/compat/npu/test_ascend_config_lifecycle.py` arrived in #425 (`b7561d7`) with `ast.parse(path.read_text())` and no encoding, so it is latent everywhere except a locale that cannot decode UTF-8; - the manifest failure: #539 (`f2c79d8`) replaced the posix-normalised `required` set with `set(files.allfiles)`, whose entries carry native separators, so the comparison only holds where `/` is the separator. The #361 version of the test compared normalised paths and was correct; - the entry-point failure: the test first appeared as `tests/test_package.py` in #1 (`dffdb14`, "Add Phase 1 plugin skeleton") and was later moved to `tests/unit/package/test_package.py` by `d1fd84f`; it always needed installation metadata. None of the three involves the transports in #506; the tests are older than that branch. On verification scope: this PR's checks are label, pre-commit and DCO -- none of them installs the plugin or runs pytest. The numbers above come from a local run on this branch; the entry-point test exercising an installed build is therefore *not* covered by CI, before or after this change. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
a7ab213 to
0f059d9
Compare
0f059d9 to
aaa4dc1
Compare
|
Could you check whether the GSM8K accuracy drop from 90.90% to 89.76% is expected with MXFP8 transport? It would help to confirm whether this comes from quantization or run-to-run variance, and whether it falls within the acceptable accuracy tolerance. |
The sync transport can only carry BF16 activations today. This adds the payload half of an opt-in MXFP8 mode: the routing carries the activation MX scale, one E8M0 exponent per 32 values and one row per token. The scale follows the same row transforms as the routing -- pad, gather and split -- because a row whose scale stays behind describes another token. Padding writes the byte 127 (2**0, the neutral exponent) through the byte view, because E8M0 stores an exponent and the collectives are not required to accept the float8 dtype. That view is what the payload fakes reject, so a transform that hands a collective the float tensor instead of its bytes fails the tests rather than passing silently. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
An operator build that predates the MXFP8 arguments would either reject the fourteen-argument call or accept it and ignore the scale. The probe answers three ways -- the schema takes x_scale and quant_mode, it does not, or it cannot be read -- so the caller decides what an unreadable build means instead of treating it as support. The check is structural, not textual. str(torch.Type) renders Optional[Tensor] in a form that does not parse back, and a bare Tensor[] would satisfy a name test while the operator expects the optional wrapper. Inspecting the types keeps both faults visible. The probe tests build their schemas with torch._C.parse_schema, so they exercise the same rendering the production path reads. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
connector_extra_config.quant_mode=4 quantizes the Attention activation to E4M3 inside the opaque A2E call, ships it with its E8M0 MX scale, and hands the pair to the experts on the FFN side without dequantizing. Mode 0 keeps the twelve-argument BF16 call, so an operator build that predates the MXFP8 arguments still runs. Construction refuses mode 4 unless the deployment can carry it: the common gate must be on Attention, both sides must have the same rank count between 1 and 32, three AIV cores are needed, and the operator probe must have said yes. The FFN receive refuses the token-ids slot in this mode, because the router and the shared experts need the model dtype. Quantizing the final wire tensor keeps the scale covering exactly the rows the operator sends; mode 4 cannot reach the padding branch because it requires equal rank counts. The scale travels with the routing, so prepare pads, gathers and shards it by the same rule as the activation rows. What consumes it fixes the layout: ALLGATHER re-lays it as (rows, cols // 2, 2) for the fused GMM, while MC2 hands the two-dimensional scale to its dispatch operator. MC2 gets a dispatcher subclass that forwards the scale and selects quant_mode=0 instead of letting upstream quantize an activation that already is; an explicit comm_quant_mode that disagrees is rejected rather than overruled. The tests drive the real prepare transforms and the real FFN entry point, with only the connector payload supplied, and pin the bytes the experts read: every scale row carries different values, so a transform that reorders, pads or re-lays it cannot keep them in place. This branch has no NPU run. The operator that accepts x_scale and quant_mode lives in vllm-project#444, so the mode is wired and reviewed but unverified on hardware. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
quant_mode now has two legal values, so the connector tables and the support matrix state what mode 4 carries and which deployment shapes it requires. The payload contract gains activation_scale, including the row-per-token and ceil(hidden / 32) shape the transport validates. The support matrix records that mode 4 has no device run in this repository yet, rather than implying the mode is verified because it is wired. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Both roles read their own `connector_extra_config.quant_mode`, and each value is individually legal, so an Attention role on 4 with an FFN role on 0 passed both local checks while the two A2E packet layouts differ in the activation offset and the element width. The FFN rank would read the peer's bytes as its own mode and hand the experts a silently wrong activation. Two checks, in the order they bite: - `init_afd_connector` gathers every rank's mode over the AFD group and fails them all when the set is not a singleton. Every rank joins that group, so the 4/0 and 0/4 pairs both fail here -- before profile, capture and the first A2E call -- rather than leaving the Attention side waiting for an E2A the FFN side will never send. - The control payload also carries the sender's mode, and the FFN role compares it in `recv_dp_metadata_list`. That path covers a peer that started before the group handshake existed; a payload without the field reads as 0, which is the right answer only for a peer that never had mode 4. A `5041bd5` peer does have it, so the handshake above is the one that keeps that combination from being silently accepted. `quant_mode=4` also needed a way to see a device whose operator build lacks the path: the operator registers the same fourteen-argument schema on every build and rejects mode 4 only at runtime, so the construction gate now checks for an Ascend A5 device. The dtype half of that build check needs a capability marker from the operator side (vllm-project#444); until it exists this stays a documented requirement rather than a check. Tests: ranks must agree on the mode (4/0 and 0/4 both raise), the mixed pair fails at the handshake, a matching pair passes, the send side declares its own mode, the codec round-trips the field and defaults a missing one to 0, and the A5 check rejects another device family. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Three cases restated behaviour another seam already pins: the two `compute_ffn_output` cases re-bound the same extracted methods that the runner case drives through the real `_ffn_forward`, and the component-level row-order case asserted the same bytes and layout as that runner case. They are gone. Three pairs with identical setup became parametrize tables, keeping every scenario: the pair contract now covers E4M3 without a scale, a scale next to BF16, E5M2, and a scale with the wrong row count in one table; MC2 dispatch covers an unset and an explicit pre-quantized comm mode; the control-payload check covers 4/0, 0/4 and 4/4. The BF16 no-scale assertion moved onto the vllm-project#503 routing case, which now states `torch.bfloat16` explicitly instead of relying on the float32 default, and the startup handshake case now also pins that `p2p` is built after the collective. Mutation re-checked after the change: deleting the runner's `activation_scale` forwarding still fails both the ALLGATHER and the MC2 implementation of `test_the_ffn_runner_carries_the_received_scale_to_the_experts`. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Round two of the same review: keep the seam that owns each behaviour and drop the rows that only restate it. - Schema probe: the float and int `x_scale` cases both hit the same OptionalType rejection, so they are gone. `Tensor[]` stays -- it is the one that catches an element-type-only check -- and it now parses directly, so the hand-built ListType scaffolding is gone with it. - The FFN runner case keeps MC2 only: the DP-gather case above it already pins the ALLGATHER three-dimensional layout, its row order and the padding byte. - The MC2 rejection table keeps `scale-dim`, `not-mxfp` and the `comm_quant_mode` conflict. The two dtype rows restated what the normal A2E to MoE path validates before the dispatcher is reached. - The control payload's `quant_mode` assertion folds into the existing JSON round-trip; the missing-field-reads-as-0 case stays on its own. - The startup check now runs through the real `init_afd_connector` for all four outcomes (0/0, 4/4, 4/0, 0/4): the fake collective asserts the mode that was actually published, and the refused cases assert `afd_moe` and `p2p` are never built. The direct helper call went away, so the test fails if the entry point stops calling the check. - Sending is one test now: the public `send_attn_output()` drives the registered op, and the operator boundary asserts one fourteen-argument call, the quantized activation, and the scale with its mode. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
- The payload padding case restates `padded()`: the ALLGATHER component case walks the same call and checks the original rows and the appended 127 bytes. The gather case with distinct per-rank bytes stays, since it proves the order. - The zero-block scale case can only be reached by hand: the connector already validates the column count as `ceil(hidden / 32)` before a routing payload exists. - In the MoE pair table, `row-count` is rejected by `AFDExternalRoutingInputs` even if the component's own check is bypassed; the payload-level row test stays, and so does `no-scale`, which is what catches a runner that stops forwarding the scale. - In the MC2 dispatch table, `scale-dtype` and `activation-dtype` restate what the normal A2E to MoE path validates before a dispatcher is reached. `scale-dim`, `not-mxfp` and the `comm_quant_mode` conflict stay. No production change in this commit; the Gloo payload field is untouched. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The operator moves a call routing ids as one `[rows, topk]` int32 block into a single UB region, so `batch_size x topk x 4` has to fit 195,584 bytes. Nothing in the connector checked it, so `--max-num-batched-tokens 8192` with `topk=6` started cleanly and the real operator would then reject the full batch. Construction now computes the largest batch the mode can send (`8,149` tokens at `topk=6`), refuses a configuration above it, and the guide states the bound next to the other mode-4 requirements. The tests cover the boundary (8,149 accepted, 8,150 refused), a configured maximum that exceeds it, and a smaller `topk` that fits under the same batch. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
A mode-4 pair can pass every construction check and still have the FFN resolve to a MoE path the transport cannot feed: the pinned selector picks ALLTOALL or FUSED_MC2 depending on batch size, EP, LoRA and MegaMoE state, and the external-routing consumer supports only ALLGATHER and MC2. The control plane is one-way, so refusing after the first control payload would leave the Attention side inside an A2E waiting for an E2A that never comes. The startup handshake grows a second step. The first collective now exchanges `[quant_mode, attention batch bound]` per rank -- only the Attention role publishes a bound, and the aggregate stays FFN first, so the FFN reads its candidate from the Attention rows alone. A mixed set fails there, both roles at once. An all-BF16 set continues without the second collective, so the ordinary deployment keeps exactly one startup collective. An all-mode-4 set asks the pinned selector, with its import, capacity lookup and call all inside the guard: a selector that raises still publishes its verdict, because a rank that exits early leaves the peer waiting. The verdict travels on the second collective and every rank acts on it -- the FFN re-raises with the original exception as its cause, the Attention side reports that the peer refused. Tests drive the real `init_afd_connector` for both roles with 1A1F and record each collective separately (`mode_gather`, `verdict_gather`), so the stop point is proved rather than inferred: mixed pairs stop at the first, an unsupported path or a raising selector stop at the second, and a mode-0 pair never reaches the second. The FFN publishes no bound, the selector is asked about Attention 1024 rather than the FFN own 512, and the failing role keeps its original cause. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Two ways a configured pair could still reach a working transport and then read or route wrongly: - The roles read `afd.compute_gate_on_attention` independently. With the gate on the A2E ids slot carries routing ids, with it off the same slot carries token ids, and both configurations pass construction, so a mixed pair exchanges payloads successfully: the FFN reads expert ids as token ids in one direction and token ids as routing ids (with zero weights) in the other. The startup handshake now carries the flag and fails both roles when they disagree, before any A2E call. - The selector pre-check only ran for mode 4. BF16 routing has the same ALLGATHER/MC2 contract, so a pair such as 8A8F with a bound beyond the MC2 capacity started, served small batches, and only failed at prefill when the pinned selector chose ALLTOALL. The verdict now runs whenever routing is carried -- mode 4 always, mode 0 when the gate is on -- so the pair is refused at startup. A gate-off BF16 pair still exchanges exactly one collective. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The regression worth pinning is not that a mismatch is detected but that neither side continues: if only the FFN rejects the gate disagreement, the Attention role walks on to the verdict collective and waits for a peer that has already left. The case now runs once per role and asserts the same stop point on both -- the group exists, the first gather happened, nothing else was created -- so a one-sided rejection fails the test instead of passing it. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
0546493 to
f862904
Compare
|
We followed up on the earlier 1.14 pp drop with repeated runs per mode on the full 1,319-question GSM8K set, evaluated with AIS Bench (
The earlier drop did not recur in this follow-up. In Graph the MXFP8 mean is 0.19 pp higher; in eager it is 0.07 pp lower -- both inside the spread of their own runs, and the two directions disagree. The two BF16 Graph runs differed by 0.76 pp, the four MXFP8 Graph runs spanned 0.68 pp, the two BF16 eager runs by 0.08 pp, and the two MXFP8 eager runs by 0.38 pp. The modes ran in separate time blocks and used different A2E builds, so this comparison leaves the cause of the earlier drop and the acceptable tolerance open. The remaining check is to alternate modes on the same #444 build -- the BF16 runs above predate it -- keep the evaluation settings fixed, and retain per-question results. For context, the measurement that raised the question: 2026-10-30/31, same tree ( The comparison for adopting this mode is separate: clean |
The synchronous gate shell sent and received on the AFD metadata stage and never yielded, so DBO/ubatching was rejected for it even though the connector already keeps one AFD group per ubatch and the FFN runner receives on the stage it is told. The shell now resolves the stage once, before the routing send, prefers the forward context's ubatch_idx and falls back to the metadata stage, writes the result back to the metadata, yields between the send and the receive, and receives on the resolved stage so a peer ubatch cannot switch it mid-transfer. The validator accepts DBO/ubatching for this shell with exactly two microbatches on either transport; the guide and the model boundary state the same contract. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
The sync gate tests now stub the DBO custom op, resolve the stage against a forward context whose ubatch_idx differs from the metadata stage, and move both stage sources inside the yield so a receive that re-reads either fails. A second case runs two ubatches through the shell and checks that each send carries its own routing and activation and that each receive stays on its own stage. The connector case drives mode 4 for two stages and checks that the E4M3 activation and its E8M0 scale cover exactly the rows of the stage being sent. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
Merge the nested DBO timing check into one condition for pre-commit, parametrize the acceptance test over both transports and both roles, and let the mode-4 send case assert the per-stage group name, an E8M0 scale and legal routing inputs. The guide, the support matrix and the connector contract drop the remaining statements that the Attention-side gate cannot run under DBO. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
quant_mode=4 always transports an E4M3 activation with its E8M0 scale, but the MoE entry only checked that the pair matched. Prepare skips requantization for a pre-quantized input, so an FP4 or BF16 expert type consumed the E4M3 bytes as its own activation format and reached the fused GMM mismatched -- the MC2 path in particular accepts W4A4MXFP through its is_mxfp check. The entry now requires an expert type whose activation is FP8 (W8A8MXFP, W4A8MXFP) whenever a scale is present, and rejects the rest with the reason. Tests cover the two rejected types and an accepted W4A8MXFP case; the stub expert type defaults to W8A8MXFP so the existing pair cases keep their intent. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
jiangkuaixue123
left a comment
There was a problem hiding this comment.
Posted by an agent on behalf of @jiangkuaixue123.
Thanks for putting the transport and DSV4 integration together. I think the CAMP2P changes are worth separating conceptually, but the MXFP8 wire format and its FFN consumer must remain one compatible feature: connector.py sends E4M3 plus E8M0 scale, while the DSV4 MoE path forwards that pair into ALLGATHER/MC2. Landing only one end would leave the mode unusable or misinterpreted.
Suggested split:
- Keep the already self-contained native DBO shell/validation work (
deepseek_v4.py,feature_validation.py, about 25 changed lines) as its own change. It is already isolated in commits81f1c28ande038326; this also avoids mixing scheduler behavior with transport quantization. - Keep the MXFP8 end-to-end contract together for correctness: A2E quantization, schema/device/shape checks and startup handshake (
connector.py,compat/npu/ops.py, about 570 added lines); scale in external-routing metadata and row alignment (external_routing.py,metadata.py, about 90); and the DSV4 consumer/model seam (deepseek_v4_external_routing_moe.py,deepseek_v4.py, FFN runner, about 190). Tests and docs are another roughly 1,300 changed lines. These are reasonable review-sized commits within one feature PR, but extracting a “CAMP2P-only” PR from the FFN changes would break the contract. - For future reuse, the A2E MX quantization/pair validation currently contributes roughly 100 lines in
connector.py:1475-1548; consider moving that transport-specific code to a small CAMP2P transport module after this feature lands. Keep connector lifecycle and handshake ownership in the connector. This narrows future wire-format work without introducing a cross-PR protocol dependency.
Two concrete follow-ups:
- The docs require Attention DP > 1 for DBO/ubatching (
docs/npu/CAM_P2P_CONNECTOR_USER_GUIDE.md:137-138), but the validator accepts DP=1 whenevernum_ubatches == 2. At DP=1 the runtime does not construct the ubatch runner, so this configuration can pass validation and silently run without DBO. Please rejectdata_parallel_size <= 1here or make the documented/runtime behavior explicit. This is also the remaining finding from the now-closed #607. - Please add a same-configuration DBO-off vs DBO-on performance comparison (same model, rank placement, input mix, graph mode and repetitions). The current reported MXFP8 vs BF16 result with DBO enabled does not isolate DBO's effect.
Other code-grounded note: the connector currently owns transport-mode setup, startup capability agreement, A2E calls, and pair validation in one file (connector.py:654-800, :1100-1200, :1475-1548). The module extraction above would make these responsibilities easier to review while retaining the startup agreement where it can fail both roles before data transfer.
| f"TP={int(parallel.tensor_parallel_size)}", | ||
| ) | ||
| if bool(parallel.enable_dbo) or bool(parallel.use_ubatching): | ||
| if (bool(parallel.enable_dbo) or bool(parallel.use_ubatching)) and int( |
There was a problem hiding this comment.
The docs require Attention DP > 1 for DBO/ubatching (docs/npu/CAM_P2P_CONNECTOR_USER_GUIDE.md:137-138), but this new validation only checks that num_ubatches == 2. At DP=1 the runtime does not construct the ubatch runner, so this configuration passes validation and silently runs without DBO. Could this also reject data_parallel_size <= 1 (or could the docs describe the actual behavior)? This is the remaining finding from the now-closed #607.
There was a problem hiding this comment.
Fixed at 3cbdcb0. When DBO/ubatching is requested, the gate validator now also rejects Attention DP = 1:
if getattr(afd_config, "role", None) == "attention" and int(
parallel.data_parallel_size
) <= 1:
raise RuntimeError(
"Attention-side gate with the synchronous connector requires "
"Attention DP > 1 for DBO/ubatching; at DP=1 the runtime does not "
"build the ubatch runner and DBO would not run. Got "
f"DP={int(parallel.data_parallel_size)}",
)So a DP=1 deployment now fails at validation instead of running without overlap, which matches what the guide already stated. Coverage: test_dbo_is_rejected_without_attention_data_parallelism asserts the rejection on the Attention role, and the accepted two-ubatch case still runs at DP=4. CI (DCO, label, pre-commit) is green on 3cbdcb0.
At DP=1 the runtime returns no ubatch runner, so the configuration passed validation and then ran without overlap. The gate validator now rejects Attention DP <= 1 when DBO/ubatching is requested, which is what the guide already stated. Adds the rejection test and keeps the accepted two-ubatch case at DP=4. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
|
Addressed the two concrete follow-ups from the latest review. DP=1 validation gap (inline finding). Fixed in 3cbdcb0: when DBO/ubatching is requested, the gate validator now also rejects Attention DP = 1, because the runtime builds no ubatch runner there and the target would silently run without overlap. This matches what the guide already stated. New rejection test, and the accepted two-ubatch case stays at DP=4; the config file is 14 passed, full DBO off vs on comparison. Not measured yet, so the Performance table stays an MXFP8-vs-BF16 result with DBO configured rather than a DBO claim. It needs a device run on the same build (same model, 4A4F placement, input mix, graph mode and repetitions) with the four DBO flags removed for the off arm; I will add it as measured data when it exists rather than extrapolate. Split guidance. Taken: the native DBO shell/validation is already isolated in |
|
Updated measurements addressing the DBO off/on comparison are below. DP=1 validation gap. Fixed at Same-configuration DBO off vs on. One build (
With DBO on and Attention-side routing, MXFP8 is 16.0% above BF16; with DBO off the gap is 5.5%. Combined configuration against the same-build default baseline. Baseline (gate-on-FFN, BF16, DBO off, Not measured: per-build split evidence, accuracy on the DBO-enabled arms, and any mechanism-level attribution of the DBO difference. |
Two constants had no callers: A2E_MXFP8_ARGUMENT_NAMES duplicates the two names the a2e schema probe compares inline, and MX_BLOCK_SIZE duplicates the A2E MX group size while the actual arithmetic uses A2E_MX_GROUP_SIZE. Deleting them keeps the probe and the scale layout with a single source each. Signed-off-by: 宋元盟 <songyuanmeng2@huawei.com>
## What changed
Adds opt-in MXFP8 activation transport and native DBO support to synchronous CAMP2P with Attention-side routing.
Attention quantizes activations with
torch_npu.npu_dynamic_mx_quantinside CAMP2P's opaque send operation, immediately before calling the nativea2eoperator. FFN consumes the transported activation and scale through ALLGATHER or MC2 without dequantizing and requantizing.quant_mode=0retains the original twelve-argument A2E call, including compatibility with operator builds that predate the MXFP8 arguments.Code volume
Relative to the merge base (
cb2eec8), the branch changes 22 files, +2,571 / -95:afd_plugin/tests/docs/The DBO change is isolated in its own commits; the MXFP8 wire format and its FFN consumer stay together in this PR because either half alone is unusable.
Deployment
CAMP2pAFDConnector, DeepSeek-V4, TP 1, andafd.compute_gate_on_attention=trueon both roles.x_scaleandquant_modeinput schema. The [WIP] feat(npu): support MXFP8 datatype for A2E operator #444 operator implementation remains a separate change.Build the compatible operator with:
SOC_VERSION=ascend950dt_9574 pip install -e . --no-build-isolationSee the CAMP2P guide for the configuration contract.
Verification
082b2b8:test_sync_attention_gate_proxy.pyandtest_camp2p_connector.pycompleted with 21 passed, no skips.Tests cover byte-level scale transforms, runner-to-expert forwarding, MC2 dispatch arguments, startup mismatch rejection, and the two-microbatch contract. DBO cases cover both transport modes and roles, preserve the stage across a yield that changes both stage sources, and verify that mode-4 sends use the correct stage's rows and communication group.
Accuracy (GSM8K, DBO disabled)
Full 1,319-question GSM8K with AIS Bench (
gsm8k,gen), Attention-side routing,max-num-batched-tokens 3072,max-num-seqs 128; 2026-11-16/17, buildsf862904(BF16) and857d8e1(MXFP8, with the #444 operator). Two to four runs per cell, DBO disabled:Run start times (host local): BF16 Graph 13:51 / 13:57 (11-16); MXFP8 Graph 16:48 / 16:53 (11-16) and 11:31 / 11:37 (11-17); BF16 eager 11:39 / 11:51 (11-16); MXFP8 eager 11:01 / 11:14 (11-17).
In these follow-up runs, MXFP8 averaged +0.19 pp relative to BF16 in Graph mode and -0.07 pp in eager mode. Thread: PR comment.
Performance
Single-node A5 4A4F, ACL Graph (
FULL_DECODE_ONLY), 8,192 input / 32 output tokens, 128 concurrent requests, two runs per arm unless noted. All rows come from one build:0433169=3cbdcb0plus the #444 A2E operator commitb5c64100. Everything is measured on 2026-11-19 except row 1, which pools the 2026-11-19 run with two 2026-11-20 repeats; only the listed flags differ. Throughput is total token throughput (AIS Bench).Throughput comparisons using the means above:
The combined configuration also reduces mean E2EL by 7.2% relative to the same-build default-configuration baseline (6 vs 1; baseline
n=3, measured on 2026-11-19 and 2026-11-20).With BF16 transport, enabling DBO reduces throughput by 2.0% with FFN-side routing (2 vs 1) and 4.3% with Attention-side routing (4 vs 3). With DBO disabled, Attention-side BF16 routing is 3.2% lower in throughput than FFN-side routing (3 vs 1). FFN-side BF16 routing also sees a 3.1% increase in mean E2EL with DBO enabled (2 vs 1).
DBO thresholds determine which model-execution steps are eligible for splitting, based on the number of tokens in the batch:
--dbo-decode-token-threshold)--dbo-prefill-token-threshold)32/512(used above)2/12Both settings use two microbatches (
--ubatch-size 2). The thresholds are batch token counts, not request counts or microbatch sizes. Lowering them to2/12makes smaller steps eligible for DBO. A separate MXFP8 run with2/12on the same build measured 329,021 total token/s (n=1).Pre-DBO transport measurements (2026-10, no DBO)
f683eb152a292df683eb1These runs predate the DBO work, so no DBO flags were set; they are history only. Against the upstream
BASE/mainrun onbcc0126(340,844, n=3), MXFP8 measured +4.3% (n=4); against gate-on-Attention BF16 on the earlier commit, the cross-commit gap was +6.1%. The historical DBO-off results show the same overall trend as the current measurements: MXFP8 improves throughput with Attention-side routing, while moving routing to Attention slightly reduces BF16 throughput. The respective changes were +6.1% and -1.9% historically, versus +5.5% and -3.2% in the current build.Earlier data (2026-11-18)
74d929c(d211266+ #444 operator)2/1274d929c2/12These rows are history only: both the build and the DBO configuration differ from the table above. The DBO configuration has a large effect on the result: with
2/12, the MXFP8 arm measured about 15% lower than with32/512on the measured build (329,021 versus 389,311 total token/s). Rows from different DBO configurations are therefore not directly comparable. Within these rows, however, MXFP8 is still 3.4% above BF16 (339,506 versus 328,367 total token/s), so the transport gain holds even under the aggressive configuration.Reproduction: Attention on devices 0-3 (port 18000), FFN on 4-7 (port 18010), FFN started first. Both roles share:
DBO arms add
--enable-dbo --dbo-decode-token-threshold 32 --dbo-prefill-token-threshold 512 --ubatch-size 2; routing moves with the commoncompute_gate_on_attentionfield. The numbers come from AIS Bench with the chat model config (DeepSeek-V4-Flashathttp://127.0.0.1:18000/v1/chat/completions),synthetic_gen, 8,192 input / 32 output tokens, 128 requests, concurrency 128: