Repository navigation
Conversation
226e748 to
b478f55
Compare
|
移除CANN的依赖 并且标明只支持CANN 9.2+ |
|
Thanks for the large integration — two requests before a detailed review, both about the vendored CANN sources. 1. Please drop the bundled CANN dependency from the diff. As it stands this PR is 1073 files / ~175k added lines, and about 134k of those (around 860 2. Please state the supported CANN version explicitly: CANN 9.2+ only. The PR body currently mentions Once the vendored tree is out, the remaining diff is small enough for a proper review of the connector/runtime logic, and @jiangkuaixue123 will take a look. Scope checked: file-level diff statistics and directory/licence layout at head |
53be9ab to
f3f24a1
Compare
20cc78e to
f3f24a1
Compare
| @@ -0,0 +1,3 @@ | |||
| # ### PATCH START: AFD standalone MegaMoE extension | |||
| from afd_plugin._C_megamoe import CommContextManager | |||
There was a problem hiding this comment.
这个文件单独保留是为了兼容某个已有的导入路径吗?仓库里只有 megamoe_split.py 从这里导入 CommContextManager,而它已经直接从 afd_plugin._C_megamoe 导入了 SPLIT_SEQUENCE_MAX。如果没有兼容需求,能否把这两个符号放在同一条 import 里,然后删掉 comm_context.py?
|
想确认一下
|
|
|
||
| # Upstream: vllm_ascend/ascend_config.py at | ||
| # bd42eea01d8312c4df469fabaed45f77a12df4d3. | ||
| # f9c8e0bdb263d7a8c0270536f1ea00fa7373a84e. |
|
|
||
| import os | ||
|
|
||
| AFD_ASYNC_CAM_LAYERED_GMM = "AFD_ASYNC_CAM_LAYERED_GMM" |
There was a problem hiding this comment.
This is already moved. Please remove
There was a problem hiding this comment.
已删除对 AFD_ASYNC_CAM_LAYERED_GMM 的 Mega 语义改写,该变量仍只控制 Async CAM layered GMM。
|
请在后续 PR 中补充一个可在 A5 上独立运行的 MegaMoE 三算子测试程序,无需启动完整的 DeepSeek-V4 服务即可验证
请一并提供 A5 上可直接复现的启动命令、所需卡数和 A/F rank 布局,方便后续算子改动时做回归对比。 |
| """Check additional NPU features after common Worker deployment admission.""" | ||
|
|
||
| afd_config = afd_config or parse_afd_config(vllm_config) | ||
| layered_gmm_requested = async_cam_layered_gmm_enabled() |
There was a problem hiding this comment.
#538 在 Async CAM FFN 统一使用 layered GMM 后删除了 AFD_ASYNC_CAM_LAYERED_GMM。这里复用旧变量名来控制 A5 MegaMoE 的 host/device 选层,又在公共入口检查它。如果部署脚本仍给所有角色导出旧变量 =1,原先可以启动的 Attention worker 现在会在这里报错。能否给 MegaMoE 选层使用独立的配置名,并仅在 MegaMoE FFN 读取和校验,让已有部署继续忽略旧变量?
There was a problem hiding this comment.
已引入独立配置 AFD_MEGA_LAYERED_GMM,仅在 Mega FFN 读取。
| return any("DeepseekV4" in str(architecture) for architecture in architectures) | ||
|
|
||
|
|
||
| def _fail_if_unsupported_dsv4_async_features( |
There was a problem hiding this comment.
There was a problem hiding this comment.
已移除重复的 DSV4 Async CAM 校验 helper;Mega 的 A5/FP4/eager/geometry 约束由独立 validate_mega_config 保留。CAMAsyncAFDConnector 的 A5 约束整理随后续 PR。
| parallel_config.enable_dbo = saved.enable_dbo | ||
| parallel_config.ubatch_size = saved.ubatch_size | ||
| parallel_config.all2all_backend = saved.all2all_backend | ||
| if saved.ubatch_size > 1 and not saved.enable_dbo: |
There was a problem hiding this comment.
Why we need this.
There was a problem hiding this comment.
ascend_platform.py 里的 --ubatch-size 归一化分支是调试时加上的,已删除。Mega 复用 configure_async_cam_dbo 接口做 DBO 归一化,该接口内部现在按 connector 区分,不影响 CAMAsyncAFDConnector。
|
建议把这次改动按两条可独立验收的执行路径拆成两个 PR:
拆分时可以按调用链判断归属:如果某段 TP 布局或收发逻辑是 MegaMoE 独立运行所必需的,可以保留最小的共享部分,同时保持原有 Async CAM 的行为。这样当前 PR 可以集中审查三算子的协议、精度和性能,后续 PR 则可以单独验证旧 CAM 的 A5 BF16/MXFP4 路径和 A3 回归。 |
Extend the DeepSeek-V4-Flash `CAMAsyncAFDConnector` path to Ascend A5 and integrate `CAMAsyncMegamoeAFDConnector` with the async-megamoe operators, per Attention-side routing and local shared-expert execution, and adds the MXFP4/MXFP8 expert path plus A5 support for the bundled async CAM operators. Shared A5 connector and FFN execution code is reused by the DeepSeek-V4.1 Flash work in vllm-project#537. - Related issue(s): vllm-project#541, vllm-project#540, vllm-project#537 - Closes # - In scope: - A5 port of the four bundled `afd_async_dispatch_send/recv` and `afd_async_combine_send/recv` operators (the `cam_moe_distribute_*` suite): A5 HCCL window and resource access, AIV communication-engine selection, `ascend950` registration and build selection, Python bindings. - A5 DSV4 adaptation: MXFP4 expert weights in `FRACTAL_NZ_C0_32` with group-32 E8M0 scales, BF16 transfer for the MXFP4 expert path, and the matching quantization and weight-layout checks. - `CAMAsyncMegamoeAFDConnector` over the UMDK-provided `cam_megamoe_dispatch_send`, `cam_megamoe_fused_moe` and `cam_megamoe_combine_recv`: one shared communication context with input, output and control windows, wrap-safe sequence matching, device-side readiness and consumption signals, and FFN scheduling that selects the layer's weights and scales from `layer_idx`. - FFN-side scheduler with bounded work descriptors per (Attention DP, FFN rank), admission only after every source publishes, fused FFN execution driven by `execute_next_experts`, and two in-flight ubatches on isolated windows. - Topology and validation coverage: asymmetric A/F counts within tiling limits, DP-grouped submissions for Attention TP > 1, local token-row routing on plain Attention TP, and DBO normalization for async Attention. - Docs and unit tests. - Out of scope: - A3 W4A8/layered execution and ACL graph capture on A5. - ModelRunnerV1, performance qualification and tuning. - The three async-megamoe operators are consumed from the UMDK CAM wheel (`umdk_cam_megamoe`) in this change. Their sources and the in-tree build integration follow in a separate commit. The four `afd_async_*` operators are built from this repository's `csrc/npu` sources. - Attention keeps routing and shared experts. Its send/receive calls map to `cam_megamoe_dispatch_send` and `cam_megamoe_combine_recv`, and the matching dispatch state is retained until combine is submitted. - FFN exposes only `execute_next_experts`. Work descriptors travel through the Store while tensor data stays in the UMDK window, PUBLISHED/TAKEN/READY/ CONSUMED handling protects window reuse, and dual-stage admission is limited to FFN workers with two ubatches. - The MXFP4 weight adapter runs once after load: transpose, NZ re-pack into the operator's C0=32 container, and keep the group-32 E8M0 scales alongside. - A5 gating requires native FP4 experts, `dynamicQuant=0`, eager MRV2, static EP, `enable_fused_mc2=0` and layered GMM off. - Restore the DBO/ubatching fields that Ascend platform validation clears, and treat `--ubatch-size > 1` as a DBO request for async Attention instead of silently dropping the intent. - A5 operators: build the four `afd_async_*` operators for `ascend950`, then run `tests/npu/async_cam_precision.py` (dispatch ordering and weighted combine) and `tests/npu/async_cam_performance.py --measure-mode cycle` on 2A2F. - A5 E2E: 2A2F eager DeepSeek-V4-Flash with each connector, prefill plus generation, repeated startup/shutdown and uneven Attention DP traffic. - Numerics, same checkpoint and routing as the native A5 run: - `cam_megamoe_fused_moe` expert output against native A5 MoE for the same layer, weights and `group_list`. - `cam_megamoe_combine_recv` result against the reference top-k weighted sum of those per-expert outputs. - `cam_megamoe_dispatch_send` routing counts and token placement against the CPU reference used by `tests/npu/async_cam_precision.py`. - Whole-model output against native A5 inference. - A5 2A2F eager bring-up with `CAMAsyncAFDConnector` succeeds; 16-concurrent decode TPOT ~84 ms and single-request TPOT ~85 ms at `--max-num-batched-tokens 4096`. Functional check only. - `CAMAsyncMegamoeAFDConnector` end-to-end numbers are not collected yet. - Files updated: `docs/npu/A5_DSV4_ASYNC.md`, `docs/npu/A5_ASYNC_CAM_VALIDATION.md`, `docs/npu/MEGAMOE_EAGER.md`, `README.md`.
Add Ascend 950 dispatch-send, fused-MoE and combine-recv operators with shared MTE communication buffers, tiling, native bindings and Python APIs. Support FP8 activations with FP8/FP4 expert weights and carry layer_index with each DP request to select W1/W2 and scales from multilayer weight banks, using stacked tensors or layer-major per-expert TensorLists. Register weight dtype/format pairs consistently with tiling. Extend a build-local copy of the SDK format whitelist for FRACTAL_NZ_C0_32 while preserving SDK validation and installed files. Synchronize the HCCL warmup before topology queries and communication-channel creation. Integrate the operators into AFD's Ascend 950 build and private OPP package. Keep shared dependencies under megamoe_utils, prefix colliding headers, and build an isolated _C_megamoe extension. Expose the Python package under afd_plugin.compat.npu.megamoe and operators under torch.ops.afd_ascend. Resolve custom ACLNN symbols from AFD's private library, retain dependency licenses in distributions, and align connector/work-queue sequencing with the non-wrapping protocol. Validation: 77 targeted CPU tests passed, plus staged include/collision, CMake configuration, SDK parser/OPC adaptation, Torch schema/Meta and packaging checks. Device compilation and execution remain unverified because the local environment has no CANN toolkit or NPU.
Remove bundled Tensor API, Blaze, OpBase and MC2 common headers. Keep only the shared host compatibility, layer selection and sequence protocol headers in a flat megamoe_utils directory. Resolve installed CANN headers for the msopgen host/codegen and kernel OPC builds, update operator and native-extension includes, and report missing SDK components during configuration. Preserve CANN license attribution outside the utility directory and update package metadata. Validation: 84 focused CPU tests passed; generated-project CMake probes covered SDK include/pkg_inc layouts, include propagation, missing headers and builds without split MegaMoE. Real CANN/A5 compilation and runtime validation remain pending.
Scope this PR to standalone A5 MegaMoE and remove the legacy Async CAM A5 operator/runtime adaptations and their dedicated tests and docs. Keep the shared integration needed by MegaMoE and restore the existing CAM execution and validation paths and upstream source attribution. Move window layout, sequence handling, dispatch tickets, PUBLISHED polling and weight preparation into the MegaMoE connector modules. Remove compat/npu/megamoe and its redundant re-exports. Consolidate native extension loading and torch registrations in compat/npu/ops.py, explaining why the pybind-only MegaMoE bindings require registration there while CAM registers operators in C++. Preserve CANN attribution and license text and update interface documentation. Use AFD_MEGAMOE_LAYERED_GMM for MegaMoE host/device layer selection, read only by the FFN role, instead of reusing the Async CAM setting. Remove the redundant ubatch-only normalization from the Ascend platform patch; AFDNPUAttentionWorker.init_device already performs it.
Rename the connector to CAMAsyncMegaAFDConnector, the native extension to _C_afd_async_mega, and connector modules and configuration to mega names. Rename the three operators to afd_async_atten_send, afd_async_mega_ffn and afd_async_atten_recv, with Arch35-qualified CANN OpTypes and kernel directories. Keep public Python and ACLNN entry points without the suffix. Move the packaged vendor artifacts to _custom_ops and update build rules, imports, registration, tests and deployment documentation. Align shape inference and template registration with the Arch35 OpTypes used by OpDef, tiling and generated inner ACLNN entry points. Correct the ACLNN prefix check so missing private operators fail before fallback, and restrict Mega SDK staging to the three A5 operators. Add registration consistency checks and update operator registry expectations. Validation: local targeted tests reported 87 passed and 3 skipped. The user confirmed the changes run successfully on A5.
13583e3 to
3407263
Compare
72be4ae to
3407263
Compare
Keep dispatch activations and combine results on Attention ranks. Place compact routes, per-source task metadata and expert counts on FFN ranks. Size the FFN tables by the number of Attention sources and use the same offsets in all three operators when accessing peer windows. Host tiling checks now use the payload end for the local role.
Read the rank from the Attention and FFN union group before building the split layout. Compute the registered window size from the diagnostic and control areas plus the aligned sections owned by that role. Supply the same byte count to the communication context and the operator configuration so allocation agrees with the kernel offsets.
Have one AIV core scan consecutive Attention DP groups from a cursor in the local FFN window. Claim a group when all its ranks have published the expected generation, record the claim and advance the cursor. The FFN operator receives the group width instead of a host-selected rank list. Update the tiling, ACLNN, pybind and Torch interfaces together, and reserve space for the cursor and per-DP generations after the count table.
Submit the stage window and the full layer weight bank to the FFN operator. Agree on layer slots at startup and publish the slot with each Attention dispatch. Remove the Mega host work queue, publication-flag readback and optional host-selected layer path. For two stages, keep a separate window, stream and completion event per stage so either stage can accept work independently. Update the existing connector tests and deployment notes to match.
这部分代码来自算子给出的python extension接口,放到connector下更合适。窗口、布局、sequence、ticket 及三算子协议参数已移至 |
Restore shared Ascend vendor paths and public helpers, associated build and test references, and agent documentation. Restore upstream source paths and native MegaMoE terminology for A3 and other model paths. Keep the A5 Mega operator and connector names and their required callers unchanged.
| @@ -0,0 +1,38 @@ | |||
| CANN Open Software License Agreement Version 2.0 | |||
There was a problem hiding this comment.
这个license是需要放到这个目录下的吗?
| @@ -0,0 +1,82 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project | ||
| """Single-window CAM handoff through the native DBO receive hook.""" | ||
| """Receive ownership for shared and per-stage CAM windows.""" |
There was a problem hiding this comment.
这个文件的改动和megaconnector功能有关吗?
| layout: CAMDispatchLayout | ||
|
|
||
|
|
||
| def plain_tp_layout( |
There was a problem hiding this comment.
这个文件的改动和megaconnector功能有关吗?

Purpose
Implement the split async Mega operators and connector proposed in RFC #540, and support DeepSeek-V4-Flash on Ascend A5 with
CAMAsyncMegaAFDConnector, as proposed in RFC #541.The three Mega operators separate asynchronous Attention dispatch/combine from fused FFN receive, expert computation and result transmission. This preserves asynchronous A/F scheduling while reducing the FFN-side launch and data-movement overhead of separate communication and compute operators.
This also provides shared A5 infrastructure for RFC #537. V4.1 model adaptation is outside this PR.
Issue
Scope
CAMAsyncMegaAFDConnector, including bounded scheduling, sequence/acknowledgement handling, optional device layer selection, TP and two-ubatch DBO.Out of scope: V4.1 model adaptation, ModelRunnerV1, A5 graphs, MTP, EPLB, the
CAMAsyncAFDConnectorA5 adaptation and completed numerical/performance qualification.Supported Environment
CANN 9.2+ only, with the ops-transformer, MC2, OpBase, Blaze and Tensor API development headers available in the installed SDK. The tested environment uses CANN 9.2.1; compatibility with every later release is not established.
5df63a7203cdb9328d824a27b682e77c7b9928bef9c8e0bdb263d7a8c0270536f1ea00fa7373a84eImplementation Notes
Operators, naming and packaging. The three split operators live under
csrc/npu/ascend_kernels/afd_async_atten_send_arch35/,csrc/npu/ascend_kernels/afd_async_mega_ffn_arch35/andcsrc/npu/ascend_kernels/afd_async_atten_recv_arch35/(dispatch-send, fused-MoEand combine-receive, respectively). The
_arch35suffix marks the Atlas A5architecture and appears only on the operator directory / op-registration names;
the public function names (
aclnnAfdAsyncAttenSend, pybindafd_async_atten_send,torch.ops.afd_ascend.afd_async_atten_send, etc.) stay without the suffix.csrc/npu/pybind/afd_async_mega/exposes them through_C_afd_async_mega,torch.ops.afd_ascend.afd_async_*and AFD's privatelibcust_opapi.so; thewindow/sequence/ticket protocol lives under
afd_plugin/connectors/npu/(
mega_runtime.py,mega_layout.py,mega_work.py,mega_layers.py,mega_weights.py). No external UMDK wheel is required.SDK dependencies. 9f409c6 removes the bundled MC2, OpBase, Blaze and Tensor API dependency sources.
csrc/npu/ascend_kernels/cmake_files/cmake/mega_sdk.cmakeresolves the installed SDK headers and fails early with actionable errors when required components are missing. Explicit minimum-version enforcement remains a follow-up; the current checks validate SDK components.Scheduling and synchronization.
afd_plugin/connectors/npu/async_mega.pycoordinates TP-grouped submissions and bounded work admission. Metadata travels through Store descriptors; tensors use MTE windows. The sequence ring and PUBLISHED/TAKEN/READY/CONSUMED acknowledgements protect input and output reuse. DBO uses two communication windows/groups.Layer selection. Host-selected layers remain the default.
AFD_MEGA_LAYERED_GMM=1on every FFN worker enables device selection using agreed layer mappings and static TensorLists referencing the loaded weights.A5 precision and weights. Mega dispatch quantizes BF16 activations to MXFP8. FP4 C0=32 weights and group-32 E8M0 scales are prepared after loading.
Runtime integration. bd1e4d2 integrates A5 CAM execution with main's updated validation and FFN transaction flow. A5 BF16 reception permits absent receive scales; A3 retains its quantized path and graph tensor-lifetime handling. Mega remains eager-only and supports Attention DBO within the operator topology limits.
Source attribution. 7b99a20 preserves the original CANN notices and adds adaptation descriptions and history.
csrc/npu/ascend_kernels/SOURCE_NOTES.mdrecords provenance; the retained license iscsrc/npu/ascend_kernels/LICENSE-CANN.Test Plan
Build against the installed CANN SDK and the pinned Python environment:
Rebuild both the OPP package and Python extension after operator or protocol changes.
Run the host protocol and connector tests:
On A5, verify operator accuracy and DSV4 prefill/decode with
CAMAsyncMegaAFDConnector, including TP, uneven DP workloads, empty routes, sequence wraparound, host/device layer selection, DBO and repeated startup/shutdown. Measure performance and device overlap against comparable baselines.Test Result
A5 validation. A fresh A5 service startup test passed on the updated branch after switching to installed SDK dependencies. This establishes bring-up; it does not complete numerical, long-run or performance qualification.
Host regression. After merging main's validation/runtime changes, the affected suites passed 619 tests, with 4 skipped and 10 additional subtests passed. After the SDK dependency update in
9f409c6, the selected build/packaging suites passed 36 tests. These results cover different stages and are not presented as one final-head full-suite run.Remaining validation. Full operator accuracy, extended topology coverage, long-run protocol behavior, device overlap and performance comparisons remain pending. The acceptance criteria of #537, #540 and #541 are not yet fully established.
Follow-up
A5 support for
CAMAsyncAFDConnector. Kept out of this Mega-focused PR: a follow-up PR carries the existing connector's A5 adaptation — the A5 port of the fourafd_async_*communication operators, thedynamicQuant=0BF16 receive handling inasync_cam.py,AsyncCAMMXFPExecutor, and the matching configuration validation, tests and docs. It verifies the old CAM A5 BF16/MXFP4 path and A3 regression separately from this PR.Operator tests and CANN version enforcement. Add a standalone three-operator precision/performance test that runs on A5 without launching a full DeepSeek-V4 service, plus an explicit CANN minimum-version check and consistent CANN 9.2+ only wording across the A5 documentation.
Docs Impact
csrc/npu/README.md: build instructions and installed SDK header requirements.csrc/npu/ascend_kernels/SOURCE_NOTES.md: source attribution and license location.docs/npu/A5_DSV4_ASYNC.md: environment pins, execution paths and integration notes.docs/npu/MEGA_EAGER.md: Mega setup and execution.docs/design/module/ffn_runtime.md: FFN runtime integration and test ownership.Essential PR Checklist