Skip to content

[NPU] Add async MegaMoE and A5 DeepSeek-V4-Flash integration - #603

Open
Nova-xiao wants to merge 14 commits into
vllm-project:mainfrom
Nova-xiao:feat/a5-async-with-megamoe
Open

Nova-xiao wants to merge 14 commits into
vllm-project:mainfrom
Nova-xiao:feat/a5-async-with-megamoe

Conversation

@Nova-xiao

@Nova-xiao Nova-xiao commented Nov 17, 2026 •

Copy link
Copy Markdown

Purpose

Implement the split async Mega operators and connector proposed in RFC #540, and support DeepSeek-V4-Flash on Ascend A5 with CAMAsyncMegaAFDConnector, as proposed in RFC #541.

The three Mega operators separate asynchronous Attention dispatch/combine from fused FFN receive, expert computation and result transmission. This preserves asynchronous A/F scheduling while reducing the FFN-side launch and data-movement overhead of separate communication and compute operators.

This also provides shared A5 infrastructure for RFC #537. V4.1 model adaptation is outside this PR.

Issue

Scope

  • Three split Mega operators, native bindings and private Ascend950 OPP packaging.
  • CAMAsyncMegaAFDConnector, including bounded scheduling, sequence/acknowledgement handling, optional device layer selection, TP and two-ubatch DBO.
  • DeepSeek-V4-Flash model and weight adaptation for A5.
  • Integration with main's configuration validation and async FFN runtime.
  • Tests, build instructions and source attribution.

Out of scope: V4.1 model adaptation, ModelRunnerV1, A5 graphs, MTP, EPLB, the CAMAsyncAFDConnector A5 adaptation and completed numerical/performance qualification.

Supported Environment

CANN 9.2+ only, with the ops-transformer, MC2, OpBase, Blaze and Tensor API development headers available in the installed SDK. The tested environment uses CANN 9.2.1; compatibility with every later release is not established.

Component Target
Hardware Ascend A5
Model DeepSeek-V4-Flash with native FP4 routed experts
Execution Eager ModelRunnerV2, static EP, Attention-side routing
vLLM v0.32.0, 5df63a7203cdb9328d824a27b682e77c7b9928be
vLLM-Ascend f9c8e0bdb263d7a8c0270536f1ea00fa7373a84e
Tested CANN 9.2.1

Implementation Notes

Operators, naming and packaging. The three split operators live under
csrc/npu/ascend_kernels/afd_async_atten_send_arch35/,
csrc/npu/ascend_kernels/afd_async_mega_ffn_arch35/ and
csrc/npu/ascend_kernels/afd_async_atten_recv_arch35/ (dispatch-send, fused-MoE
and combine-receive, respectively). The _arch35 suffix marks the Atlas A5
architecture and appears only on the operator directory / op-registration names;
the public function names (aclnnAfdAsyncAttenSend, pybind afd_async_atten_send,
torch.ops.afd_ascend.afd_async_atten_send, etc.) stay without the suffix.
csrc/npu/pybind/afd_async_mega/ exposes them through _C_afd_async_mega,
torch.ops.afd_ascend.afd_async_* and AFD's private libcust_opapi.so; the
window/sequence/ticket protocol lives under afd_plugin/connectors/npu/
(mega_runtime.py, mega_layout.py, mega_work.py, mega_layers.py,
mega_weights.py). No external UMDK wheel is required.

SDK dependencies. 9f409c6 removes the bundled MC2, OpBase, Blaze and Tensor API dependency sources. csrc/npu/ascend_kernels/cmake_files/cmake/mega_sdk.cmake resolves the installed SDK headers and fails early with actionable errors when required components are missing. Explicit minimum-version enforcement remains a follow-up; the current checks validate SDK components.

Scheduling and synchronization. afd_plugin/connectors/npu/async_mega.py coordinates TP-grouped submissions and bounded work admission. Metadata travels through Store descriptors; tensors use MTE windows. The sequence ring and PUBLISHED/TAKEN/READY/CONSUMED acknowledgements protect input and output reuse. DBO uses two communication windows/groups.

Layer selection. Host-selected layers remain the default. AFD_MEGA_LAYERED_GMM=1 on every FFN worker enables device selection using agreed layer mappings and static TensorLists referencing the loaded weights.

A5 precision and weights. Mega dispatch quantizes BF16 activations to MXFP8. FP4 C0=32 weights and group-32 E8M0 scales are prepared after loading.

Runtime integration. bd1e4d2 integrates A5 CAM execution with main's updated validation and FFN transaction flow. A5 BF16 reception permits absent receive scales; A3 retains its quantized path and graph tensor-lifetime handling. Mega remains eager-only and supports Attention DBO within the operator topology limits.

Source attribution. 7b99a20 preserves the original CANN notices and adds adaptation descriptions and history. csrc/npu/ascend_kernels/SOURCE_NOTES.md records provenance; the retained license is csrc/npu/ascend_kernels/LICENSE-CANN.

Test Plan

Build against the installed CANN SDK and the pinned Python environment:

export ASCEND_CANN_PACKAGE_PATH="$ASCEND_HOME_PATH"

AFD_BUILD_ASCEND_OPS=1 SOC_VERSION=950 \
pip install -e . -v --no-build-isolation

Rebuild both the OPP package and Python extension after operator or protocol changes.

Run the host protocol and connector tests:

python -m pytest -q \
  tests/unit/connectors/npu/test_mega_work.py \
  tests/unit/connectors/npu/test_mega_admission.py \
  tests/unit/connectors/npu/test_mega_sequence_ring.py \
  tests/unit/connectors/npu/test_mega_layers.py \
  tests/unit/connectors/npu/test_mega_topology.py

On A5, verify operator accuracy and DSV4 prefill/decode with CAMAsyncMegaAFDConnector, including TP, uneven DP workloads, empty routes, sequence wraparound, host/device layer selection, DBO and repeated startup/shutdown. Measure performance and device overlap against comparable baselines.

Test Result

A5 validation. A fresh A5 service startup test passed on the updated branch after switching to installed SDK dependencies. This establishes bring-up; it does not complete numerical, long-run or performance qualification.

Host regression. After merging main's validation/runtime changes, the affected suites passed 619 tests, with 4 skipped and 10 additional subtests passed. After the SDK dependency update in 9f409c6, the selected build/packaging suites passed 36 tests. These results cover different stages and are not presented as one final-head full-suite run.

Remaining validation. Full operator accuracy, extended topology coverage, long-run protocol behavior, device overlap and performance comparisons remain pending. The acceptance criteria of #537, #540 and #541 are not yet fully established.

Follow-up

A5 support for CAMAsyncAFDConnector. Kept out of this Mega-focused PR: a follow-up PR carries the existing connector's A5 adaptation — the A5 port of the four afd_async_* communication operators, the dynamicQuant=0 BF16 receive handling in async_cam.py, AsyncCAMMXFPExecutor, and the matching configuration validation, tests and docs. It verifies the old CAM A5 BF16/MXFP4 path and A3 regression separately from this PR.

Operator tests and CANN version enforcement. Add a standalone three-operator precision/performance test that runs on A5 without launching a full DeepSeek-V4 service, plus an explicit CANN minimum-version check and consistent CANN 9.2+ only wording across the A5 documentation.

Docs Impact

  • csrc/npu/README.md: build instructions and installed SDK header requirements.
  • csrc/npu/ascend_kernels/SOURCE_NOTES.md: source attribution and license location.
  • docs/npu/A5_DSV4_ASYNC.md: environment pins, execution paths and integration notes.
  • docs/npu/MEGA_EAGER.md: Mega setup and execution.
  • docs/design/module/ffn_runtime.md: FFN runtime integration and test ownership.

Essential PR Checklist
  • Purpose is clear and linked to public context.
  • Scope is bounded.
  • Compatibility with vLLM v0.32.0 and the pinned vLLM-Ascend source is considered.
  • No changes are made to the vLLM source checkout.
  • Plugin-owned classes or explicit dotted class paths are preferred over monkey patches.
  • Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested.
  • Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated.
  • Validation evidence and pending device qualification are stated.
  • Documentation impact is stated.

@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes labels Nov 17, 2026
@Nova-xiao
Nova-xiao force-pushed the feat/a5-async-with-megamoe branch from 226e748 to b478f55 Compare November 17, 2026 07:26
@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

移除CANN的依赖 并且标明只支持CANN 9.2+

@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

Thanks for the large integration — two requests before a detailed review, both about the vendored CANN sources.

1. Please drop the bundled CANN dependency from the diff. As it stands this PR is 1073 files / ~175k added lines, and about 134k of those (around 860 .h files) are csrc/npu/ascend_kernels/megamoe_utils/ops-tensor/**, which carries its own LICENSE and is a copy of an upstream CANN/Ascend toolkit rather than code owned by this repo (csrc/npu/ascend_kernels/megamoe_utils/LICENSE, .../opbase/LICENSE, .../ops-tensor/LICENSE, afd_plugin/compat/npu/megamoe/LICENSE). The plugin-side change is only ~1.4k lines across 33 files under afd_plugin/, plus ~770 lines of tests and ~310 lines of docs. Vendoring the toolkit makes the PR effectively unreviewable, pins us to one snapshot for security and upstream fixes, and adds third-party licensing/attribution obligations to this tree. Please consume these headers from the installed CANN toolkit (e.g. resolve them from ASCEND_HOME_PATH/the toolkit's include dirs at build time in setup.py / the CMake files under csrc/npu/ascend_kernels/cmake_files/) and fail the build with a clear message when the toolkit is missing or too old, instead of committing them here.

2. Please state the supported CANN version explicitly: CANN 9.2+ only. The PR body currently mentions CANN 9.2.1 only as part of the pinned test matrix. If these kernels require 9.2 or newer, make that a documented, enforced requirement — in docs/npu/* and the install/build instructions, and as a build-time/runtime check next to the existing A5 feature validation (afd_plugin/compat/npu/feature_validation.py) — so that an older toolkit fails fast with an actionable error rather than at kernel load.

Once the vendored tree is out, the remaining diff is small enough for a proper review of the connector/runtime logic, and @jiangkuaixue123 will take a look.

Scope checked: file-level diff statistics and directory/licence layout at head b478f5540466944e64d3ea0e058f52f80e002ebd; PR body and linked RFCs #540/#541/#537. Not checked yet: kernel correctness, connector/runtime logic, tests, or any build/run on NPU hardware.

@Nova-xiao
Nova-xiao force-pushed the feat/a5-async-with-megamoe branch from 53be9ab to f3f24a1 Compare November 18, 2026 09:34
@Nova-xiao
Nova-xiao requested a review from ronenkat as a code owner November 18, 2026 09:34
@Nova-xiao
Nova-xiao force-pushed the feat/a5-async-with-megamoe branch from 20cc78e to f3f24a1 Compare November 18, 2026 09:59
@@ -0,0 +1,3 @@
# ### PATCH START: AFD standalone MegaMoE extension
from afd_plugin._C_megamoe import CommContextManager

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个文件单独保留是为了兼容某个已有的导入路径吗?仓库里只有 megamoe_split.py 从这里导入 CommContextManager,而它已经直接从 afd_plugin._C_megamoe 导入了 SPLIT_SEQUENCE_MAX。如果没有兼容需求,能否把这两个符号放在同一条 import 里,然后删掉 comm_context.py?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已移除comm_context.py

@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

想确认一下 afd_plugin/compat/npu/megamoe/common/ 的来源和维护边界。megamoe_split.py 管理通信窗口、序列号和 dispatch ticket,megamoe_split_layout.py 定义与 kernel 对齐的窗口布局。能否说明一下:

  1. 这套代码来自哪个上游仓库、哪个 tag 或 commit?如果是从上游移植的,能否给出对应的源文件路径,并标明哪些部分是 AFD 新增的?
  2. 相对该版本,AFD 的 patch 具体改了哪些行为,为什么需要这些改动?例如,comm_context.py 标记了从 afd_plugin._C_megamoe 导入 CommContextManager,但从标记里看不出原来的导入路径,也看不出是否需要保留这层转导出。
  3. 把整个 common/ 放在 compat/npu 下,是为了跟随上游版本同步,还是需要为其他调用方提供稳定接口?如果窗口布局、序列号和 ticket 属于 AFD 自己维护的传输协议,是否更适合放到 MegaMoE connector 下面,让 compat/npu 只负责原生扩展的加载和适配?


# Upstream: vllm_ascend/ascend_config.py at
# bd42eea01d8312c4df469fabaed45f77a12df4d3.
# f9c8e0bdb263d7a8c0270536f1ea00fa7373a84e.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont modify this

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已恢复

Comment thread afd_plugin/envs.py

import os

AFD_ASYNC_CAM_LAYERED_GMM = "AFD_ASYNC_CAM_LAYERED_GMM"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already moved. Please remove

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已删除对 AFD_ASYNC_CAM_LAYERED_GMM 的 Mega 语义改写,该变量仍只控制 Async CAM layered GMM。

@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

请在后续 PR 中补充一个可在 A5 上独立运行的 MegaMoE 三算子测试程序,无需启动完整的 DeepSeek-V4 服务即可验证 megamoe_dispatch_send → megamoe_fused_moe → megamoe_combine_recv。希望这个程序能覆盖两项:

  • 精度:固定输入、路由和专家权重,运行完整三算子链路,将最终的加权输出与参考实现对比,给出误差和通过阈值。
  • 性能:在相同输入和拓扑下预热并重复采样,分别报告三个算子的耗时及整条链路的延迟,注明计时范围,便于区分设备执行与跨 rank 等待。

请一并提供 A5 上可直接复现的启动命令、所需卡数和 A/F rank 布局,方便后续算子改动时做回归对比。

"""Check additional NPU features after common Worker deployment admission."""

afd_config = afd_config or parse_afd_config(vllm_config)
layered_gmm_requested = async_cam_layered_gmm_enabled()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#538 在 Async CAM FFN 统一使用 layered GMM 后删除了 AFD_ASYNC_CAM_LAYERED_GMM。这里复用旧变量名来控制 A5 MegaMoE 的 host/device 选层,又在公共入口检查它。如果部署脚本仍给所有角色导出旧变量 =1,原先可以启动的 Attention worker 现在会在这里报错。能否给 MegaMoE 选层使用独立的配置名,并仅在 MegaMoE FFN 读取和校验,让已有部署继续忽略旧变量?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已引入独立配置 AFD_MEGA_LAYERED_GMM,仅在 Mega FFN 读取。

return any("DeepseekV4" in str(architecture) for architecture in architectures)


def _fail_if_unsupported_dsv4_async_features(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#600 已经把这个 helper 的检查归并到对应入口。compute_gate_on_attention 在 validation.py 的 NPU 准入中已检查,AFDAsyncExtraInfo 的类型在上一层刚检查过,而且入口已经拒绝非 DSV4 的 Async CAM,所以调用处的 if is_dsv4 恒为真。能否沿用 #600 的分工,只在现有 Async CAM 校验中补上 A5 的 FP4、eager 等约束,并按平台区分 dynamicQuant(A5 为 0,非 A5 保留 1)?这样以后改准入条件只需维护一处。

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已移除重复的 DSV4 Async CAM 校验 helper;Mega 的 A5/FP4/eager/geometry 约束由独立 validate_mega_config 保留。CAMAsyncAFDConnector 的 A5 约束整理随后续 PR。

parallel_config.enable_dbo = saved.enable_dbo
parallel_config.ubatch_size = saved.ubatch_size
parallel_config.all2all_backend = saved.all2all_backend
if saved.ubatch_size > 1 and not saved.enable_dbo:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need this.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ascend_platform.py 里的 --ubatch-size 归一化分支是调试时加上的,已删除。Mega 复用 configure_async_cam_dbo 接口做 DBO 归一化,该接口内部现在按 connector 区分,不影响 CAMAsyncAFDConnector。

@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

建议把这次改动按两条可独立验收的执行路径拆成两个 PR:

  • 当前 PR 聚焦 A5 MegaMoE:保留 CAMAsyncMegamoeAFDConnector、三个 split 算子、原生绑定,以及这条路径必需的 DSV4 接入和测试。建议把 MegaMoE 专属的权重转换和窗口策略放在新 connector 一侧,让这个 PR 可以独立构建、运行和验证。
  • 后续 PR 单独支持已有的 CAMAsyncAFDConnector 在 A5 上运行:请把旧 CAM 四个 afd_async_* 通信算子的 A5 适配、async_cam.py 中 dynamicQuant=0 的 BF16 接收处理、AsyncCAMMXFPExecutor,以及对应的配置校验、测试和文档移到后续 PR。

拆分时可以按调用链判断归属:如果某段 TP 布局或收发逻辑是 MegaMoE 独立运行所必需的,可以保留最小的共享部分,同时保持原有 Async CAM 的行为。这样当前 PR 可以集中审查三算子的协议、精度和性能,后续 PR 则可以单独验证旧 CAM 的 A5 BF16/MXFP4 路径和 A3 回归。

Nova-xiao and others added 9 commits November 20, 2026 14:22
Extend the DeepSeek-V4-Flash `CAMAsyncAFDConnector` path to Ascend A5 and
integrate `CAMAsyncMegamoeAFDConnector` with the async-megamoe operators, per
Attention-side routing and local shared-expert execution, and adds the
MXFP4/MXFP8 expert path plus A5 support for the bundled async CAM operators.
Shared A5 connector and FFN execution code is reused by the DeepSeek-V4.1 Flash
work in vllm-project#537.

- Related issue(s): vllm-project#541, vllm-project#540, vllm-project#537
- Closes #

- In scope:
  - A5 port of the four bundled `afd_async_dispatch_send/recv` and
    `afd_async_combine_send/recv` operators (the `cam_moe_distribute_*` suite):
    A5 HCCL window and resource access, AIV communication-engine selection,
    `ascend950` registration and build selection, Python bindings.
  - A5 DSV4 adaptation: MXFP4 expert weights in `FRACTAL_NZ_C0_32` with
    group-32 E8M0 scales, BF16 transfer for the MXFP4 expert path, and the
    matching quantization and weight-layout checks.
  - `CAMAsyncMegamoeAFDConnector` over the UMDK-provided
    `cam_megamoe_dispatch_send`, `cam_megamoe_fused_moe` and
    `cam_megamoe_combine_recv`: one shared communication context with input,
    output and control windows, wrap-safe sequence matching, device-side
    readiness and consumption signals, and FFN scheduling that selects the
    layer's weights and scales from `layer_idx`.
  - FFN-side scheduler with bounded work descriptors per (Attention DP, FFN
    rank), admission only after every source publishes, fused FFN execution
    driven by `execute_next_experts`, and two in-flight ubatches on isolated
    windows.
  - Topology and validation coverage: asymmetric A/F counts within tiling
    limits, DP-grouped submissions for Attention TP > 1, local token-row
    routing on plain Attention TP, and DBO normalization for async Attention.
  - Docs and unit tests.
- Out of scope:
  - A3 W4A8/layered execution and ACL graph capture on A5.
  - ModelRunnerV1, performance qualification and tuning.

- The three async-megamoe operators are consumed from the UMDK CAM wheel
  (`umdk_cam_megamoe`) in this change. Their sources and the in-tree build
  integration follow in a separate commit. The four `afd_async_*` operators are
  built from this repository's `csrc/npu` sources.
- Attention keeps routing and shared experts. Its send/receive calls map to
  `cam_megamoe_dispatch_send` and `cam_megamoe_combine_recv`, and the matching
  dispatch state is retained until combine is submitted.
- FFN exposes only `execute_next_experts`. Work descriptors travel through the
  Store while tensor data stays in the UMDK window, PUBLISHED/TAKEN/READY/
  CONSUMED handling protects window reuse, and dual-stage admission is limited
  to FFN workers with two ubatches.
- The MXFP4 weight adapter runs once after load: transpose, NZ re-pack into the
  operator's C0=32 container, and keep the group-32 E8M0 scales alongside.
- A5 gating requires native FP4 experts, `dynamicQuant=0`, eager MRV2, static
  EP, `enable_fused_mc2=0` and layered GMM off.
- Restore the DBO/ubatching fields that Ascend platform validation clears, and
  treat `--ubatch-size > 1` as a DBO request for async Attention instead of
  silently dropping the intent.

- A5 operators: build the four `afd_async_*` operators for `ascend950`, then
  run `tests/npu/async_cam_precision.py` (dispatch ordering and weighted
  combine) and `tests/npu/async_cam_performance.py --measure-mode cycle` on
  2A2F.
- A5 E2E: 2A2F eager DeepSeek-V4-Flash with each connector, prefill plus
  generation, repeated startup/shutdown and uneven Attention DP traffic.
- Numerics, same checkpoint and routing as the native A5 run:
  - `cam_megamoe_fused_moe` expert output against native A5 MoE for the same
    layer, weights and `group_list`.
  - `cam_megamoe_combine_recv` result against the reference top-k weighted sum
    of those per-expert outputs.
  - `cam_megamoe_dispatch_send` routing counts and token placement against the
    CPU reference used by `tests/npu/async_cam_precision.py`.
  - Whole-model output against native A5 inference.

- A5 2A2F eager bring-up with `CAMAsyncAFDConnector` succeeds; 16-concurrent
  decode TPOT ~84 ms and single-request TPOT ~85 ms at
  `--max-num-batched-tokens 4096`. Functional check only.
- `CAMAsyncMegamoeAFDConnector` end-to-end numbers are not collected yet.

- Files updated: `docs/npu/A5_DSV4_ASYNC.md`,
  `docs/npu/A5_ASYNC_CAM_VALIDATION.md`, `docs/npu/MEGAMOE_EAGER.md`,
  `README.md`.
Add Ascend 950 dispatch-send, fused-MoE and combine-recv operators with
shared MTE communication buffers, tiling, native bindings and Python APIs.
Support FP8 activations with FP8/FP4 expert weights and carry layer_index
with each DP request to select W1/W2 and scales from multilayer weight
banks, using stacked tensors or layer-major per-expert TensorLists.

Register weight dtype/format pairs consistently with tiling. Extend a
build-local copy of the SDK format whitelist for FRACTAL_NZ_C0_32 while
preserving SDK validation and installed files. Synchronize the HCCL warmup
before topology queries and communication-channel creation.

Integrate the operators into AFD's Ascend 950 build and private OPP package.
Keep shared dependencies under megamoe_utils, prefix colliding headers,
and build an isolated _C_megamoe extension. Expose the Python package under
afd_plugin.compat.npu.megamoe and operators under torch.ops.afd_ascend.
Resolve custom ACLNN symbols from AFD's private library, retain dependency
licenses in distributions, and align connector/work-queue sequencing with
the non-wrapping protocol.

Validation: 77 targeted CPU tests passed, plus staged include/collision,
CMake configuration, SDK parser/OPC adaptation, Torch schema/Meta and
packaging checks. Device compilation and execution remain unverified
because the local environment has no CANN toolkit or NPU.
Remove bundled Tensor API, Blaze, OpBase and MC2 common headers. Keep only
the shared host compatibility, layer selection and sequence protocol
headers in a flat megamoe_utils directory.

Resolve installed CANN headers for the msopgen host/codegen and kernel OPC
builds, update operator and native-extension includes, and report missing
SDK components during configuration. Preserve CANN license attribution
outside the utility directory and update package metadata.

Validation: 84 focused CPU tests passed; generated-project CMake probes
covered SDK include/pkg_inc layouts, include propagation, missing headers
and builds without split MegaMoE. Real CANN/A5 compilation and runtime
validation remain pending.
Scope this PR to standalone A5 MegaMoE and remove the legacy Async CAM
A5 operator/runtime adaptations and their dedicated tests and docs.
Keep the shared integration needed by MegaMoE and restore the existing
CAM execution and validation paths and upstream source attribution.

Move window layout, sequence handling, dispatch tickets, PUBLISHED
polling and weight preparation into the MegaMoE connector modules.
Remove compat/npu/megamoe and its redundant re-exports. Consolidate
native extension loading and torch registrations in compat/npu/ops.py,
explaining why the pybind-only MegaMoE bindings require registration
there while CAM registers operators in C++. Preserve CANN attribution
and license text and update interface documentation.

Use AFD_MEGAMOE_LAYERED_GMM for MegaMoE host/device layer selection,
read only by the FFN role, instead of reusing the Async CAM setting.

Remove the redundant ubatch-only normalization from the Ascend platform
patch; AFDNPUAttentionWorker.init_device already performs it.
Rename the connector to CAMAsyncMegaAFDConnector, the native extension to
_C_afd_async_mega, and connector modules and configuration to mega names.
Rename the three operators to afd_async_atten_send, afd_async_mega_ffn
and afd_async_atten_recv, with Arch35-qualified CANN OpTypes and kernel
directories. Keep public Python and ACLNN entry points without the suffix.
Move the packaged vendor artifacts to _custom_ops and update build rules,
imports, registration, tests and deployment documentation.

Align shape inference and template registration with the Arch35 OpTypes
used by OpDef, tiling and generated inner ACLNN entry points. Correct the
ACLNN prefix check so missing private operators fail before fallback,
and restrict Mega SDK staging to the three A5 operators. Add registration
consistency checks and update operator registry expectations.

Validation: local targeted tests reported 87 passed and 3 skipped.
The user confirmed the changes run successfully on A5.
@Nova-xiao
Nova-xiao force-pushed the feat/a5-async-with-megamoe branch from 13583e3 to 3407263 Compare November 20, 2026 06:33
@lqy16
lqy16 force-pushed the feat/a5-async-with-megamoe branch from 72be4ae to 3407263 Compare November 20, 2026 06:47
Keep dispatch activations and combine results on Attention ranks. Place compact routes, per-source task metadata and expert counts on FFN ranks. Size the FFN tables by the number of Attention sources and use the same offsets in all three operators when accessing peer windows. Host tiling checks now use the payload end for the local role.
Read the rank from the Attention and FFN union group before building the split layout. Compute the registered window size from the diagnostic and control areas plus the aligned sections owned by that role. Supply the same byte count to the communication context and the operator configuration so allocation agrees with the kernel offsets.
Have one AIV core scan consecutive Attention DP groups from a cursor in the local FFN window. Claim a group when all its ranks have published the expected generation, record the claim and advance the cursor. The FFN operator receives the group width instead of a host-selected rank list. Update the tiling, ACLNN, pybind and Torch interfaces together, and reserve space for the cursor and per-DP generations after the count table.
Submit the stage window and the full layer weight bank to the FFN operator. Agree on layer slots at startup and publish the slot with each Attention dispatch. Remove the Mega host work queue, publication-flag readback and optional host-selected layer path. For two stages, keep a separate window, stream and completion event per stage so either stage can accept work independently. Update the existing connector tests and deployment notes to match.
@Nova-xiao

Copy link
Copy Markdown
Author

想确认一下 afd_plugin/compat/npu/megamoe/common/ 的来源和维护边界。megamoe_split.py 管理通信窗口、序列号和 dispatch ticket,megamoe_split_layout.py 定义与 kernel 对齐的窗口布局。能否说明一下:

  1. 这套代码来自哪个上游仓库、哪个 tag 或 commit?如果是从上游移植的,能否给出对应的源文件路径,并标明哪些部分是 AFD 新增的?
  2. 相对该版本,AFD 的 patch 具体改了哪些行为,为什么需要这些改动?例如,comm_context.py 标记了从 afd_plugin._C_megamoe 导入 CommContextManager,但从标记里看不出原来的导入路径,也看不出是否需要保留这层转导出。
  3. 把整个 common/ 放在 compat/npu 下,是为了跟随上游版本同步,还是需要为其他调用方提供稳定接口?如果窗口布局、序列号和 ticket 属于 AFD 自己维护的传输协议,是否更适合放到 MegaMoE connector 下面,让 compat/npu 只负责原生扩展的加载和适配?

这部分代码来自算子给出的python extension接口,放到connector下更合适。窗口、布局、sequence、ticket 及三算子协议参数已移至 connectors/npu/megamoe_runtime.py、megamoe_layout.py。整个 compat/npu/megamoe/ 目录已删除,算子加载/注册合入 compat/npu/ops.py,该处补注册 schema/Meta/impl,是因为沿用的算子绑定只通过 pybind 导出函数、不生成 torch.ops 入口。

Restore shared Ascend vendor paths and public helpers, associated build and test references, and agent documentation. Restore upstream source paths and native MegaMoE terminology for A3 and other model paths. Keep the A5 Mega operator and connector names and their required callers unchanged.
@@ -0,0 +1,38 @@
CANN Open Software License Agreement Version 2.0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个license是需要放到这个目录下的吗?

@@ -0,0 +1,82 @@
# SPDX-License-Identifier: Apache-2.0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image 建议这几个文件统一挪到
afd_plugin/connectors/npu/async_mega/ 下面
(async_mega这个名字可以调整)

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project
"""Single-window CAM handoff through the native DBO receive hook."""
"""Receive ownership for shared and per-stage CAM windows."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个文件的改动和megaconnector功能有关吗?

layout: CAMDispatchLayout


def plain_tp_layout(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个文件的改动和megaconnector功能有关吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants