Repository navigation
Description
This RFC proposes an optional Attention-side router/shared-expert placement for synchronous A5 AFD, together with MXFP8 activation pass-through to the FFN role.
Collaborators: @ksiyuan, @wangyao-i, @cqq70605.
1. Motivation
AFD allows Attention (A) and FFN (F) resources to be provisioned independently. An important deployment pattern is to have multiple Attention instances feed a smaller shared FFN pool. Aggregating tokens from the Attention DP replicas gives FFN workers larger expert-computation batches, creating an opportunity to improve model FLOPs utilization (MFU) and amortize expert-weight reads over more tokens, while Attention instances retain distributed request and KV-cache capacity.
Realizing this benefit requires delivering the aggregated activations from A to F. As more Attention instances contribute tokens, A2E activation traffic grows. If A and F are placed on different hosts, that traffic also crosses the inter-node fabric. Both token aggregation and cross-host placement can put pressure on activation delivery; when available bandwidth limits execution, transfer time can reduce the benefit of larger FFN batches. Reducing A2E bytes is therefore particularly valuable for bandwidth-constrained, cross-host A/F deployments.
This RFC proposes MXFP8 activation transport and direct consumption on Ascend 950 (A5), reducing A2E activation-data bytes relative to BF16 while carrying the associated E8M0 scale. The initial implementation targets the synchronous CAMP2pAFDConnector. To preserve the transport saving through the FFN input path, it also moves router and shared-expert execution to A, where model-dtype activations are already available. F receives the quantized activation, scale, and top-k, and runs only the routed experts, removing the need to recover BF16 activations on F for router/shared-expert computation.
The initial measurements cover single-host 4A4F. Higher A:F ratios and cross-host bandwidth constraints motivate the design and the scaling evaluation described in section 5.6; they are not yet measured deployment results.
2. Proposed execution
flowchart LR
H["A: attention output h"] --> G["A: router / top-k"]
H --> S["A: shared experts"]
H --> Q["A: optional MXFP8 quantization"]
G --> T["A2E: activation + top-k + optional MX scale"]
Q --> T
T --> F["F: align and run routed experts"]
F --> R["E2A: routed output"]
R --> ADD["A: add outputs"]
S --> ADD
This role split follows the existing AFD asynchronous CAM implementation, adapting Attention-side routing and shared-expert execution to the synchronous CAMP2P path.
The goal is to reduce A2E activation traffic on Ascend 950 (A5) while allowing F to consume the transported MXFP8 activation directly. Keeping router and shared-expert computation on F requires model-dtype activations for those operations. Recovering BF16 activations adds device work and memory traffic, with larger token batches requiring more data to be processed. Moving both operations to A removes that recovery requirement and allows F to consume the received FP8 activation and scale directly for routed-expert computation.
The proposal therefore moves both the router and shared experts to A, where the model-dtype activation is already available. A computes top-k before transport quantization and computes the shared-expert output from that model-dtype input. With MXFP8 enabled, it sends the FP8 activation, E8M0 scale, and top-k IDs/weights to F, then adds the returned routed output to its shared-expert output. BF16 external routing uses the same role split without transport quantization.
F consumes the received expert IDs and weights and runs only the routed experts. The model helper preserves the target model's routing and scaling semantics: top-k weights already include model-level routed scaling, so F does not apply that factor again. Finalize returns the routed result in the original token order, with communication-specific reduction handled by the MoE implementation.
Router and shared-expert placement share one opt-in switch because both must move to enable direct consumption. Moving only the router would leave the shared experts needing a BF16 activation on F and retain the activation-recovery step.
With external routing enabled, MoE weight placement and execution are:
| Component | A weights | F weights | Computation |
|---|---|---|---|
| Router | Loaded | Retained for the native MoE structure | A computes top-k; F does not run its router |
| Shared experts | Loaded when present in the model | Not constructed or loaded | A computes the shared output |
| Routed experts | Not loaded | Loaded and expert-parallel sharded | F computes the routed output |
This table covers MoE weights, not the full checkpoint. Moving router execution does not mean removing every router parameter from F: the current native MoE still constructs and loads its copy. The proposal avoids duplicating shared and routed experts across roles. Device-memory impact remains to be measured. With the option disabled, the original router/shared/routed computation stays on F.
3. Transfer and consumption contract
3.1 Payload and configuration
Let M be the receiving F rank's local row count before MoE prepare, H the hidden width, K the model's top-k, and C = ceil(H / 32).
| Payload field | BF16 external routing | MXFP8 pass-through |
|---|---|---|
hidden_states |
BF16 [M, H] |
E4M3 FP8 [M, H] |
topk_ids |
int32 [M, K] |
int32 [M, K] |
topk_weights |
float32 [M, K] |
float32 [M, K] |
activation_scale |
absent | E8M0 [M, C] |
These fields describe the same rows on the same device. Expert IDs and weights arrive together. A quantized activation and its activation scale also arrive together; a missing half, wrong format, or mismatched shape/device is an error. Validation checks metadata without reading tensor contents back to the host.
The existing A2E IDs/weights slots carry either token IDs for the original router-on-F protocol or expert IDs/weights for external routing. They do not carry both meanings in one transfer. The wire slot named scales carries routing weights; it is distinct from the E8M0 activation scale.
The payload maps to the A2E operator as follows:
| Logical field | A producer | A2E input and corresponding received output | F consumer |
|---|---|---|---|
hidden_states |
Model-dtype activation, optionally quantized | x → received activation |
Prepare, then routed-expert computation |
topk_ids |
Router's expert selection | expert_ids → received expert IDs |
Prepare and expert selection |
topk_weights |
Router weights including routed scaling | scales → received routing weights |
Prepare and weighted expert computation |
activation_scale |
MXFP8 quantization of that activation | x_scale → received MX scale |
Prepare's pertoken_scale, then the path-specific expert consumer |
The F connector constructs the payload from these received tensors; the runner forwards the fields under the same names. The scale then follows section 3.2's alignment and layout rules. In particular, A2E's scales means routing weights, whereas MC2 dispatch's scales means activation scale. These operator parameters share a name but do not carry interchangeable data.
The proposed configuration matrix is:
afd.compute_gate_on_attention |
afd.connector_extra_config.quant_mode |
Meaning |
|---|---|---|
false or omitted |
0 or omitted |
Original router-on-F path |
true |
0 |
BF16 external routing |
true |
4 |
MXFP8 external routing and pass-through |
false |
4 |
Outside this proposal; reject rather than silently dequantize |
The default stays unchanged. Both roles use matching gate and transport modes. The gate switch belongs in the common afd block, not connector_extra_config. For example, with the role-specific fields supplied separately:
{
"afd": {
"role": "attention",
"connector": "CAMP2pAFDConnector",
"host": "127.0.0.1",
"port": 6239,
"num_attention_ranks": 4,
"num_ffn_ranks": 4,
"compute_gate_on_attention": true,
"connector_extra_config": {"quant_mode": 4}
}
}F uses the same settings with role: "ffn". This is a configuration fragment, not a complete launch recipe.
Gate placement and transport mode select the transfer's meaning, not just a local optimization. A and F need matching gate/transport modes, activation and scale formats, model dimensions, and a consistent rank topology. A mismatch is invalid; it is not a request to reinterpret the payload or fall back to another mode. Local shape/dtype checks cannot establish agreement with the peer.
Current code validates local settings but does not negotiate these fields across roles. Deployment therefore requires a consistent configuration for both roles, checked before sending traffic. This RFC defines that compatibility requirement without introducing a new version-negotiation protocol. A future startup check could use the existing control plane to reject disagreement before data transfer.
3.2 Alignment and scale layout
Transfer scale is always two-dimensional. Prepare and dispatch preserve row correspondence across activations, routing, and scale; their transformations are not independent choices.
| Path | Prepare and dispatch | Scale at consumption |
|---|---|---|
| ALLGATHER | Pad each rank to the DP-wide maximum, then gather activation, routing, and scale in the same rank order | Normalize the gathered scale at the expert boundary for the native GMM path |
| MC2 | Keep local rows aligned with the active mask and prepare padding; dispatch activation and scale to the selected experts together | Pass a two-dimensional scale into dispatch; the native downstream helper normalizes its output for GMM |
Padding uses zero activations and routing weights, a valid placeholder expert ID, and E8M0 byte 127 (scale 1). E8M0 padding and collectives can use a byte view to preserve the stored exponents. Finalize removes padding under the same token metadata used by prepare.
For the validated H=4096 model, transfer scale is [M,128]. After alignment or dispatch, the native GMM layout is [R,64,2], where R is that stage's row count, not necessarily M. The connector and runner do not perform expert reordering or convert the transfer scale into this three-dimensional layout.
MC2 has a separate quantization control from A2E transport. For E4M3 input with an E8M0 scale, dispatch receives scales=<scale> and quant_mode=0: communicate the existing quantized values and scale without dynamic quantization. An explicit incompatible comm_quant_mode is rejected. With no external scale, dispatch retains its original quantization selection. This contract depends on the installed operator implementation, not merely on the presence of a Python argument; see the pinned dispatch operator contract.
3.3 Graph and call lifetime
Capture records receives and subsequent device operations. Replay uses the storage bound to that captured graph: the receive operations refresh activation, top-k, and scale there before expert computation reads them. Python attribute assignments made during capture are not a refresh mechanism during replay.
F caches graphs by its existing per-rank token-count key. Each graph retains its own bound tensor views and lifetimes; this proposal does not introduce one mutable top-k buffer shared across graph keys. Any change that moves a receive outside capture needs an explicit update of the captured input storage before replay. Temporary Python routing/scale/mask references are cleared after the call, including exceptional exits.
Validation checks actual replay and changed inputs at a fixed graph key, then switches keys. Successful capture or higher throughput alone does not establish correct replay.
4. Plugin boundary and supported scope
The model wrappers own router/shared placement and weight loading. The connector owns transport and wire formats. The runner forwards the payload. MoE prepare, dispatch, and quantization own row alignment and kernel-specific scale layout.
Three pinned compatibility patches bridge missing upstream interfaces:
| Patch | Why needed | Removal condition |
|---|---|---|
| ALLGATHER prepare | Native prepare does not align externally supplied top-k and activation scale | Upstream prepare accepts and aligns that bundle |
| MC2 prepare | External routing and scale need the activation's padding and active-row layout | Upstream prepare carries the bundle through those transformations |
| MC2 dispatch kwargs | Native kwargs do not forward this external scale or select the pre-quantized mode | Upstream dispatch supports the same external-scale contract |
The patches preserve upstream behavior when no external routing/scale is provided. Installation needs compatibility checks against the pinned upstream implementation and idempotence. It does not modify the vLLM source checkout. Plugin-owned extension points are preferred when an equivalent hook is available.
The initial target is A5, DeepSeek-V4-Flash W4A8MXFP, synchronous CAMP2P, TP1, without sequence parallelism, DBO, or ubatching. MoE ALLGATHER and MC2 are the supported paths; ALLTOALL and FUSED_MC2 are outside the proposal. Eager and FULL_DECODE_ONLY are the execution modes under evaluation. MXFP8 additionally requires the target A2E operator's equal A/F rank-count topology and E4M3/E8M0 format; device evidence covers single-node 4A4F only.
The proposed guardrail requirement is to reject unsupported configurations as early as the relevant facts are available, before entering an incompatible device operation. This includes unsupported communication policies, incompatible A/F settings, and platform requirements. These are design requirements; completion of the corresponding checks is part of implementation review before merge.
The MXFP8 operator build needs the A2E extensions described in #444, including x_scale and quant_mode. BF16 keeps the original operator call. Async CAM, wider TP, new communication paths, multi-node qualification, and changes to the default mode are outside this RFC.
5. Evidence and alternatives
5.1 Measurement conditions and interpretation
Two datasets are reported separately: the current BF16 split-branch session and the earlier MXFP8 development-branch experiments. Comparisons stay within each dataset. Section 5.5 summarizes the validation status and measurement scope.
| Setting | Recorded value |
|---|---|
| Hardware and topology | Single-node A5 4A4F; A DP4/TP1 on devices 0–3, F DP4/TP1 on devices 4–7 |
| Model | DeepSeek-V4-Flash, W4A8MXFP |
| Runtime limits | max-model-len=9216, max-num-seqs=128, max-num-batched-tokens=3072 |
| KV settings | kv-cache-dtype=fp8, indexer_kv_dtype=fp8 |
| MoE communication | Automatic selection for every configuration below |
| Load | 128 requests, target concurrency 128, synthetic input setting 8192, output 32 |
| Reported token counts | 1,059,200 input / 4,096 output tokens in the cited benchmark logs; reported input averages 8275 per request |
| Execution | Eager uses --enforce-eager; Graph uses FULL_DECODE_ONLY, capture sizes [8,16,32,64,128] |
| Throughput metric | Benchmark-reported Total Token Throughput, input plus output tokens per second |
The recorded benchmark runs completed with 128 successes and no failed requests. Graph windows were approximately 3 seconds; eager windows were roughly 5–7 seconds. The configured input length and reported token count are shown separately. All means and ratios below are recomputed from the listed throughput values; ratios use unrounded means. n counts benchmark runs, not requests.
5.2 Current BF16 placement comparison
The BF16 comparison uses 828582a for both gate placements and bcc0126 as the clean baseline, with BF16 transport (quant_mode=0) and automatic communication selection.
| Mode | Placement / code | Each run (token/s) | Mean (token/s) | n |
|---|---|---|---|---|
| Eager | Gate on A, 828582a |
192,374 / 193,400 | 192,887 | 2 |
| Eager | Gate on F, 828582a |
166,567 / 161,427 | 163,997 | 2 |
| Eager | Clean baseline, bcc0126 |
165,081 / 164,005 | 164,543 | 2 |
| Graph | Gate on A, 828582a |
327,970 / 324,830 | 326,400 | 2 |
| Graph | Gate on F, 828582a |
333,602 / 329,309 | 331,455.5 | 2 |
| Graph | Clean baseline, bcc0126 |
334,084 / 332,730 | 333,407 | 2 |
| Comparison | Eager | Graph (FULL_DECODE_ONLY) |
|---|---|---|
| Router/shared on A versus on F, same commit | +17.6% throughput | -1.5% throughput |
| Router/shared on F versus clean baseline | -0.3% | -0.6% |
Graph gate-on-A GSM8K was 90.60% over 1319 questions. These results are also recorded in #503. They support offering a placement choice while retaining the current default.
5.3 Historical MXFP8 performance: Graph
The historical experiment used vLLM ced6857, vLLM-Ascend 8d4409d6, MXFP8 development code f683eb1, and clean baseline bcc0126. BF16 controls come from earlier development revisions; the gate-on-A control was recorded as 52a292d. The cross-revision comparison is qualified in section 5.5.
| Configuration | Each run (token/s) | Mean (token/s) | n | Observed range (token/s) |
|---|---|---|---|---|
| Clean baseline, gate on F, BF16 | 343,163 / 340,709 / 338,660 | 340,844 | 3 | 338,660–343,163 |
| Development control, gate on F, BF16 | 341,995 / 341,185 | 341,590 | 2 | 341,185–341,995 |
| Historical control, gate on A, BF16 | 336,305 / 334,131 | 335,218 | 2 | 334,131–336,305 |
| Gate on A, MXFP8 | 347,730 / 351,374 / 367,132 / 356,085 | 355,580.25 | 4 | 347,730–367,132 |
| MXFP8 comparison | Ratio of means minus one | Interpretation |
|---|---|---|
| Gate-on-A MXFP8 versus clean BF16 baseline | +4.32% | Entire proposal versus a fixed clean baseline; includes placement and transport changes |
| Gate-on-A MXFP8 versus gate-on-A BF16 | +6.07% | Historical cross-revision comparison; does not isolate quantization alone |
The observed ranges and individual runs are retained above so readers can assess the aggregate alongside run-to-run variation.
5.4 Historical MXFP8 performance: eager
| Configuration | Each run (token/s) | Mean (token/s) | n | Observed range (token/s) |
|---|---|---|---|---|
Clean baseline bcc0126, gate on F, BF16 |
177,273 | 177,273 | 1 | 177,273 |
| Development control, gate on F, BF16 | 191,229 / 188,364 | 189,796.5 | 2 | 188,364–191,229 |
| Historical control, gate on A, BF16 | 226,743 / 217,608 | 222,175.5 | 2 | 217,608–226,743 |
| Gate on A, MXFP8 | 225,276 / 224,819 | 225,047.5 | 2 | 224,819–225,276 |
The eager measurements provide workload context. Historical full GSM8K for MXFP8 was 90.30% in eager and 90.14% in Graph, each over 1319 questions.
The tables above summarize historical development measurements. The Graph aggregate was also reported in #497. These measurements span development revisions and motivate a controlled comparison on the final implementation.
5.5 Validation status, measurement scope, and alternatives
BF16 external routing has device evidence at 828582a. Historical MXFP8 measurements provide preliminary performance and full-dataset task-accuracy evidence for the development stack. The current MXFP8 revision 4d2d890 has passed CPU checks; device validation of the final stack remains pending. The original profile fix was exercised in the BF16 integration stack; its replacement described in section 6.1 requires revalidation.
Performance observations use small samples and short batch-completion windows; no statistical significance test was performed. The historical MXFP8 comparison spans development revisions, with incomplete per-run attribution for the BF16 control, and does not isolate quantization alone. Complete dependency versions were not captured for the current BF16 session, so that comparison is limited to its within-session arms.
Graph versus eager changes compilation settings as well as ACL Graph replay. Neither the mode-dependent performance mechanism nor an A2E bandwidth bottleneck has been established by profiling. Final-stack validation should record exact versions, communication-path execution, and capture/replay with changing inputs; the BF16 split-branch session did not retain per-path instrumentation. Task accuracy, throughput, and CPU interface checks serve different validation goals.
Keeping routing on F avoids the added A work and remains the default. A transport-only alternative can retain the received FP8 activation and scale for routed experts while recovering model-dtype activations for router/shared computation. Moving both operations to A removes that recovery requirement. The proposed capability supports the runtime's automatic selection between ALLGATHER and MC2.
5.6 Potential for larger gains at scale
Attention-side routing and MXFP8 pass-through provide two complementary ways to adapt AFD to deployment needs: placing router/shared computation on A creates an opportunity to improve A/F load balance, while quantized transport reduces the activation payload delivered to F. With suitable A/F resource allocation and interconnect characteristics, these capabilities offer potential for larger gains as deployments scale.
The historical MXFP8 Graph measurements above used one eight-device host, with A and F communicating through intra-node HCCL. Mean throughput was 355,580.25 token/s: +4.32% versus the clean BF16 baseline and +6.07% versus the earlier gate-on-A BF16 control. The measurement limits in section 5.5 apply. These observations motivate investigating larger deployments, where aggregation and interconnect bandwidth may make A2E quantization more valuable.
Higher A:F ratios can increase pressure on activation delivery. AFD allows multiple Attention instances to feed fewer FFN instances, combining distributed KV-cache capacity with larger expert-computation batches. As more Attention instances contribute tokens, incoming activation volume and FFN work both grow. If expert computation remains efficient while activation delivery becomes a limiting resource, a smaller A2E payload may help preserve the benefits of aggregation. This is a promising extension beyond the initial equal-rank MXFP8 target; higher A:F device ratios need corresponding operator and alignment support, not only a configuration change.
Bandwidth-constrained links can increase the value of payload reduction. When placement requires A/F traffic to cross an interconnect with lower effective bandwidth, activation delivery can consume more of the critical path. MXFP8 roughly halves the activation-data bytes relative to BF16. Activation scales, routing metadata, and other communication costs remain, so this does not halve total traffic or imply twice the throughput.
Larger gains are therefore plausible when A2E delivery limits execution, provided A-side quantization and added router/shared work do not create another bottleneck. E2A returns and expert communication are not compressed by this proposal and may limit the gain instead. This is an expectation, not a measured scaling result. Validation should vary A:F ratios, deployment size, interconnect bandwidth, and workload, and measure sustained throughput together with A2E time on the critical path.
6. Implementation PRs and validation
6.1 What is implemented
The proposal is implemented as a stack of three PRs, all open as of 2026-11-02. The PR stack is being updated to place profile-dependent expert-ID balancing behind a narrow opaque custom-op boundary, preserving the upstream profiling and compilation lifecycle. The updated stack will be restacked and revalidated before merge. The table records the posted revisions before this update.
#500 profile/compilation fix
└─ #503 BF16 external routing
└─ #506 MXFP8 pass-through
| PR | Branch | Recorded revision | Implemented scope |
|---|---|---|---|
| #500 | pr1/profile-pass-compile |
1da04df |
Keep profile execution out of compilation; guarded installation and tests |
| #503 | pr2/external-routing |
828582a |
A-side router/shared execution, top-k transport, F-side consumption, BF16 alignment on ALLGATHER and MC2 |
| #506 | pr3/mxfp8-passthrough |
4d2d890 |
FP8/E8M0 transport, scale-carrying payload, per-path scale layout, pre-quantized MC2 dispatch |
All PRs target main, so GitHub's displayed diff includes unmerged dependencies; their file and line counts are not independent changes to add together. Review the incremental ranges for #503 and #506. After a dependency merges, rebase and recheck the next PR's own diff and validation.
6.2 How the CPU checks are run
CPU tests use real PyTorch tensors for dtype, layout, padding, and scale-byte checks. Stand-ins replace vLLM/Ascend interfaces and operator calls that need the NPU environment. The maintained suite runs test files in isolated processes to avoid leaking stand-in modules between files. Representative focused commands are:
python -m pytest -q tests/unit/connectors/test_external_routing.py
python -m pytest -q tests/unit/compat/patches/npu/test_mc2_dispatch_prequant.pyThe checks cover paired fields and rejection paths, connector arguments and received payloads, inert padding and rank order, scale contents, ALLGATHER versus MC2 layout at the expert boundary, dispatch mode conflicts, installer behavior, and preservation of the BF16 path.
| PR | Reported CPU result |
|---|---|
| #500 | Maintained suite: 226 passed, 0 skipped, 0 failed |
| #503 | 99 focused cases; maintained suite: 235 passed, 0 skipped, 0 failed |
| #506 | Maintained suite: 235 passed, 0 skipped, 0 failed |
Counts are those reported by the linked PRs for the listed revisions. Device-validation status is summarized in section 5.5.
6.3 Device test scenarios and method
Device validation starts matching A/F services, waits for readiness, checks ordinary generation, and then runs ais_bench through the Attention role's vLLM API for synthetic throughput and GSM8K accuracy. Service startup time is outside the benchmark duration.
| Scenario | Procedure | What it establishes |
|---|---|---|
| BF16 role-placement comparison | On the same node/session, run gate-on-A, gate-on-F, and a clean baseline in eager and FULL_DECODE_ONLY; two benchmark runs for each of six cells |
The placement comparison in section 5.2; gate-on-A and gate-on-F share 828582a and differ in the gate switch |
| Synthetic throughput | Use 128 requests/concurrency target 128, input setting 8192, output 32, and the runtime limits in section 5.1; retain reported token counts, duration, and success/failure counts | Batch-completion throughput for this workload, including the observed effective input count |
| BF16 task accuracy | Run full GSM8K; gate-on-A scored 91.13% in eager and 90.60% in Graph | Task-level accuracy for the recorded BF16 configurations |
| Historical MXFP8 evaluation | Run the pre-quantized path in eager and Graph with automatic communication selection; benchmark values are in sections 5.3–5.4 and full GSM8K scores were 90.30% / 90.14% | Development-stack performance and full-dataset task accuracy |
For final-stack validation, run the BF16 control and MXFP8 candidate at the same final commit under matching environment and execution settings. Record task accuracy, communication-path selection, and capture/replay with changing inputs at a fixed graph key and across key changes. Include service-restart repetitions and longer runs to assess reproducibility and sustained throughput.
If accepted, this proposal updates the existing Attention, FFN, model integration, connector, and patch documents and the CAMP2P user guide.
7. Planned model adaptations
The next model-adaptation targets are MiniMax-3.1-Flash and
DeepSeek-V4.1. The aim is to extend the same A-side routing and MXFP8 A2E
pass-through capability through model-specific adapters, reusing the common
connector, payload, and MoE alignment contracts.
7.1 Model-specific integration
| Target model | Adaptation work |
|---|---|
| MiniMax-3.1-Flash | Add a plugin-owned adapter for role construction, checkpoint loading, and Attention-side routing. Preserve the native model's gate input, expert selection, top-k weighting, and output scaling. Place shared experts on A when the model has them, and connect the routed-expert backend to transported activations and scales. |
| DeepSeek-V4.1 | Extend the DeepSeek adapter after checking the native model's construction, checkpoint layout, routing behavior, and quantization interfaces against the existing DeepSeek-V4 integration. Reuse compatible components while preserving V4.1-specific behavior, including token-dependent routing where applicable. |
Each adapter obtains the hidden width, expert count, top-k, and quantization
layout from its target model and checkpoint. The DeepSeek-V4-Flash example
shapes in section 3 are not constants for these additional models. Shared experts
are constructed only when present in the model.
Router execution and any shared-expert computation use the model-dtype
activation on A. The connector carries the resulting top-k together with the
activation and, for MXFP8, its E8M0 scale. Model-specific routing and weight
semantics remain in the adapter; transport continues to use the common contract.
Direct MXFP8 consumption is enabled with the corresponding native expert-backend
support for the received activation and scale format.
7.2 Integration and validation sequence
For each model:
- Establish the native and router-on-F BF16 reference, including checkpoint
loading and end-to-end generation. - Integrate BF16 external routing, checking expert IDs, routing weights, routed
scaling, and shared/routed output composition against the native path. - Enable MXFP8 transport and direct consumption, preserving activation/scale row
correspondence through ALLGATHER and MC2 automatic selection. - Validate eager execution, then graph capture and replay with changing inputs
and batch sizes, using the narrow routing custom-op boundary for
profile-dependent behavior. - Record model-appropriate task accuracy and same-revision BF16/MXFP8 throughput,
including prefill-heavy and decode-heavy workloads, with exact model,
checkpoint, software, and execution configurations.
These adaptations are follow-up work. The performance and accuracy results in
section 5 remain specific to the recorded DeepSeek-V4-Flash configurations;
model-specific results will accompany each adaptation.
Activity
[-][RFC]: External routing and MXFP8 pass-through for synchronous A5 AFD[/-][+][RFC]: External routing and MXFP8 pass-through for synchronous AFD on Ascend 950 (A5)[/+]