Repository navigation
Description
Motivation
Elastic AFD -- resizing the Attention pool and the FFN expert pool
independently, in place -- is implemented for Ascend NPU and has been qualified on
hardware: the group lifecycle, the resize ladder, the rejection paths and the
graph-mode boundary were each measured on a single-node 8x 910_93 host.
This RFC is the design record for that work. It exists because qualification on
real hardware produced four conclusions that are design decisions rather than
implementation details, and none of them is visible from the API:
- What "correct after a resize" means on a quantized MoE deployment. With
W8A8 + MC2, moving experts between FFN ranks changes chunk boundaries and
accumulation order, so greedy decoding can legitimately take a different
continuation. The checkable property -- and the one every number here uses -- is
a resized deployment equals the same topology booted statically. An
equality claim against the pre-resize output is not achievable and should not
be written into a test. - Graph capture and the N:1 fanout do not coexist on Ascend, and the refusal
is the cheaper failure. The fanout publishes one uniform row count per step
read from runtime DP metadata; dynamo cannot bound a value it only learns at
runtime, and the step that fails is the step that restores the Attention
runtime -- so it cannot be rolled back. Startup refuses the combination. - A failure after the switch begins is unrecoverable from the plugin, so the
deployment should stop serving rather than look healthy. Drawing that line
precisely is the substance of sec. 3: most of the failure surface (preflight
refusal, drain timeout) is not destructive and must not cost availability,
while the rest has no recovery path at all. A rollback across that line was
built, measured, and removed. - On Ascend, elastic AFD currently brings Ray data-parallel device assignment
with it. Resolving Ray'sNPUaccelerator resource lives in the elastic
Attention actor, which changes what "opt-in" means on this backend (sec. Plugin
boundary).
#618 carries the implementation; this document is where the reasoning lives,
because that PR ships no in-tree documentation.
Proposed change
0. Position
| Covered | elastic resize of both roles on Ascend NPU: CAMP2pAFDConnector, TP=1, eager, divisible A:F, single node |
| Also carries | the shared executor (AFDElasticGPUExecutor), because AFDElasticNPUExecutor subclasses it: that file is on the Ascend code path whether or not a CUDA device exists. Its CUDA behaviour is CPU-tested and source-reviewed only |
| Out of scope | graph modes on Ascend (refused at startup), DBO/TBO/XBO, async connectors, TP>1, multi-node, speculative decoding, PD+AFD |
| Backend / topology envelope | Ascend 910_93, CAMP2pAFDConnector, TP=PP=PCP=DCP=1, eager, A DP >= 2, F DP >= 1, A divisible by F, single node |
1. Component map
flowchart TD
OP["operator / external policy"] -->|"POST /scale_elastic_ep"| MW["AFDScalingMiddleware<br/>scaling gate -> 503 for inference while a switch runs"]
MW --> RT["elastic route handler<br/>afd_plugin/elastic/api.py"]
RT -->|"preflight(), then resize()"| CL["ElasticAFDClient<br/>(lives in the API process)"]
CL -->|"pause / prepare_elastic_ep / commit_elastic_ep / resume / collective_rpc"| CC["DPLBAsyncMPClient (native CoreClient)"]
CL -->|"rendezvous / RPC / actor lifecycle"| FM["FFNActorManager"]
CC --> AE["A EngineCore actors, one per A DP rank<br/>AFDEngineCoreActor -> patched DPEngineCoreProc"]
AE --> AW["AFDNPUAttentionWorker<br/>AFDNPUAttentionModelRunnerV2"]
FM --> FA["FFNActor, one per F DP rank<br/>short control RPCs only"]
FA --> FW["AFDNPUFFNWorker<br/>serve loop in a worker thread"]
AW <-->|"uniform row count + activations over HCCL;<br/>DP metadata over a gloo control plane"| CON["CAMP2pAFDConnector"]
FW <--> CON
CO["vLLM DPCoordinator"] -.->|"DP wave + pause consensus"| CC
RAY["Ray head + placement groups<br/>one device bundle per rank"] -.-> AE
RAY -.-> FA
Where each piece actually runs, because the failure modes are process-local:
| Process | Count | Hosts | Created by |
|---|---|---|---|
| API / front end | 1 | route handler, middleware, ElasticAFDClient, FFNActorManager (actor handles), native CoreClient |
vllm serve |
| Attention EngineCore actor | A DP | engine busy loop (patched), UniProcExecutor, A worker + runner + connector |
native CoreEngineActorManager via EEP |
| FFN actor | F DP | UniProcExecutor, F worker + runner + connector, serve loop in a thread |
FFNActorManager |
DPCoordinator |
1 | wave accounting and pause consensus for the A set | vLLM |
Two consequences worth stating: the FFN serve loop is a thread inside an actor
that otherwise answers short RPCs, so a dead loop is invisible to a process
count; and the client lives in one API process, so the gate is per-process (risk 5).
2. The resize call chain
sequenceDiagram
autonumber
participant RT as route handler (API proc)
participant CL as ElasticAFDClient (API proc)
participant A as A engines (Ray actors)
participant F as F actors (Ray actors)
participant N as native EEP inside EngineCore
RT->>RT: preflight(size, role) -> a 400 here opens no gate at all
RT->>CL: reserve(size, role) FFN roles pre-book their devices
Note over RT,CL: an unplaceable bundle is a 408 refusal with the reason;<br/>the gate is still down, so a refused request never opens a 503 window
RT->>RT: raise scaling gate (503 window starts)
RT->>RT: drain: front-end registry + in-flight routing + dp_engines_running
RT->>CL: resize(size, role)
Note over CL,N: from here on the sequence is destructive
CL->>A: pause_scheduler(keep, clear_cache) native DP pause consensus
CL->>A: afd_send_stop only the connector's designated senders
CL->>F: join_ffn_server_loop
par release both roles concurrently
CL->>A: afd_release_link retire graphs, then connector.close()
and
CL->>F: afd_release_link
end
CL->>F: rendezvous() -> F0 returns its host + two fresh store ports
CL->>A: afd_update_topology(target)
alt role == attention
CL->>F: afd_update_topology(target)
CL->>N: prepare_elastic_ep(size) new DP group + standby handshake
CL->>N: commit_elastic_ep()
loop until every engine reports eep_scaling_state is None
CL->>N: afd_eep_complete (utility, yielding)
end
else role == ffn
CL->>F: release_roles -> resize_actors(new placement group) -> initialize_roles
end
par reconnect (connector construction is collective)
CL->>A: afd_connect
and
CL->>F: afd_connect
end
CL->>F: start_ffn_server_loop
CL->>A: afd_finish KV init + warmup, no graph capture on Ascend
CL->>RT: resume_scheduler -> gate lowered, service is back
Notes on three steps that are easy to get wrong:
- native DP pause is issued even though the gate already 503s. The gate stops
admission; only pause consensus stops a lateSTART_DP_WAVE/ dummy forward. RECONFIGURE_FINISHEDis too early. The client polls every target EngineCore
untileep_scaling_state is None, yielding so the engine can run
state.progress(). Waiting on the native event alone resumes service onto an
engine that has not finished switching.- warmup that touches remote experts must happen after both sides reconnect, and
on Ascend the F rank is a fresh actor: a kept FFN rank cannot re-establish its
HCCL communicators, so every NPU size change rebuilds the pool and reloads
weights (~15 s per rank). That is the dominant term in the FFN window, and it is
why an FFN resize costs like an Attention resize rather than the ~10 s a
weight-reload-only design would give.
3. Where the switch becomes destructive, and what we do about it
flowchart LR
Q["POST /scale_elastic_ep"] --> P{"preflight<br/>DP bound - divisibility - native expert minimum - device capacity"}
P -->|"refuse"| R400["400 - gate never opened<br/>nothing paused, nothing drained"]
P -->|"pass"| RES["reserve FFN bundles<br/>bounded wait, self-cleaning"]
RES -->|"not schedulable"| R408R["408 - gate never opened<br/>bundles removed - service never interrupted"]
RES -->|"reserved"| G["raise gate"]
G --> D{"drain within drain_timeout"}
D -->|"timeout"| R408["408 - gate lowered again<br/>deployment still serving the old topology"]
D -->|"done"| SW["switch: pause - STOP - release - rebuild - reconnect - warm"]
SW -->|"success"| OK["200 - gate lowered"]
SW -->|"any exception OR cancellation"| FC["gate STAYS closed - controller.degraded = reason - one ERROR log line<br/>requests 503 - later resizes 503 with the degraded reason<br/>only a restart clears it"]
| Failure point | Admission after | Why this is the right answer |
|---|---|---|
| preflight refusal | unchanged -- no 503 window ever opened | nothing was touched |
| FFN bundles not schedulable within the reserve deadline | refused before the gate -- 408 with the reason, bundles self-removed, service never interrupted | the capacity preflight sees what Ray reports as present; occupancy only surfaces when the bundle is waited on, so that wait is bounded and happens while the deployment is still serving |
| drain timeout | reopened | draining mutates nothing; the old topology still serves |
anything from resize() onward, including a client that hangs up |
kept closed, with a reason | a half-applied topology has no plugin-side recovery (see below), and serving-while-broken is the worse failure |
A cancellation counts as destructive on purpose, and it is the one decision here I
would like challenged: the client disconnect cancels our coroutine while the
engine-side RPCs keep running, so the plugin's own view (topology, links, F pool)
may no longer describe the deployment, and this build has no state probe to
re-establish it. A short curl -m inside a ~74 s window therefore fences the
deployment. The alternative -- reopen the gate and hope -- is what the code did
before, and it produced a deployment that answered /health with 200 and never
completed a request.
Why not roll back instead: a full rollback was built and measured. It restored
configs, the F actor pool, links and the paused schedulers, and the deployment
still died later, because nothing cancels the in-flight new-DP-group future inside
ElasticEPState -- the rollback converts "dead immediately" into "dead later". It
was removed rather than shipped half-done, and the durable fix is upstream.
4. Capacity model and pre-gate refusals
Four of the five refusals below are pure reads of live state; the fifth is a
bounded discovery. All of them happen before the gate:
| Rule | Form |
|---|---|
| DP bound | size <= parallel_config.elastic_ep_max_dp_size |
| Divisibility | A DP - A TP divisible by F DP - F TP; A ranks >= F ranks |
| Native expert minimum | shrinking A must keep physical experts per engine >= E / A_new; refused below ceil(E - A_cur / physical) -- the same arithmetic and wording native uses, applied earlier |
| Device capacity | need = A_dp-A_tp + F_dp-F_tp <= min(Ray cluster accelerator report, visible-device list); fails open when neither reports a count |
| Late placement discovery | FFN roles pre-book their bundles with a 120 s deadline right after the preflight; expiry is a readable 408, the bundles made by that call are removed, and no gate ever opens |
The visible-device term exists because Ray advertises every accelerator on the node
while the launch script pins a smaller ASCEND_RT_VISIBLE_DEVICES -- observed as Ray
reporting 16 while the server could use 4, which would have waved an impossible
scale-up straight into the store handshake.
5. Topology identity: one generation per resize
A rebuilt connector must never be able to talk through a retired communicator, so
every process group the connector creates carries a generation suffix equal to the
rendezvous port that generation negotiated:
| Group | Backend | World | Name |
|---|---|---|---|
| AFD world (one per ubatch slot) | HCCL | F + A | afd_<port>, afd1_<port>, ... |
| MoE dispatch | HCCL | F | afd_moe_<port> |
| control plane (DP metadata) | gloo | F + min(A,F) | p2p_<port> |
The port is allocated by FFN rank 0 -- the process that actually binds the stores,
not the API host -- and published to both roles through the AFD config, so names are
unique per generation yet byte-identical across all participants. Creating these
groups at all needs the Ascend stateless process-group patches: per-group ids,
disjoint port windows and thread-local store clients, because the pinned Ascend
worker registers one global group per name.
6. Data-plane contract on Ascend
For A > F (N:1 fanout) the two roles must agree on one row count per step:
- every A rank in a group publishes the same uniform row count for a step,
derived from the DP metadata rather than its own token count; - the a2e / e2a kernels move an equal share of rows per source rank, so a
non-divisible fanout is refused rather than silently misrouted; - routing ids and scales are padded to that same uniform count on the send side.
This was a live defect surfaced by the rebase: a sibling upstream change moved
BF16 routing into the same payload using the unpadded count, soa2eindexed
past the supplied rows -- the classic row-misplacement signature, output that
varies between identical requests; - the FFN runner sizes
xActiveMask/expand_xand its graph keys from the
published rows, not the real token count, otherwise MoE dispatch fails with a
mask/dimension mismatch and sibling ranks spin in vector-core wait.
7. Numerics and the correctness criterion
Digests are taken over choices[0].text only -- the response body carries a
random id and a created timestamp and is therefore never reproducible.
| Topology | Greedy digest | Agreement |
|---|---|---|
| 2Ax2F (static boot, and every post-resize visit) | 00923c4faae9... |
3 visits |
| 2Ax1F (static control, and every resized visit to 1 F) | ecf2a9ea057b... |
4 visits |
FFN rungs differ from the 2Ax2F baseline and match the same-topology static
control. That equality is the criterion: output stable across probes that also
differs from a static boot of the same topology is what a resize that left the
wrong state looks like.
8. Measured cost of a switch
One host, one checkpoint, eager, 2Ax2F boot, --elastic-ep-max-dp-size 8,
--kv-cache-memory-bytes 2GiB. Windows are wall clock of the HTTP call, measured on
the ladder run and again on the final commit:
| Rung | Window | Output vs baseline |
|---|---|---|
| Attention 2->4 | 73 s / 74.0 s | byte-identical (3 probes, plus 3 on the re-run) |
| Attention 4->2 | 10 s / 10.6 s | byte-identical |
| FFN 2->1 | 68 s | differs from 2Ax2F, equals static 2Ax1F |
| FFN 1->2 | 66 s | back to the 2Ax2F digest |
| refusals | 18-39 ms | service unaffected, no drain, no pause |
FFN 2->1 / 1->2, re-run on 7f17cc9 |
25.7 s / 64.5 s | probe output identical across all six probes |
| unsatisfiable FFN placement (device held by another job) | 408 after 120.0 s, with the reason | completions answered every 5 s through the wait; no degraded marker; the FFN pool was unchanged afterwards |
Not instrumented: the per-phase split inside a window. The FFN figures are
explained by actor rebuild + weight reload (~15 s/rank, observed during pool
bring-up); the attention-up figure is not decomposed, and I would rather leave it
undecomposed than attribute it by guess.
9. Decisions taken, and the open questions
| # | Decision | Status | Open question for reviewers |
|---|---|---|---|
| D1 | Correctness after a resize is judged against a same-topology static boot | adopted; used by every measurement above | is this the contract the project wants written into the elastic test suite? |
| D2 | Ascend elastic is eager-only, refused during config validation | implemented | accepted as a restriction with a named unblock (static per-bucket row counts and an abort path), or should the combination fail loudly at first resize instead? |
| D3 | Fail closed from the switch onward; reopen after a drain timeout; later resizes answered with the degraded reason | implemented; the gate, the 503, the log line and the reservation refusal (408 with the reason, service answering throughout the wait) are measured; the wording fix is unit-tested | is fencing on a hung-up client acceptable? After cancellation our own view may no longer describe the engines and there is no state probe; the alternative is probe-and-decide, which does not exist yet |
| D4 | Statically-unsatisfiable requests are refused before the gate exists | implemented, measured at 18-39 ms | should the same refusal shape apply to role=ffn on non-elastic configs (today a plain 400)? |
10. Remaining work, with owner
| Item | Owner | Cost | Hardware |
|---|---|---|---|
ElasticEPState.abort_prepare() -- cancel an in-flight scale-up |
upstream vLLM | not ours; plugin hookup ~1 day after it lands | reuses the existing injection case |
| Graph modes + elastic on Ascend (static per-bucket row counts; graph-manager rebuild that survives boot) | plugin | 2 -3 weeks, dominated by boot->capture->crash iterations, and gated on the row above | exclusive |
| CUDA hardware qualification of the shared/CUDA executor this design also carries | plugin | 1-2 days | yes, pinned vllm==0.32.0 |
| Move Ascend Ray-DP device assignment out of the elastic actor | plugin | small but cross-cutting | boot smoke |
Re-anchor the remaining main@544f182d patch markers onto the pinned 0.32.0 line |
plugin | 1/2-1 day | boot smoke |
In-tree documentation, if wanted anywhere (a docs/npu/ page or the recipe tree) |
plugin | 1/2 day | none |
Plugin boundary
Plugin-owned: the scaling client and its preflight; the FFN actor pool,
rendezvous and expert-layout reconstruction; per-generation topology identity; the
Ascend attention/FFN workers and model runners; the CAMP2P uniform row-count
protocol; the eager-only rule and the fail-closed boundary.
Compat helper: AFDElasticGPUExecutor / AFDElasticNPUExecutor (subclasses of
native ElasticEPScalingExecutor), AFDEngineCoreActor (native
DPMoEEngineCoreActor), AFDScalingMiddleware (native ScalingMiddleware).
Compat patch -- needed because native elastic EP reaches the scaled engine from
inside its own modules, where a subclass cannot intercept: role dispatch and the
drain/gate sequence around native prepare_elastic_ep / commit_elastic_ep
(compat/patches/elastic.py), AFD-aware Attention actor construction
(elastic_actors.py), the FFN engine-core busy loop and shutdown
(engine_core.py), config validation (config_validation.py), the parallel-state
swap (elastic_parallel.py), and Ascend stateless HCCL group creation
(patches/npu/stateless_pg.py). Each pins the upstream revision it copies, delimits
changed lines with PATCH START/END, is idempotent, and carries an upstream or
removal plan. The one I would most like reviewed is patches/npu/stateless_pg.py:
it is the deepest, and it exists only because the pinned Ascend worker registers one
global group per name.
Explicit class path: --worker-cls afd_plugin.v1.worker.npu.AFDNPUAttentionWorker, plus the vllm.endpoint_plugins
entry point afd_elastic for the route.
Layering wart. Ascend + Ray data-parallel device assignment works only because
the elastic Attention actor overrides _set_visible_devices: Ray reports Ascend
accelerators under an NPU resource, which the stock actor does not resolve.
Measured: keeping the Ray-DP recipe and dropping --enable-elastic-ep yields
workers that see zero devices (Invalid device ID / value 0 ... Expected value: [0, 0)), while the identical device list with elastic enabled boots to weight
loading. The community's static Ascend recipes do not use Ray DP at all, so this gap
is created by choosing EEP rather than by breaking an existing path -- but it means
"opt-in within the same package" is stronger on Ascend than the RFC implies, and
device assignment clearly belongs in the platform or executor layer, not in an
elastic-only actor.
Risks and alternatives
- Availability after a bad resize. Options: rollback -- built, measured,
removed (it restores state but cannot cancel the in-flight group creation, so
the deployment dies later while looking fine longer); fail closed -- chosen, and
measured; upstream abort -- the actual fix, filed. There is no plugin-only option
that is simultaneously available and trustworthy today. - Eager-only on Ascend. The alternative is making the published row count
static per capture bucket so dynamo can bound it. It interacts with the FFN
runner's mask/graph-key sizing, and until an abort path exists a failure during
post-resize re-capture is unrecoverable by construction -- so it should follow
risk 1's fix, not precede it. - A rank's teardown is global. Any AFD rank's
shutdown()calls
connector.close(), destroying process groups every other rank still uses, so one
failure becomes a deployment-wide outage and survivors report
Connection closed by peer-- an error naming someone else. It predates elastic
AFD and is filed separately, and it is why reading the last traceback in a dead
deployment wastes time: the discriminator is the first exception line and the
process that raised it. - Idle deployments self-destruct. The control-plane gloo
recvcarries a
hardcoded 30-minute timeout on both backends; with every Attention engine paused
nobody publishes metadata, the recv expires, that exception is fatal to the FFN
loop, and its teardown destroys the shared groups. Thirty minutes of silence is
therefore equivalent to a crash, and/healthnever reflects it. Both directions
measured: silence bricks, while a probe every 8 minutes kept a deployment
answering 8/8 over 64 minutes with byte-identical output. Not elastic-specific --
the timeout is an unchanged context line in this work -- but it defines the
operating envelope for every number above. - One client, one gate. The scaling gate and the degraded flag live in the API
process that owns the client. The recipe says one API process; nothing in this
build enforces it with--api-server-count > 1, where each front end would hold
its own view and two concurrent resizes could interleave. Either the launcher
refuses elastic AFD with multiple API processes, or the gate moves behind shared
state. Flagged rather than left to be discovered. - Unvalidated combinations are refused, not assumed. Only
P2pNcclAFDConnectorandCAMP2pAFDConnectorare accepted; async-CAM,
DBO/microbatch overlap, TP>1, multi-node, speculative decoding and prefix-cache
retention across a resize are unvalidated. Of these only the Ascend graph-mode
refusal is enforced at startup; the rest are unvalidated rather than
supported, which is the gap D2 has to be honest about.
Feedback period
Two weeks, or before graph-mode support on Ascend is scheduled: D2 and D3 decide
what that work can promise, and the device-assignment note decides whether Ascend
elastic can be strictly opt-in at all.
Implementation: #618. Two companion reports are deliberately not filed yet,
because filing both before a decision here would be premature: the missing abort for
an in-flight elastic-EP scale-up is upstream (vLLM), and "an idle control-plane recv
is fatal / a rank's teardown destroys shared groups" is a connector-level change.
Everything needed to file them is in Risks 3 and 4; say the word and I will open
both with the measured evidence attached.
CC list
The elastic AFD owners and the Ascend connector owners -- specifically for D1 (the
correctness contract), D3 (the cancellation rule) and the device-assignment
layering note.
Activity