Skip to content

[RFC]: Elastic AFD on Ascend NPU — resize semantics proven on hardware, and the decisions it forces #620

Open

Description

@yzchang-plus

Motivation

Elastic AFD -- resizing the Attention pool and the FFN expert pool
independently, in place -- is implemented for Ascend NPU and has been qualified on
hardware: the group lifecycle, the resize ladder, the rejection paths and the
graph-mode boundary were each measured on a single-node 8x 910_93 host.

This RFC is the design record for that work. It exists because qualification on
real hardware produced four conclusions that are design decisions rather than
implementation details, and none of them is visible from the API:

  1. What "correct after a resize" means on a quantized MoE deployment. With
    W8A8 + MC2, moving experts between FFN ranks changes chunk boundaries and
    accumulation order, so greedy decoding can legitimately take a different
    continuation. The checkable property -- and the one every number here uses -- is
    a resized deployment equals the same topology booted statically. An
    equality claim against the pre-resize output is not achievable and should not
    be written into a test.
  2. Graph capture and the N:1 fanout do not coexist on Ascend, and the refusal
    is the cheaper failure.
    The fanout publishes one uniform row count per step
    read from runtime DP metadata; dynamo cannot bound a value it only learns at
    runtime, and the step that fails is the step that restores the Attention
    runtime -- so it cannot be rolled back. Startup refuses the combination.
  3. A failure after the switch begins is unrecoverable from the plugin, so the
    deployment should stop serving rather than look healthy.
    Drawing that line
    precisely is the substance of sec. 3: most of the failure surface (preflight
    refusal, drain timeout) is not destructive and must not cost availability,
    while the rest has no recovery path at all. A rollback across that line was
    built, measured, and removed.
  4. On Ascend, elastic AFD currently brings Ray data-parallel device assignment
    with it.
    Resolving Ray's NPU accelerator resource lives in the elastic
    Attention actor, which changes what "opt-in" means on this backend (sec. Plugin
    boundary).

#618 carries the implementation; this document is where the reasoning lives,
because that PR ships no in-tree documentation.

Proposed change

0. Position

Covered elastic resize of both roles on Ascend NPU: CAMP2pAFDConnector, TP=1, eager, divisible A:F, single node
Also carries the shared executor (AFDElasticGPUExecutor), because AFDElasticNPUExecutor subclasses it: that file is on the Ascend code path whether or not a CUDA device exists. Its CUDA behaviour is CPU-tested and source-reviewed only
Out of scope graph modes on Ascend (refused at startup), DBO/TBO/XBO, async connectors, TP>1, multi-node, speculative decoding, PD+AFD
Backend / topology envelope Ascend 910_93, CAMP2pAFDConnector, TP=PP=PCP=DCP=1, eager, A DP >= 2, F DP >= 1, A divisible by F, single node

1. Component map

flowchart TD
    OP["operator / external policy"] -->|"POST /scale_elastic_ep"| MW["AFDScalingMiddleware<br/>scaling gate -> 503 for inference while a switch runs"]
    MW --> RT["elastic route handler<br/>afd_plugin/elastic/api.py"]
    RT -->|"preflight(), then resize()"| CL["ElasticAFDClient<br/>(lives in the API process)"]
    CL -->|"pause / prepare_elastic_ep / commit_elastic_ep / resume / collective_rpc"| CC["DPLBAsyncMPClient (native CoreClient)"]
    CL -->|"rendezvous / RPC / actor lifecycle"| FM["FFNActorManager"]
    CC --> AE["A EngineCore actors, one per A DP rank<br/>AFDEngineCoreActor -> patched DPEngineCoreProc"]
    AE --> AW["AFDNPUAttentionWorker<br/>AFDNPUAttentionModelRunnerV2"]
    FM --> FA["FFNActor, one per F DP rank<br/>short control RPCs only"]
    FA --> FW["AFDNPUFFNWorker<br/>serve loop in a worker thread"]
    AW <-->|"uniform row count + activations over HCCL;<br/>DP metadata over a gloo control plane"| CON["CAMP2pAFDConnector"]
    FW <--> CON
    CO["vLLM DPCoordinator"] -.->|"DP wave + pause consensus"| CC
    RAY["Ray head + placement groups<br/>one device bundle per rank"] -.-> AE
    RAY -.-> FA
Loading

Where each piece actually runs, because the failure modes are process-local:

Process Count Hosts Created by
API / front end 1 route handler, middleware, ElasticAFDClient, FFNActorManager (actor handles), native CoreClient vllm serve
Attention EngineCore actor A DP engine busy loop (patched), UniProcExecutor, A worker + runner + connector native CoreEngineActorManager via EEP
FFN actor F DP UniProcExecutor, F worker + runner + connector, serve loop in a thread FFNActorManager
DPCoordinator 1 wave accounting and pause consensus for the A set vLLM

Two consequences worth stating: the FFN serve loop is a thread inside an actor
that otherwise answers short RPCs
, so a dead loop is invisible to a process
count; and the client lives in one API process, so the gate is per-process (risk 5).

2. The resize call chain

sequenceDiagram
    autonumber
    participant RT as route handler (API proc)
    participant CL as ElasticAFDClient (API proc)
    participant A as A engines (Ray actors)
    participant F as F actors (Ray actors)
    participant N as native EEP inside EngineCore
    RT->>RT: preflight(size, role)  -> a 400 here opens no gate at all
    RT->>CL: reserve(size, role)    FFN roles pre-book their devices
    Note over RT,CL: an unplaceable bundle is a 408 refusal with the reason;<br/>the gate is still down, so a refused request never opens a 503 window
    RT->>RT: raise scaling gate (503 window starts)
    RT->>RT: drain: front-end registry + in-flight routing + dp_engines_running
    RT->>CL: resize(size, role)
    Note over CL,N: from here on the sequence is destructive
    CL->>A: pause_scheduler(keep, clear_cache)   native DP pause consensus
    CL->>A: afd_send_stop                        only the connector's designated senders
    CL->>F: join_ffn_server_loop
    par release both roles concurrently
        CL->>A: afd_release_link   retire graphs, then connector.close()
    and
        CL->>F: afd_release_link
    end
    CL->>F: rendezvous() -> F0 returns its host + two fresh store ports
    CL->>A: afd_update_topology(target)
    alt role == attention
        CL->>F: afd_update_topology(target)
        CL->>N: prepare_elastic_ep(size)   new DP group + standby handshake
        CL->>N: commit_elastic_ep()
        loop until every engine reports eep_scaling_state is None
            CL->>N: afd_eep_complete (utility, yielding)
        end
    else role == ffn
        CL->>F: release_roles -> resize_actors(new placement group) -> initialize_roles
    end
    par reconnect (connector construction is collective)
        CL->>A: afd_connect
    and
        CL->>F: afd_connect
    end
    CL->>F: start_ffn_server_loop
    CL->>A: afd_finish      KV init + warmup, no graph capture on Ascend
    CL->>RT: resume_scheduler -> gate lowered, service is back
Loading

Notes on three steps that are easy to get wrong:

  • native DP pause is issued even though the gate already 503s. The gate stops
    admission; only pause consensus stops a late START_DP_WAVE / dummy forward.
  • RECONFIGURE_FINISHED is too early. The client polls every target EngineCore
    until eep_scaling_state is None, yielding so the engine can run
    state.progress(). Waiting on the native event alone resumes service onto an
    engine that has not finished switching.
  • warmup that touches remote experts must happen after both sides reconnect, and
    on Ascend the F rank is a fresh actor: a kept FFN rank cannot re-establish its
    HCCL communicators, so every NPU size change rebuilds the pool and reloads
    weights (~15 s per rank). That is the dominant term in the FFN window, and it is
    why an FFN resize costs like an Attention resize rather than the ~10 s a
    weight-reload-only design would give.

3. Where the switch becomes destructive, and what we do about it

flowchart LR
    Q["POST /scale_elastic_ep"] --> P{"preflight<br/>DP bound - divisibility - native expert minimum - device capacity"}
    P -->|"refuse"| R400["400 - gate never opened<br/>nothing paused, nothing drained"]
    P -->|"pass"| RES["reserve FFN bundles<br/>bounded wait, self-cleaning"]
    RES -->|"not schedulable"| R408R["408 - gate never opened<br/>bundles removed - service never interrupted"]
    RES -->|"reserved"| G["raise gate"]
    G --> D{"drain within drain_timeout"}
    D -->|"timeout"| R408["408 - gate lowered again<br/>deployment still serving the old topology"]
    D -->|"done"| SW["switch: pause - STOP - release - rebuild - reconnect - warm"]
    SW -->|"success"| OK["200 - gate lowered"]
    SW -->|"any exception OR cancellation"| FC["gate STAYS closed - controller.degraded = reason - one ERROR log line<br/>requests 503 - later resizes 503 with the degraded reason<br/>only a restart clears it"]
Loading
Failure point Admission after Why this is the right answer
preflight refusal unchanged -- no 503 window ever opened nothing was touched
FFN bundles not schedulable within the reserve deadline refused before the gate -- 408 with the reason, bundles self-removed, service never interrupted the capacity preflight sees what Ray reports as present; occupancy only surfaces when the bundle is waited on, so that wait is bounded and happens while the deployment is still serving
drain timeout reopened draining mutates nothing; the old topology still serves
anything from resize() onward, including a client that hangs up kept closed, with a reason a half-applied topology has no plugin-side recovery (see below), and serving-while-broken is the worse failure

A cancellation counts as destructive on purpose, and it is the one decision here I
would like challenged: the client disconnect cancels our coroutine while the
engine-side RPCs keep running, so the plugin's own view (topology, links, F pool)
may no longer describe the deployment, and this build has no state probe to
re-establish it. A short curl -m inside a ~74 s window therefore fences the
deployment. The alternative -- reopen the gate and hope -- is what the code did
before, and it produced a deployment that answered /health with 200 and never
completed a request.

Why not roll back instead: a full rollback was built and measured. It restored
configs, the F actor pool, links and the paused schedulers, and the deployment
still died later, because nothing cancels the in-flight new-DP-group future inside
ElasticEPState -- the rollback converts "dead immediately" into "dead later". It
was removed rather than shipped half-done, and the durable fix is upstream.

4. Capacity model and pre-gate refusals

Four of the five refusals below are pure reads of live state; the fifth is a
bounded discovery. All of them happen before the gate:

Rule Form
DP bound size <= parallel_config.elastic_ep_max_dp_size
Divisibility A DP - A TP divisible by F DP - F TP; A ranks >= F ranks
Native expert minimum shrinking A must keep physical experts per engine >= E / A_new; refused below ceil(E - A_cur / physical) -- the same arithmetic and wording native uses, applied earlier
Device capacity need = A_dp-A_tp + F_dp-F_tp <= min(Ray cluster accelerator report, visible-device list); fails open when neither reports a count
Late placement discovery FFN roles pre-book their bundles with a 120 s deadline right after the preflight; expiry is a readable 408, the bundles made by that call are removed, and no gate ever opens

The visible-device term exists because Ray advertises every accelerator on the node
while the launch script pins a smaller ASCEND_RT_VISIBLE_DEVICES -- observed as Ray
reporting 16 while the server could use 4, which would have waved an impossible
scale-up straight into the store handshake.

5. Topology identity: one generation per resize

A rebuilt connector must never be able to talk through a retired communicator, so
every process group the connector creates carries a generation suffix equal to the
rendezvous port that generation negotiated
:

Group Backend World Name
AFD world (one per ubatch slot) HCCL F + A afd_<port>, afd1_<port>, ...
MoE dispatch HCCL F afd_moe_<port>
control plane (DP metadata) gloo F + min(A,F) p2p_<port>

The port is allocated by FFN rank 0 -- the process that actually binds the stores,
not the API host -- and published to both roles through the AFD config, so names are
unique per generation yet byte-identical across all participants. Creating these
groups at all needs the Ascend stateless process-group patches: per-group ids,
disjoint port windows and thread-local store clients, because the pinned Ascend
worker registers one global group per name.

6. Data-plane contract on Ascend

For A > F (N:1 fanout) the two roles must agree on one row count per step:

  • every A rank in a group publishes the same uniform row count for a step,
    derived from the DP metadata rather than its own token count;
  • the a2e / e2a kernels move an equal share of rows per source rank, so a
    non-divisible fanout is refused rather than silently misrouted;
  • routing ids and scales are padded to that same uniform count on the send side.
    This was a live defect surfaced by the rebase: a sibling upstream change moved
    BF16 routing into the same payload using the unpadded count, so a2e indexed
    past the supplied rows -- the classic row-misplacement signature, output that
    varies between identical requests;
  • the FFN runner sizes xActiveMask / expand_x and its graph keys from the
    published rows, not the real token count, otherwise MoE dispatch fails with a
    mask/dimension mismatch and sibling ranks spin in vector-core wait.

7. Numerics and the correctness criterion

Digests are taken over choices[0].text only -- the response body carries a
random id and a created timestamp and is therefore never reproducible.

Topology Greedy digest Agreement
2Ax2F (static boot, and every post-resize visit) 00923c4faae9... 3 visits
2Ax1F (static control, and every resized visit to 1 F) ecf2a9ea057b... 4 visits

FFN rungs differ from the 2Ax2F baseline and match the same-topology static
control
. That equality is the criterion: output stable across probes that also
differs from a static boot of the same topology is what a resize that left the
wrong state looks like.

8. Measured cost of a switch

One host, one checkpoint, eager, 2Ax2F boot, --elastic-ep-max-dp-size 8,
--kv-cache-memory-bytes 2GiB. Windows are wall clock of the HTTP call, measured on
the ladder run and again on the final commit:

Rung Window Output vs baseline
Attention 2->4 73 s / 74.0 s byte-identical (3 probes, plus 3 on the re-run)
Attention 4->2 10 s / 10.6 s byte-identical
FFN 2->1 68 s differs from 2Ax2F, equals static 2Ax1F
FFN 1->2 66 s back to the 2Ax2F digest
refusals 18-39 ms service unaffected, no drain, no pause
FFN 2->1 / 1->2, re-run on 7f17cc9 25.7 s / 64.5 s probe output identical across all six probes
unsatisfiable FFN placement (device held by another job) 408 after 120.0 s, with the reason completions answered every 5 s through the wait; no degraded marker; the FFN pool was unchanged afterwards

Not instrumented: the per-phase split inside a window. The FFN figures are
explained by actor rebuild + weight reload (~15 s/rank, observed during pool
bring-up); the attention-up figure is not decomposed, and I would rather leave it
undecomposed than attribute it by guess.

9. Decisions taken, and the open questions

# Decision Status Open question for reviewers
D1 Correctness after a resize is judged against a same-topology static boot adopted; used by every measurement above is this the contract the project wants written into the elastic test suite?
D2 Ascend elastic is eager-only, refused during config validation implemented accepted as a restriction with a named unblock (static per-bucket row counts and an abort path), or should the combination fail loudly at first resize instead?
D3 Fail closed from the switch onward; reopen after a drain timeout; later resizes answered with the degraded reason implemented; the gate, the 503, the log line and the reservation refusal (408 with the reason, service answering throughout the wait) are measured; the wording fix is unit-tested is fencing on a hung-up client acceptable? After cancellation our own view may no longer describe the engines and there is no state probe; the alternative is probe-and-decide, which does not exist yet
D4 Statically-unsatisfiable requests are refused before the gate exists implemented, measured at 18-39 ms should the same refusal shape apply to role=ffn on non-elastic configs (today a plain 400)?

10. Remaining work, with owner

Item Owner Cost Hardware
ElasticEPState.abort_prepare() -- cancel an in-flight scale-up upstream vLLM not ours; plugin hookup ~1 day after it lands reuses the existing injection case
Graph modes + elastic on Ascend (static per-bucket row counts; graph-manager rebuild that survives boot) plugin 2 -3 weeks, dominated by boot->capture->crash iterations, and gated on the row above exclusive
CUDA hardware qualification of the shared/CUDA executor this design also carries plugin 1-2 days yes, pinned vllm==0.32.0
Move Ascend Ray-DP device assignment out of the elastic actor plugin small but cross-cutting boot smoke
Re-anchor the remaining main@544f182d patch markers onto the pinned 0.32.0 line plugin 1/2-1 day boot smoke
In-tree documentation, if wanted anywhere (a docs/npu/ page or the recipe tree) plugin 1/2 day none

Plugin boundary

Plugin-owned: the scaling client and its preflight; the FFN actor pool,
rendezvous and expert-layout reconstruction; per-generation topology identity; the
Ascend attention/FFN workers and model runners; the CAMP2P uniform row-count
protocol; the eager-only rule and the fail-closed boundary.

Compat helper: AFDElasticGPUExecutor / AFDElasticNPUExecutor (subclasses of
native ElasticEPScalingExecutor), AFDEngineCoreActor (native
DPMoEEngineCoreActor), AFDScalingMiddleware (native ScalingMiddleware).

Compat patch -- needed because native elastic EP reaches the scaled engine from
inside its own modules, where a subclass cannot intercept: role dispatch and the
drain/gate sequence around native prepare_elastic_ep / commit_elastic_ep
(compat/patches/elastic.py), AFD-aware Attention actor construction
(elastic_actors.py), the FFN engine-core busy loop and shutdown
(engine_core.py), config validation (config_validation.py), the parallel-state
swap (elastic_parallel.py), and Ascend stateless HCCL group creation
(patches/npu/stateless_pg.py). Each pins the upstream revision it copies, delimits
changed lines with PATCH START/END, is idempotent, and carries an upstream or
removal plan. The one I would most like reviewed is patches/npu/stateless_pg.py:
it is the deepest, and it exists only because the pinned Ascend worker registers one
global group per name.

Explicit class path: --worker-cls afd_plugin.v1.worker.npu.AFDNPUAttentionWorker, plus the vllm.endpoint_plugins
entry point afd_elastic for the route.

Layering wart. Ascend + Ray data-parallel device assignment works only because
the elastic Attention actor overrides _set_visible_devices: Ray reports Ascend
accelerators under an NPU resource, which the stock actor does not resolve.
Measured: keeping the Ray-DP recipe and dropping --enable-elastic-ep yields
workers that see zero devices (Invalid device ID / value 0 ... Expected value: [0, 0)), while the identical device list with elastic enabled boots to weight
loading. The community's static Ascend recipes do not use Ray DP at all, so this gap
is created by choosing EEP rather than by breaking an existing path -- but it means
"opt-in within the same package" is stronger on Ascend than the RFC implies, and
device assignment clearly belongs in the platform or executor layer, not in an
elastic-only actor.

Risks and alternatives

  1. Availability after a bad resize. Options: rollback -- built, measured,
    removed (it restores state but cannot cancel the in-flight group creation, so
    the deployment dies later while looking fine longer); fail closed -- chosen, and
    measured; upstream abort -- the actual fix, filed. There is no plugin-only option
    that is simultaneously available and trustworthy today.
  2. Eager-only on Ascend. The alternative is making the published row count
    static per capture bucket so dynamo can bound it. It interacts with the FFN
    runner's mask/graph-key sizing, and until an abort path exists a failure during
    post-resize re-capture is unrecoverable by construction -- so it should follow
    risk 1's fix, not precede it.
  3. A rank's teardown is global. Any AFD rank's shutdown() calls
    connector.close(), destroying process groups every other rank still uses, so one
    failure becomes a deployment-wide outage and survivors report
    Connection closed by peer -- an error naming someone else. It predates elastic
    AFD and is filed separately, and it is why reading the last traceback in a dead
    deployment wastes time: the discriminator is the first exception line and the
    process that raised it.
  4. Idle deployments self-destruct. The control-plane gloo recv carries a
    hardcoded 30-minute timeout on both backends; with every Attention engine paused
    nobody publishes metadata, the recv expires, that exception is fatal to the FFN
    loop, and its teardown destroys the shared groups. Thirty minutes of silence is
    therefore equivalent to a crash, and /health never reflects it. Both directions
    measured: silence bricks, while a probe every 8 minutes kept a deployment
    answering 8/8 over 64 minutes with byte-identical output. Not elastic-specific --
    the timeout is an unchanged context line in this work -- but it defines the
    operating envelope for every number above.
  5. One client, one gate. The scaling gate and the degraded flag live in the API
    process that owns the client. The recipe says one API process; nothing in this
    build enforces it with --api-server-count > 1, where each front end would hold
    its own view and two concurrent resizes could interleave. Either the launcher
    refuses elastic AFD with multiple API processes, or the gate moves behind shared
    state. Flagged rather than left to be discovered.
  6. Unvalidated combinations are refused, not assumed. Only
    P2pNcclAFDConnector and CAMP2pAFDConnector are accepted; async-CAM,
    DBO/microbatch overlap, TP>1, multi-node, speculative decoding and prefix-cache
    retention across a resize are unvalidated. Of these only the Ascend graph-mode
    refusal is enforced at startup; the rest are unvalidated rather than
    supported, which is the gap D2 has to be honest about.

Feedback period

Two weeks, or before graph-mode support on Ascend is scheduled: D2 and D3 decide
what that work can promise, and the device-assignment note decides whether Ascend
elastic can be strictly opt-in at all.

Implementation: #618. Two companion reports are deliberately not filed yet,
because filing both before a decision here would be premature: the missing abort for
an in-flight elastic-EP scale-up is upstream (vLLM), and "an idle control-plane recv
is fatal / a rank's teardown destroys shared groups" is a connector-level change.
Everything needed to file them is in Risks 3 and 4; say the word and I will open
both with the measured evidence attached.

CC list

The elastic AFD owners and the Ascend connector owners -- specifically for D1 (the
correctness contract), D3 (the cancellation rule) and the device-assignment
layering note.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    RFCRequest for comments

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions