Skip to content

feat(elastic): AFD elastic scaling on Ascend NPU - #618

Open
yzchang-plus wants to merge 1 commit into
vllm-project:rfc/elastic_scalingfrom
yzchang-plus:feat/npu-elastic-afd-rfc
Open

yzchang-plus wants to merge 1 commit into
vllm-project:rfc/elastic_scalingfrom
yzchang-plus:feat/npu-elastic-afd-rfc

Conversation

@yzchang-plus

@yzchang-plus yzchang-plus commented Nov 19, 2026 •

Copy link
Copy Markdown

Purpose

Elastic scaling for Attention-FFN Disaggregation (AFD) on Ascend NPU: resize the
Attention pool (DP 2->4->2) and the FFN pool (2->1->2) at runtime through the
existing /scale_elastic_ep route (role=attention|ffn), with requests
drained, DP pause consensus reached, topology switched, and service resumed
automatically. All NPU integration ships inside this plugin (runtime patch
modules plus worker/runner subclasses); no edits to the vLLM or vLLM-Ascend
source trees are required.

Design record. The reasoning, the component and resize-sequence diagrams, the
failure-ownership boundary and the decisions taken live in #620 (RFC: Elastic
AFD on Ascend NPU). This PR deliberately ships no in-tree documentation, so that
issue is where the design is described and argued; #485 is the original project RFC
that asked for independently resizable AFD pools, and nothing here restates it.

What this PR covers: the Ascend implementation, plus the shared CUDA executor it
necessarily carries -- AFDElasticNPUExecutor subclasses AFDElasticGPUExecutor,
so that file is on the Ascend code path whether or not a CUDA device exists. The
CUDA behaviour in it is CPU-tested and source-reviewed only. Graph modes,
DBO/microbatch overlap, async connectors, TP>1 and multi-node are out of scope; on
Ascend graph modes are refused at startup rather than half-supported.

Supersedes #523, which was based on main. This version is rebased onto
rfc/elastic_scaling, whose head is still 2bede02 (checked 2026-11-19), per the
staging branch created for this work.

rfc/elastic_scaling has no commits of its own (ahead 0 / behind 22 against
main at ae19c62), so this is the branch's first content. Both a rebase onto
current main and a merge of main into it were tried locally and conflict in
the same 7 files, all of them worker/runner files that the measured windows above
come from; a rebase therefore needs those runs repeated, and this PR keeps the
target it was cut for.

Issue

Scope

In scope

  • NPU elastic AFD on 910_93, TP=1, divisible A:F fanout, eager execution only.
  • Stateless HCCL group plumbing for standby topologies (per-group ids,
    generation-unique names, port windows, thread-local store clients), delivered
    as runtime patches.
  • CAMP2P uniform per-step row counts for N:1 fanout, and topology validation
    that rejects non-divisible A:F at the API boundary.
  • Rejections raised before the scaling gate exists, so an unsatisfiable
    request pauses nothing, drains nothing and opens no 503 window: the
    divisibility rule, --elastic-ep-max-dp-size, the native expert-redundancy
    minimum, and a new device-capacity check (Ray cluster report and the
    deployment's visible-device list, whichever is tighter; fails open when
    neither reports a count).
  • Startup rejection of graph mode + elastic on NPU (see Known limitations).
  • CPU unit suites, N0/N1 NPU qualification suites, and the NPU elastic recipe.

Out of scope

  • Non-divisible A:F topologies (e.g. 3Ax2F): the CAMP2P a2e/e2a kernels move a
    fixed share of rows per source rank, so an unequal fanout leaves sources
    unwindowed -- rejected with a 400 rather than silently misrouted.
  • Elastic TP>1.
  • Graph modes on NPU: refused at startup.
  • GPU hardware validation for this stack: the code path is shared, but the
    qualification here was run on Ascend.

Implementation Notes

Extension points; no upstream source was edited.

  • afd_plugin/elastic/gpu.py ships in this PR even though the CUDA hardware
    run is pending, because AFDElasticNPUExecutor subclasses
    AFDElasticGPUExecutor: the NPU path inherits the orchestration, so the file
    is on the NPU code path whether or not a CUDA device exists.
  • One divergence worth recording while it is cheap: the design targets
    ModelRunnerV2, and the Ascend path in this PR does run MRV2
    (v1/worker/npu/attention_model_runner_v2.py), but the CUDA build carried here
    is still the MRV1 one (VLLM_USE_V2_MODEL_RUNNER=0). That is inherited from the
    shared CUDA executor, not introduced here, and closing it belongs with that
    executor's CUDA qualification rather than being folded into an Ascend PR.
  • pyproject.toml registers exactly one vLLM entry point --
    vllm.endpoint_plugins: afd_elastic = afd_plugin.elastic.api:AFDElasticEndpointPlugin,
    which installs the /scale_elastic_ep route. Everything else activates through
    the plugin's existing patch registry. The only new dependency is msgspec
    (the control-payload codec already shared with the connectors).
  • Behaviour lives in plugin-owned classes: AFDElasticGPUExecutor /
    AFDElasticNPUExecutor (subclasses of upstream ElasticEPScalingExecutor),
    AFDEngineCoreActor, FFNActor / FFNActorManager, ElasticAFDClient,
    AFDScalingMiddleware, AFDElasticEndpointPlugin, and the NPU worker/runner
    subclasses under afd_plugin/v1/worker/npu/.
  • The parts that cannot be subclasses are runtime patches in
    afd_plugin/compat/patches/ (elastic.py, elastic_actors.py,
    elastic_parallel.py, config_validation.py, async_dp_engine.py,
    npu/stateless_pg.py, npu/ascend_config.py), because upstream reaches the
    scaled engine from inside its own modules. Each keeps the pinned upstream
    revision and the patch reason in its header, delimits changed lines with
    ### PATCH START/END, is idempotent on re-install, and is covered by CPU tests
    (tests/unit/compat/patches/, tests/unit/elastic/).

Intentional deviations from stock behaviour.

  • prepare_new_worker() is deliberately a no-op, _warm_target_groups() returns
    immediately, and warm_and_capture() runs under skip_dp_coordination():
    stock posts a standby warm-up collective that a joining AFD engine never
    matches, so exempting both sides is what turns the hang into a return.
  • config.py refuses graph modes on NPU at validation time rather than letting
    re-capture fail mid-resize.
  • Unsatisfiable requests are rejected in client.preflight() before the
    elastic-EP gate is set, so a 400 pauses nothing, drains nothing and opens no
    503 window.
  • The elastic Attention actor also carries Ascend Ray-DP device assignment
    (_set_visible_devices) because stock Ray-DP does not resolve the NPU
    accelerator resource. Device assignment is not an elastic concern; it is
    recorded as follow-up work in the linked issue instead of being silently
    restructured here.
  • connector.close() is left alone: a rank's shutdown still destroys groups
    shared with the rest of the deployment (Known limitation 3). Changing base
    teardown semantics inside an elastic PR would be the wrong scope.

Test Plan

Environment: single container on node73-148, 8x Ascend 910_93, Python 3.12.13,
vLLM tag v0.32.0 (5df63a72, editable), vLLM-Ascend 0cc580bb9
(PR#20321 head, which contains the bd42eea0 pinned in docs/installation.md),
CANN 9.1.0 (the installation table lists 9.2 as tested), DeepSeek-V2-Lite
W8A8, --enforce-eager, --elastic-ep-max-dp-size 8,
--kv-cache-memory-bytes 2GiB, 2Ax2F boot.

Commands, in the order run:

  1. pytest tests/unit -m "not gpu and not vllm_runtime" on a host without
    torch or vLLM (Python 3.12 + pytest only) -- the boundary AGENTS.md requires.
    Also the runtime-gated profile pytest tests/unit -m "not gpu and not npu".
  2. tests/npu/test_stateless_hccl_pg.py (N0) and
    tests/npu/test_eep_group_lifecycle.py (N1).
  3. Resize ladder over the live route, each rung followed by three identical
    greedy probes, plus the rejection paths.
  4. Same-topology control boot (static 2Ax1F) to judge FFN rungs -- see the
    criterion below.
  5. Deliberate failure injection (a target topology with more ranks than devices)
    and a graph-mode boot.

Steps 3-5 drove a live deployment through the /scale_elastic_ep route with
per-rung greedy probes and same-topology control boots. The throwaway driver
scripts for that are deliberately not in this PR (they owned no production
behaviour and would have needed their own maintenance); the permanent suites that
pin the same behaviours are tests/unit/elastic, tests/unit/connectors and
tests/npu. Ask me if a reproduction script would help a specific review point
and I will add one under tests/ with the maintenance rules it needs.
6. Failure-ownership cases on the live route: a drain timeout, and a failure
injected immediately after the switch begins (the natural over-capacity
failure can no longer be reached, because preflight refuses it before the
gate). Each outcome is also pinned by a CPU unit test.
7. Keep-alive soak: one greedy completion every 8 minutes, 8 rounds, on an
elastic 2Ax2F boot, compared against a silent 30-minute gap on the same
build. Digests are taken over choices[0].text only -- the response body
carries a random id and a created timestamp and is never reproducible.

Coverage follows docs/UT_MAINTENANCE.md: each behaviour this PR changes has a
test that fails when the change is removed, not only a test that passes.

Test Result

Check Result
N0 stateless HCCL process groups passed, 82.68s
N1 EEP standby-group lifecycle 2->4 passed, 79.07s -- 7 stages x 4 ranks
Attention 2->4 200, window 73s; 3/3 probes byte-identical to baseline
Attention 4->2 200, window 10s; byte-identical
FFN 2->1 200, window 68s; differs from the 2Ax2F baseline, identical to a static 2Ax1F boot (four independent visits now agree on ecf2a9ea057b...)
FFN 1->2 200, window 66s; back to the baseline digest
Attention 9 (> max_dp 8) / Attention 3 (non-divisible) 400 with the intended messages; service healthy afterwards
Attention 4 on a 4-device deployment (capacity preflight) 400 in 18ms, message that topology needs 6 devices but only 4 can be scheduled here; no drain, no pause, no store handshake started, and the deployment keeps serving afterwards
Keep-alive soak, elastic 2Ax2F, one completion every 8 min x 8 rounds 8/8 answered over 64 min (>2x the connector's 30-minute recv timeout), no timeout logged, and all 8 completions byte-identical -- the positive control for limitation 3
Failure ownership, drain timeout 408 Timed out draining elastic AFD requests; deployment kept serving, gate released
Failure ownership, injected mid-switch failure 503 with the reason on the failed resize; completions 503; one ERROR line logged; a second resize 503 (still worded "currently scaling, try again later" on the measured build); restart restored service. The wording fix is unit-tested + mutation-checked, not re-run on hardware
Failure ownership, unsatisfiable FFN placement 408 with the reason after 120.0 s (4Ax1F on six devices with the sixth held by a throwaway Ray placement group, so the capacity preflight passes on Ray's presence report; requesting ffn 2 then needs two bundles nobody can place); completions answered throughout the wait; no degraded marker; FFN pool unchanged
FFN 2->1 / 1->2 re-run on 7f17cc9 200, 25.7 s / 64.5 s; probe output identical across all six probes
Resize ladder re-run on the shipped commit 4506552 attention 2->4 200 in 74.0 s, 4->2 200 in 10.6 s; baseline + 6 probes byte-identical; is_scaling_elastic_ep false at both ends; no degraded log; pools back at 2 A / 2 F. This is the success path on the build that fails closed, i.e. the refactor did not disturb the normal exit

Digests: 2Ax2F 00923c4faae9..., 2Ax1F ecf2a9ea057b... (static boot == both
post-resize visits, 9 probes total).

How the FFN rungs are judged. An FFN resize moves experts and changes how many
rows each rank takes, so MC2 chunking and W8A8 accumulation order change and greedy
decoding can take a different continuation. The property that holds, and the one the
table above records, is a resized deployment matches the same topology booted
statically
. Output stable across probes but different from a same-topology
control would be the signature of a resize that left the wrong state.

Window lengths above replace the "~10s for FFN" figure quoted earlier: a kept FFN
actor cannot re-establish its HCCL communicators, so every NPU size change
rebuilds the actor pool and reloads weights (~15s per rank).

Known limitations (stated, not papered over)

  1. Graph modes are refused at startup on NPU. The N:1 fanout publishes one
    uniform row count per step taken from the runtime DP metadata, which dynamo
    cannot bound: two FULL_DECODE_ONLY runs returned 500 during re-capture of an
    FFN resize (Could not guard on data-dependent expression 16 < Max(16, u0, u1), then Pending unbacked symbols {...}). Supporting it means making the
    published count static per captured bucket.

  2. A resize that fails after the drain leaves the deployment unsafe to trust;
    restart it.
    Elastic EP creates the new DP group asynchronously inside
    EngineCore and the client's timeout does not cancel it, so an unsatisfiable
    request leaves a group creation in flight against a topology the plugin has
    already put back. Reproduced: 2A2F on four devices asked for attention 4 and
    hit the store timeout after the drain; the deployment then served correctly
    for ~30 minutes and one EngineCore exited with DistStoreError: wait timeout after 1800000ms from _prepare_future.result() -- the abandoned future
    failing on its own clock. A plugin-side rollback was implemented
    and measured (it restored the F actor pool, the configs and the paused
    schedulers) but it converts "dead immediately" into "dead later", so it was
    removed rather than shipped half-done.

    What this PR does instead is fail closed at the point the switch becomes
    destructive
    (compat/patches/elastic.py + controller.degraded): a
    preflight rejection opens no 503 window at all; a drain timeout reopens
    admission, because draining mutates nothing; and anything from
    controller.resize() onward -- including a client that hangs up mid-switch,
    which cancels our coroutine while the engines keep working -- leaves admission
    closed with a logged reason and refuses further resizes until restart. On hardware the
    gate, the 503 with its reason, the log line and the recovery-by-restart were
    all observed; what hardware also revealed is that the scaling middleware
    answers a later resize with the native "currently scaling, try again later"
    forever, which is wrong for a permanently degraded deployment, so the
    middleware now substitutes the degraded reason. That last piece is
    unit-tested and mutation-checked but has not been re-run on hardware.

    Cancellation counts as destructive too: a hung-up client cancels the
    orchestration coroutine while engine-side RPCs keep running, so the plugin's
    view of the topology may no longer describe the engines and this build has no
    probe to re-establish it. The alternative would need an engine-side state
    query that does not exist yet.
    The real fix is an abort path upstream; see the separate
    issue.

  3. An idle AFD deployment dies after 30 minutes of silence, and this branch
    is not why.
    The FFN connector loop blocks in a control-plane recv on a group
    the connector creates with a 30-minute timeout; with every Attention engine
    paused nobody publishes metadata, the recv expires, and that exception is
    fatal to the FFN loop -- whose teardown then destroys the process groups the
    surviving ranks are still using. Measured: last request 06:26:37, both FFN
    loops out at 06:56:37 with Timed out waiting 1800000ms for recv operation to complete, Attention Connection closed by peer at 07:56:47, /health
    answering 200 the whole time. Nothing in that chain consults elastic state:
    the 30-minute timeout is an unchanged context line in this diff and
    recv_dp_metadata_list is not touched by it at all, and the CUDA connector
    sets the same 30 minutes. So this PR neither introduces nor fixes it -- the
    fix belongs to the connector (an idle recv must not be fatal, and a rank's
    teardown must not destroy groups it does not exclusively own), which is the
    subject of a separate issue. Measured in the other direction too: an elastic
    2Ax2F boot probed every 8 minutes for 64 minutes -- twice the window --
    answered 8/8 and never logged a recv timeout, so the trigger is the silence,
    not elastic and not uptime. A flag-off control is not available here: this
    recipe cannot boot at all without --enable-elastic-ep on Ascend with the
    Ray DP backend, because resolving Ray's NPU accelerator resource lives in
    the elastic Attention actor's _set_visible_devices. That is a real
    layering problem (device assignment is not an elastic concern) and is
    recorded in the same issue as follow-up work, not smuggled into this PR.
    Both rules -- keep a completion arriving, and the discriminator against the
    unrelated 1800s store timeout of limitation 2 -- are in Operator notes
    below.

  4. Re-anchor main@544f182d markers in the patch modules predate this stack; the
    environment now validated is vLLM tag 5df63a72. One anchor was verified
    against 0.32.0 and corrected; the rest are re-anchoring work still owed.

  5. This host runs CANN 9.1.0 while docs/installation.md lists 9.2 as tested;
    the results above are 9.1.0 results.

  6. Static gates run on the author's machine: ruff check, ruff format,
    mypy-3.11, typos, markdownlint-cli2, shellcheck,
    check-spdx-header, check-yaml, check-merge-conflict,
    debug-statements, end-of-file-fixer, mixed-line-ending,
    trailing-whitespace, check-buildkite -- all passing over the PR range.
    actionlint matches no file in this diff (nothing under .github/
    changes); its hook environment did install on a retry and was run over the
    repository's two workflow files, where it passes -- so this is verified
    rather than assumed to be irrelevant.

Operator notes

Everything a deployment needs that is not in the repo as documentation:

  • Enable: VLLM_PLUGINS=afd,afd_elastic (without afd_elastic the route is
    never installed; the ascend platform entry stays as the NPU install sets it),
    --enable-elastic-ep, --elastic-ep-max-dp-size <N> (mandatory -- a resize
    beyond it is refused), --data-parallel-backend ray,
    --distributed-executor-backend uni, --enforce-eager (mandatory on NPU), an
    explicit --kv-cache-memory-bytes (--gpu-memory-utilization is not a
    substitute; the run above used 2147483648), --worker-cls afd_plugin.v1.worker.npu.AFDNPUAttentionWorker, and
    additional_config: {"afd": {"role": "attention", "connector": "CAMP2pAFDConnector", "num_attention_ranks": A, "num_ffn_ranks": F}, "afd_elastic": {"ffn_tensor_parallel_size": 1}}. TP=1 and A divisible by F.
  • Resize: POST /scale_elastic_ep {"new_data_parallel_size": N, "role": "attention"|"ffn", "drain_timeout": 120}; N is the selected role's DP size.
    POST /is_scaling_elastic_ep stays readable during a switch. Measured windows:
    attention up 73 s, attention down 10 s, FFN down 68 s, FFN up 66 s -- an NPU
    resize rebuilds the FFN actor pool and reloads weights (~15 s per rank),
    because a kept FFN rank cannot re-establish its HCCL communicators.
  • Keep traffic: an AFD deployment idle for 30 minutes bricks itself
    (pre-existing connector behaviour, limitation 3). Keep a real completion
    arriving at least every ~20 minutes; /health does not count, it never
    reaches the engines.
  • After a failed resize: a request refused before the gate costs nothing. A
    drain timeout (408) leaves the deployment serving. A failure from the switch
    onward leaves admission closed with a degraded reason and later resizes answer
    503 -- restart the deployment; 503 is the expected response until you do.
  • Correctness check after a resize: compare against the same topology
    booted statically
    , not against the pre-resize output.

Docs Impact

  • No in-tree documentation in this PR: nothing under docs/ changes. The rules a
    deployment needs are stated in Operator notes above, and a docs page can follow
    as a separate commit at whichever location the project prefers.
  • No default changes for existing deployments: elastic AFD engages only when
    both additional_config.afd_elastic and --enable-elastic-ep are present.

Files changed

59 files, +7217 / -88, one commit 7f17cc9 (base 2bede02 = upstream
rfc/elastic_scaling): afd_plugin/ and tests/ plus the pyproject.toml
entry point. Nothing else ships -- no docs/, no recipe/, and the throwaway
validation scripts that drove Test Plan steps 3-6 were never part of this
commit.


Essential PR Checklist
  • Purpose is clear and linked to public context when possible ([RFC]: Elastic AFD on Ascend NPU — resize semantics proven on hardware, and the decisions it forces #620, [RFC]: Elastic AFD: independently scale Attention and FFN capacity #485).
  • Scope is bounded: eager only, TP=1, divisible A:F, Ascend NPU; graph modes
    and non-divisible topologies are refused, not half-supported.
  • Compatibility with vLLM v0.32.0 and the pinned vLLM-Ascend source is
    considered: validated on tag 5df63a72 + vLLM-Ascend 0cc580bb9
    (which contains the bd42eea0 named in docs/installation.md), CANN
    9.1.0. The remaining Re-anchor main@544f182d markers are listed as
    Known limitation 4 rather than left implicit.
  • No changes are made to the vLLM source checkout.
  • Plugin-owned classes or explicit dotted class paths are preferred over
    monkey patches -- see Implementation Notes for the seven seams that cannot
    be classes and why.
  • Any compat shim or monkey patch is isolated, idempotent, version-guarded,
    documented, and tested.
  • Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated.
    tests/unit is green on a host with no torch and no vLLM, which is the
    boundary this claim needs. The NPU worker modules import the elastic
    lifecycle seams on first use, not at module scope.
  • Validation evidence is included, including skipped GPU tests when
    applicable: section B of the known-P1 tripwire needs a GPU host and is
    reported as not covered, not as green.
  • Documentation impact is stated: intentionally none in-tree; the rules
    an operator needs are in Operator notes, and I will add a page anywhere the
    maintainers prefer.

@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes NVIDIA NVIDIA GPU, CUDA, and related changes labels Nov 19, 2026

@jiangkuaixue123 jiangkuaixue123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The single-node eager ladder does not cover the CAMP2P wire-count mismatch under DBO or the placement failure paths below. Please address those before merging, and preserve the existing non-elastic graph and upstream configuration paths. This review is based on the pinned source; I did not run hardware E2E.

# smaller one (negative padding) and allocate a negative-sized tensor.
# The two conventions must not both act on the same step.
if self.attn_size > self.ffn_size and len(self.dp_metadata_list) <= 1:
uniform = _camp2p_uniform_batch(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep runtime DP counts out of the traced forward for static N:1 graphs. This single-stage branch also runs with elastic EP disabled. _camp2p_uniform_batch() calls .flatten().tolist() on the DP tensor, and the result controls Python max and the padding branch before the opaque send op. This is the data-dependent capture path the PR rejects for elastic NPU, but static CAMP2P FULL/FULL_DECODE_ONLY is an existing supported mode. Compute the wire count outside the trace or preserve the static graph path, and verify a static 2A×1F graph capture and replay.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: the branch I added runs with elastic EP disabled too, and it read
num_tokens_across_dp_cpu.flatten().tolist() inside the traced forward -- a
data-dependent value in exactly the path that static FULL/FULL_DECODE_ONLY
CAMP2P is expected to capture. That is a regression against a supported mode,
and it was the same failure signature I had only ever blamed on resize.

Fixed by moving the decision out of the trace instead of deleting it. The
eager control-plane update already runs on every step before the forward --
including the capture and replay publications, which both go through
send_dp_metadata -- so the uniform count is resolved there:

def update_state_from_dp_metadata(self, payload):
    connector.dp_metadata_list = self._physical_payload(payload).dp_metadata_list
    ...
    connector.uniform_tokens = {stage: count for stage in ... }   # ints

and send_attn_output only does self.uniform_tokens.get(ubatch_idx). Frozen
at capture time, which matches what this data path already does with padding
(send_attn_output_impl says so in its own comment).

Evidence, in order of strength:

  1. a unit test that poisons dp_metadata_list after the update (any attribute
    read raises) and still requires the send to declare the correct row count --
    so the traced region provably touches no DP tensor;
  2. on 910_93, a graph-mode boot (FULL_DECODE_ONLY, 2Ax2F, N:1 fanout) where
    the eager-only startup refusal was temporarily bypassed on the test host
    and reverted afterwards
    : capture completed
    (Graph capturing finished in 1 secs, took 0.33 GiB), zero
    unbacked / Could not guard on data-dependent lines in the log, and three
    identical requests returned identical text.

What that does not show, and what I do not want to imply: a resize under
graph mode. That is a different failure -- re-capture is the step that restores
the Attention runtime, and nothing cancels the in-flight scale-up inside
ElasticEPState, so it stays refused at startup. Two questions for you:
(a) with the capture path no longer data-dependent, should the rule narrow
from "eager only" to "eager for resizes, graphs allowed statically"? and (b) I
did not run the static two-process/local-DP recipe (this host's static graph
scripts all predate elastic AFD); if you want that specific configuration
demonstrated rather than the capture mechanism, say so and I will build it.

Comment thread afd_plugin/elastic/ffn.py Resolved
Comment thread afd_plugin/elastic/client.py Outdated
return
self.preflight(size, role)
await self.prepare()
host, (link_port, role_port) = await self.ffn.rendezvous()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Select the rendezvous on the FFN rank zero that will actually bind it. For an NPU FFN resize, resize_actors() kills all existing F actors and allocates fresh placement groups with no affinity to the old F0 node. If Ray puts the replacement F0 on another node, all ranks still receive the old host/ports selected here and cannot rendezvous. For the FFN branch, create/select the target actors first, then ask the new F0 for its endpoint before publishing the configs (or explicitly enforce single-node placement if that is the intended contract).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and single-node tests could not see it: on Ascend the F-role branch
kills every FFN actor and allocates fresh placement groups, so asking for the
endpoint before spawn() publishes an address whose owner no longer exists.

The F-role branch now orders it as release -> spawn -> rendezvous -> publish, so
the host and both ports come from the rank zero that will actually bind the
stores. The Attention branch is unchanged (its F pool keeps its identity), and
I added a test for that too, so the fix cannot silently widen into a behaviour
change for A resizes.

Tests: a double whose rank-0 endpoint changes with every spawn() -- the
published A topology must carry the post-spawn host, and the event order must
be spawn before afd_update_topology. Both fail if the rendezvous moves back
up. test_attention_resize_keeps_the_existing_rank_zero guards the other side.

On scope: multi-node scaling is deliberately not part of this round -- the PR
envelope is single-node, and the reordering is what removes the defect for when
that envelope widens. I have not exercised a cross-node run and do not claim it.
If you would rather the single-node envelope be enforced explicitly (rejecting
a multi-node elastic AFD launch at config time), I will add the check; otherwise
it stays an assumption of the launch recipe, as today.

# See https://github.com/pytorch/pytorch/issues/174288
from vllm.distributed.nixl_utils import is_nixl_available

if is_nixl_available():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the pinned XPU EPLB default. ParallelConfig.__post_init__ at vLLM 5df63a72 selects torch_xccl first on XPU; this copy falls through to nixl, pynccl, or torch_gloo instead. The replacement is installed by general-plugin registration, so even an ordinary non-AFD XPU model with EPLB and no explicit communicator changes backend. Restore the upstream current_platform.is_xpu() branch outside the AFD delta.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed -- a copy defect, and the worst kind, because the replacement is
installed for every process by general-plugin registration: a non-AFD XPU
deployment with EPLB and no explicit communicator would silently move off
torch_xccl. Restored verbatim from 5df63a72:

if current_platform.is_xpu():
    self.eplb_config.communicator = "torch_xccl"
elif is_nixl_available(): ...

tests/unit/elastic/test_parallel_patch.py pins the whole ladder (5
parametrised cases: XPU with and without NIXL, NIXL only, elastic EP without
NIXL, neither), which is the first test this seam ever had. Deleting the branch
fails exactly the two XPU cases.

This also strengthens the case for the re-anchoring work listed as a known
limitation: the seam's header still says "copied from vLLM 0.30.0", and this is
a 0.32 difference that survived it. I would like to close that by re-copying
each seam against 5df63a72 with a fidelity test per seam, in a follow-up that
does not mix into this branch's functional review -- unless you would rather it
land first.

Comment thread docs/npu/ELASTIC_AFD_NPU.md Resolved
  Scale the attention or FFN pool in and out at runtime through the
  native /scale_elastic_ep endpoint, with the drain -> DP pause -> STOP
  -> switch -> resume lifecycle driving both roles.

  On Ascend NPU the stock rendezvous cannot build process groups for
  standby ranks, so the runtime patches install stateless HCCL group
  creation with per-group ids, generation-unique group names and port
  windows, thread-local store clients, and version-anchored worker/
  runner seams. CAMP2P topologies are validated against the fixed N:1
  uniform-row fanout the transport encodes; elastic is scoped to the
  sync CAMP2P plane and the validator rejects it for async CAM.

  Requests that pass topology validation but violate the native EP
  bounds -- both --elastic-ep-max-dp-size and the minimum implied by
  fixed physical experts per engine -- are refused before any pause,
  STOP, connector release or config rewrite. The elastic finish path
  reuses the KV allocation its runner already applied instead of
  re-entering initialize_from_config, so a device-sized KV budget is
  never resident twice during startup or a resize, and the CUDA
  resize rebuilds the V2 runner the attention worker ships.

  Includes CPU unit suites, the N0/N1 NPU qualification suites, and the
  NPU elastic serving recipe.

Rebased onto rfc/elastic_scaling. Two integration fixes came with the move:
b33e20d added the Attention-side routing branch beside the uniform row-count
publication, so routing now pads to the row count the transfer declares instead
of the unpadded token count -- expecting 10 rows where 14 were published is the
row-misplacement case itself, the kernel writes a fixed share per source. The
BF16 routing receive test inherited from that commit asserted the pre-uniform
count and is corrected for the same reason. b33e20d also turned camp2p into a
package whose __init__ resolves only its public __all__, so the CPU token-count
test now imports _num_tokens_for_ffn_rank from the connector module.

Closes both lifecycle findings from the PR vllm-project#523 review. release_link now drops
the MRV2 captured graphs through native release_graphs() before the connector
closes, and the standby-warm exemption plus the warm_and_capture DP-coordination
bypass moved onto the hardware-neutral executor: the joining engine cannot post
a matching leg for that all_reduce without deadlocking on
sync_kv_cache_memory_size, so deferral on both sides is the only structural
option (ROCm carries the same guard upstream, and AFD runs no MoE on A). NPU
graph modes are refused at startup instead of carrying NPU-only graph
bookkeeping that nothing could validate.

Rejects graph modes for elastic AFD on Ascend. The N:1 fanout publishes one
uniform row count per step taken from the runtime DP metadata, so dynamo cannot
bound it: two FULL_DECODE_ONLY runs on 910_93 returned 500 during re-capture of
an FFN resize. Re-capture is also the step that restores the Attention runtime,
so no rollback recovers from it, and the deployment went on accepting requests
while answering none. The startup check turns that into an error before the
engines exist. GPU keeps graph modes, where the published count is not
data-dependent. Elastic resize failures after the drain still require a
restart; the in-flight new-DP-group future belongs to upstream elastic EP, which
has no abort path, and that is recorded as a limitation rather than papered over.

Restored the CPU-only boundary tests/unit guarantees. Importing the NPU workers
pulled the elastic lifecycle, which binds torch, torch_npu and the vLLM graph at
import, so 15 upstream async-CAM wiring tests could not reach their fixtures on
a host without the Ascend runtime; the lifecycle seams are now imported on first
use inside the worker methods. The stateless-PG import moved to its point of use
in apply_afd_ascend_patches_if_needed, keeping the config and MLA seams
testable, and tests/unit/compat/test_runtime stubs that seam the way it already
stubs the routing-id op. Two elastic CPU tests got the doubles their subjects
now need -- a vllm.logger stub in test_seams, a platform double in test_ffn --
rather than being skipped, and the lifecycle fixture's doubles now carry a
ModuleSpec, without which any find_spec() on them raises and no test in that
file can be selected on its own.

Validated on Ascend 910_93 with vLLM tag v0.32.0 (5df63a72) and vllm-ascend
0cc580bb9, CANN 9.1.0, eager: N0 and N1 pass, Attention 2->4 and 4->2 are
byte-identical (74s and 9s windows), FFN 2->1 differs from the 2A2F baseline but
matches a static 2A1F boot across three independent visits (69s and 94s
windows), and both the over-bound and non-divisible requests return 400 with the
service healthy afterwards. Matching a same-topology static boot, not the
pre-resize baseline, is the criterion FFN rungs are judged by on this stack.

  Preflights every rejection before the scaling gate. scale() now refuses
  unsatisfiable requests -- the divisibility rule, the --elastic-ep-max-dp-size
  bound, the native expert-redundancy minimum, and a new device-capacity check --
  while the gate is still down, so a rejected request pauses nothing, drains
  nothing and opens no 503 window. The capacity check exists because the
  drain-then-timeout path was reproduced destroying a deployment: 2A2F on four
  devices asked for attention 4, the store handshake never saw the two
  unschedulable engines, and thirty minutes later the abandoned new-DP-group
  future killed an EngineCore. The check takes the tighter of two ceilings --
  Ray's cluster report and the deployment's visible-device list -- because Ray
  counts every accelerator on the node while the launch script may pin the
  server to a subset, and the cluster number alone waved exactly the impossible
  scale-up through that later killed an engine. It fails open when neither
  source reports a usable count, because a missing report must not refuse a
  request that would otherwise have worked. Also records, in the NPU guide,
  what a failure after
  prepare() means and the operational rule that follows from it.

  Pins the standby-warm exemption with a unit test. Those two overrides were
  shipped without coverage, which made "closed" a claim rather than a check:
  test_elastic_executor_defers_the_unpaired_standby_warm fails if either override
  is dropped. Its limit is stated in the test itself -- the harness doubles the
  native parent, so it pins the plugin-side behaviour and cannot replay stock's
  collective.

Signed-off-by: yzchang-plus <1078477584@qq.com>
@yzchang-plus
yzchang-plus force-pushed the feat/npu-elastic-afd-rfc branch from 9ff6d00 to 7f17cc9 Compare November 20, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants