Repository navigation
feat(elastic): AFD elastic scaling on Ascend NPU - #618
yzchang-plus wants to merge 1 commit into
Conversation
jiangkuaixue123
left a comment
There was a problem hiding this comment.
The single-node eager ladder does not cover the CAMP2P wire-count mismatch under DBO or the placement failure paths below. Please address those before merging, and preserve the existing non-elastic graph and upstream configuration paths. This review is based on the pinned source; I did not run hardware E2E.
| # smaller one (negative padding) and allocate a negative-sized tensor. | ||
| # The two conventions must not both act on the same step. | ||
| if self.attn_size > self.ffn_size and len(self.dp_metadata_list) <= 1: | ||
| uniform = _camp2p_uniform_batch( |
There was a problem hiding this comment.
[P1] Keep runtime DP counts out of the traced forward for static N:1 graphs. This single-stage branch also runs with elastic EP disabled. _camp2p_uniform_batch() calls .flatten().tolist() on the DP tensor, and the result controls Python max and the padding branch before the opaque send op. This is the data-dependent capture path the PR rejects for elastic NPU, but static CAMP2P FULL/FULL_DECODE_ONLY is an existing supported mode. Compute the wire count outside the trace or preserve the static graph path, and verify a static 2A×1F graph capture and replay.
There was a problem hiding this comment.
Confirmed: the branch I added runs with elastic EP disabled too, and it read
num_tokens_across_dp_cpu.flatten().tolist() inside the traced forward -- a
data-dependent value in exactly the path that static FULL/FULL_DECODE_ONLY
CAMP2P is expected to capture. That is a regression against a supported mode,
and it was the same failure signature I had only ever blamed on resize.
Fixed by moving the decision out of the trace instead of deleting it. The
eager control-plane update already runs on every step before the forward --
including the capture and replay publications, which both go through
send_dp_metadata -- so the uniform count is resolved there:
def update_state_from_dp_metadata(self, payload):
connector.dp_metadata_list = self._physical_payload(payload).dp_metadata_list
...
connector.uniform_tokens = {stage: count for stage in ... } # intsand send_attn_output only does self.uniform_tokens.get(ubatch_idx). Frozen
at capture time, which matches what this data path already does with padding
(send_attn_output_impl says so in its own comment).
Evidence, in order of strength:
- a unit test that poisons
dp_metadata_listafter the update (any attribute
read raises) and still requires the send to declare the correct row count --
so the traced region provably touches no DP tensor; - on 910_93, a graph-mode boot (
FULL_DECODE_ONLY, 2Ax2F, N:1 fanout) where
the eager-only startup refusal was temporarily bypassed on the test host
and reverted afterwards: capture completed
(Graph capturing finished in 1 secs, took 0.33 GiB), zero
unbacked/Could not guard on data-dependentlines in the log, and three
identical requests returned identical text.
What that does not show, and what I do not want to imply: a resize under
graph mode. That is a different failure -- re-capture is the step that restores
the Attention runtime, and nothing cancels the in-flight scale-up inside
ElasticEPState, so it stays refused at startup. Two questions for you:
(a) with the capture path no longer data-dependent, should the rule narrow
from "eager only" to "eager for resizes, graphs allowed statically"? and (b) I
did not run the static two-process/local-DP recipe (this host's static graph
scripts all predate elastic AFD); if you want that specific configuration
demonstrated rather than the capture mechanism, say so and I will build it.
| return | ||
| self.preflight(size, role) | ||
| await self.prepare() | ||
| host, (link_port, role_port) = await self.ffn.rendezvous() |
There was a problem hiding this comment.
[P1] Select the rendezvous on the FFN rank zero that will actually bind it. For an NPU FFN resize, resize_actors() kills all existing F actors and allocates fresh placement groups with no affinity to the old F0 node. If Ray puts the replacement F0 on another node, all ranks still receive the old host/ports selected here and cannot rendezvous. For the FFN branch, create/select the target actors first, then ask the new F0 for its endpoint before publishing the configs (or explicitly enforce single-node placement if that is the intended contract).
There was a problem hiding this comment.
Confirmed, and single-node tests could not see it: on Ascend the F-role branch
kills every FFN actor and allocates fresh placement groups, so asking for the
endpoint before spawn() publishes an address whose owner no longer exists.
The F-role branch now orders it as release -> spawn -> rendezvous -> publish, so
the host and both ports come from the rank zero that will actually bind the
stores. The Attention branch is unchanged (its F pool keeps its identity), and
I added a test for that too, so the fix cannot silently widen into a behaviour
change for A resizes.
Tests: a double whose rank-0 endpoint changes with every spawn() -- the
published A topology must carry the post-spawn host, and the event order must
be spawn before afd_update_topology. Both fail if the rendezvous moves back
up. test_attention_resize_keeps_the_existing_rank_zero guards the other side.
On scope: multi-node scaling is deliberately not part of this round -- the PR
envelope is single-node, and the reordering is what removes the defect for when
that envelope widens. I have not exercised a cross-node run and do not claim it.
If you would rather the single-node envelope be enforced explicitly (rejecting
a multi-node elastic AFD launch at config time), I will add the check; otherwise
it stays an assumption of the launch recipe, as today.
| # See https://github.com/pytorch/pytorch/issues/174288 | ||
| from vllm.distributed.nixl_utils import is_nixl_available | ||
|
|
||
| if is_nixl_available(): |
There was a problem hiding this comment.
[P2] Preserve the pinned XPU EPLB default. ParallelConfig.__post_init__ at vLLM 5df63a72 selects torch_xccl first on XPU; this copy falls through to nixl, pynccl, or torch_gloo instead. The replacement is installed by general-plugin registration, so even an ordinary non-AFD XPU model with EPLB and no explicit communicator changes backend. Restore the upstream current_platform.is_xpu() branch outside the AFD delta.
There was a problem hiding this comment.
Confirmed -- a copy defect, and the worst kind, because the replacement is
installed for every process by general-plugin registration: a non-AFD XPU
deployment with EPLB and no explicit communicator would silently move off
torch_xccl. Restored verbatim from 5df63a72:
if current_platform.is_xpu():
self.eplb_config.communicator = "torch_xccl"
elif is_nixl_available(): ...tests/unit/elastic/test_parallel_patch.py pins the whole ladder (5
parametrised cases: XPU with and without NIXL, NIXL only, elastic EP without
NIXL, neither), which is the first test this seam ever had. Deleting the branch
fails exactly the two XPU cases.
This also strengthens the case for the re-anchoring work listed as a known
limitation: the seam's header still says "copied from vLLM 0.30.0", and this is
a 0.32 difference that survived it. I would like to close that by re-copying
each seam against 5df63a72 with a fidelity test per seam, in a follow-up that
does not mix into this branch's functional review -- unless you would rather it
land first.
c6d2bee to
4506552
Compare
59ee211 to
9ff6d00
Compare
Scale the attention or FFN pool in and out at runtime through the native /scale_elastic_ep endpoint, with the drain -> DP pause -> STOP -> switch -> resume lifecycle driving both roles. On Ascend NPU the stock rendezvous cannot build process groups for standby ranks, so the runtime patches install stateless HCCL group creation with per-group ids, generation-unique group names and port windows, thread-local store clients, and version-anchored worker/ runner seams. CAMP2P topologies are validated against the fixed N:1 uniform-row fanout the transport encodes; elastic is scoped to the sync CAMP2P plane and the validator rejects it for async CAM. Requests that pass topology validation but violate the native EP bounds -- both --elastic-ep-max-dp-size and the minimum implied by fixed physical experts per engine -- are refused before any pause, STOP, connector release or config rewrite. The elastic finish path reuses the KV allocation its runner already applied instead of re-entering initialize_from_config, so a device-sized KV budget is never resident twice during startup or a resize, and the CUDA resize rebuilds the V2 runner the attention worker ships. Includes CPU unit suites, the N0/N1 NPU qualification suites, and the NPU elastic serving recipe. Rebased onto rfc/elastic_scaling. Two integration fixes came with the move: b33e20d added the Attention-side routing branch beside the uniform row-count publication, so routing now pads to the row count the transfer declares instead of the unpadded token count -- expecting 10 rows where 14 were published is the row-misplacement case itself, the kernel writes a fixed share per source. The BF16 routing receive test inherited from that commit asserted the pre-uniform count and is corrected for the same reason. b33e20d also turned camp2p into a package whose __init__ resolves only its public __all__, so the CPU token-count test now imports _num_tokens_for_ffn_rank from the connector module. Closes both lifecycle findings from the PR vllm-project#523 review. release_link now drops the MRV2 captured graphs through native release_graphs() before the connector closes, and the standby-warm exemption plus the warm_and_capture DP-coordination bypass moved onto the hardware-neutral executor: the joining engine cannot post a matching leg for that all_reduce without deadlocking on sync_kv_cache_memory_size, so deferral on both sides is the only structural option (ROCm carries the same guard upstream, and AFD runs no MoE on A). NPU graph modes are refused at startup instead of carrying NPU-only graph bookkeeping that nothing could validate. Rejects graph modes for elastic AFD on Ascend. The N:1 fanout publishes one uniform row count per step taken from the runtime DP metadata, so dynamo cannot bound it: two FULL_DECODE_ONLY runs on 910_93 returned 500 during re-capture of an FFN resize. Re-capture is also the step that restores the Attention runtime, so no rollback recovers from it, and the deployment went on accepting requests while answering none. The startup check turns that into an error before the engines exist. GPU keeps graph modes, where the published count is not data-dependent. Elastic resize failures after the drain still require a restart; the in-flight new-DP-group future belongs to upstream elastic EP, which has no abort path, and that is recorded as a limitation rather than papered over. Restored the CPU-only boundary tests/unit guarantees. Importing the NPU workers pulled the elastic lifecycle, which binds torch, torch_npu and the vLLM graph at import, so 15 upstream async-CAM wiring tests could not reach their fixtures on a host without the Ascend runtime; the lifecycle seams are now imported on first use inside the worker methods. The stateless-PG import moved to its point of use in apply_afd_ascend_patches_if_needed, keeping the config and MLA seams testable, and tests/unit/compat/test_runtime stubs that seam the way it already stubs the routing-id op. Two elastic CPU tests got the doubles their subjects now need -- a vllm.logger stub in test_seams, a platform double in test_ffn -- rather than being skipped, and the lifecycle fixture's doubles now carry a ModuleSpec, without which any find_spec() on them raises and no test in that file can be selected on its own. Validated on Ascend 910_93 with vLLM tag v0.32.0 (5df63a72) and vllm-ascend 0cc580bb9, CANN 9.1.0, eager: N0 and N1 pass, Attention 2->4 and 4->2 are byte-identical (74s and 9s windows), FFN 2->1 differs from the 2A2F baseline but matches a static 2A1F boot across three independent visits (69s and 94s windows), and both the over-bound and non-divisible requests return 400 with the service healthy afterwards. Matching a same-topology static boot, not the pre-resize baseline, is the criterion FFN rungs are judged by on this stack. Preflights every rejection before the scaling gate. scale() now refuses unsatisfiable requests -- the divisibility rule, the --elastic-ep-max-dp-size bound, the native expert-redundancy minimum, and a new device-capacity check -- while the gate is still down, so a rejected request pauses nothing, drains nothing and opens no 503 window. The capacity check exists because the drain-then-timeout path was reproduced destroying a deployment: 2A2F on four devices asked for attention 4, the store handshake never saw the two unschedulable engines, and thirty minutes later the abandoned new-DP-group future killed an EngineCore. The check takes the tighter of two ceilings -- Ray's cluster report and the deployment's visible-device list -- because Ray counts every accelerator on the node while the launch script may pin the server to a subset, and the cluster number alone waved exactly the impossible scale-up through that later killed an engine. It fails open when neither source reports a usable count, because a missing report must not refuse a request that would otherwise have worked. Also records, in the NPU guide, what a failure after prepare() means and the operational rule that follows from it. Pins the standby-warm exemption with a unit test. Those two overrides were shipped without coverage, which made "closed" a claim rather than a check: test_elastic_executor_defers_the_unpaired_standby_warm fails if either override is dropped. Its limit is stated in the test itself -- the harness doubles the native parent, so it pins the plugin-side behaviour and cannot replay stock's collective. Signed-off-by: yzchang-plus <1078477584@qq.com>
9ff6d00 to
7f17cc9
Compare
Purpose
Elastic scaling for Attention-FFN Disaggregation (AFD) on Ascend NPU: resize the
Attention pool (DP 2->4->2) and the FFN pool (2->1->2) at runtime through the
existing
/scale_elastic_eproute (role=attention|ffn), with requestsdrained, DP pause consensus reached, topology switched, and service resumed
automatically. All NPU integration ships inside this plugin (runtime patch
modules plus worker/runner subclasses); no edits to the vLLM or vLLM-Ascend
source trees are required.
Design record. The reasoning, the component and resize-sequence diagrams, the
failure-ownership boundary and the decisions taken live in #620 (RFC: Elastic
AFD on Ascend NPU). This PR deliberately ships no in-tree documentation, so that
issue is where the design is described and argued; #485 is the original project RFC
that asked for independently resizable AFD pools, and nothing here restates it.
What this PR covers: the Ascend implementation, plus the shared CUDA executor it
necessarily carries --
AFDElasticNPUExecutorsubclassesAFDElasticGPUExecutor,so that file is on the Ascend code path whether or not a CUDA device exists. The
CUDA behaviour in it is CPU-tested and source-reviewed only. Graph modes,
DBO/microbatch overlap, async connectors, TP>1 and multi-node are out of scope; on
Ascend graph modes are refused at startup rather than half-supported.
Supersedes #523, which was based on
main. This version is rebased ontorfc/elastic_scaling, whose head is still2bede02(checked 2026-11-19), per thestaging branch created for this work.
rfc/elastic_scalinghas no commits of its own (ahead 0 / behind 22againstmainatae19c62), so this is the branch's first content. Both a rebase ontocurrent
mainand a merge ofmaininto it were tried locally and conflict inthe same 7 files, all of them worker/runner files that the measured windows above
come from; a rebase therefore needs those runs repeated, and this PR keeps the
target it was cut for.
Issue
this work) and [RFC]: Elastic AFD: independently scale Attention and FFN capacity #485 (the original project RFC for elastic AFD)
scope, and one upstream gap is listed under Known limitations.
Scope
In scope
generation-unique names, port windows, thread-local store clients), delivered
as runtime patches.
that rejects non-divisible A:F at the API boundary.
request pauses nothing, drains nothing and opens no 503 window: the
divisibility rule,
--elastic-ep-max-dp-size, the native expert-redundancyminimum, and a new device-capacity check (Ray cluster report and the
deployment's visible-device list, whichever is tighter; fails open when
neither reports a count).
graph mode + elasticon NPU (see Known limitations).Out of scope
fixed share of rows per source rank, so an unequal fanout leaves sources
unwindowed -- rejected with a 400 rather than silently misrouted.
qualification here was run on Ascend.
Implementation Notes
Extension points; no upstream source was edited.
afd_plugin/elastic/gpu.pyships in this PR even though the CUDA hardwarerun is pending, because
AFDElasticNPUExecutorsubclassesAFDElasticGPUExecutor: the NPU path inherits the orchestration, so the fileis on the NPU code path whether or not a CUDA device exists.
ModelRunnerV2, and the Ascend path in this PR does run MRV2
(
v1/worker/npu/attention_model_runner_v2.py), but the CUDA build carried hereis still the MRV1 one (
VLLM_USE_V2_MODEL_RUNNER=0). That is inherited from theshared CUDA executor, not introduced here, and closing it belongs with that
executor's CUDA qualification rather than being folded into an Ascend PR.
pyproject.tomlregisters exactly one vLLM entry point --vllm.endpoint_plugins: afd_elastic = afd_plugin.elastic.api:AFDElasticEndpointPlugin,which installs the
/scale_elastic_eproute. Everything else activates throughthe plugin's existing patch registry. The only new dependency is
msgspec(the control-payload codec already shared with the connectors).
AFDElasticGPUExecutor/AFDElasticNPUExecutor(subclasses of upstreamElasticEPScalingExecutor),AFDEngineCoreActor,FFNActor/FFNActorManager,ElasticAFDClient,AFDScalingMiddleware,AFDElasticEndpointPlugin, and the NPU worker/runnersubclasses under
afd_plugin/v1/worker/npu/.afd_plugin/compat/patches/(elastic.py,elastic_actors.py,elastic_parallel.py,config_validation.py,async_dp_engine.py,npu/stateless_pg.py,npu/ascend_config.py), because upstream reaches thescaled engine from inside its own modules. Each keeps the pinned upstream
revision and the patch reason in its header, delimits changed lines with
### PATCH START/END, is idempotent on re-install, and is covered by CPU tests(
tests/unit/compat/patches/,tests/unit/elastic/).Intentional deviations from stock behaviour.
prepare_new_worker()is deliberately a no-op,_warm_target_groups()returnsimmediately, and
warm_and_capture()runs underskip_dp_coordination():stock posts a standby warm-up collective that a joining AFD engine never
matches, so exempting both sides is what turns the hang into a return.
config.pyrefuses graph modes on NPU at validation time rather than lettingre-capture fail mid-resize.
client.preflight()before theelastic-EP gate is set, so a 400 pauses nothing, drains nothing and opens no
503 window.
(
_set_visible_devices) because stock Ray-DP does not resolve theNPUaccelerator resource. Device assignment is not an elastic concern; it is
recorded as follow-up work in the linked issue instead of being silently
restructured here.
connector.close()is left alone: a rank's shutdown still destroys groupsshared with the rest of the deployment (Known limitation 3). Changing base
teardown semantics inside an elastic PR would be the wrong scope.
Test Plan
Environment: single container on node73-148, 8x Ascend 910_93, Python 3.12.13,
vLLM tag v0.32.0 (
5df63a72, editable), vLLM-Ascend0cc580bb9(PR#20321 head, which contains the
bd42eea0pinned indocs/installation.md),CANN 9.1.0 (the installation table lists 9.2 as tested), DeepSeek-V2-Lite
W8A8,
--enforce-eager,--elastic-ep-max-dp-size 8,--kv-cache-memory-bytes 2GiB, 2Ax2F boot.Commands, in the order run:
pytest tests/unit -m "not gpu and not vllm_runtime"on a host withouttorch or vLLM (Python 3.12 + pytest only) -- the boundary AGENTS.md requires.
Also the runtime-gated profile
pytest tests/unit -m "not gpu and not npu".tests/npu/test_stateless_hccl_pg.py(N0) andtests/npu/test_eep_group_lifecycle.py(N1).greedy probes, plus the rejection paths.
criterion below.
and a graph-mode boot.
Steps 3-5 drove a live deployment through the
/scale_elastic_eproute withper-rung greedy probes and same-topology control boots. The throwaway driver
scripts for that are deliberately not in this PR (they owned no production
behaviour and would have needed their own maintenance); the permanent suites that
pin the same behaviours are
tests/unit/elastic,tests/unit/connectorsandtests/npu. Ask me if a reproduction script would help a specific review pointand I will add one under
tests/with the maintenance rules it needs.6. Failure-ownership cases on the live route: a drain timeout, and a failure
injected immediately after the switch begins (the natural over-capacity
failure can no longer be reached, because preflight refuses it before the
gate). Each outcome is also pinned by a CPU unit test.
7. Keep-alive soak: one greedy completion every 8 minutes, 8 rounds, on an
elastic 2Ax2F boot, compared against a silent 30-minute gap on the same
build. Digests are taken over
choices[0].textonly -- the response bodycarries a random
idand acreatedtimestamp and is never reproducible.Coverage follows
docs/UT_MAINTENANCE.md: each behaviour this PR changes has atest that fails when the change is removed, not only a test that passes.
Test Result
ecf2a9ea057b...)max_dp8) / Attention 3 (non-divisible)that topology needs 6 devices but only 4 can be scheduled here; no drain, no pause, no store handshake started, and the deployment keeps serving afterwardsTimed out draining elastic AFD requests; deployment kept serving, gate releasedffn 2then needs two bundles nobody can place); completions answered throughout the wait; no degraded marker; FFN pool unchanged7f17cc94506552is_scaling_elastic_epfalse at both ends; no degraded log; pools back at 2 A / 2 F. This is the success path on the build that fails closed, i.e. the refactor did not disturb the normal exitDigests: 2Ax2F
00923c4faae9..., 2Ax1Fecf2a9ea057b...(static boot == bothpost-resize visits, 9 probes total).
How the FFN rungs are judged. An FFN resize moves experts and changes how many
rows each rank takes, so MC2 chunking and W8A8 accumulation order change and greedy
decoding can take a different continuation. The property that holds, and the one the
table above records, is a resized deployment matches the same topology booted
statically. Output stable across probes but different from a same-topology
control would be the signature of a resize that left the wrong state.
Window lengths above replace the "~10s for FFN" figure quoted earlier: a kept FFN
actor cannot re-establish its HCCL communicators, so every NPU size change
rebuilds the actor pool and reloads weights (~15s per rank).
Known limitations (stated, not papered over)
Graph modes are refused at startup on NPU. The N:1 fanout publishes one
uniform row count per step taken from the runtime DP metadata, which dynamo
cannot bound: two
FULL_DECODE_ONLYruns returned 500 during re-capture of anFFN resize (
Could not guard on data-dependent expression 16 < Max(16, u0, u1), thenPending unbacked symbols {...}). Supporting it means making thepublished count static per captured bucket.
A resize that fails after the drain leaves the deployment unsafe to trust;
restart it. Elastic EP creates the new DP group asynchronously inside
EngineCore and the client's timeout does not cancel it, so an unsatisfiable
request leaves a group creation in flight against a topology the plugin has
already put back. Reproduced: 2A2F on four devices asked for attention 4 and
hit the store timeout after the drain; the deployment then served correctly
for ~30 minutes and one EngineCore exited with
DistStoreError: wait timeout after 1800000msfrom_prepare_future.result()-- the abandoned futurefailing on its own clock. A plugin-side rollback was implemented
and measured (it restored the F actor pool, the configs and the paused
schedulers) but it converts "dead immediately" into "dead later", so it was
removed rather than shipped half-done.
What this PR does instead is fail closed at the point the switch becomes
destructive (
compat/patches/elastic.py+controller.degraded): apreflight rejection opens no 503 window at all; a drain timeout reopens
admission, because draining mutates nothing; and anything from
controller.resize()onward -- including a client that hangs up mid-switch,which cancels our coroutine while the engines keep working -- leaves admission
closed with a logged reason and refuses further resizes until restart. On hardware the
gate, the 503 with its reason, the log line and the recovery-by-restart were
all observed; what hardware also revealed is that the scaling middleware
answers a later resize with the native "currently scaling, try again later"
forever, which is wrong for a permanently degraded deployment, so the
middleware now substitutes the degraded reason. That last piece is
unit-tested and mutation-checked but has not been re-run on hardware.
Cancellation counts as destructive too: a hung-up client cancels the
orchestration coroutine while engine-side RPCs keep running, so the plugin's
view of the topology may no longer describe the engines and this build has no
probe to re-establish it. The alternative would need an engine-side state
query that does not exist yet.
The real fix is an abort path upstream; see the separate
issue.
An idle AFD deployment dies after 30 minutes of silence, and this branch
is not why. The FFN connector loop blocks in a control-plane
recvon a groupthe connector creates with a 30-minute timeout; with every Attention engine
paused nobody publishes metadata, the recv expires, and that exception is
fatal to the FFN loop -- whose teardown then destroys the process groups the
surviving ranks are still using. Measured: last request 06:26:37, both FFN
loops out at 06:56:37 with
Timed out waiting 1800000ms for recv operation to complete, AttentionConnection closed by peerat 07:56:47,/healthanswering 200 the whole time. Nothing in that chain consults elastic state:
the 30-minute timeout is an unchanged context line in this diff and
recv_dp_metadata_listis not touched by it at all, and the CUDA connectorsets the same 30 minutes. So this PR neither introduces nor fixes it -- the
fix belongs to the connector (an idle recv must not be fatal, and a rank's
teardown must not destroy groups it does not exclusively own), which is the
subject of a separate issue. Measured in the other direction too: an elastic
2Ax2F boot probed every 8 minutes for 64 minutes -- twice the window --
answered 8/8 and never logged a recv timeout, so the trigger is the silence,
not elastic and not uptime. A flag-off control is not available here: this
recipe cannot boot at all without
--enable-elastic-epon Ascend with theRay DP backend, because resolving Ray's
NPUaccelerator resource lives inthe elastic Attention actor's
_set_visible_devices. That is a reallayering problem (device assignment is not an elastic concern) and is
recorded in the same issue as follow-up work, not smuggled into this PR.
Both rules -- keep a completion arriving, and the discriminator against the
unrelated 1800s store timeout of limitation 2 -- are in Operator notes
below.
Re-anchor main@544f182dmarkers in the patch modules predate this stack; theenvironment now validated is vLLM tag
5df63a72. One anchor was verifiedagainst 0.32.0 and corrected; the rest are re-anchoring work still owed.
This host runs CANN 9.1.0 while
docs/installation.mdlists 9.2 as tested;the results above are 9.1.0 results.
Static gates run on the author's machine:
ruff check,ruff format,mypy-3.11,typos,markdownlint-cli2,shellcheck,check-spdx-header,check-yaml,check-merge-conflict,debug-statements,end-of-file-fixer,mixed-line-ending,trailing-whitespace,check-buildkite-- all passing over the PR range.actionlintmatches no file in this diff (nothing under.github/changes); its hook environment did install on a retry and was run over the
repository's two workflow files, where it passes -- so this is verified
rather than assumed to be irrelevant.
Operator notes
Everything a deployment needs that is not in the repo as documentation:
VLLM_PLUGINS=afd,afd_elastic(withoutafd_elasticthe route isnever installed; the
ascendplatform entry stays as the NPU install sets it),--enable-elastic-ep,--elastic-ep-max-dp-size <N>(mandatory -- a resizebeyond it is refused),
--data-parallel-backend ray,--distributed-executor-backend uni,--enforce-eager(mandatory on NPU), anexplicit
--kv-cache-memory-bytes(--gpu-memory-utilizationis not asubstitute; the run above used
2147483648),--worker-cls afd_plugin.v1.worker.npu.AFDNPUAttentionWorker, andadditional_config: {"afd": {"role": "attention", "connector": "CAMP2pAFDConnector", "num_attention_ranks": A, "num_ffn_ranks": F}, "afd_elastic": {"ffn_tensor_parallel_size": 1}}. TP=1 and A divisible by F.POST /scale_elastic_ep {"new_data_parallel_size": N, "role": "attention"|"ffn", "drain_timeout": 120};Nis the selected role's DP size.POST /is_scaling_elastic_epstays readable during a switch. Measured windows:attention up 73 s, attention down 10 s, FFN down 68 s, FFN up 66 s -- an NPU
resize rebuilds the FFN actor pool and reloads weights (~15 s per rank),
because a kept FFN rank cannot re-establish its HCCL communicators.
(pre-existing connector behaviour, limitation 3). Keep a real completion
arriving at least every ~20 minutes;
/healthdoes not count, it neverreaches the engines.
drain timeout (408) leaves the deployment serving. A failure from the switch
onward leaves admission closed with a degraded reason and later resizes answer
503 -- restart the deployment; 503 is the expected response until you do.
booted statically, not against the pre-resize output.
Docs Impact
docs/changes. The rules adeployment needs are stated in Operator notes above, and a docs page can follow
as a separate commit at whichever location the project prefers.
both
additional_config.afd_elasticand--enable-elastic-epare present.Files changed
59 files, +7217 / -88, one commit
7f17cc9(base2bede02= upstreamrfc/elastic_scaling):afd_plugin/andtests/plus thepyproject.tomlentry point. Nothing else ships -- no
docs/, norecipe/, and the throwawayvalidation scripts that drove Test Plan steps 3-6 were never part of this
commit.
Essential PR Checklist
and non-divisible topologies are refused, not half-supported.
considered: validated on tag
5df63a72+ vLLM-Ascend0cc580bb9(which contains the
bd42eea0named indocs/installation.md), CANN9.1.0. The remaining
Re-anchor main@544f182dmarkers are listed asKnown limitation 4 rather than left implicit.
monkey patches -- see Implementation Notes for the seven seams that cannot
be classes and why.
documented, and tested.
tests/unitis green on a host with no torch and no vLLM, which is theboundary this claim needs. The NPU worker modules import the elastic
lifecycle seams on first use, not at module scope.
applicable: section B of the known-P1 tripwire needs a GPU host and is
reported as not covered, not as green.
an operator needs are in Operator notes, and I will add a page anywhere the
maintainers prefer.