Skip to content

[GPU] Run the DeepEP M2N connector on afdEP-comm's role API - #588

Open
Etelis wants to merge 4 commits into
vllm-project:mainfrom
ms-llmd:m2n-afdep-buffer
Open

Etelis wants to merge 4 commits into
vllm-project:mainfrom
ms-llmd:m2n-afdep-buffer

Conversation

@Etelis

@Etelis Etelis commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Purpose

The DeepEP M2N connector drove afdEP-comm's low-level ElasticBuffer directly: it built one per lane, added DeepEP's dummy expert slots, kept its own empty and zero tensors, and set the per-buffer communication stream variable. afdEP-comm now offers this as its role API, so the connector uses that instead:

Connector Before Now
build lanes ElasticBuffer(...) per lane Lanes(group, Config(...), lanes=N)
Attention send dispatch(...) source_dispatch
FFN receive dispatch(empty tensors) worker_receive
FFN reply combine(rows, handle) worker_reply
Attention collect combine(zero rows, handle) source_collect

No behaviour change is intended.

Issue

Closes #598

Scope

  • In scope: the connector's transport calls, the afdEP-comm pin, and the unit-test stand-in for afdep.
  • Out of scope: bucket planning, the network plan, the control plane, DBO order and stream handling, and the experts. All unchanged.

Implementation Notes

  • afdEP-comm is pinned to v1.2.0, the first release with Lanes.
  • Config takes the real expert count; afdEP-comm adds the empty Attention blocks.
  • afdEP-comm checks exact dtypes, so the connector casts routing ids and weights before source_dispatch. These are the casts ElasticBuffer did internally.
  • The connector no longer tracks pending Attention handles or rejects receive rows on Attention ranks; afdEP-comm does both.
  • After building the lanes, the connector checks that afdEP-comm gives this rank the same role as the plugin's layout, and on FFN ranks the same FFN index. Both compute the FFN placement, and a mismatch would otherwise give wrong results silently.
  • DeepEPM2NLayout.deepep_num_experts is removed; afdEP-comm's Config.logical_experts computes it.
  • Depends on Close the lanes already built when Lanes construction fails afdEP/afdEP-comm#37 (close the lanes already built when Lanes construction fails), which needs an afdEP-comm release before this can merge.

Test Plan

  • CPU: pytest tests/unit.
  • GPU: the M2N GPU tests and the DeepSeek and GLM-5.3 recipes, with and without DBO, checking identical tokens and unchanged throughput.

Test Result

  • CPU: pytest tests/unit passes; pre-commit (ruff, mypy, typos) passes.
  • GPU: not run yet, so this is a draft.

Docs Impact

  • Files updated: docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md.

afdEP-comm main after v1.1.0 adds Lanes (one Buffer per lane) and exports
the receipt type from the package.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
The connector built an ElasticBuffer per lane and called dispatch and
combine with its own empty and zero tensors. It now builds afdEP-comm's
Lanes and calls source_dispatch, source_collect, worker_receive and
worker_reply. afdEP-comm takes the real expert count, holds the empty
and zero tensors, and sets the per-buffer communication stream.

No behaviour change is intended: buckets, the control plane, DBO order
and stream handling are as before.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes labels Nov 15, 2026
@Etelis
Etelis marked this pull request as ready for review November 15, 2026 15:39
@jiangkuaixue123

Copy link
Copy Markdown
Collaborator

cc @ronenkat

# One lane runs on the caller's stream: no fork/join cost.
if self.num_lanes > 1:
self.lane_streams.append(torch.cuda.Stream(self.device))
self.lanes = self._new_lanes()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if lanes fails while constructing a later buffer, 'self.lanes` is None but the previously created buffers won’t be closed so make Lanes.init have a try @Etelis

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address.
suggested fix: in afdEP-comm, have Lanes.init close the buffers it already built when one fails.

@Etelis Etelis Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in afdEP-comm: afdEP/afdEP-comm#37

@ronenkat ronenkat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please open an issue for migrating M2N connector to afdEP-comm external API, and link to this PR.
  2. Need before/after performance numbers.
  3. a new afdEP-comm version tag is required to merge.
  4. Nit: dead code. DeepEPM2NLayout.deepep_num_experts (layout.py:60) is no longer used in production; only a test assertion still reads it (test_deepep_m2n_connector.py:373). afdEP-comm's Config.logical_experts now does this calculation. Remove it, or keep it as a test cross-check against Config.

python -m pip uninstall -y afdep
python -m pip install --no-deps \
"afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@<tag pinned in pyproject.toml>"
"afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@<revision pinned in pyproject.toml>"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a version tag and not a commit.

Comment thread pyproject.toml Outdated
vllm = ["vllm==0.32.0"]
afdep = [
"afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@v1.1.0",
"afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@510172c5018eb96dd97a59db5b43774247179a5f",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should use a version tag. not commit.
Please change to DRAFT pull-request until so.
afdEP-comm MUST use not reuse v1.1.0 with the addition of lanes

# One lane runs on the caller's stream: no fork/join cost.
if self.num_lanes > 1:
self.lane_streams.append(torch.cuda.Stream(self.device))
self.lanes = self._new_lanes()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address.
suggested fix: in afdEP-comm, have Lanes.init close the buffers it already built when one fails.

if self.num_lanes > 1:
self.lane_streams.append(torch.cuda.Stream(self.device))
self.lanes = self._new_lanes()
# One lane runs on the caller's stream: no fork/join cost.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The layout/afdEP-comm duplicate FFN-placement formula carries risk. If the two copies ever disagree, an FFN rank would get rows for experts it didn't load, and the results would be silently wrong. The new role API gives a cheap way to catch this. Right after self.lanes = self._new_lanes() (connector.py:1151), assert:

  • self.lanes[0].role == ("ffn" if self.is_ffn else "attention")
  • on FFN ranks, self.lanes[0].role_rank == self.role_rank

@Etelis
Etelis marked this pull request as draft November 16, 2026 09:14
The first release with Lanes.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
… count

The plugin's layout and afdEP-comm both place the FFN ranks. After the
lanes are built, check that afdEP-comm gives this rank the same role, and
on FFN ranks the same FFN index; a mismatch would return wrong results
silently. DeepEPM2NLayout.deepep_num_experts is no longer used:
afdEP-comm's Config.logical_experts computes it.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis

Etelis commented Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

@ronenkat thanks, addressed

@ronenkat

Copy link
Copy Markdown
Collaborator

Thanks @Etelis

  1. please run GPU tests per the test plan
  2. What is the performance impact? if any?

Etelis added a commit to afdEP/afdEP-comm that referenced this pull request Nov 18, 2026
If building a later lane failed (for example out of memory registering
its buffer), or two lanes ended up on one stream, `Lanes.__init__`
raised without closing the Buffers it had already built, and the caller
had nothing to close them with. It now closes them before re-raising.
Close is collective, so the cleanup completes where every rank failed
alike; an error during cleanup does not hide the original one.

Raised in review of vllm-project/afd-plugin#588.

## Checks

- [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format
--check .` pass
- No native changes.
@ronenkat

Copy link
Copy Markdown
Collaborator

Benchmark

  • Baseline: upstream/main @ e4bddb3
  • Post-PR: m2n-afdep-buffer @ 778e408a
  • Model: Qwen3-30B-A3B
  • Layout: 2A2F, graph + DBO
  • Workload: random 512 input / 1024 output, 8,192 prompts, concurrency 1,024
  • vLLM: 0.32.0; afdEP-comm: 1.2.0; 4× H100 80GB

Result

  • Before: 13,615 out tok/s, 38.58 s median TTFT, 37.31 ms median TPOT, 96.91 ms p99 ITL
  • After: 13,638 out tok/s, 38.59 s median TTFT, 37.33 ms median TPOT, 96.04 ms p99 ITL
  • Delta: +0.16% throughput, +0.02% median TTFT, +0.06% median TPOT, −0.90% p99 ITL
  • Both runs completed 8,192/8,192 requests with 0 failures.

Interpretation

  • Performance is effectively unchanged; the differences are within single-run noise.
  • No regression from the PR.

@ronenkat
ronenkat marked this pull request as ready for review November 20, 2026 07:32
@ronenkat ronenkat added the ready Used to trigger ready CI in PRs. label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes ready Used to trigger ready CI in PRs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Migrate the DeepEP M2N connector to afdEP-comm's external API

4 participants