Repository navigation
Conversation
afdEP-comm main after v1.1.0 adds Lanes (one Buffer per lane) and exports the receipt type from the package. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
The connector built an ElasticBuffer per lane and called dispatch and combine with its own empty and zero tensors. It now builds afdEP-comm's Lanes and calls source_dispatch, source_collect, worker_receive and worker_reply. afdEP-comm takes the real expert count, holds the empty and zero tensors, and sets the per-buffer communication stream. No behaviour change is intended: buckets, the control plane, DBO order and stream handling are as before. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
|
cc @ronenkat |
| # One lane runs on the caller's stream: no fork/join cost. | ||
| if self.num_lanes > 1: | ||
| self.lane_streams.append(torch.cuda.Stream(self.device)) | ||
| self.lanes = self._new_lanes() |
There was a problem hiding this comment.
if lanes fails while constructing a later buffer, 'self.lanes` is None but the previously created buffers won’t be closed so make Lanes.init have a try @Etelis
There was a problem hiding this comment.
Please address.
suggested fix: in afdEP-comm, have Lanes.init close the buffers it already built when one fails.
ronenkat
left a comment
There was a problem hiding this comment.
- Please open an issue for migrating M2N connector to afdEP-comm external API, and link to this PR.
- Need before/after performance numbers.
- a new afdEP-comm version tag is required to merge.
- Nit: dead code. DeepEPM2NLayout.deepep_num_experts (layout.py:60) is no longer used in production; only a test assertion still reads it (test_deepep_m2n_connector.py:373). afdEP-comm's Config.logical_experts now does this calculation. Remove it, or keep it as a test cross-check against Config.
| python -m pip uninstall -y afdep | ||
| python -m pip install --no-deps \ | ||
| "afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@<tag pinned in pyproject.toml>" | ||
| "afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@<revision pinned in pyproject.toml>" |
There was a problem hiding this comment.
Please use a version tag and not a commit.
| vllm = ["vllm==0.32.0"] | ||
| afdep = [ | ||
| "afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@v1.1.0", | ||
| "afdep-comm @ git+https://github.com/afdEP/afdEP-comm.git@510172c5018eb96dd97a59db5b43774247179a5f", |
There was a problem hiding this comment.
should use a version tag. not commit.
Please change to DRAFT pull-request until so.
afdEP-comm MUST use not reuse v1.1.0 with the addition of lanes
| # One lane runs on the caller's stream: no fork/join cost. | ||
| if self.num_lanes > 1: | ||
| self.lane_streams.append(torch.cuda.Stream(self.device)) | ||
| self.lanes = self._new_lanes() |
There was a problem hiding this comment.
Please address.
suggested fix: in afdEP-comm, have Lanes.init close the buffers it already built when one fails.
| if self.num_lanes > 1: | ||
| self.lane_streams.append(torch.cuda.Stream(self.device)) | ||
| self.lanes = self._new_lanes() | ||
| # One lane runs on the caller's stream: no fork/join cost. |
There was a problem hiding this comment.
The layout/afdEP-comm duplicate FFN-placement formula carries risk. If the two copies ever disagree, an FFN rank would get rows for experts it didn't load, and the results would be silently wrong. The new role API gives a cheap way to catch this. Right after self.lanes = self._new_lanes() (connector.py:1151), assert:
- self.lanes[0].role == ("ffn" if self.is_ffn else "attention")
- on FFN ranks, self.lanes[0].role_rank == self.role_rank
The first release with Lanes. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
… count The plugin's layout and afdEP-comm both place the FFN ranks. After the lanes are built, check that afdEP-comm gives this rank the same role, and on FFN ranks the same FFN index; a mismatch would return wrong results silently. DeepEPM2NLayout.deepep_num_experts is no longer used: afdEP-comm's Config.logical_experts computes it. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
|
@ronenkat thanks, addressed |
|
Thanks @Etelis
|
If building a later lane failed (for example out of memory registering its buffer), or two lanes ended up on one stream, `Lanes.__init__` raised without closing the Buffers it had already built, and the caller had nothing to close them with. It now closes them before re-raising. Close is collective, so the cleanup completes where every rank failed alike; an error during cleanup does not hide the original one. Raised in review of vllm-project/afd-plugin#588. ## Checks - [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format --check .` pass - No native changes.
|
Benchmark
Result
Interpretation
|
Purpose
The DeepEP M2N connector drove afdEP-comm's low-level
ElasticBufferdirectly: it built one per lane, added DeepEP's dummy expert slots, kept its own empty and zero tensors, and set the per-buffer communication stream variable. afdEP-comm now offers this as its role API, so the connector uses that instead:ElasticBuffer(...)per laneLanes(group, Config(...), lanes=N)dispatch(...)source_dispatchdispatch(empty tensors)worker_receivecombine(rows, handle)worker_replycombine(zero rows, handle)source_collectNo behaviour change is intended.
Issue
Closes #598
Scope
afdep.Implementation Notes
v1.2.0, the first release withLanes.Configtakes the real expert count; afdEP-comm adds the empty Attention blocks.source_dispatch. These are the castsElasticBufferdid internally.DeepEPM2NLayout.deepep_num_expertsis removed; afdEP-comm'sConfig.logical_expertscomputes it.Lanesconstruction fails), which needs an afdEP-comm release before this can merge.Test Plan
pytest tests/unit.Test Result
pytest tests/unitpasses; pre-commit (ruff, mypy, typos) passes.Docs Impact
docs/gpu/DEEPEP_M2N_CONNECTOR_USER_GUIDE.md.