Skip to content

Close the lanes already built when Lanes construction fails - #37

Merged
Etelis merged 4 commits into
mainfrom
lanes-close-on-failure
Nov 18, 2026
Merged

Etelis merged 4 commits into
mainfrom
lanes-close-on-failure

Conversation

@Etelis

@Etelis Etelis commented Nov 16, 2026

Copy link
Copy Markdown
Member

If building a later lane failed (for example out of memory registering its buffer), or two lanes ended up on one stream, Lanes.__init__ raised without closing the Buffers it had already built, and the caller had nothing to close them with. It now closes them before re-raising. Close is collective, so the cleanup completes where every rank failed alike; an error during cleanup does not hide the original one.

Raised in review of vllm-project/afd-plugin#588.

Checks

  • python -m pytest tests/unit, ruff check . and ruff format --check . pass
  • No native changes.

If a later Buffer failed to build, or two lanes shared a stream, the
Buffers built before it were never closed and the caller had no handle
to close them. Lanes now closes them before re-raising.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>

@almogtavor almogtavor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@almogtavor almogtavor left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If only one rank fails, close() waits for the others forever. The case that can happen is that you have 4 ranks. Rank 1 failed so it enters the except, but the other 3 proceed and don't call close(). The current fix only works when all ranks fail together. we'll need to solve it with an all_gather since the ranks need to agree first on success/failure, right after the ElasticBuffer is created (before the barrier, so that all ranks raise together). It runs once at construction, so negligible cost

The checks between NCCL setup and the native constructor ran on each rank
alone, so a failure there left the peers waiting inside the collective
constructor while the failing rank waited for them in close(). The ranks
now exchange their result first and raise together, which lets Lanes close
the buffers it already built. A failure inside the native constructor is
still one-sided.
@almogtavor

almogtavor commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

I pushed a commit on top of this branch so before calling the native DeepEP constructor, the ranks use an all_gather to share whether their setup steps (domain and buffer sizing) succeeded, and if any rank fails they all raise an error together.

This way the cleanup in Lanes runs on every rank, and close() doesn't get stuck waiting for a rank that already exited.

I measured it on 4x H100 with 4 ranks and 3 lanes, injecting an error on rank 1 while it builds the third lane (Gloo group timeout 60 s):

Case d87fb27 fadf845
No failure built in 2.9 s, closed built in 2.9 s, closed
Rank 1 fails before the native constructor rank 1 raises after 62.1 s, ranks 0, 2, 3 hang all 4 ranks raise within 3.6 s
Rank 1 fails inside the native constructor rank 1 raises after 62.1 s, ranks 0, 2, 3 hang same as before

The last row is the remaining limit. A failure inside the native constructor is still having this behavior but fixing it needs a change in the native code, which belongs in a separate PR.

almogtavor
almogtavor previously approved these changes Nov 16, 2026
@ronenkat

Copy link
Copy Markdown
Collaborator

please add a test that verify a failure inside the native constructor. It should fail (be checked for failure) until the follow-up PR.

@Etelis

Etelis commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

Anyways if this stall occurs gloo timeout on vLLM side will solve it

…structor

One rank raises inside the native constructor of the last lane. Only that rank
raises today; its peers stay inside the collective constructor. The check exits 0
while this holds and exits 1 once every rank raises.
@almogtavor

almogtavor commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Hi Ronen, I added the test in tests/gpu/construct_failure.py.

Rank 1 raises inside the native constructor of the last lane (4 ranks, 2 FFN ranks, 3 lanes) and we expect a failure. it currently exits with 0 while only the failing rank raises, and exits with 1 when every rank raises. The follow-up PR has to remove the expected failure branch.

I ran it on 4 H100 GPUs.
It is a GPU script like the other files in tests/gpu, so CI does not run it.

@Etelis
Etelis merged commit 593c851 into main Nov 18, 2026
4 checks passed
@Etelis
Etelis deleted the lanes-close-on-failure branch November 18, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants