Repository navigation
Close the lanes already built when Lanes construction fails - #37
Conversation
If a later Buffer failed to build, or two lanes shared a stream, the Buffers built before it were never closed and the caller had no handle to close them. Lanes now closes them before re-raising. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
There was a problem hiding this comment.
If only one rank fails, close() waits for the others forever. The case that can happen is that you have 4 ranks. Rank 1 failed so it enters the except, but the other 3 proceed and don't call close(). The current fix only works when all ranks fail together. we'll need to solve it with an all_gather since the ranks need to agree first on success/failure, right after the ElasticBuffer is created (before the barrier, so that all ranks raise together). It runs once at construction, so negligible cost
The checks between NCCL setup and the native constructor ran on each rank alone, so a failure there left the peers waiting inside the collective constructor while the failing rank waited for them in close(). The ranks now exchange their result first and raise together, which lets Lanes close the buffers it already built. A failure inside the native constructor is still one-sided.
|
I pushed a commit on top of this branch so before calling the native DeepEP constructor, the ranks use an This way the cleanup in I measured it on 4x H100 with 4 ranks and 3 lanes, injecting an error on rank 1 while it builds the third lane (Gloo group timeout 60 s):
The last row is the remaining limit. A failure inside the native constructor is still having this behavior but fixing it needs a change in the native code, which belongs in a separate PR. |
|
please add a test that verify a failure inside the native constructor. It should fail (be checked for failure) until the follow-up PR. |
|
Anyways if this stall occurs gloo timeout on vLLM side will solve it |
…structor One rank raises inside the native constructor of the last lane. Only that rank raises today; its peers stay inside the collective constructor. The check exits 0 while this holds and exits 1 once every rank raises.
|
Hi Ronen, I added the test in Rank 1 raises inside the native constructor of the last lane (4 ranks, 2 FFN ranks, 3 lanes) and we expect a failure. it currently exits with 0 while only the failing rank raises, and exits with 1 when every rank raises. The follow-up PR has to remove the expected failure branch. I ran it on 4 H100 GPUs. |
If building a later lane failed (for example out of memory registering its buffer), or two lanes ended up on one stream,
Lanes.__init__raised without closing the Buffers it had already built, and the caller had nothing to close them with. It now closes them before re-raising. Close is collective, so the cleanup completes where every rank failed alike; an error during cleanup does not hide the original one.Raised in review of vllm-project/afd-plugin#588.
Checks
python -m pytest tests/unit,ruff check .andruff format --check .pass