Repository navigation
dsv41: a rank whose peer dies ends with exit 75 (peer link watch) - #40
Conversation
When rank 1 died while rank 0 was idle, rank 0 stayed up: /health said 200 and the next request hung without an error or exit. When rank 1 died during a decode, the live streams hung too: the round's GPU waits on the RDMA ring got no error completion, so the abort word was never set (release fault F3). - peer_watch.zig: a thread that polls the ranks' TCP links for the peer's end (POLLRDHUP, HUP, ERR; it reads nothing). On a peer that is gone without a done frame, it aborts the RDMA ring (Ring.abort: every wait gives up, so a live round fails and its requests get the error) and, with TF_DS_EXIT_ON_FAULT=1, ends the process with exit 75 after TF_DS_EXIT_GRACE plus 2 s (the lane host's own fault exit comes first when a round ran). It waits 1 s first, for rank 0's done frame and the close that follows it. - link.zig: a keepalive (10 s idle, 3 probes 3 s apart), so a peer whose node vanished shows within about 20 s. - lanes.zig: rank 0 watches its followers' links from Lanes.init and stops the watch before its done frame; a follower watches its link to rank 0 while it follows.
Elevatormusic
added
the
type: fix
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6dd239777
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An error unwind (rank 0's link gone while the follower reads it) left through a deferred stop, which marked the watch orderly and cancelled its exit 75. The watch now stays running on an error unwind, so its exit 75 still ends the process if the normal exit hangs on kernels that wait on the gone rank.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6dd239777
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31466020f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ails World 4 keeps its rings in rdma_rings (all4, col, pair) or neighbor_gather (first, second), not in ring, so a dead peer aborted no GPU wait there. The watch now takes every ring the rank's waits read. A start that fails after Lanes.init (a bad TF_DS_FIX_K, a vision load error) closed the model with the watch still polling its links and holding its rings; native.open now stops the watch on that path.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84a0e16a22
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…eer watch Model.close frees the RDMA rings after follow() returns an error, while the watch can still abort them. The follower now calls Watch.release on the error path: it aborts the rings (so close does not wait on a gone peer) and forgets them under a lock, and keeps the exit 75 armed.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56b1a68d68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Lanes.deinit stopped the watch before the done frame on every teardown, so an error teardown after a follower died marked the loss orderly and cancelled the exit 75. Now the watch stops only after a done frame that goes out with no link end seen. Otherwise rank 0 releases the rings from the watch and holds (at most the grace plus 4 s) for the watch's exit 75.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bbb861b7af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Every way a rank ends now calls Watch.end before the model frees its links and rings: rank 0's Lanes.deinit, a follower's done frame or error, and a start that fails after Lanes.init. On an error it aborts every RDMA ring (TP=2 and TP=4), stops and joins the watch thread, frees the watch, and, when a peer's link is gone and TF_DS_EXIT_ON_FAULT=1, exits 75 before Model.close. Watch.stop, release and hold are gone. Link sends use MSG_NOSIGNAL: a send to a gone follower fails with PeerDown instead of a SIGPIPE that ends rank 0 with no exit 75. Tests: a failed start with the peer up, an idle follower whose rank 0 dies, a sending leader whose follower dies, a follower that gets PeerDown before the watch acts, and a done frame then the peer's close.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a339278f9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A keepalive only covers an idle link. When a peer's node loses power while this rank has sent data that is not acknowledged, the kernel retransmits for many minutes and the peer watch gets no POLLERR. TCP_USER_TIMEOUT at 20 s (the keepalive's bound) ends such a link with ETIMEDOUT. Test: both ends of a loopback link have the timeout and the keepalive.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6749974bb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…er's When the peer watch aborts a ring while a round waits in synchronize, the collect kernels leave their waits and can copy stale peer slots. synchronize checked only the neighbor gather, so a TP=2 or TP=4 round could return its results. It now checks every ring (Model.rdmaRings, the same list the watch aborts) and fails with RdmaTransportFailed. Test: rdma.anyFailed finds an aborted ring.
|
@codex review |
1 similar comment
|
@codex review |
…, POLLNVAL) - A requested stop (the server's SIGTERM) ends rank 0's watch orderly even when a follower went away during it, so a clean stop exits 0, not 75. - Watch.start takes the exit grace (faultGrace() in the engine, null in tests), so an exported TF_DS_EXIT_ON_FAULT cannot end the test runner. - The abort comes before any log, and every watch log is written only when stderr takes it at once, so a blocked stderr cannot hold the abort or the exit 75. - faulted() also polls the links now: a FIN that the watch thread has not seen yet still makes rank 0's teardown a peer fault. - POLLNVAL counts as a link end, and a failed poll sleeps 200 ms (no spin). With TF_DS_EXIT_ON_FAULT=1 a watch that cannot start fails the start. - The follower starts its watch after its kept states, so on an error the rings are aborted (and exit 75 comes) before their device memory is freed. Tests: POLLNVAL on a closed pipe end; the exit comes from Watch.start's grace alone, with TF_DS_EXIT_ON_FAULT=1 exported.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4012d8c65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Model.pass returns the drafter's drafts after the drafter syncs its own stream (draft.zig), so the ring check in synchronize never ran for it. A ring aborted during a draft pass now fails the pass (RdmaTransportFailed) before its drafts are used. The check is one function, ringsOk, for both.
|
Review fixes (a code review while the Codex review stalled), in f4012d8, cd5970c and 8e7d31f:
The description now states that the bounded exit needs @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e7d31f516
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…oll snapshot) note() wrote to fd 2 after a poll(POLLOUT) snapshot. Another thread can fill the pipe between the poll and the write, so the write could block after the ring abort and before the exit 75. Watch.start now opens /proc/self/fd/2 again with O_NONBLOCK (a new open file description, so fd 2 stays blocking for the other threads); note writes only through it, and a socket stderr gets send with MSG_DONTWAIT. A new test fills a pipe through the copy and checks that the pipe's own descriptor stays blocking.
…g-peerwatch # Conflicts: # zig/src/families/dsv41/lanes.zig
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
What this changes
The release fault campaign found that a rank whose peer dies does not end (F3, severity 1). Two GB10 nodes at TP=2, the 2.8 bpw pack,
TF_DS_EXIT_ON_FAULT=1:/healthsays 200, and the next request gets no reply in 120 s, no error and no exit 75. Rank 0 was still up 5 minutes later. Seen in 3 of 3 runs.futex_do_wait, 2 inib_uverbs_event_read, 7 inpoll_schedule_timeout; none in a TCP read. The round waits on the RDMA ring's arrival flags; no error completion came, so the abort word that every wait reads was never set.This change:
peer_watch.zig(new): a thread that polls the ranks' TCP links for the peer's end (POLLRDHUP,POLLHUP,POLLERR). It reads nothing. When a peer is gone and this rank does not end (an idle rank, or a round whose waits got no error completion), it waits 1 s (for an orderly stop), aborts every RDMA ring (every GPU wait gives up, so a live round fails through the lane host's fault path and its requests get the error), and withTF_DS_EXIT_ON_FAULT=1ends the process with exit 75 afterTF_DS_EXIT_GRACEplus 2 s.Watch.end(how): the one teardown. Every way a rank ends calls it beforeModel.closefrees the links and rings: rank 0'sLanes.deinit, a follower's done frame or error, and a start that fails afterLanes.init. In order: (1) on an error, abort every RDMA ring (TP=2's ring, TP=4's three, or the neighbor gather's two); (2) stop the watch thread and join it, so no thread reads the rings after it returns; (3) free the watch (the caller frees the rings next); (4) on an error with a peer's link gone (the thread saw it, or a poll that does not wait finds it now) andTF_DS_EXIT_ON_FAULT=1, exit 75 with_exit, beforeModel.closecan wait on the gone rank. It returnsorderly,failed(an error with every peer up) orpeer_fault.lanes.zig: rank 0 watches its followers' links fromLanes.init. At teardown it sends the done frame, then callsend(.done)when the frame went out and no link end was seen before it, elseend(.failed). A follower watches its link to rank 0 while it follows:end(.done)on the done frame,end(.failed)on any error. If the watch cannot start, the ranks run as before.model.zig:Model.rdmaRingsis the one list of every RDMA ring (TP=2's ring, TP=4's three, the neighbor gather's two). The watch aborts that list, andModel.synchronizechecks it after each sync: work that waited on an aborted ring fails withRdmaTransportFailedinstead of returning results from stale peer slots (before, only the neighbor gather was checked).native.zig: a start that fails afterLanes.initcallsendWatch(.failed)before the model closes.link.zig: a keepalive on the link (10 s idle, then 3 probes 3 s apart) andTCP_USER_TIMEOUT20 s (sent data that is not acknowledged), so a peer whose node vanished (no FIN) is found, idle or during a round (not tested on hardware; see the known gaps). Sends useMSG_NOSIGNAL: a send to a gone follower fails withPeerDown, not a SIGPIPE that ends rank 0 oftf-dsv41-laneswith no exit 75.Review fixes (a code review while the Codex review stalled): a requested stop (the server's SIGTERM,
stop_requested) ends rank 0's watch orderly even when a follower went away during the stop, so a clean stop exits 0, not 75;Watch.starttakes the exit grace (faultGrace()in the engine,nullin the tests), so an exportedTF_DS_EXIT_ON_FAULTcannot end the test runner; the ring abort comes before any log, and every watch log is written only when stderr takes it at once, so a blocked stderr cannot hold the abort or the exit;faulted()also polls the links, so a FIN that the watch thread has not seen yet still makes rank 0's teardown a peer fault;POLLNVALcounts as a link end and a failed poll sleeps 200 ms (no spin); withTF_DS_EXIT_ON_FAULT=1, a watch that cannot start fails the start; the follower starts its watch after its kept prompt states, so on an error the rings are aborted before their device memory is freed.The bounded exit needs
TF_DS_EXIT_ON_FAULT=1on both ranks (the release launch sets it). Without it, the watch still aborts the rings, so rank 0 stays up and every request fails with an error instead of hanging; there is no exit 75, and a follower'sModel.closeafter an error has no timer.Known gaps:
TCP_USER_TIMEOUT20 s and the keepalive probes, the exit is expected about 22-30 s after a power loss, plus the grace.Not changed: NCCL is not aborted from the watch thread (
ncclCommAbortthere could race the main thread's use of the comm); a prefill that waits in NCCL on a gone rank ends with the process exit.Receipt
dsv41-zigplus this branch, indsv41-zig-dev@cc9be34e(with dsv41 Zig: load the EXL3 2.8 bpw pack (bf16 router weights and confidence head, one source a tensor) #38 and dsv41: special-token text in request strings is plain text #39). Zig 0.17.0. Two NVIDIA GB10 nodes at TP=2. DeepSeek-V4.1-Flash-1017 EXL3 2.8 bpw.peer_watch.zig, loopback links, one for each teardown path): a failed start with the peer up endsfailed; an idle follower whose rank 0 dies: the watch acts, thenendfindspeer_fault; a sending leader whose follower dies getsPeerDown(no SIGPIPE) andendfindspeer_fault; a follower that getsPeerDowncallsendbefore the watch acts and still getspeer_fault; a done frame, then the peer's close, endsorderly.zig build test-cuda-hostatdsv41-zig-dev@87cf5678(this branch @0e92350a): rc 0.TF_DS_EXIT_ON_FAULT=1): 4.5 (SIGKILL to rank 1 during a 4-stream decode): 1 round fault on rank 0, the 4 streams end with the errorPeerDown, rank 0 exits 75 in 7 s. 4.7 (SIGKILL to rank 1 when idle): rank 0 exits 75 in 8 s. 4.5b (SIGKILL to rank 0 during decode): rank 1 exits 75 in 1 s. Before this change, 4.5 and 4.7 hung in 4 of 4 runs. The clean stops still pass: 4.4a (SIGTERM during a 256K prefill: exit 0 in 6 s, the request endscancelled), 4.4b (SIGTERM during decode: exit 0 in 5 s), 4.6 (restart: 30 s to a reply).zig build test-cuda-hostatdsv41-zig-dev@d58d25cc(this branch @b862c70d): rc 0.tensorfold-dsv41-zig:d58d25cc, 2026-10-22 20:18-20:29Z,TF_DS_EXIT_ON_FAULT=1): 4.5 (SIGKILL to rank 1 during a 4-stream decode): 1 round fault, the 4 streams end withPeerDown, rank 0 exits 75 in 7 s; 4.7 (rank 1 killed when idle): rank 0 exits 75 in 8 s; 4.5b (rank 0 killed during decode): rank 1 exits 75 in 2 s; 4.4a (SIGTERM in a 256K prefill: exit 0 in 6 s, requestcancelled), 4.4b (SIGTERM during decode: exit 0 in 5 s, streamscancelled) and 4.6 (restart: 33 s to a reply) pass. The same 6 cases also passed on952398c5(beforeTCP_USER_TIMEOUTand the ring check).zig build test-cuda-hostatdsv41-zig-dev@cc9be34e(this branch @8e7d31f5, with the review fixes and the draft-pass ring check): rc 0.tensorfold-dsv41-zig:cc9be34e, 2026-10-23 03:28-03:37Z,TF_DS_EXIT_ON_FAULT=1): 4.5: 1 round fault, the 4 streams end withRdmaTransportFailed(the ring check now fires before the link error), rank 0 exits 75 in 7 s; 4.7: rank 0 exits 75 in 9 s; 4.5b: rank 1 exits 75 in 1 s; 4.4a (exit 0 in 6 s, requestcancelled), 4.4b (exit 0 in 4 s, streamscancelled) and 4.6 (33 s to a reply) pass. No decode check on this image (the change afterd58d25ccadds one ring-state read per pass).Checklist