Repository navigation
dsv41: kept prompt states: rank 0 copies before the frame, a failed follower restore stops - #41
Conversation
…ollower restore stops snapSaveFn put the new state into self.snaps before the frame and before rank 0's copy. When the flush or the copy failed, the errdefers freed the state, but its pointer stayed in the list (a use after free in the next placement), and rank 1 kept a buffer that was never dropped. Now the list has room first, rank 0 copies, then sends, then appends. The rank 0 restore sent its frame before its own copies, and rank 1 logged a failed restore copy and continued with a state different from rank 0's. Now rank 0 copies first (a failed copy sends no frame), and a follower whose restore copy fails stops. kept_order.zig holds the order, with 6 CPU tests (std only).
Elevatormusic
added
the
type: fix
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…very pull request) zig fmt 0.17.0 changes only layout and @intFromEnum to @backingInt. The Zig format check fails on these five base-branch files in every open dsv41-zig pull request, also where the pull request does not touch them.
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
Two error-path defects in the kept prompt states (the prompt cache's saved states) of the dsv41 TP=2 engine. Both can make rank 0 and rank 1 hold different states. An audit of the session cache for rank divergence found them. The audit context is Refs jayleaton/deepseek-v41-tensorfold-spark#65. That fault (each rank trims its disk tier by its own file sizes) cannot occur here: rank 0 makes every keep, evict, budget and restore decision, and dsv41 has no disk tier.
lanes.zigsnapSaveFn(rank 0). The new state went intoself.snapsbefore the frame flush and before rank 0's copy. If the flush or the copy failed, the errdefers freed the state, but its pointer stayed in the list. The nextusable,placeKeptorstaleOverthen read freed memory. Also, rank 1 kept that id's buffer (snapBytes of device memory), and no drop was ever sent for it.Fix: make room in the list first, then rank 0's copy, then the frame, then add the state to the list.
next_snapgoes up before the save, so an id is never used again.lanes.zigrestore, both ranks. Rank 0 sentsnap_restorebefore its own copies. Rank 1 logged a failedcopyExtentorsnapCopy(follow_refused) and continued with a state different from rank 0's: a silent divergence for that stream.Fix: rank 0 does its copies first, so a failed copy on rank 0 sends no frame. A follower whose restore copy fails returns the error and stops.
kept_order.zig(new, std only) holds the order of these steps. The three call sites inlanes.ziguse it.Known gaps
dsv41-zigwithout dsv41: a rank whose peer dies ends with exit 75 (peer link watch) #40, rank 1's stop makes rank 0 wait on the gone rank (the F3 hang). With dsv41: a rank whose peer dies ends with exit 75 (peer link watch) #40, both ranks exit 75.UnknownSnapshot. That is a loud stop, not a divergence.Receipt
dsv41-zig@84ded67dplus this branch. Zig 0.17.0, CPU only (no GPU work).zig test kept_order.zigwith the old order in the helpers: 1 passed, 5 failed. With this change: 6 of 6 pass.zig ast-checkandzig fmt --checkon the changed files: pass.zig build test-cuda-hostatda420678(CPU container, no GPU work, run without the GPU lock): rc 0 in 173 s. Its 5 "failed command" lines are the usual notes for test steps that print warnings.Checklist
kept_order.zig. 5 fail on the old order.