fix(session-runtime): don't charge a retired turn's late frames to the next execution#618
Merged
Merged
Conversation
…e next execution The codex app-server can emit a turn's `turn/completed` seconds after the `error` frame that already ended that turn. When a user retries in that window, the late frame arrives after the pump has already called `forget()` on the finished execution — which erased the turn->execution mapping the late-frame guard depends on. `execution_for_line` then found no owner and fell through to the active execution, charging a dead turn's `status: failed` to the retry and terminating it before its prompt ever reached the model. Observed in prod: a session hit the codex usage limit, and the retry died 40ms after birth with the misleading `turn completed with status failed before final answer` — the real reason (out of credits) was masked. `forget()` now retires the execution's turn/item ids into a bounded FIFO tombstone instead of dropping them, so late frames stay attributable to the execution that owned them and are discarded.
gbasin
enabled auto-merge
July 20, 2026 14:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
Session
362d3277in prod hit the codex usage limit. The retry ("Retry the failed turn.") died 40ms after birth with a misleadingturn completed with status failed before final answer— and never reached the model at all. Its prompt was silently swallowed.Root cause
The codex app-server emitted
turn/completedfor the already-errored turn 7.5s after theerrorframe that ended it — by which time the retry was the active execution:error(usage limit) endsexe_f8be…; pump records terminal, callsforget()exe_03b05…, becomes activeturn/completedfor the old turn arrives → charged to the retry → kills itforget()purgedturn_execution_by_id, erasing the very mapping the existing late-frame guard (execution_for_line, lib.rs:5919) depends on. With no owner found, the frame fell through toactive_execution_id.Fix
forget()now retires the execution's turn/item ids into a bounded FIFO tombstone (1024 ids) instead of dropping them, so late frames stay attributable to the execution that owned them and are discarded rather than misattributed.This would corrupt any retry landing inside the window where a prior turn's completion frame is still in flight — not specific to the rate-limit case.
Testing
stdout_state_drops_late_terminal_frame_from_a_retired_turnreproduces the exact prod sequence. Verified it fails without the fix (assertsSome("exe-retry")vsNone) and passes with it.cargo test -p centaur-session-runtime --lib— 130 passedcargo clippy --all-targetsclean,cargo fmt --checkcleanNot run: the live-cluster e2e. This is a pure state-machine change in the stdout pump, proven by unit test; standing up k8s to exercise a tombstone map seemed disproportionate. Happy to run it if you'd rather.