Conversation
…ed retries process_lost_retry (and the equivalent scheduleBoundedRetryForRun codex-transient path) transferred an issue's checkoutRunId/executionRunId to the new retry run with a naive `executionRunId = <dying run id>` WHERE guard. If the issue's lock had already drifted away from that exact run id by the time the transfer ran (BBO-1541: the lock pointed at a third, older, already-terminal run — neither the timed-out run nor the new retry run), the UPDATE silently affected 0 rows. The retry run was created and woken, but the issue kept pointing at a dead run forever, and every subsequent checkout/release/PATCH from the new run was rejected as an ownership conflict with no automatic recovery (BBO-1547). Both call sites now lock the issue row and decide from its live state instead of trusting the stale run snapshot: transfer authority to the new run whenever the current lock already belongs to the dying run, is unset, or points at a run that is itself terminal/missing. Only refuse when a different, still-live run genuinely owns the issue. Adds a regression test reproducing the exact BBO-1541 divergence and asserting checkout/assertCheckoutOwner/release all succeed against the retry run afterward.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
process_lost_retry(and the equivalentscheduleBoundedRetryForRuncodex-transient path) transferred an issue'scheckoutRunId/executionRunIdto the new retry run using a naiveexecutionRunId = <dying run id>WHERE guard. If the lock had already drifted off that exact run id (BBO-1541: pointed at a third, already-terminal run), the UPDATE silently affected 0 rows — the retry run was created and woken, but the issue stayed pointed at a dead run forever, permanently rejecting every subsequent checkout/release/PATCH from the new run as an ownership conflict.SELECT ... FOR UPDATEthe issue row and decide from its live state instead of trusting the stalerunsnapshot: transfer authority to the new run whenever the current lock already belongs to the dying run, is unset, or points at a run that is itself terminal/missing (viaTERMINAL_HEARTBEAT_RUN_STATUSES). Only refuse when a different, still-live run genuinely owns the issue.server/src/__tests__/heartbeat-process-loss-retry-lock-transfer.test.ts, reproducing the exact BBO-1541 divergence and asserting checkout/assertCheckoutOwner/release all succeed against the retry run afterward.Test plan
pnpm exec vitest run server/src/__tests__/heartbeat-process-loss-retry-lock-transfer.test.ts— new regression test passespnpm exec vitest runacross 7 related heartbeat lock/ownership/retry test files (heartbeat-lock-release-on-reassignment,heartbeat-retry-scheduling,heartbeat-process-recovery,heartbeat-start-lock,heartbeat-responsible-user-invariant,heartbeat-zombie-guard,heartbeat-stale-queue-invalidation) — 133/133 passing, no regressionspnpm --filter @paperclipai/server typecheck— cleanRefs: BBO-1551, BBO-1547