Wait briefly for the restore target instead of failing the deployment - #168
Merged
Conversation
A deployment takes two backups minutes apart, and both verify into the same disposable database. The exclusivity check refused outright whenever another session or prepared transaction was present, with no wait and no retry, so a session that had not finished closing failed the post-cutover backup and took the whole deployment with it. That happened three times on 2026-07-30. Each occurrence left the maintenance window open at failed-after-database-mutation with the site returning 502 until the deployment was re-run, and each re-run succeeded because the competing session had gone by then. Wait for the target to become exclusive, bounded and abortable, before asserting. The assertion itself is unchanged: a target that never frees is still reported busy and the destructive restore still refuses to run. Only the transient case changes. The competing session is often not identifiable afterwards, which is why this does not try to name it. An earlier note in ROADMAP blamed the hourly backup cron; the timings disprove that and the entry has been corrected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A deployment takes two backups minutes apart and both verify into the same disposable database. The exclusivity check refused outright whenever another session or prepared transaction was present — no wait, no retry — so a session that had not finished closing failed the post-cutover backup and took the whole deployment with it.
That happened three times on 2026-07-30. Each left the maintenance window open at
failed-after-database-mutationwith the site returning 502 until the deployment was re-run, and each re-run succeeded because the competing session had gone.Now waits for the target to become exclusive — bounded and abortable — before asserting. The assertion is unchanged: a target that never frees is still reported busy and the destructive restore still refuses to run. Only the transient case changes.
Mutation-verified against removing the wait, making it unbounded, and ignoring prepared transactions.
I first recorded this in ROADMAP as the hourly backup cron firing mid-cutover. The timings disprove that — cron backups ran at 15:00, 16:00 and 17:00 while the failing deployments started at 15:32 and 16:17 — so the entry is corrected to what the evidence actually shows.