fix: bound Wire recovery across restarts - #449
Merged
Merged
Conversation
…w-126-replay-budget
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and change
An exhausted replay incident previously opened another archive request after a restart, and paused/failed restarts omitted saved retry/range/ETag evidence. Reject requests before opening them when incident capacity is exhausted and restore retained transport evidence. Correct the documented readiness semantics: a paused lane remains unready.
Add an opt-in absolute UTC
REPLAY_CANARY_EXPIRES_ATsetting for Wire so a separately reviewed recovery canary can stop across owner restarts. Expiry cancels the Wire lane, joins runner cleanup before explicitly releasing its lease, and parks the lane without supervisor retries. An already-expired owner opens no database connection and starts no lease or archive work. AppView intake is independent; unset preserves existing behavior. Invalid timestamps or AppView use fail validation.Rollout and limits
Every eligible owner must run the guard and retain the same configured UTC expiry before activation. Verify synchronized clocks, the exact owner configuration prefixes, and separately reviewed byte/disk/queue limits. Cleanup can finish after cancellation; an uncooperative operation is joined, never detached. Existing in-flight byte overshoot remains. This PR does not activate recovery, change hosted settings, or reset counters, checkpoints, source identity, or recovery evidence.
The runbook records the stalled Development recovery seam and separates local intake from Development viewers' Production Corpus serving path. Original download buckets have expired, so the original 5 GiB cannot be apportioned between useful progress and retries. Recovery completion remains unresolved.
Validation
d772a5376139216f646c9e9a4a2d9183b533461d: full local Go race suite, 132 tests/subtests passed, 11 existing PostgreSQL integrations skipped, zero failures.2b3a6901e40f6656a1567bc3024eab69157aed3amerges main864add735bee2a42f5878c9a14d284c4e99a9389without conflicts; Jetstream source/tests are byte-identical to the locally tested implementation. Strict Required CI is rerunning on that exact head.gofmtandgit diff --checkpassed.Tracks TSW-126 under TSW-92. Preserve Development-first promotion. Restart-budget fix PR450 is deployed in Development; canary guard PR457 is deployed successfully in Development at merge revision
cee73e33cc07228861072c51aea5e7a320f0ab9f, with expiry unset and checkpoint/usage continuity verified. No automatic merge or recovery activation.