Skip to content

fix: bound Wire recovery across restarts - #449

Merged
CountableNewt merged 4 commits into
mainfrom
codex/tsw-126-replay-budget
Sep 15, 2026
Merged

CountableNewt merged 4 commits into
mainfrom
codex/tsw-126-replay-budget

Conversation

@CountableNewt

@CountableNewt CountableNewt commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Problem and change

An exhausted replay incident previously opened another archive request after a restart, and paused/failed restarts omitted saved retry/range/ETag evidence. Reject requests before opening them when incident capacity is exhausted and restore retained transport evidence. Correct the documented readiness semantics: a paused lane remains unready.

Add an opt-in absolute UTC REPLAY_CANARY_EXPIRES_AT setting for Wire so a separately reviewed recovery canary can stop across owner restarts. Expiry cancels the Wire lane, joins runner cleanup before explicitly releasing its lease, and parks the lane without supervisor retries. An already-expired owner opens no database connection and starts no lease or archive work. AppView intake is independent; unset preserves existing behavior. Invalid timestamps or AppView use fail validation.

Rollout and limits

Every eligible owner must run the guard and retain the same configured UTC expiry before activation. Verify synchronized clocks, the exact owner configuration prefixes, and separately reviewed byte/disk/queue limits. Cleanup can finish after cancellation; an uncooperative operation is joined, never detached. Existing in-flight byte overshoot remains. This PR does not activate recovery, change hosted settings, or reset counters, checkpoints, source identity, or recovery evidence.

The runbook records the stalled Development recovery seam and separates local intake from Development viewers' Production Corpus serving path. Original download buckets have expired, so the original 5 GiB cannot be apportioned between useful progress and retries. Recovery completion remains unresolved.

Validation

  • Tested implementation head d772a5376139216f646c9e9a4a2d9183b533461d: full local Go race suite, 132 tests/subtests passed, 11 existing PostgreSQL integrations skipped, zero failures.
  • Corresponding Development guard PR457: 133 passed, 11 PostgreSQL integrations skipped, zero failures.
  • New deterministic coverage: expired restarts/no database work, in-flight transport cancellation, delayed cleanup joining, sibling isolation/no supervisor restart, parent cancellation, and standalone/controller/named-Wire configuration prefixes.
  • Earlier restart-budget change separately passed 134 PostgreSQL-enabled tests; that result does not validate the newly added guard's database lifecycle. Required CI must provide the current-head PostgreSQL validation before promotion.
  • Current PR head 2b3a6901e40f6656a1567bc3024eab69157aed3a merges main 864add735bee2a42f5878c9a14d284c4e99a9389 without conflicts; Jetstream source/tests are byte-identical to the locally tested implementation. Strict Required CI is rerunning on that exact head.
  • gofmt and git diff --check passed.

Tracks TSW-126 under TSW-92. Preserve Development-first promotion. Restart-budget fix PR450 is deployed in Development; canary guard PR457 is deployed successfully in Development at merge revision cee73e33cc07228861072c51aea5e7a320f0ab9f, with expiry unset and checkpoint/usage continuity verified. No automatic merge or recovery activation.

@linear-code

linear-code Bot commented Sep 15, 2026

Copy link
Copy Markdown

TSW-126

@CountableNewt CountableNewt changed the title fix: preserve replay budget pauses across restarts fix: bound Wire recovery across restarts Sep 15, 2026
@CountableNewt
CountableNewt merged commit ddfe2d9 into main Sep 15, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant