Skip to content

fix(test): accept a losing owner that retires the replaced token - #3323

Merged
freeqaz merged 3 commits into
mainfrom
flake-2257
Oct 23, 2026
Merged

freeqaz merged 3 commits into
mainfrom
flake-2257

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

postgres-prepared-token-service.test.ts "rejected fresh GitHub material recovers once across separate owners without replay" failed on main (PostgreSQL Platform, run 39972061000) at

assert.deepEqual(results.sort(), ["committed", "idle"])   // got ["committed", "retired"]

Two owners run one maintenance pass at once over a family whose current token was just rejected. A pass is discover, then retire, then prepare, and it reports prepare's result, or the retire step's result when prepare claimed nothing. The winner claims, issues the replacement and commits it, and that commit moves the rejected token into retirement custody. The loser claims nothing, so it reports its own retire step:

  • idle when its retire step ran before the winner's commit (the next pass then retires the rejected token);
  • retired when its retire step ran after that commit and revoked the rejected token.

Both are correct; retirement is shared by every owner and is never replayed. The test only accepted idle. Delaying the loser by about 30 ms or more is enough to produce retired, so a slow runner hits it. The failing test took 510 ms; nothing hung. Since the test landed (#2487), this is 1 failure in about 230 completed PostgreSQL Platform jobs.

This is not a product bug: the loser never claims the family, never issues, and the token it retires is the superseded, rejected one.

Change

Test only.

  • New shared helper raceMaintenance(owners, signal) in tests/helpers/github-metadata-exchange.mjs: runs one pass on each owner at once, asserts exactly one commits, and returns the sorted losing results. Both prepared GitHub race tests use it.
  • postgres-prepared-github-lifecycle.test.mjs (cold family, nothing to retire) still requires the loser to be idle.
  • postgres-prepared-token-service.test.ts accepts idle or retired for the loser and checks retirement directly, by counting the provider revocations of the rejected token: one before the race (the test's own), a second right after the race only if the loser reported retired, exactly two after the next ready pass, and still two at the end. The new final count also catches a retirement replay that the old test missed. Before the test revokes the replacement itself, it checks that no maintenance pass revoked it.

Verification

  • Forced race (loser delayed by a random 0-300 ms): the old test failed 9 of 10 runs with the CI message; the new test passed 16 of 16 on fresh databases, with both idle and retired losers.
  • Mutant: leaving a retired token in retirement custody (so later passes revoke it again) passes the old test 3/3 and fails the new one 3/3.
  • Pins at 2 CPUs, each run on a fresh database: 90/90 as three parallel loops on the first head, 30/30 with CPU stress on the same cores, and 90/90 as three parallel loops on the final head. The lifecycle test passes 3/3.
  • tsc --build --force, eslint and prettier are clean on the touched files.

Risks

Low, test only. The race test is a bit more lenient about the losing result, but the invariants it protects are now asserted directly: exactly one commit, one new issue, and exactly one retirement of the rejected token with no replay.

Two owners race one maintenance pass over a family whose rejected token is
still current. The winner claims, issues and commits the replacement, and
that commit moves the rejected token into retirement custody. A losing pass
that reaches its retirement step after the commit revokes that token and
reports `retired`; one that reaches it earlier reports `idle` and the next
pass retires it. Both are correct, but the test only accepted `idle`.

Add a shared `raceMaintenance` helper that asserts exactly one owner commits
and returns the losers, use it in both prepared GitHub race tests, and in
the token-service case accept `idle` or `retired` while counting the
rejected token's revocations: one only when the loser retired it, exactly
one retirement by the next pass, and none replayed by later passes.
@freeqaz
freeqaz requested a review from a team as a code owner October 23, 2026 15:19
@clawsweeper

clawsweeper Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@freeqaz
freeqaz merged commit 9924351 into main Oct 23, 2026
43 checks passed
@freeqaz
freeqaz deleted the flake-2257 branch October 23, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant