[freeqaz]

Problem

A flake sweep of today's CI (no real flakes) found four preventive items:

• The two convergence deadline tests from fix(worker): judge the convergence deadline by the database clock #2281 and fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283 ran a real 10 s deadline and took 10.7-11.1 s. A runner stall of more than 10 s before their first pass would fail them (the finding 1171 pattern). The fix(worker): judge the convergence deadline by the database clock #2281 test also replaced Date.now by hand.
• postgres-worker-agent-revision.test.mjs grew from 101 to 129 s (worst 153 s) as fix(occ): judge deployment claim liveness by the database clock #2275, fix(worker): judge the convergence deadline by the database clock #2281 and fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283 added cases.
• The SIGTERM termination test reports each phase's stop time only in a test diagnostic, so lane results can't show how close each phase comes to its 15 s budget.
• A k3d OAuth result file said cleanup-failed when a newer push had cancelled the job. GitHub cancels a step by signalling and then killing its shell, never the runner. The runner then finishes as an orphan and races the job's always() Cleanup lane step. In run 38349577861 the result file ended 3 s after the step was marked cancelled.

Change

• Database-clock deadline test (fix(worker): judge the convergence deadline by the database clock #2281):
  • It now uses two deployments against a 60 s deadline.
  • With the controller clock 5 minutes ahead, a fresh deployment must stay pending.
  • With the clock 5 minutes behind, a deployment admitted 120 s ago by the database clock (the fixture's workAge) must fail on its first pass with CONVERGENCE_DEADLINE_EXCEEDED and retire its bootstrap.
  • Neither result depends on when the first pass runs. Date.now is replaced through context.mock, which restores it when the test ends, even on failure.
• Outage-at-deadline test (fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283):
  • Its scripted passes must run before the deadline, so the deadline stays real time and goes from 10 to 30 s.
  • Each deployment must also fail at or after the deadline by the database clock.
• Split: the 34 credential-source cases (withdrawals, and listed sources that lose their grant or readiness) move unchanged into postgres-worker-agent-revision-credential-sources.test.mjs, together with their helpers. The new file is registered in PostgreSQL Application's parallelFiles and expectedTests.
  • The lane still runs 8 files at a time, and the short files fill slots as they free up.
  • Expected time: about 50 s for the new file and about 100 s for the remaining file.
  • The slowest file is still teardown, at about 145 s.
• SIGTERM headroom: the termination test records a runtime-sigterm measurement for each wrapper and phase: stopped, elapsed time and budget.
  • The reporter allowlists the measurement's 7 case names and its integer times.
  • It appears in each file's measurements in the lane result.
• Cancelled cleanup: if a file's cleanup fails after the runner's parent shell is gone, the runner reports cleanup status cancelled with issue cleanup-cancelled, not cleanup-failed. The lane still fails, and aggregation treats a cancelled cleanup as failed.

Verification

• node --test on ci-runner.test.mjs passes locally for:
  • the allowlisted measurements test, which now has a valid runtime-sigterm sample and three rejected ones;
  • a new test that runs the runner under bash and kills the shell from the fake cleanup. With the parent check forced off, this test fails (status: 'failed' instead of cancelled).
• No database locally (hosted CI is the proof): both revision files load and skip as expected, 50 and 34 tests, the same 84 as before the split.
• eslint and prettier --check pass on the touched files.
• Neither docs page grows: controller-worker.md stays at 2,493 words, after a two-word trim in its neighbouring bullet; the other two are far below the limit.
• An independent review approved the change with nits, which are applied: the controller-ahead first pass gets 45 s, the lane check fails any cleanup that did not pass, and the new runner test is bounded.
• Pre-fix reasoning:
  • fix(worker): judge the convergence deadline by the database clock #2281: before it, work age was Date.now() - createdAt. The fresh deployment's first pass under +5 min would then read 300 s, past 60 s, and fail at once, so the "young and pending" assertion fails. The pre-aged deployment under -5 min would read 120 - 300 < 0 s and never fail within the wait.
  • fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283: before it, a deadline pass that hit an outage never retired the bootstrap, so cleanupRows(running) stays empty and the test fails.
• Hosted CI on head 9f2b968 (full run, green):
  • PostgreSQL Application: the revision file took 117 s (it was 128-153 s) and -credential-sources 64 s. The lane is still bounded by teardown at 142 s.
  • The fix(worker): judge the convergence deadline by the database clock #2281 test took 0.3 s and the fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283 test 30.6 s.
  • Image Runtime Startup 2 recorded all 7 runtime-sigterm measurements. The slowest stop was 966 ms against 15 s.
• Head 637bd9b: the cancelled-cleanup test now starts the runner under a node parent instead of bash -c, after code scanning flagged the shell command. CI Required is green on this head.

Risks

• Test and CI-runner only; no product code changes.
• The split moves withCredentialGateway out of the revision file. Open PR Support OpenAI-compatible Codex model endpoints #913 adds one line to that helper, so it now conflicts in that file. To resolve, add validateSourceConfig() {}, to the helper's new location.
• No other open PR gains a conflict (trial merges: feat: add Agent workload tags #115, feat(occdev): enable local Keycloak sign-in and manual launcher qualification #1138, fix(ci): retain nested failure causes in diagnostics #2002, fix(ci): synchronize Agent watches before starting test files #2012, test: mint isolated QA observer tokens from the GitHub App #2106, fix(worker): keep an active runtime when repository authority fails #2237).
• The fix(worker): retire an oauth bootstrap last seen running when the deadline pass hits an outage #2283 test now takes about 31 s instead of 11 s. Its file stays well under the lane's slowest file.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

ClawSweeper status: review started.

I am starting a fresh review of this pull request: test(worker): harden the convergence deadline tests and split the revision file This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.