Skip to content

fix(worker): judge the convergence deadline by the database clock - #2281

Merged
freeqaz merged 3 commits into
mainfrom
fix-1187
Oct 13, 2026
Merged

freeqaz merged 3 commits into
mainfrom
fix-1187

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The worker measured a work item's age as Date.now() - claim.createdAt, but createdAt is occ.controller_work.created_at, written with PostgreSQL's clock_timestamp(). Skew between the controller and the database moved the convergence deadline by the skew. Since #2269, that deadline also retires a pending OAuth bootstrap (its Job and seed Secret), so a controller clock running ahead retired bootstraps early, and one running behind kept them past the deadline. #2275 and #2277 fixed the same clock mixing for lease liveness; this is the last controller-clock consumer of a database timestamp on the worker path.

Change

  • PostgresWorkQueue.claim() and heartbeat() return PostgreSQL's own age for the work (clock_timestamp() - created_at) as ClaimedWork.ageMs.
  • The worker anchors that age on its monotonic clock when it claims and re-anchors it on every lease renewal. One helper, workAgeMs(claim), replaces every Date.now() - claim.createdAt: the revision and lifecycle convergence deadlines, the refusal-wait deadline, the stop-yield cutoff, age-scaled rechecks, the deploy/stop duration metrics and the elapsedMs log field. Comparisons between two database timestamps are unchanged.
  • docs/reference/controller/reconciliation.md and the OCC_WORKER_CONVERGENCE_TIMEOUT_MS row in docs/reference/settings/operations.md say the deadline is measured on PostgreSQL's clock.

Verification

  • New real-PostgreSQL test the convergence deadline judges work age by the database clock, not the controller clock, registered in postgres-application.json. With the worker's Date.now five minutes ahead, a pending OAuth bootstrap stays pending and is not retired. With it five minutes behind, the work fails with CONVERGENCE_DEADLINE_EXCEEDED once PostgreSQL counts it past the 10 s deadline, and the bootstrap retirement row commits with the failure. The previous code fails both halves.
  • deployment progress names Compute's pending reason and old pending work rechecks less often used to age its work by skewing Date.now. The application role cannot rewrite created_at, so the fixture's revision() now takes workAge and admits the reconcile row that long ago by PostgreSQL's clock; the test compares a fresh deployment's 500 ms recheck with a five-minute-old one's 5 s recheck.
  • Shared-file check: open PRs touching these files (fix: prepare native admin access on the first Agent deployment #2259, fix(worker): keep an active runtime when repository authority fails #2237, Support OpenAI-compatible Codex model endpoints #913, feat: add Agent workload tags #115; only feat: add Agent workload tags #115 touches the revision fixture helper, which already conflicts with main) have the same conflicted files against this branch as against main; no new conflicts.
  • tsc --build, ESLint and Prettier on the touched files pass locally. PostgreSQL suites: hosted CI (no local database).

Risks

  • Low. The age comes from the latest claim or renewal statement plus local monotonic time, so it trails PostgreSQL's age by at most one round trip and never ends work early.
  • Finding 1165 (a deadline pass that hits a transient dependency failure leaves the seed) is unchanged and stays open.

The worker measured a work item's age as Date.now() - createdAt, but
createdAt is occ.controller_work.created_at on PostgreSQL's clock. Skew
between the controller and the database moved the convergence deadline,
which since #2269 also retires a pending OAuth bootstrap, early or late.

The claim (and each renewal) now returns PostgreSQL's own age for the work,
clock_timestamp() - created_at. The worker anchors that age on its monotonic
clock when it claims and uses one helper, workAgeMs, for every age: the
convergence deadlines, the stop-yield cutoff, age-scaled rechecks and the
operation duration metrics.

Closes finding 1187.
Review follow-up. Worker renewals go through renewClaim, which refreshes the
database-clock age anchor, so a suspended controller cannot judge in-flight
work younger than PostgreSQL does. The missing-anchor error names the work,
the new test reports the work's age before its first-pass assertion, and the
operations settings table says the deadline uses PostgreSQL's clock.
@freeqaz
freeqaz requested a review from a team as a code owner October 13, 2026 06:22
@clawsweeper

clawsweeper Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Oct 13, 2026
CI's application role cannot update controller_work.created_at, so the
recheck-cadence test could not age its work in place. The fixture's
revision() now takes workAge and inserts the reconcile row that long ago by
the database clock; the test compares a fresh deployment's 500 ms recheck
with a five-minute-old one's 5 s recheck.
@clawsweeper

clawsweeper Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(worker): judge the convergence deadline by the database clock This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@freeqaz
freeqaz merged commit 617f725 into main Oct 13, 2026
41 of 44 checks passed
@freeqaz
freeqaz deleted the fix-1187 branch October 13, 2026 06:33

This branch had an error being deployed

1 failed deployment
integration-qa-pr — 5a1c82c8 Deployed Oct 13, 2026 by freeqaz via QA advisory (compose) #550
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant