Repository navigation
Conversation
The worker measured a work item's age as Date.now() - createdAt, but createdAt is occ.controller_work.created_at on PostgreSQL's clock. Skew between the controller and the database moved the convergence deadline, which since #2269 also retires a pending OAuth bootstrap, early or late. The claim (and each renewal) now returns PostgreSQL's own age for the work, clock_timestamp() - created_at. The worker anchors that age on its monotonic clock when it claims and uses one helper, workAgeMs, for every age: the convergence deadlines, the stop-yield cutoff, age-scaled rechecks and the operation duration metrics. Closes finding 1187.
Review follow-up. Worker renewals go through renewClaim, which refreshes the database-clock age anchor, so a suspended controller cannot judge in-flight work younger than PostgreSQL does. The missing-anchor error names the work, the new test reports the work's age before its first-pass assertion, and the operations settings table says the deadline uses PostgreSQL's clock.
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
freeqaz
had a problem deploying
to
integration-qa-pr
October 13, 2026 06:22 — with
GitHub Actions
Error
freeqaz
had a problem deploying
to
integration-qa-pr
October 13, 2026 06:22 — with
GitHub Actions
Error
CI's application role cannot update controller_work.created_at, so the recheck-cadence test could not age its work in place. The fixture's revision() now takes workAge and inserts the reconcile row that long ago by the database clock; the test compares a fresh deployment's 500 ms recheck with a five-minute-old one's 5 s recheck.
freeqaz
had a problem deploying
to
integration-qa-pr
October 13, 2026 06:29 — with
GitHub Actions
Failure
freeqaz
had a problem deploying
to
integration-qa-pr
October 13, 2026 06:29 — with
GitHub Actions
Failure
Contributor
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(worker): judge the convergence deadline by the database clock This is item 1/1 in the current shard. Shard 0/1. This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
This was referenced Oct 13, 2026
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The worker measured a work item's age as
Date.now() - claim.createdAt, butcreatedAtisocc.controller_work.created_at, written with PostgreSQL'sclock_timestamp(). Skew between the controller and the database moved the convergence deadline by the skew. Since #2269, that deadline also retires a pending OAuth bootstrap (its Job and seed Secret), so a controller clock running ahead retired bootstraps early, and one running behind kept them past the deadline. #2275 and #2277 fixed the same clock mixing for lease liveness; this is the last controller-clock consumer of a database timestamp on the worker path.Change
PostgresWorkQueue.claim()andheartbeat()return PostgreSQL's own age for the work (clock_timestamp() - created_at) asClaimedWork.ageMs.workAgeMs(claim), replaces everyDate.now() - claim.createdAt: the revision and lifecycle convergence deadlines, the refusal-wait deadline, the stop-yield cutoff, age-scaled rechecks, the deploy/stop duration metrics and theelapsedMslog field. Comparisons between two database timestamps are unchanged.docs/reference/controller/reconciliation.mdand theOCC_WORKER_CONVERGENCE_TIMEOUT_MSrow indocs/reference/settings/operations.mdsay the deadline is measured on PostgreSQL's clock.Verification
the convergence deadline judges work age by the database clock, not the controller clock, registered inpostgres-application.json. With the worker'sDate.nowfive minutes ahead, a pending OAuth bootstrap stays pending and is not retired. With it five minutes behind, the work fails withCONVERGENCE_DEADLINE_EXCEEDEDonce PostgreSQL counts it past the 10 s deadline, and the bootstrap retirement row commits with the failure. The previous code fails both halves.deployment progress names Compute's pending reason and old pending work rechecks less oftenused to age its work by skewingDate.now. The application role cannot rewritecreated_at, so the fixture'srevision()now takesworkAgeand admits the reconcile row that long ago by PostgreSQL's clock; the test compares a fresh deployment's 500 ms recheck with a five-minute-old one's 5 s recheck.tsc --build, ESLint and Prettier on the touched files pass locally. PostgreSQL suites: hosted CI (no local database).Risks