Skip to content

fix(worker): order revisions against stops by the database clock - #2293

Merged
freeqaz merged 3 commits into
mainfrom
fix-1195
Oct 13, 2026
Merged

freeqaz merged 3 commits into
mainfrom
fix-1195

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Problem

A stop, an Agent deletion and a Namespace deletion own the revisions admitted before they were requested. The worker decided that ordering by comparing two clocks: agent_revisions.admitted_at was stamped by the controller (OpenClawController.timestamp()), and controller_work.created_at is stamped by PostgreSQL (clock_timestamp()). Four worker sites compared them (worker.ts STOP_SUPERSEDED check and stop cleanup set, the Namespace-deletion credential close in observe(), and beforeSource in repository-credentials.ts), and so did the cleanup SQL in postgres-work-queue.ts.

With the controller clock ahead of PostgreSQL's by d, an active revision admitted less than d before a stop read as admitted after it. The stop then finished STOP_SUPERSEDED (success) without stopping anything: desiredRuntimeState became stopped while the runtime kept running. Finding 1195, the same family as finding 1187 (#2281).

Change

  • PostgresPlatformState.createRevision stamps admitted_at with clock_timestamp() and returns the stored stamp as createdAt (RETURNING admitted_at). Deploy responses already use the returned revision, so the API's revision timestamps are on the database clock too. The caller's createdAt is no longer stored. The in-memory store is unchanged.
  • One helper, revisionAdmittedBeforeWork (controller-work.ts), replaces the four comparisons and documents the tie rule: both reads are truncated to milliseconds, and a tie counts as earlier, so the stop still stops the revision.
  • The SQL predicates are unchanged; they now compare two database-clock stamps. Comments note it.
  • No schema change and no migration. Revisions admitted before this change keep their controller stamps.
  • repositoryCredentials.deadlineWallMs stays on the controller's clock, which the worker checks it against. It is taken at admission, before runtime credentials are provisioned, so it no longer equals createdAt plus the session duration exactly. repository-credentials-platform.test.mjs now checks that the deadline base comes before the stored admission and less than a minute earlier.
  • Docs: reconciliation.md says the stop ordering uses PostgreSQL's clock.

Verification

  • New real-PostgreSQL test, registered in postgres-application.json: "a stop orders the running revision by the database clock, not the controller clock". For one Agent the admitting clock runs 5 minutes ahead of PostgreSQL's, for another 5 minutes behind (new controllerSkewMs fixture knob). It checks the stored admitted_at is on the database clock and equals the returned createdAt, then deploys, stops and expects AGENT_STOPPED, a Compute stopRevision call, and a cleared active pointer. Before the fix the ahead case finishes STOP_SUPERSEDED without stopping. Hosted CI is the proof for PostgreSQL tests.
  • The store contract now accepts a store-stamped createdAt (assertCreatedRevision). The memory runs pass locally: platform-state-memory, repository-sessions-memory (48/48).
  • Local: tsc --build clean, eslint and prettier clean on touched files, repository-credentials-source-boundary, repository-credentials-package and configuration-occ pass (126/126).

Risks

  • Low. Revisions admitted before the upgrade keep controller stamps, so the skew can still affect them until they are replaced. Without skew nothing changes.
  • The SQL comparisons of deadlineWallMs with clock_timestamp() are an existing cross-clock spot, unchanged here.
  • A revision admitted after a stop in the same millisecond reads as earlier in the worker (the SQL is exact to the microsecond). Every admission also sets desiredRuntimeState to running, and the stop rechecks that before each effect, so this cannot stop a newer deployment. In closeRevision such a tie can make the cleanup registration refuse and roll back the pass; this was already possible before the change.

agent_revisions.admitted_at came from the controller clock while
controller_work.created_at comes from PostgreSQL's, and four worker sites
plus the cleanup SQL compared them. With the controller ahead, a stop could
read the running revision as admitted after it and finish STOP_SUPERSEDED
without stopping it. The PostgreSQL store now stamps admitted_at with
clock_timestamp() and returns the stored stamp; the worker compares through
one helper, revisionAdmittedBeforeWork. Closes finding 1195.
…dmission

The controller's clock starts the repository credential deadline when it
admits a deployment, and PostgreSQL now stamps the revision's createdAt a
moment later, after runtime credentials are provisioned. The platform test
asserted they were equal; it now checks the deadline base precedes the
stored admission by less than a minute.
@freeqaz
freeqaz requested a review from a team as a code owner October 13, 2026 08:12
@clawsweeper

clawsweeper Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Oct 13, 2026
@clawsweeper

clawsweeper Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge.

What this changes

This PR uses PostgreSQL timestamps to order revision admissions against stop and deletion work.

Example: PostgreSQL admits a revision at 12:00:00 while the controller clock reads 12:05:00, then records a stop at 12:00:01.

  • Before: The revision appears newer than the stop, which succeeds as STOP_SUPERSEDED while the runtime remains running.
  • After: The stored admission time is 12:00:00, so the stop includes the revision and clears its active pointer.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Focused, source-correct repair with meaningful PostgreSQL coverage, held below a pass by missing populated-state compatibility verification.
Proof confidence 🦐 gold shrimp (3/6) The maintainer-authored ordinary live-proof gate does not apply, and exact-head PostgreSQL CI supports the fresh-admission regression; populated-state upgrade proof for the changed persisted timestamp semantics is still absent.
Patch quality 🦐 gold shrimp (3/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A requested stop can succeed while the Agent runtime continues running when the controller clock is ahead of PostgreSQL.
Reason: The maintainer-authored change restores the existing stop contract for new admissions and explicitly accepts retaining historical timestamps.

Merge readiness

⛔ Blocked before merge - 1 item remains

Keep this PR open: it fixes a source-confirmed stop-ordering bug within the existing lifecycle contract, and no definite introduced bug was found.

Priority: P2
Reviewed head: 3557bb0afab064cbc501e58a1f009f1c2fad158c

Before merge

  • Add data-model compatibility proof - The review found that existing stored data may not work after upgrade. Show that existing data still loads and works with this change.

Findings

None.

Tests

  • Missing end-to-end proof: Supply a populated-state upgrade scenario covering retained controller-stamped revisions, new database-stamped admissions, stop ordering, and unchanged repository credential deadlines.
Agent review details

How this fits together

OCC admission persists immutable Agent revisions and durable work; the controller worker reads their ordering, checks ownership and authorization, and invokes Compute and repository-credential cleanup.

flowchart TD
  A[Authorized deployment] --> B[PostgreSQL revision admission]
  C[Authorized stop or deletion] --> D[PostgreSQL controller work]
  B --> E[Controller worker ordering]
  D --> E
  E --> F[Compute lifecycle]
  E --> G[Repository credential cleanup]
Loading

Technical review

Best possible solution:

Keep database-owned admission timestamps and verify the explicitly retained old-state behavior without rewriting immutable revision history.

Do we have a high-confidence way to reproduce the issue?

Current-main source establishes the failure with an ahead controller clock; the added PostgreSQL scenario targets that path, but a base-versus-head failing execution was not supplied.

Is this the best way to solve the issue?

Stamping admission at its persistence owner fixes the clock mismatch across both worker and SQL comparisons while preserving lifecycle authorization and cleanup ownership.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against dc92ed5c8292.

Provenance checked

  • Revision admission timestamp ownership changes intended behavior with a stated reason (unknown: Scoped history and blame reach the initial checkout boundary for caller-provided timestamps; the original clock choice is not established.)
  • Stop and repository cleanup ordering keeps the original intent (Repository credentials: platform lifecycle and Kubernetes delivery #235: Exclude later admissions while retaining exact revision ownership and durable cleanup obligations.)
  • Store and credential deadline assertions changes intended behavior with a stated reason (Test repository session continuity after token refresh #674: Protect retained session identity and its original deadline across recovery and token refresh.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Review metrics

Metric Value Why it matters
Production and regression coverage Production +41/-10; tests and fixtures +96/-12 Growth is concentrated in a real-PostgreSQL skew regression and adapting assertions to store-owned timestamps.

Labels

Label changes:

  • add P2: Clock skew can make an Agent stop report success while leaving runtime running; the repair is focused and no broader outage is established.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.

Label justifications:

  • P2: Clock skew can make an Agent stop report success while leaving runtime running; the repair is focused and no broader outage is established.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Verify retained pre-upgrade revisions and new admissions together, preserving credential deadlines and recording the expected stop outcomes.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 13, 2026, 4:16 AM ET / 08:16 UTC.

@freeqaz
freeqaz merged commit 93ced2a into main Oct 13, 2026
45 of 47 checks passed
@freeqaz
freeqaz deleted the fix-1195 branch October 13, 2026 08:17

This branch had an error being deployed

1 failed deployment
integration-qa-pr — 3557bb0a Deployed Oct 13, 2026 by freeqaz via QA advisory (kubernetes) #591
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant