Repository navigation
Merged
Conversation
…ail a quarantined one Native prepared Git issues a version's first GitHub token only after its active pointer is published, so a deployment reported succeeded with no warning while every Git request answered 503 when GitHub refused the App key. The worker now reads the version's token state from its prepared owner after activation: it waits at REPOSITORY_GIT_MATERIAL_PENDING until the token commits and fails with REPOSITORY_GIT_TOKEN_QUARANTINED when the preparation was quarantined, which no later pass retries. Maintenance of running versions is unchanged.
…nt the deployment check The status read now happens on the second check after the pointer is published, once the first REPOSITORY_GIT_MATERIAL_PENDING deferral is recorded. Document the first-token wait and the quarantined failure in the native Git reference and the repository credential flow; move older flow changelog entries to its history page to keep the page under the word limit.
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
clawsweeper
Bot
added
clawsweeper:intake:bug
Contributor
|
Codex review: needs maintainer review before merge. What this changesNative repository deployments wait for their first GitHub token and fail explicitly when token preparation is quarantined. Example: GitHub returns HTTP 401 when preparing a native Agent revision's first repository token.
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep this PR open: it addresses a current deployment-status bug, follows an explicit maintainer decision, and has no concrete introduced defect identified in this review. Priority: P2 Before mergeNone. FindingsNone. Tests
Agent review detailsHow this fits togetherThe controller worker reconciles admitted Agent revisions, reads token readiness from its prepared Git owner, and publishes deployment progress or failure through OCC. flowchart TB
A[Agent deployment] --> B[Controller worker]
B --> C[Publish active revision]
C --> D[Prepared Git owner]
D --> E[GitHub token preparation]
E --> F[PostgreSQL token metadata]
F --> B
B --> G[Deployment status]
Technical reviewBest possible solution: Land the focused readiness gate while preserving existing token issuance, authorization, cleanup, and published-runtime ownership. Do we have a high-confidence way to reproduce the issue? Source confirms that native token discovery requires the published active pointer while main completes deployment without observing token readiness; no target code was executed during this review. Is this the best way to solve the issue? The existing prepared owner is the appropriate readiness source, and checking both publication paths fixes deployment reporting without moving issuance into reconciliation or changing send authorization. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 53c84b112f91. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
Likely related people:
LabelsLabel changes:
Label justifications:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 25, 2026, 2:34 AM ET / 06:34 UTC. |
- Name unreachable GitHub in REPOSITORY_GIT_TOKEN_QUARANTINED and say to deploy a new version when the key and registry are correct. - Keep an unfinished issuing attempt pending: its own run can still commit it after the dispatch window (a suspended worker host or a delayed commit), and a false quarantine would fail the deployment for good. The reference now says other waits end at the convergence deadline. - Test that a running version's maintenance pass ignores a quarantined renewal. - Move the misplaced doc comment, use the store's material state type, and say that earlier versions stay until the next deployment. - Breaking-change notice for native repository deployments.
The new 2026-10-25 entry put breaking-changes.md over the 2500-word limit.
A live run on a k3d install showed the PR's claim wrong: like any repository version that fails for good, the failed version's work transfers to terminal retirement, which stops its runtime and clears the Agent's active pointer. The earlier version was already retired when the new one was published, so the Agent has no running version until a deployment succeeds. - Reference, flow, runbook and breaking-change notice now say so. - The quarantined worker test waits for the retirement and asserts the stopped runtime and the cleared pointer instead of the pointer right after failure. - The runbook says to wait about two minutes for the mounted App key Secret to update before deploying: a version deployed earlier can still be signed with the rejected key and fail again (seen live).
Collaborator
Author
|
Free's clanker here (an AI agent working on Free's behalf). Not landing this yet. What blocks it:
The code review is done: no deadlock, a read-only query, and maintenance passes are not gated (a test pins this). The live checks on a remote test VM passed: a good key held briefly and then succeeded, a rejected key failed with the new code, and a restored key succeeded again. Also open: a docs follow-up. After a GitHub App key Secret changes, the worker can sign with the old key for one to two minutes, so a deployment in that window fails as quarantined. The runbook now tells operators to wait. |
Resolve controller-work.ts on main's tightened messages, add REPOSITORY_GIT_TOKEN_QUARANTINED to the failure-step table and fit its message in the 256-character cap. Drop the two breaking-change pages main deleted (#3930) and keep the flow changelog consistent with main.
…git token is quarantined Owner decision 2026-10-28 (finding 2425): a new Agent with repository access and no Git access fails; an existing Agent keeps running so it can report the problem. A quarantined first token now fails the deployment and, when an earlier revision of the same service principal was activated by its own deployment (stopped or superseded completions do not count), keeps the version running: the deployment fails with continuingRevision and its maintenance chain starts in one transaction under Namespace then Agent locks. Maintenance accepts that failure code. A new Agent's version is still retired. An exhausted published deployment of a new Agent without a token is retired instead of kept. A token still pending at the convergence deadline retires the version for every Agent, since it could still commit later.
…e flow under the word limit The existing-Agent case now checks that nothing retired the predecessor only after the failed pass's completion event, so a retirement made after the failure commit cannot race the assertion (a retire-predecessor mutant survived before). Move the flow's changelog entries dated 2026-10-21 to 2026-10-25 to its history page and tighten the first-token paragraph (2477 words).
freeqaz
changed the title
…git token is pending at the deadline Finding 2894: a native deployment still waiting for its version's first GitHub token at the convergence deadline ended CONVERGENCE_DEADLINE_EXCEEDED and retired the version for every Agent, so an existing Agent went fully offline (owner decision 2026-10-28 says it stays running). At the deadline, when the last wait was REPOSITORY_GIT_MATERIAL_PENDING and the token state is still pending (or quarantined), the deployment goes through continueRevisionWithoutGit: an existing Agent's deployment fails with REPOSITORY_GIT_TOKEN_PENDING, keeps the version and queues its maintenance in one transaction; maintenance accepts that code as it does the quarantined one. A new Agent's version is still retired with CONVERGENCE_DEADLINE_EXCEEDED. A token committed later gives the kept version Git: its session stays open. Finding 2895: drop the tautological principal check in activatedEarlierRevision (a revision's principal is the Agent's own) and pin that maintenance refuses a version whose deployment failed with any other code. Sharper existing-Agent assertion and a later fixed deployment step from the independent review.
… to stay under the word limit
…ent's version at the deadline The finding 2894 path keeps an existing Agent's version at the convergence deadline only when the deployment's last wait was the first Git token and the token is still pending. Two guards had no test: a published version whose activation never completes (WORKSPACE_NODE_PENDING) with its token pending, and Compute's Sandbox install wait (REPOSITORY_GIT_MATERIAL_PENDING, same code as the token wait) once the token is committed. Both must still fail with CONVERGENCE_DEADLINE_EXCEEDED and retire the version. Each case kills one mutant that the existing suite let survive (dropping the last-wait check; keeping the version when the re-read token state is ready).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the native prepared Git route (no Sandbox), a deployment could report
succeededwithwarnings: []anderror: nullwhile the version had no repository Git token and every Git request answered a bare 503 (dogfood finding 2415: a rejected App key, and a stale grant after a registry switch). The worker log named the revision (PREPARED_GITHUB_RECOVERY_REQUIRED); nothing reached the deployment status,agent getor the console.The first version of this PR failed such a deployment and then retired the version like any failed repository version. The earlier version had already stopped serving when the new one was published, so an existing Agent went fully offline, chat included (finding 2425).
Change
Owner decision 2026-10-28: a new Agent with configured Git repositories and no Git access fails; an existing Agent keeps running so it can report the issue.
PreparedRepositoryGitStore.revisionMaterial(revisionId)(native broker only, reads no material) answersready,pendingorquarantined. The worker checks it for a native repository deployment after it publishes the pointer and before it retires earlier versions:pendingholds atREPOSITORY_GIT_MATERIAL_PENDING,quarantinedfails withREPOSITORY_GIT_TOKEN_QUARANTINED. Maintenance claims and protected (OpenShell) admissions are not gated. An unfinished attempt stayspending; it ends at the convergence deadline.REVISION_ACTIVATEDorREVISION_ALREADY_ACTIVE, throughcontrollerWorkDeploymentStatus). A deployment that completed stopped or superseded does not count.continueRevisionWithoutGitfails the deployment withcontinuingRevisionand queues the version's maintenance in one transaction, under Namespace then Agent locks, after it rechecks the exact active, running revision, its service principal, a ready Namespace and the repository deadline. The version keeps running without Git, so users can still chat with the Agent and it can say Git is unavailable. Nothing in this path retires the predecessor.REPOSITORY_GIT_MATERIAL_PENDINGand the token state read at the deadline is stillpending(no attempt finished: an App key the worker cannot read after a Secret rotation, a worker restart during a request, an issuer change), an existing Agent's deployment fails with a new code,REPOSITORY_GIT_TOKEN_PENDING, through the samecontinueRevisionWithoutGitpath (same locks, rechecks and maintenance in one transaction). A token state ofquarantinedat that moment usesREPOSITORY_GIT_TOKEN_QUARANTINED. A new Agent's version is still retired withCONVERGENCE_DEADLINE_EXCEEDED. The state is read again because the Sandbox install wait shares the pending code; a failed read keeps the deadline failure. If the token commits later, the kept version gets Git: its session stays open, and it is the running Agent's active, maintained version.REPOSITORY_GIT_TOKEN_QUARANTINED,REPOSITORY_GIT_TOKEN_PENDING); it still refuses every other failed deployment, now pinned by a test.controller-work.ts: two messages (249 and 239 characters, under the 256-character cap; both say to ask an Installation administrator) anddeploy-againsteps indeployment-failure-steps.ts.controllerWorkDeploymentStatusis exported from the OCC package index for the worker.prepared-native-git.md("Deployment and the first token": a new/existing table), the team runbook (rejected App key), and flow section 6. Older flow changelog entries moved to the history page to stay under the word limit.Not done (needs an owner call or RFC): keeping the previous version serving until the new one gets Git. The Kubernetes Compute Driver stops a dedicated deployment's predecessors before it prepares the candidate (
requiresStoppedPredecessors), and an embedded activation replaces the predecessor's Gateway. Holding the predecessor would change Driver behavior. This PR keeps the Agent available through the new version instead.Verification
postgres-worker-revision-fixture.mjsand-support.mjs):postgres-worker-agent-revision.test.mjs, a matrix: a new Agent with a ready, quarantined or pending (deadline) token; an existing Agent with a quarantined token or a token still pending at the deadline (REPOSITORY_GIT_TOKEN_PENDING): the deployment fails, the version stays active with its repository session open, is not stopped, the predecessor is not retired, its maintenance pass then succeeds withREVISION_ALREADY_ACTIVE, and a later version whose token is issued replaces it normally; and an Agent whose earlier version completedREVISION_STOPPED(treated as new, retired).postgres-worker-agent-revision-teardown.test.mjs: a new Agent's published deployment that lost its lease (stale-claim recovery,ACTIVE_REVISION_RECOVERY) and then gets a quarantined token, or then exhausts its attempts on an unavailable dependency. Both fail and retire.postgres-worker-agent-revision-teardown.test.mjs(exhausted published deployment, maintenance declared): maintenance queued later for a version whose deployment failed with another code (DEPENDENCY_UNAVAILABLE) is refused withREVISION_SUPERSEDEDand prepares nothing.postgres-worker-agent-revision.test.mjs, deadline guards: an existing Agent whose published version never activates (WORKSPACE_NODE_PENDING) while its token is pending, and one waiting at Compute's Sandbox install (REPOSITORY_GIT_MATERIAL_PENDING) with its token committed, still fail withCONVERGENCE_DEADLINE_EXCEEDEDand retire the version: only the first-token wait keeps it.postgres-prepared-github-git-use) and "maintenance ignores a quarantined token renewal".CONVERGENCE_DEADLINE_EXCEEDEDinstead ofREPOSITORY_GIT_TOKEN_PENDING); every other test passes.postgres-applicationrun: counting a rawsucceededwork row as an earlier activation; locking the Agent before its Namespace (the fixture's lock-order check); retiring the predecessor in the kept path (now caught at once by aworker.errorwithCLAIM_LOST); keeping an exhausted new Agent's version without a token; maintenance refusing the kept version; queuing no maintenance in the kept path; maintenance exempting any failure code (the new refusal test); the deadline path retiring an existing Agent's version; maintenance not exemptingREPOSITORY_GIT_TOKEN_PENDING. The deadline-guard tests catch two more: keeping the version for any last wait, and keeping it when the token state read at the deadline isready. The earlier "accept another principal's activation" mutant no longer applies: the principal comparison it mutated was always true (a revision's principal is read from its Agent's row) and is removed; the earlier-activation check is scoped to the Agent.postgres-applicationgreen,tsc --buildclean, the matrix (6), the two new-Agent recovery tests and both exhausted-deployment tests pinned 20/20 at 2 CPUs (every run executed all of them), docs word count within limits (flow page 2477). ESLint and Prettier pass on the touched files.tsc --buildclean,postgres-applicationgreen, the matrix, recovery and exhausted-deployment tests 20/20 at 2 CPUs, all 9 mutants above caught again; the two deadline-guard tests 20/20 at 2 CPUs and each catches its mutant.Risks
REPOSITORY_GIT_TOKEN_PENDING) instead of going offline. If the token commits later, the version gets Git without a new deployment. It is the Agent's active, authorized and maintained version, so that is the access a successful deployment would have given; the deployment still shows as failed.LEASE_EXPIREDwith its runtime kept, by queue recovery SQL that cannot read the token state (unchanged).