Skip to content

fix(repository): refuse an App key that cannot sign at worker startup - #3258

Merged
freeqaz merged 2 commits into
mainfrom
fix-2217
Oct 23, 2026
Merged

freeqaz merged 2 commits into
mainfrom
fix-2217

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The worker read and parsed repository.preparedGitHub.privateKeyPath only when it first prepared a token. A missing file, a public key, a passphrase-protected key or a non-RSA key in the App-key Secret let the worker report ready. Every maintenance pass then logged PREPARED_GITHUB_MAINTENANCE_FAILED with no reason, and revisions waited for Git material that never arrived (finding 2217).

Change

  • startPreparedGitHubGitOwner now reads the App key before it starts. It parses the key the way issuance does: a private regular file, an unencrypted private key, RSA of 2048 to 8192 bits. It zeroes the file bytes afterwards. A key that cannot sign throws PreparedGitHubAppKeyError. Maintenance still reads the key again on each pass, so a rotated Secret still applies.
  • The worker logs worker.startup-error with code PREPARED_GITHUB_APP_KEY_INVALID and one of three fixed reasons: app_key_unreadable, app_key_unparsable or app_key_unsupported. The log never contains key bytes or the path.
  • Docs:
    • The team runbook has a new section, "Fix an App key the worker refuses". It lists the code and the reasons, and says that a key that goes bad while the worker runs is still found only at the next preparation.
    • The controller reference names the code.
    • breaking-changes.md has a 2026-10-23 entry: a bad key now stops the worker, and the upgrade check can't catch it.
  • The chart's pre-upgrade installation check is unchanged. It deliberately mounts no private key or custody file, so it can't check this key without widening where the key is mounted.

Verification

  • New test in configuration-startup: it boots the real worker against an unreachable database. It covers a valid key, which gets past the owner to PERSISTENCE_UNAVAILABLE, and five bad keys:

    • group-writable;
    • missing;
    • a public key;
    • an encrypted PKCS#8 key;
    • an EC key.

    Each bad key asserts the code and reason, and that stderr contains neither PEM text nor the key directory. On main the test fails: the bad keys reach PERSISTENCE_UNAVAILABLE.

  • qa-repository-installation no longer checks the mounted key by hand. Starting the owner now does that check.

  • Locally (Node 24): the whole configuration-startup and qa-repository-installation files pass (40/40). The two touched tests passed 20/20 at 2 CPUs on the final head. tsc --build, eslint, Prettier and docs:check (word limit, links, specs) are clean.

  • Internal review: approve with nits. All applied:

    • breaking-change notice;
    • the signing key is closed if the key-version export throws;
    • room is reserved for the longest socket path;
    • the test checks stderr for any PEM header;
    • a runbook subsection and a controller reference note.

    Follow-up, not done here: give PREPARED_GITHUB_MAINTENANCE_FAILED a reason when a key goes bad at runtime. That needs TokenService to pass the error through.

Risks

  • An installation whose App key could never sign used to keep its worker running for non-repository work. Its worker now stops at startup until the Secret holds a usable key. This is intended: such an installation can't prepare any repository material.
  • The worker reads the key once more, at startup. There is no new mount and no new process that reads it.

The prepared GitHub owner read and parsed repository.preparedGitHub.privateKeyPath
only at its first token preparation, so a missing, public, encrypted or non-RSA
key let the worker report ready while every maintenance pass logged
PREPARED_GITHUB_MAINTENANCE_FAILED without a reason (finding 2217).

The owner now reads and parses the key the way issuance does before it starts,
zeroes the file bytes, and refuses with PreparedGitHubAppKeyError. The worker
logs PREPARED_GITHUB_APP_KEY_INVALID with a fixed reason class
(app_key_unreadable, app_key_unparsable, app_key_unsupported), never key bytes
or the path. Maintenance still re-reads the key, so a rotated Secret applies.
The chart's upgrade check mounts no private key and is unchanged.
@freeqaz
freeqaz requested a review from a team as a code owner October 23, 2026 02:13
@clawsweeper

clawsweeper Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

Add a breaking-changes entry: a bad App key now stops the worker, which the
upgrade check cannot catch. Give the runbook its own section for the refusal
and note that a key that goes bad at runtime is still read only at the next
preparation. Mention the code in the controller reference.

Close the signing key if the key-version export throws, note why the signer's
refusal can only be the key type, reserve socket room for the longest case's
directory, and check stderr for any PEM header.
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 23, 2026
@clawsweeper

clawsweeper Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

What this changes

This PR validates the GitHub App key during worker startup and reports a fixed diagnostic before readiness when validation fails.

Example: The prepared-Git worker mounts an EC P-256 private key instead of an RSA App key.

  • Before: The worker can report ready while repository preparation repeatedly fails.
  • After: Startup exits with PREPARED_GITHUB_APP_KEY_INVALID and reason app_key_unsupported.

Review scores

Measure Result What it means
Overall readiness 🦪 silver shellfish (2/6) The implementation is focused and has meaningful subprocess coverage, but startup compatibility evidence is incomplete and the required flow update is absent.
Proof confidence 🦪 silver shellfish (2/6) Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality 🐚 platinum hermit (4/6) 1 actionable review finding remain.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Repository-enabled revisions can wait for credentials that cannot be prepared while the worker reports ready and emits an uninformative maintenance failure.
Reason: Early validation exposes an unusable dependency through fixed diagnostics, and the maintainer author has approved the resulting worker-wide startup refusal.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep this PR open: it addresses a current startup failure, and its maintainer author explicitly accepts stopping the worker when its App key is unusable.

Priority: P2
Reviewed head: ebe9918a520dabf07d5d8c92da16a320f793a0fa

Before merge

  • Document the fatal App-key check in the existing startup flow (P3) - This call adds a consequential startup branch: an invalid App key now stops the entire worker before readiness. AGENTS.md requires behavior changes to update the affected flow, and local-dev explicitly includes consequential failure handling. The PR changes guides and reference text but leaves docs/flows/production-startup.md without this ordering or refusal; the related documentation PR also omits it. Add a focused source-backed trace there, including worker-only key access, refusal before readiness, and continued key reloads during preparation.

Findings

  • [P3] Document the fatal App-key check in the existing startup flow — apps/controller/src/backends/repository-credentials/prepared-git-owner.ts:204-206

Tests

  • Missing end-to-end proof: The supplied evidence does not demonstrate fresh successful readiness and an existing-Installation upgrade followed by App-key Secret correction and recovery.
Agent review details

How this fits together

The controller worker owns prepared GitHub credentials, reads its private App-key mount, and uses the existing signing owner to prepare repository material for authorized Agent revisions.

flowchart TD
  A[Worker startup] --> B[Load repository configuration]
  B --> C[Validate mounted App key]
  C --> D[Fixed startup refusal]
  C --> E[Start prepared credential owner]
  E --> F[Reload key during preparation]
Loading

Technical review

Best possible solution:

Retain validation in the existing prepared credential owner, document its startup ordering, and establish that an existing Installation can upgrade and recover with its settings intact.

Do we have a high-confidence way to reproduce the issue?

Current-main source establishes that startup can finish without reading the App key and that preparation later encounters the invalid file; the author reports a base-failing regression test, which this read-only review did not execute.

Is this the best way to solve the issue?

Reusing the issuer's existing key restrictions at startup is a focused repair that preserves worker-only custody and rotation behavior; runtime maintenance diagnostics remain an explicitly separate follow-up.

Full review comments:

  • [P3] Document the fatal App-key check in the existing startup flow — apps/controller/src/backends/repository-credentials/prepared-git-owner.ts:204-206
    This call adds a consequential startup branch: an invalid App key now stops the entire worker before readiness. AGENTS.md requires behavior changes to update the affected flow, and local-dev explicitly includes consequential failure handling. The PR changes guides and reference text but leaves docs/flows/production-startup.md without this ordering or refusal; the related documentation PR also omits it. Add a focused source-backed trace there, including worker-only key access, refusal before readiness, and continued key reloads during preparation.
    Confidence: 0.97

Overall correctness: patch is correct
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against a651091ac5c4.

Provenance checked

  • Prepared GitHub key loading and startup changes intended behavior with a stated reason (3115679: Credential Proxy Implementation #2487 placed issuance and cleanup with the worker-owned prepared credential owner; its key read occurred during preparation.)
  • QA mounted-key verification keeps the original intent (beb627d: test(qa): move the QA matrix repository leg onto prepared native Git #3240 manually checked the key because starting the owner did not read it; the new startup validation makes that manual parsing redundant.)
  • Worker startup diagnostics keeps the original intent (ff020a4: Existing startup handling emits classified diagnostics without exposing raw configuration or exception text; the new typed refusal preserves that approach.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

Acceptance criteria:

  • python3 .agents/skills/local-dev/scripts/validate_flow_doc.py --kind flow-doc --doc docs/flows/production-startup.md
  • pnpm docs:check
  • git diff --check

What I checked:

Labels

Label changes:

  • add P2: This improves diagnosis of an operator-configured repository failure; the evidence does not establish an urgent regression affecting valid installations.
  • add merge-risk: 🚨 compatibility: An existing Installation with an unusable App key will now lose all worker processing until correction; the maintainer author explicitly accepts and documents that tradeoff.
  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

  • P2: This improves diagnosis of an operator-configured repository failure; the evidence does not establish an urgent regression affecting valid installations.
  • merge-risk: 🚨 compatibility: An existing Installation with an unusable App key will now lose all worker processing until correction; the maintainer author explicitly accepts and documents that tradeoff.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 22, 2026, 10:35 PM ET / October 23, 2026, 02:35 UTC.

@freeqaz
freeqaz merged commit 3850356 into main Oct 23, 2026
83 checks passed
@freeqaz
freeqaz deleted the fix-2217 branch October 23, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant