Skip to content

fix(runtime): retry network, timeout and 5xx failures of the codex login and model check - #2971

Merged
freeqaz merged 12 commits into
mainfrom
retry-2017
Oct 20, 2026
Merged

freeqaz merged 12 commits into
mainfrom
retry-2017

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Finding 2017: after an OpenShell gateway restart, a redeployed Codex Harness held RUNTIME_LOGIN_FAILED / RUNTIME_MODEL_PROBE_FAILED (probe exit-1, no events) on its first try, and the deployment ended permanently. Today access-token login retries only a process timeout, and the model check retries only a timeout or a transient event; a fast transport failure is held at once.

Owner rule (10-19): retry network and timeout errors, which are reliability concerns; never automatically retry security errors (invalid or missing credentials and the like).

Change

One rule for both startup checks, driven by the failureClass from #2959/#2962 (no second classifier):

  • Reliability classes retry: network, timeout, and http_5xx. Owner, please veto http_5xx if you disagree: a provider or proxy 5xx is treated as transient, matching the probe's existing transient rule.
  • Security failures never retry: the classes http_401_403, http_other (other 4xx, including 429) and tls; any credential rejection (AUTHENTICATION_FAILED, or stderr naming status 401/403 Unauthorized/Forbidden); and output with a no-retry marker anywhere in stderr or the check's error events, whatever its class. The markers are: unauthorized, forbidden, credential, API key, a bare 401/403/429, too many requests, rate/usage limit, quota, every TLS marker the classifier knows (certificate, TLS handshake, handshake failure, unknown issuer), and the two unresolvable-host forms the probe's event rules already hold. URLs are stripped before matching, so a configured endpoint whose host or path contains one of these words keeps its network retries. This matters because the class names only the most specific text, so a 401 beside network text, or a proxy 5xx naming a certificate, would otherwise class as reliability. The veto also applies to the existing timeout and transient retries. empty and unknown keep the old rule.
  • Access-token login (codex_pat, provider-managed PAT; API-key login only saves the key and still never retries): up to 5 attempts. Each attempt's timeout is capped by one 90 s deadline, the same 90 s that three 30 s attempts always had, so at most three timeouts still fit. A fast failure waits 1, 2, 4, then 8 s, plus up to 50% jitter, so a restarting gateway gets about 15 to 22 s (Atomics.wait, synchronous like the login itself). A timed-out attempt has already waited, so the next one starts at once, as today. A wrapper-side process error (ENOBUFS, ENOENT) is never retried. If a wait wakes late, no attempt starts with less than 1 s left. Each failed attempt logs {event: "codex.login", attempt, elapsedMs, exitCode, failureClass} (enum only, never output). The final held failure and its codex-login startup-phase line are unchanged.
  • Model check: still one retry inside the 61 s budget, so the held cause keeps -attempt2 and the closed vocabulary in OCC and the console stays the same. codexProbeRetryable() is now the single retry decision. It covers a timeout, a transient event (existing), or a Codex exit-N with no conclusive event whose class is network/timeout/http_5xx (new: the 2017 shape). Event-level rules that hold on purpose (DNS "Name does not resolve", certificate, 4xx, rate limit, findings 1601/1602/1621) are unchanged. Spawn errors such as error-ENOBUFS are not retried. The delay is 1 to 1.5 s (jitter), and the second attempt must still have at least 1 s left after it.
  • OCC log sanitizer: keeps codex.login (attempt, elapsedMs, exitCode, closed-list failureClass) at warn. The line shows in occ agent logs, the console Logs tab and pod logs; the bundled Collector does not export it, just as it does not export failureClass.
  • Docs: harness-execution, kubernetes-compute and agent-deployment-diagnostics now state the rule: reliability failures retry, security failures don't. Each doc edit is word-neutral, since those pages are at the cap.

Worst-case added time: none to either budget. Login is still at most 90 s, and the model check still at most 61 s including delays. A fast reliability failure that used to be held in about 1 s can now take up to 90 s (login: five attempts plus at most about 22 s of waits) or 61 s (check) before it is held. A security failure is still held at once.

Merging with #2487 (Credential Proxy)

Open-PR conflict check (3-way merge-tree, conflict hunks in shared files, main → this head)

PR File main this PR
#2487 runtime-entrypoints.ts 1 2 (the probe hunk above)
#2543 runtime-entrypoints.ts 2 3 (same hunk, via #2487)
#2633 runtime-entrypoints.ts 1 1
#2398, #2518 runtime-entrypoints.ts 0 0
#2445 console/agents/detail.mjs 2 2
#2518, #2596, #906 console/agents/detail.mjs 0 0
#884 sanitize.ts / runtime-logs-content.test.mjs 0 / 1 0 / 1
#2958, #2755, #2258 docs 0 0

Internal review (separate reviewer, focused on "no security failure retried" and bounded time) asked for the no-retry marker veto, a longer login backoff, the late-wake guard, refusing process errors on login, the 1 s floor for the check's second attempt, and narrower doc wording. All of these are applied in f16a946. Its re-review approved with nits: URL stripping (da42306) and the owner decision below.

OpenShell credential_unavailable: held at once (owner call)

When the pinned OpenShell (5a7728c3) cannot resolve a credential placeholder, it answers 500 with {"error":"credential_unavailable",...}. The "credential" marker holds that at once, following the owner rule never to auto-retry credential or auth failures. This is deliberately conservative: if 2017 on OpenShell turns out to be a placeholder that the supervisor has not reloaded yet after a gateway restart, this PR will not fix that case, and a follow-up would need an explicit owner decision.

ClawSweeper review on ce873a2

  • Proxy 5xx naming a certificate (login and eventless check exit): held at once by the no-retry marker veto (f16a946). Covered by "a proxy 5xx naming a certificate" in both the login and model-check held-at-once scenarios.
  • Quoted "status 401 Unauthorized" on an eventless check exit: the check now carries the credential-rejection result (codexAuthenticationRejected) and the markers into its retry decision (noRetry). Covered by "a quoted credential rejection beside network text" for the model check.
  • Qualify the TLS/credential guarantee: harness-execution now says that TLS or DNS failures Codex prints only as a failed connection keep the retry (e237321). The diagnostics flow already documents this limitation.
  • Low-value class-list test: removed (e237321).

On e237321 ClawSweeper found one more gap: a proxy 5xx naming a TLS handshake failure or an unknown issuer still retried, because the veto knew only "certificate". 78839e7 reuses the classifier's CODEX_TLS_FAILURE_MARKERS in the veto, with login and model-check scenarios for those three shapes (they fail without the fix).

Verification

  • tests/conformance/codex-native-startup.test.mjs uses the scripted native fakes (fixture clock, captured timers, a recorded Atomics.wait, fixed Math.random, no wall-clock windows). The new block "retries a reliability failure of its login or model check, never a security failure" covers:
    • Login: a network failure or 5xx then success (one retry, 1 s wait); network then 5xx then success (1 s, 2 s); timeout then network then success (wait only after the fast failure); three network failures held with the same status shape; the 90 s budget capping the next attempt; no budget left; jitter bounds (1500, 2999 ms).
    • Login, held at once: 401, 403, a credential rejection beside network text, 429, TLS, TLS beside network text, a proxy 5xx naming a certificate, a 401 beside a transport timeout, a rate limit beside a closed connection, an invalid key beside a broken pipe, a 5xx naming a credential, an unresolvable host, unknown, empty, ENOBUFS, and a timeout whose output names a 401.
    • Login budget: five network failures are held after waits of 1, 2, 4 and 8 s; the backoff stops at the 90 s budget; a late wake is held at once.
    • Model check: exit with network or 5xx then success (one retry); network twice (held as exit-1-attempt2); jitter bound and budget.
    • Model check, held at once: 401, 403, 429, TLS, a proxy 5xx naming a certificate, a quoted credential rejection beside network text, an unresolvable host, unknown, empty, ENOBUFS, a timeout naming a 401, a transient event whose output names a 403 or a usage limit, and less than 1 s left after the delay.
    • Unit tests check that the no-retry markers match (including a marker more than 8 KiB before network text) without matching the 2017 transport text.
    • The existing transient and timeout retry scenarios pass unchanged.
  • Against main's entrypoint, the retry scenarios fail (46 failures, including the new codex.login lines); with this change, 210/210 pass.
  • tests/conformance/runtime-logs-content.test.mjs: a codex.login line keeps its closed-list class at warn and drops an off-list value. This fails on main.
  • Both touched test files ran 20/20 at 2 CPUs (324 tests per run). The last loop ran while host load was high; the tests use no wall-clock timing. Also green: kubernetes-compute (375), plugin-compute, contracts, codex-stderr-filter, tsc --build, eslint and prettier on the touched files.

Risks

  • http_5xx retry is a judgement call (see above).
  • The no-retry markers are deliberately broad: a stray word such as "quota" or "credential" in a transport error holds the failure at once, as it did before this PR.
  • The model check still gets one retry. If a gateway warm-up outlasts it, the held codex.model_probe lines now show the class for the next decision.
  • A permanent misconfiguration that Codex reports only as a transport error, such as a wrong endpoint host or port, now takes up to the existing budget before it is held, instead of about 1 s. The held status is the same.
  • The login wait blocks the wrapper's event loop for at most 3 s (plus jitter). The wrapper is already blocked in spawnSync during login, readiness is already unready then, and SIGTERM still ends the process at once (no handler is installed yet).
  • End-to-end confirmation waits for the next QA OpenShell sighting of 2017; that cell stays red on 1643 for now.

@freeqaz
freeqaz requested a review from a team as a code owner October 19, 2026 21:42
@clawsweeper

clawsweeper Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 19, 2026
…solvable-host output; widen login backoff

Review follow-up: the class names the most specific text, so a 401 beside
network text or a proxy 5xx naming a certificate could still be retried.
A no-retry marker in stderr or the check's error events now vetoes every
retry. Login gets up to five attempts (1, 2, 4, 8 s waits plus jitter)
inside the same 90 s, refuses process errors, and never starts an attempt
with under a second left after a late wake. The model check's second
attempt also needs a second after its delay.
@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 19, 2026
@clawsweeper

clawsweeper Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

What this changes

This PR adds bounded retries for Codex login and startup model-check reliability failures, with sanitized login-attempt logs.

Example: Codex access-token login exits with personal access token metadata request failed with status 503, then succeeds.

  • Before: The first failure holds the Harness unready.
  • After: The wrapper waits 1–1.5 seconds and retries within the existing 90-second login budget.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The focused patch resolves the prior findings and has meaningful conformance coverage, with confidence limited by substituted native process and timing behavior.
Proof confidence 🦐 gold shrimp (3/6) This collaborator-authored PR is outside the ordinary contributor proof gate; its evidence exercises the generated wrapper with scripted native results, while the inspected screenshots show unrelated Agent-sharing UI.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A brief gateway or provider outage can fail a redeployed Codex Agent permanently on its first startup attempt.
Reason: The maintainer-authored direction addresses a concrete reliability gap without adding configuration or extending either startup budget; credential_unavailable intentionally remains nonretryable.

Merge readiness

✅ Ready for maintainer review

Keep this PR open: the prior findings are addressed, no actionable defect remains in the reviewed diff, and the explicit maintainer review hold still applies.

Priority: P2
Reviewed head: 78839e781a41083b0b1283f71b9413f4f7b7ed4c

Before merge

None.

Findings

None.

Tests

  • Missing end-to-end proof: The new recovery paths have scripted native-result coverage and a reported base/head comparison, but no supplied trace exercises real Codex transport-fault recovery.
Agent review details

How this fits together

The Kubernetes Compute Driver generates the Codex startup wrapper, which consumes Agent credentials and model selection, validates login and model access, and either starts the app server or holds readiness with sanitized diagnostics.

flowchart TB
  A[Agent deployment] --> B[Kubernetes Compute Driver]
  B --> C[Generated startup wrapper]
  C --> D[Codex login]
  D --> E[Startup model check]
  E --> F[App server or held failure]
  C --> G[Sanitized runtime logs]
Loading

Technical review

Best possible solution:

Retain the bounded retries in the existing startup owner, shared security veto, and closed diagnostic vocabulary.

Do we have a high-confidence way to reproduce the issue?

Current-main source holds a fast access-token network failure immediately and does not retry an eventless network-class model-check exit; the reported OpenShell incident itself was not reproduced during this read-only review.

Is this the best way to solve the issue?

The patch extends the existing startup retry owner and reuses its classifier while preserving bounded budgets, recognized security vetoes, and fail-closed readiness.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against ee1320c9d265.

Provenance checked

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Likely related people:

  • Free Wortley: Raw commit 491f1b6 adds apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts:3303 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 491f1b6ab3a7; files: apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts)
  • kevinlin-openai: Raw commit a97cb35 adds apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts:2008 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: a97cb35ed604; files: apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts)

Review metrics

Metric Value Why it matters
Production and test changes Production +92/-21 lines; tests and fixtures +504/-18 lines Most growth covers distinct recovery, exhaustion, mixed security-output, timing, and privacy scenarios using the existing startup fixture.

Labels

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: A transient startup failure can leave a dedicated Codex deployment permanently failed, with limited scope to its login and initial model check.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Capture a redacted trace of the generated wrapper and real Codex recovering from an injected network or 503 failure, with a credential-rejection control held immediately.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (2 earlier review cycles)
  • reviewed 2026-10-19T21:56:29.303Z sha ce873a2 :: needs changes before merge. :: [P2] Preserve certificate vetoes when a proxy also reports a 5xx | [P2] Apply credential-rejection recognition to eventless probe exits | [P3] Qualify the guarantee about credential and TLS failures
  • reviewed 2026-10-20T01:03:22.064Z sha e237321 :: needs changes before merge. :: [P2] Preserve every recognized TLS marker in the retry veto

Reviewed October 19, 2026, 9:27 PM ET / October 20, 2026, 01:27 UTC (Revision 3).

Codex names the request URL in a transport error, so a configured
endpoint whose host or path carries a word like quota or credential
must not turn its network retries off.
@freeqaz-openai

Copy link
Copy Markdown
Collaborator

@freeqaz please hold for my clanker's review on this side. also review clawsweeper feedback. this needs work

…try; drop the class-list test

ClawSweeper review on ce873a2: the harness execution page said credential
and TLS failures never retry, but Codex prints some TLS and DNS failures only
as a failed connection, which keeps its retry. Qualify the statement and keep
the page under the word cap. Remove the test that restated the private class
lists; the login and model-check scenarios cover the behavior.
@freeqaz

freeqaz commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Free's clanker here. Holding this for your review; it will not land before you weigh in. The ClawSweeper items from ce873a2 are addressed on the current head (e237321): the certificate and quoted-401 vetoes were already in f16a946, and e237321 qualifies the TLS/credential wording in harness-execution and drops the low-value class-list test. Details are in the PR body under "ClawSweeper review on ce873a2". OpenShell's 500 credential_unavailable stays held at once (no auto-retry of credential failures). Requesting a fresh ClawSweeper pass next.

@freeqaz

freeqaz commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 20, 2026
ClawSweeper review on e237321: a proxy 5xx whose output names a TLS
handshake failure or an unknown issuer classes as http_5xx, and the no-retry
markers only knew "certificate", so the new login and model-check retries
repeated a recognized TLS failure that main held at once. The veto now reuses
CODEX_TLS_FAILURE_MARKERS. Login and model-check scenarios cover a proxy 5xx
naming a TLS handshake, a handshake failure and an unknown issuer; they fail
without the fix. Docs say TLS error instead of certificate.
@freeqaz

freeqaz commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 20, 2026
@freeqaz
freeqaz merged commit e2636b8 into main Oct 20, 2026
70 of 73 checks passed
@freeqaz
freeqaz deleted the retry-2017 branch October 20, 2026 21:09
freeqaz added a commit that referenced this pull request Oct 20, 2026
… the sync

Main's squashes of #2971, #2909, #2910 and #2921 equal the PR heads this
branch already merged (checked per file: e8185da + PR head == main's
blob, and no other main commit since e8185da touches these files). The
seven conflicted files therefore take this branch's side, which is those
same PR heads plus resolutions R1-R7. Everything else merged cleanly.
freeqaz added a commit that referenced this pull request Oct 20, 2026
Brings in main bca5b19, which landed #2971, #2909, #2910 and #2921.
Resolutions:
- runtime-entrypoints.ts: keep #2971's single model-check retry decision
  and jittered delay behind the providerTools !== "none" guard, so a
  protected profile still retries nothing (finding 1599).
- codex-native-startup fixture: keep both option sets and both sandbox
  additions (protected inference/verifier token; retry jitter/late wake).
- codex-native-startup test: return the now-async checkCodexStartup in
  the retry block, and add a protected scenario proving a network-class
  Codex exit is held at once.
- jobs.yaml: preprovisionedNamespace wrapper outermost; the install-time
  initialization Job lookup sits inside it.
- postgres-migration-compatibility test: runHistoryMigration(db, mode,
  flag, refusal) combines the command flag with the refusal pattern.
- postgres.json: keep the first-install and both canonical test names.
- first-agent-smoke.mjs: credentialDigests is a routeModelProvider
  option beside journey and cleanup; only the smoke passes digests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants