Skip to content

fix(runtime): hold provider 4xx and configuration failures at startup without a retry - #2650

Merged
freeqaz merged 3 commits into
mainfrom
fix-1601
Oct 16, 2026
Merged

freeqaz merged 3 commits into
mainfrom
fix-1601

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Follow-ups to #2639 (startup model probe retry), found in a bug hunt:

  • 1601: codexTransientFailure retried any provider 4xx whose text held a transient word, such as unexpected status 404 Not Found: overloaded or a 402 with a server_error body. A 400 is worse: Codex prints it as its bare body with no status, so a 400 body saying overloaded or server_error was retried too. That cost a second, possibly billed, model call and up to about 31 s before the failure was held as turn-failed-attempt2.
  • 1602: Codex reports a content-filter stop, a rejected certificate and a provider host name that does not resolve as stream disconnected before completion: …, so they also got the retry, though they fail the same way again.
  • 1603: agent-deployment-diagnostics.md implied the Agent Gateway sees the held code. Its ws client never reads the 503 body and logs only Unexpected server response: 503.
  • 1604: the reference table, the contract description and the OpenAPI file listed exact cause tokens only, so a caller matching detail === "rate_limit" missed rate_limit-attempt2.
  • 1605: three wrapper HTTP handlers in runtime-entrypoints.ts (the native worker readiness server, the held-failure server and the plugin/runtime status server) called new URL(request.url, base) unguarded. A request target of // threw, which ends the wrapper; the container restarts and runs a new startup model probe. Only peers the NetworkPolicy allows can reach these ports, and the HTTPRoute replaces the path.

Change

  • codexTransientFailure returns false for any provider 4xx status before the 5xx/marker test, and for an invalid_request_error body (how a 400 reaches the probe).
  • It also returns false for content_filter, certificate, and the musl resolver's Name does not resolve and Name has no usable address (the shipped Codex is the musl build). Codex's HTTP client prints a failed connection as Connection failed: error sending request without its cause, so a DNS or TLS failure reported only that way cannot be told from a dropped connection: it keeps its retry, and a code comment says why. musl's Try again (EAI_AGAIN) also stays retried.
  • A shared runtimeRequestPathname helper answers 400 {"error":"bad_request"} (no-store) when the target does not parse; all three handlers use it. The held-failure server still checks the bearer token first.
  • Docs: the flow doc now says the Gateway logs only the status and Compute's authenticated status read carries the code. The reference, the contract detail description and the regenerated OpenAPI say any Codex detail may end in the single -attempt2 suffix. The reference page stays under the word limit (a few words trimmed in the same section).

No change to how credentials reach the probe, the retry budget, the cause vocabulary or the schema.

Verification

Internal review: first pass requested changes (400 shape, musl resolver text, suffix scope), applied in the second commit; second pass approved with nits (captured certificate text, doc wording), applied in the third.

  • Message shapes: I ran the pinned Codex build (0.163.0-alpha.2 linux-x64 musl, registry integrity checked) with a synthetic key against local fake providers and a .invalid host, over HTTPS Responses and WebSocket, with and without Codex's own retries. It printed a 400 as the bare body, a 402/404 as unexpected status 404 Not Found: overloaded, url: …, a WebSocket DNS failure as stream disconnected before completion: failed to lookup address information: Name does not resolve, a self-signed certificate over WebSocket as … invalid peer certificate: Other(OtherError(CaUsedAsEndEntity)), and every HTTPS connect failure (DNS, TLS, refused) as the same Connection failed: error sending request. The fixtures use those shapes.
  • New tests fail on main: 7 "held at once, with no second probe" cases in codex-native-startup.test.mjs (404 overloaded, 402 server_error, 400 bare body, content filter, certificate, two DNS forms) and 4 cases in openshell-harness-status.test.mjs (held server, Compute status port, native worker readiness port, and a raw GET // request line over a real socket that leaves the wrapper serving).
  • Kept-retry controls: a 500, Connection failed: error sending request and a Try again DNS lookup still recover on the second attempt.
  • 21 mutants, 19 killed: the 4xx guard removed, its upper bound shifted either way or moved after the 5xx test, the exclusion removed or each marker dropped, the exclusion widened to EAI_AGAIN or the bare request error, the helper not answering / answering 404 / rethrowing, each of the three call sites reverted, and the held server parsing before its token check. The two survivors move the guard's lower bound from 400 to 401; they are equivalent in practice, because Codex never prints a 400 status and a 401 is already held as a credential failure.
  • Touched test files 20/20 at 2 CPUs. codex-native-startup 105/105, openshell-harness-status 23/23; kubernetes-compute, plugin-compute, kubernetes-runtime-credentials, runtime-model-probe-settlement, runtime-probe-cpu-counter, workspace-node-supervisor pass.
  • tsc, eslint and prettier on touched files clean; openapi:check current.
  • No Docker or cluster run; First Agent Smoke runs in CI.
  • Open PRs touching these files: Credential Proxy Implementation #2487 and feat(runtime): apply Gateway tool policy to dedicated OCE deployments #2633 have the same conflicted files and hunk count against this head as against main (24/56 and 9/16), and none of this change's lines sits in a conflict block.

Risks

  • Low. A provider 4xx that really would pass on a second try (for example a 408) is now held at once; startup can be retried by redeploying. A 5xx whose body names a certificate (a TLS-inspecting proxy's) is held too, which matches a configuration fault.
  • A DNS or certificate failure reported only as the bare error sending request still gets one retry, as before.

… without a retry

A provider 4xx is never retried, whatever its body says (finding 1601).
A content-filter stop, a rejected certificate and a host name that does
not resolve are held at once (1602). Wrapper status servers answer 400 to
a request target that is not a path instead of crashing (1605). Docs:
the Gateway logs only the 503 status (1603), and the -attempt2 suffix is
documented in the reference, contract and OpenAPI (1604).
@freeqaz
freeqaz requested a review from a team as a code owner October 16, 2026 22:33
@clawsweeper

clawsweeper Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review blocked

Automated review did not run, so no review verdict was produced.

Reason: The input-safety check rejected material in this revision. No detected value, path, or scanner output is reproduced here.

Automatic review is on hold, including after source changes. Request a fresh re-review after resolving the failure; maintainers can also release the hold through an intentional scanner-policy update.

Next step: If this is a genuine credential, remove and rotate it. If it is an intentional test fixture, a maintainer must review and qualify it.

View the workflow run.

@freeqaz
freeqaz temporarily deployed to integration-qa-pr October 16, 2026 22:33 — with GitHub Actions Inactive
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 16, 2026
… messages

Internal review: Codex prints a 400 as its bare body, so the 4xx guard
never saw one; hold an invalid_request_error body. The shipped Codex is
built against musl, whose resolver says "Name does not resolve" and
"Name has no usable address", not glibc's text. Fixtures now use the
messages the pinned build printed against local fake providers. The
reference says any Codex detail may end -attempt2.
…a malformed request

Second internal review nits: the certificate fixture now carries the text
the pinned Codex printed over WebSocket for a self-signed provider, and
the reference says the provider rejected the malformed request again.

This branch was previously deployed

1 inactive deployment
integration-qa-pr — e9f7330e Deployed Oct 16, 2026 by freeqaz via QA advisory (kubernetes) #1600
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant