Skip to content

fix(runtime): run the native codex binary so the model check timeout ends it - #3449

Merged
freeqaz merged 2 commits into
mainfrom
fix-2274
Oct 24, 2026
Merged

freeqaz merged 2 commits into
mainfrom
fix-2274

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The Codex Harness startup model check runs spawnSync("codex", ...) with a 30 s timeout and SIGKILL. Access-token login does the same, for up to five attempts. In the runtime image, codex is the npm launcher (@openai/codex/bin/codex.js). It starts the platform package's native binary with inherited stdio, and on SIGTERM it forwards the signal to that binary. SIGKILL cannot be forwarded, so the timeout killed only the launcher. The native process kept running after the check gave up, still retrying against the model provider with the check's credential, and it held the inherited pipes. So the 30 s cap was not a hard limit on the attempt's process (finding 2274). After a login timeout, the next attempt started while the previous native login process was still running with the credential.

Change

  • The model check and native login run the binary that the launcher would start, found the same way, through one helper, codexExecutable(). The helper takes the first codex on PATH. If it resolves to @openai/codex/bin/codex.js, the helper resolves @openai/codex-linux-<arch>/package.json from the launcher, as the launcher does, and falls back to the launcher's own vendor directory. The binary is vendor/<arch>-unknown-linux-musl/bin/codex. The timeout's SIGKILL now ends the process that makes the model calls.
  • Any other codex on PATH (a native install or a test fixture) still runs as codex. The arguments, environment, timeout, output limit and result handling do not change. The launcher sets only update and doctor metadata (CODEX_MANAGED_*) that codex exec does not use. The native binary finds its bundled resources relative to its own path.
  • app-server still goes through the launcher. It is long-lived, and forwardTermination sends SIGTERM, which the launcher forwards.

Why not #3353: that draft makes the check async so it can kill a process group. That needs the vm-sandbox startup harnesses reworked, and it takes the dedicated Harness command over the 64 KiB exec-string budget (66,072 bytes). This change stays synchronous. It changes about 15 lines and keeps the command at 65,237 bytes. #3353's other half, stopping at the first 401, is a separate question and stays with that PR.

Verification

  • tests/integration/codex-model-probe.test.mjs:
    • New scenario "timeout then success behind the npm launcher". The fixture lays out the npm packages as the image does: the launcher on PATH starts the native fixture, which hangs on the first call. Every scenario now also checks the container's process table after the check settles. A held failure leaves no Codex process, and a ready check leaves only app-server.
    • On main the new scenario fails: the timed-out native codex ... exec process is still running after the check settled. With the fix all 12 cases pass. Run locally against a Node 24 image.
  • kubernetes-provider-harness conformance: the Disable unbounded connection retries for protected Codex launches #3310 command-shape regex now names codexExecutable(). The vm-sandboxed startup harnesses (codex-native-startup, openshell-harness-status) still see codex, because their stubbed fs has no realpathSync.
  • kubernetes-provider-harness plus codex-native-startup pass 20/20, pinned to 2 CPUs. The new scenario passed 20/20 before the login change. On the final head it passed 18/20, five runs at a time, each pinned to 2 CPUs, under a local Podman engine. Both failures were the existing 5 s docker exec timeout of the status snapshot helper, not an assertion. Main's unchanged "timeout then success" scenario hit the same timeout in 1 of 20 runs under the same load.
  • Internal review (high effort): land with nits. Applied: login uses the helper too, and the helper is shorter.
  • These pass: codex-native-startup, kubernetes-compute (exec-string budget), kubernetes-runtime-credentials, openshell-harness-status, plugin-compute, workspace-node-supervisor, contracts, codex-stderr-filter and egress-enrollment (899 tests). ESLint and Prettier pass.
  • Image lanes: the entrypoint is controller-rendered, so no image recipe, pin or version changes, and the native image check does not apply. The Images and Packaging lane runs codex-model-probe. Image Runtime Startup runs the real launcher and binary in the runtime image.

Risks

…timeout ends it

The image's codex command is the npm launcher, which starts the native binary
with inherited stdio. The model check's 30 s spawnSync timeout killed only the
launcher; the native child kept retrying with the pipes open, so spawnSync
waited on and the timeout was no hard limit. The check now runs the binary the
launcher would start, resolved the same way, so the SIGKILL ends it. Any other
codex on PATH runs as before. Finding 2274.
@freeqaz
freeqaz requested a review from a team as a code owner October 24, 2026 12:44
@clawsweeper

clawsweeper Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

Review: access-token login retries a timed-out attempt up to five times, and
each timeout's SIGKILL ended only the npm launcher, so a native login process
kept running with the credential while the next attempt started. Login now
uses the same helper, renamed codexExecutable and moved to the program's top
level so the login block can call it. The helper is folded to one variable,
which saves bytes in the rendered command (65,237 of 65,536).
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 24, 2026
@clawsweeper

clawsweeper Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge.

What this changes

This PR runs native Codex directly for startup login and model checks so their timeout kills the process doing the work.

Example: A Codex startup model check exceeds its 30-second timeout.

  • Before: SIGKILL ends the npm launcher while its native child retains the pipes and continues running.
  • After: SIGKILL targets the native Codex process directly.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) A focused, source-supported repair with useful container regression coverage and green required checks, held below a pass by unresolved stored-command compatibility.
Proof confidence 🦐 gold shrimp (3/6) The ordinary contributor proof gate does not apply to this collaborator-authored PR; supplied verification describes the generated launcher running in Node 24 containers with a substituted CLI. The stored-command compatibility gate still applies: no result shows an existing OpenShell Sandbox surviving controller upgrade or following a supported replacement path with its valid state intact.
Patch quality 🦐 gold shrimp (3/6) 1 actionable review finding remain.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A startup check can exceed its advertised timeout while native Codex continues running, including overlapping credential-login attempts.
Reason: The collaborator-authored direction restores the existing timeout contract without adding configuration or changing retry policy.

Merge readiness

⛔ Blocked before merge - 2 items remain

Keep this PR open: it addresses a current startup timeout defect with a focused implementation; the required runtime-flow documentation still needs updating.

Priority: P2
Reviewed head: 5bb44b04edcb08b3a58fac9eca76f5442ca1b06a

Before merge

  • Document direct native execution in the existing startup flow (P3) - The helper changes which process owns login and model-check execution and receives the timeout signal, but this PR updates no runtime flow. AGENTS.md and local-dev require a source-backed flow update for consequential runtime-path changes. Update step 5 of docs/flows/native-service-account-credential-delivery.md to explain native executable resolution and timeout termination, while distinguishing the launcher-backed app-server path; the reference page's word limit does not prevent this focused flow update.
  • Add data-model compatibility proof - The review found that existing stored data may not work after upgrade. Show that existing data still loads and works with this change.

Findings

  • [P3] Document direct native execution in the existing startup flow — apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts:3445-3454

Tests

  • Missing end-to-end proof: Existing-Sandbox upgrade behavior remains unverified; the reported base-failing/head-passing container scenario exercises a fixture CLI rather than native Codex.
Agent review details

How this fits together

The Kubernetes Compute Driver renders the Harness launcher, which consumes projected credentials, performs Codex login and a model check, then publishes readiness and starts app-server.

flowchart TB
  Compute[Kubernetes Compute Driver] --> Launcher[Generated Harness launcher]
  Credentials[Projected model credentials] --> Launcher
  Launcher --> Native[Native Codex login and model check]
  Native --> Provider[Model provider]
  Native --> Gate[Startup result and readiness]
  Gate --> Server[Codex app server]
Loading

Technical review

Best possible solution:

Keep direct native execution in the existing launcher, document its process boundary, and establish compatible handling of previously stored Sandbox commands.

Do we have a high-confidence way to reproduce the issue?

Current-main spawnSync calls, the runtime image symlink, and the pinned upstream launcher establish the surviving-child path; the author also reports a base-failing container regression, but this review executed no target code.

Is this the best way to solve the issue?

Directly invoking the same native executable is a small repair that preserves synchronous startup, retry decisions, and the existing command budget without requiring the draft PR's asynchronous conversion.

Full review comments:

  • [P3] Document direct native execution in the existing startup flow — apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts:3445-3454
    The helper changes which process owns login and model-check execution and receives the timeout signal, but this PR updates no runtime flow. AGENTS.md and local-dev require a source-backed flow update for consequential runtime-path changes. Update step 5 of docs/flows/native-service-account-credential-delivery.md to explain native executable resolution and timeout termination, while distinguishing the launcher-backed app-server path; the reference page's word limit does not prevent this focused flow update.
    Confidence: 0.98

Overall correctness: patch is correct
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 6f81df8e4df5.

Provenance checked

Testing

Proof path: in-process harness.

Security

None.

Evidence

Acceptance criteria:

  • python3 .agents/skills/local-dev/scripts/validate_flow_doc.py --kind flow-doc --doc docs/flows/native-service-account-credential-delivery.md
  • pnpm docs:check
  • git diff --check

What I checked:

Likely related people:

  • Free Wortley: Raw commit e2636b8 adds apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts:3588 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: e2636b8f01c4; files: apps/controller/src/drivers/compute/kubernetes/runtime-entrypoints.ts)

Review metrics

Metric Value Why it matters
Production and test changes Production +15/-2 lines; tests +62/-5 lines The test growth adds a distinct launcher-child timeout regression and observable process-cleanup assertions.

Labels

Label changes:

  • add P2: The defect affects dedicated Harness startup timeout enforcement and can leave native login or model-check processes running.
  • add merge-risk: 🚨 compatibility: Changing the generated command changes a stored Sandbox specification that must match exactly during reuse.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

  • P2: The defect affects dedicated Harness startup timeout enforcement and can leave native login or model-check processes running.
  • merge-risk: 🚨 compatibility: Changing the generated command changes a stored Sandbox specification that must match exactly during reuse.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 24, 2026, 9:16 AM ET / 13:16 UTC.

@freeqaz
freeqaz merged commit bb7df09 into main Oct 24, 2026
79 of 81 checks passed
@freeqaz
freeqaz deleted the fix-2274 branch October 24, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant