Skip to content

Contain Codex child stdin release races - #2506

Open
ymichael wants to merge 1 commit into
mainfrom
bb/ci-flake-fix-codex-topology-late-child-release-r-thr_wqsu3g5tkg
Open

Contain Codex child stdin release races#2506
ymichael wants to merge 1 commit into
mainfrom
bb/ci-flake-fix-codex-topology-late-child-release-r-thr_wqsu3g5tkg

Conversation

@ymichael

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

The recurring packages failure was amplified by package-shard CPU oversubscription but rooted in two lifecycle races. First, the topology test started a 200 ms real-time request deadline before the app-server subprocess had executed, then required the fake to have logged both spawn: and exit:; under contention the runtime correctly released the construction before the fake ran its first line, so spawned() === 1 && exited() === 1 was impossible forever. Instrumentation reproduced that state in all 40/40 stressed failures (spawned=0, no live PID, no failed-release warning). Second, once the test made the construction boundary deterministic, release could kill the app-server while the Codex bridge was making its next initialization write; the resulting child-stdin EPIPE was unhandled and crashed the shared bridge. This is distinct from #2352: its predicate-before-deadline polling fix remains intact, and these failures were real lifecycle states that never satisfied the predicate. The independently verified merge base is 524b2fe4b0df74185bdd29a274fb77cd6765b2b3.

What changed

  • Treat an absent, unwritable, or asynchronously broken app-server stdin as a connection failure: reject pending child requests, terminate the unusable child through the existing SIGTERM-to-SIGKILL policy, and let the normal exit event finalize the connection. Child termination is now idempotent, so an EPIPE during an already-started release cannot schedule duplicate kill sequences or crash the bridge.
  • Add a real child-process regression that closes fd 0, remains alive, and proves the connection contains the broken pipe, rejects the request, and reaps the child.
  • Make the topology scenario deterministic without increasing any timeout: stall the fake thread/start, freeze only the runtime request clock until the real child has started, advance the existing 200 ms deadline, and assert the recorded PID is actually gone. This covers both graceful SIGTERM and the connection's intentional SIGKILL escalation.

There is no server/daemon wire-contract change, so HOST_DAEMON_PROTOCOL_VERSION is unchanged. There are no CLI, guide, or configuration-surface changes.

How you verified

  • Red-capable exact stress harness, 64 parallel focused runs on 16 logical CPUs:
    • Before: 62/64 passed; 2/64 failed with the exact Timed out after 10000ms waiting for the late-constructed child was released signature. Test-body p50/p95/max were 2.59 s / 3.01 s / 12.62 s.
    • After, under the same 4x oversubscription: 64/64 passed, 0 exact signatures. Test-body p50/p95/max were 1.94 s / 2.47 s / 2.70 s.
  • Broken-stdin regression: before the connection fix, the request never received a connection failure and the child stayed alive until Vitest's 5 s wrapper failed; after the fix, the focused test passes in 25 ms.
  • pnpm exec turbo run test --filter=@bb/agent-runtime --force -- --run src/runtime.codex-topology.test.ts — 1 file / 4 tests passed.
  • pnpm exec turbo run test --filter=bb-plugin-provider-codex --force -- --run src/bridge/app-server-connection.test.ts — 1 file / 1 test passed.
  • pnpm exec turbo run test --filter=@bb/agent-runtime --filter=bb-plugin-provider-codex --force — agent-runtime 22 files / 318 tests and Codex provider 25 files / 239 tests passed.
  • pnpm exec turbo run typecheck --filter=@bb/agent-runtime --filter=bb-plugin-provider-codex --force — passed.
  • pnpm exec turbo run build --filter=@bb/agent-runtime --filter=bb-plugin-provider-codex --force — applicable Turbo build graph passed (the two scoped packages have no direct build scripts).
  • pnpm exec prettier --check packages/agent-runtime/src/runtime.codex-topology.test.ts plugins/provider-codex/src/bridge/app-server-connection.ts plugins/provider-codex/src/bridge/app-server-connection.test.ts plugins/provider-codex/src/bridge/fake-codex-app-server.mjs — passed.

Fixes: no issue — exact and fuzzy open issue/PR searches found no active recurrence fix.

AGENT GENERATED: by GPT-5.6-Sol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant