Repository navigation
fix(codex): the provider supervisor outlives its provider group when stopped - #23466
Conversation
…stopped A signalled supervisor forwards the signal to the provider group, escalates to SIGKILL after the grace, and exits only once the group is gone, so recovery's proof that the recorded pid is dead also proves the provider is. It refuses to spawn when its parent is already not the owner named in its spec, and watches that owner rather than whichever parent it first saw. The grace is a spec field. Recovery's SIGTERM stage now outlasts the supervisor's own stop, since a SIGKILL that lands first cannot be handled and leaves the group running.
…its provider group When Orca dies, the supervisor's stdout pipe has no reader. Provider output in the window before the parent-death watch fired raised an unhandled EPIPE that exited the supervisor with the provider group still running.
…ways covers it Recovery sized its SIGTERM stage from the default grace, so a launch with a longer grace would be SIGKILLed mid-stop and orphan its group with no test noticing. The spec now refuses any grace above one exported maximum, and recovery derives its SIGTERM stage from that maximum.
Owner death, stdin end after the grace, and a signal to the supervisor now all take one path: SIGTERM the provider group, SIGKILL it after the grace, and exit only once it is gone. The signal handlers are registered before the provider is spawned, so a stop that lands in the spawn window still reaps it. The longest stop grows to two graces plus the reap wait, and both recovery's SIGTERM stage and the connection's graceful close now wait that long before forcing, since forcing the supervisor sooner can orphan its group.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (5)
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review. 📝 WalkthroughWalkthroughThe POSIX provider supervisor now accepts owner and grace-period options and uses SIGTERM followed by bounded SIGKILL escalation to stop the provider process group. Connection closing and structured recovery use the supervisor stop limit for their waits. New unit and integration tests cover supervisor shutdown, owner loss, connection-close timing, and recovery after supervisor exit. Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to The POSIX provider shutdown changes appear ready to merge after normal checks. No actionable shutdown or recovery regression was established. Security Architecture ReviewSecurity architecture risk: 🟡 Moderate · up to The shutdown sequence is designed to keep the supervisor alive until its provider stops, and callers now allow more time for that sequence. Unusual cleanup failures can still leave provider lifetime uncertain, so session ownership during recovery merits attention. Retained concerns Security review detailsSecurity Blast Radius
Security Findings and Attack Paths
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
- Unified provider-group stop. Every supervisor trigger now SIGTERMs the detached provider group, escalates to SIGKILL after the spec grace, and exits only once
kill(-pgid, 0)reports the group gone; a received signal is re-raised afterward so a watcher still sees death by signal. - Owner pid and grace travel in the spec.
supervisedPosixLaunchstampsownerPid/graceMs, refuses a grace aboveMAX_PROVIDER_SUPERVISOR_GRACE_MS, and the supervisor refuses to spawn when its parent is already notownerPid. The handlers are registered before the spawn. - Closed output pipe no longer ends the supervisor.
errorhandlers onprocess.stdout/process.stderrjoin the existing ones, so a dead owner'sEPIPEcannot kill the pid before it reaps the group. - Recovery and
close()wait out the full stop.STOP_PROBES.SIGTERMrises 4 → 17 (≈4.25 s) andclose()waitsPROVIDER_SUPERVISOR_MAX_STOP_MS(4 s) on POSIX, both derived from the supervisor's max stop so the SIGKILL cannot land mid-stop. - Tests. New real-process integration coverage for signal-handed stops, the spawn window, owner death, the grace bound, and recovery eviction; connection unit tests use the new close bound.
ℹ️ The reaping wait depends on the host reaping orphaned grandchildren
The stop logic is correct, but the new integration tests assume the supervisor's kill(-pgid, 0) probe clears once the group is SIGKILLed. On a host whose PID 1 does not reap orphans, the SIGTERM-ignoring grandchild stays a zombie with a process-table entry, the group never clears, and the supervisor takes the full grace + 1500 ms before exiting 1. I reproduced this here and confirmed the pre-existing teardown integration test fails identically, so it is host behavior rather than a regression in this PR; GitHub's runner reaps and passes. No change requested.
Technical details
# Zombie members keep `kill(-pgid, 0)` true
## Affected sites
- `src/main/codex/codex-app-server-posix-supervisor.ts` — `providerGroupExists()` / `waitForProviderGroupExit()` clear only when the group has no process-table entry, which a zombie still does.
- `src/main/codex/codex-app-server-posix-supervisor.integration.test.ts` — the SIGTERM and owner-death cases assert the group clears inside their windows.
- `src/main/codex/codex-app-server-teardown.integration.test.ts` — pre-existing, same assumption.
## What I observed
- In this sandbox (PID 1 = `su`, no reaper) the grandchild is SIGKILLed but reads `/proc/<pid>/stat` state `Z` with PPID 1, and `kill(-pgid, 0)` keeps succeeding, so `reapOwnedProviderGroup()` returns false and `stopProviderGroup` exits 1.
- 5 of the new integration tests fail for this reason; the pre-existing teardown integration test fails the same way.
## Required outcome
- No code change requested. The bounded wait plus exit-1 fallback keeps the invariant sound (a zombie cannot run user code); this note only flags that the tests are host-reaper sensitive.DeepSeek Flash (free via Pullfrog for OSS) | 𝕏
|
Extra QA: does a Codex tool command outlive an Orca crash? PASS, 3 of 3 runs on this PR ( Codex was running Why this PR matters here: Codex starts each shell command in its own process group, so the supervisor's group signal can't reach the command directly; Codex kills it during its own shutdown. On main the supervisor force-kills Codex about 100 ms after Orca dies, with no grace, which can beat that shutdown and leave the command running. An earlier run on main did leave a |
|
Status: ready for review. Head Review: two review loops; the second was clean. The design was challenged by an adversarial review before it was built. Readiness check: passed with no release-blocking findings. It ran at the end of the review only, not at the start as well; that's disclosed here. One minor finding, deferred: the provider-neutral supervisor still lives in a Codex module, and recovery imports its timing from there; it moves when #23476 reuses it for Claude. Live QA:
Follow-up (not this PR): the Codex connection still picks its stop wait with its own platform check ( |
…RM to flush before SIGKILL The supervisor's stop was stdin end, 1.25 s, SIGTERM, 1.25 s, SIGKILL. Codex now gets 3 s after SIGTERM to flush its state. The two graces are separate constants, the longest stop they derive becomes 5.5 s, and a test keeps it inside quit's 8 s child-eviction bound.
|
| await waitForProcessExitUntil( | ||
| exitPromise, | ||
| process.platform === 'win32' ? GRACEFUL_EXIT_MS : PROVIDER_SUPERVISOR_MAX_STOP_MS | ||
| ) |
There was a problem hiding this comment.
Forced stop can orphan children
If the host is under load, the supervisor's timers can run late, but close() forces teardown at exactly the nominal 5,500 ms stop time. If the provider has exited while a child in its process group ignores SIGTERM, teardown can stop the supervisor before it sends SIGKILL to that group. The reparented child may be missing from the descendant snapshot and keep running after close. Recovery has the same narrow timing margin: it waits only one extra 250 ms probe before SIGKILLing the supervisor. Both paths need time to observe the supervisor finishing its stop.
Knowledge Base Used: Agent workspaces and sessions
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Incremental pass over the one commit after 3b374050c6, which splits the supervisor's single grace into the two waits the PR body now documents. No new defects; the ordering guarantees still hold at the wider bound.
- Split the supervisor grace into two named waits.
DEFAULT/MAX_PROVIDER_SUPERVISOR_GRACE_MS(both 1250) becamePROVIDER_STDIN_END_GRACE_MS(1000, given before SIGTERM) andPROVIDER_SIGTERM_GRACE_MS(3000, given before SIGKILL); the spec now carriesstdinEndGraceMs/sigtermGraceMs, each bounded by its own default inassertGraceWithin, so no spec can carry a wait recovery andclose()don't cover. - Raised the longest stop to 5.5 s.
PROVIDER_SUPERVISOR_MAX_STOP_MSis now the sum of both graces plus the 1.5 s reap, and both recovery's SIGTERM stage (ceil(MAX/250)+1= 23 probes, ≈5.75 s) and POSIXclose()derive from it, keeping recovery's SIGKILL after the supervisor's own group SIGKILL (≤4.5 s). - New timing coverage. Integration tests assert the stdin-end fast path adds no wait and that SIGTERM lands ≥1 s after stdin end with SIGKILL ≥3 s later; the teardown test pins
MAX_STOP + 1 sunder quit's 8 s child-eviction budget, and the connection unit tests use the new close bound.
DeepSeek Flash (free via Pullfrog for OSS) | 𝕏
Eviction drains the sink for up to 1 s before it stops the child, inside the same 8 s bound.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Incremental pass over 189fce772f, the one commit after 20e81ad43f. It makes the quit-budget test count the eviction's pre-stop drain, so the phase bound is accurate rather than undercounted. No new defects.
- Drain timeout exported and folded into the quit budget.
SNAPSHOT_DRAIN_TIMEOUT_MSis now exported fromstructured-agent-session-eviction.ts, andstructured-agent-session-host-teardown.test.tsasserts1_000 + 5_500 + 1_000 < 8_000. Both terms genuinely run inside theevict-owned-sessionsphase —snapshot-before-stopdrains the sink (≤1 s) beforestop-provider-childcloses the Codex connection (≤5.5 s on POSIX) — so the tightened assertion reflects the real worst case.
The invariant and its ordering guarantees are unchanged from the prior pass: while any member of the provider group is alive, the supervisor pid is alive, and recovery's SIGTERM stage plus POSIX close() both outlast the supervisor's longest stop.
DeepSeek Flash (free via Pullfrog for OSS) | 𝕏






ELI5
On macOS and Linux, Orca runs each Codex chat under a small watcher process. Orca remembers the watcher's process ID and later judges whether the chat's agent is still running by checking whether that ID is alive. Today, if something politely asks the watcher to stop, it dies on the spot and leaves the real Codex process running. Orca then sees the watcher gone, concludes Codex is gone too, and lets a second Codex attach to the same conversation while the first one is still writing to it. This change makes the watcher stop Codex first and only then exit, so "watcher gone" really means "Codex gone".
What Changed
In the code the watcher is the POSIX provider supervisor, an inline Node script, and the process it runs (Codex) is the provider. The provider runs in its own process group, so the supervisor can signal it and the processes it starts together. The supervisor's own behaviour changes in seven ways, and two callers now wait long enough for it.
The supervisor
PROVIDER_STDIN_END_GRACE_MS,PROVIDER_SIGTERM_GRACE_MS) and two spec fields instead of one hard-coded number. Codex keeps the defaults, and building a spec refuses either wait above its default, so no launch can carry a wait that recovery and close do not wait out.EPIPEthat killed the supervisor with the provider group still running. The supervisor now ignores errors on its own stdout and stderr, as it already did on its stdin and the provider's pipes, so the watch still stops the group first.The callers that wait for it
PROVIDER_SUPERVISOR_MAX_STOP_MS: stdin end, 1 s, SIGTERM, 3 s, SIGKILL, then at most 1.5 s for the group to disappear. That is 1000 + 3000 + 1500 = 5500 ms, so recovery keeps checking for about 5.75 s before SIGKILL. It is derived from that maximum in code, so raising the maximum widens the stage with it, and a longer wait fails at spec construction instead of silently reopening the gap. Recovery returns as soon as the owner is proven gone, so it only waits that long when a provider ignores SIGTERM.close()ends the supervisor's stdin, then forces the process tree if it has not exited. On POSIX it used to force after 1.5 s. Now it waitsPROVIDER_SUPERVISOR_MAX_STOP_MS(5.5 s), and Windows keeps 1.5 s. Under SIGTERM-first stopping, the 1.5 s force landed mid-stop:The 40-launch teardown test caught this: all 40 child processes survived. A Codex that exits on stdin end still closes as fast as before (about 5 ms). Only a provider that ignores stdin end waits for the supervisor's own stop.
Quit budget. On quit, Orca gives the shutdown of all chats 8 s (
CHILD_EVICTION_TIMEOUT_MSinstructured-agent-session-host-teardown.ts; chats shut down in parallel). Before stopping each agent it waits up to 1 s for already-delivered events to be saved, so the worst case is 1 s plus the 5.5 s longest stop, which leaves 1.5 s to spare. A test fails if those two plus 1 s of headroom reach 8 s.How a second Codex got in, before: recovery SIGTERMs the recorded pid, which is the supervisor → the supervisor dies by default action → the next check finds the pid gone → recovery treats the owner as proven dead and clears it → a new Codex is made the owner while the old Codex app-server is still running on the same thread.
After: the SIGTERM reaches the supervisor's handler, which was registered before the provider existed → the provider group gets SIGTERM, then SIGKILL 3 s later if anything in it is still running → the supervisor exits only once the group is empty → the next check finds the pid gone, and that is now true of the provider too → clearing the owner is sound.
The window before the owner was known. The provider used to be spawned before the supervisor read its parent's pid. If Orca died in that gap, the supervisor recorded its new parent (PID 1, or whichever process the system hands orphaned processes to) as its owner, and the parent-death watch never fired.
That gap is narrower than it sounds. On main, Orca's death already closes the supervisor's stdin, and the stdin-end path stops the provider group after a short wait, even inside the window. A provider is stranded only if some other process holds the write end of that pipe open. This PR makes the protection direct instead of relying on that: the supervisor checks its parent against the owner pid from the spec before it spawns anything, so a supervisor whose parent is already not Orca never starts a provider.
Why
Everything that judges Codex by its recorded pid (the owner check, recovery, the exit relay) assumed the watcher dies only after Codex does. The watcher broke that by dying on any polite stop, which let recovery hand a live conversation to a second Codex. Making the watcher stop Codex first restores that guarantee in one process, so no reader needs to learn a new field.
Put as one rule: while any member of the provider group is alive, the supervisor pid is alive. The alternatives each add a guard somewhere else instead:
The pre-spawn check follows the same rule: re-derive the owner from a fact the spawner states, rather than trust whatever the supervisor happened to observe after the spawn. No field is persisted. The spec lives only in the supervisor's environment, and the script and the spec are always built by the same build, so no version skew is possible.
What SIGKILL to the supervisor still cannot cover: an uncatchable SIGKILL sent before its stop finishes. Orca's own senders are covered. Recovery sends SIGTERM first and waits out the supervisor's longest stop. The Codex connection's close waits the same bound before its forced teardown. That teardown freezes the supervisor, kills the captured child processes by identity, and only then kills the supervisor. One pre-existing exception is unchanged here: if the descendant snapshot cannot be taken, teardown SIGKILLs the supervisor directly (
codex-app-server-process-teardown.ts:117-119). A SIGKILL from outside Orca (a user'skill -9) still orphans the group. The stdin-close backstop cannot help there either, because the provider's stdin belongs to the dead supervisor.Differences from the common pattern
The usual way to build a watcher like this: pass SIGINT or SIGTERM on to the program it runs, SIGKILL the program if it has not exited after a grace period, and exit only once the program and everything it started are gone. This PR does the same. Where it differs:
Linked Issue
N/A (maintainer follow-up to #23460, which removes the startup sweep that killed processes by their inherited spawn tag).
Visual Proof
This is a process-lifetime change with no UI change. The evidence is timings from a live run on a macOS machine: an isolated build and profile, a structured Codex chat running a 90 s loop, a watcher polling every 10 ms, and a second watcher inside the provider group that logs the signal it receives. Base =
9795563695, PR =3b374050c6. The build was confirmed per run from the built bundle.kill -9of Orca's main mid-tool, provider ignoring stdin endkill -9of Orca's main mid-tool, stock CodexAfter a crash and relaunch, and after a clean restart, a send is answered.

Testing
Real-process tests (macOS). Each new test was confirmed red with its core line deleted, then restored from the commit.
codex-app-server-posix-supervisor.integration.test.tstouchmarker is never created). Deleting the check makes the provider run.--requirepreload wrapsspawnto SIGTERM the supervisor the moment the provider exists. The supervisor dies by SIGTERM and the provider is gone. With the handler registration moved back after the spawn, the provider survives (red).codex-app-server-posix-supervisor.test.ts) Either wait one millisecond above its default is refused; the default itself is accepted. Deleting the bound check turns this test red.EPIPEfirst and the provider survives (red 3 of 3, green 3 of 3).structured-agent-session-recovery-resolution.test.ts: recovery runs against a real supervisor whose provider ignores SIGTERM, and clears the owner on proof of death only after the provider is gone. With main's recovery timing restored, the provider is still alive when the owner is cleared. Deleting the SIGTERM handler has the same result.structured-agent-session-host-teardown.test.ts: the 1 s wait before the stop, plus the longest stop, plus 1 s, stays under quit's 8 s (7500 ms). Raising the SIGTERM wait to 6 s turns it red (9500 vs 8000), and so does raising the 1 s wait to 2 s (8500 vs 8000).3b374050c6(1.25 s and 1.25 s), after = this head.Existing suites pass:
src/main/codex,src/main/native-chat/agent-session-wire, andsrc/shared/child-process, including the child-process import ratchet and the 40-iteration teardown integration tests (about 8 to 9 s each now that each close can wait 5.5 s).close()back at its 1.5 s force, that teardown test goes red: all 40 SIGTERM-ignoring child processes survive.src/main/codexandsrc/shared/child-process1743 pass;src/main/native-chat/agent-session-wire821 pass. One lease-renewal test that never starts a supervisor (structured-agent-session-lease-renewer.test.ts) failed once under load average about 35 and passed 3 of 3 on its own.pnpm tc:node,pnpm tc:web,oxlinton the changed files,check:code-quality:changedandaudit:anti-slopall pass.Not run: Linux (the tests are POSIX and not macOS-specific, and CI covers Linux), and a live Codex app-server stop. Windows is unaffected, because it does not use the supervisor.
I manually tested these changes locally
Automated tests added/updated, or explained why not below
AI Disclosure
Written with AI assistance (Claude), and reviewed and validated by the author.
Review
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
codex-app-server-process-teardown.ts:117-119), which can orphan the provider group.Author X handle: @BrennanKB5
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)