Skip to content

Fix run_command cleanup after shell closes - #494

Merged
sambitcreate merged 11 commits into
mainfrom
feature/run-command-process-cleanup
Oct 16, 2026
Merged

sambitcreate merged 11 commits into
mainfrom
feature/run-command-process-cleanup

Conversation

@sambitcreate

@sambitcreate sambitcreate commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Keep run_command's Unix process-group termination promise alive after the shell closes. On requested termination, the tool waits for the group to disappear or for the one-second SIGKILL escalation before settling.
  • Preserve the cancellation reason, output/error formatting, and successful-command fast path. Timeout and output overflow use the same cleanup path.
  • Add a real Node worker that ignores SIGTERM and writes a heartbeat. The regression covers early shell closure, a resistant foreground process, timeout, and output overflow. Its cancellation case fails on the unchanged original base because the descendant keeps writing.
  • Record the ownership decision and test friction in .memory/ and .papercuts/.

This adapts the owned, awaited cleanup approach from bb #5374 to Aiden's shell process group. No upstream code was copied.

Fixes #424.

Latest-main integration

Merged origin/main through 5661bae0e with ordinary merge commits, without rebasing or force-pushing. The latest merge was conflict-free and includes #496's Device Hub lifecycle fix. Remote revision 33, its fixture, and the root test scripts match main; the run_command source and tests remain this PR's only production/test diff against current main. The only conflict was in CodingToolOptions; the resolved interface retains this PR's commandTimeoutMs and main's native fileMutatorBinary. The root package.json matches main exactly, preserving its full test chain.

Local verification on the merged head

  • Native file-mutator build: passed.
  • Focused real-process run_command and edit_file tests: 8/8 passed.
  • Root and CLI type checks: passed.
  • Focused ESLint and git diff --check: passed.
  • npm run test:cli: 99/99 passed, including the shared bundled tool path.
  • Full main/services/coding-tools.test.ts: 66/67 passed. The unrelated foreground glob agrees with native glob on normal patterns, hidden paths and safe links test expects **/* not to descend through a fixture symlink. It fails identically on unchanged original production source under macOS Node 26.10.0. This is a deterministic baseline failure, not a passing rerun. The latest release-metadata-only main merge was followed by another focused 8/8 pass and CLI 99/99 pass.

Unix process tests ran on macOS. Linux was unavailable because the configured Docker daemon was stopped; Windows process-tree handling remains outside this issue. Hosted checks and bot reviews must complete on the exact pushed head before merge.

Current head: c842aaa30f6ae2bfc2fda5abe51c2d0ed06c13c8

After merging main at 5661bae0e, local Device Hub service/host lifecycle tests passed 80/80, focused run_command/edit_file tests passed 8/8, and root/CLI type checks, scoped ESLint, desktop build, and git diff --check passed. The agent-device shim that can be put on run_command's PATH is covered by the Device Hub tests. No merge conflict occurred.

Exact-head CI passed, including the E2E shards and CI required. Pullfrog found no new issues; there are no unresolved review threads. Hermes completed a full review at 21:37 UTC on this SHA with confidence 5/5 and findings []. A separate Hermes Review Bot (follow-up) status still records its 19:46 UTC engine failure from before quota recovery. The full review supersedes it as code-review evidence, but the failed status remains a gate. The bot skips an incremental follow-up when the checkpoint already equals HEAD, so one /hermes review full request was submitted to rerun that context through the normal interface. Await its result; no status was modified manually and this PR remains unmerged.

Prior-head validation: 691997c0503ee21a651a36b712c4af35f8b5c23d

After merging main at 264b537a4, focused process/edit tests passed 8/8; Remote Bot Files/session/router/protocol passed 192/192; Bot authority/application and input-modality passed 84/84. Root, CLI, and E2E TypeScript checks, scoped ESLint, CLI bundle build, desktop build, Android AidenBotContractTest, and iOS AidenBotSessionTests plus AidenRemoteClientTests on an iPhone 18 Pro simulator passed. package.json matches main. Pullfrog completed that head's check with no new issues and no open review threads. Its CI run and any Hermes result apply only to that earlier head.

Prior-head local verification: 7f1d40d520063ebdcd848396255cb5a0541b2497

The new head includes the conflict-free merge of main at 716636cf9 and integration notes. Locally, eight focused process/edit cases, 162 Remote Bot Files/router/service cases (one skip), root and CLI type checks, scoped ESLint, native helper and desktop builds, Android AidenBotContractTest, and iOS AidenRemoteClientTests on an iPhone 18 Pro simulator passed. The merged Bot Files owner-audience wiring leaves the CLI run_command path intact. package.json matches main exactly.

Two local CLI cases failed during the full 97/99 run after taking roughly 16 minutes: Bots retain one-shot notice, exact Custom tools, revocation and independent rollback anchors ended with generation authority revoked, and bundled codemode executes nested tools with current workspace authority (revoke=false) exited 143. Both passed on one isolated rerun. The focused environment-devices-stream.spec.ts:107 first timed out waiting for a saved assistant reply while generation remained active, before reaching the #508 assertion; one rerun passed in 14.6 seconds. These are flaky local results and remain recorded in .papercuts/. Local runs have no hosted run links. That head's CI and review results do not validate the current head.

Prior-head hosted review status on 2e7c26d4681a8c8683ab9eb5ac1fbaa9c4896301

Hermes Review Bot failed during its AGY engine phase with agy failed (3) before producing a code-review verdict. The deployed AGY service is reporting RESOURCE_EXHAUSTED (429) for individual quota; this is a bot infrastructure failure, not a code finding. No Aiden source change or manual Hermes trigger is proposed before quota recovery. The exact-head CI run passed on its one permitted same-head failed-job rerun, but the first-attempt E2E failures below remain bugs. Pullfrog reviewed this exact head and found no new issues; there are no unresolved review threads. Keep merge blocked until Hermes produces a real successful verdict and all exact-head gates remain green.

Exact-head CI failures, run 38854140261

Head: 2e7c26d4681a8c8683ab9eb5ac1fbaa9c4896301. The workflow reached terminal failure before any failed-job rerun.

  • Linux build x64 job 116822110938: tests/e2e/settings-model-picker.spec.ts:104, "every Settings destination renders and a one-model local inventory stays usable", saw zero [cmdk-item] rows where the one seeded model was expected. The assertion failed on the initial Playwright attempt and its built-in retry. Its sanitized receipt reports only electron-e2e/test-failed; CI does not upload the screenshot or trace.
  • Electron E2E (2/3) job 116835682784: tests/e2e/environment-devices-stream.spec.ts:107, "paints MJPEG through the proxy, sends touches, reconnects input, and attaches a screenshot", re-floated at x=285 while the prior bounding box was x=285.1875. It passed the built-in Playwright retry, so --fail-on-flaky-tests correctly failed the job. Its sanitized receipt also omits the trace and screenshot. Dependent Linux status jobs and CI required failed because these jobs were red.

Neither spec nor its model-picker/device UI code changed in this PR. Current unchanged main (515de673) passed its CI run 38853835724, including Linux build x64 and Electron E2E (2/3), so these observations do not establish a deterministic main failure. The same model-picker spec failed on independent PR #499, and separate PR #508 targeted the device fractional-coordinate assertion and is now included in the current head. No test timeout, assertion, retry count, or production behavior was changed here. Run attempt 2 passed on the same head; no second failed-job rerun will be requested. Local macOS testing passed the device spec, while the model-picker spec stopped earlier because a toast intercepted an Appearance heading click, so it did not classify the Linux missing-row symptom.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes Reviewed the process-group lifecycle fix, its callers, and behavioral coverage for cancellation, timeout, and output overflow.

  • Owned termination cleanup: Shell closure no longer cancels SIGKILL escalation while descendants remain; command settlement awaits cleanup, with an immediate path when the process group is already absent.
  • Behavioral coverage: Real SIGTERM-resistant workers verify that background and foreground processes stop writing before the tool settles, while existing cancellation reasons and error formatting are preserved.
  • Project context: Memory and papercut notes document the ownership decision and platform-specific validation limits.

Independent Linux validation passed: all 59 coding-tools tests, npm run type-check, and ESLint on both changed TypeScript files. Additional real-tool checks confirmed successful-command output and immediate cancellation when no process-group members remain. Windows was not exercised, and hosted CI is still in progress.

Pullfrog  | View workflow run | Using gpt-6.1-sol | 𝕏

@very-hermes-bot

very-hermes-bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Hermes Review Bot

Confidence: 5

Reviewed the whole PR again, with the previous Hermes checkpoint and prior reviews as context.

Engine: agy/gemini-3.8-flash-high
Review mode: full
Scope: full
Head: c842aaa30f6ae2bfc2fda5abe51c2d0ed06c13c8
Generated: 2026-10-16T21:47:13+00:00
Reviews: 2

Summary

Resolves issue #424 by maintaining process-group cleanup ownership after the shell exits, ensuring background descendants that resist SIGTERM are reliably torn down via SIGKILL. When termination is requested due to abort, timeout, or output overflow, run_command awaits a dedicated terminationCleanup promise that either verifies the absence of the process group (process.kill(-child.pid, 0)) or escalates to SIGKILL after a one-second timer. Normal completions bypass escalation with an immediate exit check, while Windows platforms preserve single-process termination semantics without attempting process-group signaling. A maintainer should double-check that the fixed one-second escalation window remains suitable for workloads where non-resistant background processes require several hundred milliseconds to complete graceful cleanup following shell exit.

Confidence Score: 5/5

Fully traced the process-group lifecycle across normal execution, abort, timeout, output overflow, and error paths on POSIX and Windows, verified Node.js process-group signal behaviors directly, and verified test coverage against descendant process status and heartbeat files.

📁 Important Files Changed
  • main/services/coding-tools.ts: Replaces premature timer clearing on shell close with an awaited process-group cleanup promise (terminationCleanup), inspects group presence via process.kill(-child.pid, 0), and exposes an optional commandTimeoutMs setting in CodingToolOptions for test deadlines.
  • main/services/coding-tools.test.ts: Adds behavioral test coverage using a SIGTERM-resistant Node worker across cancellation, timeout, output overflow, and foreground execution, verifying heartbeat cessation and process status.
  • .memory/run-command-process-cleanup.md: Records design context, successive integration merges from main, and process-group lifecycle ownership decisions.
  • .papercuts/run-command-process-cleanup.md: Documents test environment prerequisites, local execution notes, and merge conflict resolution.

Findings

No findings.

📊 Sequence Diagram

Unchanged from the previous review.

Machine-Readable Findings

[]

🕑 Review History
Round Scope Commit Confidence Status
1 initial 513f26b1e696 5/5 completed
1 initial 691997c0503e 5/5 completed
1 initial c842aaa30f6a 5/5 completed
2 full c842aaa30f6a 5/5 latest

Last reviewed commit: c842aaa30f6a
Reviews (2) · Comment /hermes review to trigger a new review · /hermes review full for full re-review

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes Reviewed the merges from main and integration notes added since the prior review at 513f26b1e.

  • Preserved both options: The CodingToolOptions conflict resolution retains commandTimeoutMs alongside fileMutatorBinary, with the shared builder forwarding each to the appropriate tools.
  • Retained cleanup behavior: The merged source preserves the reviewed process-group termination fix and its real-process regression coverage; the root package.json has no changes against main, preserving its test chain.
  • Documented integration checks: Updated memory and papercut notes record both main merges, validation results, and platform limitations.

Independent Linux validation on 2e7c26d46 passed: native file-mutator build, all 67 coding-tools tests, and root npm run type-check. Windows was not exercised, and CLI tests were not rerun in this incremental review. Hosted CI on this exact head is still pending.

Pullfrog  | View workflow run | Using gpt-6.1-sol | 𝕏

@sambitcreate

Copy link
Copy Markdown
Owner Author

/hermes review

1 similar comment
@sambitcreate

Copy link
Copy Markdown
Owner Author

/hermes review

@sambitcreate

Copy link
Copy Markdown
Owner Author

/hermes review full

@sambitcreate
sambitcreate merged commit aa86223 into main Oct 16, 2026
27 checks passed
@sambitcreate
sambitcreate deleted the feature/run-command-process-cleanup branch October 16, 2026 21:59
sambitcreate added a commit that referenced this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P2: run_command stop can leave a SIGTERM-resistant descendant running

2 participants