Skip to content

fix(orchestration): report a wait-for-setup readiness timeout as a setup wait - #25221

Open
brynnclaw wants to merge 2 commits into
stablyai:mainfrom
brynnclaw:fix/orchestration-setup-wait-timeout
Open

brynnclaw wants to merge 2 commits into
stablyai:mainfrom
brynnclaw:fix/orchestration-setup-wait-timeout

Conversation

@brynnclaw

Copy link
Copy Markdown
Contributor

ELI5

Some repos run a long setup script (install, build) before Orca lets a worker agent start. When the 60-second start budget ran out, worker-start always blamed the agent (agent_readiness / timeout), whatever setup was doing. Now, when the budget runs out under wait-for-setup, worker-start asks the setup terminal what setup actually did and reports that: still running (with the flag that gives it more time), failed with its exit code, or finished (then the agent really was the slow part).

What Changed

  • Before: with startupPolicy: wait-for-setup, a readiness timeout always gave failedStage: agent_readiness, lastError: "timeout", and setup.state: running in the same receipt, even after setup had exited. Nothing pointed at --timeout-ms.
  • After, for a terminal worker, worker-start checks the setup terminal for up to 1 s when the readiness wait times out:
    • Setup still running: failedStage: setup_wait, lastError: "Setup was still running after 60000 ms (startup policy wait-for-setup), so the agent could not start. Retry with a larger --timeout-ms.", setup.state: running.
    • Setup exited non-zero: failedStage: setup_wait, lastError: "Setup failed (exit N), so the agent could not start (startup policy wait-for-setup).", setup.state: failed. A longer timeout would not help here, so the message does not suggest one.
    • Setup exited 0: failedStage: agent_readiness, lastError: "timeout" as before, but now with setup.state: succeeded.
    • No setup terminal to ask (an in-process hook, or the setup terminal is already gone): unchanged, agent_readiness / timeout.
  • For a structured (claude/codex chat) worker, the setup gate already races the real setup-completion signal. When that race times out, the start now reports setup_wait with the "still running" message instead of agent_readiness.
  • Mechanism: the setup receipt cannot answer the question. It is set to running when the worktree is created and only changes when a readiness wait returns. A wait that rejects with timeout leaves it at running. A setup that fails also leaves the agent pane's shell alive (the gate prints "Setup failed; skipping agent startup." and the agent never starts), so the readiness wait times out in that case too. The new setupErrorAtReadinessTimeout in worker-setup-gate.ts asks waitForSetupTerminalCompletion (the same signal the structured gate uses), raced against a 1 s probe and cancelled by an AbortSignal so no observer is left behind. An exit it sees is persisted through the existing persistWorkerSetupWaitOutcome, so the receipt and the stored dispatch both say failed or succeeded.
  • Not changed: the default budget (still 60 s), task delivery, timeouts under start-immediately, a blocked agent, and a readiness wait that returns exited for a failed setup.

Why

Under wait-for-setup the agent's start is held behind the setup hook. Callers read failedStage and lastError to decide what to do. A receipt that blames the agent while setup is still working makes them tear down a healthy worker. A receipt that says "still running, retry with a larger timeout" after setup has failed sends them into a longer retry that fails the same way. The probe gives each case its own answer.

Alternatives considered: trusting setup.state at the timeout (it is stale on this path, see above); raising the default budget when a setup hook exists, or making the budget follow setup activity (points 2-3 of the issue); or reporting a non-terminal outcome instead of failed (point 1). Points 1-3 change behavior for every repo or change the worker lifecycle and retry rules, so they need a maintainer decision and are out of scope. This PR addresses point 4 (name --timeout-ms when it would help) and labels the failed stage from what setup actually did.

Linked Issue

Addresses #21512

This addresses point 4 of the issue and the stage label. The receipt still says state: failed for a setup that is still running, and points 1-3 stay open (see Notes).

Visual Proof

N/A. No visual change: this only changes the failedStage, lastError and setup.state fields of the orchestration worker-start RPC/CLI receipt for readiness timeouts under wait-for-setup. No UI shows them.

Testing

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below

In src/main/runtime/rpc/methods/orchestration/worker/workers-new-worktree.test.ts:

  • reports a readiness timeout while wait-for-setup is still running as a setup wait: setup terminal still running at the timeout gives setup_wait, the "Setup was still running after 1000 ms" text and --timeout-ms, and the setup terminal is asked. Fails on main.
  • reports a wait-for-setup readiness timeout after setup failed as a setup failure: setup exited 3 gives setup_wait, "Setup failed (exit 3)", setup.state: failed, and no "still running". Fails on main.
  • keeps a wait-for-setup readiness timeout after setup succeeded at agent readiness: setup exited 0 gives agent_readiness / timeout with setup.state: succeeded. Fails on main (setup.state stays running).
  • reports a structured worker whose setup gate timed out as a setup wait: a structured worker whose setup-completion race times out gives setup_wait. Fails on main.
  • keeps a wait-for-setup readiness timeout with no running setup at agent readiness: a skipped setup keeps agent_readiness and the setup terminal is not asked. Passes before and after.
  • keeps a start-immediately readiness timeout at agent readiness: control. Passes before and after.

Each guard in the new code was removed once on purpose. Every removal made at least one of these tests fail.

Ran on Linux: the 119 orchestration and worker-start test files (two mail test files timed out under host load and passed 48/48 when rerun alone), pnpm lint, tsc --noEmit for tsconfig.tc.cli.json, and tsc --noEmit with the tsconfig.node.json settings over the three changed files. The full pnpm test, tsc for the whole tsconfig.node.json and tsconfig.tc.web.json, and pnpm build were not run locally (memory limits on the test machine). Not tested on macOS or Windows. The change is platform-neutral TypeScript.

AI Disclosure

Claude Opus 5.5 (claude-opus-5-5), Anthropic. An automation session wrote the change, tests and this description.

Review

Reviewed before submission by an automated reviewer against the change and its tests.

  • Cross-platform: TypeScript control flow and message strings only. The probe uses the existing waitForSetupTerminalCompletion, which reads the completion marker that both the POSIX (bash) and the Windows (powershell.exe) observed setup commands print. No paths, shells, shortcuts or OS calls are added.
  • SSH / remote / local: changes only the local worker-start path (local-worker-start.ts, worker-setup-gate.ts). Federated (remote-host) worker starts in federation.ts keep their current behavior; that can be a follow-up. No RPC param or wire shape changes: failedStage already uses setup_wait for a failed setup.
  • Agents and integrations: provider-neutral. It applies to every terminal agent, and to structured (claude/codex) sessions, only under the repo's wait-for-setup policy.
  • Performance: only on the failure path. A setup that has exited normally answers at once, from the completion marker in its replayed output or from its terminal's exit. A setup that is still running adds at most 1 s to a start that has already used its whole budget. The observer is aborted when the probe ends.
  • UI: none.
  • Security: none. The error text holds only the budget in milliseconds, an exit code and a flag name.

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Issue scope. The template asks for "Fixes #", but this PR does not resolve the whole report. The receipt still says state: failed while setup is still running, and points 1-3 of the issue (a non-terminal outcome, an activity-based budget, a policy-derived default) are left for a maintainer decision. Maintainers may prefer to keep worker-start reports state:failed / agent_readiness while setup.state is still 'running' under wait-for-setup #21512 open for those.
  • Design choice for maintainers. The probe window is a fixed 1 s. An exited setup normally answers at once from its replayed completion marker; a setup that has not answered in 1 s is reported as still running. If the setup terminal is gone, or the hook ran in-process with no setup terminal, there is no evidence and the old agent_readiness / timeout stays.
  • Overlap with open PRs. Several open PRs edit local-worker-start.ts. Start orchestration workers with their brief on the launch command line (step 4 of 7) #24081 rewrites this readiness block (launch-brief workers wait on a real setup-completion race), so whichever lands second needs a rebase, and the "still running / failed / succeeded" split would move into its setup gate. fix: failed worktree setup left the gated agent terminal silently waiting for two hours #16891 changes the setup gates so a failed setup no longer leaves the agent waiting; with it, more failed setups would reach the existing exited path, and this PR's probe still covers the rest. fix(orchestration): cancel timed out setup observers #22259 cancels timed-out observers in the structured gate; the probe here already aborts its own observer.
  • Callers that matched failedStage === 'agent_readiness' for a timeout while setup was running, or after setup failed, now see setup_wait, the stage that already means "the setup gate did not let the agent start".

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred). pnpm lint and the CLI typecheck config pass locally. The node and web typecheck configs, the full test suite and the build were not run locally; CI covers them.

…tup wait

Under startupPolicy wait-for-setup the agent cannot become ready before
the setup hook exits, so a readiness budget that runs out while setup is
still running is the setup gate's timeout. worker-start now reports
failedStage setup_wait with an error that says setup was still running
after N ms and names --timeout-ms, instead of agent_readiness / timeout.

Addresses stablyai#21512
…or-setup readiness timeout

The setup receipt stays `running` on a terminal readiness timeout: only a
wait that returns updates it, and a failed setup leaves the agent pane's
shell alive, so readiness times out then too. Reporting "setup was still
running" from that receipt mislabels a failed or finished setup.

At the timeout, worker-start now probes the setup terminal (1 s, aborted
afterwards) and reports what it saw: still running -> setup_wait with the
--timeout-ms hint; exited non-zero -> setup_wait "Setup failed (exit N)"
and setup.state failed; exited 0 -> agent_readiness with setup.state
succeeded; no setup terminal -> unchanged. A structured worker whose own
setup-completion race timed out reports setup_wait.

Refs stablyai#21512
@coderabbitai

coderabbitai Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (1)
AGENTS.md — auto-discovered

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e5068a7d-847d-4350-8a6b-77c286c0c160
📥 Commits

Reviewing files that changed from the base of the PR and between 8c61730 and ef8ccb7.

📒 Files selected for processing (3)
  • src/main/runtime/rpc/methods/orchestration/worker/local-worker-start.ts
  • src/main/runtime/rpc/methods/orchestration/worker/worker-setup-gate.ts
  • src/main/runtime/rpc/methods/orchestration/worker/workers-new-worktree.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

Worker startup now checks setup state when agent-readiness waiting times out. A running setup can be reported as a setup_wait timeout, and a nonzero or unknown setup exit can be reported as a setup failure. Successful or skipped setup preserves the agent-readiness timeout classification. Tests cover these outcomes for terminal and structured workers, as well as start-immediately startup.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to ef8cc

The change appears mergeable after normal checks. No concrete issue with setup-timeout classification remains identified.

Security Architecture Review

Security architecture risk: 🔵 Low · up to ef8cc

This changes how setup-related start failures are reported, not who can launch work or access resources. No new security weakness was established, but interruption and recovery behavior remains partly verified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated effect is limited to the affected dispatch's failure classification and setup evidence. The new branch neither sends terminal commands nor continues startup after a readiness timeout.

Trust Boundaries and Controls

  • inferred — The inspected caller relationship retains coordinator/run fencing and task ownership validation. Timeout classification remains downstream of those checks and upstream of worker authority acquisition; no bypass was established in that path.

Resilience and Maintainability Implications

  • observed — The terminal probe clears its timer and aborts its completion observer on every settlement path. Runtime cancellation unsubscribes data observation and cancels the internal exit wait without stopping setup. Failed starts still enter centralized teardown and failure receipt handling.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: reporting readiness timeouts under wait-for-setup as setup-wait failures.
Description check ✅ Passed The description covers the required sections and explains the change, rationale, linked issue, testing, review considerations, and checklist. It also states which checks and platforms were not tested …
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the full 3-file diff at ef8ccb7228 — the new setup-terminal probe for wait-for-setup readiness timeouts, its wiring into startLocalWorker, and the added tests.

  • worker-setup-gate.ts — new setupErrorAtReadinessTimeout. Under wait-for-setup with a running setup receipt, races waitForSetupTerminalCompletion against a 1 s timer and an AbortController, then maps the outcome to a still-running error, a persisted failed/succeeded receipt (via persistWorkerSetupWaitOutcome), or null to keep the agent-readiness timeout.
  • local-worker-start.ts — stage relabeling. Wraps the readiness wait in try/catch and, on a 'timeout' rejection (or a structured gate timeout), calls the probe before rethrowing, setting failedStage = 'setup_wait' when setup blocked the agent.
  • workers-new-worktree.test.ts — five scenarios plus controls. still-running, exited-nonzero, exited-zero, no-setup-terminal, the structured-gate timeout, and a start-immediately control, all asserted on failedStage / lastError / setup.state / effects.

I verified the probe's race, timer, and abort lifecycle (the late rejection after abort.abort() is handled by Promise.race's own reaction — no unhandled rejection or orphaned observer), that both terminal readiness lanes reject with the exact 'timeout' message the catch matches, and that an observed exit is persisted through the same receipt object the failure path returns so setup, effects, and failedStage agree. The --timeout-ms the message names is the real worker-start flag. The exited-zero test genuinely fails on main (setup.state stays running).

ℹ️ Remote-host worker starts still report the old stage

src/main/runtime/rpc/methods/orchestration/federation/federation.ts:220 runs its own readiness wait without the probe, so a federated (remote-host) worker start under wait-for-setup keeps failedStage: agent_readiness / lastError: 'timeout' on a readiness timeout even when setup was still running or had failed. The PR body discloses this as a follow-up; flagging it so remote parity stays tracked, since this change closes #21512 for local workers only.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash (free via Pullfrog for OSS) | 𝕏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant