Repository navigation
fix(orchestration): let a worker start helpers on the Run it works for - #29205
Conversation
A dispatched worker could only start helper workers by taking a coordinator seat on a new Run, and taking a seat moved its inbox. worker-start now also accepts a worker on an active Dispatch: its helpers run in the same Run, report to it (worker_done, heartbeat, escalation, status, ask), and it answers their questions. The worker's own mailbox never moves. When it settles, whatever its helpers sent that it never handled goes to the Run's coordinator. Authority comes from the worker's own Dispatch, re-checked inside the reservation transaction. Helpers are fresh agents on this host; a worker may retry only its own helper's latest attempt. Remote workers keep the old own-Run instructions. No saved field or wire change.
|
Workers previously had to create a new Run to start helpers, which moved their inbox and made coordinator follow-ups difficult to receive. This PR lets a local dispatched worker start helpers under its current work and receive their reports without changing inboxes. The two earlier recovery defects have been addressed on 640232c, and fresh independent verification is pending. Unanswered helper questions (including ones already read) and unread completion, escalation and status messages now move together to the Run when their creator settles; the same transfer covers messages sent during a stop. Delivery state is reset, old deliveries are fenced, and the new owner is notified after the transaction commits. Recovery uses the recipient mailbox and Run, so a status message does not need a payload. The required three-model design panel and fresh plan check completed before implementation. The source trace covers rollback, committed-row revalidation, existing restart recovery, replay attention and mailbox-scoped late delivery stamps. Saved-data host downgrades remain outside the supported contract; no RPC parameter, stream opcode or stored-column change was introduced. Remote workers keep their existing launch and relay path. Update for head Verified: the full PR workflow 38995786372 passed on this exact head, including static analysis, typecheck, all unit-test shards and the final PR checks passed gate. Real runtime QA on macOS with a real Grok agent (main vs this head, results and screenshots in the PR description) passed the core flow: main refuses a worker's direct helper; this head keeps worker and helper on one Run, delivers the helper's completion and the coordinator's follow-up to the worker's own check, and the worker finishes last. The three-level hand-back and the older own-Run path were checked from source and the flow tests, not live. The PR description now says exactly what moves to the Run (reports and unanswered questions, not heartbeats) and that #29006 touches the same delivery files. The bot thread about heartbeats was answered and resolved under the panel ruling. Generation and process-incarnation concerns without concrete failing inputs remain deferred; local-only helper scope and legacy own-Run nesting remain temporary boundaries tracked in STA-10701. For Brennan's review Human judgment points are the intended shared Run fallback when a creator settles and the temporary remote-helper exclusion; no new product decision is requested. #29006 changes the same delivery files, so whichever lands second needs a real merge and a fresh CI run. Main-sync update: Current-head integration validation passed on Windows low spec: 43 real source CLI commands against the production RPC server/runtime and on-disk database, with two named suites passing all six tests. Coverage includes helper creation, same-Run parentage/depth, scope refusals, report/question delivery and replies, coordinator follow-ups, completion before and after creator settlement, acknowledgement replay, pending-question/payload-free report hand-back, stop-window/rollback recovery, and database reopen. Agent launch/readiness use automated fixtures; no current-head real-agent or rendered-app claim is made. The orchestration/CLI QA requirement is covered by these tests. The earlier macOS real-agent QA and screenshots remain evidence only for the previous head. The full PR workflow passed on this exact head. One supplemental WSL job timed out during Ubuntu distro import, before dependencies or Orca tests; the same stage timed out on main, the workflow is unchanged here, and other distro jobs passed. The failed-jobs-only retry passed, including Ubuntu distro import and both Orca WSL phases. Review and integration validation are complete on this head, with no new code changes needed. Non-product observations: existing Vite configuration warnings and an install notice for an unapproved browser skill did not block the integration tests; no approval/install of that skill was performed. Initial diagnostic attempts needed fixture identity and durable result logging corrections. Final command logs, runner/process identities and a zero-survivor check are saved privately; initial short-lived install/test subprocess identity recording was incomplete. Real configuration hashes remained unchanged and the QA slot and terminals were released. For Brennan's review The intended behavior is that helpers remain visible to the Run coordinator and receive Run-wide group sends, and a surviving coordinator receives eligible mail after the creator finishes. The temporary remote-helper boundary is described above. Existing main-versus-branch screenshots in the PR body show the prior-head core flow; they are historical evidence. #29006 is still open and changes overlapping delivery paths: whichever lands second requires a merge review and fresh checks. |
…s live delivery When a worker that started helpers settled, its helpers' unhandled mail was re-addressed to the Run by settlement SQL alone. Nothing woke the Run's coordinator, delivered rows kept their old delivery stamp so pointers and the restart scan skipped them, payload-free status reports were left behind because selection required payload.dispatchId, and a question asked while the creator was being stopped stayed with the stopped creator. - Select handed-back mail by its recipient mailbox and Run (unread reports plus pending questions, even ones already read), reset delivery and pointer state on the moved rows, and fence the old mailbox's Deliveries. - After the settling transaction ends, the runtime re-reads exactly the moved rows and wakes the Run with their real types; a rollback wakes no one and a failure falls back to the existing repoint retry. Restart recovery is the existing unread-undelivered scan. - Hand mail back again at the final stop settlement, which also closes the stopped Dispatch's own questions asked during the stop. - A worker_done receipt replay re-offers the handed-back targets. - Scope markAsDelivered to the mailbox it pointed, so a late pointer stamp from the old owner cannot hide mail now owned by the Run.
…r-29205 # Conflicts: # src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments: 1
skill-guides/orchestration/references/worker-contract.md: 1
56-69: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument the unresolved hand-back gap for helper questions.
Line 69 says a helper still running after you settle reports to the coordinator's Run. The PR comments report that an unanswered helper question moves to the Run mailbox without waking the coordinator. The helper can stay blocked.
The guide tells workers to wait for every helper's
worker_donebefore settling. That rule limits the exposure. It does not remove it for stop or failure settlement. Fix the recovery defect inquestion-threads.ts. Until then, do not describe hand-back as reliable.
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
e8a83e09-a702-4f37-8365-a7cf77c936cc
📒 Files selected for processing: 36
skill-guides/orchestration.mdskill-guides/orchestration/references/coordinator-loop.mdskill-guides/orchestration/references/messaging-and-gates.mdskill-guides/orchestration/references/worker-contract.mdsrc/cli/bundled-skill-guides.tssrc/main/runtime/orca-runtime-test-orchestration-messages.spec.tssrc/main/runtime/orchestration/db-message-timestamp.test.tssrc/main/runtime/orchestration/db-messages.test.tssrc/main/runtime/orchestration/db-undelivered-mailboxes.test.tssrc/main/runtime/orchestration/db/messages/helper-mailbox-hand-back.tssrc/main/runtime/orchestration/db/messages/message-inbox.tssrc/main/runtime/orchestration/db/orchestration-db.tssrc/main/runtime/orchestration/db/questions/question-threads.tssrc/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-start.tssrc/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-stop.tssrc/main/runtime/orchestration/dispatch-report-mailbox.tssrc/main/runtime/orchestration/mailbox-pointer-stage.tssrc/main/runtime/orchestration/message-batch-atomicity.test.tssrc/main/runtime/orchestration/preamble.test.tssrc/main/runtime/orchestration/preamble.tssrc/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.tssrc/main/runtime/orchestration/structured-mailbox-pointer-delivery.tssrc/main/runtime/rpc/methods/orchestration/federation/federation.tssrc/main/runtime/rpc/methods/orchestration/messaging/ask-methods.tssrc/main/runtime/rpc/methods/orchestration/messaging/check.test.tssrc/main/runtime/rpc/methods/orchestration/messaging/helper-question-answerer.tssrc/main/runtime/rpc/methods/orchestration/messaging/message-methods.tssrc/main/runtime/rpc/methods/orchestration/messaging/send-methods.tssrc/main/runtime/rpc/methods/orchestration/messaging/send-point-to-point.tssrc/main/runtime/rpc/methods/orchestration/worker/local-worker-start.tssrc/main/runtime/rpc/methods/orchestration/worker/worker-start-authority.tssrc/main/runtime/rpc/methods/orchestration/worker/worker-topology.tssrc/main/runtime/rpc/methods/orchestration/worker/workers.tssrc/main/runtime/structured-chat-worker-helpers.test.tssrc/main/runtime/terminal/layout/terminal-resize-listeners.tssrc/main/runtime/workspaces/persisted-surfaces/automation-operations.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.
… mail back Every settlement runs the hand-back, so a full dispatch row read there added a statement per settlement. Also expect the mailbox-scoped delivered stamp in two delivery tests from main.
…e replay A helper creator's worker_done hands its helpers' unread reports and pending questions to the Run. The send stored the Run's extra wake targets in the mutation receipt, but the executor rewrites that receipt on completion and on each replay, so an identical retry only woke the original destination. Derive the targets at replay time instead: for the settled Dispatch (when it has same-Run helpers), notify run:<id> for each type of unread report or pending question there. Notification only; nothing is re-settled or inserted. The receipt augmentation is removed.
# Conflicts: # src/main/runtime/rpc/methods/orchestration/worker/workers.ts
ELI5
An agent that Orca hands a task to (a "worker") sometimes needs helpers of its own, for example a review loop that wants two reviewers. Orca refused to let the worker start them on the Run it was working for. The only way around it was for the worker to start a whole new Run of its own, and doing that silently moved the worker's inbox, so messages from the agent that gave it the task stopped arriving where the worker was looking. Now a worker starts helpers as itself, the helpers report back to it, and its inbox never moves.
What Changed
The problem
ctx_cd859ba985d0, Runrun_0c6e0ad8ad5b) ranorca orchestration worker-start --run run_0c6e0ad8ad5b --from <itself> --parent <its task> --agent codex ...and gotconsumer_fenced: worker-start requires the coordinator terminal currently bound to the Task Run.No helper was created and the review of Chat: dropped files hold Send until they're attached #29173 finished incomplete.run-create). That makes the worker the coordinator of a new Run, and Orca then delivers everything addressed to the worker into that new Run's mailbox. Reading the old Run (check --run <old>) returnsconsumer_fenced. On 10-15 a watcher that hid that error missed about an hour of the coordinator's mail; on 10-14 several workers then ranrun-useon the Run they worked for and took it away from its real coordinator (Stop workers from taking over the Run they work for #29008).What changes for agents and the people supervising them
worker-start --spec ... --agent <agent>on the Run it works for. The helper's task appears in that Run, under the worker's task, and records which worker started it.worker_done, heartbeats, escalations, status messages and questions (ask) go to the worker that started it, and the worker reads them with its normalcheck. It answers a helper's question withreply. The Run's coordinator does not get this traffic.check, andrun-currentstill shows no Run for it.worker_done, escalations and status messages) and any question it never answered move to the Run's coordinator, which is woken for them. Heartbeats stay behind: liveness is already recorded on the helper's task, so an old heartbeat is not work for the coordinator. Helpers that keep running after that report to the coordinator.worker-startandreplyfor helpers instead ofrun-create. Remote workers (started with--on) keep the oldrun-createinstructions; see Notes.Mechanism
dispatchReportMailbox: a task's reports go to the task that started it while that one is still active in the same Run, otherwise to the Run. It reads the existingcreator_dispatch_idon each task attempt and is evaluated on every send, so nothing is stored and nothing has to be cleaned up. Used bysend(report types only),ask, the unexpected-exit escalation and the setup-finished notice.worker-startauthority (resolveWorkerStartRun): the Run's coordinator, as before (it takes precedence), or a worker on an active task, resolved by the same lookup that stamps the helper's creator, so who may start a helper and where it reports always agree. The creator is checked again inside the transaction that reserves the helper, so a worker that finished in between creates nothing.--terminaland--onare refused for workers, so a worker can't take over another agent's pane or start a helper that can't report back);--taskonly for a task whose latest attempt was the worker's own helper (a retry), never the coordinator's tasks. The existing depth limit applies unchanged.replyfrom the worker whose task a question was addressed to answers it from that task, checked against the task's current attempt inside the answer transaction. The Run's coordinator can still answer any question in its Run.closeQuestionsForDispatch; it now first moves the settled task's unread helper reports and unanswered helper questions to the Run's mailbox.Why
creator_dispatch_id, task status), so there is no new saved field and nothing that can strand.worker-list. Orca's Run is the unit a coordinator supervises.--on) keep the old own-Run path, because their reports travel through the Run's home host to the Run mailbox. Follow-up: STA-10701.run-create(old prompts and skills do it), and that still moves its inbox. Retiring it is a product call, listed in STA-10701.Linked Issue
Linear: STA-10701
Related, not closed by this PR: #29008 and #29006 (open drafts in the same area; this PR is based on
mainand does not depend on them). #29006 also changes mail delivery inmessage-inbox.ts,mailbox-pointer-stage.tsandstructured-mailbox-pointer-delivery.ts, which this PR touches too (it scopes the "delivered" mark to the mailbox that received the mail), so whichever lands second needs a real merge of those files and a fresh CI run, #28922, #24974, #16784.Current-head validation (2026-10-22)
Reviewed head:
f2b0fc8c6a91e0d6d83c7dc3381ca4677e42bb6c, after merging main and updating formatting. Independent checks of worker authority, dispatch identity/depth, mailbox delivery, pending questions, settlement recovery and replay found no proven semantic conflict. The import conflict keeps the helper-authority path and main's relocated request schema. #29006 remains open and its delivery redesign is absent from this head.This is an orchestration/CLI PR: the review policy requires integration tests running the real CLI (
qa=tests). On Windows low spec, 43 actual source CLI commands ran against this head's production RPC server/runtime and on-disk database; two named suites passed all six tests. They exercised helper creation on the same Run, scope/depth refusals, reports and questions in the creator's ordinary inbox, replies, coordinator follow-ups, helper completion before and after creator settlement, mailbox replay, stop-window and rollback recovery, and database close/reopen recovery. Automated agent launch/readiness fixtures were used; this is not current-head rendered-app or real-provider QA. The earlier macOS real-agent screenshots below remain evidence for39ceaf3db5conly.The full PR workflow passed on the current head, including static analysis, typecheck, unit tests and cross-version wire compatibility. The supplemental WSL workflow also passed after retrying an Ubuntu distro-import setup timeout; no code change was needed. No new code change was needed. The dedicated performance pass covered all six review questions from source; large-backlog latency and live cross-platform resource use were not measured.
Visual Proof
Real runtime QA on macOS (hidden test copies of
origin/mainand this PR at39ceaf3db5c, same fixture repo with its setup script and default tabs, a real Grok agent as the worker and helper, throwaway homes; real hook config hashes unchanged).run-create)consumer_fenced, no helperworker_donecheckand is acknowledged therecheckNot exercised live (checked from source and covered by the flow tests): the three-level hand-back when a middle worker finishes with a helper's question unanswered, and the older own-Run launch path. The branch build showed a first-use "Search every agent session" introduction on its fresh profile; that comes from the main merge, not this PR.
Testing
I manually tested these changes locally
Automated tests added/updated, or explained why not below
New
structured-chat-worker-helpers.test.ts, one flow on the chat coordinator rig: a dispatched chat worker passesworker-startauthority on the Run it works for; its helper's escalation and question arrive in the worker's owncheck; the worker'sreplyunblocks the helper'sask; the coordinator's mailbox stays empty and the worker is still bound to no Run; after the worker acknowledges and finishes, a later helper escalation and the helper'sworker_donereach the coordinator. Fails with the send routing reverted.Updated the preamble pins for the new SUB-DISPATCH text.
Focused Node vitest on macOS: the orchestration messaging, worker, federation and DB suites plus the runtime mailbox specs, in two runs (79 files / 880 tests and 120 files / 952 tests), all passing. Typecheck and lint are left to CI.
AI Disclosure
Review
Agent skill upstream boundary
Notes
--onrefused for workers). Nothing here treats loss of contact as a finished task; the hand-back runs only when a task actually settles on its own host.worker-listand as subtasks of the worker's task. A coordinator's group sends (@all) reach helpers too, since they are live tasks of the Run. If a worker finishes before reading what its helpers sent, the coordinator receives those messages.Author: @BrennanKB5
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)