Skip to content

fix(orchestration): native chats get each orchestration message whole instead of a check pointer - #28567

Open
brennanb2025 wants to merge 50 commits into
mainfrom
brennanb2025/orchestration-mail-content
Open

brennanb2025 wants to merge 50 commits into
mainfrom
brennanb2025/orchestration-mail-content

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Worker reports previously reached native chats as a mailbox-check notice. This change puts each report into the chat's ordinary durable message queue, attributed to its sender, with a 4,000-character excerpt and orca orchestration inbox --id for the full text. Reports wait while the chat is working or waiting for a person's answer, then run one per full turn in order.

Stop keeps reports that were already waiting paused until the person sends or uses Resume. A provably new report after Stop can wake the chat while the person's paused messages stay paused. Reports whose saved order around Stop is unknown stay held. Existing queue Edit, Delete, Move, Send and Resume controls apply to report cards.

Durable queue creation marks the source report delivered. Inbox read/check acknowledgement remains separate, and later card edits, deletion or turn failure do not offer the source report again. This is rebuilt directly on main and has no dependency on the closed #32108.

Current head: 39a039ba3101d90a8aad764467a548a3287d6fe5. The normal commit hook passed, and targeted production-path and mounted-consumer tests passed. Independent source, reference and accounting qualification is in progress. Some ACP/runtime tests could not collect locally because fflate is missing; their execution must be confirmed in CI. Targeted real Claude and Codex runs have not started. This PR is not ready to merge.

The work is tracked in STA-8718. The single review status comment retains historical screenshots and will receive the current-head verification evidence.

…ing turn

A native-chat coordinator's mail notice always queued as a card until the whole turn ended, so a
coordinator busy in a long command never saw its workers' reports. Agents that take a message
mid-turn (Claude, Codex, Pi) now get the notice at once; agents that would cancel their turn for
it (ACP) still queue it.
@brennanb2025
brennanb2025 force-pushed the brennanb2025/orchestration-mail-steer branch 2 times, most recently from abbb2ea to 4f1d256 Compare October 14, 2026 05:23
… instead of a check pointer

Each mail message is sent to a native-chat agent as its own chat message,
with the sender, type, subject, body, payload and exact reply command, and
is marked read once the chat accepts it. Heartbeats are marked read, never
sent. Mail an unacknowledged check batch holds stays with that batch.
@brennanb2025
brennanb2025 force-pushed the brennanb2025/orchestration-mail-content branch from 3ff5b50 to c8ec903 Compare October 14, 2026 08:08
@brennanb2025

brennanb2025 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Worker reports now use the native chat's ordinary durable queue, with their own content and sender attribution. Busy chats and pending questions hold reports until the current turn completes; each report then gets its own turn. Stop keeps waiting reports paused, while provably fresh reports can wake the chat without releasing the person's paused messages. Unknown saved order stays held until Send or Resume.

Not ready to merge. Frozen head 39a039ba3101d90a8aad764467a548a3287d6fe5 is committed and non-force-pushed, and the PR now targets main with no dependency on the closed #32108. The normal hook passed and 49 post-repair tests passed; earlier overlapping test runs are not added to that count. Three fresh source/reference reviewers are running. Final accounting, CI execution of locally uncollected ACP/runtime targets, and targeted real Claude/Codex flows remain unqualified.

The current-head real runs will cover busy delivery, answered questions with ordered separate report turns, Stop and restart, fresh reports without personal-queue release, and existing Resume including a report-only backlog. No current-head QA pass or readiness marker is claimed.

The screenshots and outcomes below are historical evidence from the earlier implementation, including its recorded failure. They do not qualify the rebuilt delivery/Stop flows.

Earlier review and historical screenshots

Native chats currently receive a notice asking the agent to check its mailbox. This PR puts each worker report into the conversation as a separate attributed message, with a 4,000-character excerpt and an exact-message command for the full text. Real Grok QA found that sending reports into a running turn can cancel the person's answer and lose a report. The approved revision holds mail while any agent is working or waiting for the person, then starts one report turn at a time, oldest first. Terminal delivery stays unchanged.

Current review on fe6c675fbed87b2742bf6257627a02d141cf3fcd: blocked; not ready to merge. Two review rounds of the revised timing design are complete, after six source-review rounds of the earlier design. The independent test-only repair is committed and pushed; shared source repairs remain pending. Independent correctness, architecture and readiness/performance reviews and the three-seat architecture ruling are complete. The ruling keeps mailbox-owned pending delivery because other consumers can explicitly read and acknowledge the same mail; putting another executable copy into the chat's editable queue would change ownership and show reports before the required idle point.

The review found these remaining repairs and integration checks:

  • Pressing Stop can release a report that was already waiting, restart the agent and unpause the person's queued follow-up. Protecting that pause also requires a reliable release wake when the person later uses Resume. Following the existing Stop ruling, reports already held or owed at Stop remain stopped until the person sends or resumes; fresh reports follow ordinary idle-send behavior. Automatic mail must preserve the person’s own queue pause. This is settled, not a new product decision.
  • Busy worker history updates repeatedly rebuild the chat's history before discovering that new mail must wait. Remove that recurring work or establish the required operation-count performance contract while preserving settlement of an already-issued report.
  • The obsolete steering test is repaired: two reports remain invisible during personal work, then each gets its own completed attributed turn in order. The targeted test passes and the named coordinator file now has 23 passes and one failure. The unchanged failure is the ambiguous failed-start recovery interaction to reconcile with fix: recover orchestration mail after interrupted chats #32108; the working hold was preserved. Ordering across the chat's direct, Run and Dispatch mailboxes also needs the composed contract; sorting within one mailbox does not prove that result.

This round changed only the coordinator integration test (+84/−41); no production source changed. An independent Codex reviewer agreed with the whole-flow contract and its size. The normal commit hook passed, the non-force push is verified, and the worktree is clean. No main synchronization or QA launch occurred. Shared eligibility, consuming-turn recovery and outstanding-delivery repair remain with #32108, which is still open. Production work at that boundary waits for its handoff. The plan now records the repair contracts and an estimate of 100–150 additional production lines, with 120–180 changed test lines; the estimate will be checked against the landed interface before the shared repairs. The independent slice has updated the obsolete busy-turn test without changing recovery code.

Verification limits: the named real-host coordinator test used the scripted protocol fixture; its tested/reviewed patch exactly matches this commit. The file still has the known recovery failure. This is not real-agent QA. The older green full workflow and real-agent screenshots below are historical. Only the housekeeping workflow is registered on this head; full PR Checks, final mergeability/conflict verification, the final main synchronization, and real Codex/Grok/Claude QA on the resulting head remain required. No completion marker is claimed.

For Brennan's review: final-head main-versus-branch screenshots remain outstanding. The historical comparison and the Grok failure are preserved below. The three places where behavior depends on the admission and release contract are mail admission, idle release, and the person's Stop pause.


Historical QA only; these images do not prove the current timing revision:

QA on head 021e26d (M4Air, real signed-in Codex and Grok): FAIL, needs a fix before this can merge.

Flow Result Screenshot
Busy Codex chat gets the report as its own message PASS codex busy
Busy Grok chat gets the report as its own message PASS grok busy
Reports held while Grok's question waits PASS held
Reports after the question is answered FAIL after answer
Long report shortened; full text by message id; older-CLI guidance PASS long report
Clicking the sender opens the sender PASS sender
Main for comparison: old "check your mailbox" notice as expected main

The failure: Grok can't take a message while a turn runs, so Orca cancels the turn to send one. Held reports are released the moment the person answers, and each report is now its own send, so the second report cancels the first before Grok sees it. Grok's saved history has no trace of report A while Orca marks it read and delivered. On main the single notice gets both reports read. Separately, the first report also cancels the turn that carries the answer, so Grok re-asks its question; that also happens on main (main evidence). The fix is being decided; QA reruns after it. Final main sync, its CI and the completion marker also remain.

@brennanb2025
brennanb2025 marked this pull request as ready for review October 15, 2026 20:11
@brennanb2025
brennanb2025 changed the base branch from brennanb2025/orchestration-mail-steer to main October 16, 2026 21:45
@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
Walkthrough

The PR changes structured orchestration mail from pointer notices to individually formatted messages sent through the agent-session queue. It adds journal mail boundaries and source metadata to control eligibility across Stop, Resume, and queue operations. Mail creation receipts support recovery after restarts and uncertain outcomes. The inbox RPC and CLI gain exact message-ID lookup. Tests cover delivery, queue behavior, sender lookup, database query work, and runtime recovery.

Priority: Normal

Merge Risk: Moderate · up to 39a03

The change delivers orchestration reports as ordinary queued messages. Before merging, fix the failing CI gates and the broken status-sink test, and fix the delivery loop so older reports are not overtaken and a superseded report does not abort the batch. Also check that Stop works when the mail database has not been opened yet.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description explains the behavior and current validation status, but it does not follow the required template. It omits the required linked-issue format, visual-proof status, checklist sections, A… Update the description to include every required template section. Add an actual issue reference such as Fixes #<issue>, provide before-and-after visual proof or state N/A with a reason, complete the Testing checklist, add AI Disclosure…
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: native chats receive complete orchestration messages instead of check pointers.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the behavior and current validation status, but it does not follow the required template. It omits the required linked-issue format, visual-proof status, checklist sections, AI disclosure, review notes, and explicit testing checklist entries.

Resolution

Update the description to include every required template section. Add an actual issue reference such as Fixes #&lt;issue&gt;, provide before-and-after visual proof or state N/A with a reason, complete the Testing checklist, add AI Disclosure when applicable, complete Review, Agent skill upstream boundary, Notes, and Checklist sections, and retain the stated CI and readiness limitations.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…28540's hold

Each orchestration message still reaches a native chat as its own message, now through the
merged plain send for every agent and its admission hold: a message held for a waiting prompt
stops the pass so later mail cannot overtake it, and the mailbox is parked before the first
read. Heartbeats are skipped in this lane only and stay unread. A mailbox with an unacknowledged
check batch waits for its ack, as on main. Chat text is capped at 4,000 characters with the
command that shows the whole message.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments: 2
src/cli/specs/orchestration.ts: 1

133-134: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Restore --json in the inbox usage text.

GLOBAL_FLAGS is probably the source of --json, so the flag stays valid. The handler still reads json and prints JSON output. The usage string no longer lists it, and the other specs do. Users cannot find the flag in help output.

Proposed fix
-    usage: 'orca orchestration inbox [--id <id>] [--limit <n>] [--terminal <handle>] [--full]',
+    usage:
+      'orca orchestration inbox [--id <id>] [--limit <n>] [--terminal <handle>] [--full] [--json]',
src/cli/handlers/orchestration/message-inbox-handlers.ts: 1

44-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Filter messages without mutating the RPC response.

The code mutates result.result in place. The behavior is correct today. A copy is safer if the client later caches or shares response objects.

Proposed fix
-    if (id) {
-      result.result.messages = result.result.messages.filter((message) => message.id === id)
-      result.result.count = result.result.messages.length
-    }
-    printResult(result, json, (value) => {
+    const response = id
+      ? {
+          ...result,
+          result: {
+            ...result.result,
+            messages: result.result.messages.filter((message) => message.id === id),
+            count: result.result.messages.filter((message) => message.id === id).length
+          }
+        }
+      : result
+    printResult(response, json, (value) => {

ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1337d471-c548-46ac-b759-814998ea2690
📥 Commits

Reviewing files that changed from the base of the PR and between c8ec903 and f2cbf3a.

📒 Files selected for processing: 27
  • config/reliability-gates.jsonc
  • config/scripts/vitest-sqlite-database-runtime-files.mjs
  • src/cli/handlers/orchestration-legacy-read-only.test.ts
  • src/cli/handlers/orchestration/message-inbox-handlers.ts
  • src/cli/specs/orchestration.ts
  • src/main/acp/acp-structured-launch-resolution.test.ts
  • src/main/runtime/orca-runtime-tests/repository-project-operations-part-02.spec.ts
  • src/main/runtime/orchestration/agent-facing-parity.test.ts
  • src/main/runtime/orchestration/db/messages/mailbox-push-query.ts
  • src/main/runtime/orchestration/formatter.test.ts
  • src/main/runtime/orchestration/formatter.ts
  • src/main/runtime/orchestration/mailbox-pointer-eligibility.ts
  • src/main/runtime/orchestration/send-agent-turn.ts
  • src/main/runtime/orchestration/structured-mailbox-message-delivery.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-backlog.test.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-batch.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-host.test.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-host.ts
  • src/main/runtime/orchestration/types.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/check.test.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/message-methods.ts
  • src/main/runtime/structured-chat-coordinator-mail-rig.test-fixture.ts
  • src/main/runtime/structured-chat-coordinator-mail.test.ts
  • src/main/runtime/structured-chat-dispatch-worker-mail.test.ts
  • src/shared/rpc-contract/orchestration-params.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

Comment thread config/reliability-gates.jsonc Outdated
Comment thread src/main/runtime/structured-chat-coordinator-mail.test.ts Outdated
@brennanb2025
brennanb2025 marked this pull request as draft October 17, 2026 01:42
@brennanb2025
brennanb2025 marked this pull request as ready for review October 17, 2026 01:42
A mail send admitted into a running turn cancels that turn for agents that
cannot take a message mid-turn (ACP: Grok, OpenCode, OMP, Antigravity): the
second of two reports cancelled the first before the agent saw it, and the
first report cut off the turn carrying the person's answer. Mail now waits
while the chat is working or a prompt waits on the person, for every agent,
and goes out at the idle edge, one message per turn, oldest first.
@brennanb2025
brennanb2025 changed the base branch from main to brennanb2025/fix-stuck-codex-review October 25, 2026 16:26
@brennanb2025
brennanb2025 changed the base branch from brennanb2025/fix-stuck-codex-review to main October 28, 2026 03:40

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 75560694-81a8-431d-9d47-062624e79233
Commits

Reviewing files that changed from the base of the PR and between 5c181f8 and 39a039b.

Files selected for processing: 122
  • config/reliability-gates.jsonc
  • mobile/src/session/use-mobile-structured-agent-session-queued.test.tsx
  • src/cli/handlers/orchestration/message-inbox-handlers.ts
  • src/main/acp/acp-mail-performance.test-support.ts
  • src/main/acp/acp-structured-mail-admission.test.ts
  • src/main/acp/acp-structured-steer.test.ts
  • src/main/agent-hooks/server-agent-presence.test.ts
  • src/main/native-chat/agent-session-journal/command-receipt-schema.spec.ts
  • src/main/native-chat/agent-session-journal/journal-conversation-history.ts
  • src/main/native-chat/agent-session-journal/journal-database-schema.ts
  • src/main/native-chat/agent-session-journal/journal-database.spec.ts
  • src/main/native-chat/agent-session-journal/journal-epoch-controller.ts
  • src/main/native-chat/agent-session-journal/journal-epoch-replacement.spec.ts
  • src/main/native-chat/agent-session-journal/journal-epoch-replacement.ts
  • src/main/native-chat/agent-session-journal/journal-epoch-rollover.ts
  • src/main/native-chat/agent-session-journal/journal-mail-boundary.ts
  • src/main/native-chat/agent-session-journal/journal-mail-holds.test.ts
  • src/main/native-chat/agent-session-journal/journal-mail-holds.ts
  • src/main/native-chat/agent-session-journal/journal-queue-events.ts
  • src/main/native-chat/agent-session-journal/journal-queue-fold.ts
  • src/main/native-chat/agent-session-journal/journal-queue-retention.ts
  • src/main/native-chat/agent-session-journal/journal-queue-selection.ts
  • src/main/native-chat/agent-session-journal/journal-queued-messages.ts
  • src/main/native-chat/agent-session-journal/journal-reducer.ts
  • src/main/native-chat/agent-session-journal/journal-row-schema.ts
  • src/main/native-chat/agent-session-journal/journal-row-unusable-annotations.ts
  • src/main/native-chat/agent-session-journal/journal-row-writer.ts
  • src/main/native-chat/agent-session-journal/journal-stop-acceptance.ts
  • src/main/native-chat/agent-session-journal/journal-stop-and-resume-rows.ts
  • src/main/native-chat/agent-session-journal/journal-store-collaborators.ts
  • src/main/native-chat/agent-session-journal/journal-store-contracts.ts
  • src/main/native-chat/agent-session-journal/journal-store.ts
  • src/main/native-chat/agent-session-journal/journal-submission-fold.ts
  • src/main/native-chat/agent-session-journal/journal-submission-writer.ts
  • src/main/native-chat/agent-session-journal/queued-message-edit-leases.ts
  • src/main/native-chat/agent-session-journal/queued-message-pause.ts
  • src/main/native-chat/agent-session-journal/queued-message-table.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-adopted-import.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-adopted-import.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-chat-stop-target.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-chat-stop.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-client-delivery.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-conversation-open.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-create-adoption.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-current-work.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-host-mutations.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-host-types.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-mail-queue.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-mutation-plans.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-action-receipts.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-edit.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-messages.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-messages.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-mutations.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-pause-lift.test.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-pause.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-publication.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-queued-send.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-stop-acceptance.ts
  • src/main/native-chat/agent-session-wire/structured-agent-session-turns.ts
  • src/main/runtime/agent-session-founding-admission.ts
  • src/main/runtime/agent-session-record-store.test.ts
  • src/main/runtime/agent-session/ownership/agent-session-reservation-admission.ts
  • src/main/runtime/orca-runtime-structured-status-sink-wiring.test.ts
  • src/main/runtime/orchestration/agent-facing-parity.test.ts
  • src/main/runtime/orchestration/db/federation/federation-relay-import.ts
  • src/main/runtime/orchestration/db/messages/mail-source-order.ts
  • src/main/runtime/orchestration/db/messages/mailbox-routing-page.ts
  • src/main/runtime/orchestration/db/messages/message-inbox.ts
  • src/main/runtime/orchestration/db/messages/message-insert.ts
  • src/main/runtime/orchestration/db/orchestration-db.ts
  • src/main/runtime/orchestration/db/reset/orchestration-reset.ts
  • src/main/runtime/orchestration/federation-control-message.ts
  • src/main/runtime/orchestration/formatter.ts
  • src/main/runtime/orchestration/mail-handoff.test-fixture.ts
  • src/main/runtime/orchestration/mail-sql-work.test-support.ts
  • src/main/runtime/orchestration/mailbox-pointer-eligibility.ts
  • src/main/runtime/orchestration/mailbox-pointer-release-query.test.ts
  • src/main/runtime/orchestration/mailbox-pointer-submit.test.ts
  • src/main/runtime/orchestration/send-agent-turn.ts
  • src/main/runtime/orchestration/session-mail-stop-inventory.test.ts
  • src/main/runtime/orchestration/session-mail-stop-inventory.ts
  • src/main/runtime/orchestration/structured-mail-source.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-host.test.ts
  • src/main/runtime/orchestration/structured-mailbox-pointer-host.ts
  • src/main/runtime/orchestration/structured-pointer-operation-id.test.ts
  • src/main/runtime/orchestration/structured-pointer-operation-id.ts
  • src/main/runtime/orchestration/structured-session-mail-target.ts
  • src/main/runtime/orchestration/terminal-pane-mail-routing.ts
  • src/main/runtime/orchestration/types.ts
  • src/main/runtime/rpc/methods/orchestration-structured-worker-session.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/inbox-exact.test.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/inbox-method.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/mailbox-message-receipt.ts
  • src/main/runtime/rpc/methods/orchestration/messaging/message-methods.ts
  • src/main/runtime/runtime-message-waiters.ts
  • src/main/runtime/runtime-orchestration-sender-names.test.ts
  • src/main/runtime/structured-agent-session-host-assembly.ts
  • src/main/runtime/structured-agent-session-runtime.ts
  • src/main/runtime/structured-chat-coordinator-fake-codex-fixture.ts
  • src/main/runtime/structured-chat-coordinator-mail-rig.test-fixture.ts
  • src/main/runtime/structured-chat-coordinator-mail.test.ts
  • src/main/runtime/structured-chat-coordinator-observation-clock.test-fixture.ts
  • src/main/runtime/structured-chat-dispatch-worker-mail.test.ts
  • src/main/runtime/structured-chat-dispatch-worker.test.ts
  • src/main/runtime/structured-chat-mail-person-fixture.ts
  • src/main/runtime/structured-chat-mail-recovery.test.ts
  • src/main/runtime/structured-chat-mail-stop.test.ts
  • src/main/runtime/workspaces/persisted-surfaces/automation-operations.ts
  • src/main/runtime/workspaces/workspace-process-listing.ts
  • src/renderer/src/components/native-chat/use-structured-agent-session-queued-messages.resume.test.tsx
  • src/renderer/src/components/native-chat/use-structured-agent-session-read.test.tsx
  • src/renderer/src/lib/open-agent-message-sender.ts
  • src/shared/agent-session-conversation-view.ts
  • src/shared/agent-session-record.ts
  • src/shared/agent-session/agent-session-message-source.test.ts
  • src/shared/agent-session/agent-session-message-source.ts
  • src/shared/rpc-contract/orchestration-params.ts
  • src/shared/structured-agent-session-queued-cards.ts
  • src/shared/wsl-move/wsl-move-storage-dispositions.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Comment on lines +2091 to +2092
"reconciles a crashed immutable cohort read-only (%s) before delivering an independent survivor",
"does not turn expiry of an uncertain original operation into proof it never ran"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Supported by static analysis

Script executed:

sed -n '2070,2140p' config/reliability-gates.jsonc
rg -n 'crashed immutable cohort|expiry of an uncertain original' src config | head

Repository: stablyai/orca

Length of output: 8238


Script executed:

# Locate the checker and every place it reads the gate "assertions" field.
rg --files -g 'check-reliability-gates.mjs' || test "$?" -eq 1
rg -n -F -e 'assertions' -e 'structured-send-at-most-once' -- scripts config/scripts 2>/dev/null || test "$?" -eq 1
for f in $(rg --files -g 'check-reliability-gates.mjs' 2>/dev/null); do
  echo "== $f"
  rg -n -e 'assertion' -e 'readFileSync' -e 'includes\(' -e 'test\(' -- "$f" || test "$?" -eq 1
done

Repository: stablyai/orca

Length of output: 9226


Replace the stale assertion references in the reliability gate.

agent-session.structured-send-at-most-once maps two assertion names to src/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.ts. Neither name appears under src/, so the mapped file does not contain them. The manifest therefore records recovery coverage that the referenced file does not provide. Replace these references with the current test that covers this behavior, or record the coverage gap. The manifest checker validates that each reference names a listed test file and has a non-empty assertion array. It does not check that the assertion names exist in that file, so this mismatch passes validation.

Comment on lines +102 to +103
startedAt: Date.now(),
completedAt: Date.now(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Maintainability & Code Quality · Quick win

Replace Date.now() with fixed timestamps to pass the flaky-test gate.

The test-no-wall-clock ast-grep gate reports 4 findings in this file, and its baseline allows 0. The 4 findings are the Date.now() calls on Line 102, Line 103, Line 157 and Line 210. The static analysis and typecheck job fails until they are removed. These turn items only need stable, ordered timestamps, so fixed values are enough.

Proposed fix
+const TURN_AT = 1_800_000_000_000
@@
-      startedAt: Date.now(),
-      completedAt: Date.now(),
+      startedAt: TURN_AT,
+      completedAt: TURN_AT + 1,
@@
-      startedAt: Date.now(),
+      startedAt: TURN_AT,
@@
-      startedAt: Date.now(),
+      startedAt: TURN_AT,

Also applies to: 157-157, 210-210

Source: Pipeline failures

@@ -1,4 +1,6 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import type { AgentSessionRecord } from '../../shared/agent-session-record'
import type { MailLineageScope } from '../native-chat/agent-session-journal/journal-mail-holds'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Supported by static analysis

Script executed:

rg -n 'MailLineageScope' src | head
sed -n '1,100p' src/main/runtime/orca-runtime-structured-status-sink-wiring.test.ts
sed -n '1,40p' src/main/native-chat/agent-session-journal/journal-mail-holds.ts
sed -n '265,295p' src/main/runtime/workspaces/workspace-process-listing.ts

Repository: stablyai/orca

Length of output: 6342


Script executed:

rg -n -F -- 'captureSessionMail' src/main/runtime/orca-runtime.ts src/main/runtime/workspaces/workspace-process-listing.ts src/main/native-chat/agent-session-journal/journal-mail-holds.ts src/main/runtime/orca-runtime-structured-status-sink-wiring.test.ts
sed -n '45,75p' src/main/runtime/orca-runtime-structured-status-sink-wiring.test.ts
sed -n '270,290p' src/main/runtime/workspaces/workspace-process-listing.ts

Repository: stablyai/orca

Length of output: 3014


Update this test to the zero-argument captureSessionMail contract.

MailLineageScope is not exported, so the test fails typecheck. The installed callback ignores all supplied arguments and calls this.captureSessionMail() without arguments, so the current forwarding assertions fail when the test runs.

Proposed fix
-import type { AgentSessionRecord } from '../../shared/agent-session-record'
-import type { MailLineageScope } from '../native-chat/agent-session-journal/journal-mail-holds'
@@
-    const records: AgentSessionRecord[] = []
-    const scope: MailLineageScope = { root: 'session', aliases: ['session'] }
-    const capture = vi.spyOn(runtime, 'captureSessionMail').mockImplementation(function (
-      this: OrcaRuntimeService,
-      sessionId,
-      actualRecords,
-      actualScope
-    ) {
+    const capture = vi.spyOn(runtime, 'captureSessionMail').mockImplementation(function (
+      this: OrcaRuntimeService
+    ) {
       expect(this).toBe(runtime)
-      expect(sessionId).toBe('session')
-      expect(actualRecords).toBe(records)
-      expect(actualScope).toBe(capture.mock.calls.length === 1 ? scope : null)
       return {}
     })
@@
-    installedCapture('session', records, scope)
-    installedCapture('session', records, null)
-    expect(capture).toHaveBeenCalledTimes(2)
+    expect(installedCapture()).toEqual({})
+    expect(capture).toHaveBeenCalledOnce()

Comment on lines +260 to +278
beforeRun: () => {
const latest = db.getMessageById(message.id)
const held = db.getOutstandingMailboxDelivery(mailbox)
if (
!current() ||
!latest ||
latest.read !== 0 ||
latest.delivered_at !== null ||
latest.to_handle !== mailbox ||
held?.messageIds.includes(message.id) ||
messageTypeHasOrchestrationWaiter(
this.deps.getMessageWaiters(mailbox),
latest.type
) ||
interest.reservedTypes?.has(latest.type)
) {
throw new Error('source creation superseded')
}
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

Supported by static analysis

Script executed:

sed -n '169,300p' src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts
sed -n '45,85p' src/main/runtime/orchestration/structured-mailbox-pointer-host.ts
rg -n 'beforeRun|beforeInsert' src/main/native-chat src/main/runtime/orchestration --glob '!*.test.ts'

Repository: stablyai/orca

Length of output: 7313


Script executed:

set -o pipefail
printf '%s\n' '--- relevant declarations and callers ---'
rg -n -F --glob '*.ts' -e 'function sendAgentTurn' -e 'const sendAgentTurn' -e 'maybeQueueStructuredAgentSessionSend' -e 'deliverForHandle' -e 'ordinary mail creation failed' -e 'delete.*recovery' -e 'recovery' src/main/runtime/orchestration src/main/native-chat
printf '%s\n' '--- send-agent-turn ---'
sed -n '1,220p' src/main/runtime/orchestration/send-agent-turn.ts
printf '%s\n' '--- queued message path ---'
sed -n '80,210p' src/main/native-chat/agent-session-journal/journal-queued-messages.ts
sed -n '90,210p' src/main/native-chat/agent-session-wire/structured-agent-session-queued-messages.ts
printf '%s\n' '--- delivery owner ---'
sed -n '1,180p' src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts

Repository: stablyai/orca

Length of output: 42765


Script executed:

printf '%s\n' '--- delivery control flow ---'
nl -ba src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts | sed -n '55,115p;235,315p'
printf '%s\n' '--- sendStructuredSessionTurn ---'
nl -ba src/main/runtime/orchestration/send-agent-turn.ts | sed -n '125,185p'
printf '%s\n' '--- queue admission ---'
nl -ba src/main/native-chat/agent-session-wire/structured-agent-session-queued-messages.ts | sed -n '100,180p'
printf '%s\n' '--- journal insertion ---'
nl -ba src/main/native-chat/agent-session-journal/journal-queued-messages.ts | sed -n '90,145p'

Repository: stablyai/orca

Length of output: 12987


Handle source creation superseded per message.

beforeRun can throw after a batch selects a message. The exception propagates through host.send, rejects deliver, and skips the remaining messages in the current batch. The outer handler logs ordinary mail creation failed and clears the mailbox recovery marker. The skipped messages can remain undelivered until a later trigger or coalesced retry.

Catch a dedicated error for message-level supersession around host.send, then continue the loop. Keep a separate cancellation path for !current(): do not continue with a stale target. Re-throw all other errors.

Comment on lines +289 to +294
if (
outcome.kind === 'queued' ||
this.deps.host.readCreation(operationId, target.sessionId) === 'accepted'
) {
db.markAsDelivered([message.id], mailbox)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Supported by static analysis

Script executed:

#!/bin/bash
rg -nP -C4 'nextSendableQueuedCard|mailSource\??\.sequence|ORDER BY' src/main/native-chat/agent-session-journal/journal-queue-selection.ts src/main/native-chat/agent-session-journal/journal-queued-messages.ts src/main/native-chat/agent-session-journal/queued-message-table.ts src/main/native-chat/agent-session-journal/journal-queue-fold.ts

Repository: stablyai/orca

Length of output: 2143


Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- delivery references and loop ---'
rg -n -F -- 'host.send' src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts
nl -ba src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts | sed -n '240,315p'
printf '%s\n' '--- queue selection implementation ---'
rg -n -F -- 'function nextSendableQueuedCard' src/main/native-chat/agent-session-journal
rg -n -F -- 'nextSendableQueuedCard' src/main/native-chat/agent-session-journal/queued-message-pause.ts
nl -ba src/main/native-chat/agent-session-journal/queued-message-pause.ts | sed -n '1,220p'
printf '%s\n' '--- queued message listing and ordering ---'
rg -n -F -e 'list()' -e 'sort(' -e 'mailSource' src/main/native-chat/agent-session-journal/journal-queued-messages.ts src/main/native-chat/agent-session-journal/queued-message-table.ts src/main/native-chat/agent-session-journal/journal-queue-fold.ts
nl -ba src/main/native-chat/agent-session-journal/journal-queued-messages.ts | sed -n '1,180p'
nl -ba src/main/native-chat/agent-session-journal/queued-message-table.ts | sed -n '1,240p'
printf '%s\n' '--- capacity test references ---'
rg -n -F -C8 -- 'queuedMessagesPublishedBytesRefusal' src/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.ts src/main/runtime/orchestration

Repository: stablyai/orca

Length of output: 24374


Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- host send contract and refusal ---'
rg -n -F -e 'send(input' -e 'queuedMessagesPublishedBytesRefusal' -e 'publishedBytes' -e "kind: 'held'" src/main/runtime/orchestration/structured-mailbox-pointer-host.ts src/main/runtime/orchestration
nl -ba src/main/runtime/orchestration/structured-mailbox-pointer-host.ts | sed -n '130,280p'
printf '%s\n' '--- queue selection order ---'
nl -ba src/main/native-chat/agent-session-journal/queued-message-pause.ts | sed -n '216,310p'
nl -ba src/main/native-chat/agent-session-journal/journal-queue-selection.ts | sed -n '1,240p'
printf '%s\n' '--- delivery tests ---'
rg -n -i -F -e 'capacity' -e 'published' -e 'refusal' -e 'message A' -e 'message B' src/main/runtime/orchestration/*structured-mailbox-pointer-delivery*.test.ts src/main/runtime/orchestration/*mailbox*.test.ts || test "$?" -eq 1

Repository: stablyai/orca

Length of output: 8201


Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- host send block ---'
nl -ba src/main/runtime/orchestration/structured-mailbox-pointer-host.ts | sed -n '1,95p'
printf '%s\n' '--- capacity test ---'
nl -ba src/main/runtime/orchestration/structured-mailbox-pointer-delivery.test.ts | sed -n '1,110p'

Repository: stablyai/orca

Length of output: 9008


Stop delivery after the first declined report.

JournalQueuedMessages appends accepted cards in position order. If an older report returns held, the loop currently attempts newer reports. A newer report can then enter the queue before the older report, which breaks oldest-first delivery.

🐛 Suggested fix
--- "a/src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts"
+++ "b/src/main/runtime/orchestration/structured-mailbox-pointer-delivery.ts"
@@ -286,13 +286,16 @@
         if (!current()) {
           return
         }
         if (
           outcome.kind === 'queued' ||
           this.deps.host.readCreation(operationId, target.sessionId) === 'accepted'
         ) {
           db.markAsDelivered([message.id], mailbox)
+          continue
         }
+        // Preserve oldest-first: a newer report must not overtake a declined one.
+        return
       }
     }
   }
 }

Update the capacity test so that it expects delivery to stop after the first held result.

Comment on lines +65 to +96
it.each(['exit-then-throw', 'throw-then-exit', 'before-echo'] as const)(
'never reoffers chat-owned reports after provider failure, restart or repeated source edges (%s)',
async (fault) => {
observationClock.start()
// A death-status edge formerly re-pointed unread mail and restarted the provider every second.
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
providerFaults.dieBeforeEveryEcho = true
const before = codex.connections.length
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
// A fixed window, not a poll: a respawn loop would restart it several times in it.
await observationClock.observe(1_500)
await edgesAnswered()
expect(codex.connections.length - before).toBe(0)
expect(providerFaults.turnStarts).toBe(1)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)
})

it.each(['exit-then-throw', 'throw-then-exit'] as const)(
'does not restart a provider that crashed while taking the pointer turn (%s)',
async (crash) => {
observationClock.start()
// A failed send records unknown with the connection error; status edges must not re-point it into a respawn loop.
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
providerFaults.crashOnTurnStart = crash
const before = providerFaults.starts
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
await observationClock.observe(1_500)
await edgesAnswered()
expect(providerFaults.starts - before).toBe(0)
expect(providerFaults.turnStarts).toBe(1)
expect(await pointerSends()).toHaveLength(1)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)
if (fault === 'before-echo') {
providerFaults.dieBeforeEveryEcho = true
} else {
providerFaults.crashOnTurnStart = fault
}
)

it('points the next result once after a transient death, then holds nothing', async () => {
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
const second = await call(
'orchestration.taskCreate',
{ spec: 'more' },
{ sessionId: COORDINATOR }
)
providerFaults.dieBeforeEveryEcho = true
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
await edgesAnswered()
providerFaults.dieBeforeEveryEcho = false
const before = providerFaults.starts
await finishWorker(idOf(second.task), { handle: 'term_worker_2', paneKey: WORKER_2_PANE })
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(2), WAIT)
const revived = connectionFor(COORDINATOR)
expect(turnText(revived.turns.at(-1)!)).toBe(
formatMessagePointer(2, `run:${runId}`, localOrchestrationCliCommand()).trim()
)
await settleTurn(COORDINATOR, revived.turns.length - 1)
await vi.waitFor(
() => expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toEqual([]),
WAIT
)
expect(providerFaults.starts - before).toBe(1)
expect(providerFaults.turnStarts).toBe(2)
})

it('leaves a pointer the person stopped while its agent was starting stopped', async () => {
observationClock.start()
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
await host.close(COORDINATOR, 'evict')
providerFaults.startDelayMs = 400
const before = providerFaults.starts
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.starts).toBe(before + 1), WAIT)
const cancelling = host.cancel(
{ callerKey: 'test-surface' },
{
envelope: {
sessionId: COORDINATOR,
clientOperationId: operationId(),
expectedRuntimeFence: host.deps.store.getRecord(COORDINATOR)!.lease.runtimeFence,
payloadFingerprint: computeAgentSessionPayloadFingerprint({
method: 'agentSession.cancel',
sessionId: COORDINATOR,
fields: { turnId: 'turn-x' }
})
},
turnId: 'turn-x'
}
)
await observationClock.observe(400)
const cancelled = await cancelling
expect(cancelled).toMatchObject({ ok: true })
providerFaults.startDelayMs = 0
// A fixed window, not a poll: a re-point would start the agent again in it.
await observationClock.observe(1_500)
expect(providerFaults.starts - before).toBe(1)
expect(await pointerSends()).toHaveLength(1)
await edgesAnswered()
expect(providerFaults.starts - before).toBe(1)
expect(providerFaults.turnStarts).toBe(0)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)
})

it('points a held pointer once more after Orca restarts, under a new id', async () => {
observationClock.start()
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
providerFaults.dieBeforeEveryEcho = true
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
await edgesAnswered()
const [held] = await pointerSends()

// A fresh runtime redrives restored mail once; the still-failing provider must not enter a restart loop.
const source = db.getUnreadMessages(`run:${runId}`)[0]
expect(source.delivered_at).not.toBeNull()
const id = ordinaryMailCreationId(source)
expect(host.deps.store.readCommandReceipt({ kind: 'global' }, id)).toMatchObject({
verdict: 'readable',
receipt: { status: 'accepted', result: { kind: 'queued-draft', messageId: id } }
})
restartRuntime()
const before = providerFaults.starts
await observationClock.observe(2_000)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(2), WAIT)
await observationClock.observe(1_500)
await edgesAnswered()
expect(providerFaults.starts - before).toBe(1)
expect(providerFaults.turnStarts).toBe(2)
const sends = await pointerSends()
expect(sends).toHaveLength(2)
expect(sends[0]).toBe(held)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)
})

/** Mail for a chat whose agent is stopped and whose every start is refused; resolves after it. */
async function refusedStartsFor(
refusal: () => Error
): Promise<{ runId: string; starts: number }> {
observationClock.start()
await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
await host.close(COORDINATOR, 'evict')
providerFaults.refuseStart = refusal
const before = providerFaults.starts
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.starts).toBe(before + 1), WAIT)
await vi.waitFor(
() => expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1),
WAIT
)
// A fixed window, not a poll: a retry loop would start it several times in it.
await observationClock.observe(1_500)
return { runId, starts: providerFaults.starts - before }
}

it.each([
[
'one the person must fix',
() => new AgentSessionAcquisitionRefusal('no login', 'notSignedIn')
],
[
'an account switch in progress',
() =>
new AgentSessionPreSpawnError(new Error('switching accounts'), {
reason: 'accountSwitchInProgress'
})
]
])(
'holds mail after a start refused as %s until a turn runs, adding nothing on later edges',
async (_label, refusal) => {
const { runId, starts } = await refusedStartsFor(refusal)
expect(starts).toBe(1)
const before = providerFaults.starts
for (let edge = 0; edge < 5; edge += 1) {
runtime.onStructuredSessionStatusForMail({ sessionId: COORDINATOR, status: 'idle' })
await observationClock.observe(100)
}
await edgesAnswered()
expect(providerFaults.starts).toBe(before)
expect(await pointerSends()).toHaveLength(1)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)

providerFaults.refuseStart = null
expect(await sendUserMessage(COORDINATOR, 'again')).toMatchObject({ ok: true })
await vi.waitFor(() => expect(providerFaults.starts).toBe(before + 1), WAIT)
const revived = connectionFor(COORDINATOR)
await vi.waitFor(() => expect(revived.turns.length).toBeGreaterThanOrEqual(1), WAIT)
expect(revived.turns[0]!.text).toContain('again')
await settleTurn(COORDINATOR, 0)
await vi.waitFor(() => expect(revived.turns).toHaveLength(2), WAIT)
expect(revived.turns[1]!.text).toMatch(POINTER)
}
)

it('points held mail after the next turn that runs, even once a rewind dropped its send', async () => {
const chat = await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
providerFaults.crashOnTurnStart = 'exit-then-throw'
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
await edgesAnswered()
// What a rewind's recovery does: the journal is rebuilt, and no send is on record any more.
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the host keeps its conversations private; this is the one journal call rewind recovery makes.
const open = (host as unknown as { sessions: Map<string, { journal: AgentSessionJournal }> })
.sessions
const fence = host.deps.store.getRecord(COORDINATOR)!.lease.runtimeFence
await open.get(COORDINATOR)!.journal.replaceEpochItems('handle_forked', fence, [])
expect(await pointerSends()).toEqual([])

providerFaults.crashOnTurnStart = 'off'
expect(await sendUserMessage(COORDINATOR, 'again')).toMatchObject({ ok: true })
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(2), WAIT)
const revived = connectionFor(COORDINATOR)
expect(revived).not.toBe(chat)
await settleTurn(COORDINATOR, revived.turns.length - 1)
await vi.waitFor(() => expect(turnText(revived.turns.at(-1)!)).toMatch(POINTER), WAIT)
await settleTurn(COORDINATOR, revived.turns.length - 1)
await vi.waitFor(
() => expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toEqual([]),
WAIT
)
})

it('points again under a new id once a send the host never recorded is too old to admit', async () => {
const chat = await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
// The first pointer is refused before the host records it, so the journal holds no verdict.
const realSend = host.send
const refused = vi.spyOn(host, 'send').mockImplementationOnce(async () => ({
ok: false as const,
refusal: refuse(
'agent_session_operation_invalid',
{ reason: 'conversationCommandInFlight' },
'busy'
)
}))
refused.mockImplementation((caller, params) => realSend(caller, params))
await finishWorker(taskId)
await vi.waitFor(() => expect(refused).toHaveBeenCalledTimes(1), WAIT)
const held = db.getStructuredPointerOperation(`run:${runId}`)
expect(held).toMatchObject({ session_id: COORDINATOR, operation_id: expect.any(String) })
const heldId = held!.operation_id
expect(heldId).not.toBe('')
expect(refused.mock.calls[0]?.[1].envelope.clientOperationId).toBe(heldId)
expect(await pointerSends()).toEqual([])

// No edge for a day: the host would now refuse that id as expired, on every retry.
vi.useFakeTimers({ toFake: ['Date'] })
try {
vi.setSystemTime(Date.now() + AGENT_SESSION_MAX_NEW_OPERATION_AGE_MS + 60_000)
for (let n = 0; n < 3; n++) {
await edgesAnswered()
await vi.waitFor(
() => expect(chat.turns.map(turnText)).toEqual([ptyPointer(`run:${runId}`)]),
WAIT
)
await vi.waitFor(async () => {
const sends = await pointerSends()
expect(sends).toHaveLength(1)
expect(sends[0]).toBeTruthy()
expect(sends[0]).not.toBe(heldId)
expect(db.getStructuredPointerOperation(`run:${runId}`)).toMatchObject({
session_id: COORDINATOR,
operation_id: sends[0]
})
}, WAIT)
expect(db.getUndeliveredUnreadMessages(`run:${runId}`, undefined, {})).toHaveLength(1)
} finally {
vi.useRealTimers()
}
})

it('points mail a refused turn left in doubt again with the next result, counting both', async () => {
// Failed turn/start remains unknown on replay; a new result points to both messages without waiting behind that doubt.
const chat = await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
const second = await call(
'orchestration.taskCreate',
{ spec: 'more' },
{ sessionId: COORDINATOR }
)
providerFaults.refuseTurnStarts = 1
const before = codex.connections.length
await finishWorker(taskId)
await vi.waitFor(() => expect(providerFaults.turnStarts).toBe(1), WAIT)
await edgesAnswered()
expect(chat.turns).toHaveLength(0)

await finishWorker(idOf(second.task), { handle: 'term_worker_2', paneKey: WORKER_2_PANE })
await vi.waitFor(() => expect(chat.turns).toHaveLength(1), WAIT)
expect(turnText(chat.turns[0]!)).toBe(
formatMessagePointer(2, `run:${runId}`, localOrchestrationCliCommand()).trim()
)
expect(await queuedCardTexts()).toEqual([])
expect(codex.connections.length).toBe(before)
})

it('points a coordinator whose agent is not running through the send alone, which starts it', async () => {
await openChat(COORDINATOR)
const { taskId } = await coordinatorRunAndTask()
await host.close(COORDINATOR, 'evict')
expect(host.hasSession(COORDINATOR)).toBe(false)
const before = codex.connections.length

await finishWorker(taskId)

// Nothing holds or wakes the session first: the pointer's accepted send starts its agent.
await vi.waitFor(() => expect(codex.connections.length).toBe(before + 1), WAIT)
const revived = connectionFor(COORDINATOR)
await vi.waitFor(() => expect(revived.turns).toHaveLength(1), WAIT)
expect(revived.turns[0]!.text).toMatch(POINTER)
await settleTurn(COORDINATOR, 0)
expect(await userTexts(COORDINATOR)).toEqual([expect.stringMatching(POINTER)])
})

it('steers mail that arrives mid-turn into the running turn, with who it is from', async () => {
const chat = await openChat(COORDINATOR)
const { runId, taskId } = await coordinatorRunAndTask()
expect(await sendUserMessage(COORDINATOR, 'go')).toMatchObject({ ok: true })
await vi.waitFor(() => expect(chat.turns).toHaveLength(1), WAIT)
const notify = (method: string, params: unknown) =>
chat.handlers.onNotification?.(method, params)
const echo = (id: string, clientId: string, text: string) =>
notify('item/completed', {
item: { type: 'userMessage', id, clientId, content: [{ type: 'text', text }] }
})
notify('turn/started', { turn: { id: 'turn-1' } })
echo('echo-go', chat.turns[0]!.clientUserMessageId, 'go')
await host.flushStreamedEvents(COORDINATOR)

await finishWorker(taskId)

// Into the turn that is running, as a terminal agent takes its typed nudge: no card, no
// second turn.
await vi.waitFor(() => expect(chat.steers).toHaveLength(1), WAIT)
const [steer] = chat.steers
expect(steer!.expectedTurnId).toBe('turn-1')
expect(steer!.text).toContain(ptyPointer(`run:${runId}`))
expect(await queuedCardTexts()).toEqual([])
const sent = (await host.journalSnapshot(COORDINATOR)).items.find(
(item) => item.body.kind === 'message' && item.body.from
)
expect(sent?.body).toMatchObject({
from: { kind: 'agent', orchestration: { message: 'mail-notice', mailbox: `run:${runId}` } }
await observationClock.observe(500)
expect(providerFaults.turnStarts).toBe(1)
expect(await call('orchestration.check', {}, { sessionId: COORDINATOR })).toMatchObject({
count: 1
})

echo('echo-pointer', steer!.clientUserMessageId, ptyPointer(`run:${runId}`))
notify('turn/completed', { turn: { id: 'turn-1' } })
await host.flushStreamedEvents(COORDINATOR)
runtime.onStructuredSessionStatusForMail({ sessionId: COORDINATOR, status: 'idle' })
await new Promise((resolve) => setTimeout(resolve, 200))
expect(chat.turns).toHaveLength(1)
expect(chat.steers).toHaveLength(1)
}
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Maintainability & Code Quality · Quick win

Prune the flaky-test ratchet baseline for the removed real-time wait.

This file no longer contains the real-time wait that the test-no-real-time-wait baseline still counts. CI now reports 0 findings against a baseline of 1. The ratchet allows the baseline only to shrink, and it treats a stale baseline as a regression. As a result, the ast-grep test step and ast-grep-gates.mjs fail in the static analysis job. Run pnpm check:flaky-test-pattern-ratchet --prune and commit the updated baseline file. No change to this test is needed.

Source: Pipeline failures

Comment on lines +170 to +184
captureSessionMail = () =>
captureSessionMailAtStop(this._orchestrationDb, (db) => {
if (this._orchestrationDb !== db) {
return
}
this.orchestrationStructuredMailboxPointerDelivery.resetForDatabaseChange()
this.orchestrationFederation.resetForDatabaseChange()
this.mailPointerRepointScheduler.clear()
this._orchestrationDb = null
try {
db.close()
} catch (error) {
console.warn('[mail] retiring source connection failed', error)
}
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Capture against the existing database when it has not been opened yet.

captureSessionMail passes this._orchestrationDb to the capture. That field stays null until something calls getOrchestrationDb(). After a restart, a person can press Stop before any mail code has opened the database. In that state, captureSessionMailAtStop returns {}, so the Stop writes a held boundary without a cut. journalMailSourceEligible then refuses every later report, including fresh local reports, until the person presses Resume. This contradicts the stated rule that fresh reports may run after Stop.

Use getExistingOrchestrationDb(). It opens the file only when the file exists, and it still returns null for a profile that has no mail.

Proposed fix
--- "a/src/main/runtime/workspaces/persisted-surfaces/automation-operations.ts"
+++ "b/src/main/runtime/workspaces/persisted-surfaces/automation-operations.ts"
@@ -167,8 +167,8 @@
       : null
   }
 
   captureSessionMail = () =>
-    captureSessionMailAtStop(this._orchestrationDb, (db) => {
+    captureSessionMailAtStop(this.getExistingOrchestrationDb(), (db) => {
       if (this._orchestrationDb !== db) {
         return
       }
Committable suggestion

IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
captureSessionMail = () =>
captureSessionMailAtStop(this._orchestrationDb, (db) => {
if (this._orchestrationDb !== db) {
return
}
this.orchestrationStructuredMailboxPointerDelivery.resetForDatabaseChange()
this.orchestrationFederation.resetForDatabaseChange()
this.mailPointerRepointScheduler.clear()
this._orchestrationDb = null
try {
db.close()
} catch (error) {
console.warn('[mail] retiring source connection failed', error)
}
})
captureSessionMail = () =>
captureSessionMailAtStop(this.getExistingOrchestrationDb(), (db) => {
if (this._orchestrationDb !== db) {
return
}
this.orchestrationStructuredMailboxPointerDelivery.resetForDatabaseChange()
this.orchestrationFederation.resetForDatabaseChange()
this.mailPointerRepointScheduler.clear()
this._orchestrationDb = null
try {
db.close()
} catch (error) {
console.warn('[mail] retiring source connection failed', error)
}
})

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant