fix(runtime): persist uncertainty for external tool effects

[AsperforMias]

Area

Runtime

Summary

Severity: P0 for write-capable tools. Mosoo cannot distinguish an external tool action that never started from one that succeeded after the Sandbox or Driver lost its control connection before the terminal receipt was persisted.

The Driver tracks command replay and terminal delivery only in process memory. MCP execution returns an in-memory result, but no durable effect intent, provider receipt, reconciliation record, compensation contract, or stable idempotency key is persisted or forwarded to the tool provider. On terminal Driver loss, the API marks accepted commands and the Run as failed. The recovery code only logs a future requeue decision.

A user resend or future automatic resume can therefore repeat an already successful external write; avoiding replay instead leaves an indeterminate operation incorrectly represented as failed. This affects MCP tools and any Agent tool that can deploy, mutate production data, send messages, or charge money.

Steps to reproduce

1. Check out main at b4a5f67.
2. In the Driver boundary fixture, provide an MCP executor whose tool body increments a local counter and then returns a successful result.
3. Let the Driver send the accepted command update, but make every terminal completed update throw to simulate a control socket loss after the local external effect succeeded.
4. Run the real DriverCommandDispatcher. It retries terminal delivery three times and exits without a persisted completion receipt; the counter is 1.
5. Start a fresh DriverCommandDispatcher with the same MCP command. Its in-memory command history is empty, so the tool body runs again and the counter becomes 2.
6. Separately run the real API finalization path against an accepted command. It terminalizes the command and Run as failed; it does not create an effect record, query a provider receipt, compensate, or enqueue a recovery run.

Expected behavior

Before a write-capable tool call, Mosoo should durably record an effect intent linked to the Run and command. The effect must have a stable idempotency key, lifecycle state including unknown, attempt history, and an optional external receipt.

For providers that support it, Mosoo must pass the stable idempotency key and persist the returned provider reference before treating the effect as succeeded. After Driver/Sandbox loss, recovery must reconcile unknown effects before any replay. Unsupported or non-idempotent tools must stay unknown and require explicit user or operator resolution.

Compensation should be an explicit per-tool capability, not an assumed rollback. A Saga-like compensating action may run only where the provider declares one.

Actual behavior

The DriverCommandDelivery dedupe map is process-local. A newly created Driver has no durable record of an already executed external call. The remote MCP executor invokes the tool directly and returns only output text, request id, server id, and tool name.

If the terminal receipt cannot be delivered, the Driver exits. API terminal finalization changes accepted commands to failed and changes the Session Run to failed. The reclaim module records a requeue recommendation only; it does not dispatch it. The stored state has no executed, not-executed, or unknown classification for the external action.

Evidence

• DriverCommandDelivery stores tracked commands in an in-memory Map: apps/driver/src/core/driver-command-delivery.ts.
• The MCP executor calls the external tool directly and has no Mosoo effect idempotency or receipt persistence: apps/driver/src/runtimes/mcp/remote-http-mcp-executor.ts.
• The local fault injection described above executed the fake external tool once, dropped all three terminal completion attempts, then executed the same command a second time in a fresh Driver. No Cloudflare, MCP credential, or remote system was used.
• bun --cwd apps/driver test tests/driver-runtime-boundary-state.test.ts passes, but covers duplicate delivery only while the same Driver remains alive.
• just test-file apps/api/tests/driver-finalization-repair.test.ts passes and demonstrates the current terminal behavior: accepted command and Run become failed while only a requeue decision is logged.
• just test-file apps/api/tests/session-run-reclaim-recovery.test.ts passes and confirms that recovery is a decision function, not an executed requeue.
• The product contract already says runtime loss is reported and the user must deliberately resend: docs/prd/session-lifecycle.md.

I searched current and closed issues. #329 fixes atomic Session Run admission and is no longer applicable. The merged #220 work classifies reclaim and records a requeue decision, but explicitly defers executing it and does not model external tool effects. #306 and #318 cover Cloudflare deployment resource reconciliation, not generic Sandbox tool effects.

Environment

• mosoo commit: b4a5f67
• Bun: 1.4.0-canary.1
• Local Driver test harness and SQLite D1 API fixture
• No remote Cloudflare account, Sandbox, MCP server, provider credential, model request, or billable resource

Contribution

I can test a fix.

Checklist

• I searched existing issues before opening this report.
• I included a reproducible path or explained why I cannot reproduce it locally.
• I removed or redacted secrets from logs, screenshots, and config.