diff --git a/docs/SURFACE.md b/docs/SURFACE.md index 29c6887f3..9585f606a 100644 --- a/docs/SURFACE.md +++ b/docs/SURFACE.md @@ -1736,6 +1736,32 @@ typed `run_not_found` refusal. A dropped connection, request failure, or journal may already have changed and the CLI cannot honestly claim the resume was refused before a write. +A timed-out read is handled separately: flow execution clients (`flows run` +and `flows resume`, authored and declarative) retry only `run.get`, +`journal.read`, `stream.read`, and `subscription.inspect` within a 300-second +total budget. Reads use a separate session, one in flight per body client, +with increasing attempt bounds and jitter. Interactive clients keep the +single-shot 30-second default; writes are never retried by this policy. +Unretried timeouts retain `journal client: timed out after ms`. +Exhausted reads instead name the verb, attempts, elapsed time, total read +budget, and possible CPU load. + +After a read budget expires, the CLI probes a fresh connection. A responding +daemon reports `daemon_unresponsive`; a failed probe reports +`daemon_unreachable` (also used for initial attach failures). A failed probe +cannot prove the daemon is dead: its diagnostic names both unreachability and +CPU load. Both reports exit 1, carry the known run/root id, retain +`status: running`, and include `flows resume`. This parks the CLI execution +without manufacturing a terminal journal fact; it does not claim a journaled +human park or verify journal integrity. The root worker session closes so the +kernel can recover its attempt, preserving completed work for resume. + +Worker completion waits use `run.watch` pushes with a snapshot every two +seconds for the live lease deadline. A scoped watch session closes after each +wait because the protocol has no unwatch verb. A heartbeat timeout relinquishes +worker ownership as lease loss, leaving recovery to the kernel instead of +recording `worker_error` against the body. + A step that ran and failed is **not** one of those. It reports `step_failed` with `status: failed`, for every step type and for authored TypeScript flows as well as declarative ones. `protocol_error` is reserved for an outcome the diff --git a/evidence/run-read-timeout/baseline-setup.md b/evidence/run-read-timeout/baseline-setup.md new file mode 100644 index 000000000..aa9e8e1ac --- /dev/null +++ b/evidence/run-read-timeout/baseline-setup.md @@ -0,0 +1,17 @@ +The handshake comparison used an unmodified detached checkout of +`3c58ee16d10a9e2400db980f5bbafac84e437f20`. The two probe sources are copied +verbatim from the `/tmp` scripts named in their captured output. They use this +session's absolute checkout/build paths; adjust those paths when reproducing +on another machine. + +Setup used (from the implementation checkout): + +```sh +git worktree add --detach /tmp/relayflow-read-timeout-baseline 3c58ee16d10a9e2400db980f5bbafac84e437f20 +ln -s /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk/node_modules /tmp/relayflow-read-timeout-baseline/packages/sdk/node_modules +npm run build --prefix /tmp/relayflow-read-timeout-baseline/packages/sdk +``` + +The baseline probe loads SDK code from that checkout while keeping the daemon +binary and stub fixture path identical to the current-code probe. It compares +that one fixture handshake failure, not the full suite. diff --git a/evidence/run-read-timeout/final-focused.log b/evidence/run-read-timeout/final-focused.log new file mode 100644 index 000000000..e9d5dcb40 --- /dev/null +++ b/evidence/run-read-timeout/final-focused.log @@ -0,0 +1,75 @@ +$ cd packages/sdk && RELAYFLOWD_BIN=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target/debug/relayflowd npx vitest run tests/journal-client-read-timeout.test.ts tests/journal-client.test.ts tests/journal-client-completion.test.ts tests/journal-client-subscriptions.test.ts tests/running-step-watch.test.ts tests/heartbeat-timeout.test.ts tests/run-daemon-unresponsive.test.ts tests/authored-root.test.ts tests/classify-outcome.test.ts tests/cli.test.ts tests/direct-run-worker-lease.test.ts tests/resume-worker-lease.test.ts tests/worker-lease.test.ts tests/worker-lease-lost.test.ts tests/worker-lease-lost-live.test.ts tests/worker-lease-sweep.test.ts tests/run-read-load-live.test.ts tests/flow-executor-chain.test.ts tests/agent-transcript-live.test.ts tests/human-live.test.ts tests/reuse-summary-interruption.test.ts tests/memoization.test.ts tests/authored-verifier-read-budget.test.ts tests/communication-worker.test.ts tests/communication.test.ts tests/helper-effect-lease-loss.test.ts tests/authored-helpers.test.ts tests/authored-flow-slack.test.ts tests/effect-channel.test.ts --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/cli.test.ts (71 tests) 5583ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 350ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2045ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms +(node:783010) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (36 tests) 458ms + ✓ tests/journal-client-read-timeout.test.ts (33 tests) 2247ms + ✓ an exhausted budget during reader setup frees the queue for the next read 303ms + ✓ tests/journal-client.test.ts (17 tests) 80ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 6820ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 422ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 999ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 2666ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 355ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 889ms + ✓ tests/authored-flow-slack.test.ts (7 tests) 1271ms + ✓ authored Slack helper effects > replays after SIGKILL before confirm with the same token and one successful completion 458ms + ✓ authored Slack helper effects > replays after SIGKILL before complete with the same token and one successful completion 436ms + ✓ tests/running-step-watch.test.ts (14 tests) 12309ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2115ms + ✓ cancels promptly while a lease snapshot read is in flight 2004ms + ✓ a completion push ends the wait without waiting out an in-flight snapshot 2053ms + ✓ a completion that aborts the snapshot is not an error even if a retry starts at once 2002ms + ✓ a malformed watch frame during an in-flight snapshot is an error, not a completion 2002ms + ✓ a watch dropped during an in-flight snapshot is reported at once, not after the read budget 2001ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 2755ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 662ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 686ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 792ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 614ms + ✓ tests/authored-helpers.test.ts (6 tests) 4017ms + ✓ lowers the named acceptance helpers and Slack to confirmed journal effects 352ms + ✓ runs every available provider through the real kernel and resumes completed effects without a second write 2254ms + ✓ replays after SIGKILL before confirm with the same token and one successful completion 726ms + ✓ replays after SIGKILL before complete with the same token and one successful completion 621ms + ✓ tests/classify-outcome.test.ts (11 tests) 7422ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2010ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2001ms + ✓ tests/human-live.test.ts (3 tests) 4840ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 2929ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1217ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 693ms + ✓ tests/communication-worker.test.ts (20 tests) 1826ms + ✓ tests/worker-lease.test.ts (8 tests) 12ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 20ms + ✓ tests/communication.test.ts (10 tests) 14ms + ✓ tests/effect-channel.test.ts (5 tests) 282ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 714ms + ✓ tests/run-daemon-unresponsive.test.ts (5 tests) 6ms + ✓ tests/memoization.test.ts (58 tests) 66ms + ✓ tests/run-read-load-live.test.ts (2 tests) 2979ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2064ms + ✓ drains read and watch promises before an authored flow completes 914ms + ✓ tests/reuse-summary-interruption.test.ts (6 tests) 5ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 6ms + ✓ tests/journal-client-completion.test.ts (6 tests) 109ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 10ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 11ms + ✓ tests/helper-effect-lease-loss.test.ts (1 test) 280ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 10ms + ✓ tests/authored-verifier-read-budget.test.ts (2 tests) 59ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 14ms + + Test Files 29 passed (29) + Tests 371 passed (371) + Start at 06:16:46 + Duration 66.56s (transform 1.62s, setup 189ms, collect 8.28s, tests 54.22s, environment 4ms, prepare 1.15s) + +exit=0 diff --git a/evidence/run-read-timeout/full-suite-final.log b/evidence/run-read-timeout/full-suite-final.log new file mode 100644 index 000000000..0f4c71463 --- /dev/null +++ b/evidence/run-read-timeout/full-suite-final.log @@ -0,0 +1,1970 @@ +$ cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npm test + +> @relayflows/sdk@2.0.42 test +> sh scripts/test.sh + + +> @relayflows/sdk@2.0.42 test:prep +> ( cd ../../kernel && sh ../ops/cargo.sh build ) && ( [ ! -d ../../testdata/preflight ] || find ../../testdata/preflight -name '*-cli' -type f -exec chmod +x {} + ) + + Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s + +> @relayflows/sdk@2.0.42 typecheck +> tsc --noEmit && tsc -p tsconfig.type-tests.json + + +> @relayflows/sdk@2.0.42 build +> tsc && node scripts/make-cli-executable.mjs + + +> @relayflows/sdk@2.0.42 typecheck:tests +> tsc -p tsconfig.tests.json + + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + +stdout | tests/live-kernel.test.ts +LIVE_KERNEL relayflowd=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd +LIVE_KERNEL flows=/home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk/dist/cli.js + + ✓ tests/preflight.test.ts (70 tests) 115ms + ✓ tests/cli.test.ts (71 tests) 5824ms + ✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 460ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 413ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2038ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms + ✓ tests/cloud-read.test.ts (46 tests) 55ms +stdout | tests/live-kernel.test.ts > surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once +LIVE_KERNEL kill -9 pid=44150 run=01M45TEQ6WQ827KKF8VV8WPXD1 while step=two state=Running + + ❯ tests/live-kernel.test.ts (32 tests | 8 failed) 57015ms + ✓ built flows CLI against live relayflowd > twenty-six-step reuses 25 durable completions after editing the failed final step 2821ms + ✓ built flows CLI against live relayflowd > runs rung (a), parks rung (b), and keeps JSON report-shaped 2765ms + ✓ built flows CLI against live relayflowd > allows a deterministic run to exceed the bounded request timeout 32494ms + ✓ built flows CLI against live relayflowd > follows a live worker dispatch through flows run 668ms + ✓ built flows CLI against live relayflowd > runs an agent CLI end to end through the SDK worker 466ms + ✓ built flows CLI against live relayflowd > f.agent lowers to a real agent step and dispatches through a live worker 578ms + ✓ built flows CLI against live relayflowd > can always get a parked run to a late-attaching worker 5588ms + ✓ built flows CLI against live relayflowd > reports a real manual-recovery NeedsHuman state as parked 485ms + × built flows CLI against live relayflowd > runs hn-monitor analyze-story end-to-end via a stub agent CLI (gate 2 clause 2 demo) 543ms + → expected { …(12) } to match object { output: { …(3) }, …(1) } +(22 matching properties omitted from actual) + × built flows CLI against live relayflowd > hn-monitor analyze-story FAILS verification when the CLI omits required schema fields 543ms + → expected { …(12) } to match object { …(3) } +(21 matching properties omitted from actual) + × built flows CLI against live relayflowd > agent step preserves the CliResult wrapper as output when the CLI emits non-JSON text 506ms + → expected null not to be null + × built flows CLI against live relayflowd > AgentWorker exposes wake_context to the CLI via RELAYFLOW_WAKE_CONTEXT env var (real analyzer prerequisite) 635ms + → Cannot read properties of null (reading 'story_title') + × built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_WAKE_CONTEXT UNSET when the run has no wake_context (undefined-vs-null pin) 587ms + → Cannot read properties of null (reading 'env_present') + ✓ built flows CLI against live relayflowd > AgentWorker passes a declared model to an identified wrapper as RELAYFLOW_MODEL 589ms + ✓ built flows CLI against live relayflowd > AgentWorker refuses a nonconforming journal-submitted wrapper before exposing RELAYFLOW_MODEL 522ms + ✓ built flows CLI against live relayflowd > AgentWorker executes the raw claude adapter with its real model flag 462ms + ✓ built flows CLI against live relayflowd > AgentWorker executes the raw codex adapter with its real model flag 635ms + × built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_MODEL UNSET when the step declares no model 1454ms + → Cannot read properties of null (reading 'story_title') + × built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI 29ms + → LIVE_ANALYZER_UNAVAILABLE: "/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-claude-cli" does not identify as relayflows-agent-cli-v1 — failing because gate-2 acceptance requires the real analyzer to execute. Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is not gate evidence. + ✓ built flows CLI against live relayflowd > preflights before journaling and names an unreachable socket 904ms + ✓ built flows CLI against live relayflowd > starts exactly one daemon when two runs race for one empty data dir 569ms + ✓ subprocess_gate output capture against live relayflowd > journals the gate command output and reports both tails 1007ms + ✓ surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once 882ms + × a relayflow can be scheduled: tick source against live relayflowd > a tick spawns a real run whose step reports the SCHEDULED instant 495ms + → expected null to deeply equal { schedule_id: 'heartbeat-1m', …(3) } + ✓ tests/cloud-live.test.ts (55 tests) 2058ms + ✓ argv and wiring > routes both live invocations through runCli, and refuses a missing run id 2004ms + ❯ tests/hosted-extension-isolation.test.ts (22 tests | 14 failed) 350ms + × hosted extension capability isolation > executes the exact capability-only handler for a queued receipt 9ms + → bubblewrap is unavailable + × hosted extension capability isolation > executes the exact capability-only handler for a duplicate receipt 2ms + → bubblewrap is unavailable + × hosted extension capability isolation > launches through the captured process primitive after builtin export synchronization 5ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > ignores inherited launcher overrides and decodes manifests with the captured Buffer intrinsic 5ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > streams verified bytes when the live store is replaced and no writable staging path exists 248ms + → promise rejected "Error: Hosted extension sandbox exited wi… { code: '…' }" instead of resolving + × hosted extension capability isolation > mounts pinned private Surface bytes when the live package changes before launch 4ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > refuses oversized Surface files through the bounded descriptor reader 6ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + × hosted extension capability isolation > shields verified Surface files before async settlement 4ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > preserves a typed host refusal while disclosing only a fixed marker to the child 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to be Error: private Cloud policy detail { code: '…' } // Object.is equality + × hosted extension capability isolation > denies ambient credentials, host files, writes, network, subprocesses, and undeclared context verbs 7ms + → bubblewrap is unavailable + × hosted extension capability isolation > enforces OS address-space and data bounds on native Buffer allocation 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + × hosted extension capability isolation > blocks extra handler fields and authority-bearing receipt fields at the parent port 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension capability isolation > constructs adapter authority with the captured freeze intrinsic 1ms + → bubblewrap is unavailable + × hosted extension capability isolation > writes the Surface manifest and protocol without inherited toJSON behavior 2ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ✓ tests/cloud-deploy.test.ts (117 tests) 2599ms + ✓ tests/cloud-sync.test.ts (40 tests) 736ms + ✓ tests/observer-link.test.ts (44 tests) 177ms + ✓ tests/authored-flow.test.ts (38 tests) 806ms + ✓ tests/plugin-extension.test.ts (94 tests) 464ms + ✓ tests/cloud-transcript-codex.test.ts (44 tests) 21ms + ✓ tests/worker-cli.test.ts (25 tests) 27087ms + ✓ registered CLI model defaults > passes the same priced Claude default to the real provider invocation 482ms + ✓ registered CLI model defaults > uses the explicitly supplied agent environment for the provider subprocess 427ms + ✓ registered CLI model defaults > dispatches canonical generic bytes with the preflight-proved adapter identity 493ms + ✓ direct transport lifecycle evidence > classifies only the exact Codex stdin lifecycle signature as retryable 410ms + ✓ direct transport lifecycle evidence > records a signal close separately from an ordinary nonzero exit 849ms + ✓ direct transport lifecycle evidence > records a spawn error code without treating a missing executable as transient 450ms + ✓ direct transport lifecycle evidence > journals classified lifecycle evidence and reports crashed instead of generic worker_error 610ms + ✓ step discovery environment > names the run, step, attempt and an absolute data dir for a direct agent spawn 582ms + ✓ step discovery environment > exports none of the four without a data dir, even when the worker inherited them 478ms + ✓ wrapper discovery environment > sets the four names from the dispatch and still refuses ambient values and other secrets 466ms + ✓ wrapper discovery environment > exports none of the four to a wrapper without a data dir, even when the worker inherited them 486ms + ✓ custom wrapper execution identity > passes an explicit safe environment at identification and execution 449ms + ✓ custom wrapper execution identity > refuses a wrapper symlink retarget before delivering private values 453ms + ✓ custom wrapper execution identity > bounds wrapper execution after acknowledgement 498ms + ✓ custom wrapper execution identity > bounds captured wrapper output 470ms + ✓ custom wrapper execution identity > refuses a duplicate execute protocol frame 432ms + ✓ custom wrapper execution bounds are reader-owned > resolves when a conforming wrapper leaks a stdio pipe to a background helper 697ms + ✓ custom wrapper execution bounds are reader-owned > resolves when the leaked helper inherits stderr only 673ms + ✓ custom wrapper execution bounds are reader-owned > resolves when a wrapper leaks a stdio pipe and exits before identifying 3447ms + ✓ custom wrapper execution bounds are reader-owned > journals a completionReason at the default bound when a wrapper leaks a stdio pipe 11463ms + ✓ custom wrapper execution bounds are reader-owned > accepts an execute token and an over-8KiB payload flushed in one write 434ms + ✓ custom wrapper execution bounds are reader-owned > accepts the same over-8KiB payload whether or not it coalesces with the execute token 1430ms + ✓ custom wrapper execution bounds are reader-owned > still bounds an un-terminated handshake buffer and names the bound 441ms + ✓ delivers the journaled memory pack to the real wrapper and excludes its charge from completion usage 462ms + ✓ tests/flow-extension-compose.test.ts (33 tests) 6663ms + ✓ composing flow extensions onto a base flow > composes two extensions in declaration order, and the order is the lockfile order 556ms + ✓ composing flow extensions onto a base flow > flows check reports the composition and keeps the composed triggers deliverable 1122ms + ✓ composing flow extensions onto a base flow > uses extension permissions for hosted deploy preflight and the deploy body 324ms +(node:46445) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/cli-status.test.ts (27 tests) 1171ms + ✓ flows status > resolves the run with no arguments from inside a worker-spawned agent 940ms + ✓ tests/run-state.test.ts (28 tests) 34ms + ✓ tests/cloud-run.test.ts (59 tests) 597ms + ✓ tests/relay-cli-surface.test.ts (87 tests) 42ms + ✓ tests/agent-transcript.test.ts (29 tests) 332ms +(node:46773) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (26 tests) 400ms + ✓ tests/babysitter-native-extension.test.ts (41 tests | 1 skipped) 1473ms + ✓ tests/shipped-source-model-provenance.test.ts (1 test) 109230ms + ✓ shipped-source model provenance > does not treat mutable aliases or incomplete named agents as pinned 109229ms + ✓ tests/authored-completion-detail.test.ts (59 tests) 223ms + ✓ tests/shipped-source-worker-call-forms.test.ts (1 test) 17255ms + ✓ shipped-source worker call forms > fails closed for computed keys, assertions, binds, call, and apply 17254ms + ✓ tests/step-failure-diagnostic.test.ts (26 tests) 72ms + ✓ tests/stop-process-group.test.ts (11 tests) 14220ms + ✓ every stop reaches the process group, not just the direct child > exits the run after an execution-timeout stop 970ms + ✓ every stop reaches the process group, not just the direct child > exits the run after a protocol terminate stop 486ms + ✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after a protocol terminate stop 1508ms + ✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after an execution-timeout stop 1913ms + ✓ every stop reaches the process group, not just the direct child > holds the loop open long enough for the escalation to run 1102ms + ✓ every stop reaches the process group, not just the direct child > bounds forced-stop confirmation when a group remains unprovable 1006ms + ✓ a wrapper that exits with no execution deadline still drains > reports the wrapper result and reaps a grandchild holding its pipes 707ms + ✓ a wrapper that exits with no execution deadline still drains > reaps a SIGTERM-deaf grandchild holding its pipes 1780ms + ✓ a wrapper that exits with no execution deadline still drains > settles on its own deadline when an escaped holder withholds close 4427ms + ✓ tests/mcp.test.ts (34 tests) 13995ms + ✓ MCP preflight and transports > flows check refuses an undeclared server with exit 2 and no daemon 865ms + ✓ MCP preflight and transports > flows check reports a refusing server and leaves no PID 944ms + ✓ MCP preflight and transports > kills a SIGTERM-resistant silent child after a parent-owned handshake deadline 1313ms + ✓ MCP preflight and transports > reaps a SIGTERM-resistant descendant with inherit stdio before cleanup finishes 1061ms + ✓ MCP preflight and transports > reaps a SIGTERM-resistant descendant with ignore stdio before cleanup finishes 2021ms + ✓ MCP preflight and transports > rejects close when forced group death remains unprovable 2065ms + ✓ MCP preflight and transports > cancels force escalation when close follows an exited child 1179ms + ✓ MCP preflight and transports > waits for child close when the transport cannot own a process group 1055ms + ✓ MCP preflight and transports > cancels force escalation for an already-closed child without a process group 1177ms + ✓ MCP preflight and transports > reports malformed connection configuration as config_invalid 803ms + ✓ authored MCP effects against the real kernel > reports a dropped tool connection as a failed CLI run 647ms + ✓ tests/step-attempt-history.test.ts (23 tests) 27ms + ✓ tests/daemon-lifecycle.test.ts (42 tests) 41ms + ✓ tests/close-pr-flow.test.ts (35 tests) 21131ms + ✓ close-pr journaled repair loop > reads an existing Bugbot finding, repairs in the same worktree, pushes and re-verifies before merging 2505ms + ✓ close-pr journaled repair loop > opens a PR and feeds failed CI logs into the repair agent 2128ms + ✓ close-pr journaled repair loop > pins the current generated model for codex when no override is supplied 4998ms + ✓ close-pr journaled repair loop > pins the current generated model for claude when no override is supplied 3900ms + ✓ close-pr journaled repair loop > parks after exactly three nonconverging repairs, with accumulated blockers 2385ms + ✓ close-pr journaled repair loop > can converge on the third repair 2486ms + ✓ close-pr journaled repair loop > executes the deterministic commit and force-push steps against a local Git remote, including a no-op repair 2527ms + ❯ tests/authored-node-runtime.test.ts (16 tests | 16 skipped) 12ms + ✓ tests/artifact-gates.test.ts (27 tests) 242ms + ❯ tests/hosted-extension-protocol.test.ts (27 tests | 8 failed) 40086ms + × hosted extension hostile protocol > uses captured JSON intrinsics for the complete parent boundary 6ms + → bubblewrap is unavailable + × hosted extension hostile protocol > rejects an import-time different PR frame with zero adapter calls 5ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects an import-time different delivery frame with zero adapter calls 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects an import-time different event frame with zero adapter calls 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects two forged calls after the authoritative first outcome settles 10004ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > waits for a pending adapter to reject after a forged child error 10004ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > waits for a pending adapter to resolve after a forged child error 10005ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > returns a typed adapter rejection even when the hostile child hangs 10005ms + → hostile child did not invoke the adapter + ✓ tests/hosted-base-snapshot.test.ts (18 tests) 1873ms + ✓ hosted base private snapshot > stops streaming project entries at the shared count limit 1797ms + ✓ tests/cloud-connect.test.ts (24 tests) 3023ms + ✓ hosted verbs connect before they submit > flows run --cloud submits once the prompt connected the integration 2120ms + ✓ tests/journal-client.test.ts (17 tests) 81ms + ✓ tests/authored-node-result.test.ts (45 tests) 20ms + ✓ tests/verb-field-lint.test.ts (99 tests) 389ms + ✓ tests/bundle.test.ts (26 tests) 9078ms + ✓ immutable bundles > returns exit 2 naming a byte-flipped payload and refuses to reuse corruption 406ms + ✓ immutable bundles > verifies with --verify in any position and answers --json with one object 799ms + ✓ immutable bundles > refuses --out with --verify rather than ignoring the destination 401ms + ✓ immutable bundles > builds and verifies the canonical YAML fixture through the compiled CLI 1253ms + ✓ immutable bundles > emits the ephemeral warning on CLI stderr and uses the default output directory 793ms + ✓ immutable bundles > refuses build-provable CLI resolution errors without environment probes 431ms + ✓ immutable bundles > builds a standalone TS fixture twice with identical executable hashes 2353ms + ✓ immutable bundles > refuses to label installed dependency drift with lockfile pins 384ms + ✓ immutable bundles > refuses invalid CLI arguments %j 396ms + ✓ immutable bundles > refuses invalid CLI arguments "--out" 396ms + ✓ immutable bundles > refuses invalid CLI arguments "--verify" 397ms + ✓ immutable bundles > refuses invalid CLI arguments "--verify" 394ms + ✓ immutable bundles > refuses invalid CLI arguments "--out" 529ms + ✓ tests/validate.test.ts (68 tests) 34ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 9665ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 842ms + ✓ flow executor LLM and output-binding chain > runs a dollar-budgeted authored Claude agent with the same default used by preflight 547ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 1465ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 3225ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 761ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 1096ms + ✓ tests/agent-relay-transport.test.ts (17 tests) 2272ms + ✓ Relay completion at the journal boundary > does not complete at readiness and journals exact output, receipt, and priced accounting 1006ms + ✓ Relay completion at the journal boundary > aborts polling on rejected renewal and never writes a stale completion 1000ms + ✓ tests/shipped-source-flow-provenance-repairs.test.ts (1 test) 30332ms + ✓ shipped-source flow provenance repairs > audits repaired writer, alias, binding, and cyclic provenance forms 30331ms + ✓ tests/tick-source.test.ts (33 tests) 21ms + ✓ tests/authored-step-graph.test.ts (25 tests) 837ms + ✓ the authored step DAG > does not walk a long-running step's own polling chain to find its dependents' edges 337ms + ✓ tests/cloud-mirror.test.ts (14 tests) 33ms + ✓ tests/pr-review-post.test.ts (21 tests) 1982ms + ✓ tests/authored-flow-lifecycle-executor.test.ts (27 tests) 587ms + ✓ tests/step-failure-excerpt.test.ts (42 tests) 195ms + ✓ tests/authored-step-terminal-completion.test.ts (13 tests) 4878ms + ✓ readCompletedStepOutput on an adopted gated child whose gate has not started > refuses a producer success when the wait ends with the run still not terminal 2006ms + ✓ readCompletedStepOutput given a suspended known status (#441) > refuses a producer success while the child is still not terminal 2006ms +(node:52566) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/cloud-mirror-step.test.ts (17 tests) 12ms + ✓ tests/cli-replay.test.ts (38 tests) 1314ms + ✓ flows replay > --json is byte-identical across two CLI invocations (diff) 941ms + ✓ tests/authored-flow-slack.test.ts (7 tests) 1753ms + ✓ authored Slack helper effects > replays after SIGKILL before confirm with the same token and one successful completion 560ms + ✓ authored Slack helper effects > replays after SIGKILL before complete with the same token and one successful completion 528ms + ✓ tests/wrapper-execution-duration.test.ts (7 tests) 10822ms + ✓ keeps the handshake deadline independent of the removed execution deadline 10054ms + ✓ still lets a lease abort stop an unlimited wrapper before it produces output 511ms + ✓ tests/tick-runner.test.ts (22 tests) 2354ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms fractional as an invocation error 393ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms exponent notation as an invocation error 385ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms hex as an invocation error 385ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms trailing text as an invocation error 390ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms empty as an invocation error 391ms + ✓ CLI argument parsing refuses coercion rather than accepting it > accepts an exact integer and proceeds past parsing 385ms + ✓ tests/authored-step-index.test.ts (17 tests) 16ms + ✓ tests/named-gate-diagnostics.test.ts (20 tests) 640ms + ✓ tests/authored-agent-artifacts.test.ts (5 tests) 325ms + ✓ tests/direct-input.test.ts (6 tests) 6022ms + ✓ direct .flow.ts input through the built CLI and live runtime > returns exit 3 for an authored human handoff and persists its outcome 592ms + ✓ direct .flow.ts input through the built CLI and live runtime > returns exit 1 for an authored step_failed verdict and persists its outcome 632ms + ✓ direct .flow.ts input through the built CLI and live runtime > executes inline and file JSON input through relayflowd 2421ms + ✓ direct .flow.ts input through the built CLI and live runtime > refuses missing and malformed input before contacting relayflowd 1517ms + ✓ direct .flow.ts input through the built CLI and live runtime > does not run the authored body before daemon availability 478ms + ✓ direct .flow.ts input through the built CLI and live runtime > refuses oversized file input before contacting relayflowd 381ms + ✓ tests/worker-transcript.test.ts (9 tests) 866ms + ✓ tests/gate-contract.test.ts (20 tests) 151ms + ❯ tests/canonical-software-factory.test.ts (25 tests | 25 failed) 152ms + × canonical software-factory metadata contract > opens the actual catalog flow with the ticket title and exactly one GitHub closing line 11ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > writes a Linear closing reference that the GitHub integration links back 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > accepts a Linear team key that carries digits 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier () 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier (not-an-issue) 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when the Linear closing reference is duplicated in the final body 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when the body carries a foreign closing reference 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > does not mistake ordinary closing-verb prose for a reference 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > normalizes whitespace and caps the title at 240 Unicode code points 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when GitHub identity is missing or the final body duplicates its closing line 0ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for explicit defects 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for no verdict 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory blocked and unverified 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory passed and unverified 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory passed and blocked 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for empty unverified 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > distinguishes a missing verification prerequisite from a defect 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > preserves the passed body without a scope marker or draft 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects a forged scope for local before publication 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects a forged scope for github before publication 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > scopes the post-review hook refusal 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > scopes the merge-gate hook refusal 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "" before publication 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "not-a-sha" before publication 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "a'; touch injected; #" before publication 6ms + → expected an @relayflows/surface flow handle + ✓ tests/authored-human.test.ts (13 tests) 91ms + ✓ tests/wrapper-exit-drain.test.ts (10 tests) 3512ms + ✓ reports a signalled wrapper death while a pipe is held, with its output intact 379ms + ✓ lets a lease abort outrank a successful exit still being drained 537ms + ✓ does not let a later drain overwrite abort while group confirmation is pending 631ms + ✓ tests/cloud-schedule.test.ts (17 tests) 5275ms + ✓ schedule lowering > marks a non-grid cron as Cloud-only rather than approximating it, with a silence budget from its own cadence 2913ms + ✓ flows check prints declared schedules > shows the lowering for a fixed interval and the Cloud-only note for a real cron 1956ms + ✓ tests/authored-run-failure-evidence.test.ts (9 tests) 848ms + ✓ the child index after the process that wrote it is gone > still names every child, with its own run id, after a daemon restart 484ms + ✓ tests/worker-cli-result-exit.test.ts (7 tests) 34844ms + ✓ a Claude agent step completes on its result, not only on process exit > settles a hung, successful run within the grace and stops its whole tree 32619ms + ✓ a Claude agent step completes on its result, not only on process exit > maps an error result on a hung run to a failed exit 31619ms + ✓ a Claude agent step completes on its result, not only on process exit > leaves a hang before any result to the existing stops 32014ms + ✓ an agent tree does not outlive the process that spawned it > kills the agent group when the run process is terminated by SIGTERM 795ms + ✓ an agent tree does not outlive the process that spawned it > removes a wrapper alias on host exit after group death stays unprovable 1341ms + ✓ tests/cli-hn-monitor.test.ts (16 tests) 95ms + ✓ tests/spec-parity.test.ts (46 tests) 505ms + ✓ tests/authored-activity.test.ts (16 tests) 108ms + ✓ tests/run-projection.test.ts (13 tests) 12ms + ✓ tests/agent-artifacts-live.test.ts (5 tests) 4163ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > journals the files the agent wrote, including under a dot-directory, and every artifact gate passes on that journal 1030ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > fails the run when the artifact_exists gate names a file the agent did not write 803ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > fails the run with the author reason when a predicate gate returns false, journaling the verdict 877ms + ✓ review follow-ups > applies a predicate gate on a helper step too, and journals its verdict 790ms + ✓ review follow-ups > records predicate verdicts on the root run so a resume reuses them instead of re-running the closure 662ms + ✓ tests/daemon-lifecycle-live.test.ts (9 tests) 6889ms + ✓ flows run against a data dir with no daemon (§6 test 7) > cold start spawns exactly one daemon, the run succeeds, and the daemon outlives the CLI 492ms + ✓ flows run against a data dir with no daemon (§6 test 7) > polls, bounded, for a daemon that holds the lock before it binds 1369ms + ✓ flows run against a data dir with no daemon (§6 test 7) > attaches to a serving daemon that has not published a connection file 505ms + ✓ flows run against a data dir with no daemon (§6 test 7) > a second run attaches to the daemon the first one started, spawning nothing 964ms + ✓ flows run against a data dir with no daemon (§6 test 7) > detects a stale connection file left by a hard kill and starts a fresh daemon 1010ms + ✓ concurrent invocations against one empty data dir (§6 test 15) > ends with exactly one daemon owning the socket, and both runs succeed 1203ms + ✓ refusals from a spawn that cannot produce a daemon > names relayflowd_not_found rather than falling through to PATH 413ms + ✓ refusals from a spawn that cannot produce a daemon > names daemon_start_failed and quotes the daemon log when startup dies 470ms + ✓ refusals from a spawn that cannot produce a daemon > refuses a daemon speaking another protocol version instead of binding over it 464ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 3517ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 852ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 806ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 885ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 973ms + ✓ tests/resume-local-agent.test.ts (8 tests) 13ms +(node:55014) Warning: Transcript tail for run-9/analyze attempt 1 (stdout) could not be written; the step continues without it: EACCES: permission denied, mkdir '/tmp/transcript-tail-1nBIzB/runs/run-9/steps' +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/transcript-tail.test.ts (11 tests) 1192ms + ✓ direct agent spawn > tees stdout and stderr into tail files that name the dispatch 527ms + ✓ direct agent spawn > completes the step when the tail directory cannot be created 384ms + ✓ tests/advisory-outcome.test.ts (23 tests) 72ms + ✓ tests/shipped-source-cubic-regressions.test.ts (5 tests) 11768ms + ✓ shipped-source adversarial provenance regressions > retains callable writes through member mutation, spreads, this, holes, and alternate receivers 6294ms + ✓ shipped-source adversarial provenance regressions > does not manufacture intrinsic writes or comma-left callables 2442ms + ✓ shipped-source adversarial provenance regressions > treats constructor and implicit-this receiver escapes as possible worker writes 666ms + ✓ shipped-source adversarial provenance regressions > maps pure helper parameters and refuses mutations or conditional assignments as proofs 1755ms + ✓ shipped-source adversarial provenance regressions > does not apply later alias assignments retroactively to copied values 610ms + ✓ tests/agent-cwd-live.test.ts (4 tests) 2928ms + ✓ agents drive two checkouts under one run root > runs each CLI in its declared directory, and the run is not refused as an unknown field 868ms + ✓ agents drive two checkouts under one run root > gates on an artifact path relative to the agent directory 779ms + ✓ a declaration this contract cannot honour is refused at its own edge > refuses a path that climbs out of the run root without starting a run at all 600ms + ✓ a declaration this contract cannot honour is refused at its own edge > fails the step with the reason when the directory is not there, rather than running somewhere else 680ms + ✓ tests/authored-helpers.test.ts (6 tests) 3363ms + ✓ runs every available provider through the real kernel and resumes completed effects without a second write 1593ms + ✓ replays after SIGKILL before confirm with the same token and one successful completion 712ms + ✓ replays after SIGKILL before complete with the same token and one successful completion 741ms + ✓ tests/cloud-mirror-session.test.ts (12 tests) 14ms + ✓ tests/agent-cwd.test.ts (53 tests) 50ms + ✓ tests/pty-sidechannel.test.ts (14 tests) 8465ms + ✓ view attach preserves worker completion and marks only drive 970ms + ✓ drive attach preserves worker completion and marks only drive 460ms + ✓ passthrough attach preserves worker completion and marks only drive 907ms + ✓ none attach preserves worker completion and marks only drive 947ms + ✓ none subscriber lets an unattended CLI read EOF 506ms + ✓ view subscriber lets an unattended CLI read EOF 497ms + ✓ passthrough subscriber lets an unattended CLI read EOF 494ms + ✓ incomplete subscriber lets an unattended CLI read EOF 513ms + ✓ unattended Codex receives closed stdin before startup instead of entering its additional-input lifecycle 591ms + ✓ ends driven stdin when the last drive peer disconnects after spawn 730ms + ✓ rejects drive after EOF without marking human intervention 951ms + ✓ delivers all drive bytes in order across child stdin backpressure 866ms + ✓ tests/authored-flow-operation.test.ts (24 tests) 507ms + ✓ tests/budget-attribution.test.ts (10 tests) 9ms + ✓ tests/helper-reference.test.ts (30 tests) 22ms + ✓ tests/cloud-mirror-live.test.ts (3 tests) 2782ms + ✓ a local run on the Cloud dashboard > registers, publishes its steps and its log, and reports terminal last 547ms + ✓ a local run on the Cloud dashboard > sends nothing at all unless it is asked to 1358ms + ✓ a local run on the Cloud dashboard > publishes a composed authored flow as one connected dashboard DAG 835ms + ✓ tests/flow-requirements.test.ts (14 tests) 720ms + ✓ flows check prints REQUIRES > names the helper, the harness and the mcp server of an authored flow 508ms + ✓ tests/backlog-picker.test.ts (14 tests) 41ms + ✓ tests/plugin-store-bounds.test.ts (11 tests) 69ms + ✓ tests/classify-outcome.test.ts (11 tests) 7417ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2007ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2000ms + ✓ tests/authored-completion-recovery.test.ts (5 tests) 375ms + ✓ tests/backlog-picker-flow.test.ts (6 tests) 262ms + ✓ tests/hosted-extension-protocol-intrinsics.test.ts (6 tests) 13ms + ✓ tests/preflight-permissions-unenforced.test.ts (17 tests) 517ms + ✓ an authored TypeScript body is out of reach, and spec.ts says so > checks clean on a .flow.ts whose body declares permissions 473ms + ✓ tests/budget-unmetered-live.test.ts (4 tests) 1245ms + ✓ unmetered budget spend through the live kernel > runs an unpriced step under a dollar budget without tripping it, journaling unknown dollars 444ms + ❯ tests/stuck-run-triage.test.ts (22 tests | 22 failed) 74ms + × stuck-run-triage input validation > refuses an 8-character run-id prefix: Cloud has no prefix lookup 4ms + → expected [Function] to throw error matching /not full Cloud run ids: c649fe14/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses the whole batch when any id is invalid, rather than dropping it 1ms + → expected [Function] to throw error matching /not full Cloud run ids: nope!/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses an empty batch 0ms + → expected [Function] to throw error matching /needs runIds/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses a batch too large for the edge step lease 0ms + → expected [Function] to throw error matching /exceeds the 8 that fit/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > accepts eight ids — the incident batch is inside the bound 3ms + → promise rejected "TypeError: expected an @relayflows/surfac…" instead of resolving + × stuck-run-triage apiUrl > refuses to send the Cloud bearer token to an unapproved origin 0ms + → expected [Function] to throw error matching /refusing to send the Cloud bearer to…/\ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage apiUrl > refuses a non-URL apiUrl 1ms + → expected [Function] to throw error matching /is not a URL/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage apiUrl > allows an approved origin and uses it in the curl 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage apiUrl > defaults to production Cloud 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage apiUrl > never publishes a run record the fetch did not produce 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > names the Worker on every wrangler invocation 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > accepts caller-supplied Workers and rejects option-shaped ones 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > falls back when GNU timeout is absent, as it is on macOS 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > runs the tails concurrently so wall time does not scale with the batch 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > records wrangler's own exit status rather than head's 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage shell text > parses under both sh and bash 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage shell text > collects tails with no GNU timeout on PATH, as on a stock macOS 58ms + → expected an @relayflows/surface flow handle + × stuck-run-triage agents > declares read-only permissions on every agent 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage agents > tells the forensics agents their evidence is untrusted 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage fan-out > refuses a duplicate run id: two tails would share one evidence file 1ms + → expected [Function] to throw error matching /duplicate runIds: c649fe14-0c2e-4e51-…/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage fan-out > refuses a duplicate Worker name for the same reason 0ms + → expected [Function] to throw error matching /duplicate workers: w-one/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage fan-out > bounds ids x workers, not just ids 0ms + → expected [Function] to throw error matching /24 concurrent tails, over the 16/ but got 'expected an @relayflows/surface flow …' + ✓ tests/hosted-extension-routing.test.ts (7 tests) 7ms + ✓ tests/webhook.test.ts (9 tests) 784ms + ✓ webhook ingress > checks TS declarations against flows.json without invoking handlers 675ms + ✓ tests/redact.test.ts (54 tests) 10ms + ✓ tests/cli-probe.test.ts (17 tests) 1830ms + ✓ tests/authored-advisory-run-live.test.ts (16 tests) 883ms + ✓ tests/webhook-live.test.ts (6 tests) 9746ms + ✓ executes and deduplicates 'app_mention' only for its provider and matching payload 1471ms + ✓ executes and deduplicates 'reaction_added' only for its provider and matching payload 1447ms + ✓ executes and deduplicates 'pull_request' only for its provider and matching payload 1448ms + ✓ flows serve-webhook writes JSON before the daemon starts, then journals and archives exactly once 1444ms + ✓ replays a dropped file after SIGKILL before spawn 478ms + ✓ resumes the same journal after SIGKILL after spawn and before acknowledgement 3457ms + ✓ tests/authored-parallel-agents.test.ts (8 tests) 8916ms + ✓ authored steps under local workers with capacity > runs more concurrent f.llm calls than the worker holds side by side, never more than its capacity 1086ms + ✓ authored steps under local workers with capacity > completes more concurrent f.agent calls than the worker holds: the overflow waits for a slot instead of parking 1473ms + ✓ authored steps under local workers with capacity > runs agents in distinct working directories side by side (the kernel carries cwd) 596ms + ✓ authored steps under local workers with capacity > serializes agents whose cwd is a symlink alias of the same directory 1014ms + ✓ authored steps under local workers with capacity > serializes agents whose cwd is a directory nested inside the other 1036ms + ✓ authored steps under local workers with capacity > never starts queued agents once the body has failed 1568ms + ✓ authored steps under local workers with capacity > never starts a queued agent when the agent holding the only slot fails 1570ms + ✓ authored steps under local workers with capacity > parks the overflow when the body is not told the capacity (the defect this closes) 573ms + ✓ tests/local-agent-live.test.ts (7 tests) 42312ms + ✓ built CLI local agent against a real daemon > dispatches through the wrapper and keeps --json stdout report-shaped 766ms + ✓ built CLI local agent against a real daemon > runs beyond the initial 30-second lease without a second invocation 35817ms + ✓ built CLI local agent against a real daemon > renders actual agent completion in text output 888ms + ✓ built CLI local agent against a real daemon > returns a failed run when the agent process fails 813ms + ✓ built CLI local agent against a real daemon > prints a remedy that runs the parked authored flow to completion 1608ms + ✓ built CLI local agent against a real daemon > prints a remedy that runs, for a run started from an input file too long to be one 1818ms + ✓ built CLI local agent against a real daemon > refuses a workspace it cannot pin before invoking the agent 600ms + ✓ tests/cli-watch.test.ts (10 tests) 16514ms + ✓ flows check --watch > rechecks syntax errors, clears once, and returns the last refusal on Ctrl-C 1335ms + ✓ flows check --watch > streams JSON lines without ANSI, recovers after atomic saves, and exits zero after repair 1867ms + ✓ flows check --watch > coalesces 20 concurrent saves into at most two rechecks 1870ms + ✓ flows check --watch > watches transitive relative use imports, cycles, and nearest config changes 2461ms + ✓ flows check --watch > refreshes the import graph and notices missing imports being created 2520ms + ✓ flows check --watch > reloads authored TypeScript instead of reusing the first imported definition 1804ms + ✓ flows check --watch > detects a nearer config appearing and falls back after it is deleted 1973ms + ✓ flows check --watch > keeps watching after the target is deleted and recreated 1905ms + ✓ flows check --watch > queues changes during a slow check without overlapping checks 775ms + ✓ tests/check-worker-surface.test.ts (11 tests) 80ms + ✓ tests/human-live.test.ts (3 tests) 6825ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 3963ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1759ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 1102ms +(node:58939) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-status-detail.test.ts (20 tests) 124ms + ✓ tests/authored-step-failed.test.ts (10 tests) 42ms + ✓ tests/budget-preflight.test.ts (25 tests) 18ms + ✓ tests/authored-retried-child-resume-live.test.ts (2 tests) 1732ms + ✓ an authored step whose child run retried an attempt > resumes after a kill mid-step and reads the terminal completion, not the crashed one 1619ms + ✓ tests/provider-trigger-contract.test.ts (7 tests) 872ms + ✓ provider trigger contract > accepts one subscription from each of five providers and refuses a bogus event on any of them 510ms + ✓ provider trigger contract > fails `flows check` before deployment and passes once the event is real 357ms + ✓ tests/shipped-source-flow-header-provenance.test.ts (1 test) 2910ms + ✓ shipped-source flow header provenance > fails closed for unsafe authored and declarative headers 2910ms + ✓ tests/work-package-consumer.test.ts (13 tests) 111ms + ✓ tests/helpers-fanout.test.ts (96 tests) 159ms +(node:59280) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-step-graph-live.test.ts (1 test) 944ms + ✓ the authored step DAG through the live kernel > carries labels and predecessors on every index record and journal step, ids unchanged 943ms + ✓ tests/named-gate-journal.test.ts (5 tests) 1760ms + ✓ a lowered subprocess_gate journals the gate command's streams > persists what the gate printed before a timeout killed it 828ms + ✓ a gate on an agent step > journals the command's streams and surfaces them in the authored failure 647ms + ✓ tests/generate-triggers.test.ts (7 tests) 1403ms + ✓ discovers new adapters, preserves exact event names, and prefers adapter-local mappings 404ms + ✓ tests/local-agent-remedy.test.ts (16 tests) 13ms + ✓ tests/yaml-local-agent-live.test.ts (9 tests) 6529ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked step CLI and model and journals done 601ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked named CLI and model and journals done 622ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked flow CLI and model and journals done 590ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked project CLI and model and journals done 582ms + ✓ YAML --local-agent through the built CLI and real daemon > still parks without --local-agent 539ms + ✓ YAML --local-agent through the built CLI and real daemon > resumes a parked spec run with --local-agent instead of parking identically again 1004ms + ✓ YAML --local-agent through the built CLI and real daemon > parks a second time with a different message when the attached worker is not eligible 1020ms + ✓ YAML --local-agent through the built CLI and real daemon > reports the agent process failure 852ms + ✓ YAML --local-agent through the built CLI and real daemon > preserves declared workspace surfaces that the local worker cannot pin 717ms + ✓ tests/yaml-helpers.test.ts (34 tests) 77ms + ✓ tests/direct-run-failure.test.ts (12 tests) 41ms + ✓ tests/authored-agent-permissions.test.ts (27 tests) 902ms + ✓ tests/authored-detail-live.test.ts (3 tests) 3737ms + ✓ carries done("step_failed", { detail }) into the report, the journal and flows status 2146ms + ✓ reports a detail a caller sliced through an emoji, instead of hanging on it 603ms + ✓ leaves a one-argument done("step_failed") reporting exactly as it always did 987ms + ✓ tests/live-event-activities.test.ts (2 tests) 202ms + ✓ tests/software-garden-babysitter-composition.test.ts (6 tests | 2 skipped) 43ms + ✓ tests/worker-lease.test.ts (7 tests) 18ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 25ms + ✓ tests/webhook-hardening.test.ts (11 tests) 58ms + ✓ tests/authored-probe-cache.test.ts (5 tests) 21050ms + ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 1 6998ms + ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 4 5396ms + ✓ authored run CLI probe cache > does not serialize nine starts behind nine probes, and probes again on a new run 7027ms + ✓ authored run CLI probe cache > caches probe failures while refusing all nine calls 397ms + ✓ authored run CLI probe cache > shares probe results across agent calls too 1232ms + ✓ tests/yaml-declared-streams-live.test.ts (8 tests) 5063ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked step CLI and model and journals done 609ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked named CLI and model and journals done 599ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked flow CLI and model and journals done 606ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked project CLI and model and journals done 587ms + ✓ YAML declared streams through the built CLI and real daemon > still parks without --local-agent 539ms + ✓ YAML declared streams through the built CLI and real daemon > reports the agent process failure 613ms + ✓ YAML declared streams through the built CLI and real daemon > resumes the journaled stream declarations after the input YAML changes 1001ms + ✓ YAML declared streams through the built CLI and real daemon > preserves declared workspace surfaces that the local worker cannot pin 508ms + ✓ tests/human-to.test.ts (8 tests) 13ms +(node:61719) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-dispatch-live.test.ts (2 tests) 2417ms + ✓ durable child flow dispatch > runs a declared child in the parent DAG and replays it without repeating effects 1621ms + ✓ durable child flow dispatch > lowers a named gate attached to the dispatch receipt 794ms + ✓ tests/plugin-loader.test.ts (9 tests) 195ms + ✓ tests/worker-lease-drift-live.test.ts (3 tests) 36205ms + ✓ worker lease under clock skew between worker and daemon > renews before the daemon deadline when the worker clock runs 75s behind the daemon 36062ms + ✓ tests/deploy.test.ts (11 tests) 5316ms + ✓ flows deploy file buckets > publishes the full signed layout byte-for-byte and redeploys as a noop 824ms + ✓ flows deploy file buckets > answers --json with one object per outcome 830ms + ✓ flows deploy file buckets > reports a refusal as JSON under --json 398ms + ✓ flows deploy file buckets > refuses a missing local bundle before creating the bucket 394ms + ✓ flows deploy file buckets > refuses an unreachable bucket before copying 403ms + ✓ flows deploy file buckets > refuses an unwritable bucket 410ms + ✓ flows deploy file buckets > refuses local tampering of spec.canonical.json 405ms + ✓ flows deploy file buckets > refuses local tampering of identity.json 419ms + ✓ flows deploy file buckets > refuses asset bundles instead of using daemon-relative files 392ms + ✓ flows deploy file buckets > never labels a corrupt existing deployment as a noop 813ms + ✓ tests/bin.test.ts (12 tests) 5976ms + ✓ built flows binary > prints the installed SDK version for --version 383ms + ✓ built flows binary > prints the installed SDK version for -V 389ms + ✓ built flows binary > keeps version flags strict when extra arguments are supplied 377ms + ✓ built flows binary > prints the installed SDK version from the standalone binary for --version 973ms + ✓ built flows binary > prints the installed SDK version from the standalone binary for -V 987ms + ✓ built flows binary > refuses through a symlink to the built artifact 383ms + ✓ built flows binary > refuses through a symlinked directory component 400ms + ✓ built flows binary > classifies a signal-terminated auth probe as probe_failed 859ms + ✓ built flows binary > classifies an unavailable PATH resolver as probe_failed 420ms + ✓ built flows binary > does not describe a present non-executable CLI as missing 403ms + ✓ built flows binary > runs one auth probe for three steps sharing a flow CLI 399ms + ✓ tests/babysitter-catalog-export.test.ts (14 tests) 759ms + ✓ Babysitter catalog artifact export > CLI refuses an existing output and leaves no file on validation failure 606ms + ✓ tests/journal-client-read-timeout.test.ts (13 tests) 1065ms + ✓ a recovered read timeout does not become an authored callback failure 337ms + ✓ tests/communication.test.ts (10 tests) 13ms + ✓ tests/communication-review.test.ts (6 tests) 324ms + ✓ tests/transport-evidence-bounds.test.ts (7 tests) 11ms + ✓ tests/typed-output.test.ts (14 tests) 209ms + ✓ tests/shipped-source-models.test.ts (2 tests) 39331ms + ✓ first-party shipped source model pins > gives every TypeScript agent and LLM an explicit supported pair or a pinned named-agent declaration 39293ms + ✓ tests/agent-timeout-live.test.ts (3 tests) 4254ms + ✓ journals timeout, stops the process, runs a predicate gate and publishes under a budget header 1156ms + ✓ replays timeout after killing the root and daemon, without executing the agent again 1973ms + ✓ lets an author reject timeout through a predicate gate 1123ms + ✓ tests/effect-channel.test.ts (5 tests) 397ms + ✓ tests/worker-slots.test.ts (8 tests) 7ms + ✓ tests/step-lease.test.ts (36 tests) 66445ms + ✓ f.run leases against the live kernel > enforces 10000 ms for 'sleep 5; printf ok' 5065ms + ✓ f.run leases against the live kernel > enforces 40000 ms for 'sleep 31; printf ok' 31101ms + ✓ f.run leases against the live kernel > enforces 30000 ms for 'sleep 31; printf ok' 30086ms + ✓ tests/json-schema-bound.test.ts (71 tests) 2320ms + ✓ JSON Schema termination bound > walks a deep schema with an explicit stack rather than recursion 1896ms + ✓ tests/cloud-mirror-transport.test.ts (5 tests) 9ms + ✓ tests/mcp-lifecycle.test.ts (4 tests) 19ms + ✓ tests/communication-worker.test.ts (17 tests) 1708ms + ✓ tests/model-selection.test.ts (10 tests) 17ms + ✓ tests/f-memory.test.ts (7 tests) 781ms + ✓ tests/relayflowd-path.test.ts (10 tests) 5ms + ✓ tests/agent-artifacts.test.ts (9 tests) 20ms + ✓ tests/authored-plugin-effect.test.ts (6 tests) 65ms + ✓ tests/agent-timeout-worker.test.ts (9 tests) 2190ms + ✓ stops claude at its declared limit, preserving work and stopping descendants 523ms + ✓ stops codex at its declared limit, preserving work and stopping descendants 517ms + ✓ stops wrapper at its declared limit, preserving work and stopping descendants 536ms + ✓ tests/hosted-hardening.test.ts (11 tests) 15ms + ✓ tests/local-dev-ux.test.ts (8 tests) 14ms + ✓ tests/authored-preflight.test.ts (5 tests) 214ms + ✓ tests/deterministic-llm.test.ts (5 tests) 53ms + ✓ tests/relay-cli-surface-live.test.ts (3 tests) 416ms + ✓ tests/authored-declined.test.ts (13 tests) 51ms + ✓ tests/resume-failure.test.ts (2 tests) 5ms + ✓ tests/dependency-validation.test.ts (6 tests) 759ms + ✓ dependency validation > accepts a valid 10,000-step reverse chain through every direct public boundary 417ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 878ms + ✓ reports journal success after completion rejects with lease_conflict 315ms + ✓ tests/authored-hooks.test.ts (5 tests) 6ms + ✓ tests/input-binding.test.ts (12 tests) 201ms + ✓ tests/yaml-helper-effect.test.ts (4 tests) 83ms + ✓ tests/cloud-mirror-ledger.test.ts (7 tests) 1121ms + ✓ the mirror ledger > keeps the newest entries when the count cap bites 1109ms + ✓ tests/scope-preflight.test.ts (6 tests) 8ms + ✓ tests/run-from-digest.test.ts (6 tests) 4580ms + ✓ flows run digest input > submits the sealed canonical spec through the normal journal path without checkout 431ms + ✓ flows run digest input > uses a verified cache hit even after the bucket is removed 453ms + ✓ flows run digest input > resolves deploy.bucket from flows.json and honors explicit override 1216ms + ✓ flows run digest input > refuses an unconfigured bucket 809ms + ✓ flows run digest input > refuses tampered spec.canonical.json before creating run data 810ms + ✓ flows run digest input > refuses tampered identity.json before creating run data 859ms + ✓ tests/build-gate.test.ts (3 tests) 1201ms + ✓ flows build gates on flows check green (#318) > refuses a flow with an unresolvable named-agent CLI and leaves no artifacts 388ms + ✓ flows build gates on flows check green (#318) > --json emits one CheckReport object on stdout on refusal, exits 2, no artifacts 394ms + ✓ flows build gates on flows check green (#318) > builds the bundle on success (regression: gate must not block valid flows) 418ms + ✓ tests/scope-compiler.test.ts (25 tests) 11ms + ✓ tests/worker-cli-abort.test.ts (3 tests) 3748ms + ✓ stops claude and its process group when lease ownership is lost 1365ms + ✓ stops wrapper.mjs and its process group when lease ownership is lost 1350ms + ✓ fails closed when abort cannot prove the process group is gone 1033ms + ✓ tests/hn-poller.test.ts (6 tests) 7ms + ✓ tests/plugin-add.test.ts (7 tests) 1144ms + ✓ typechecks the augmented verb and rejects unknown namespaces 855ms + ✓ tests/authored-step-failed-exit.test.ts (3 tests) 9ms + ✓ tests/run-read-load-live.test.ts (2 tests) 3060ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2431ms + ✓ drains read and watch promises before an authored flow completes 628ms + ✓ tests/retried-step-failure.test.ts (1 test) 904ms + ✓ a retried step failing differently through the live kernel > reports both attempts and says the evidence differs 903ms + ✓ tests/dir-watcher-poller.test.ts (6 tests) 7ms + ✓ tests/subscription-report.test.ts (8 tests) 5ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 9ms + ✓ tests/model-pricing.test.ts (10 tests) 5ms + ✓ tests/authored-use-loader.test.ts (6 tests) 693ms + ✓ tests/yaml-helper-live.test.ts (1 test) 957ms + ✓ runs compiled YAML helpers through the built CLI and kernel effect journal 957ms + ✓ tests/provider-trigger-executor.test.ts (4 tests) 228ms + ✓ tests/journal-client-completion.test.ts (6 tests) 104ms + ✓ tests/transcript-tail-close.test.ts (2 tests) 1303ms + ✓ a stalled transcript-tail close > does not hold the spawn open past its bounded window 647ms + ✓ a stalled tail close beside a transcript that finished > still journals the transcript pointer 654ms + ✓ tests/wrapper-artifacts-cwd.test.ts (2 tests) 123ms + ✓ tests/hello-deterministic.test.ts (5 tests) 17ms + ✓ tests/transcript-exclusion-timeout.test.ts (1 test) 287ms + ✓ tests/cli-adapter.test.ts (4 tests) 5ms + ✓ tests/communication-mixed-resume.test.ts (1 test) 169ms + ✓ tests/work-package-validator.test.ts (7 tests) 5ms + ✓ tests/activity-preflight.test.ts (10 tests) 21ms + ✓ tests/authored-declined-live.test.ts (1 test) 1606ms + ✓ runs an input guard and resumes its completed declined root without repeated effects 1605ms + ✓ tests/cli-answer.test.ts (15 tests) 10ms + ✓ tests/bundle-preflight.test.ts (4 tests) 880ms + ✓ bundle execution preflight > ignores surrounding cache configuration on a verified cache hit 448ms + ✓ bundle execution preflight > uses the built alias for a nameless flow even in a digest-only cache directory 416ms + ✓ tests/agent-timeout-outcome.test.ts (6 tests) 18ms + ✓ tests/agent-relay-hardening.test.ts (12 tests) 16ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 6ms + ✓ tests/communication-preflight.test.ts (13 tests) 31ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 9ms + ↓ tests/real-cli-adapters.test.ts (3 tests | 3 skipped) + ✓ tests/memoization.test.ts (57 tests) 54ms + ✓ tests/shipped-source-call-array-forwarding.test.ts (1 test) 2781ms + ✓ shipped-source call-array forwarding > maps rest indexes and nested same-helper actuals for workers and flows 2780ms + ✓ tests/local-agent-environment.test.ts (8 tests) 6ms + ✓ tests/fs-descriptor.test.ts (1 test) 4ms + ✓ tests/running-step-watch.test.ts (2 tests) 2120ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2117ms + ✓ tests/runtime-dependencies.test.ts (1 test) 210ms + ✓ tests/parse-json-output.test.ts (7 tests) 4ms + ✓ tests/reported-cost.test.ts (4 tests) 3ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 8ms + ✓ tests/authored-model-probe-kinds.test.ts (2 tests) 199ms + ✓ tests/communication-environment-preflight.test.ts (6 tests) 5ms + ✓ tests/budget-authored-live.test.ts (2 tests) 181ms + ✓ tests/slack-writeback.test.ts (1 test) 257ms + ✓ tests/authored-surface-authority.test.ts (2 tests) 19ms + ✓ tests/adapters/claude.test.ts (7 tests) 4ms + ✓ tests/worker-cli-cwd.test.ts (2 tests) 360ms + ✓ runAgentCli — cwd propagation (flows#357) > omits cwd when not provided (inherits parent cwd) 354ms + ✓ tests/adapters/codex.test.ts (7 tests) 5ms + ✓ tests/shipped-source-parameter-provenance.test.ts (1 test) 1630ms + ✓ shipped-source parameter provenance > keeps caller-supplied identifier and destructured keys unknown 1629ms + ✓ tests/slack-block-kit.test.ts (5 tests) 14ms + ✓ tests/agent-timeout.test.ts (23 tests) 35ms + ✓ tests/communication-history.test.ts (1 test) 3ms + ✓ tests/adapters/registry.test.ts (4 tests) 4ms + ✓ tests/authored-declined-report.test.ts (6 tests) 7ms + ✓ tests/promise-ancestry.test.ts (2 tests) 260ms + ✓ tests/agent-cwd-validation.test.ts (2 tests) 397ms + ✓ declarative agent cwd > is refused by `flows check` on a YAML flow before anything runs 394ms + ✓ tests/communication-refusal.test.ts (1 test) 14ms + ✓ tests/bundle-transport.test.ts (20 tests) 2491ms + ✓ digest references > accepts and deploys the build output for hello 463ms + ✓ digest references > accepts and deploys the build output for Hello 413ms + ✓ digest references > accepts and deploys the build output for hello.world 408ms + ✓ digest references > accepts and deploys the build output for hello_world 397ms + ✓ digest references > accepts and deploys the build output for 123 408ms + ✓ digest references > accepts and deploys the build output for A_b.c-1 399ms + ✓ tests/cli-cloud-mirror-flag.test.ts (4 tests) 3ms + ✓ tests/catalog-plugins.test.ts (2 tests) 4ms + ✓ tests/check-command-cwd.test.ts (1 test) 23ms + ✓ tests/communication-lazy.test.ts (1 test) 5ms + ✓ tests/cli-progress-wait.test.ts (2 tests) 4ms + ✓ tests/run-digest-live.test.ts (1 test) 924ms + ✓ executes a deployed digest on the real kernel after deleting the authoring tree 923ms + ✓ tests/run-daemon-unresponsive.test.ts (2 tests) 4ms + ✓ tests/placement.test.ts (54 tests) 18ms + ✓ tests/preflight-run-cache.test.ts (1 test) 5ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 15ms + ✓ tests/canonical-tree.test.ts (1 test) 2ms + ✓ tests/local-agent-stream-selection.test.ts (1 test) 5ms + ✓ tests/run-digest.test.ts (4 tests) 1552ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {invalid json 397ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{}} 383ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{"bucket":123}} 380ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{"bucket":""}} 391ms + ✓ tests/communication-tools.test.ts (1 test) 64ms + ✓ tests/authored-admission.test.ts (2 tests) 3ms + ✓ tests/memory.test.ts (18 tests) 8ms + ✓ tests/worker-platform.test.ts (1 test) 3ms +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs surface /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK @relayflows/surface@2.0.42: package/dist/index.js, package/dist/index.d.ts, package/dist/runtime.js, package/dist/runtime.d.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs sdk /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK @relayflows/sdk@2.0.42: package/dist/index.js, package/dist/index.d.ts, package/dist/cli.js +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs relayflows /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK relayflows@2.0.42: package/bin/flows.js +$ npm install --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-sdk-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 150 packages in 5s +$ /tmp/flows-cli-package-h2FUqF/local/node_modules/.bin/flows check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node /tmp/flows-cli-package-h2FUqF/local/node_modules/relayflows/bin/flows.js check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ npm install -g --prefix /tmp/flows-cli-package-h2FUqF/prefix --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-sdk-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 294 packages in 5s +$ npm install --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 140 packages in 2s +$ /tmp/flows-cli-package-h2FUqF/prefix/bin/flows check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node /tmp/flows-cli-package-h2FUqF/prefix/lib/node_modules/relayflows/bin/flows.js check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +CLI_PACKAGE_OK: local and global installs + ✓ tests/cli-package-gate.test.ts (1 test) 19640ms + ✓ checks authored flows with locally and globally installed release tarballs 19639ms + ✓ tests/event-await-cli.test.ts (1 test) 13231ms + ✓ parks, restarts, and replays two event wakes through the actual CLI 13228ms + ✓ tests/authored-parallel-llm.test.ts (6 tests) 115215ms + ✓ parallel llm capacity 1 > deduplicates concurrent preflight probes 3257ms + ✓ parallel llm capacity 1 > completes nine calls without expired child leases during slow preflight 8921ms + ✓ parallel llm capacity 1 > keeps the durable root lease alive across two cold models 47937ms + ✓ parallel llm capacity 4 > deduplicates concurrent preflight probes 1356ms + ✓ parallel llm capacity 4 > completes nine calls without expired child leases during slow preflight 7251ms + ✓ parallel llm capacity 4 > keeps the durable root lease alive across two cold models 46493ms + ✓ tests/shipped-source-worker-invocations.test.ts (1 test) 756344ms + ✓ shipped-source worker invocation resolution > fails closed across direct, extracted, bound, mutable, and escaped callables 756343ms + +⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-node-runtime.test.ts [ tests/authored-node-runtime.test.ts ] +AssertionError: expected '1.3.6' to be '1.4.0' // Object.is equality + +Expected: "1.4.0" +Received: "1.3.6" + + ❯ tests/authored-node-runtime.test.ts:18:77 + 16| + 17| beforeAll(() => { + 18| expect(spawnSync(bun, ['--version'], { encoding: 'utf8' }).stdout.tr… + | ^ + 19| expect(existsSync(daemon), 'build the current kernel or set RELAYFLO… + 20| stage = mkdtempSync(join(tmpdir(), 'authored-standalone-build-')); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/78]⎯ + +⎯⎯⎯⎯⎯⎯ Failed Tests 77 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > opens the actual catalog flow with the ticket title and exactly one GitHub closing line +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:89:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > writes a Linear closing reference that the GitHub integration links back +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:106:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > accepts a Linear team key that carries digits +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:115:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[4/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier () + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier (not-an-issue) +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:126:28 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[5/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when the Linear closing reference is duplicated in the final body +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:138:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[6/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when the body carries a foreign closing reference +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:150:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[7/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > does not mistake ordinary closing-verb prose for a reference +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:162:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[8/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > normalizes whitespace and caps the title at 240 Unicode code points +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:171:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[9/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when GitHub identity is missing or the final body duplicates its closing line +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ tests/canonical-software-factory.test.ts:180:24 + 178| + 179| it('fails closed before push when GitHub identity is missing or the … + 180| const definition = getFlowDefinition(softwareFactory); + | ^ + 181| const commands: string[] = []; + 182| let completionReason = ''; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[10/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for explicit defects + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for no verdict + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory blocked and unverified + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory passed and unverified + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory passed and blocked + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for empty unverified +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:214:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[11/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > distinguishes a missing verification prerequisite from a defect +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:224:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[12/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > preserves the passed body without a scope marker or draft +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:235:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[13/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects a forged scope for local before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects a forged scope for github before publication +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:243:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[14/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > scopes the post-review hook refusal + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > scopes the merge-gate hook refusal +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:250:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[15/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "" before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "not-a-sha" before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "a'; touch injected; #" before publication +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:259:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[16/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > executes the exact capability-only handler for a queued receipt + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > executes the exact capability-only handler for a duplicate receipt +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:221:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[17/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > launches through the captured process primitive after builtin export synchronization +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:283:11 + 281| input: descriptor(), + 282| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 283| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 284| } finally { + 285| process.execPath = originalExecPath; + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:277:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[18/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > ignores inherited launcher overrides and decodes manifests with the captured Buffer intrinsic +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:373:11 + 371| input: descriptor('delivery-options'), + 372| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 373| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 374| } finally { + 375| Buffer.prototype.toString = bufferToString; + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:367:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[19/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > streams verified bytes when the live store is replaced and no writable staging path exists +AssertionError: promise rejected "Error: Hosted extension sandbox exited wi… { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:431:7 + 429| return { receiptId: 'receipt-1', status: 'queued' }; + 430| } }, + 431| })).resolves.toEqual({ completionReason: 'success', capabilityCall… + | ^ + 432| expect(calls).toBe(1); + 433| expect(readFileSync(join(installed.directory, 'babysitter.flow.ts'… + +Caused by: Error: Hosted extension sandbox exited without a valid completion (exit 1): /tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:20 +if (child.error) throw child.error; + ^ + +Error: spawnSync /usr/bin/bwrap ENOENT + at Object.spawnSync (node:internal/child_process:1103:20) + at spawnSync (node:child_process:911:24) + at Object. (/tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:19:15) + at Module._compile (node:internal/modules/cjs/loader:1809:14) + at Object..js (node:internal/modules/cjs/loader:1940:10) + at Module.load (node:internal/modules/cjs/loader:1530:32) + at Module._load (node:internal/modules/cjs/loader:1332:12) + at wrapModuleLoad (node:internal/modules/cjs/loader:255:19) + at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5) + at node:internal/main/run_main_module:33:47 { + errno: -2, + code: 'ENOENT', + syscall: 'spawnSync /usr/bin/bwrap', + path: '/usr/bin/bwrap', + spawnargs: [ + '--unshare-all', + '--die-with-parent', + '--new-session', + '--clearenv', + '--cap-drop', + 'ALL', + '--dir', + '/usr', + '--ro-bind', + '/usr/lib', + '/usr/lib', + '--ro-bind', + '/usr/lib64', + '/usr/lib64', + '--ro-bind', + '/usr/lib', + '/lib', + '--ro-bind', + '/usr/lib64', + '/lib64', + '--proc', + '/proc', + '--dev', + '/dev', + '--tmpfs', + '/tmp', + '--dir', + '/runtime', + '--dir', + '/extension', + '--dir', + '/extension/node_modules', + '--dir', + '/extension/node_modules/@relayflows', + '--dir', + '/extension/node_modules/@relayflows/surface', + '--dir', + '/extension/node_modules/@relayflows/surface/dist', + '--dir', + '/extension/node_modules/@relayflows/surface/dist/helpers', + '--dir', + '/extension/node_modules/@relayflows/surface/dist/triggers', + '--dir', + '/extension/src', + '--perms', + '0500', + '--ro-bind-data', + '4', + '/runtime/node', + '--perms', + '0400', + '--ro-bind-data', + '5', + '/runtime/runner.mjs', + '--perms', + '0400', + '--ro-bind-data', + '6', + '/extension/node_modules/@relayflows/surface/package.json', + '--perms', + '0400', + '--ro-bind-data', + '7', + '/extension/node_modules/@relayflows/surface/index.js', + '--perms', + '0400', + '--ro-bind-data', + '8', + '/extension/node_modules/@relayflows/surface/runtime.js', + '--perms', + '0400', + '--ro-bind-data', + '9', + '/extension/node_modules/@relayflows/surface/dist/flow.js', + '--perms', + '0400', + '--ro-bind-data', + '10', + '/extension/node_modules/@relayflows/surface/dist/helpers/providers.js', + '--perms', + '0400', + '--ro-bind-data', + '11', + '/extension/node_modules/@relayflows/surface/dist/provider-trigger.js', + '--perms', + '0400', + '--ro-bind-data', + '12', + '/extension/node_modules/@relayflows/surface/dist/schedule.js', + '--perms', + '0400', + '--ro-bind-data', + '13', + '/extension/node_modules/@relayflows/surface/dist/triggers.js', + '--perms', + '0400', + '--ro-bind-data', + '14', + '/extension/node_modules/@relayflows/surface/dist/triggers/github.js', + '--perms', + ... 27 more items + ] +} + +Node.js v25.6.0 + + ❯ Object. ../../../../../../../tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:19:15 + ❯ refuse src/hosted-extension-protocol.ts:135:21 + ❯ ChildProcess. src/hosted-extension-protocol.ts:234:21 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[20/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > mounts pinned private Surface bytes when the live package changes before launch +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:456:7 + 454| return { receiptId: 'receipt-1', status: 'queued' }; + 455| } }, + 456| })).resolves.toEqual({ completionReason: 'success', capabilityCall… + | ^ + 457| expect(calls).toBe(1); + 458| expect(readFileSync(join(surfaceRoot, 'dist/flow.js'), 'utf8')).to… + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:443:18 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[21/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > refuses oversized Surface files through the bounded descriptor reader +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + +- Expected ++ Received + +- Object { ++ PluginError { + "code": "plugin_unsupported", +- "message": StringContaining "cannot read pinned Surface runtime flow.js", + } + + ❯ tests/hosted-extension-isolation.test.ts:489:5 + 487| }); + 488| let calls = 0; + 489| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 490| artifact: await artifact(), + 491| manifest: validateFlowExtensionManifest(manifest()), + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[22/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > shields verified Surface files before async settlement +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:532:9 + 530| surfaceRoot, + 531| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-1'… + 532| })).resolves.toEqual({ completionReason: 'success', capabilityCa… + | ^ + 533| } finally { + 534| if (previous === undefined) delete (Array.prototype as { then?: … + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:525:20 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[23/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > preserves a typed host refusal while disclosing only a fixed marker to the child +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to be Error: private Cloud policy detail { code: '…' } // Object.is equality + +- Expected ++ Received + +- [Error: private Cloud policy detail] ++ [Error: bubblewrap is unavailable] + + ❯ tests/hosted-extension-isolation.test.ts:560:5 + 558| provider: 'github', eventType: 'pull_request.labeled', deliveryI… + 559| }); + 560| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 561| artifact: await artifact(source), manifest: validateFlowExtensio… + 562| babysitterTurn: { queue: async () => { throw refusal; } }, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[24/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > denies ambient credentials, host files, writes, network, subprocesses, and undeclared context verbs +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:624:13 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[25/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > enforces OS address-space and data bounds on native Buffer allocation +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + +- Expected ++ Received + +- Object { ++ PluginError { + "code": "plugin_unsupported", +- "message": StringMatching /(?:Failed to allocate memory|Array buffer allocation failed)/u, + } + + ❯ tests/hosted-extension-isolation.test.ts:646:5 + 644| }); + 645| let calls = 0; + 646| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 647| artifact: installed, + 648| manifest: validateFlowExtensionManifest(manifest()), + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[26/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > blocks extra handler fields and authority-bearing receipt fields at the parent port +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + +- Expected ++ Received + +- Object { +- "code": "plugin_event_unroutable", ++ PluginError { ++ "code": "plugin_unsupported", + } + + ❯ tests/hosted-extension-isolation.test.ts:675:5 + 673| }); + 674| let calls = 0; + 675| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 676| artifact: await artifact(source), manifest: validateFlowExtensio… + 677| babysitterTurn: { queue: async () => { calls += 1; return { rece… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[27/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > constructs adapter authority with the captured freeze intrinsic +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:745:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[28/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > writes the Surface manifest and protocol without inherited toJSON behavior +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:788:11 + 786| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 787| timeoutMs: 3_000, + 788| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 789| } finally { + 790| if (previous === undefined) delete (Object.prototype as { toJS… + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:781:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[29/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > uses captured JSON intrinsics for the complete parent boundary +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-protocol.test.ts:326:24 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[30/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different PR frame with zero adapter calls + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different delivery frame with zero adapter calls + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different event frame with zero adapter calls +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + +- Expected ++ Received + +- Object { +- "code": "plugin_event_unroutable", ++ PluginError { ++ "code": "plugin_unsupported", + } + + ❯ tests/hosted-extension-protocol.test.ts:447:5 + 445| ])('rejects an import-time %s frame with zero adapter calls', async … + 446| let calls = 0; + 447| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 448| artifact: await artifact(hostileImport([frame, { type: 'error', … + 449| manifest: validateFlowExtensionManifest(manifest()), dispatch: d… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[31/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects two forged calls after the authoritative first outcome settles + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > waits for a pending adapter to reject after a forged child error + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > waits for a pending adapter to resolve after a forged child error + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > returns a typed adapter rejection even when the hostile child hangs +Error: hostile child did not invoke the adapter + ❯ Timeout._onTimeout tests/hosted-extension-protocol.test.ts:135:45 + 133| async function waitForInvocation(invoked: Promise): Promise((resolve, reject) => { + 135| const timeout = setTimeout(() => reject(new Error('hostile child d… + | ^ + 136| void invoked.then(() => { clearTimeout(timeout); resolve(); }, rej… + 137| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[32/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > runs hn-monitor analyze-story end-to-end via a stub agent CLI (gate 2 clause 2 demo) +AssertionError: expected { …(12) } to match object { output: { …(3) }, …(1) } +(22 matching properties omitted from actual) + +- Expected ++ Received + + Object { +- "output": Object { +- "reasoning": "stub agent runtime — deterministic output for gate-2 clause-2 demo", +- "relevance_score": 5, +- "story_title": "stub", +- }, ++ "output": null, + "verification": Object { +- "gate": "json_schema", +- "verdict": "pass", ++ "gate": "execution", ++ "verdict": "fail", + }, + } + + ❯ tests/live-kernel.test.ts:657:36 + 655| && (entry as { step_id?: string }).step_id === 'analyze-story', + 656| ) as { payload: { output: unknown; verification: unknown } } | und… + 657| expect(stepCompleted?.payload).toMatchObject({ + | ^ + 658| output: { + 659| story_title: 'stub', + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[33/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story FAILS verification when the CLI omits required schema fields +AssertionError: expected { …(12) } to match object { …(3) } +(21 matching properties omitted from actual) + +- Expected ++ Received + + Object { +- "completionReason": "retries_exhausted", ++ "completionReason": "worker_error", + "output": null, + "verification": Object { +- "gate": "json_schema", ++ "gate": "execution", + "verdict": "fail", + }, + } + + ❯ tests/live-kernel.test.ts:752:36 + 750| // its verification record names the json_schema rejection. The re… + 751| // parsed value is nulled before the completion is persisted. + 752| expect(stepCompleted?.payload).toMatchObject({ + | ^ + 753| completionReason: 'retries_exhausted', + 754| output: null, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[34/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > agent step preserves the CliResult wrapper as output when the CLI emits non-JSON text +AssertionError: expected null not to be null + ❯ tests/live-kernel.test.ts:823:24 + 821| // here (parseJsonOutput returned null on non-JSON stdout) and + 822| // these assertions would all fail. + 823| expect(output).not.toBeNull(); + | ^ + 824| expect(output.exit_code).toBe(0); + 825| expect(output.stdout_tail).toContain('looked at the story'); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[35/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker exposes wake_context to the CLI via RELAYFLOW_WAKE_CONTEXT env var (real analyzer prerequisite) +TypeError: Cannot read properties of null (reading 'story_title') + ❯ tests/live-kernel.test.ts:891:42 + 889| ) as { payload: { output: { story_title: string; reasoning: string… + 890| expect(stepCompleted).toBeDefined(); + 891| expect(stepCompleted!.payload.output.story_title).toBe(`echoed:${s… + | ^ + 892| expect(stepCompleted!.payload.output.reasoning).toContain(String(s… + 893| + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[36/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_WAKE_CONTEXT UNSET when the run has no wake_context (undefined-vs-null pin) +TypeError: Cannot read properties of null (reading 'env_present') + ❯ tests/live-kernel.test.ts:958:38 + 956| ) as { payload: { output: { env_present: boolean } } } | undefined; + 957| expect(completed).toBeDefined(); + 958| expect(completed!.payload.output.env_present).toBe(false); + | ^ + 959| + 960| delete process.env.RELAYFLOW_WAKE_CONTEXT; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[37/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_MODEL UNSET when the step declares no model +TypeError: Cannot read properties of null (reading 'story_title') + ❯ tests/live-kernel.test.ts:1194:38 + 1192| expect(completed).toBeDefined(); + 1193| // UNSET, not EMPTY and not the leaked parent value. + 1194| expect(completed!.payload.output.story_title).toBe('model:UNSET'); + | ^ + 1195| + 1196| delete process.env.RELAYFLOW_MODEL; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[38/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI +Error: LIVE_ANALYZER_UNAVAILABLE: "/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-claude-cli" does not identify as relayflows-agent-cli-v1 — failing because gate-2 acceptance requires the real analyzer to execute. Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is not gate evidence. + ❯ tests/live-kernel.test.ts:1223:15 + 1221| const notice = `LIVE_ANALYZER_UNAVAILABLE: ${readiness.detail}`; + 1222| if (process.env['RELAYFLOWS_ALLOW_ANALYZER_SKIP'] !== '1') { + 1223| throw new Error( + | ^ + 1224| `${notice} — failing because gate-2 acceptance requires the … + 1225| + 'Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is … + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[39/78]⎯ + + FAIL tests/live-kernel.test.ts > a relayflow can be scheduled: tick source against live relayflowd > a tick spawns a real run whose step reports the SCHEDULED instant +AssertionError: expected null to deeply equal { schedule_id: 'heartbeat-1m', …(3) } + +- Expected: +Object { + "lag_ms": 43000, + "schedule_id": "heartbeat-1m", + "scheduled_for_ms": 1764000000000, + "slot": 29400000, +} + ++ Received: +null + + ❯ tests/live-kernel.test.ts:1739:39 + 1737| // The bound: the run reports the grid instant and its own lag, so… + 1738| // backfilled run can tell it is running for a slot from the past. + 1739| expect(completed!.payload.output).toEqual({ + | ^ + 1740| schedule_id: 'heartbeat-1m', + 1741| slot: 29_400_000, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[40/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses an 8-character run-id prefix: Cloud has no prefix lookup +AssertionError: expected [Function] to throw error matching /not full Cloud run ids: c649fe14/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/not full Cloud run ids: c649fe14/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[41/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses the whole batch when any id is invalid, rather than dropping it +AssertionError: expected [Function] to throw error matching /not full Cloud run ids: nope!/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/not full Cloud run ids: nope!/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[42/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses an empty batch +AssertionError: expected [Function] to throw error matching /needs runIds/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/needs runIds/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[43/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses a batch too large for the edge step lease +AssertionError: expected [Function] to throw error matching /exceeds the 8 that fit/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/exceeds the 8 that fit/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[44/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > accepts eight ids — the incident batch is inside the bound +AssertionError: promise rejected "TypeError: expected an @relayflows/surfac…" instead of resolving + ❯ tests/stuck-run-triage.test.ts:62:40 + 60| it('accepts eight ids — the incident batch is inside the bound', asy… + 61| const ids = Array.from({ length: 8 }, (_, i) => `${ID_A.slice(0, -… + 62| await expect(drive({ runIds: ids })).resolves.toBeDefined(); + | ^ + 63| }); + 64| }); + +Caused by: TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + ❯ tests/stuck-run-triage.test.ts:62:18 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[45/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > refuses to send the Cloud bearer token to an unapproved origin +AssertionError: expected [Function] to throw error matching /refusing to send the Cloud bearer to…/\ but got 'expected an @relayflows/surface flow …' + +- Expected: +/refusing to send the Cloud bearer token to https:\/\/evil\.example/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[46/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > refuses a non-URL apiUrl +AssertionError: expected [Function] to throw error matching /is not a URL/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/is not a URL/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[47/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > allows an approved origin and uses it in the curl +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:77:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[48/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > defaults to production Cloud +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:82:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[49/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > never publishes a run record the fetch did not produce +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:89:34 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[50/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > names the Worker on every wrangler invocation +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:98:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[51/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > accepts caller-supplied Workers and rejects option-shaped ones +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:107:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[52/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > falls back when GNU timeout is absent, as it is on macOS +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:115:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[53/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > runs the tails concurrently so wall time does not scale with the batch +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:123:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[54/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > records wrangler's own exit status rather than head's +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:129:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[55/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage shell text > parses under both sh and bash +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:137:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[56/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage shell text > collects tails with no GNU timeout on PATH, as on a stock macOS +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:157:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[57/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage agents > declares read-only permissions on every agent +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:176:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[58/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage agents > tells the forensics agents their evidence is untrusted +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:182:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[59/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > refuses a duplicate run id: two tails would share one evidence file +AssertionError: expected [Function] to throw error matching /duplicate runIds: c649fe14-0c2e-4e51-…/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/duplicate runIds: c649fe14-0c2e-4e51-9a6a-4f0d1b0f77aa/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[60/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > refuses a duplicate Worker name for the same reason +AssertionError: expected [Function] to throw error matching /duplicate workers: w-one/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/duplicate workers: w-one/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[61/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > bounds ids x workers, not just ids +AssertionError: expected [Function] to throw error matching /24 concurrent tails, over the 16/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/24 concurrent tails, over the 16/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[62/78]⎯ + +⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯ + +Vitest caught 1 unhandled error during the test run. +This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected. + +⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯ +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-protocol.test.ts:471:17 + ❯ node_modules/@vitest/runner/dist/index.js:533:5 + ❯ runTest node_modules/@vitest/runner/dist/index.js:1056:11 + ❯ runSuite node_modules/@vitest/runner/dist/index.js:1205:15 + ❯ runSuite node_modules/@vitest/runner/dist/index.js:1205:15 + ❯ runFiles node_modules/@vitest/runner/dist/index.js:1262:5 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +This error originated in "tests/hosted-extension-protocol.test.ts" test file. It doesn't mean the error was thrown inside the file itself, but while it was running. +The latest test that might've caused the error is "rejects two forged calls after the authoritative first outcome settles". It might mean one of the following: +- The error was thrown, while Vitest was running this test. +- If the error occurred after the test had been completed, this was the last documented test before it was thrown. +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ + + Test Files 6 failed | 248 passed | 1 skipped (255) + Tests 77 failed | 3774 passed | 22 skipped (3873) + Errors 1 error + Start at 10:40:22 + Duration 757.43s (transform 4.18s, setup 1.09s, collect 64.50s, tests 1765.01s, environment 32ms, prepare 10.44s) + +exit=1 diff --git a/evidence/run-read-timeout/handshake-probe-baseline.mjs b/evidence/run-read-timeout/handshake-probe-baseline.mjs new file mode 100644 index 000000000..eb71885c3 --- /dev/null +++ b/evidence/run-read-timeout/handshake-probe-baseline.mjs @@ -0,0 +1,25 @@ +import {mkdtempSync,readFileSync,rmSync} from 'node:fs'; +import {join} from 'node:path'; +import {tmpdir} from 'node:os'; +import {spawn} from 'node:child_process'; +const root='/home/daytona/.relayflow-v2-supervisor/durable/repository'; +const {JournalClient}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/journal-client.js'); +const {AgentWorker}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/worker.js'); +const {socketPathFor}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/daemon-connection.js'); +const data=mkdtempSync(join(tmpdir(),'rf-read-probe-')); +const daemon=spawn('/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd',['--data-dir',data,'serve'],{stdio:'ignore'}); +let client,worker; +try { + for(let i=0;i<100;i++) { + client=new JournalClient(socketPathFor(data)); + try {await client.connect();await client.hello('probe');break;} catch {client.close();await new Promise(r=>setTimeout(r,20));} + } + worker=new AgentWorker(client,{workerId:'probe',pins:{workspace:[{surface:'repo',revision_id:'rev-a'}],streams:[]}}); + await worker.attach(); + const spec=JSON.parse(readFileSync(root+'/testdata/hn-monitor.spec.canonical.json','utf8')); + for(const step of spec.steps) if(step.id==='analyze-story')step.cli=root+'/testdata/preflight/analyze-story-stub-cli'; + const outcome=await client.eventSubmit(spec,{type:'hn.story_posted',payload:{id:42000042,type:'story'}}); + for(let i=0;i<100;i++){if((await client.runGet(outcome.run.run_id)).steps['analyze-story'].state==='done')break;await new Promise(r=>setTimeout(r,50));} + const {entries}=await client.journalRead(outcome.run.run_id,1,1000); + console.log(JSON.stringify(entries.filter(e=>e.entry_type==='step.completed'),null,2)); +}finally{await worker?.close();client?.close();daemon.kill('SIGTERM');await new Promise(r=>daemon.once('exit',r));rmSync(data,{recursive:true,force:true});} diff --git a/evidence/run-read-timeout/handshake-probe.mjs b/evidence/run-read-timeout/handshake-probe.mjs new file mode 100644 index 000000000..60241ad88 --- /dev/null +++ b/evidence/run-read-timeout/handshake-probe.mjs @@ -0,0 +1,25 @@ +import {mkdtempSync,readFileSync,rmSync} from 'node:fs'; +import {join} from 'node:path'; +import {tmpdir} from 'node:os'; +import {spawn} from 'node:child_process'; +const root='/home/daytona/.relayflow-v2-supervisor/durable/repository'; +const {JournalClient}=await import(root+'/packages/sdk/dist/journal-client.js'); +const {AgentWorker}=await import(root+'/packages/sdk/dist/worker.js'); +const {socketPathFor}=await import(root+'/packages/sdk/dist/daemon-connection.js'); +const data=mkdtempSync(join(tmpdir(),'rf-read-probe-')); +const daemon=spawn('/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd',['--data-dir',data,'serve'],{stdio:'ignore'}); +let client,worker; +try { + for(let i=0;i<100;i++) { + client=new JournalClient(socketPathFor(data)); + try {await client.connect();await client.hello('probe');break;} catch {client.close();await new Promise(r=>setTimeout(r,20));} + } + worker=new AgentWorker(client,{workerId:'probe',pins:{workspace:[{surface:'repo',revision_id:'rev-a'}],streams:[]}}); + await worker.attach(); + const spec=JSON.parse(readFileSync(root+'/testdata/hn-monitor.spec.canonical.json','utf8')); + for(const step of spec.steps) if(step.id==='analyze-story')step.cli=root+'/testdata/preflight/analyze-story-stub-cli'; + const outcome=await client.eventSubmit(spec,{type:'hn.story_posted',payload:{id:42000042,type:'story'}}); + for(let i=0;i<100;i++){if((await client.runGet(outcome.run.run_id)).steps['analyze-story'].state==='done')break;await new Promise(r=>setTimeout(r,50));} + const {entries}=await client.journalRead(outcome.run.run_id,1,1000); + console.log(JSON.stringify(entries.filter(e=>e.entry_type==='step.completed'),null,2)); +}finally{await worker?.close();client?.close();daemon.kill('SIGTERM');await new Promise(r=>daemon.once('exit',r));rmSync(data,{recursive:true,force:true});} diff --git a/evidence/run-read-timeout/kernel-build.log b/evidence/run-read-timeout/kernel-build.log new file mode 100644 index 000000000..6b859fb94 --- /dev/null +++ b/evidence/run-read-timeout/kernel-build.log @@ -0,0 +1,3 @@ +$ cd kernel && CARGO_TARGET_DIR=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target cargo build + Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s +exit=0 diff --git a/evidence/run-read-timeout/live-failure-probe-baseline.log b/evidence/run-read-timeout/live-failure-probe-baseline.log new file mode 100644 index 000000000..947cc0382 --- /dev/null +++ b/evidence/run-read-timeout/live-failure-probe-baseline.log @@ -0,0 +1,45 @@ +$ node /tmp/relayflow-read-probe-baseline.mjs +[ + { + "at_ms": 1791197583957, + "attempt": 1, + "entry_type": "step.completed", + "payload": { + "budget": { + "dollars": "0", + "dollars_unmetered": true, + "tokens_in": 0, + "tokens_out": 0 + }, + "completed_by": "probe", + "completionReason": "worker_error", + "disposition": "step_done", + "effects": [], + "end_pins": { + "streams": [], + "workspace": [] + }, + "input_hash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "next_attempt_at_ms": null, + "output": null, + "spend": { + "dollars": 0, + "dollars_unmetered": true, + "tokens_input": 0, + "tokens_output": 0, + "wallclock_ms": 888 + }, + "step_spec_hash": "12d9633edd0ffd882a0b1586fc347ce36c84d32656fe82f7e0b8d9d97cecc087", + "verification": { + "detail": "{\"artifacts\":[],\"exit_code\":null,\"stderr_tail\":\"CLI \\\"/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-stub-cli\\\" exited before completing the relayflows-agent-cli-v1 same-process handshake.\",\"stdout_tail\":\"\"}", + "gate": "execution", + "verdict": "fail" + } + }, + "run_id": "01M45V45PHWH45AD8128KRDAR7", + "segment_id": 1, + "seq": 7, + "step_id": "analyze-story" + } +] +exit=0 diff --git a/evidence/run-read-timeout/live-failure-probe.log b/evidence/run-read-timeout/live-failure-probe.log new file mode 100644 index 000000000..7d969c681 --- /dev/null +++ b/evidence/run-read-timeout/live-failure-probe.log @@ -0,0 +1,45 @@ +$ node /tmp/relayflow-read-probe.mjs +[ + { + "at_ms": 1791197554045, + "attempt": 1, + "entry_type": "step.completed", + "payload": { + "budget": { + "dollars": "0", + "dollars_unmetered": true, + "tokens_in": 0, + "tokens_out": 0 + }, + "completed_by": "probe", + "completionReason": "worker_error", + "disposition": "step_done", + "effects": [], + "end_pins": { + "streams": [], + "workspace": [] + }, + "input_hash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "next_attempt_at_ms": null, + "output": null, + "spend": { + "dollars": 0, + "dollars_unmetered": true, + "tokens_input": 0, + "tokens_output": 0, + "wallclock_ms": 723 + }, + "step_spec_hash": "12d9633edd0ffd882a0b1586fc347ce36c84d32656fe82f7e0b8d9d97cecc087", + "verification": { + "detail": "{\"artifacts\":[],\"exit_code\":null,\"stderr_tail\":\"CLI \\\"/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-stub-cli\\\" exited before completing the relayflows-agent-cli-v1 same-process handshake.\",\"stdout_tail\":\"\"}", + "gate": "execution", + "verdict": "fail" + } + }, + "run_id": "01M45V38MZYZ0F87JWZVAJBZ6F", + "segment_id": 1, + "seq": 7, + "step_id": "analyze-story" + } +] +exit=0 diff --git a/evidence/run-read-timeout/mutation-completion-aborts-snapshot.log b/evidence/run-read-timeout/mutation-completion-aborts-snapshot.log new file mode 100644 index 000000000..437214d3e --- /dev/null +++ b/evidence/run-read-timeout/mutation-completion-aborts-snapshot.log @@ -0,0 +1,55 @@ +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +snapshotRead?.abort(new Error(`step "${runningStep.id}" completed during the lease snapshot`)); +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'a completion push ends the wait' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 3014ms + × a completion push ends the wait without waiting out an in-flight snapshot 3013ms + → expected 'still waiting' to be 'returned' // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > a completion push ends the wait without waiting out an in-flight snapshot +AssertionError: expected 'still waiting' to be 'returned' // Object.is equality + +Expected: "returned" +Received: "still waiting" + + ❯ tests/running-step-watch.test.ts:152:21 + 150| const waiting = waitForRunningStep(client, 'run', { id: 'step', ty… + 151| const outcome = await Promise.race([waiting.then(() => 'returned')… + 152| expect(outcome).toBe('returned'); + | ^ + 153| expect(await client.runGet('run', { signal: AbortSignal.timeout(50… + 154| } finally { client.close(); await new Promise(resolve => serve… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:29 + Duration 3.51s (transform 191ms, setup 16ms, collect 329ms, tests 3.01s, environment 0ms, prepare 41ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'a completion push ends the wait' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2062ms + ✓ a completion push ends the wait without waiting out an in-flight snapshot 2061ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:33 + Duration 2.57s (transform 204ms, setup 17ms, collect 350ms, tests 2.06s, environment 0ms, prepare 41ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-heartbeat.log b/evidence/run-read-timeout/mutation-heartbeat.log new file mode 100644 index 000000000..e94f0a2c0 --- /dev/null +++ b/evidence/run-read-timeout/mutation-heartbeat.log @@ -0,0 +1,53 @@ +Source: packages/sdk/src/worker-lease.ts +Original SHA256: 4c58d4670422dc2169764af16eb082c55e72c5321ed9d6001af3607f03955fa3 +Replaced: + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/heartbeat-timeout.test.ts (1 test | 1 failed) 19ms + × a heartbeat timeout is lease loss rather than a worker body failure 18ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/heartbeat-timeout.test.ts > a heartbeat timeout is lease loss rather than a worker body failure +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + ❯ tests/heartbeat-timeout.test.ts:16:5 + 14| try { + 15| await client.connect(); + 16| await expect(withWorkerLease(client, { + | ^ + 17| run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', … + 18| lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed (1) + Start at 06:16:36 + Duration 460ms (transform 168ms, setup 17ms, collect 294ms, tests 19ms, environment 0ms, prepare 42ms) + +exit=1 + +RESTORED SHA256: 4c58d4670422dc2169764af16eb082c55e72c5321ed9d6001af3607f03955fa3 +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/heartbeat-timeout.test.ts (1 test) 16ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 06:16:37 + Duration 435ms (transform 164ms, setup 16ms, collect 277ms, tests 16ms, environment 0ms, prepare 32ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-reader-reconnect.log b/evidence/run-read-timeout/mutation-reader-reconnect.log new file mode 100644 index 000000000..75cf57455 --- /dev/null +++ b/evidence/run-read-timeout/mutation-reader-reconnect.log @@ -0,0 +1,61 @@ +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: + this.drop(reader); + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'reconnects the reader after it disconnects' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 1007ms + × reconnects the reader after it disconnects within the read budget 1006ms + → journal client: run.get read session was interrupted after 76 attempts in 1000ms (read budget 1000ms): journal client: not connected (run.get): journal client: connection closed + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > reconnects the reader after it disconnects within the read budget +JournalReadInterruptedError: journal client: run.get read session was interrupted after 76 attempts in 1000ms (read budget 1000ms): journal client: not connected (run.get): journal client: connection closed + ❯ exhausted src/journal-read-policy.ts:75:9 + 73| let last: unknown; + 74| const exhausted = () => last instanceof JournalReadInterruptedError + 75| ? new JournalReadInterruptedError(verb, attempts, performance.no… + | ^ + 76| : new JournalRequestTimeoutError(verb, timeoutMs, attempts, perf… + 77| // A caller's cancellation settles this read and drains its queued… + ❯ Timeout. src/journal-read-policy.ts:90:48 + +Caused by: Error: journal client: not connected (run.get): journal client: connection closed + ❯ src/journal-connection.ts:183:16 + ❯ JournalClient.requestOnce src/journal-connection.ts:176:12 + ❯ src/journal-budgeted-reads.ts:60:29 + ❯ src/journal-read-policy.ts:102:18 + +Caused by: Error: journal client: connection closed + ❯ Socket. src/journal-connection.ts:89:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:11 + Duration 1.49s (transform 189ms, setup 18ms, collect 324ms, tests 1.01s, environment 0ms, prepare 46ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'reconnects the reader after it disconnects' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 91ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:13 + Duration 559ms (transform 180ms, setup 18ms, collect 295ms, tests 91ms, environment 0ms, prepare 45ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-reader-setup-retry.log b/evidence/run-read-timeout/mutation-reader-setup-retry.log new file mode 100644 index 000000000..e72403820 --- /dev/null +++ b/evidence/run-read-timeout/mutation-reader-setup-retry.log @@ -0,0 +1,63 @@ +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: + if (this.reader === reader) { + this.reader = undefined; + this.readerReady = undefined; + } + throw error; + +With: + return undefined; + + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'retries a reader setup that timed out' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 84ms + × retries a reader setup that timed out instead of reading on the primary for good 83ms + → expected 1 to be 2 // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > retries a reader setup that timed out instead of reading on the primary for good +AssertionError: expected 1 to be 2 // Object.is equality + +- Expected ++ Received + +- 2 ++ 1 + + ❯ tests/journal-client-read-timeout.test.ts:203:18 + 201| expect(await client.runGet('run')).toMatchObject({ status: 'complete… + 202| expect(await client.runGet('run')).toMatchObject({ status: 'complete… + 203| expect(hellos).toBe(2); + | ^ + 204| expect(served).toHaveLength(2); + 205| expect(served.every(socket => socket !== primary)).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:14 + Duration 535ms (transform 173ms, setup 17ms, collect 299ms, tests 84ms, environment 0ms, prepare 42ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'retries a reader setup that timed out' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 74ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:15 + Duration 534ms (transform 180ms, setup 17ms, collect 303ms, tests 74ms, environment 0ms, prepare 41ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-reader.log b/evidence/run-read-timeout/mutation-reader.log new file mode 100644 index 000000000..c4822afb2 --- /dev/null +++ b/evidence/run-read-timeout/mutation-reader.log @@ -0,0 +1,57 @@ +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: +return await reader.requestOnce(verb, params, remaining, attemptSignal); +With: +return await this.primary.requestOnce(verb, params, remaining, attemptSignal); + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 113ms + × serves a bounded read while an unbounded command is in flight 112ms + → expected true to be false // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > serves a bounded read while an unbounded command is in flight +AssertionError: expected true to be false // Object.is equality + +- Expected ++ Received + +- false ++ true + + ❯ tests/journal-client-read-timeout.test.ts:38:23 + 36| await inFlight; + 37| expect(await client.runGet('run')).toMatchObject({ status: 'running'… + 38| expect(commandDone).toBe(false); + | ^ + 39| await command; + 40| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:09 + Duration 589ms (transform 185ms, setup 18ms, collect 300ms, tests 113ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 106ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:10 + Duration 542ms (transform 172ms, setup 15ms, collect 309ms, tests 106ms, environment 0ms, prepare 28ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-root-parking.log b/evidence/run-read-timeout/mutation-root-parking.log new file mode 100644 index 000000000..342ecd71d --- /dev/null +++ b/evidence/run-read-timeout/mutation-root-parking.log @@ -0,0 +1,124 @@ +Source: packages/sdk/src/authored-root.ts +Original SHA256: 8e3bd4ebe8b470c1486f1ab6adcf2fd81ba19e6a173e2531e7cbc59b16bcba29 +Replaced: + if (isReadInterruptionError(error) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/authored-root.test.ts (36 tests | 3 failed | 33 skipped) 31ms + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=false) 22ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=true) 4ms + → expected AuthoredFlowExecutionError: daemon_unresp… { …(7) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + × durable authored root > leaves the root resumable when the read session disconnects while the root worker is healthy 4ms + → expected JournalReadInterruptedError: journal clie… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 3 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=false) +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalRequestTimeoutError: journal client: run.get timed out after 10ms] + + ❯ tests/authored-root.test.ts:424:5 + 422| }); + 423| const journal = new RootJournal(); + 424| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 425| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 426| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/3]⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=true) +AssertionError: expected AuthoredFlowExecutionError: daemon_unresp… { …(7) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { ++ AuthoredFlowExecutionError { + "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", + "rootRunId": "root-run", + } + + ❯ tests/authored-root.test.ts:424:5 + 422| }); + 423| const journal = new RootJournal(); + 424| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 425| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 426| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/3]⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable when the read session disconnects while the root worker is healthy +AssertionError: expected JournalReadInterruptedError: journal clie… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalReadInterruptedError: journal client: run.get read session was interrupted after 2 attempts in 40ms (read budget 300000ms): journal client: connection closed] + + ❯ tests/authored-root.test.ts:437:5 + 435| const loaded = await fixture(false, 0, async () => { throw interru… + 436| const journal = new RootJournal(); + 437| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 438| { dataDir: '/unused', admissionKey: 'read-disconnect' })).reject… + 439| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/3]⎯ + + Test Files 1 failed (1) + Tests 3 failed | 33 skipped (36) + Start at 06:16:37 + Duration 1.33s (transform 681ms, setup 15ms, collect 1.12s, tests 31ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: 8e3bd4ebe8b470c1486f1ab6adcf2fd81ba19e6a173e2531e7cbc59b16bcba29 +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/authored-root.test.ts (36 tests | 33 skipped) 23ms + + Test Files 1 passed (1) + Tests 3 passed | 33 skipped (36) + Start at 06:16:39 + Duration 1.29s (transform 674ms, setup 14ms, collect 1.11s, tests 23ms, environment 0ms, prepare 41ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-snapshot-cancel.log b/evidence/run-read-timeout/mutation-snapshot-cancel.log new file mode 100644 index 000000000..d94ca66ae --- /dev/null +++ b/evidence/run-read-timeout/mutation-snapshot-cancel.log @@ -0,0 +1,52 @@ +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +const signal = options.signal === undefined ? read.signal + : AbortSignal.any([options.signal, read.signal]); +With: +const signal = read.signal; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'cancels promptly while a lease snapshot read is in flight' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 2224ms + × cancels promptly while a lease snapshot read is in flight 2223ms + → expected 'still waiting' to be an instance of Error + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > cancels promptly while a lease snapshot read is in flight +AssertionError: expected 'still waiting' to be an instance of Error + ❯ tests/running-step-watch.test.ts:83:21 + 81| controller.abort(); + 82| const settled = await Promise.race([outcome, sleep(200).then(() =>… + 83| expect(settled).toBeInstanceOf(Error); + | ^ + 84| expect((settled as Error).message).toContain('was canceled'); + 85| } finally { client.close(); await new Promise(resolve => serve… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:22 + Duration 3.35s (transform 482ms, setup 30ms, collect 848ms, tests 2.22s, environment 0ms, prepare 75ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'cancels promptly while a lease snapshot read is in flight' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2009ms + ✓ cancels promptly while a lease snapshot read is in flight 2008ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:26 + Duration 2.60s (transform 223ms, setup 20ms, collect 393ms, tests 2.01s, environment 0ms, prepare 50ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-watch-cadence.log b/evidence/run-read-timeout/mutation-watch-cadence.log new file mode 100644 index 000000000..d4d5dc347 --- /dev/null +++ b/evidence/run-read-timeout/mutation-watch-cadence.log @@ -0,0 +1,51 @@ +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +const LEASE_POLL_MS = 2_000; +With: +const LEASE_POLL_MS = 50; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 2109ms + × uses pushes for completion with lease-cadence reads and releases its watcher 2108ms + → expected 41 to be less than or equal to 1 + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > uses pushes for completion with lease-cadence reads and releases its watcher +AssertionError: expected 41 to be less than or equal to 1 + ❯ tests/running-step-watch.test.ts:39:19 + 37| await waitForRunningStep(client, 'run', { id: 'step', type: 'agent… + 38| expect(performance.now() - start).toBeGreaterThan(2000); + 39| expect(reads).toBeLessThanOrEqual(1); + | ^ + 40| await sleep(10); + 41| expect(watchClosed).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:16 + Duration 2.59s (transform 194ms, setup 16ms, collect 322ms, tests 2.11s, environment 0ms, prepare 41ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2117ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2116ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:19 + Duration 2.56s (transform 186ms, setup 14ms, collect 298ms, tests 2.12s, environment 0ms, prepare 33ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutations.py b/evidence/run-read-timeout/mutations.py new file mode 100644 index 000000000..f1ac34ac3 --- /dev/null +++ b/evidence/run-read-timeout/mutations.py @@ -0,0 +1,76 @@ +"""Reproduce the reviewed mutations against the current source; always restore source bytes.""" +from pathlib import Path +import hashlib +import shlex +import subprocess + +ROOT = Path(__file__).resolve().parents[2] +SDK = ROOT / 'packages/sdk' +CASES = [ + ('reader', 'packages/sdk/src/journal-budgeted-reads.ts', + 'return await reader.requestOnce(verb, params, remaining, attemptSignal);', + 'return await this.primary.requestOnce(verb, params, remaining, attemptSignal);', + 'tests/journal-client-read-timeout.test.ts', 'serves a bounded read'), + ('reader-reconnect', 'packages/sdk/src/journal-budgeted-reads.ts', + """ this.drop(reader); +""", '', 'tests/journal-client-read-timeout.test.ts', 'reconnects the reader after it disconnects'), + ('reader-setup-retry', 'packages/sdk/src/journal-budgeted-reads.ts', + """ if (this.reader === reader) { + this.reader = undefined; + this.readerReady = undefined; + } + throw error; +""", """ return undefined; +""", 'tests/journal-client-read-timeout.test.ts', 'retries a reader setup that timed out'), + ('watch-cadence', 'packages/sdk/src/cli/running-step.ts', + 'const LEASE_POLL_MS = 2_000;', 'const LEASE_POLL_MS = 50;', + 'tests/running-step-watch.test.ts', 'uses pushes for completion'), + ('snapshot-cancel', 'packages/sdk/src/cli/running-step.ts', + 'const signal = options.signal === undefined ? read.signal\n : AbortSignal.any([options.signal, read.signal]);', + 'const signal = read.signal;', + 'tests/running-step-watch.test.ts', 'cancels promptly while a lease snapshot read is in flight'), + ('completion-aborts-snapshot', 'packages/sdk/src/cli/running-step.ts', + 'snapshotRead?.abort(new Error(`step "${runningStep.id}" completed during the lease snapshot`));', + '', + 'tests/running-step-watch.test.ts', 'a completion push ends the wait'), + ('heartbeat', 'packages/sdk/src/worker-lease.ts', + """ if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } +""", '', 'tests/heartbeat-timeout.test.ts', 'a heartbeat timeout'), + ('root-parking', 'packages/sdk/src/authored-root.ts', + """ if (isReadInterruptionError(error) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } +""", '', 'tests/authored-root.test.ts', 'leaves the root resumable'), +] + +for name, source, old, new, test, pattern in CASES: + path = ROOT / source + original = path.read_bytes() + assert original.count(old.encode()) == 1, (name, 'mutation must match once') + digest = hashlib.sha256(original).hexdigest() + command = ['npx', 'vitest', 'run', test, '-t', pattern, '--maxWorkers=1', '--minWorkers=1'] + log = ROOT / 'evidence/run-read-timeout' / f'mutation-{name}.log' + with log.open('w') as output: + output.write(f'Source: {source}\nOriginal SHA256: {digest}\nReplaced:\n{old}\nWith:\n{new}\n') + try: + path.write_bytes(original.replace(old.encode(), new.encode())) + output.write('\nREVERTED\n$ cd packages/sdk && ' + shlex.join(command) + '\n') + output.flush() + failed = subprocess.run(command, cwd=SDK, stdout=output, stderr=subprocess.STDOUT) + output.write(f'exit={failed.returncode}\n') + finally: + path.write_bytes(original) + assert path.read_bytes() == original + output.write(f'\nRESTORED SHA256: {hashlib.sha256(path.read_bytes()).hexdigest()}\n') + output.write('$ cd packages/sdk && ' + shlex.join(command) + '\n') + output.flush() + passed = subprocess.run(command, cwd=SDK, stdout=output, stderr=subprocess.STDOUT) + output.write(f'exit={passed.returncode}\n') + print(f'{name}: reverted exit={failed.returncode}; restored exit={passed.returncode}; {log.relative_to(ROOT)}', flush=True) + assert failed.returncode == 1 and passed.returncode == 0, name diff --git a/evidence/run-read-timeout/resume-live.log b/evidence/run-read-timeout/resume-live.log new file mode 100644 index 000000000..7169d526d --- /dev/null +++ b/evidence/run-read-timeout/resume-live.log @@ -0,0 +1,13 @@ +$ cd packages/sdk && RELAYFLOWD_BIN=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target/debug/relayflowd npx vitest run tests/read-timeout-resume-live.test.ts + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/read-timeout-resume-live.test.ts (1 test) 860ms + ✓ parks an unreadable authored root and resumes without repeating its journaled effect 860ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 06:17:53 + Duration 2.17s (transform 643ms, setup 20ms, collect 1.02s, tests 860ms, environment 0ms, prepare 82ms) + +exit=0 diff --git a/evidence/run-read-timeout/typecheck-final.json b/evidence/run-read-timeout/typecheck-final.json new file mode 100644 index 000000000..2c0063d8c --- /dev/null +++ b/evidence/run-read-timeout/typecheck-final.json @@ -0,0 +1,10 @@ +{ + "command": "cd packages/sdk && npx tsc --noEmit && npx tsc -p tsconfig.tests.json", + "result": { + "chunk_id": "10aa44", + "wall_time_seconds": 0.000001521, + "exit_code": 0, + "original_token_count": 0, + "output": "" + } +} diff --git a/evidence/run-read-timeout/verification.md b/evidence/run-read-timeout/verification.md new file mode 100644 index 000000000..3fa45b5d5 --- /dev/null +++ b/evidence/run-read-timeout/verification.md @@ -0,0 +1,665 @@ +Captured verification at the review-fix source. Commands and output below are verbatim (absolute paths included): relayflowd is built from this tree by the kernel-build command, and both live suites run against that binary. [full-suite-final.log](full-suite-final.log) is the earlier full-suite transcript from before the review fixes (head a50a8de); it records that run's environment failures and does not cover the review-fix changes. + +## kernel-build.log + +```text +$ cd kernel && CARGO_TARGET_DIR=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target cargo build + Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s +exit=0 +``` + +## final-focused.log + +```text +$ cd packages/sdk && RELAYFLOWD_BIN=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target/debug/relayflowd npx vitest run tests/journal-client-read-timeout.test.ts tests/journal-client.test.ts tests/journal-client-completion.test.ts tests/journal-client-subscriptions.test.ts tests/running-step-watch.test.ts tests/heartbeat-timeout.test.ts tests/run-daemon-unresponsive.test.ts tests/authored-root.test.ts tests/classify-outcome.test.ts tests/cli.test.ts tests/direct-run-worker-lease.test.ts tests/resume-worker-lease.test.ts tests/worker-lease.test.ts tests/worker-lease-lost.test.ts tests/worker-lease-lost-live.test.ts tests/worker-lease-sweep.test.ts tests/run-read-load-live.test.ts tests/flow-executor-chain.test.ts tests/agent-transcript-live.test.ts tests/human-live.test.ts tests/reuse-summary-interruption.test.ts tests/memoization.test.ts tests/authored-verifier-read-budget.test.ts tests/communication-worker.test.ts tests/communication.test.ts tests/helper-effect-lease-loss.test.ts tests/authored-helpers.test.ts tests/authored-flow-slack.test.ts tests/effect-channel.test.ts --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/cli.test.ts (71 tests) 5583ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 350ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2045ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms +(node:783010) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (36 tests) 458ms + ✓ tests/journal-client-read-timeout.test.ts (33 tests) 2247ms + ✓ an exhausted budget during reader setup frees the queue for the next read 303ms + ✓ tests/journal-client.test.ts (17 tests) 80ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 6820ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 422ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 999ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 2666ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 355ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 889ms + ✓ tests/authored-flow-slack.test.ts (7 tests) 1271ms + ✓ authored Slack helper effects > replays after SIGKILL before confirm with the same token and one successful completion 458ms + ✓ authored Slack helper effects > replays after SIGKILL before complete with the same token and one successful completion 436ms + ✓ tests/running-step-watch.test.ts (14 tests) 12309ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2115ms + ✓ cancels promptly while a lease snapshot read is in flight 2004ms + ✓ a completion push ends the wait without waiting out an in-flight snapshot 2053ms + ✓ a completion that aborts the snapshot is not an error even if a retry starts at once 2002ms + ✓ a malformed watch frame during an in-flight snapshot is an error, not a completion 2002ms + ✓ a watch dropped during an in-flight snapshot is reported at once, not after the read budget 2001ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 2755ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 662ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 686ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 792ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 614ms + ✓ tests/authored-helpers.test.ts (6 tests) 4017ms + ✓ lowers the named acceptance helpers and Slack to confirmed journal effects 352ms + ✓ runs every available provider through the real kernel and resumes completed effects without a second write 2254ms + ✓ replays after SIGKILL before confirm with the same token and one successful completion 726ms + ✓ replays after SIGKILL before complete with the same token and one successful completion 621ms + ✓ tests/classify-outcome.test.ts (11 tests) 7422ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2010ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2001ms + ✓ tests/human-live.test.ts (3 tests) 4840ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 2929ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1217ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 693ms + ✓ tests/communication-worker.test.ts (20 tests) 1826ms + ✓ tests/worker-lease.test.ts (8 tests) 12ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 20ms + ✓ tests/communication.test.ts (10 tests) 14ms + ✓ tests/effect-channel.test.ts (5 tests) 282ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 714ms + ✓ tests/run-daemon-unresponsive.test.ts (5 tests) 6ms + ✓ tests/memoization.test.ts (58 tests) 66ms + ✓ tests/run-read-load-live.test.ts (2 tests) 2979ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2064ms + ✓ drains read and watch promises before an authored flow completes 914ms + ✓ tests/reuse-summary-interruption.test.ts (6 tests) 5ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 6ms + ✓ tests/journal-client-completion.test.ts (6 tests) 109ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 10ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 11ms + ✓ tests/helper-effect-lease-loss.test.ts (1 test) 280ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 10ms + ✓ tests/authored-verifier-read-budget.test.ts (2 tests) 59ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 14ms + + Test Files 29 passed (29) + Tests 371 passed (371) + Start at 06:16:46 + Duration 66.56s (transform 1.62s, setup 189ms, collect 8.28s, tests 54.22s, environment 4ms, prepare 1.15s) + +exit=0 +``` + +## resume-live.log + +```text +$ cd packages/sdk && RELAYFLOWD_BIN=/home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/kernel-target/debug/relayflowd npx vitest run tests/read-timeout-resume-live.test.ts + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/read-timeout-resume-live.test.ts (1 test) 860ms + ✓ parks an unreadable authored root and resumes without repeating its journaled effect 860ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 06:17:53 + Duration 2.17s (transform 643ms, setup 20ms, collect 1.02s, tests 860ms, environment 0ms, prepare 82ms) + +exit=0 +``` + +The mutation sections below were regenerated by `cd packages/sdk && python3 ../../evidence/run-read-timeout/mutations.py` (the command used) against the review-fix source; each mutation fails its test and the restored source passes. + +## mutation-reader.log + +```text +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: +return await reader.requestOnce(verb, params, remaining, attemptSignal); +With: +return await this.primary.requestOnce(verb, params, remaining, attemptSignal); + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 113ms + × serves a bounded read while an unbounded command is in flight 112ms + → expected true to be false // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > serves a bounded read while an unbounded command is in flight +AssertionError: expected true to be false // Object.is equality + +- Expected ++ Received + +- false ++ true + + ❯ tests/journal-client-read-timeout.test.ts:38:23 + 36| await inFlight; + 37| expect(await client.runGet('run')).toMatchObject({ status: 'running'… + 38| expect(commandDone).toBe(false); + | ^ + 39| await command; + 40| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:09 + Duration 589ms (transform 185ms, setup 18ms, collect 300ms, tests 113ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 106ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:10 + Duration 542ms (transform 172ms, setup 15ms, collect 309ms, tests 106ms, environment 0ms, prepare 28ms) + +exit=0 +``` + +## mutation-reader-reconnect.log + +```text +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: + this.drop(reader); + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'reconnects the reader after it disconnects' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 1007ms + × reconnects the reader after it disconnects within the read budget 1006ms + → journal client: run.get read session was interrupted after 76 attempts in 1000ms (read budget 1000ms): journal client: not connected (run.get): journal client: connection closed + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > reconnects the reader after it disconnects within the read budget +JournalReadInterruptedError: journal client: run.get read session was interrupted after 76 attempts in 1000ms (read budget 1000ms): journal client: not connected (run.get): journal client: connection closed + ❯ exhausted src/journal-read-policy.ts:75:9 + 73| let last: unknown; + 74| const exhausted = () => last instanceof JournalReadInterruptedError + 75| ? new JournalReadInterruptedError(verb, attempts, performance.no… + | ^ + 76| : new JournalRequestTimeoutError(verb, timeoutMs, attempts, perf… + 77| // A caller's cancellation settles this read and drains its queued… + ❯ Timeout. src/journal-read-policy.ts:90:48 + +Caused by: Error: journal client: not connected (run.get): journal client: connection closed + ❯ src/journal-connection.ts:183:16 + ❯ JournalClient.requestOnce src/journal-connection.ts:176:12 + ❯ src/journal-budgeted-reads.ts:60:29 + ❯ src/journal-read-policy.ts:102:18 + +Caused by: Error: journal client: connection closed + ❯ Socket. src/journal-connection.ts:89:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:11 + Duration 1.49s (transform 189ms, setup 18ms, collect 324ms, tests 1.01s, environment 0ms, prepare 46ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'reconnects the reader after it disconnects' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 91ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:13 + Duration 559ms (transform 180ms, setup 18ms, collect 295ms, tests 91ms, environment 0ms, prepare 45ms) + +exit=0 +``` + +## mutation-reader-setup-retry.log + +```text +Source: packages/sdk/src/journal-budgeted-reads.ts +Original SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +Replaced: + if (this.reader === reader) { + this.reader = undefined; + this.readerReady = undefined; + } + throw error; + +With: + return undefined; + + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'retries a reader setup that timed out' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (33 tests | 1 failed | 32 skipped) 84ms + × retries a reader setup that timed out instead of reading on the primary for good 83ms + → expected 1 to be 2 // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > retries a reader setup that timed out instead of reading on the primary for good +AssertionError: expected 1 to be 2 // Object.is equality + +- Expected ++ Received + +- 2 ++ 1 + + ❯ tests/journal-client-read-timeout.test.ts:203:18 + 201| expect(await client.runGet('run')).toMatchObject({ status: 'complete… + 202| expect(await client.runGet('run')).toMatchObject({ status: 'complete… + 203| expect(hellos).toBe(2); + | ^ + 204| expect(served).toHaveLength(2); + 205| expect(served.every(socket => socket !== primary)).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 32 skipped (33) + Start at 06:16:14 + Duration 535ms (transform 173ms, setup 17ms, collect 299ms, tests 84ms, environment 0ms, prepare 42ms) + +exit=1 + +RESTORED SHA256: fe56924a3b780e35507350c7cef74a4cb2815fee0588cf952542cb8119332710 +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'retries a reader setup that timed out' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (33 tests | 32 skipped) 74ms + + Test Files 1 passed (1) + Tests 1 passed | 32 skipped (33) + Start at 06:16:15 + Duration 534ms (transform 180ms, setup 17ms, collect 303ms, tests 74ms, environment 0ms, prepare 41ms) + +exit=0 +``` + +## mutation-watch-cadence.log + +```text +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +const LEASE_POLL_MS = 2_000; +With: +const LEASE_POLL_MS = 50; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 2109ms + × uses pushes for completion with lease-cadence reads and releases its watcher 2108ms + → expected 41 to be less than or equal to 1 + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > uses pushes for completion with lease-cadence reads and releases its watcher +AssertionError: expected 41 to be less than or equal to 1 + ❯ tests/running-step-watch.test.ts:39:19 + 37| await waitForRunningStep(client, 'run', { id: 'step', type: 'agent… + 38| expect(performance.now() - start).toBeGreaterThan(2000); + 39| expect(reads).toBeLessThanOrEqual(1); + | ^ + 40| await sleep(10); + 41| expect(watchClosed).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:16 + Duration 2.59s (transform 194ms, setup 16ms, collect 322ms, tests 2.11s, environment 0ms, prepare 41ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2117ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2116ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:19 + Duration 2.56s (transform 186ms, setup 14ms, collect 298ms, tests 2.12s, environment 0ms, prepare 33ms) + +exit=0 +``` + +## mutation-snapshot-cancel.log + +```text +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +const signal = options.signal === undefined ? read.signal + : AbortSignal.any([options.signal, read.signal]); +With: +const signal = read.signal; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'cancels promptly while a lease snapshot read is in flight' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 2224ms + × cancels promptly while a lease snapshot read is in flight 2223ms + → expected 'still waiting' to be an instance of Error + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > cancels promptly while a lease snapshot read is in flight +AssertionError: expected 'still waiting' to be an instance of Error + ❯ tests/running-step-watch.test.ts:83:21 + 81| controller.abort(); + 82| const settled = await Promise.race([outcome, sleep(200).then(() =>… + 83| expect(settled).toBeInstanceOf(Error); + | ^ + 84| expect((settled as Error).message).toContain('was canceled'); + 85| } finally { client.close(); await new Promise(resolve => serve… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:22 + Duration 3.35s (transform 482ms, setup 30ms, collect 848ms, tests 2.22s, environment 0ms, prepare 75ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'cancels promptly while a lease snapshot read is in flight' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2009ms + ✓ cancels promptly while a lease snapshot read is in flight 2008ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:26 + Duration 2.60s (transform 223ms, setup 20ms, collect 393ms, tests 2.01s, environment 0ms, prepare 50ms) + +exit=0 +``` + +## mutation-completion-aborts-snapshot.log + +```text +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +Replaced: +snapshotRead?.abort(new Error(`step "${runningStep.id}" completed during the lease snapshot`)); +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'a completion push ends the wait' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/running-step-watch.test.ts (14 tests | 1 failed | 13 skipped) 3014ms + × a completion push ends the wait without waiting out an in-flight snapshot 3013ms + → expected 'still waiting' to be 'returned' // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > a completion push ends the wait without waiting out an in-flight snapshot +AssertionError: expected 'still waiting' to be 'returned' // Object.is equality + +Expected: "returned" +Received: "still waiting" + + ❯ tests/running-step-watch.test.ts:152:21 + 150| const waiting = waitForRunningStep(client, 'run', { id: 'step', ty… + 151| const outcome = await Promise.race([waiting.then(() => 'returned')… + 152| expect(outcome).toBe('returned'); + | ^ + 153| expect(await client.runGet('run', { signal: AbortSignal.timeout(50… + 154| } finally { client.close(); await new Promise(resolve => serve… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 13 skipped (14) + Start at 06:16:29 + Duration 3.51s (transform 191ms, setup 16ms, collect 329ms, tests 3.01s, environment 0ms, prepare 41ms) + +exit=1 + +RESTORED SHA256: 2a5f6adeca146e6188ac5e6d129db004e71574cf09df397fd4b9127c50e9b5aa +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'a completion push ends the wait' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/running-step-watch.test.ts (14 tests | 13 skipped) 2062ms + ✓ a completion push ends the wait without waiting out an in-flight snapshot 2061ms + + Test Files 1 passed (1) + Tests 1 passed | 13 skipped (14) + Start at 06:16:33 + Duration 2.57s (transform 204ms, setup 17ms, collect 350ms, tests 2.06s, environment 0ms, prepare 41ms) + +exit=0 +``` + +## mutation-heartbeat.log + +```text +Source: packages/sdk/src/worker-lease.ts +Original SHA256: 4c58d4670422dc2169764af16eb082c55e72c5321ed9d6001af3607f03955fa3 +Replaced: + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/heartbeat-timeout.test.ts (1 test | 1 failed) 19ms + × a heartbeat timeout is lease loss rather than a worker body failure 18ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/heartbeat-timeout.test.ts > a heartbeat timeout is lease loss rather than a worker body failure +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + ❯ tests/heartbeat-timeout.test.ts:16:5 + 14| try { + 15| await client.connect(); + 16| await expect(withWorkerLease(client, { + | ^ + 17| run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', … + 18| lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed (1) + Start at 06:16:36 + Duration 460ms (transform 168ms, setup 17ms, collect 294ms, tests 19ms, environment 0ms, prepare 42ms) + +exit=1 + +RESTORED SHA256: 4c58d4670422dc2169764af16eb082c55e72c5321ed9d6001af3607f03955fa3 +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/heartbeat-timeout.test.ts (1 test) 16ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 06:16:37 + Duration 435ms (transform 164ms, setup 16ms, collect 277ms, tests 16ms, environment 0ms, prepare 32ms) + +exit=0 +``` + +## mutation-root-parking.log + +```text +Source: packages/sdk/src/authored-root.ts +Original SHA256: 8e3bd4ebe8b470c1486f1ab6adcf2fd81ba19e6a173e2531e7cbc59b16bcba29 +Replaced: + if (isReadInterruptionError(error) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ❯ tests/authored-root.test.ts (36 tests | 3 failed | 33 skipped) 31ms + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=false) 22ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=true) 4ms + → expected AuthoredFlowExecutionError: daemon_unresp… { …(7) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + × durable authored root > leaves the root resumable when the read session disconnects while the root worker is healthy 4ms + → expected JournalReadInterruptedError: journal clie… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 3 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=false) +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalRequestTimeoutError: journal client: run.get timed out after 10ms] + + ❯ tests/authored-root.test.ts:424:5 + 422| }); + 423| const journal = new RootJournal(); + 424| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 425| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 426| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/3]⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch (Node frame=true) +AssertionError: expected AuthoredFlowExecutionError: daemon_unresp… { …(7) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { ++ AuthoredFlowExecutionError { + "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", + "rootRunId": "root-run", + } + + ❯ tests/authored-root.test.ts:424:5 + 422| }); + 423| const journal = new RootJournal(); + 424| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 425| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 426| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/3]⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable when the read session disconnects while the root worker is healthy +AssertionError: expected JournalReadInterruptedError: journal clie… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalReadInterruptedError: journal client: run.get read session was interrupted after 2 attempts in 40ms (read budget 300000ms): journal client: connection closed] + + ❯ tests/authored-root.test.ts:437:5 + 435| const loaded = await fixture(false, 0, async () => { throw interru… + 436| const journal = new RootJournal(); + 437| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 438| { dataDir: '/unused', admissionKey: 'read-disconnect' })).reject… + 439| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/3]⎯ + + Test Files 1 failed (1) + Tests 3 failed | 33 skipped (36) + Start at 06:16:37 + Duration 1.33s (transform 681ms, setup 15ms, collect 1.12s, tests 31ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: 8e3bd4ebe8b470c1486f1ab6adcf2fd81ba19e6a173e2531e7cbc59b16bcba29 +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/khaliqgant/Projects/AgentWorkforce/flows-worktrees/pr613/packages/sdk + + ✓ tests/authored-root.test.ts (36 tests | 33 skipped) 23ms + + Test Files 1 passed (1) + Tests 3 passed | 33 skipped (36) + Start at 06:16:39 + Duration 1.29s (transform 674ms, setup 14ms, collect 1.11s, tests 23ms, environment 0ms, prepare 41ms) + +exit=0 +``` diff --git a/kernel/DAEMON-LIFECYCLE.md b/kernel/DAEMON-LIFECYCLE.md index 3d739fab2..257db138d 100644 --- a/kernel/DAEMON-LIFECYCLE.md +++ b/kernel/DAEMON-LIFECYCLE.md @@ -559,3 +559,15 @@ SDK (`npm test` in `packages/sdk`): Test 15 is the one that matters most and is the hardest to fake: it must spawn real processes against a real temp data dir, because the property under test is enforced by `flock(2)`, not by any code we could stub. + +## Reads during long-running commands + +The daemon handles frames sequentially per connection; `run.start`, +`run.resume`, and `step.complete` can drive deterministic commands before +replying. The SDK's flow execution clients therefore send read-only requests +on an unconditional lazy reader session, serialized and retried within a +bounded budget. Worker registrations, leases, writes, and watches retain their +own session ordering. Watch pushes come from the hub and can arrive while a +command is in flight. A read timeout is not a terminal run fact: the CLI probes +a fresh connection and reports a resumable interruption (see `docs/SURFACE.md` +§5). No daemon protocol or runtime release is required for this policy. diff --git a/packages/sdk/src/authored-flow-error.ts b/packages/sdk/src/authored-flow-error.ts index d43aeeb96..307343791 100644 --- a/packages/sdk/src/authored-flow-error.ts +++ b/packages/sdk/src/authored-flow-error.ts @@ -35,6 +35,9 @@ export type AuthoredFlowExecutionErrorCode = * re-dispatch the root to it. The run is not failed: completed steps are * journaled and `flows resume ` continues it. */ + | 'daemon_unresponsive' + /** The root run completed; only reading back its stored result failed. */ + | 'result_unreadable' | 'root_lease_lost' | 'unsupported_completion' | 'unsupported_gate' diff --git a/packages/sdk/src/authored-helper-effect.ts b/packages/sdk/src/authored-helper-effect.ts index f10a7b641..7a627fdc4 100644 --- a/packages/sdk/src/authored-helper-effect.ts +++ b/packages/sdk/src/authored-helper-effect.ts @@ -8,7 +8,7 @@ import type { AuthoredFlowJournalStep } from './authored-flow-executor.js'; import { compileSpec, toKernelSpec } from './compile.js'; import { SPEC_SCHEMA_VERSION, type KernelAgentStep } from './spec.js'; import type { StepDispatchEvent } from './protocol.js'; -import { withWorkerLease } from './worker-lease.js'; +import { isLeaseLost, withWorkerLease } from './worker-lease.js'; import { helperProviders, type HelperCall } from '@relayflows/surface/runtime'; import { helperWriteback, HelperDeliveryError } from './helper-writeback.js'; import { checkSlackHelpers } from './slack-preflight.js'; @@ -101,7 +101,8 @@ async function driveHelperEffect( } await completed; } catch (error) { - if (error instanceof AuthoredFlowExecutionError) throw error; + // A lost lease belongs to the kernel's sweep and retry, not to this step. + if (error instanceof AuthoredFlowExecutionError || isLeaseLost(error)) throw error; throw new AuthoredFlowExecutionError('step_failed', error instanceof Error ? error.message : 'Helper effect failed', 'worker_error', runId); } finally { client.close(); } } diff --git a/packages/sdk/src/authored-node-entry.ts b/packages/sdk/src/authored-node-entry.ts index ac6f6fdab..1ecd4a265 100644 --- a/packages/sdk/src/authored-node-entry.ts +++ b/packages/sdk/src/authored-node-entry.ts @@ -1,3 +1,4 @@ +import { FLOW_READ_BUDGET_MS, isReadInterruptionError } from './journal-read-policy.js'; import { Worker } from 'node:worker_threads'; import { createHash, createHmac } from 'node:crypto'; import { readFileSync, writeSync } from 'node:fs'; @@ -74,7 +75,7 @@ try { const loaded = await loadPinnedAuthoredSource(request.metadata, true); if (request.localAgentStream !== request.metadata.localAgentStream) throw new Error('authored root local agent surface mismatch'); if (request.workerCapacity !== undefined && !isAgentCapacity(request.workerCapacity)) throw new Error('invalid authored worker capacity'); - client = new JournalClient(request.socketPath); + client = new JournalClient(request.socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS, budgetHandshake: true }); await client.connect(); await client.hello('flows-authored-node'); const result = await executeAuthoredFlow(loaded.handle, client, request.metadata.inputPresent ? request.metadata.input : undefined, { @@ -99,6 +100,10 @@ try { // `parkCause` travels for the same reason: it is the difference between // "attach a worker" and "a human has to recover this", and this process is // the only one that saw the child's classification. + // Preserve a read interruption across the Node/Bun error frame so the + // lease-owning parent can leave the root resumable. + ...(isReadInterruptionError(error) + ? { code: 'daemon_unresponsive' } : {}), ...(error instanceof AuthoredFlowExecutionError ? { code: error.code, completionReason: error.completionReason, runId: error.runId, details: error.details, diff --git a/packages/sdk/src/authored-node-runner.ts b/packages/sdk/src/authored-node-runner.ts index cf2c8f075..a530b4a1b 100644 --- a/packages/sdk/src/authored-node-runner.ts +++ b/packages/sdk/src/authored-node-runner.ts @@ -1,4 +1,5 @@ import { JournalClient } from './journal-client.js'; +import { FLOW_READ_BUDGET_MS } from './journal-read-policy.js'; import { spawn, spawnSync } from 'node:child_process'; import { createHash, createHmac, randomBytes, timingSafeEqual } from 'node:crypto'; import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; @@ -171,7 +172,7 @@ export async function runAuthoredInNode( else resolve(result); }); }); - await verifyAuthoredNodeResult(result, metadata, rootRunId, socketPath); + await verifyAuthoredNodeResult(result, metadata, rootRunId, socketPath, options.signal); return result; } finally { await rm(directory, { recursive: true, force: true }); } } @@ -323,7 +324,7 @@ function frameCount(frame: Record, key: string): number | undef /** The IPC frame is a claim, not a durable terminal fact or a sandbox boundary. */ export async function verifyAuthoredNodeResult( result: AuthoredFlowExecutionResult, metadata: AuthoredRootMetadata, - rootRunId: string, socketPath: string, + rootRunId: string, socketPath: string, signal?: AbortSignal, ): Promise { const invalid = (why = ''): never => { throw new Error(`authored runtime result has no matching durable completion${process.env['FLOWS_VERIFIER_DEBUG'] && why ? ` (${why})` : ''}`); }; if (result.rootRunId !== rootRunId || result.name !== metadata.flowName @@ -354,7 +355,9 @@ export async function verifyAuthoredNodeResult( if (!authoredIds.has(gate.id.slice(0, -'.gate'.length))) invalid(`orphan ${gate.id}`); } const runs = new Set(); - const journal = new JournalClient(socketPath); + // The verifier reads the same journal under the same CPU load as the body. + const journal = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS, budgetHandshake: true, + ...(signal === undefined ? {} : { readSignal: signal }) }); await journal.connect(); try { await journal.hello('flows-authored-result-verifier'); diff --git a/packages/sdk/src/authored-root-readback.ts b/packages/sdk/src/authored-root-readback.ts new file mode 100644 index 000000000..e80d25ee8 --- /dev/null +++ b/packages/sdk/src/authored-root-readback.ts @@ -0,0 +1,100 @@ +import { isDurableCompletionDetail, isLoweredCompletion } from './authored-completion.js'; +import type { AuthoredFlowExecutionResult } from './authored-flow-executor.js'; +import { AuthoredFlowExecutionError } from './authored-flow-error.js'; +import { isSurfaceCompletionReason } from './authored-step-output.js'; +import { JournalFrameError, JournalProtocolError, type JournalClient } from './journal-client.js'; +import type { RunOutcome } from './protocol.js'; + +/* + * Reading an authored root back from its journal, and what a failed read + * means. Only a definite answer from the daemon may change how a run is + * treated; an unanswered, canceled or failed read decides nothing. + */ + +const NO_ROOT_STEP = new Error('authored root journal has no root step'); + +export async function rootKernelStep(journal: JournalClient, runId: string): Promise> { + const entries = (await journal.journalRead(runId, 1)).entries as Array>; + const spawned = entries.find(entry => entry.entry_type === 'run.spawned'); + const payload = spawned?.payload as { spec?: { steps?: unknown[] } } | undefined; + const step = payload?.spec?.steps?.[0]; + if (typeof step !== 'object' || step === null || Array.isArray(step)) throw NO_ROOT_STEP; + return step as Record; +} + +/** Codes relayflowd uses when it failed to read its own storage (server/protocol.rs). */ +const DAEMON_READ_FAILURES = new Set(['journal_write_failed', 'internal']); + +/** + * The root step of `runId`, or undefined when the daemon definitively answered + * that it has none: a refusal of the read (run_not_found, an unreadable early + * epoch, ...) or a journal without a root step. A daemon-side read failure, an + * interruption or a cancellation propagates instead, so a delayed, canceled or + * failing daemon never makes resume guess the declarative path. + */ +export async function rootStepIfPresent(journal: JournalClient, runId: string): Promise | undefined> { + return rootKernelStep(journal, runId).catch(error => { + if ((error instanceof JournalProtocolError && !DAEMON_READ_FAILURES.has(error.code)) || error === NO_ROOT_STEP) return undefined; + throw error; + }); +} + +/** + * The stored result of a root the daemon already reports completed. If reading + * it back is interrupted, the outcome is still known: say so, rather than + * recasting a finished run as resumable. + */ +export function completedResult(journal: JournalClient, outcome: RunOutcome, rootRunId: string): Promise { + return completedRootResult(journal, rootRunId).catch(error => { + // A bad frame or a journal with no stored result is a definite fact; any + // other failure — interrupted, canceled, or a daemon-side read error such + // as journal_write_failed — left the verdict unread, not the run unfinished. + if (error instanceof JournalFrameError || error === NO_DURABLE_RESULT) throw error; + // The kernel completes the root step with success whatever the body + // declared; the authored verdict lives only in the unread output. + const unread = new AuthoredFlowExecutionError('result_unreadable', + `run ${rootRunId} completed, but its stored result (the flow's verdict) could not be read: ${error.message}`, + undefined, rootRunId); + unread.rootRunId = rootRunId; + throw unread; + }); +} + +const NO_DURABLE_RESULT = new Error('completed authored root has no durable result'); + +async function completedRootResult( + journal: JournalClient, + rootRunId: string, +): Promise { + const entries = (await journal.journalRead(rootRunId, 1)).entries as Array>; + const completed = entries.slice().reverse().find(entry => entry.entry_type === 'step.completed' + && entry.step_id === 'authored-root' + && (entry.payload as { completionReason?: unknown } | undefined)?.completionReason === 'success'); + const output = (completed?.payload as { output?: unknown } | undefined)?.output; + if (!isCompletedRootOutput(output)) throw NO_DURABLE_RESULT; + // The stored detail, never a freshly computed one: redaction reads the + // CURRENT environment, so recomputing here would let a completed run report + // something its journal does not hold. + return Object.freeze({ + name: output.name, + completionReason: output.completionReason, + ...(output.completionDetail === undefined ? {} : { completionDetail: output.completionDetail }), + journalSteps: Object.freeze(output.journalSteps.map(step => Object.freeze({ ...step }))), + rootRunId, + }); +} + +function isCompletedRootOutput(value: unknown): value is Omit { + if (typeof value !== 'object' || value === null || Array.isArray(value)) return false; + const output = value as Partial; + return typeof output.name === 'string' + && isLoweredCompletion(output.completionReason) + // A malformed detail fails the whole readback rather than being dropped: + // silently discarding it would report the verdict without the evidence + // the journal says it was recorded with. + && (output.completionDetail === undefined || isDurableCompletionDetail(output.completionDetail)) + && Array.isArray(output.journalSteps) + && output.journalSteps.every(step => typeof step === 'object' && step !== null + && typeof step.id === 'string' && typeof step.runId === 'string' + && isSurfaceCompletionReason(step.completionReason)); +} diff --git a/packages/sdk/src/authored-root.ts b/packages/sdk/src/authored-root.ts index 1929fd381..ce5a70aca 100644 --- a/packages/sdk/src/authored-root.ts +++ b/packages/sdk/src/authored-root.ts @@ -1,3 +1,4 @@ +import { isReadInterruptionError } from './journal-read-policy.js'; import { loadPinnedAuthoredSource } from './authored-source-authority.js'; import { assertAuthoredRuntimeAvailable, runAuthoredInNode } from './authored-node-runner.js'; import { createHash, randomUUID } from 'node:crypto'; @@ -5,7 +6,6 @@ import { readFile } from 'node:fs/promises'; import { canonicalize } from './canonical.js'; import { compileSpec, toKernelSpec } from './compile.js'; import { executeAuthoredFlow, type AuthoredFlowExecutionResult, type AuthoredFlowSuspendedResult } from './authored-flow-executor.js'; -import { isDurableCompletionDetail, isLoweredCompletion } from './authored-completion.js'; import { AUTHORED_ROOT_KIND } from './authored-verdict.js'; import { loadAuthoredFlow, @@ -19,9 +19,9 @@ import type { RunLifecycleOptions } from './cli/run.js'; import { isLeaseLost, withWorkerLease } from './worker-lease.js'; import { AuthoredFlowExecutionError, AuthoredHumanParked } from './authored-flow-error.js'; import { readOpenHumanWaits, resumeCommand } from './authored-human.js'; -import { isSurfaceCompletionReason } from './authored-step-output.js'; import { readSubscriptionPark } from './authored-subscription-park.js'; import { localAgentCredentialEnvironment } from './local-agent-environment.js'; +import { completedResult, rootKernelStep, rootStepIfPresent } from './authored-root-readback.js'; export type DurableAuthoredFlowResult = | (AuthoredFlowExecutionResult & { readonly rootRunId: string }) @@ -119,7 +119,7 @@ export async function executeDurableAuthoredFlow( options.onAdmitted?.(outcome.run_id); if (outcome.status === 'completed') { dispatchWait.cancel(); - return await completedRootResult(journal, outcome.run_id); + return await completedResult(journal, outcome, outcome.run_id); } assertRootCanDispatch(outcome); options.lifecycle?.onRunStarted?.({ runId: outcome.run_id, flow: definition.name }); @@ -168,7 +168,7 @@ export async function resumeDurableAuthoredFlow( const outcome = await journal.runResume(rootRunId); if (outcome.status === 'completed') { dispatchWait.cancel(); - return await completedRootResult(journal, rootRunId); + return await completedResult(journal, outcome, rootRunId); } assertRootCanDispatch(outcome); await assertNoOpenHumanWait(journal, outcome); @@ -199,7 +199,7 @@ export async function readAuthoredRootMetadata( journal: JournalClient, runId: string, ): Promise { - const step = await rootKernelStep(journal, runId).catch(() => undefined); + const step = await rootStepIfPresent(journal, runId); if (step === undefined || step.id !== 'authored-root' || !hasRootStream(step)) return undefined; if (typeof step.instruction !== 'string') { throw new Error('authored root journal has malformed authority metadata'); @@ -320,38 +320,43 @@ async function driveRoot( ): Promise { try { const result = await withWorkerLease(peer, dispatch, async rootSignal => { - const callerSignal = options.lifecycle?.signal; - const signal = callerSignal === undefined ? rootSignal : AbortSignal.any([callerSignal, rootSignal]); - if (process.versions['bun'] !== undefined) { - return runAuthoredInNode(metadata, journal.socketPath, dispatch.run_id, { - dataDir: options.dataDir, localAgentStream: options.localAgentStream, - ...(options.agentEnvironment === undefined ? {} : { - agentEnvironment: localAgentCredentialEnvironment(options.agentEnvironment), - }), - ...(options.workerCapacity === undefined ? {} : { workerCapacity: options.workerCapacity }), - ...options.lifecycle, signal, - }); - } - return await executeAuthoredFlow( - loaded.handle, - journal, - metadata.inputPresent ? metadata.input : undefined, - { - getDefinition: loaded.getDefinition, - dataDir: options.dataDir, - flowPath: metadata.flowPath, - localAgentStream: options.localAgentStream, - ...(options.agentEnvironment === undefined ? {} : { agentEnvironment: options.agentEnvironment }), - ...(options.workerCapacity === undefined ? {} : { workerCapacity: options.workerCapacity }), - rootRunId: dispatch.run_id, - extensions: loaded.extensions, - flowGraph: loaded.graph, - ...options.lifecycle, - signal: callerSignal === undefined - ? rootSignal - : AbortSignal.any([callerSignal, rootSignal]), - }, - ); + // A lost root lease must also stop body reads still inside their budget. + const releaseReads = journal.scopeReads?.(rootSignal) ?? (() => {}); + try { + const callerSignal = options.lifecycle?.signal; + const signal = callerSignal === undefined ? rootSignal : AbortSignal.any([callerSignal, rootSignal]); + if (process.versions['bun'] !== undefined) { + // Awaited, so the read scope below covers the child's whole run. + return await runAuthoredInNode(metadata, journal.socketPath, dispatch.run_id, { + dataDir: options.dataDir, localAgentStream: options.localAgentStream, + ...(options.agentEnvironment === undefined ? {} : { + agentEnvironment: localAgentCredentialEnvironment(options.agentEnvironment), + }), + ...(options.workerCapacity === undefined ? {} : { workerCapacity: options.workerCapacity }), + ...options.lifecycle, signal, + }); + } + return await executeAuthoredFlow( + loaded.handle, + journal, + metadata.inputPresent ? metadata.input : undefined, + { + getDefinition: loaded.getDefinition, + dataDir: options.dataDir, + flowPath: metadata.flowPath, + localAgentStream: options.localAgentStream, + ...(options.agentEnvironment === undefined ? {} : { agentEnvironment: options.agentEnvironment }), + ...(options.workerCapacity === undefined ? {} : { workerCapacity: options.workerCapacity }), + rootRunId: dispatch.run_id, + extensions: loaded.extensions, + flowGraph: loaded.graph, + ...options.lifecycle, + signal: callerSignal === undefined + ? rootSignal + : AbortSignal.any([callerSignal, rootSignal]), + }, + ); + } finally { releaseReads(); } }); await peer.stepComplete( dispatch.run_id, dispatch.step_id, dispatch.attempt, @@ -368,6 +373,20 @@ async function driveRoot( // A lost lease is owned by the kernel's retry path, not by this attempt's // terminalization or suspension handling. if (isLeaseLost(error)) throw error; + // A caller cancel is the caller's, not a body failure: reads aborted by it + // (including the Bun verifier's) leave the root resumable, never terminal. + const signal = options.lifecycle?.signal; + if (aborted(signal) && (error === signal!.reason || (error instanceof Error && error.name === 'AbortError'))) { + if (error instanceof Error) (error as Error & { rootRunId?: string }).rootRunId = dispatch.run_id; + throw error; + } + if (isReadInterruptionError(error) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } if (error instanceof AuthoredFlowExecutionError && error.code === 'subscription_suspended' && error.suspension !== undefined) { @@ -450,16 +469,6 @@ function rootSpec(metadata: AuthoredRootMetadata, stream: string) { })); } -async function rootKernelStep(journal: JournalClient, runId: string): Promise> { - const entries = (await journal.journalRead(runId, 1)).entries as Array>; - const spawned = entries.find(entry => entry.entry_type === 'run.spawned'); - const payload = spawned?.payload as { spec?: { steps?: unknown[] } } | undefined; - const step = payload?.spec?.steps?.[0]; - if (typeof step !== 'object' || step === null || Array.isArray(step)) { - throw new Error('authored root journal has no root step'); - } - return step as Record; -} function rootStreamFromMetadata(step: Record): string { const surfaces = step.surfaces as { streams?: Array<{ stream?: unknown }> } | undefined; @@ -543,44 +552,6 @@ function nextRootDispatch(peer: JournalClient, timeoutMs = 30_000): { return { promise, cancel }; } -async function completedRootResult( - journal: JournalClient, - rootRunId: string, -): Promise { - const entries = (await journal.journalRead(rootRunId, 1)).entries as Array>; - const completed = entries.slice().reverse().find(entry => entry.entry_type === 'step.completed' - && entry.step_id === 'authored-root' - && (entry.payload as { completionReason?: unknown } | undefined)?.completionReason === 'success'); - const output = (completed?.payload as { output?: unknown } | undefined)?.output; - if (!isCompletedRootOutput(output)) { - throw new Error('completed authored root has no durable result'); - } - // The stored detail, never a freshly computed one: redaction reads the - // CURRENT environment, so recomputing here would let a completed run report - // something its journal does not hold. - return Object.freeze({ - name: output.name, - completionReason: output.completionReason, - ...(output.completionDetail === undefined ? {} : { completionDetail: output.completionDetail }), - journalSteps: Object.freeze(output.journalSteps.map(step => Object.freeze({ ...step }))), - rootRunId, - }); -} - -function isCompletedRootOutput(value: unknown): value is Omit { - if (typeof value !== 'object' || value === null || Array.isArray(value)) return false; - const output = value as Partial; - return typeof output.name === 'string' - && isLoweredCompletion(output.completionReason) - // A malformed detail fails the whole readback rather than being dropped: - // silently discarding it would report the verdict without the evidence - // the journal says it was recorded with. - && (output.completionDetail === undefined || isDurableCompletionDetail(output.completionDetail)) - && Array.isArray(output.journalSteps) - && output.journalSteps.every(step => typeof step === 'object' && step !== null - && typeof step.id === 'string' && typeof step.runId === 'string' - && isSurfaceCompletionReason(step.completionReason)); -} function assertRootCanDispatch(outcome: RunOutcome): void { // The kernel reports `parked` after handing an agent lease to its worker; diff --git a/packages/sdk/src/cli/direct-run.ts b/packages/sdk/src/cli/direct-run.ts index 904573700..9bf868107 100644 --- a/packages/sdk/src/cli/direct-run.ts +++ b/packages/sdk/src/cli/direct-run.ts @@ -1,3 +1,5 @@ +import { completedResultUnreadableReport, daemonUnresponsiveReport, isReadInterruption } from './journal-timeout.js'; +import { FLOW_READ_BUDGET_MS } from '../journal-read-policy.js'; import { onWorkerFailure } from '../worker-lease.js'; import { McpStepError } from '../authored-mcp.js'; import { randomUUID } from 'node:crypto'; @@ -67,7 +69,8 @@ export async function runDirectFlow( } const socketPath = socketFor(dataDir); const base: RunReport = { ...emptyReport('run'), path }; - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS, + ...(options.signal === undefined ? {} : { readSignal: options.signal }) }); const connected = await connect(client, 'run', dataDir, base, options); if (connected !== undefined) return connected; @@ -212,6 +215,10 @@ export async function runDirectFlow( if (error instanceof AuthoredFlowExecutionError && (error.code === 'step_failed' || error.code === 'gate_failed')) { return authoredStepFailure('run', base, socketPath, error); } + if (error instanceof AuthoredFlowExecutionError && error.code === 'result_unreadable') { + return completedResultUnreadableReport('run', base, socketPath, error); + } + if (isReadInterruption(error)) return daemonUnresponsiveReport('run', base, socketPath, error, base.rootRunId, options, dataDir); if (error instanceof AuthoredFlowExecutionError && error.code === 'root_lease_lost') { return rootLeaseLostReport('run', base, socketPath, error); } diff --git a/packages/sdk/src/cli/journal-timeout.ts b/packages/sdk/src/cli/journal-timeout.ts new file mode 100644 index 000000000..a6a20c6d0 --- /dev/null +++ b/packages/sdk/src/cli/journal-timeout.ts @@ -0,0 +1,49 @@ +import { probeSocket } from '../daemon-connection.js'; +import { AuthoredFlowExecutionError } from '../authored-flow-error.js'; +import { resumeCommand } from '../authored-human.js'; +import { isReadInterruptionError, JournalReadInterruptedError } from '../journal-read-policy.js'; +import type { CheckReport } from './check.js'; +import type { RunExecution, RunReport, RunLifecycleOptions, RunCommand } from './run.js'; + +export function isReadInterruption(error: unknown): boolean { + return isReadInterruptionError(error) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive'); +} + +/** + * A completed run whose stored result could not be read back. The run is + * finished — nothing to resume — but its authored verdict is unknown, so the + * report claims neither success nor a reason, and says how to read it later. + */ +export function completedResultUnreadableReport(command: RunCommand, base: CheckReport | RunReport, + socketPath: string, error: AuthoredFlowExecutionError): RunExecution { + const runId = error.rootRunId ?? error.runId; + return { exitCode: 1, report: { + ...base, command, ok: false, socketPath, status: 'completed', + ...(runId === undefined ? {} : { runId, rootRunId: runId }), + diagnostics: [...base.diagnostics, { severity: 'warning', kind: 'result_unreadable', + message: `${error.message}. The run is finished; its verdict is unconfirmed` + + (runId === undefined ? '.' : ` — read it with \`flows status ${runId}\` once relayflowd answers.`) }], + } as RunExecution['report'] }; +} + +export async function daemonUnresponsiveReport(command: RunCommand, + base: CheckReport | RunReport, socketPath: string, error: unknown, fallbackRunId?: string, + options: RunLifecycleOptions = {}, dataDir?: string): Promise { + const runId = error instanceof AuthoredFlowExecutionError ? error.rootRunId ?? fallbackRunId : fallbackRunId; + const reachable = (await probeSocket(socketPath)).reachable; + const kind = reachable ? 'daemon_unresponsive' : 'daemon_unreachable'; + const detail = error instanceof JournalReadInterruptedError + ? (reachable ? 'relayflowd answers a fresh connection but the run read session kept being interrupted.' + : 'relayflowd did not answer a fresh connection after the run read session was interrupted; it may be unreachable or delayed by CPU load.') + : reachable ? 'relayflowd answers a fresh connection but the run read timed out; CPU load may be delaying it.' + : 'relayflowd did not answer a fresh connection; it may be unreachable or delayed by CPU load.'; + return { exitCode: 1, report: { + ...base, command, ok: false, socketPath, status: 'running', + ...(runId === undefined ? {} : { runId, rootRunId: runId }), + diagnostics: [...base.diagnostics, { severity: 'failure', kind, + message: `${detail} ${error instanceof Error ? error.message : String(error)}` + + (runId === undefined ? '' : ` The run remains resumable. Continue with: ${resumeCommand(runId, dataDir ?? options.dataDir, options.localAgent === true)}.`), + }], + } }; +} diff --git a/packages/sdk/src/cli/reuse.ts b/packages/sdk/src/cli/reuse.ts index cc9df22ba..0fae54c61 100644 --- a/packages/sdk/src/cli/reuse.ts +++ b/packages/sdk/src/cli/reuse.ts @@ -1,4 +1,5 @@ import type { JournalClient } from '../journal-client.js'; +import type { RunExecution } from './run.js'; /** Count durable facts, including failed executions, across paginated reads. */ export async function reuseSummary(client: JournalClient, runId: string, fromRunId: string): Promise<{ @@ -21,3 +22,27 @@ export async function reuseSummary(client: JournalClient, runId: string, fromRun } return { fromRunId, reusedSteps: reused.size, executedSteps: executed.size }; } + +/** + * Add the reuse summary to an already classified run. The summary is an extra + * journal read after the outcome is known: if it cannot be answered, report + * that as a warning and keep the classified outcome rather than recasting a + * finished run as resumable or failed. + */ +export async function attachReuseSummary(execution: RunExecution, client: JournalClient, + runId: string, fromRunId: string): Promise { + // Classification already ended on an unanswered read: the run is resumable and + // another read would only spend a fresh budget against the same delayed daemon. + if (execution.report.diagnostics.some(diagnostic => diagnostic.kind === 'daemon_unresponsive' + || diagnostic.kind === 'daemon_unreachable' || diagnostic.kind === 'inspection_interrupted')) return execution; + try { + execution.report.reuse = await reuseSummary(client, runId, fromRunId); + } catch (error) { + // Optional detail after the outcome is decided: no failure here — an + // interruption, a cancellation, a bad frame — may replace that outcome. + const why = error instanceof Error ? error.message : String(error); + execution.report.diagnostics = [...execution.report.diagnostics, { severity: 'warning', kind: 'reuse_summary_unavailable', + message: `The run's outcome is final, but the summary of steps reused from ${fromRunId} could not be read: ${why}` }]; + } + return execution; +} diff --git a/packages/sdk/src/cli/run.ts b/packages/sdk/src/cli/run.ts index 5ec492868..20733bfe6 100644 --- a/packages/sdk/src/cli/run.ts +++ b/packages/sdk/src/cli/run.ts @@ -1,10 +1,14 @@ +import { waitForRunningStep, type RunningStep } from './running-step.js'; +export { LEASE_SWEEP_GRACE_MS } from './running-step.js'; +import { completedResultUnreadableReport, daemonUnresponsiveReport, isReadInterruption } from './journal-timeout.js'; +import { FLOW_READ_BUDGET_MS } from '../journal-read-policy.js'; import { onWorkerFailure } from '../worker-lease.js'; import { communicationInstruction } from '../communication/spec.js'; import { checkCommunicationEnvironment, CommunicationEnvironmentError } from '../communication/preflight.js'; import { parseHumanRecipient } from '../human-to.js'; import { parseDigestReference } from '../bundle-transport.js'; import { prepareDigestRun } from './run-digest.js'; -import { reuseSummary } from './reuse.js'; +import { attachReuseSummary } from './reuse.js'; import { resumeHelperEffect } from '../authored-helper-effect.js'; import { AuthoredFlowExecutionError, AuthoredHumanParked, type AuthoredHumanWait } from '../authored-flow-error.js'; import { answerCommand, resumeCommand } from '../authored-human.js'; @@ -203,7 +207,8 @@ async function executeCheckedFlow( catch (error) { return { exitCode: 2, report: { ...base, diagnostics: [...base.diagnostics, { severity: 'refusal', kind: 'probe_failed', message: error instanceof Error ? error.message : 'Communication environment could not be checked.' }] } }; } } - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS, + ...(options.signal === undefined ? {} : { readSignal: options.signal }) }); const connected = await connect(client, 'run', dataDir, base, options); if (connected !== undefined) return connected; @@ -225,6 +230,7 @@ async function executeCheckedFlow( } if (options.onJournalEntry !== undefined) client.on('entry', options.onJournalEntry); const outcome = await startWatched(client, spec, options); + base.runId = outcome.run_id; // `run.start { watch: true }` is an event stream, not the source of the // root identity. A short deterministic run can finish before its first // watched entry is delivered (and an older daemon may refuse `watch`), so @@ -234,11 +240,10 @@ async function executeCheckedFlow( // truthful "not projected" diagnostic. options.onRunReceipt?.({ runId: outcome.run_id, flow: spec.name ?? 'flow' }); const execution = await classifyOutcome(client, 'run', outcome, base, socketPath, { ...options, dataDir }); - if (options.reuseFromRunId !== undefined) { - execution.report.reuse = await reuseSummary(client, outcome.run_id, options.reuseFromRunId); - } - return execution; + return options.reuseFromRunId === undefined ? execution + : await attachReuseSummary(execution, client, outcome.run_id, options.reuseFromRunId); } catch (error) { + if (isReadInterruption(error)) return daemonUnresponsiveReport('run', base, socketPath, error, base.runId, options, dataDir); if (error instanceof CommunicationEnvironmentError) return { exitCode: 2, report: { ...base, diagnostics: [...base.diagnostics, { severity: 'refusal', kind: 'probe_failed', message: error.message }] } }; if (error instanceof JournalProtocolError && ( @@ -289,7 +294,8 @@ export async function resumeFlow( const agentEnvironment = localAgentEnvironment(); const socketPath = socketFor(dataDir); const base = emptyReport('resume'); - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS, + ...(options.signal === undefined ? {} : { readSignal: options.signal }) }); const connected = await connect(client, 'resume', dataDir, base, options); if (connected !== undefined) return connected; @@ -408,6 +414,10 @@ export async function resumeFlow( if (error instanceof AuthoredFlowExecutionError && (error.code === 'step_failed' || error.code === 'gate_failed')) { return authoredStepFailure('resume', base, socketPath, error, runId); } + if (error instanceof AuthoredFlowExecutionError && error.code === 'result_unreadable') { + return completedResultUnreadableReport('resume', base, socketPath, error); + } + if (isReadInterruption(error)) return daemonUnresponsiveReport('resume', base, socketPath, error, runId, options, dataDir); if (error instanceof AuthoredFlowExecutionError && error.code === 'root_lease_lost') { return rootLeaseLostReport('resume', base, socketPath, error, runId); } @@ -822,6 +832,19 @@ export async function connect( /// needs a stubbed client rather than a real run -- the integration test that /// found it reproduced the bug roughly 1 time in 12. export async function classifyOutcome( + client: JournalClient, command: RunCommand, outcome: RunOutcome, + base: CheckReport | RunReport, socketPath: string, options: RunLifecycleOptions, +): Promise { + try { return await classifyOutcomeInner(client, command, outcome, base, socketPath, options); } + catch (error) { + // Authored child calls have no CLI socket path: let their durable root + // handle the interruption, rather than turn it into a child step failure. + if (socketPath !== '' && isReadInterruption(error)) return daemonUnresponsiveReport(command, base, socketPath, error, outcome.run_id, options); + throw error; + } +} + +async function classifyOutcomeInner( client: JournalClient, command: RunCommand, outcome: RunOutcome, @@ -929,6 +952,7 @@ export async function classifyOutcome( kind: current.completion_reason, message: `Run "${current.run_id}" failed with completionReason: ${current.completion_reason}.`, }; + let interrupted: RunDiagnostic | undefined; if (current.completion_reason === 'step_failed') { let details: StepFailedDetails | undefined; try { @@ -936,6 +960,10 @@ export async function classifyOutcome( } catch (error) { // Inspection must not erase the already known run failure. diagnostic.message += ` Could not inspect the failed step: ${errorMessage(error)}`; + // Mark an unanswered read so optional follow-up reads do not spend a + // fresh budget on the same delayed daemon. + if (isReadInterruption(error)) interrupted = { severity: 'warning', kind: 'inspection_interrupted', + message: `The failed step could not be inspected: ${errorMessage(error)}` }; } if (details !== undefined) { Object.assign(diagnostic, details); @@ -952,7 +980,7 @@ export async function classifyOutcome( exitCode: 1, report: { ...report, - diagnostics: [...report.diagnostics, diagnostic], + diagnostics: [...report.diagnostics, diagnostic, ...(interrupted === undefined ? [] : [interrupted])], }, }; } @@ -1021,10 +1049,6 @@ interface OutOfBandInspection { backoffStep?: ParkedStep; } -interface RunningStep extends ParkedStep { - leaseDeadlineMs: number; -} - // Bound on re-polling a run that reports `running` with no identifiable step. // 40 x 50ms = 2s, far longer than the sub-second window observed in #179, and // short enough that a genuinely stuck run still reports rather than hangs. @@ -1071,63 +1095,7 @@ async function inspectOutOfBandStep( }; } -// Match kernel/relayflowd/src/server/client.rs: allow the lease sweep to dispatch a retry. -export const LEASE_SWEEP_GRACE_MS = 5_000; -// How often to re-ask the daemon about a step whose lease looks expired here. const EXPIRED_LEASE_POLL_MS = 250; -// A deadline unchanged for a whole lease (relayflowd LEASE_RENEWAL_MS, 30s) -// plus the sweep grace was renewed by nobody: the attempt is dead and the -// daemon never swept it. Fail rather than hang. -const STALE_LEASE_MS = 30_000 + LEASE_SWEEP_GRACE_MS; - -async function waitForRunningStep( - client: JournalClient, - runId: string, - runningStep: RunningStep, - options: RunLifecycleOptions, -): Promise { - let leaseDeadlineMs = runningStep.leaseDeadlineMs; - if (!Number.isFinite(leaseDeadlineMs)) { - throw new Error(`running step "${runningStep.id}" omitted lease_deadline_ms`); - } - options.onWait?.({ - runId, - stepId: runningStep.id, - stepType: runningStep.type, - leaseDeadlineMs, - }); - // When the lease deadline last changed, on this process's monotonic clock. - // A live worker renews every ~10s, moving the deadline; a dead lease the - // daemon has not swept keeps it. That -- not a comparison of the daemon's - // deadline with our wall clock -- is what separates the two under skew. - let deadlineSeenAt = performance.now(); - while (true) { - throwIfCanceled(options.signal, runningStep.id); - const remainingMs = leaseDeadlineMs + LEASE_SWEEP_GRACE_MS - Date.now(); - if (performance.now() - deadlineSeenAt > STALE_LEASE_MS) { - throw new Error( - `worker lease for step "${runningStep.id}" expired at ${leaseDeadlineMs} without completion`, - ); - } - // Past the deadline by THIS clock alone is not an expiry: the daemon may - // still hold the lease (skew) or be sweeping it into a retry. Keep - // following the step, at a slower poll, until the daemon says otherwise. - await delay(remainingMs <= 0 ? EXPIRED_LEASE_POLL_MS : Math.min(50, remainingMs), options.signal); - const snapshot = await client.runGet(runId); - const step = snapshot.steps[runningStep.id]; - if (step?.state !== 'running') return; - if (step.lease_deadline_ms !== undefined && step.lease_deadline_ms !== leaseDeadlineMs) { - leaseDeadlineMs = step.lease_deadline_ms; - deadlineSeenAt = performance.now(); - options.onWait?.({ - runId, - stepId: runningStep.id, - stepType: runningStep.type, - leaseDeadlineMs, - }); - } - } -} export function protocolFailure( command: RunCommand, diff --git a/packages/sdk/src/cli/running-step.ts b/packages/sdk/src/cli/running-step.ts new file mode 100644 index 000000000..2b3ca12cf --- /dev/null +++ b/packages/sdk/src/cli/running-step.ts @@ -0,0 +1,140 @@ +import { JournalFrameError, JournalProtocolError, type JournalClient } from '../journal-client.js'; +import { AuthoredFlowExecutionError } from '../authored-flow-error.js'; +import type { JournalEvent } from '../journal-reader.js'; +import type { RunLifecycleOptions, ParkedStep } from './run.js'; + +export const LEASE_SWEEP_GRACE_MS = 5_000; +const LEASE_POLL_MS = 2_000; +const STALE_LEASE_MS = 30_000 + LEASE_SWEEP_GRACE_MS; +export interface RunningStep extends ParkedStep { leaseDeadlineMs: number } + +/** Watch completion; snapshots only refresh the non-journaled live lease deadline. */ +export async function waitForRunningStep(client: JournalClient, runId: string, + runningStep: RunningStep, options: RunLifecycleOptions): Promise { + let leaseDeadlineMs = runningStep.leaseDeadlineMs; + if (!Number.isFinite(leaseDeadlineMs)) { + throw new Error(`running step "${runningStep.id}" omitted lease_deadline_ms`); + } + const notify = () => options.onWait?.({ runId, stepId: runningStep.id, + stepType: runningStep.type, leaseDeadlineMs }); + notify(); + let deadlineSeenAt = performance.now(); + // The protocol has no unwatch. Closing this scoped, non-worker session removes + // the server watcher, even on cancellation or a read/registration failure. + const watch = client.createPeer(); + let settled = false; + let ready = false; + let finish: () => void = () => {}; + let cancel: () => void = () => {}; + let fail: (error: unknown) => void = () => {}; + const canceled = () => new Error(`worker wait for step "${runningStep.id}" was canceled`); + const completed = new Promise((resolve, reject) => { + finish = resolve; + cancel = () => reject(canceled()); + fail = reject; + }); + // The subscription handshake itself may be pending when cancellation arrives. + void completed.catch(() => {}); + // A completion push aborts the lease snapshot in flight: the answer is known. + let snapshotRead: AbortController | undefined; + const onEntry = (entry: JournalEvent) => { + if (entry.run_id !== runId) return; + if (entry.entry_type === 'run.completed') settled = true; + else if (entry.step_id === runningStep.id) { + if (entry.entry_type === 'step.attempt.started') settled = false; + if (entry.entry_type === 'step.completed' || entry.entry_type === 'wait.human') settled = true; + } + // Fold the entire replay before deciding: an older attempt may have failed. + if (ready && settled) { + finish(); + snapshotRead?.abort(new Error(`step "${runningStep.id}" completed during the lease snapshot`)); + } + }; + watch.on('entry', onEntry); + // A malformed frame after registration has no pending request to reject: fail closed here. + let protocolFailure: unknown; + const onProtocolError = (error: unknown) => { protocolFailure ??= error; fail(error); snapshotRead?.abort(error); }; + watch.on('protocol_error', onProtocolError); + // A dropped watch loses its completion pushes: a resumable read interruption. + const onDisconnected = (error: Error) => { + if (settled) return; + const lost = new AuthoredFlowExecutionError('daemon_unresponsive', + `the completion watch for step "${runningStep.id}" disconnected: ${error.message}`); + protocolFailure ??= lost; + fail(lost); + snapshotRead?.abort(lost); + }; + watch.on('disconnected', onDisconnected); + options.signal?.addEventListener('abort', cancel, { once: true }); + if (options.signal?.aborted) cancel(); + try { + // Every transport failure up to and including the watch registration is a + // read interruption: the step may still be running and the run resumable. + let unconfirmed: unknown; + await Promise.race([watch.connect().then(() => watch.hello('flows-step-watch')) + .then(() => watch.runWatch(runId)).catch(error => { + // The replay precedes the response, but a drop may have truncated it + // (an older completion, the retry not yet seen): confirm below. + if (settled) { unconfirmed = error; return; } + // Refusals and malformed frames are protocol facts, not lost transport: fail closed. + if (error instanceof JournalProtocolError || error instanceof JournalFrameError) throw error; + throw new AuthoredFlowExecutionError('daemon_unresponsive', + `could not establish the completion watch: ${error instanceof Error ? error.message : String(error)}`); + }), completed]); + if (unconfirmed !== undefined) { + const snapshot = await client.runGet(runId, options.signal === undefined ? {} : { signal: options.signal }); + if (snapshot.steps[runningStep.id]?.state !== 'running') return; + throw new AuthoredFlowExecutionError('daemon_unresponsive', + `could not establish the completion watch: ${unconfirmed instanceof Error ? unconfirmed.message : String(unconfirmed)}`); + } + ready = true; + if (settled) return; + while (true) { + let timer: ReturnType | undefined; + let releaseTimer: () => void = () => {}; + try { + const elapsed = await Promise.race([ + new Promise(resolve => { releaseTimer = () => resolve(false); timer = setTimeout(() => resolve(true), LEASE_POLL_MS); }), + completed.then(() => false), + ]); + if (!elapsed) return; + } finally { clearTimeout(timer); releaseTimer(); } + // Drain the read before leaving the authored promise scope. A raced but + // unresolved read would look like unawaited derived work to that scope, + // so cancellation aborts the read itself and it settles at once. + const read = snapshotRead = new AbortController(); + const signal = options.signal === undefined ? read.signal + : AbortSignal.any([options.signal, read.signal]); + const snapshot = await client.runGet(runId, { signal }) + .catch(error => { + if (options.signal?.aborted) throw canceled(); + if (protocolFailure !== undefined) throw protocolFailure; + // A completion push aborted this read: completion was observed, and + // finish() is one-shot, even if a retry attempt started right after. + if (settled || read.signal.aborted) return undefined; + throw error; + }) + .finally(() => { snapshotRead = undefined; }); + if (options.signal?.aborted) throw canceled(); + if (protocolFailure !== undefined) throw protocolFailure; + if (settled || snapshot === undefined || snapshot.steps[runningStep.id]?.state !== 'running') return; + const deadline = snapshot.steps[runningStep.id]?.lease_deadline_ms; + if (deadline !== undefined && deadline !== leaseDeadlineMs) { + leaseDeadlineMs = deadline; + deadlineSeenAt = performance.now(); + notify(); + } else if (performance.now() - deadlineSeenAt > STALE_LEASE_MS) { + // A stalled sweep/renewal is not evidence that the body itself failed. + throw new AuthoredFlowExecutionError('daemon_unresponsive', + `worker lease for step "${runningStep.id}" expired at ${leaseDeadlineMs} without completion; relayflowd's lease sweep may be delayed by CPU load`); + } + } + } finally { + finish(); + options.signal?.removeEventListener('abort', cancel); + watch.off('entry', onEntry); + watch.off('protocol_error', onProtocolError); + watch.off('disconnected', onDisconnected); + watch.close(); + } +} diff --git a/packages/sdk/src/communication/local.ts b/packages/sdk/src/communication/local.ts index 60b7f4440..0b4dfa7e6 100644 --- a/packages/sdk/src/communication/local.ts +++ b/packages/sdk/src/communication/local.ts @@ -1,5 +1,6 @@ import { onWorkerFailure } from '../worker-lease.js'; import { randomUUID } from 'node:crypto'; +import { FLOW_READ_BUDGET_MS } from '../journal-read-policy.js'; import { JournalClient } from '../journal-client.js'; import { AgentWorker } from '../worker.js'; import type { KernelRunSpec } from '../spec.js'; @@ -18,6 +19,9 @@ export async function attachCommunicationWorkers(spec: KernelRunSpec, socketPath let failure: unknown; const close = async () => { const outcomes = await Promise.allSettled(workers.map(async ({ client, worker }) => { + // Stop budgeted reads first: draining the worker must not wait out a + // delayed history read on shutdown. + client.cancelReads(); try { await worker.close(); } finally { client.close(); } })); const rejected = outcomes.find(result => result.status === 'rejected'); @@ -28,7 +32,8 @@ export async function attachCommunicationWorkers(spec: KernelRunSpec, socketPath if (step.type !== 'agent') continue; requireCommunicationCli(step.cli ?? spec.cli); if (step.surfaces?.workspace?.length) throw new Error('Local communication workers support stream surfaces only'); - const client = new JournalClient(socketPath); + // History reads on a retried attempt meet the same CPU load as flow reads. + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS }); const worker = new AgentWorker(client, { workerId: `communication-${randomUUID()}`, dataDir, environment, requiredStreams: [channelName(step.id, '$receipts')], pins: { workspace: [], streams: step.surfaces?.streams?.map(({ stream }) => ({ stream, read_offset: 0 })) ?? [] } }); diff --git a/packages/sdk/src/communication/worker.ts b/packages/sdk/src/communication/worker.ts index 410af0510..a485e9863 100644 --- a/packages/sdk/src/communication/worker.ts +++ b/packages/sdk/src/communication/worker.ts @@ -5,7 +5,8 @@ import { setTimeout as delay } from 'node:timers/promises'; import type { JournalClient } from '../journal-client.js'; import type { StepDispatchEvent } from '../protocol.js'; import type { ResolvedKernelAgentStep } from '../resolved-cli-identity.js'; -import { withWorkerLease } from '../worker-lease.js'; +import { isLeaseLost, withWorkerLease } from '../worker-lease.js'; +import { isReadInterruptionError } from '../journal-read-policy.js'; import { workerInstruction } from '../worker-input.js'; import { resolveCliModel } from '../cli-adapter.js'; import { resolveAgentCwd } from '../agent-cwd.js'; @@ -26,9 +27,16 @@ export async function completeCommunicationDispatch(client: JournalClient, dispa let output: unknown; let completionReason: 'success' | 'worker_error' = 'success'; try { - output = await withWorkerLease(client, dispatch, signal => - run(client, dispatch, instruction, spec, dataDir, signal, runRoot, environment)); + output = await withWorkerLease(client, dispatch, signal => { + // A lost lease stops this attempt's history reads instead of letting + // them run out the budget beside the kernel's retry. + const releaseReads = client.scopeReads?.(signal) ?? (() => {}); + return run(client, dispatch, instruction, spec, dataDir, signal, runRoot, environment).finally(releaseReads); + }); } catch (error) { + // A lost lease or an unanswered read belongs to the kernel's sweep and + // retry: never complete the attempt here as a worker failure. + if (isLeaseLost(error) || isReadInterruptionError(error)) throw error; completionReason = 'worker_error'; output = { error: error instanceof Error ? error.message : String(error) }; } diff --git a/packages/sdk/src/failure-kinds.ts b/packages/sdk/src/failure-kinds.ts index 7a05c98a2..5050806e9 100644 --- a/packages/sdk/src/failure-kinds.ts +++ b/packages/sdk/src/failure-kinds.ts @@ -149,6 +149,7 @@ export const RUN_FAILURE_KINDS = [ * reach this process. The run is NOT failed (`status: running`): its * completed steps are journaled and `flows resume ` continues it. */ + 'daemon_unresponsive', 'root_lease_lost', ] as const; @@ -160,6 +161,9 @@ export const RUN_FAILURE_KINDS = [ */ export const RUN_WARNING_KINDS = [ 'connection_file_stale', + 'reuse_summary_unavailable', + 'result_unreadable', + 'inspection_interrupted', ] as const; /** diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts index 50909fa8b..d69f03a0d 100644 --- a/packages/sdk/src/index.ts +++ b/packages/sdk/src/index.ts @@ -217,7 +217,7 @@ export type { } from './protocol.js'; export { JOURNAL_WRITE_FAILED, PROTOCOL_VERSION } from './protocol.js'; -export { JournalClient, type JournalClientOptions } from './journal-client.js'; +export { JournalClient, JournalRequestTimeoutError, type JournalClientOptions } from './journal-client.js'; export { AgentWorker, type AgentWorkerOptions } from './worker.js'; export { diff --git a/packages/sdk/src/journal-budgeted-reads.ts b/packages/sdk/src/journal-budgeted-reads.ts new file mode 100644 index 000000000..01ffbc4f3 --- /dev/null +++ b/packages/sdk/src/journal-budgeted-reads.ts @@ -0,0 +1,134 @@ +import { JournalFrameError, JournalProtocolError } from './journal-connection.js'; +import type { JournalClient } from './journal-client.js'; +import { JournalReadInterruptedError, JournalReadPolicy, JournalRequestTimeoutError } from './journal-read-policy.js'; +import type { VerbContract } from './protocol.js'; + +/** + * Budgeted, retried reads for one flow client, on a dedicated reader session. + * + * Reads go to a second connection so they never queue behind the primary's + * unbounded commands; the policy serializes them and retries timeouts and lost + * reader transport within the budget. Protocol refusals and malformed frames + * fail closed. A refused reader handshake falls back to the primary session, + * whose own failures keep their meaning because it also carries writes. + */ +export class BudgetedReads { + private readonly policy = new JournalReadPolicy(); + private reader: JournalClient | undefined; + private readerReady: Promise | undefined; + private readonly scoped = new Set(); + + constructor( + private readonly primary: JournalClient, + private readonly budgetMs: number, + private readonly lifecycle?: AbortSignal, + ) {} + + /** Add an abort source (a root attempt's lease) for as long as the returned release is not called. */ + scope(signal: AbortSignal): () => void { + this.scoped.add(signal); + return () => { this.scoped.delete(signal); }; + } + + /** + * Stop all reads. An unexpected primary drop (`interrupted`) ends in-flight + * reads as a read interruption, so the run stays resumable; a caller close + * keeps its plain error. + */ + close(cause?: unknown, interrupted = false): void { + this.policy.close(interrupted ? new JournalReadInterruptedError('journal', 1, 0, undefined, { cause }) : undefined); + this.reader?.close(cause); + } + + read(verb: V, params: VerbContract[V]['params'], timeoutMs: number, + signal?: AbortSignal): Promise { + const sources = [signal, this.lifecycle, ...this.scoped].filter((source): source is AbortSignal => source !== undefined); + const combined = sources.length <= 1 ? sources[0] : AbortSignal.any(sources); + return this.policy.read(verb, timeoutMs, this.budgetMs, async (bound, attemptSignal) => { + const started = performance.now(); + const reader = await this.untilAborted(this.session(), attemptSignal).catch(error => { + attemptSignal.throwIfAborted(); + // A malformed handshake frame is a protocol violation: fail closed. + if (error instanceof JournalFrameError) throw error; + throw new JournalReadInterruptedError(verb, 1, performance.now() - started, undefined, { cause: error }); + }); + attemptSignal.throwIfAborted(); + const remaining = bound - (performance.now() - started); + if (remaining <= 0) throw new JournalRequestTimeoutError(verb, bound); + if (reader === undefined) return this.primary.requestOnce(verb, params, remaining, attemptSignal); + try { + return await reader.requestOnce(verb, params, remaining, attemptSignal); + } catch (error) { + // relayflowd answers one frame at a time per connection, so a canceled + // request still occupies this reader: drop it for a fresh session. + if (attemptSignal.aborted) { this.drop(reader); throw error; } + // A timed-out request still occupies this sequential connection: retry + // on a fresh reader rather than queueing behind it. + if (error instanceof JournalRequestTimeoutError) { this.drop(reader); throw error; } + if (error instanceof JournalProtocolError || error instanceof JournalFrameError) throw error; + // Only the dedicated reader's transport failed; reconnect within the budget. + this.drop(reader); + throw new JournalReadInterruptedError(verb, 1, performance.now() - started, undefined, { cause: error }); + } + }, combined); + } + + /** The primary session's own `hello`, retried within the same budget as reads. */ + handshake(params: VerbContract['hello']['params'], timeoutMs: number): Promise { + return this.policy.read('hello', timeoutMs, this.budgetMs, + (bound, attemptSignal) => this.primary.requestOnce('hello', params, bound, attemptSignal).catch(error => { + if (attemptSignal.aborted || error instanceof JournalRequestTimeoutError + || error instanceof JournalProtocolError || error instanceof JournalFrameError) throw error; + // The primary's own transport failed (ECONNRESET, ...): a read + // interruption, not a body failure, and not retryable on this session. + throw new JournalReadInterruptedError('hello', 1, 0, undefined, { cause: error, retryable: false }); + }), this.lifecycle); + } + + private session(): Promise { + if (this.readerReady === undefined) { + const reader = this.primary.createPeer(); + this.reader = reader; + this.readerReady = reader.connect().then(() => reader.hello('flows-reader')).then(() => reader) + .catch(error => { + reader.close(); + // A daemon that refuses the session will keep refusing it: read on the primary. + if (error instanceof JournalProtocolError) return undefined; + // A connect or hello that failed under load is retried with a fresh session, + // never by queueing reads behind the primary's unbounded commands for good. + if (this.reader === reader) { + this.reader = undefined; + this.readerReady = undefined; + } + throw error; + }); + } + return this.readerReady; + } + + /** + * Settle with the session, or at once with the attempt's abort — abandoning a + * setup still in progress, so the serialized queue never waits on it. + */ + private untilAborted(setup: Promise, signal: AbortSignal): Promise { + return new Promise((resolve, reject) => { + const onAbort = () => { + const pending = this.reader; + if (pending !== undefined) this.drop(pending); + reject(signal.reason); + }; + if (signal.aborted) { onAbort(); return; } + signal.addEventListener('abort', onAbort, { once: true }); + setup.then(value => { signal.removeEventListener('abort', onAbort); resolve(value); }, + error => { signal.removeEventListener('abort', onAbort); reject(error); }); + }); + } + + /** Forget a reader whose transport failed so the next attempt reconnects one. */ + private drop(reader: JournalClient): void { + if (this.reader !== reader) return; + this.reader = undefined; + this.readerReady = undefined; + reader.close(); + } +} diff --git a/packages/sdk/src/journal-client.ts b/packages/sdk/src/journal-client.ts index 9fe998eed..4eebb0233 100644 --- a/packages/sdk/src/journal-client.ts +++ b/packages/sdk/src/journal-client.ts @@ -7,26 +7,36 @@ // AGENTS.md rule 4). The kernel binary (`kernel/relayflowd serve`) speaks // this transport, including out-of-band leases, watches, events, and durable // stream plumbing. The client's framing and failure behavior is covered by a -// loopback double in tests. - -import { EventEmitter } from 'node:events'; +// loopback double in tests. Framing and the socket lifecycle live in +// journal-connection.ts; budgeted reads in journal-budgeted-reads.ts. + +import { READ_ONLY_VERBS } from './journal-read-policy.js'; +import { JournalConnection, JournalProtocolError } from './journal-connection.js'; +export { JournalFrameError, JournalProtocolError } from './journal-connection.js'; +import { BudgetedReads } from './journal-budgeted-reads.js'; +export { JournalReadInterruptedError, JournalRequestTimeoutError } from './journal-read-policy.js'; export { walkJournal, JournalReadError, type JournalEvent, type JournalReadFailure } from './journal-reader.js'; -import { randomUUID } from 'node:crypto'; -import { createConnection, type Socket } from 'node:net'; import type { VerbContract, EventEmitParams, EventSubmitParams, ReportedCost } from './protocol.js'; import { PROTOCOL_VERSION, type CompletionReason, type EffectRef, type Pins, - type Request, - type Response, - type ServerEvent, type StepUsage, } from './protocol.js'; import type { KernelRunSpec, StepType } from './spec.js'; export interface JournalClientOptions { + /** Opt-in total budget for read-only requests; interactive clients stay single-shot. */ + readBudgetMs?: number; + /** Cancels every budgeted read, so a run's lifecycle abort never waits out the budget. */ + readSignal?: AbortSignal; + /** + * Retry this session's own `hello` within the read budget. Only for sessions + * opened after a run is admitted (an authored child, its verifier); a + * top-level command's handshake stays a single-shot compatibility check. + */ + budgetHandshake?: boolean; /** Override the timeout for bounded protocol requests (ms). Default 30000. */ requestTimeoutMs?: number; /** @@ -42,42 +52,21 @@ export interface JournalClientOptions { connectTimeoutMs?: number; } -interface Pending { - resolve: (value: unknown) => void; - reject: (err: Error) => void; - timer?: ReturnType; -} - -/** A structured rejection returned by relayflowd over the journal protocol. */ -export class JournalProtocolError extends Error { - readonly code: string; - verb?: string; +export class JournalClient extends JournalConnection { + private readonly budgeted: BudgetedReads | undefined; + private readonly budgetHandshake: boolean; + /** Additive request capabilities the daemon advertised at `hello`; none until then. */ + private features: ReadonlySet = new Set(); - constructor(code: string, message: string) { - super(`${code}: ${message}`); - this.name = 'JournalProtocolError'; - this.code = code; + constructor(socketPath: string, options: JournalClientOptions = {}) { + super(socketPath, options.requestTimeoutMs ?? 30_000, options.connectTimeoutMs ?? 2_000); + this.budgetHandshake = options.budgetHandshake === true; + this.budgeted = options.readBudgetMs === undefined ? undefined + : new BudgetedReads(this, options.readBudgetMs, options.readSignal); } -} -export class JournalClient extends EventEmitter { - /** Additive request capabilities the daemon advertised at `hello`; none until then. */ - private features: ReadonlySet = new Set(); - private socket: Socket | null = null; - /** Why the connection ended, so a later "not connected" names its cause rather than hiding it. */ - private disconnectCause: Error | undefined; - private buffer = ''; - private readonly pending = new Map(); - private readonly requestTimeoutMs: number; - private readonly connectTimeoutMs: number; - - constructor( - readonly socketPath: string, - options: JournalClientOptions = {}, - ) { - super(); - this.requestTimeoutMs = options.requestTimeoutMs ?? 30_000; - this.connectTimeoutMs = options.connectTimeoutMs ?? 2_000; + protected override connectionClosed(cause: unknown, unexpected: boolean): void { + this.budgeted?.close(cause, unexpected); } /** Independent session for an SDK helper worker; preserves the caller's registration. */ @@ -87,139 +76,41 @@ export class JournalClient extends EventEmitter { }); } - /** Open the unix socket connection. Rejects on connect failure (fail-closed). */ - connect(): Promise { - return new Promise((resolve, reject) => { - if (this.socket) return resolve(); - const socket = createConnection({ path: this.socketPath }); - const timer = setTimeout(() => { - socket.removeAllListeners(); - socket.destroy(); - this.failAll(new Error(`journal client: connect timed out after ${this.connectTimeoutMs}ms`)); - reject(new Error(`journal client: connect timed out after ${this.connectTimeoutMs}ms`)); - }, this.connectTimeoutMs); - const onError = (err: Error): void => { - clearTimeout(timer); - socket.removeAllListeners(); - this.failAll(err); - reject(new Error(`journal client: connect failed: ${err.message}`)); - }; - socket.once('error', onError); - socket.once('connect', () => { - clearTimeout(timer); - socket.removeListener('error', onError); - socket.on('error', (err) => { this.disconnectCause ??= err; this.failAll(err); }); - socket.on('data', (chunk) => this.onData(chunk)); - socket.on('close', () => { - const closed = new Error('journal client: connection closed'); - this.disconnectCause ??= closed; - this.failAll(closed); - }); - this.disconnectCause = undefined; - this.socket = socket; - resolve(); - }); - }); + private request(verb: V, params: VerbContract[V]['params'], + timeoutMs: number | null = this.requestTimeoutMs, signal?: AbortSignal): Promise { + // Every allowlisted read takes the policy, even after the primary dropped: its + // closed state then answers with the typed interruption recorded at the drop. + if (this.budgeted !== undefined && timeoutMs !== null && READ_ONLY_VERBS.has(verb)) { + return this.budgeted.read(verb, params, timeoutMs, signal); + } + return this.requestOnce(verb, params, timeoutMs, signal); } /** - * Close the connection and reject any pending requests. A `cause` (for - * example the worker failure that forced the close) is carried into every - * later "not connected" rejection. + * Cancel budgeted reads also when `signal` aborts — a root attempt's lease — + * until the returned release is called. A no-op without a read budget. */ - close(cause?: unknown): void { - const closed = cause === undefined - ? new Error('journal client: closed by caller') - : new Error(`journal client: closed after ${cause instanceof Error ? cause.message : String(cause)}`, { cause }); - this.disconnectCause ??= closed; - this.failAll(closed); - this.socket?.destroy(); - this.socket = null; - this.buffer = ''; - } - - private onData(chunk: Buffer): void { - this.buffer += chunk.toString('utf8'); - let nl: number; - while ((nl = this.buffer.indexOf('\n')) !== -1) { - const line = this.buffer.slice(0, nl); - this.buffer = this.buffer.slice(nl + 1); - if (line.length > 0) this.onLine(line); - } + scopeReads(signal: AbortSignal): () => void { + return this.budgeted?.scope(signal) ?? (() => {}); } - private onLine(line: string): void { - let msg: Response | ServerEvent; - try { - msg = JSON.parse(line) as Response | ServerEvent; - } catch { - // A malformed frame is a protocol violation; fail closed. - this.failAll(new Error('journal client: malformed frame from server')); - return; - } - - if (typeof (msg as Response).id === 'string' && 'ok' in (msg as Response)) { - const res = msg as Response; - const pending = this.pending.get(res.id); - if (!pending) return; // reply for an already-timed-out request - this.pending.delete(res.id); - if (pending.timer !== undefined) clearTimeout(pending.timer); - if (res.ok) pending.resolve(res.result); - else pending.reject(new JournalProtocolError(res.error.code, res.error.message)); - } else { - const ev = msg as ServerEvent; - this.emit(ev.event, ev.data); - this.emit('event', ev); - } - } - - private failAll(err: Error): void { - for (const [, p] of this.pending) { - if (p.timer !== undefined) clearTimeout(p.timer); - p.reject(err); - } - this.pending.clear(); - } - - private request( - verb: V, - params: VerbContract[V]['params'], - timeoutMs: number | null = this.requestTimeoutMs, - ): Promise { - return new Promise((resolve, reject) => { - if (!this.socket || this.socket.destroyed) { - const cause = this.disconnectCause; - reject(new Error( - `journal client: not connected (${verb})${cause === undefined ? '' : `: ${cause.message}`}`, - cause === undefined ? undefined : { cause }, - )); - return; - } - const id = randomUUID(); - const frame: Request = { id, verb: verb as string, params }; - const timer = timeoutMs === null ? undefined : setTimeout(() => { - this.pending.delete(id); - reject(new Error(`journal client: ${verb} timed out after ${timeoutMs}ms`)); - }, timeoutMs); - this.pending.set(id, { resolve: resolve as (v: unknown) => void, reject, timer }); - this.socket.write(JSON.stringify(frame) + '\n', (err) => { - if (err) { - const p = this.pending.get(id); - if (p) { - if (p.timer !== undefined) clearTimeout(p.timer); - this.pending.delete(id); - p.reject(new Error(`journal client: ${verb} write failed: ${err.message}`)); - } - } - }); - }); + /** + * End every budgeted read now and refuse later ones; the session itself stays + * open. The reads end as read interruptions, so a worker hands its attempt + * back to the kernel rather than completing it as a failure. + */ + cancelReads(): void { + this.budgeted?.close(new Error('journal client: reads canceled'), true); } // --- Typed verb methods (gate 1 minimal set, kernel DESIGN.md §5) -------- /** Handshake; version mismatch is a hard error. Records the daemon's additive features. */ async hello(client: string): Promise { - const result = await this.request('hello', { protocol: PROTOCOL_VERSION, client }); + const params = { protocol: PROTOCOL_VERSION, client }; + // A flow client's handshake meets the same CPU load as its reads. + const result = this.budgeted === undefined || !this.budgetHandshake ? await this.request('hello', params) + : await this.budgeted.handshake(params, this.requestTimeoutMs); this.features = new Set(Array.isArray(result.features) ? result.features.filter(f => typeof f === 'string') : []); return result; } @@ -255,13 +146,15 @@ export class JournalClient extends EventEmitter { } /** Snapshot for legibility. */ - runGet(runId: string): Promise { - return this.request('run.get', { run_id: runId }); + runGet(runId: string, options: { signal?: AbortSignal } = {}): Promise { + return this.request('run.get', { run_id: runId }, this.requestTimeoutMs, options.signal); } /** - * Open a push stream of every appended entry. Resolves once subscribed; - * entries arrive as `'entry'` events: `client.on('entry', (entry) => …)`. + * Open a push stream of every appended entry. The daemon writes the run's + * existing entries before this resolves (server.rs `watch_with_replay`), so + * the replay is fully delivered first; later entries arrive as they are + * appended. Entries arrive as `'entry'` events: `client.on('entry', (entry) => …)`. */ runWatch(runId: string): Promise { return this.request('run.watch', { run_id: runId }); diff --git a/packages/sdk/src/journal-connection.ts b/packages/sdk/src/journal-connection.ts new file mode 100644 index 000000000..b2cd78ab2 --- /dev/null +++ b/packages/sdk/src/journal-connection.ts @@ -0,0 +1,222 @@ +// Journal-protocol v0 client (kernel DESIGN.md §5). +// +// A real client implementation of the wire protocol — newline-delimited JSON +// over a unix socket. It correlates requests by `id`, demultiplexes +// server-pushed events, and is fail-closed: a connection drop or write error +// rejects every pending request (a journal write that fails fails the step; +// AGENTS.md rule 4). The kernel binary (`kernel/relayflowd serve`) speaks +// this transport, including out-of-band leases, watches, events, and durable +// stream plumbing. The client's framing and failure behavior is covered by a +// loopback double in tests. + +import { EventEmitter } from 'node:events'; +import { randomUUID } from 'node:crypto'; +import { createConnection, type Socket } from 'node:net'; +import { JournalRequestTimeoutError } from './journal-read-policy.js'; +import type { Request, Response, ServerEvent, VerbContract } from './protocol.js'; + +interface Pending { + resolve: (value: unknown) => void; + reject: (err: Error) => void; + timer?: ReturnType; +} + +/** A structured rejection returned by relayflowd over the journal protocol. */ +export class JournalProtocolError extends Error { + readonly code: string; + verb?: string; + + constructor(code: string, message: string) { + super(`${code}: ${message}`); + this.name = 'JournalProtocolError'; + this.code = code; + } +} + +/** The daemon sent a frame this client cannot parse: a protocol violation, never retried. */ +export class JournalFrameError extends Error { + constructor(message = 'journal client: malformed frame from server') { + super(message); + this.name = 'JournalFrameError'; + } +} + +/** Framing, socket lifecycle and request correlation; the typed verbs live on JournalClient. */ +export class JournalConnection extends EventEmitter { + protected socket: Socket | null = null; + private connectingSocket: Socket | undefined; + /** Why the connection ended, so a later "not connected" names its cause rather than hiding it. */ + protected disconnectCause: Error | undefined; + private buffer = ''; + private readonly pending = new Map(); + + constructor( + readonly socketPath: string, + protected readonly requestTimeoutMs: number, + protected readonly connectTimeoutMs: number, + ) { + super(); + } + + /** Hook for sessions layered on this one: `unexpected` is a drop the caller did not ask for. */ + protected connectionClosed(_cause: unknown, _unexpected: boolean): void {} + + /** Open the unix socket connection. Rejects on connect failure (fail-closed). */ + connect(): Promise { + return new Promise((resolve, reject) => { + if (this.socket) return resolve(); + const socket = createConnection({ path: this.socketPath }); + this.connectingSocket = socket; + const timer = setTimeout(() => { + socket.removeAllListeners(); + socket.destroy(); + this.failAll(new Error(`journal client: connect timed out after ${this.connectTimeoutMs}ms`)); + reject(new Error(`journal client: connect timed out after ${this.connectTimeoutMs}ms`)); + }, this.connectTimeoutMs); + const onError = (err: Error): void => { + clearTimeout(timer); + socket.removeAllListeners(); + this.failAll(err); + reject(new Error(`journal client: connect failed: ${err.message}`)); + }; + socket.once('error', onError); + socket.once('connect', () => { + clearTimeout(timer); + socket.removeListener('error', onError); + socket.on('error', (err) => { this.disconnectCause ??= err; this.failAll(err); }); + socket.on('data', (chunk) => this.onData(chunk)); + socket.on('close', () => { + const closed = new Error('journal client: connection closed'); + this.connectionClosed(closed, this.socket === socket); + this.disconnectCause ??= closed; + this.failAll(closed); + // Only a drop the caller did not ask for; close() clears this.socket first. + // Sessions with nothing pending (a registered watch) must hear it too. + if (this.socket === socket) this.emit('disconnected', closed); + }); + this.disconnectCause = undefined; + this.connectingSocket = undefined; + this.socket = socket; + resolve(); + }); + }); + } + + /** + * Close the connection and reject any pending requests. A `cause` (for + * example the worker failure that forced the close) is carried into every + * later "not connected" rejection. + */ + close(cause?: unknown): void { + const closed = cause === undefined + ? new Error('journal client: closed by caller') + : new Error(`journal client: closed after ${cause instanceof Error ? cause.message : String(cause)}`, { cause }); + this.connectionClosed(cause, false); + this.connectingSocket?.destroy(closed); + this.disconnectCause ??= closed; + this.failAll(closed); + this.socket?.destroy(); + this.socket = null; + this.buffer = ''; + } + + private onData(chunk: Buffer): void { + this.buffer += chunk.toString('utf8'); + let nl: number; + while ((nl = this.buffer.indexOf('\n')) !== -1) { + const line = this.buffer.slice(0, nl); + this.buffer = this.buffer.slice(nl + 1); + if (line.length > 0) this.onLine(line); + } + } + + private onLine(line: string): void { + let msg: Response | ServerEvent; + try { + msg = JSON.parse(line) as Response | ServerEvent; + } catch { + // A malformed frame is a protocol violation; fail closed. + const error = new JournalFrameError(); + this.failAll(error); + // Sessions with nothing pending (a registered watch) must hear it too. + this.emit('protocol_error', error); + return; + } + + if (typeof (msg as Response).id === 'string' && 'ok' in (msg as Response)) { + const res = msg as Response; + const pending = this.pending.get(res.id); + if (!pending) return; // reply for an already-timed-out request + this.pending.delete(res.id); + if (pending.timer !== undefined) clearTimeout(pending.timer); + if (res.ok) pending.resolve(res.result); + else pending.reject(new JournalProtocolError(res.error.code, res.error.message)); + } else { + const ev = msg as ServerEvent; + this.emit(ev.event, ev.data); + this.emit('event', ev); + } + } + + private failAll(err: Error): void { + for (const [, p] of this.pending) { + if (p.timer !== undefined) clearTimeout(p.timer); + p.reject(err); + } + this.pending.clear(); + } + + /** @internal One unretried request on this session; flow reads go through `request`. */ + requestOnce( + verb: V, + params: VerbContract[V]['params'], + timeoutMs: number | null = this.requestTimeoutMs, + signal?: AbortSignal, + ): Promise { + return new Promise((resolve, reject) => { + if (signal?.aborted) { + reject(signal.reason); + return; + } + if (!this.socket || this.socket.destroyed) { + const cause = this.disconnectCause; + reject(new Error( + `journal client: not connected (${verb})${cause === undefined ? '' : `: ${cause.message}`}`, + cause === undefined ? undefined : { cause }, + )); + return; + } + const id = randomUUID(); + const started = performance.now(); + const frame: Request = { id, verb: verb as string, params }; + // Cancellation forgets the request; a late reply is ignored like a timed-out one. + const onAbort = () => { + const p = this.pending.get(id); + if (p === undefined) return; + if (p.timer !== undefined) clearTimeout(p.timer); + this.pending.delete(id); + p.reject(signal!.reason); + }; + const settle = (fn: (arg: A) => void) => (arg: A) => { + signal?.removeEventListener('abort', onAbort); + fn(arg); + }; + const timer = timeoutMs === null ? undefined : setTimeout(() => { + this.pending.delete(id); + settle(reject)(new JournalRequestTimeoutError(verb, timeoutMs, 1, performance.now() - started)); + }, timeoutMs); + this.pending.set(id, { resolve: settle(resolve as (v: unknown) => void), reject: settle(reject), timer }); + signal?.addEventListener('abort', onAbort, { once: true }); + this.socket.write(JSON.stringify(frame) + '\n', (err) => { + if (err) { + const p = this.pending.get(id); + if (p) { + if (p.timer !== undefined) clearTimeout(p.timer); + this.pending.delete(id); + p.reject(new Error(`journal client: ${verb} write failed: ${err.message}`)); + } + } + }); + }); + } +} diff --git a/packages/sdk/src/journal-read-policy.ts b/packages/sdk/src/journal-read-policy.ts new file mode 100644 index 000000000..14b403b5a --- /dev/null +++ b/packages/sdk/src/journal-read-policy.ts @@ -0,0 +1,123 @@ +import { setMaxListeners } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; + +export const READ_ONLY_VERBS = new Set(['run.get', 'journal.read', 'stream.read', 'subscription.inspect']); +export const FLOW_READ_BUDGET_MS = 300_000; + +export class JournalRequestTimeoutError extends Error { + constructor( + readonly verb: string, + readonly timeoutMs: number, + readonly attempts = 1, + readonly elapsedMs = timeoutMs, + readonly readBudgetMs?: number, + ) { + super(readBudgetMs === undefined + ? `journal client: ${verb} timed out after ${timeoutMs}ms` + : `journal client: ${verb} timed out after ${attempts} attempts in ${Math.round(elapsedMs)}ms (read budget ${readBudgetMs}ms); relayflowd may be delayed by CPU load`); + this.name = 'JournalRequestTimeoutError'; + } +} + +/** + * The dedicated read session lost its transport (closed, reset, not connected) + * while the primary session may still be healthy. Like a timeout, this says + * nothing about the run itself: retry within the budget, then leave the run + * resumable. Protocol refusals and write failures are never this error. + */ +export class JournalReadInterruptedError extends Error { + constructor( + readonly verb: string, + readonly attempts = 1, + readonly elapsedMs = 0, + readonly readBudgetMs?: number, + options?: ErrorOptions & { retryable?: boolean }, + ) { + const cause = options?.cause instanceof Error ? `: ${options.cause.message}` : ''; + super(readBudgetMs === undefined + ? `journal client: ${verb} read session was interrupted${cause}` + : `journal client: ${verb} read session was interrupted after ${attempts} attempts in ${Math.round(elapsedMs)}ms (read budget ${readBudgetMs}ms)${cause}`, + options); + this.name = 'JournalReadInterruptedError'; + this.retryable = options?.retryable !== false; + } + + /** False when the session it ran on is gone, so retrying it cannot help. */ + readonly retryable: boolean; +} + +/** A read that could not be answered — by timeout or lost read transport — not a run failure. */ +export function isReadInterruptionError(error: unknown): error is JournalRequestTimeoutError | JournalReadInterruptedError { + return (error instanceof JournalRequestTimeoutError + && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch' || error.verb === 'hello')) + || error instanceof JournalReadInterruptedError; +} + +/** Serial admission bounds read amplification, including retries, per body client. */ +export class JournalReadPolicy { + private tail: Promise = Promise.resolve(); + private readonly closed = new AbortController(); + + constructor() { + // Concurrent children share this cancellation signal; each waiter removes its listener. + setMaxListeners(0, this.closed.signal); + } + + close(reason: unknown = new Error('journal client: closed by caller')): void { this.closed.abort(reason); } + + async read(verb: string, timeoutMs: number, budgetMs: number, + request: (timeoutMs: number, signal: AbortSignal) => Promise, caller?: AbortSignal): Promise { + const started = performance.now(); + const deadline = started + budgetMs; + let attempts = 0; + let last: unknown; + const exhausted = () => last instanceof JournalReadInterruptedError + ? new JournalReadInterruptedError(verb, attempts, performance.now() - started, budgetMs, { cause: last.cause }) + : new JournalRequestTimeoutError(verb, timeoutMs, attempts, performance.now() - started, budgetMs); + // A caller's cancellation settles this read and drains its queued attempt. + const signal = caller === undefined ? this.closed.signal : AbortSignal.any([this.closed.signal, caller]); + // Budget exhaustion also aborts the queued work, so a stalled setup or + // request does not hold the serialized queue after its caller gave up. + const budget = new AbortController(); + const attemptSignal = AbortSignal.any([signal, budget.signal]); + let timer: ReturnType | undefined; + let onClose: () => void = () => {}; + let deliver: (value: T) => void = () => {}; + let fail: (error: unknown) => void = () => {}; + const response = new Promise((resolve, reject) => { + deliver = resolve; + fail = reject; + timer = setTimeout(() => { const error = exhausted(); budget.abort(error); reject(error); }, budgetMs); + onClose = () => reject(signal.reason); + signal.addEventListener('abort', onClose, { once: true }); + if (signal.aborted) onClose(); + }); + const job = this.tail.then(async () => { + while (true) { + attemptSignal.throwIfAborted(); + const remaining = deadline - performance.now(); + if (remaining <= 0) throw exhausted(); + attempts += 1; + try { + return await request(Math.min(remaining, timeoutMs * 2 ** Math.max(0, attempts - 2)), attemptSignal); + } catch (error) { + attemptSignal.throwIfAborted(); + if (!(error instanceof JournalRequestTimeoutError + || (error instanceof JournalReadInterruptedError && error.retryable))) throw error; + last = error; + const left = deadline - performance.now(); + if (left <= 0) throw exhausted(); + // Jitter plus escalating bounds keep a slow daemon from collecting a retry storm. + await sleep(Math.min(left, Math.min(1000, timeoutMs / 4) * (0.5 + Math.random())), undefined, { signal: attemptSignal }); + } + } + }); + this.tail = job.catch(() => {}); + void job.then(deliver, fail); + try { return await response; } + finally { + clearTimeout(timer); + signal.removeEventListener('abort', onClose); + } + } +} diff --git a/packages/sdk/src/worker-lease.ts b/packages/sdk/src/worker-lease.ts index 8396f165a..575f69093 100644 --- a/packages/sdk/src/worker-lease.ts +++ b/packages/sdk/src/worker-lease.ts @@ -1,4 +1,5 @@ -import { JournalProtocolError, type JournalClient } from './journal-client.js'; +import { JournalProtocolError, JournalRequestTimeoutError, type JournalClient } from './journal-client.js'; +import { isReadInterruptionError } from './journal-read-policy.js'; import type { StepDispatchEvent } from './protocol.js'; export class WorkerLeaseLostError extends Error { @@ -22,7 +23,9 @@ export function isLeaseLost(error: unknown): boolean { export function onWorkerFailure(label: string, fatal: (error: unknown) => void) { return (error: unknown, dispatch?: StepDispatchEvent): void => { // Without the dispatch there is no attempt to hand back to the kernel: fail closed. - if (!isLeaseLost(error) || dispatch === undefined) { fatal(error); return; } + // A lost lease and an unanswered read both leave the attempt to the kernel, + // which sweeps the lease and retries it. + if (!(isLeaseLost(error) || isReadInterruptionError(error)) || dispatch === undefined) { fatal(error); return; } // The kernel owns this attempt's fate; its journal supplies the run outcome. // stderr keeps this diagnostic out of structured reports on stdout. process.emitWarning( @@ -72,7 +75,12 @@ export async function withWorkerLease( const sentAt = performance.now(); const result = await untilAborted(client.stepHeartbeat( dispatch.run_id, dispatch.step_id, dispatch.attempt, dispatch.lease_id, - ), controller.signal); + ), controller.signal).catch(error => { + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + throw error; + }); controller.signal.throwIfAborted(); // A response handled after local expiry cannot revive ownership, even // if its future deadline was issued before this event loop stalled. diff --git a/packages/sdk/tests/authored-root.test.ts b/packages/sdk/tests/authored-root.test.ts index 589c64fb7..ab7867413 100644 --- a/packages/sdk/tests/authored-root.test.ts +++ b/packages/sdk/tests/authored-root.test.ts @@ -12,9 +12,19 @@ import { resumeDurableAuthoredFlow, type AuthoredRootMetadata, } from '../src/authored-root.js'; -import { JournalProtocolError, type JournalClient } from '../src/journal-client.js'; +import { JournalProtocolError, JournalRequestTimeoutError, type JournalClient } from '../src/journal-client.js'; import type { RunOutcome, StepDispatchEvent } from '../src/protocol.js'; -import { AuthoredHumanParked } from '../src/authored-flow-error.js'; +import { AuthoredFlowExecutionError, AuthoredHumanParked } from '../src/authored-flow-error.js'; +import { JournalReadInterruptedError } from '../src/journal-read-policy.js'; + +const nodeRunner = vi.hoisted(() => ({ override: undefined as undefined | ((...args: unknown[]) => Promise) })); +vi.mock('../src/authored-node-runner.js', async importOriginal => { + const actual = await importOriginal(); + return { ...actual, + assertAuthoredRuntimeAvailable: () => { if (nodeRunner.override === undefined) actual.assertAuthoredRuntimeAvailable(); }, + runAuthoredInNode: (...args: Parameters) => + nodeRunner.override !== undefined ? nodeRunner.override(...args) : actual.runAuthoredInNode(...args) }; +}); vi.mock('../src/authored-flow-loader.js', async importOriginal => ({ ...await importOriginal(), @@ -383,6 +393,153 @@ describe('durable authored root', () => { expect(journal.peer.completions).toEqual([{ attempt: 2, reason: 'success' }]); }); + it('does not terminalize a root whose heartbeat times out', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + const controller = new AbortController(); + journal.peer.stepHeartbeat = async () => { + controller.abort(); + throw new JournalRequestTimeoutError('step.heartbeat', 10); + }; + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'heartbeat-timeout', lifecycle: { signal: controller.signal } })) + .rejects.toThrow(); + expect(journal.peer.completions).toEqual([]); + }); + + it('still terminalizes a timed-out mutation as a body failure', async () => { + const loaded = await fixture(false, 0, async () => { throw new JournalRequestTimeoutError('stream.append', 10); }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'write-timeout' })).rejects.toMatchObject({ verb: 'stream.append' }); + expect(journal.peer.completions).toEqual([{ attempt: 1, reason: 'worker_error' }]); + }); + + it.each([false, true])('leaves the root resumable after a read timeout without waiting for redispatch (Node frame=%s)', async nodeFrame => { + const timeout = new JournalRequestTimeoutError('run.get', 10); + const loaded = await fixture(false, 0, async () => { + throw nodeFrame ? new AuthoredFlowExecutionError('daemon_unresponsive', timeout.message) : timeout; + }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.toMatchObject({ + code: 'daemon_unresponsive', rootRunId: 'root-run', message: expect.stringContaining('flows resume'), + }); + expect(journal.peer.completions).toEqual([]); + expect(journal.resumeCalls).toBe(1); + }); + + it('leaves the root resumable when the read session disconnects while the root worker is healthy', async () => { + const interrupted = new JournalReadInterruptedError('run.get', 2, 40, 300_000, + { cause: new Error('journal client: connection closed') }); + const loaded = await fixture(false, 0, async () => { throw interrupted; }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'read-disconnect' })).rejects.toMatchObject({ + code: 'daemon_unresponsive', rootRunId: 'root-run', message: expect.stringContaining('flows resume'), + }); + expect(journal.peer.completions).toEqual([]); + }); + + it('never terminalizes the root for a caller cancellation', async () => { + const controller = new AbortController(); + const loaded = await fixture(false, 0, async () => { + controller.abort(new Error('caller canceled the run')); + throw controller.signal.reason; + }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'caller-cancel', lifecycle: { signal: controller.signal } })) + .rejects.toMatchObject({ message: 'caller canceled the run', rootRunId: 'root-run' }); + expect(journal.peer.completions).toEqual([]); + }); + + it('scopes body reads to the root attempt lease for the life of the attempt', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + const release = vi.fn(); + const scopeReads = vi.fn((_signal: AbortSignal) => release); + (journal as unknown as { scopeReads: typeof scopeReads }).scopeReads = scopeReads; + await executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'scoped-reads' }).catch(() => undefined); + expect(scopeReads).toHaveBeenCalledTimes(1); + expect(scopeReads.mock.calls[0]![0]).toBeInstanceOf(AbortSignal); + expect(release).toHaveBeenCalledTimes(1); + }); + + it('propagates a read interruption while detecting an authored root instead of guessing declarative', async () => { + const { readAuthoredRootMetadata } = await import('../src/authored-root.js'); + const interrupted = { journalRead: async () => { throw new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000); } }; + await expect(readAuthoredRootMetadata(interrupted as unknown as JournalClient, 'run')).rejects.toBeInstanceOf(JournalRequestTimeoutError); + const plain = { journalRead: async () => ({ entries: [] }) }; + await expect(readAuthoredRootMetadata(plain as unknown as JournalClient, 'run')).resolves.toBeUndefined(); + }); + + it('keeps the lease read scope for the whole Bun-run attempt', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + const release = vi.fn(); + (journal as unknown as { scopeReads: (signal: AbortSignal) => () => void }).scopeReads = () => release; + let finish!: (value: unknown) => void; + nodeRunner.override = () => new Promise(resolve => { finish = resolve; }); + Object.defineProperty(process.versions, 'bun', { value: '1.4.0', configurable: true }); + try { + const execution = executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'bun-scope' }).catch(() => undefined); + await vi.waitFor(() => expect(finish).toBeTypeOf('function')); + expect(release).not.toHaveBeenCalled(); // the child is still running + finish({ name: 'f', completionReason: 'success', journalSteps: [] }); + await execution; + expect(release).toHaveBeenCalledTimes(1); + } finally { + delete (process.versions as Record)['bun']; + nodeRunner.override = undefined; + } + }); + + it('keeps a completed root outcome when reading back its result is interrupted', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + (journal as unknown as { startStatus: RunOutcome }).startStatus = outcome('root-run', 'completed', 'success'); + journal.journalRead = async () => { throw new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000); }; + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'completed-unread' })).rejects.toMatchObject({ + code: 'result_unreadable', completionReason: undefined, rootRunId: 'root-run', + }); + }); + + it('propagates a cancellation while detecting an authored root, but trusts a definite answer', async () => { + const { readAuthoredRootMetadata } = await import('../src/authored-root.js'); + const canceled = { journalRead: async () => { throw new DOMException('This operation was aborted', 'AbortError'); } }; + await expect(readAuthoredRootMetadata(canceled as unknown as JournalClient, 'run')).rejects.toMatchObject({ name: 'AbortError' }); + const absent = { journalRead: async () => { throw new JournalProtocolError('run_not_found', 'no such run'); } }; + await expect(readAuthoredRootMetadata(absent as unknown as JournalClient, 'run')).resolves.toBeUndefined(); + }); + + it('keeps a completed root outcome when reading back its result is canceled', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + (journal as unknown as { startStatus: RunOutcome }).startStatus = outcome('root-run', 'completed', 'success'); + journal.journalRead = async () => { throw new DOMException('This operation was aborted', 'AbortError'); }; + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'completed-canceled' })).rejects.toMatchObject({ code: 'result_unreadable', rootRunId: 'root-run' }); + }); + + it('propagates a daemon-side read failure while detecting an authored root', async () => { + const { readAuthoredRootMetadata } = await import('../src/authored-root.js'); + const storage = { journalRead: async () => { throw new JournalProtocolError('journal_write_failed', 'storage failed'); } }; + await expect(readAuthoredRootMetadata(storage as unknown as JournalClient, 'run')).rejects.toMatchObject({ code: 'journal_write_failed' }); + }); + + it('keeps a completed root outcome when its readback meets a storage protocol error', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + (journal as unknown as { startStatus: RunOutcome }).startStatus = outcome('root-run', 'completed', 'success'); + journal.journalRead = async () => { throw new JournalProtocolError('journal_write_failed', 'storage failed'); }; + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'completed-storage' })).rejects.toMatchObject({ code: 'result_unreadable', rootRunId: 'root-run' }); + }); + it('terminalizes a returned body failure without replaying semantic side effects', async () => { const loaded = await fixture(true); const journal = new RootJournal(); diff --git a/packages/sdk/tests/authored-verifier-read-budget.test.ts b/packages/sdk/tests/authored-verifier-read-budget.test.ts new file mode 100644 index 000000000..b26ab1fe9 --- /dev/null +++ b/packages/sdk/tests/authored-verifier-read-budget.test.ts @@ -0,0 +1,33 @@ +import { expect, it, vi } from 'vitest'; +import { FLOW_READ_BUDGET_MS } from '../src/journal-read-policy.js'; + +const constructed = vi.hoisted(() => [] as unknown[]); +vi.mock('../src/journal-client.js', async original => { + const actual = await original(); + class RecordingClient extends actual.JournalClient { + constructor(socketPath: string, options?: ConstructorParameters[1]) { + super(socketPath, options); + constructed.push(options); + } + override async connect(): Promise { throw new Error('stop after construction'); } + } + return { ...actual, JournalClient: RecordingClient }; +}); + +it('verifies a Bun-run authored result with the flow read budget, not single-shot reads', async () => { + const { verifyAuthoredNodeResult } = await import('../src/authored-node-runner.js'); + const result = { rootRunId: 'root', name: 'flow', completionReason: 'success', + journalSteps: [{ id: 'complete-1' }] }; + await verifyAuthoredNodeResult(result as never, { flowName: 'flow' } as never, 'root', '/unused.sock').catch(() => undefined); + expect(constructed).toContainEqual(expect.objectContaining({ readBudgetMs: FLOW_READ_BUDGET_MS, budgetHandshake: true })); +}); + +it('hands the lifecycle signal to the verifier so cancellation stops its reads', async () => { + const { verifyAuthoredNodeResult } = await import('../src/authored-node-runner.js'); + constructed.length = 0; + const controller = new AbortController(); + const result = { rootRunId: 'root', name: 'flow', completionReason: 'success', journalSteps: [{ id: 'complete-1' }] }; + await verifyAuthoredNodeResult(result as never, { flowName: 'flow' } as never, 'root', '/unused.sock', controller.signal) + .catch(() => undefined); + expect(constructed).toContainEqual(expect.objectContaining({ readSignal: controller.signal })); +}); diff --git a/packages/sdk/tests/classify-outcome.test.ts b/packages/sdk/tests/classify-outcome.test.ts index 47cb65562..d46da85ac 100644 --- a/packages/sdk/tests/classify-outcome.test.ts +++ b/packages/sdk/tests/classify-outcome.test.ts @@ -1,3 +1,4 @@ +import { EventEmitter } from 'node:events'; import { describe, expect, it } from 'vitest'; import { classifyOutcome } from '../src/cli/run.js'; @@ -163,6 +164,9 @@ describe('the remedy on a worker park', () => { const until = Date.now() + 3_000; let resumes = 0; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: async () => (Date.now() < until ? backoff : completed), runResume: async (): Promise => { resumes += 1; @@ -245,6 +249,9 @@ describe('the remedy on a worker park', () => { budget: { tokens_in: 0, tokens_out: 0, dollars: '0' }, }; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: async () => (Date.now() < until ? running : { ...running, status: 'completed', steps: { answer: { type: 'agent', state: 'done' } } }), runResume: async (): Promise => diff --git a/packages/sdk/tests/cli.test.ts b/packages/sdk/tests/cli.test.ts index c8b6138ad..e9d9c86be 100644 --- a/packages/sdk/tests/cli.test.ts +++ b/packages/sdk/tests/cli.test.ts @@ -1010,6 +1010,7 @@ describe('flows run/resume CLI over the journal protocol', () => { let snapshots = 0; await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-worker', status: 'parked', @@ -1059,6 +1060,7 @@ describe('flows run/resume CLI over the journal protocol', () => { let snapshots = 0; await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-stale-worker', status: 'parked', @@ -1066,7 +1068,7 @@ describe('flows run/resume CLI over the journal protocol', () => { completed_steps: 1, }), 'run.get': (ctx) => { - const running = snapshots++ < 4; + const running = snapshots++ < 1; sendResult(ctx, { run_id: 'run-stale-worker', status: running ? 'running' : 'completed', @@ -1095,13 +1097,14 @@ describe('flows run/resume CLI over the journal protocol', () => { expect(output.stderr.join('\n')).toContain('WAITING [worker_lease]'); expect(output.stderr.join('\n')).toContain(`until ${leaseDeadlineMs}`); expect(output.stderr.join('\n')).not.toContain('worker lease for step "answer" expired'); - expect(snapshots).toBeGreaterThan(4); + expect(snapshots).toBeGreaterThan(1); }); it('allows a caller to cancel a worker-lease wait', async () => { const dataDir = temporaryProject('flows-run-cancel-'); await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-cancel', status: 'parked', diff --git a/packages/sdk/tests/communication-worker.test.ts b/packages/sdk/tests/communication-worker.test.ts index f1956e767..555047a4d 100644 --- a/packages/sdk/tests/communication-worker.test.ts +++ b/packages/sdk/tests/communication-worker.test.ts @@ -92,3 +92,44 @@ it('uses the proved identity for a canonical generic communication executable', }), })); }); +it('leaves a heartbeat-lost attempt to the kernel instead of completing it as worker_error', async () => { + const { JournalRequestTimeoutError } = await import('../src/journal-client.js'); + const { isLeaseLost } = await import('../src/worker-lease.js'); + mocks.spawn.mockImplementation(async () => ({ name: 'managed-agent', generation: 'generation', release: mocks.release, waitForReady: mocks.ready })); + const f = fixture(); + f.client.stepHeartbeat.mockImplementation(async () => { throw new JournalRequestTimeoutError('step.heartbeat', 10); }); + (f as unknown as { client: { stepHeartbeat: unknown } }).client.stepHeartbeat = f.client.stepHeartbeat; + const dispatchDeadline = Date.now() + 200; + const error = await completeCommunicationDispatch(f.client as unknown as JournalClient, { + run_id: 'run', step_id: 'agent', attempt: 1, idempotency_key: 'key', pins: {}, lease_id: 'lease', + lease_deadline_ms: dispatchDeadline, spec: { type: 'agent', cli: 'claude', instruction: '' }, + } as StepDispatchEvent, { type: 'relayflows.communication.v1', instruction: 'test', incoming: ['peer'], outgoing: [], timeoutMs: 60_000 }, + '/tmp/data').catch(caught => caught); + expect(isLeaseLost(error)).toBe(true); + expect(f.client.stepComplete).not.toHaveBeenCalled(); +}); +it('leaves a retried attempt whose history read was interrupted to the kernel', async () => { + const { JournalRequestTimeoutError } = await import('../src/journal-client.js'); + const { isReadInterruptionError } = await import('../src/journal-read-policy.js'); + const f = fixture(); + (f.client as unknown as { journalRead: () => Promise }).journalRead = async () => { + throw new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000); + }; + const error = await completeCommunicationDispatch(f.client as unknown as JournalClient, { + run_id: 'run', step_id: 'agent', attempt: 2, idempotency_key: 'key', pins: {}, lease_id: 'lease', + lease_deadline_ms: Date.now() + 30_000, spec: { type: 'agent', cli: 'claude', instruction: '' }, + } as StepDispatchEvent, { type: 'relayflows.communication.v1', instruction: 'test', incoming: ['peer'], outgoing: [], timeoutMs: 1000 }, + '/tmp/data').catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); + expect(f.client.stepComplete).not.toHaveBeenCalled(); +}); +it('scopes the attempt history reads to its lease and releases them after', async () => { + const f = fixture(); + const release = vi.fn(); + const scopeReads = vi.fn((_signal: AbortSignal) => release); + (f.client as unknown as { scopeReads: typeof scopeReads }).scopeReads = scopeReads; + await f.execute(); + expect(scopeReads).toHaveBeenCalledTimes(1); + expect(scopeReads.mock.calls[0]![0]).toBeInstanceOf(AbortSignal); + expect(release).toHaveBeenCalledTimes(1); +}); diff --git a/packages/sdk/tests/direct-run-worker-lease.test.ts b/packages/sdk/tests/direct-run-worker-lease.test.ts index 2b1bbe763..f0d2e9eeb 100644 --- a/packages/sdk/tests/direct-run-worker-lease.test.ts +++ b/packages/sdk/tests/direct-run-worker-lease.test.ts @@ -1,3 +1,4 @@ +import { AuthoredFlowExecutionError } from '../src/authored-flow-error.js'; import { afterEach, expect, it, vi } from 'vitest'; import { runDirectFlow } from '../src/cli/direct-run.js'; import { JournalClient, JournalProtocolError } from '../src/journal-client.js'; @@ -32,3 +33,17 @@ it.each([false, true])('direct run handles LLM errors with leaseLost=%s', async expect(result.exitCode).toBe(1); expect(JSON.stringify(result.report)).toContain(leaseLost ? 'connection closed' : 'cli exploded'); }); + +it('reports a read interruption without losing the resumable root', async () => { + vi.spyOn(JournalClient.prototype, 'connect').mockResolvedValue(); + vi.spyOn(JournalClient.prototype, 'hello').mockResolvedValue({} as never); + vi.spyOn(LlmWorker.prototype, 'attach').mockResolvedValue(); + const error = new AuthoredFlowExecutionError('daemon_unresponsive', 'read timed out under CPU load'); + error.rootRunId = 'saved-root'; + vi.mocked(executeDurableAuthoredFlow).mockRejectedValueOnce(error); + const result = await runDirectFlow('flow.ts', '{}', '/unused'); + expect(result).toMatchObject({ exitCode: 1, report: { + command: 'run', runId: 'saved-root', rootRunId: 'saved-root', status: 'running', + diagnostics: [{ kind: 'daemon_unresponsive', message: expect.stringContaining('flows resume --data-dir /unused saved-root') }], + } }); +}); diff --git a/packages/sdk/tests/heartbeat-timeout.test.ts b/packages/sdk/tests/heartbeat-timeout.test.ts new file mode 100644 index 000000000..e4db308b7 --- /dev/null +++ b/packages/sdk/tests/heartbeat-timeout.test.ts @@ -0,0 +1,24 @@ +import { once } from 'node:events'; +import { expect, it } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { withWorkerLease, WorkerLeaseLostError } from '../src/worker-lease.js'; +import { sendOk, sockPath, startLoopback } from './journal-client-loopback.js'; + +it('a heartbeat timeout is lease loss rather than a worker body failure', async () => { + const path = sockPath(); + let heartbeats = 0; + const server = startLoopback(path, { hello: sendOk, 'step.heartbeat': () => { heartbeats++; } }); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 10 }); + let executed = false; + try { + await client.connect(); + await expect(withWorkerLease(client, { + run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', spec: {}, + lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease_ttl_ms: 30_000, + idempotency_key: 'key', pins: { workspace: [], streams: [] }, + }, async () => { executed = true; })).rejects.toBeInstanceOf(WorkerLeaseLostError); + expect(executed).toBe(false); + expect(heartbeats).toBe(1); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); diff --git a/packages/sdk/tests/helper-effect-lease-loss.test.ts b/packages/sdk/tests/helper-effect-lease-loss.test.ts new file mode 100644 index 000000000..6ae0fa913 --- /dev/null +++ b/packages/sdk/tests/helper-effect-lease-loss.test.ts @@ -0,0 +1,54 @@ +import { once } from 'node:events'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, expect, it, vi } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { sendOk, sendResult, sockPath, startLoopback } from './journal-client-loopback.js'; + +vi.mock('../src/worker-lease.js', async original => { + const actual = await original(); + return { ...actual, withWorkerLease: async () => { + throw new actual.WorkerLeaseLostError('renewal_expired', 'journal client: step.heartbeat timed out after 10ms'); + } }; +}); + +const dirs: string[] = []; +afterEach(() => { for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); }); + +it('propagates heartbeat lease loss from a helper effect instead of failing the step', async () => { + const { runHelperEffect } = await import('../src/authored-helper-effect.js'); + const { isLeaseLost } = await import('../src/worker-lease.js'); + const dataDir = mkdtempSync(join(tmpdir(), 'helper-lease-')); + dirs.push(dataDir); + vi.stubEnv('RELAYFLOWS_SLACK_MOCK', '1'); + let completions = 0; + const path = sockPath(); + const server = startLoopback(path, { + hello: sendOk, + 'run.start': ctx => sendResult(ctx, { run_id: 'helper-run', status: 'running', completion_reason: null, completed_steps: 0 }), + // The dispatch goes to the attached worker's own connection, after attach. + 'worker.attach': ctx => { + sendResult(ctx, {}); + setTimeout(() => ctx.send({ event: 'step.dispatch', data: { run_id: 'helper-run', step_id: 'notify', step_type: 'agent', + attempt: 1, idempotency_key: 'key', lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, pins: {}, + spec: { type: 'agent' } } }), 20); + }, + 'run.resume': ctx => sendResult(ctx, { run_id: 'helper-run', status: 'running', completion_reason: null, completed_steps: 0 }), + 'step.complete': ctx => { completions += 1; sendResult(ctx, {}); }, + }); + await once(server, 'listening'); + const journal = new JournalClient(path); + try { + await journal.connect(); + const error = await runHelperEffect(journal, 'flow', 'notify', + { provider: 'slack', verb: 'post', params: { channel: '#c', text: 'hi' } } as never, dataDir, []) + .catch(caught => caught); + expect(isLeaseLost(error)).toBe(true); + expect(completions).toBe(0); + } finally { + journal.close(); + vi.unstubAllEnvs(); + await new Promise(resolve => server.close(() => resolve())); + } +}); diff --git a/packages/sdk/tests/journal-client-loopback.ts b/packages/sdk/tests/journal-client-loopback.ts index caa434a74..0ef0d2f0a 100644 --- a/packages/sdk/tests/journal-client-loopback.ts +++ b/packages/sdk/tests/journal-client-loopback.ts @@ -24,35 +24,39 @@ export interface FrameCtx { } export interface LoopbackHandlers { - 'subscription.inspect'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.deliver'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.fence_overflow'?: (ctx: FrameCtx, params: Record) => void; - hello?: (ctx: FrameCtx) => void; - 'run.start'?: (ctx: FrameCtx, params: Record) => void; - 'run.resume'?: (ctx: FrameCtx, params: Record) => void; - 'run.cancel'?: (ctx: FrameCtx, params: Record) => void; - 'run.get'?: (ctx: FrameCtx, params: Record) => void; - 'run.watch'?: (ctx: FrameCtx, params: Record) => void; - 'worker.attach'?: (ctx: FrameCtx, params: Record) => void; - 'step.heartbeat'?: (ctx: FrameCtx, params: Record) => void; - 'effect.record'?: (ctx: FrameCtx, params: Record) => void; - 'effect.confirm'?: (ctx: FrameCtx, params: Record) => void; - 'step.complete'?: (ctx: FrameCtx, params: Record) => void; - 'step.wait'?: (ctx: FrameCtx, params: Record) => void; - 'event.emit'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.open'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.next'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.close'?: (ctx: FrameCtx, params: Record) => void; - 'journal.read'?: (ctx: FrameCtx, params: Record) => void; - 'stream.append'?: (ctx: FrameCtx, params: Record) => void; - 'stream.read'?: (ctx: FrameCtx, params: Record) => void; + 'subscription.inspect'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.deliver'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.fence_overflow'?: (ctx: FrameCtx, params: Record) => void | Promise; + hello?: (ctx: FrameCtx) => void | Promise; + 'run.start'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.resume'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.cancel'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.get'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.watch'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'worker.attach'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.heartbeat'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'effect.record'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'effect.confirm'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.complete'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.wait'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'event.emit'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.open'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.next'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.close'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'journal.read'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'stream.append'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'stream.read'?: (ctx: FrameCtx, params: Record) => void | Promise; } -export function startLoopback(path: string, handlers: LoopbackHandlers): Server { +export function startLoopback(path: string, handlers: LoopbackHandlers, options: { serialize?: boolean } = {}): Server { const server = createServer((socket) => { let buffer = ''; + let queue = Promise.resolve(); + // A client may hang up before a reply lands (a dropped reader, a closed + // session); like the daemon, the double ignores the failed write. + socket.on('error', () => {}); const send = (obj: unknown): void => { - socket.write(JSON.stringify(obj) + '\n'); + if (!socket.destroyed) socket.write(JSON.stringify(obj) + '\n'); }; socket.on('data', (chunk) => { buffer += chunk.toString('utf8'); @@ -62,11 +66,14 @@ export function startLoopback(path: string, handlers: LoopbackHandlers): Server buffer = buffer.slice(nl + 1); if (line.length === 0) continue; const req = JSON.parse(line) as { id: string; verb: keyof LoopbackHandlers; params: Record }; - const handler = handlers[req.verb] as ((ctx: FrameCtx, params: Record) => void) | undefined; + const handler = handlers[req.verb] as ((ctx: FrameCtx, params: Record) => void | Promise) | undefined; if (handler === undefined) { send({ id: req.id, ok: false, error: { code: 'unknown_verb', message: req.verb } }); } else { - handler({ id: req.id, socket, send }, req.params); + const handle = () => handler({ id: req.id, socket, send }, req.params); + // server.rs handles one frame at a time per connection. + if (options.serialize) queue = queue.then(handle); + else void handle(); } } }); diff --git a/packages/sdk/tests/journal-client-read-timeout.test.ts b/packages/sdk/tests/journal-client-read-timeout.test.ts new file mode 100644 index 000000000..c815b082a --- /dev/null +++ b/packages/sdk/tests/journal-client-read-timeout.test.ts @@ -0,0 +1,437 @@ +import { once } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; +import type { Server } from 'node:net'; +import { afterEach, expect, it } from 'vitest'; +import { JournalClient, JournalRequestTimeoutError } from '../src/journal-client.js'; +import { JournalReadInterruptedError, isReadInterruptionError } from '../src/journal-read-policy.js'; +import { JournalFrameError } from '../src/journal-client.js'; +import { HELLO_SPEC, sendOk, sendResult, sockPath, startLoopback, type LoopbackHandlers } from './journal-client-loopback.js'; + +const clients: JournalClient[] = []; +const servers: Server[] = []; +afterEach(async () => { + for (const client of clients.splice(0)) client.close(); + await Promise.all(servers.splice(0).map(server => new Promise(resolve => server.close(() => resolve())))); +}); +async function setup(handlers: LoopbackHandlers, budget: number | undefined = 1000) { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, ...handlers }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 50, readBudgetMs: budget }); + clients.push(client); + await client.connect(); + return client; +} + +it('serves a bounded read while an unbounded command is in flight', async () => { + let started!: () => void; + const inFlight = new Promise(resolve => { started = resolve; }); + const client = await setup({ + 'run.start': async ctx => { started(); await sleep(100); sendResult(ctx, { status: 'completed' }); }, + 'run.get': ctx => sendResult(ctx, { status: 'running' }), + }); + let commandDone = false; + const command = client.runStart(HELLO_SPEC).then(result => { commandDone = true; return result; }); + await inFlight; + expect(await client.runGet('run')).toMatchObject({ status: 'running' }); + expect(commandDone).toBe(false); + await command; +}); + +it('retries a read with a fresh id and ignores the late first reply', async () => { + const ids: string[] = []; + let first: Parameters[0]; + const client = await setup({ 'run.get': ctx => { + ids.push(ctx.id); + if (ids.length === 1) first = ctx; + else { sendResult(first, { status: 'stale' }); sendResult(ctx, { status: 'completed' }); } + } }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); + expect(new Set(ids).size).toBe(2); +}); + +it('never retries mutating verbs', async () => { + let writes = 0; + const client = await setup({ 'stream.append': () => { writes += 1; } }); + await expect(client.streamAppend('run', 'stream', {})).rejects.toMatchObject({ + verb: 'stream.append', attempts: 1, message: 'journal client: stream.append timed out after 50ms', + }); + expect(writes).toBe(1); +}); + +it('exhausts a total read budget with a typed diagnostic', async () => { + const client = await setup({ 'run.get': () => {} }, 250); + const error = await client.runGet('run').catch(error => error); + expect(error).toBeInstanceOf(JournalRequestTimeoutError); + expect(error.attempts).toBeGreaterThan(1); + expect(error.message).toContain('read budget 250ms'); + expect(error.message).toContain('CPU load'); +}); + +it('keeps interactive reads single-shot by default', async () => { + let reads = 0; + // Explicitly omit the opt-in. + const client = await setup({ 'run.get': () => { reads += 1; } }, 0); + client.close(); + const interactive = new JournalClient(client.socketPath, { requestTimeoutMs: 10 }); + clients.push(interactive); + await interactive.connect(); + await expect(interactive.runGet('run')).rejects.toMatchObject({ attempts: 1, readBudgetMs: undefined }); + expect(reads).toBe(1); +}); + +it('serializes concurrent reads and stops queued reads on close', async () => { + let reads = 0; + let received!: () => void; + const entered = new Promise(resolve => { received = resolve; }); + const client = await setup({ 'run.get': () => { reads += 1; received(); } }); + const results = Promise.allSettled(Array.from({ length: 8 }, () => client.runGet('run'))); + await entered; + client.close(); + expect((await results).every(result => result.status === 'rejected')).toBe(true); + expect(reads).toBe(1); +}); + +it('does not retry protocol rejections', async () => { + let reads = 0; + const client = await setup({ 'run.get': ctx => { + reads += 1; + ctx.send({ id: ctx.id, ok: false, error: { code: 'run_not_found', message: 'absent' } }); + } }); + await expect(client.runGet('absent')).rejects.toMatchObject({ code: 'run_not_found' }); + expect(reads).toBe(1); +}); + +it('a recovered read timeout does not become an authored callback failure', async () => { + const { flow } = await import('@relayflows/surface'); + const { executeAuthoredFlow } = await import('../src/authored-flow-executor.js'); + let reads = 0; + const client = await setup({ + 'run.start': (ctx, params) => sendResult(ctx, { run_id: (params['spec'] as { steps: Array<{ id: string }> }).steps[0]!.id, status: 'completed', completion_reason: 'success', completed_steps: 1 }), + 'journal.read': (ctx, params) => { + if (++reads === 1) return; + sendResult(ctx, { entries: [{ seq: 1, run_id: params['run_id'], step_id: params['run_id'], entry_type: 'step.completed', + payload: { completionReason: 'success', output: { stdout_tail: 'ok', exit_code: 0 } } }] }); + }, + }); + const result = await executeAuthoredFlow(flow('retried-read', async f => { + await f.run('echo ok'); + f.done('success'); + }), client, undefined); + expect(result.completionReason).toBe('success'); + expect(reads).toBe(3); +}); + +it('falls back to the primary connection if reader setup is refused', async () => { + const client = await setup({ + hello: ctx => ctx.send({ id: ctx.id, ok: false, error: { code: 'busy', message: 'reader unavailable' } }), + 'run.get': ctx => sendResult(ctx, { status: 'completed' }), + }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); +}); + +it.each(['run.get', 'journal.read', 'stream.read', 'subscription.inspect'] as const)('retries only the allowlisted read %s', async verb => { + let attempts = 0; + const client = await setup({ [verb]: ctx => { if (++attempts === 2) sendResult(ctx, {}); } }); + const result = verb === 'run.get' ? client.runGet('run') + : verb === 'journal.read' ? client.journalRead('run', 1) + : verb === 'stream.read' ? client.streamRead('run', 'stream', 0) + : client.subscriptionInspect({ run_id: 'run', subscription_id: 'subscription' }); + await result; + expect(attempts).toBe(2); +}); + +it('reconnects the reader after it disconnects within the read budget', async () => { + let reads = 0; + const client = await setup({ + 'run.get': ctx => { + reads += 1; + if (reads === 1) return; // time out first + if (reads === 2) { ctx.socket.destroy(); return; } // then lose the reader transport + sendResult(ctx, { status: 'completed' }); + }, + 'stream.append': ctx => sendResult(ctx, { seq: 1 }), + }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); + expect(reads).toBe(3); + // The primary session that carries writes was never involved. + await expect(client.streamAppend('run', 'stream', {})).resolves.toBeDefined(); +}); + +it('surfaces an exhausted reader disconnect as a typed read interruption, not a body failure', async () => { + const client = await setup({ 'run.get': ctx => { ctx.socket.destroy(); } }, 250); + const error = await client.runGet('run').catch(error => error); + expect(error).toBeInstanceOf(JournalReadInterruptedError); + expect(isReadInterruptionError(error)).toBe(true); + expect(error.verb).toBe('run.get'); + expect(error.attempts).toBeGreaterThan(1); + expect(error.message).toContain('read budget 250ms'); +}); + +it('a canceled read settles promptly and frees the serial read queue', async () => { + let reads = 0; + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const client = await setup({ 'run.get': ctx => { + reads += 1; + if (reads === 1) { entered(); return; } // never answers + sendResult(ctx, { status: 'completed' }); + } }, 60_000); + const controller = new AbortController(); + const pending = client.runGet('run', { signal: controller.signal }); + await started; + const begun = performance.now(); + controller.abort(new Error('caller canceled')); + await expect(pending).rejects.toThrow('caller canceled'); + expect(performance.now() - begun).toBeLessThan(40); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); +}); + +it('retries a reader setup that timed out instead of reading on the primary for good', async () => { + let hellos = 0; + let primary: unknown; + const served: unknown[] = []; + const client = await setup({ + hello: ctx => { if (++hellos > 1) sendOk(ctx); }, // the first reader hello times out under load + 'run.get': ctx => { served.push(ctx.socket); sendResult(ctx, { status: 'completed' }); }, + 'stream.append': ctx => { primary = ctx.socket; sendResult(ctx, { seq: 1 }); }, + }, 5_000); + await client.streamAppend('run', 'stream', {}); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); + expect(hellos).toBe(2); + expect(served).toHaveLength(2); + expect(served.every(socket => socket !== primary)).toBe(true); +}); + +it('fails closed on a malformed reader frame instead of retrying it as a transport loss', async () => { + let reads = 0; + const client = await setup({ 'run.get': ctx => { reads += 1; ctx.socket.write('{not json\n'); } }, 5_000); + const error = await client.runGet('run').catch(caught => caught); + expect(error).toBeInstanceOf(JournalFrameError); + expect(isReadInterruptionError(error)).toBe(false); + expect(reads).toBe(1); +}); + +it('a client read signal cancels every budgeted read promptly', async () => { + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, 'journal.read': () => entered() }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const controller = new AbortController(); + const client = new JournalClient(path, { requestTimeoutMs: 50, readBudgetMs: 60_000, readSignal: controller.signal }); + clients.push(client); + await client.connect(); + const pending = client.journalRead('run', 1); + await started; + const begun = performance.now(); + controller.abort(new Error('run canceled')); + await expect(pending).rejects.toThrow('run canceled'); + expect(performance.now() - begun).toBeLessThan(40); +}); + +it('fails closed on a malformed frame during reader setup', async () => { + let hellos = 0; + let reads = 0; + const client = await setup({ + hello: ctx => { hellos += 1; ctx.socket.write('{not json\n'); }, + 'run.get': ctx => { reads += 1; sendResult(ctx, { status: 'completed' }); }, + }, 5_000); + const error = await client.runGet('run').catch(caught => caught); + expect(error).toBeInstanceOf(JournalFrameError); + expect(hellos).toBe(1); + expect(reads).toBe(0); +}); + +async function handshakeClient(handlers: LoopbackHandlers, budget: number) { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, ...handlers }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 50, readBudgetMs: budget, budgetHandshake: true }); + clients.push(client); + await client.connect(); + return client; +} + +it('keeps a pre-admission handshake single-shot even on a budgeted flow client', async () => { + let hellos = 0; + const client = await setup({ hello: () => { hellos += 1; } }, 5_000); + await expect(client.hello('flows-run')).rejects.toMatchObject({ verb: 'hello', attempts: 1 }); + expect(hellos).toBe(1); +}); + +it('retries a delayed post-admission handshake within the read budget', async () => { + let hellos = 0; + const client = await handshakeClient({ hello: ctx => { if (++hellos > 1) sendOk(ctx); } }, 5_000); + await expect(client.hello('flows-authored-node')).resolves.toBeDefined(); + expect(hellos).toBe(2); +}); + +it('an exhausted post-admission handshake budget is a read interruption, not a body failure', async () => { + const client = await handshakeClient({ hello: () => {} }, 250); + const error = await client.hello('flows-authored-node').catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); +}); + +it('a scoped read signal (the root lease) cancels budgeted reads only while it is attached', async () => { + let entered!: () => void; + let reads = 0; + const started = new Promise(resolve => { entered = resolve; }); + const client = await setup({ 'journal.read': ctx => { reads += 1; if (reads === 1) entered(); else sendResult(ctx, { entries: [] }); } }, 60_000); + const lease = new AbortController(); + const release = client.scopeReads(lease.signal); + const pending = client.journalRead('run', 1); + await started; + lease.abort(new Error('root lease lost')); + await expect(pending).rejects.toThrow('root lease lost'); + release(); + await expect(client.journalRead('run', 1)).resolves.toEqual({ entries: [] }); +}); + +it('an unexpected primary disconnect aborts in-flight reads with a typed read interruption', async () => { + let primary: { destroy(): void } | undefined; + const client = await setup({ + 'stream.append': ctx => { primary = ctx.socket; sendResult(ctx, { seq: 1 }); }, + 'run.get': () => { primary?.destroy(); }, // the reader is healthy; the primary drops + }, 60_000); + await client.streamAppend('run', 'stream', {}); + const error = await client.runGet('run').catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); +}); + +it('a read started after the primary dropped is still a typed read interruption', async () => { + let primary: { destroy(): void } | undefined; + const client = await setup({ + 'stream.append': ctx => { primary = ctx.socket; sendResult(ctx, { seq: 1 }); }, + 'run.get': ctx => sendResult(ctx, { status: 'running' }), + }, 60_000); + await client.streamAppend('run', 'stream', {}); + const dropped = new Promise(resolve => client.once('disconnected', () => resolve())); + primary?.destroy(); + await dropped; + const error = await client.runGet('run').catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); +}); + +it('canceling a read during reader setup frees the queue for the next read at once', async () => { + let hellos = 0; + let firstHello!: () => void; + const helloSeen = new Promise(resolve => { firstHello = resolve; }); + const path = sockPath(); + const server = startLoopback(path, { + hello: ctx => { if (++hellos === 1) firstHello(); else sendOk(ctx); }, // the first reader hello hangs + 'run.get': ctx => sendResult(ctx, { status: 'running' }), + }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 20_000, readBudgetMs: 60_000 }); + clients.push(client); + await client.connect(); + const controller = new AbortController(); + const first = client.runGet('run', { signal: controller.signal }); + await helloSeen; + controller.abort(new Error('re-drive')); + await expect(first).rejects.toThrow('re-drive'); + const begun = performance.now(); + await expect(client.runGet('run')).resolves.toMatchObject({ status: 'running' }); + expect(performance.now() - begun).toBeLessThan(2_000); +}); + +it('cancelReads ends in-flight budgeted reads at once, ahead of a draining shutdown', async () => { + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const client = await setup({ 'journal.read': () => entered() }, 60_000); + const pending = client.journalRead('run', 1); + await started; + const begun = performance.now(); + client.cancelReads(); + await expect(pending).rejects.toThrow(); + expect(performance.now() - begun).toBeLessThan(40); +}); + +it('drops a reader whose in-flight read was canceled, so the next read gets a fresh session', async () => { + let reads = 0; + const sockets: unknown[] = []; + let firstRead!: () => void; + const started = new Promise(resolve => { firstRead = resolve; }); + const path = sockPath(); + // Serialized per connection, like relayflowd: a hung request blocks its socket. + const server = startLoopback(path, { + hello: sendOk, + 'run.get': ctx => { reads += 1; sockets.push(ctx.socket); if (reads === 1) firstRead(); else sendResult(ctx, { status: 'running' }); }, + }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 20_000, readBudgetMs: 60_000 }); + clients.push(client); + await client.connect(); + const lease = new AbortController(); + const release = client.scopeReads(lease.signal); + const first = client.runGet('run'); + await started; + lease.abort(new Error('root lease lost')); + await expect(first).rejects.toThrow('root lease lost'); + release(); + const begun = performance.now(); + await expect(client.runGet('run')).resolves.toMatchObject({ status: 'running' }); + expect(performance.now() - begun).toBeLessThan(2_000); + expect(sockets[1]).not.toBe(sockets[0]); +}); + +it('canceled reads are typed read interruptions, so workers hand attempts back', async () => { + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const client = await setup({ 'journal.read': () => entered() }, 60_000); + const pending = client.journalRead('run', 1); + await started; + client.cancelReads(); + const error = await pending.catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); +}); + +it('an exhausted budget during reader setup frees the queue for the next read', async () => { + let hellos = 0; + const path = sockPath(); + const server = startLoopback(path, { + hello: ctx => { if (++hellos > 1) sendOk(ctx); }, // the first reader handshake stalls + 'run.get': ctx => sendResult(ctx, { status: 'running' }), + }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 20_000, readBudgetMs: 300 }); + clients.push(client); + await client.connect(); + await expect(client.runGet('run')).rejects.toMatchObject({ readBudgetMs: 300 }); + const begun = performance.now(); + await expect(client.runGet('run')).resolves.toMatchObject({ status: 'running' }); + expect(performance.now() - begun).toBeLessThan(2_000); +}); + +it('a transport error during a budgeted handshake is a read interruption, not a body failure', async () => { + const client = await handshakeClient({ hello: ctx => { ctx.socket.destroy(); } }, 60_000); + const begun = performance.now(); + const error = await client.hello('flows-authored-node').catch(caught => caught); + expect(isReadInterruptionError(error)).toBe(true); + expect(performance.now() - begun).toBeLessThan(2_000); // the session is gone; no futile retries +}); + +it('retries a timed-out read on a fresh reader instead of queueing behind it', async () => { + let reads = 0; + const sockets: unknown[] = []; + const path = sockPath(); + const server = startLoopback(path, { + hello: sendOk, + 'run.get': ctx => { reads += 1; sockets.push(ctx.socket); if (reads > 1) sendResult(ctx, { status: 'running' }); }, + }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 100, readBudgetMs: 10_000 }); + clients.push(client); + await client.connect(); + await expect(client.runGet('run')).resolves.toMatchObject({ status: 'running' }); + expect(sockets[1]).not.toBe(sockets[0]); +}); diff --git a/packages/sdk/tests/memoization.test.ts b/packages/sdk/tests/memoization.test.ts index 2622e1eae..11a85e3db 100644 --- a/packages/sdk/tests/memoization.test.ts +++ b/packages/sdk/tests/memoization.test.ts @@ -56,3 +56,29 @@ it.each([ rmSync(dir, { recursive: true, force: true }); } }); + +it('cancels runFlow promptly while a classification read is delayed', async () => { + const dir = mkdtempSync(join(tmpdir(), 'run-cancel-read-')); + let reading!: () => void; + const inRead = new Promise(resolve => { reading = resolve; }); + const server = startLoopback(socketPathFor(dir), { + hello: ctx => sendOk(ctx), + 'run.start': ctx => ctx.send({ id: ctx.id, ok: true, result: { run_id: 'r1', status: 'running', completion_reason: null, completed_steps: 0 } }), + 'run.get': () => reading(), // the daemon never answers under load + }); + await once(server, 'listening'); + const controller = new AbortController(); + try { + const running = runFlow(fileURLToPath(new URL('../../../testdata/hello-deterministic.flow.yaml', import.meta.url)), + dir, { daemon: { spawn: false }, signal: controller.signal }); + await inRead; + const begun = performance.now(); + controller.abort(); + const settled = await Promise.race([running.then(() => 'settled', () => 'settled'), new Promise(resolve => setTimeout(() => resolve('still reading'), 2_000))]); + expect(settled).toBe('settled'); + expect(performance.now() - begun).toBeLessThan(2_000); + } finally { + await new Promise(resolve => server.close(() => resolve())); + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/packages/sdk/tests/read-timeout-resume-live.test.ts b/packages/sdk/tests/read-timeout-resume-live.test.ts new file mode 100644 index 000000000..677309874 --- /dev/null +++ b/packages/sdk/tests/read-timeout-resume-live.test.ts @@ -0,0 +1,60 @@ +import { readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { expect, it } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { loadAuthoredFlow } from '../src/authored-flow-loader.js'; +import { executeDurableAuthoredFlow, resumeDurableAuthoredFlow } from '../src/authored-root.js'; +import { LlmWorker } from '../src/llm-worker.js'; +import { chainFixture, shellWord } from './flow-chain-fixture.js'; + +it('parks an unreadable authored root and resumes without repeating its journaled effect', async () => { + const fixture = chainFixture(); + const journal = await fixture.connect(); + const { pid } = JSON.parse(readFileSync(join(fixture.data, 'connection.json'), 'utf8')); + const worker = new LlmWorker(journal, 'park-read-worker'); + const client = new JournalClient(journal.socketPath, { requestTimeoutMs: 100, readBudgetMs: 500 }); + const effects = join(fixture.root, 'effects'); + writeFileSync(fixture.flowPath, `import {flow} from '@relayflows/surface'; +export default flow('park-read', async f => { + await f.run(${JSON.stringify(`echo saved >> ${shellWord(effects)}`)}); + await f.llm('hello', {model:'test-model', output:{type:'object'}}); + await f.run(${JSON.stringify(`echo after >> ${shellWord(effects)}`)}); + f.done('success'); +}); +`); + const get = client.runGet.bind(client); + let interrupted = false; + // Suspend service exactly when the body first reads its running child. The + // request and retry timers are real; no synthetic rejection is injected. + client.runGet = runId => { + if (!interrupted) { interrupted = true; process.kill(pid, 'SIGSTOP'); } + return get(runId); + }; + let rootRunId = ''; + try { + await client.connect(); + await worker.attach(); + const loaded = await loadAuthoredFlow(fixture.flowPath); + try { + await expect(executeDurableAuthoredFlow(loaded, client, undefined, { + dataDir: fixture.data, admissionKey: 'park-read', onAdmitted: id => { rootRunId = id; }, + })).rejects.toMatchObject({ code: 'daemon_unresponsive', rootRunId: expect.any(String) }); + } finally { process.kill(pid, 'SIGCONT'); client.runGet = get; } + expect(interrupted).toBe(true); + expect(readFileSync(effects, 'utf8')).toBe('saved\n'); + const entries = (await journal.journalRead(rootRunId, 1, 1000)).entries as Array<{ + entry_type: string; step_id: string; payload: { completionReason?: string }; + }>; + expect(entries.some(entry => entry.step_id === 'authored-root' + && entry.entry_type === 'step.completed' && entry.payload.completionReason === 'worker_error')).toBe(false); + expect(await resumeDurableAuthoredFlow(rootRunId, client, { dataDir: fixture.data })) + .toMatchObject({ rootRunId, completionReason: 'success' }); + expect(readFileSync(effects, 'utf8')).toBe('saved\nafter\n'); + expect(await journal.runGet(rootRunId)).toMatchObject({ status: 'completed' }); + } finally { + process.kill(pid, 'SIGCONT'); + client.close(); + await worker.close(); + await fixture.close(); + } +}, 30_000); diff --git a/packages/sdk/tests/resume-worker-lease.test.ts b/packages/sdk/tests/resume-worker-lease.test.ts index 91a5b304c..102f2e259 100644 --- a/packages/sdk/tests/resume-worker-lease.test.ts +++ b/packages/sdk/tests/resume-worker-lease.test.ts @@ -1,3 +1,4 @@ +import { AuthoredFlowExecutionError } from '../src/authored-flow-error.js'; import { afterEach, expect, it, vi } from 'vitest'; import { JournalClient, JournalProtocolError } from '../src/journal-client.js'; import { LlmWorker } from '../src/llm-worker.js'; @@ -34,3 +35,17 @@ it.each([false, true])('resume handles LLM errors with leaseLost=%s', async leas expect(JSON.stringify(result.report)).toContain(leaseLost ? 'connection closed' : 'cli exploded'); expect(warning).toHaveBeenCalledTimes(leaseLost ? 1 : 0); }); + +it('reports a read interruption without losing the resumable root', async () => { + vi.spyOn(JournalClient.prototype, 'connect').mockResolvedValue(); + vi.spyOn(JournalClient.prototype, 'hello').mockResolvedValue({} as never); + vi.spyOn(LlmWorker.prototype, 'attach').mockResolvedValue(); + const error = new AuthoredFlowExecutionError('daemon_unresponsive', 'read timed out under CPU load'); + error.rootRunId = 'saved-root'; + vi.mocked(resumeDurableAuthoredFlow).mockRejectedValueOnce(error); + const result = await resumeFlow('saved-root', '/unused', { localAgent: true }); + expect(result).toMatchObject({ exitCode: 1, report: { + command: 'resume', runId: 'saved-root', rootRunId: 'saved-root', status: 'running', + diagnostics: [{ kind: 'daemon_unresponsive', message: expect.stringContaining('flows resume --data-dir /unused --local-agent saved-root') }], + } }); +}); diff --git a/packages/sdk/tests/reuse-summary-interruption.test.ts b/packages/sdk/tests/reuse-summary-interruption.test.ts new file mode 100644 index 000000000..71230bf93 --- /dev/null +++ b/packages/sdk/tests/reuse-summary-interruption.test.ts @@ -0,0 +1,60 @@ +import { expect, it } from 'vitest'; +import { attachReuseSummary } from '../src/cli/reuse.js'; +import { JournalFrameError, JournalReadInterruptedError, JournalRequestTimeoutError, type JournalClient } from '../src/journal-client.js'; +import type { RunExecution } from '../src/cli/run.js'; + +const completed = (): RunExecution => ({ exitCode: 0, report: { + command: 'run', ok: true, status: 'completed', completionReason: 'success', runId: 'run-1', diagnostics: [], +} as never }); + +it.each([ + ['timeout', new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000)], + ['interruption', new JournalReadInterruptedError('journal.read', 3, 300_000, 300_000)], +])('keeps the classified outcome when the reuse summary read hits a %s', async (_name, error) => { + const client = { journalRead: async () => { throw error; } } as unknown as JournalClient; + const execution = await attachReuseSummary(completed(), client, 'run-1', 'prior'); + expect(execution.exitCode).toBe(0); + expect(execution.report).toMatchObject({ ok: true, status: 'completed', completionReason: 'success' }); + expect(execution.report.reuse).toBeUndefined(); + expect(execution.report.diagnostics).toEqual([expect.objectContaining({ + severity: 'warning', kind: 'reuse_summary_unavailable', message: expect.stringContaining('prior'), + })]); + expect(JSON.stringify(execution.report)).not.toContain('flows resume'); +}); + +it('still attaches the summary when the read succeeds, and never lets the optional read discard the outcome', async () => { + const ok = { journalRead: async (_run: string, from: number) => ({ entries: from === 1 + ? [{ seq: 1, entry_type: 'step.completed', step_id: 'a', payload: { reused_from: 'prior' } }] : [] }) } as unknown as JournalClient; + expect((await attachReuseSummary(completed(), ok, 'run-1', 'prior')).report.reuse) + .toEqual({ fromRunId: 'prior', reusedSteps: 1, executedSteps: 0 }); + for (const failure of [new Error('invalid journal sequence'), new JournalFrameError(), + Object.assign(new Error('run canceled'), { name: 'AbortError' })]) { + const broken = { journalRead: async () => { throw failure; } } as unknown as JournalClient; + const execution = await attachReuseSummary(completed(), broken, 'run-1', 'prior'); + expect(execution.report).toMatchObject({ ok: true, status: 'completed' }); + expect(execution.report.diagnostics.at(-1)).toMatchObject({ kind: 'reuse_summary_unavailable', message: expect.stringContaining(failure.message) }); + } +}); + +it.each(['daemon_unresponsive', 'daemon_unreachable'])('does not start another read after classification reported %s', async kind => { + let reads = 0; + const client = { journalRead: async () => { reads += 1; return { entries: [] }; } } as unknown as JournalClient; + const interrupted: RunExecution = { exitCode: 1, report: { command: 'run', ok: false, status: 'running', runId: 'run-1', + diagnostics: [{ severity: 'failure', kind, message: 'read timed out' }] } as never }; + const execution = await attachReuseSummary(interrupted, client, 'run-1', 'prior'); + expect(reads).toBe(0); + expect(execution).toBe(interrupted); + expect(execution.report.reuse).toBeUndefined(); +}); + +it('does not start another read after failed-step inspection was interrupted', async () => { + let reads = 0; + const client = { journalRead: async () => { reads += 1; return { entries: [] }; } } as unknown as JournalClient; + const failed: RunExecution = { exitCode: 1, report: { command: 'run', ok: false, status: 'failed', runId: 'run-1', diagnostics: [ + { severity: 'failure', kind: 'step_failed', message: 'Run failed. Could not inspect the failed step: read timed out' }, + { severity: 'warning', kind: 'inspection_interrupted', message: 'The failed step could not be inspected: read timed out' }, + ] } as never }; + const execution = await attachReuseSummary(failed, client, 'run-1', 'prior'); + expect(reads).toBe(0); + expect(execution).toBe(failed); +}); diff --git a/packages/sdk/tests/run-daemon-unresponsive.test.ts b/packages/sdk/tests/run-daemon-unresponsive.test.ts new file mode 100644 index 000000000..21305b76d --- /dev/null +++ b/packages/sdk/tests/run-daemon-unresponsive.test.ts @@ -0,0 +1,60 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { classifyOutcome } from '../src/cli/run.js'; +import { JournalRequestTimeoutError, type JournalClient } from '../src/journal-client.js'; +import { probeSocket } from '../src/daemon-connection.js'; +import { JournalReadInterruptedError } from '../src/journal-read-policy.js'; +vi.mock('../src/daemon-connection.js', async original => ({ + ...await original(), probeSocket: vi.fn(), +})); +afterEach(() => vi.clearAllMocks()); +it.each([true, false])('reports an unreadable declarative run as resumable (reachable=%s)', async reachable => { + vi.mocked(probeSocket).mockResolvedValue({ reachable }); + const client = { runGet: async () => { throw new JournalRequestTimeoutError('run.get', 10, 3, 100, 100); } }; + const result = await classifyOutcome(client as unknown as JournalClient, 'run', { + run_id: 'saved-run', status: 'parked', completion_reason: null, completed_steps: 30, + }, { command: 'run', ok: true, diagnostics: [] } as never, '/socket', {}); + expect(result).toMatchObject({ exitCode: 1, report: { + runId: 'saved-run', rootRunId: 'saved-run', status: 'running', + diagnostics: [{ kind: reachable ? 'daemon_unresponsive' : 'daemon_unreachable', + message: expect.stringContaining('flows resume saved-run') }], + } }); + expect(probeSocket).toHaveBeenCalledWith('/socket'); +}); + +it('reports an interrupted read session as resumable rather than a failed run', async () => { + vi.mocked(probeSocket).mockResolvedValue({ reachable: true }); + const client = { runGet: async () => { + throw new JournalReadInterruptedError('run.get', 3, 100, 100, { cause: new Error('journal client: connection closed') }); + } }; + const result = await classifyOutcome(client as unknown as JournalClient, 'run', { + run_id: 'saved-run', status: 'parked', completion_reason: null, completed_steps: 30, + }, { command: 'run', ok: true, diagnostics: [] } as never, '/socket', {}); + expect(result).toMatchObject({ exitCode: 1, report: { + runId: 'saved-run', status: 'running', + diagnostics: [{ kind: 'daemon_unresponsive', message: expect.stringContaining('flows resume saved-run') }], + } }); +}); + +it('reports a completed run with an unread result as finished but with an unconfirmed verdict', async () => { + const { completedResultUnreadableReport } = await import('../src/cli/journal-timeout.js'); + const { AuthoredFlowExecutionError } = await import('../src/authored-flow-error.js'); + const error = new AuthoredFlowExecutionError('result_unreadable', 'run r completed, but its stored result (the flow\'s verdict) could not be read', undefined, 'r'); + error.rootRunId = 'r'; + const execution = completedResultUnreadableReport('resume', { command: 'resume', ok: true, diagnostics: [] } as never, '/socket', error); + expect(execution.exitCode).toBe(1); + expect(execution.report).toMatchObject({ ok: false, status: 'completed', runId: 'r', + diagnostics: [{ severity: 'warning', kind: 'result_unreadable', message: expect.stringContaining('flows status r') }] }); + expect(execution.report).not.toHaveProperty('completionReason'); + expect(JSON.stringify(execution.report)).not.toContain('flows resume'); +}); + +it('marks a step_failed run whose inspection was interrupted', async () => { + vi.mocked(probeSocket).mockResolvedValue({ reachable: true }); + const client = { journalRead: async () => { throw new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000); }, + runGet: async () => { throw new JournalRequestTimeoutError('run.get', 10, 3, 300_000, 300_000); } }; + const result = await classifyOutcome(client as unknown as JournalClient, 'run', { + run_id: 'failed-run', status: 'failed', completion_reason: 'step_failed', completed_steps: 1, + }, { command: 'run', ok: true, diagnostics: [] } as never, '/socket', {}); + expect(result.report).toMatchObject({ status: 'failed', completionReason: 'step_failed' }); + expect(result.report.diagnostics).toContainEqual(expect.objectContaining({ severity: 'warning', kind: 'inspection_interrupted' })); +}); diff --git a/packages/sdk/tests/run-read-load-live.test.ts b/packages/sdk/tests/run-read-load-live.test.ts new file mode 100644 index 000000000..5b7e6de02 --- /dev/null +++ b/packages/sdk/tests/run-read-load-live.test.ts @@ -0,0 +1,68 @@ +import { execFileSync } from 'node:child_process'; +import { availableParallelism } from 'node:os'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { expect, it } from 'vitest'; +import { toKernelSpec } from '../src/compile.js'; +import { JournalClient } from '../src/journal-client.js'; +import { chainFixture, shellWord } from './flow-chain-fixture.js'; + +it('completes a CPU-saturating deterministic flow with reads in flight and preserves its journal', async () => { + const fixture = chainFixture(); + const primary = await fixture.connect(); + const client = new JournalClient(primary.socketPath, { requestTimeoutMs: 100, readBudgetMs: 15_000 }); + let processors = availableParallelism(); + if (process.platform === 'linux') { + // Saturate the daemon's entire CPU allocation without starving unrelated + // test fixtures whose synthetic worker leases last only 30 milliseconds. + const cpu = readFileSync('/proc/self/status', 'utf8').match(/^Cpus_allowed_list:\s*(\d+)/m)![1]!; + const { pid } = JSON.parse(readFileSync(join(fixture.data, 'connection.json'), 'utf8')); + execFileSync('taskset', ['-apc', cpu, String(pid)]); + processors = 1; + } + const stamp = join(fixture.root, 'effects'); + let started!: (runId: string) => void; + const admitted = new Promise(resolve => { started = resolve; }); + client.on('entry', entry => { if (entry.entry_type === 'run.spawned') started(entry.run_id); }); + // One runnable CPU burner per available processor, launched by the flow itself. + // The shell trap owns all burners even if the deterministic step is terminated. + const command = `pids=''; trap 'kill $pids 2>/dev/null; wait' EXIT; ` + + `i=0; while [ "$i" -lt ${processors} ]; do yes >/dev/null & pids="$pids $!"; i=$((i+1)); done; sleep 2`; + try { + await client.connect(); + await client.hello('load-regression'); + const execution = client.runStart(toKernelSpec({ version: '0.1.0', name: 'read-under-load', steps: [ + { id: 'saved', type: 'deterministic', command: `echo saved >> ${shellWord(stamp)}` }, + { id: 'cpu', type: 'deterministic', command, dependsOn: ['saved'] }, + { id: 'after', type: 'deterministic', command: 'echo survived', dependsOn: ['cpu'] }, + ] }), undefined, undefined, true); + const runId = await admitted; + const reads = Promise.all(Array.from({ length: 8 }, () => client.runGet(runId))); + const [outcome, snapshots] = await Promise.all([execution, reads]); + expect(outcome).toMatchObject({ status: 'completed', completion_reason: 'success', completed_steps: 3 }); + expect(snapshots.every(snapshot => snapshot.run_id === runId)).toBe(true); + const entries = (await client.journalRead(runId, 1, 1000)).entries as Array<{ entry_type: string }>; + expect(entries.filter(entry => entry.entry_type === 'step.completed')).toHaveLength(3); + expect(await client.runResume(runId)).toMatchObject({ status: 'completed', completion_reason: 'success' }); + expect(readFileSync(stamp, 'utf8')).toBe('saved\n'); + } finally { client.close(); await fixture.close(); } +}, 30_000); + +it('drains read and watch promises before an authored flow completes', async () => { + const { flow } = await import('@relayflows/surface'); + const { executeAuthoredFlow } = await import('../src/authored-flow-executor.js'); + const { LlmWorker } = await import('../src/llm-worker.js'); + const fixture = chainFixture(); + const workerClient = await fixture.connect(); + const worker = new LlmWorker(workerClient, 'read-policy-worker'); + const client = new JournalClient(workerClient.socketPath, { readBudgetMs: 5_000 }); + try { + await client.connect(); + await worker.attach(); + const result = await executeAuthoredFlow(flow('read-policy-scope', async f => { + await f.llm('hello', { model: 'test-model', output: { type: 'object' } }); + f.done('success'); + }), client, undefined, { flowPath: fixture.flowPath }); + expect(result.completionReason).toBe('success'); + } finally { await worker.close(); client.close(); await fixture.close(); } +}, 30_000); diff --git a/packages/sdk/tests/running-step-watch.test.ts b/packages/sdk/tests/running-step-watch.test.ts new file mode 100644 index 000000000..b161921d9 --- /dev/null +++ b/packages/sdk/tests/running-step-watch.test.ts @@ -0,0 +1,310 @@ +import { once } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { expect, it } from 'vitest'; +import { JournalClient, JournalFrameError } from '../src/journal-client.js'; +import { waitForRunningStep } from '../src/cli/running-step.js'; +import { isReadInterruption } from '../src/cli/journal-timeout.js'; +import { AuthoredFlowExecutionError } from '../src/authored-flow-error.js'; +import { sendOk, sendResult, sockPath, startLoopback } from './journal-client-loopback.js'; + +it('uses pushes for completion with lease-cadence reads and releases its watcher', async () => { + let reads = 0; + let watchClosed = false; + const path = sockPath(); + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': (ctx, params) => { + const entry = (run: string, type: string) => ctx.send({ event: 'entry', data: { + run_id: run, step_id: 'step', entry_type: type, + } }); + // An older failed attempt in replay must not finish the current one. + entry('run', 'step.completed'); + entry('run', 'step.attempt.started'); + sendResult(ctx, {}); + entry('other-run', 'step.completed'); + setTimeout(() => entry(String(params['run_id']), 'step.completed'), 2100); + ctx.socket.once('close', () => { watchClosed = true; }); + }, + 'run.get': ctx => { reads++; sendResult(ctx, { steps: { step: { + state: 'running', lease_deadline_ms: Date.now() + 30_000, + } } }); }, + }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const start = performance.now(); + await waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}); + expect(performance.now() - start).toBeGreaterThan(2000); + expect(reads).toBeLessThanOrEqual(1); + await sleep(10); + expect(watchClosed).toBe(true); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('cancels a pending watch registration and removes its connection', async () => { + const path = sockPath(); + let registered!: () => void; + const registration = new Promise(resolve => { registered = resolve; }); + const server = startLoopback(path, { hello: sendOk, 'run.watch': () => registered() }); + await once(server, 'listening'); + const client = new JournalClient(path); + const controller = new AbortController(); + try { + await client.connect(); + const waiting = waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, { signal: controller.signal }); + const rejected = expect(waiting).rejects.toThrow('was canceled'); + await registration; + controller.abort(); + await rejected; + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('cancels promptly while a lease snapshot read is in flight', async () => { + const path = sockPath(); + let reading!: () => void; + const readStarted = new Promise(resolve => { reading = resolve; }); + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), + // The daemon never answers the snapshot: only cancellation can end the wait. + 'run.get': () => reading(), + }); + await once(server, 'listening'); + const client = new JournalClient(path, { readBudgetMs: 300_000 }); + const controller = new AbortController(); + try { + await client.connect(); + const waiting = waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, { signal: controller.signal }); + const outcome = waiting.then(() => 'returned', error => error); + await readStarted; + controller.abort(); + const settled = await Promise.race([outcome, sleep(200).then(() => 'still waiting')]); + expect(settled).toBeInstanceOf(Error); + expect((settled as Error).message).toContain('was canceled'); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('cancelling during the watch handshake leaves no unhandled rejection', async () => { + const path = sockPath(); + let greeted!: () => void; + const hello = new Promise(resolve => { greeted = resolve; }); + const server = startLoopback(path, { hello: () => greeted() }); // never answers the watch hello + await once(server, 'listening'); + const client = new JournalClient(path); + const controller = new AbortController(); + const unhandled: unknown[] = []; + const onUnhandled = (reason: unknown) => { unhandled.push(reason); }; + process.on('unhandledRejection', onUnhandled); + try { + await client.connect(); + const waiting = waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, { signal: controller.signal }); + const rejected = expect(waiting).rejects.toThrow('was canceled'); + await hello; + controller.abort(); + await rejected; + await sleep(50); + expect(unhandled).toEqual([]); + } finally { + process.off('unhandledRejection', onUnhandled); + client.close(); await new Promise(resolve => server.close(() => resolve())); + } +}); + +it('treats a watch connection lost during registration as a read interruption', async () => { + const path = sockPath(); + let hellos = 0; + const server = startLoopback(path, { + hello: ctx => { hellos += 1; sendOk(ctx); }, + 'run.watch': ctx => { ctx.socket.destroy(); }, // registered hello, then the socket drops + }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const error = await waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}) + .catch(caught => caught); + expect(error).toBeInstanceOf(AuthoredFlowExecutionError); + expect(error.code).toBe('daemon_unresponsive'); + expect(isReadInterruption(error)).toBe(true); + expect(hellos).toBe(1); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('a completion push ends the wait without waiting out an in-flight snapshot', async () => { + const path = sockPath(); + let push!: () => void; + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': (ctx, params) => { + sendResult(ctx, {}); + push = () => ctx.send({ event: 'entry', data: { run_id: String(params['run_id']), step_id: 'step', entry_type: 'step.completed' } }); + }, + // The snapshot never answers; the completion arrives while it is in flight. + 'run.get': () => push(), + }); + await once(server, 'listening'); + const client = new JournalClient(path, { readBudgetMs: 300_000 }); + try { + await client.connect(); + const waiting = waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}); + const outcome = await Promise.race([waiting.then(() => 'returned'), sleep(3_000).then(() => 'still waiting')]); + expect(outcome).toBe('returned'); + expect(await client.runGet('run', { signal: AbortSignal.timeout(50) }).catch(error => error.name)).toBe('TimeoutError'); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('a completion that aborts the snapshot is not an error even if a retry starts at once', async () => { + const path = sockPath(); + let push!: () => void; + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': (ctx, params) => { + sendResult(ctx, {}); + const entry = (type: string) => ctx.send({ event: 'entry', data: { run_id: String(params['run_id']), step_id: 'step', entry_type: type } }); + push = () => { entry('step.completed'); entry('step.attempt.started'); }; + }, + 'run.get': () => push(), + }); + await once(server, 'listening'); + const client = new JournalClient(path, { readBudgetMs: 300_000 }); + try { + await client.connect(); + const outcome = await Promise.race([ + waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}) + .then(() => 'returned', error => `threw: ${error.message}`), + sleep(3_000).then(() => 'still waiting'), + ]); + expect(outcome).toBe('returned'); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('fails closed on a malformed frame during watch setup', async () => { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, 'run.watch': ctx => { ctx.socket.write('{not json\n'); } }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const error = await waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}) + .catch(caught => caught); + expect(error).toBeInstanceOf(JournalFrameError); + expect(isReadInterruption(error)).toBe(false); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('fails closed on a malformed frame after the watch is registered', async () => { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, 'run.watch': ctx => { + sendResult(ctx, {}); + setTimeout(() => ctx.socket.write('{not json\n'), 20); + } }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const outcome = await Promise.race([ + waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}).then(() => 'returned', error => error), + sleep(1_000).then(() => 'still waiting'), + ]); + expect(outcome).toBeInstanceOf(JournalFrameError); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('a malformed watch frame during an in-flight snapshot is an error, not a completion', async () => { + const path = sockPath(); + let corrupt!: () => void; + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': ctx => { sendResult(ctx, {}); corrupt = () => ctx.socket.write('{not json\n'); }, + 'run.get': () => corrupt(), // the frame breaks while the snapshot is in flight + }); + await once(server, 'listening'); + const client = new JournalClient(path, { readBudgetMs: 300_000 }); + try { + await client.connect(); + const outcome = await Promise.race([ + waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}).then(() => 'returned', error => error), + sleep(4_000).then(() => 'still waiting'), + ]); + expect(outcome).toBeInstanceOf(JournalFrameError); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('a watch connection lost after registration is a read interruption, not a silent wait', async () => { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, 'run.watch': ctx => { + sendResult(ctx, {}); + setTimeout(() => ctx.socket.destroy(), 20); + } }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const outcome = await Promise.race([ + waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}).then(() => 'returned', error => error), + sleep(1_000).then(() => 'still waiting'), + ]); + expect(outcome).toBeInstanceOf(AuthoredFlowExecutionError); + expect(isReadInterruption(outcome)).toBe(true); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('a watch dropped during an in-flight snapshot is reported at once, not after the read budget', async () => { + const path = sockPath(); + let drop!: () => void; + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': ctx => { sendResult(ctx, {}); drop = () => ctx.socket.destroy(); }, + 'run.get': () => drop(), // the watch drops while the snapshot is in flight + }); + await once(server, 'listening'); + const client = new JournalClient(path, { readBudgetMs: 300_000 }); + try { + await client.connect(); + const outcome = await Promise.race([ + waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}).then(() => 'returned', error => error), + sleep(4_000).then(() => 'still waiting'), + ]); + expect(outcome).toBeInstanceOf(AuthoredFlowExecutionError); + expect((outcome as AuthoredFlowExecutionError).code).toBe('daemon_unresponsive'); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('honours a replayed completion even if the watch drops before registration answers', async () => { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, 'run.watch': (ctx, params) => { + ctx.send({ event: 'entry', data: { run_id: String(params['run_id']), step_id: 'step', entry_type: 'step.completed' } }); + setTimeout(() => ctx.socket.destroy(), 10); // replay delivered, response never sent + }, 'run.get': ctx => sendResult(ctx, { steps: { step: { state: 'completed' } } }) }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + await expect(waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {})) + .resolves.toBeUndefined(); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('confirms a replayed completion with a snapshot when the replay may be truncated', async () => { + for (const [state, expected] of [['completed', 'returned'], ['running', 'interrupted']] as const) { + const path = sockPath(); + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': (ctx, params) => { + // An older attempt's completion arrives; the drop may hide a later retry. + ctx.send({ event: 'entry', data: { run_id: String(params['run_id']), step_id: 'step', entry_type: 'step.completed' } }); + setTimeout(() => ctx.socket.destroy(), 10); + }, + 'run.get': ctx => sendResult(ctx, { steps: { step: { state, lease_deadline_ms: Date.now() + 30_000 } } }), + }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const outcome = await waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}) + .then(() => 'returned', error => (isReadInterruption(error) ? 'interrupted' : `other: ${error.message}`)); + expect(outcome).toBe(expected); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } + } +}); diff --git a/packages/sdk/tests/worker-lease-sweep.test.ts b/packages/sdk/tests/worker-lease-sweep.test.ts index 1cc66b931..c3348a308 100644 --- a/packages/sdk/tests/worker-lease-sweep.test.ts +++ b/packages/sdk/tests/worker-lease-sweep.test.ts @@ -1,3 +1,4 @@ +import { EventEmitter } from 'node:events'; import { afterEach, expect, it, vi } from 'vitest'; import { classifyOutcome, emptyReport } from '../src/cli/run.js'; import type { JournalClient } from '../src/journal-client.js'; @@ -14,10 +15,13 @@ function fixture() { }); const parked = { run_id: 'run', status: 'parked' as const, completion_reason: null, completed_steps: 0 }; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: vi.fn().mockResolvedValue(snapshot('running')), runResume: vi.fn().mockResolvedValue(parked), }; - const run = () => classifyOutcome(client as unknown as JournalClient, 'run', parked, emptyReport('run'), '/unused', {}); + const run = () => classifyOutcome(client as unknown as JournalClient, 'run', parked, emptyReport('run'), '', {}); return { client, snapshot, run }; } @@ -31,14 +35,14 @@ it('waits through an expired snapshot until the kernel retries and completes', a .mockResolvedValue({ run_id: 'run', status: 'completed', completion_reason: 'success', completed_steps: 1 }); const execution = run(); const assertion = expect(execution).resolves.toMatchObject({ exitCode: 0, report: { completionReason: 'success' } }); - await Promise.all([assertion, vi.advanceTimersByTimeAsync(1_000)]); + await Promise.all([assertion, vi.advanceTimersByTimeAsync(4_000)]); expect(client.runResume).toHaveBeenCalledTimes(2); }); it('still fails when the kernel never resolves an expired lease after sweep grace', async () => { const { run } = fixture(); const assertion = expect(run()).rejects.toThrow('without completion'); - await vi.advanceTimersByTimeAsync(35_500); + await vi.advanceTimersByTimeAsync(36_500); await assertion; }); @@ -46,7 +50,7 @@ it('bounds an unchanged lease when the CLI clock is behind the daemon', async () const { client, snapshot, run } = fixture(); client.runGet.mockResolvedValue(snapshot('running', Date.now() + 75_000)); const assertion = expect(run()).rejects.toThrow('without completion'); - await vi.advanceTimersByTimeAsync(35_500); + await vi.advanceTimersByTimeAsync(36_500); await assertion; }); @@ -69,5 +73,5 @@ it('follows a live step whose renewed deadline looks expired on a skewed CLI clo client.runGet.mockResolvedValue(snapshot('completed')); client.runResume.mockResolvedValue({ run_id: 'run', status: 'completed', completion_reason: 'success', completed_steps: 1 }); const assertion = expect(execution).resolves.toMatchObject({ exitCode: 0, report: { completionReason: 'success' } }); - await Promise.all([assertion, vi.advanceTimersByTimeAsync(1_000)]); + await Promise.all([assertion, vi.advanceTimersByTimeAsync(4_000)]); }); diff --git a/packages/sdk/tests/worker-lease.test.ts b/packages/sdk/tests/worker-lease.test.ts index 0cb168b81..6a769ac47 100644 --- a/packages/sdk/tests/worker-lease.test.ts +++ b/packages/sdk/tests/worker-lease.test.ts @@ -161,3 +161,15 @@ it('aborts the CLI when a later heartbeat response would revive an expired lease expect(client.stepComplete).not.toHaveBeenCalled(); expect(String(errors[0])).toContain('lease expired before renewal'); }); + +it('hands a worker read interruption back to the kernel instead of failing fatally', async () => { + const { onWorkerFailure } = await import('../src/worker-lease.js'); + const { JournalRequestTimeoutError } = await import('../src/journal-client.js'); + const fatal = vi.fn(); + const warn = vi.spyOn(process, 'emitWarning').mockImplementation(() => undefined); + onWorkerFailure('communication', fatal)(new JournalRequestTimeoutError('journal.read', 10, 3, 300_000, 300_000), + { run_id: 'r', step_id: 's', attempt: 2 } as never); + expect(fatal).not.toHaveBeenCalled(); + expect(warn).toHaveBeenCalled(); + warn.mockRestore(); +});