Skip to content

fix(eve): settle turn as cancelled when abort races step completion - #1389

Open
AndrewBarba wants to merge 4 commits into
mainfrom
barba/turn-cancel-race
Open

fix(eve): settle turn as cancelled when abort races step completion#1389
AndrewBarba wants to merge 4 commits into
mainfrom
barba/turn-cancel-race

Conversation

@AndrewBarba

Copy link
Copy Markdown
Collaborator

Why

A turn cancellation observed while a durable turn step is already returning loses the race to an ordinary completion: the step body can miss the abort signal and finish done, leaving the session in a completed state after the user's cancel was claimed.

What

In runTurnOwnedWorkflow, honor the claimed cancel whenever the cancellation signal has fired and the step produced no pending runtime-action batch — pending batches are exempt because their in-line wait observes the signal itself. The turn then settles through the existing cancelled-park path (finishCancelledTurn), whose parkability was already established when the cancel hook was claimed.

Includes a unit test that returns a done result only after the step's abort signal fires, asserting the turn parks as cancelled and never resumes as done.

Provenance

Extracted from #1367, where deterministic-model timing in the e2e world suites made turns complete fast enough to expose the race. Split out here so the runtime fix can be reviewed independently of the e2e suite changes (#1387).

Testing

  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/execution/turn-workflow.test.ts (23 passed)
  • Full unit + integration suites green; fmt / lint / guard:invariants / typecheck clean

A cancellation observed while a durable turn step was already returning
lost the race to an ordinary completion: the step body could miss the
abort signal and finish done, leaving the session completed after the
user had cancelled. Honor the claimed cancel whenever the abort signal
fired and the step produced no pending runtime-action batch (those
resolve in-line and observe the signal in their own wait).

Surfaced by deterministic-model timing in the e2e world suites, where
turns reach completion fast enough to expose the race (originally found
in #1367).

Signed-off-by: Andrew Barba <barba@hey.com>
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
eve-docs Ready Ready Preview Jul 30, 2026 3:11am
eve-docs-4759 Ready Ready Preview, v0 Jul 30, 2026 3:11am

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

  • No notable deltas vs main (46c6ce3).

Delta vs main (46c6ce3)

Area Metric Baseline Current Delta
Package Packed tarball 7.62 MB 7.62 MB +1.9 kB ⚠️
Package Unpacked publish size 28.79 MB 28.80 MB +10.9 kB ⚠️
Package Installed footprint 91.09 MB 91.10 MB +10.9 kB ⚠️
Package Published files 2875 2879 +4
Package Installed files 6662 6666 +4
Runtime Unique function payloads 2 2 0
Runtime Total function bytes 16.80 MB 16.82 MB +19.3 kB ⚠️
Runtime Public routes 11 11 0
Changed function payloads vs main (46c6ce3) (2)
Function Status Baseline Current Delta Route changes
functions/__server.func changed 8.40 MB 8.41 MB +9.6 kB ⚠️ none
functions/.well-known/workflow/v1/flow.func changed 8.40 MB 8.41 MB +9.6 kB ⚠️ none

eve init install

Metric Baseline Current Delta
Installed footprint 129.49 MB 129.50 MB +10.9 kB ⚠️
Installed packages 130 130 0
dependencies 4 4 0
devDependencies 2 2 0
Dependency package bytes 43.12 MB 43.13 MB +10.9 kB ⚠️
devDependency package bytes 5.04 MB 5.04 MB 0 B ➖
Build Metadata
  • Preset: vercel
  • Nitro: nitro@3.0.260610-beta
  • Output directory: apps/fixtures/weather-agent/.vercel/output
  • Build metadata timestamp: 2026-07-30T03:12:18.204Z
  • Route aliases: 11 public, 1 internal (12 total aliases)
  • Vercel routes in config: 14
  • Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small
Package Drill-Down

Package Details

  • Package: eve@0.27.13
  • Package directory: packages/eve
  • Tarball: 7.62 MB (eve-0.27.13.tgz)
  • Unpacked payload: 28.80 MB across 2879 published files
  • Installed footprint: 91.10 MB across 6666 installed files
  • Installed root package: 27.44 MB
  • Installed dependencies: 63.66 MB
  • Runtime dependencies: 2
  • Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

  • eve: 27.44 MB (30.1%)
  • @rolldown/binding-linux-x64-gnu: 19.28 MB (21.2%)
  • @rolldown/binding-wasm32-wasi: 10.66 MB (11.7%)
  • ai: 6.53 MB (7.2%)
  • @napi-rs/wasm-runtime: 6.39 MB (7.0%)
Publish payload breakdown
Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [########################] 13.15 MB 45.7%
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [###.....................] 1.51 MB 5.2%
🟡 dist/src/compiled/@vercel/sandbox/index.js       [#.......................] 632.5 kB 2.2%
🟡 dist/src/compiled/_chunks/workflow/undici-DWL... [#.......................] 502.4 kB 1.7%
🟡 dist/src/compiled/@chat-adapter/slack/index.js   [#.......................] 440.5 kB 1.5%
🔴 Other published files                            [#######################.] 12.56 MB 43.6%
Installed footprint breakdown
Installed package size
🔴 eve                             [########################] 27.44 MB 30.1%
🔴 @rolldown/binding-linux-x64-gnu [#################.......] 19.28 MB 21.2%
🔴 @rolldown/binding-wasm32-wasi   [#########...............] 10.66 MB 11.7%
🔴 ai                              [######..................] 6.53 MB 7.2%
🔴 @napi-rs/wasm-runtime           [######..................] 6.39 MB 7.0%
🔴 zod                             [####....................] 5.07 MB 5.6%
🔴 Other installed packages        [##############..........] 15.73 MB 17.3%
Runtime dependencies (2)
Package Range Notes
nitro 3.0.260610-beta
undici 8.9.0
Peer dependencies (5)
Package Range Notes
@opentelemetry/api ^1.0.0 optional peer
ai catalog:
braintrust ^3.0.0 optional peer
just-bash ^3.0.0 optional peer
microsandbox ^0.5.0 optional peer
eve init install drill-down

eve init install details

  • Command: eve init my-agent
  • Package manager: npm
  • Installed footprint: 129.50 MB across 8534 installed files
  • Installed packages: 130 total (124 transitive-only)
  • dependencies: 4 direct packages totaling 43.13 MB
  • devDependencies: 2 direct packages totaling 5.04 MB
  • Other transitive package files: 81.33 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

  • @typescript/typescript-linux-x64: 27.95 MB (21.6%)
  • eve: 27.44 MB (21.2%)
  • @rolldown/binding-linux-x64-gnu: 19.28 MB (14.9%)
  • @rolldown/binding-wasm32-wasi: 10.66 MB (8.2%)
  • zod: 9.02 MB (7.0%)
Installed footprint breakdown
Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 21.6%
🔴 eve                              [########################] 27.44 MB 21.2%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.28 MB 14.9%
🔴 @rolldown/binding-wasm32-wasi    [#########...............] 10.66 MB 8.2%
🔴 zod                              [########................] 9.02 MB 7.0%
🔴 ai                               [######..................] 6.53 MB 5.0%
🔴 @napi-rs/wasm-runtime            [#####...................] 6.39 MB 4.9%
🔴 Other installed packages         [###################.....] 22.23 MB 17.2%
dependencies (4)
Package Range Installed size Share
@vercel/connect 0.4.2 135.8 kB 0.1%
ai ^7.0.38 6.53 MB 5.0%
eve file:eve-0.27.13.tgz 27.44 MB 21.2%
zod 4.4.3 9.02 MB 7.0%
devDependencies (2)
Package Range Installed size Share
@types/node 24.x 2.54 MB 2.0%
typescript 7.0.2 2.50 MB 1.9%
Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 8.41 MB 50.0%
🔴 functions/__server.func                         [########################] 8.41 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 8.41 MB
Metric Value
Public routes /.well-known/workflow/v1/flow
Runtime nodejs24.x
Handler index.mjs
Payload 8.41 MB
Function files 8.41 MB across 44 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.31 MB (27.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [########################] 2.31 MB 27.5%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 18.9%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.7%
🟡 _chunks/sandbox.mjs             [########................] 768.8 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.33 MB 27.6%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 10 public routes, 1 internal alias • 8.41 MB
Metric Value
Public routes /
/eve/v1/callback/[token]
/eve/v1/connections/[name]/callback/[token]
/eve/v1/health
/eve/v1/info
/eve/v1/session
/eve/v1/session/[sessionId]
/eve/v1/session/[sessionId]/cancel
/eve/v1/session/[sessionId]/stream
/eve/v1/session/reset
Internal aliases /__server
Runtime nodejs24.x
Handler index.mjs
Payload 8.41 MB
Function files 8.41 MB across 44 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.31 MB (27.5%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [########################] 2.31 MB 27.5%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 18.9%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.7%
🟡 _chunks/sandbox.mjs             [########................] 768.8 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.32 MB 27.6%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

  • Build pipeline: 2.21 s -> 1.85 s (-360.1 ms) vs main (46c6ce3).
  • Timing is informational: shared GitHub runners are too variable for a hard timing budget.
Detailed phase timings vs `main (46c6ce3)`
Phase Baseline Current Delta
extension.check 9.7 ms 7.1 ms -2.6 ms
project.resolve 3.5 ms 2.5 ms -1.0 ms
workspace.create 1.2 ms 1.2 ms 0.0 ms
host.prepare 518.2 ms 160.9 ms -357.3 ms
vercel.service-prefix.resolve 1.7 ms 1.7 ms 0.0 ms
nitro.create 193.0 ms 191.3 ms -1.7 ms
sandbox.prewarm 187.8 ms 193.9 ms +6.1 ms
nitro.cache.prepare 0.3 ms 0.3 ms 0.0 ms
nitro.prepare 0.8 ms 0.8 ms 0.0 ms
nitro.public-assets 0.7 ms 0.7 ms 0.0 ms
nitro.prerender 0.5 ms 0.5 ms 0.0 ms
nitro.bundle 1.26 s 1.26 s -3.4 ms
nitro.cache.write 0.3 ms 0.3 ms 0.0 ms
vercel.workflow-function.materialize 21.4 ms 21.1 ms -0.3 ms
agent-summary.emit 0.4 ms 0.5 ms +0.1 ms
nitro.close 0.1 ms 0.1 ms 0.0 ms
output.publish 3.2 ms 3.2 ms 0.0 ms
workspace.remove 1.8 ms 1.9 ms +0.1 ms

The e2e cancellation flake has a second face beyond the settle race: on
the Vercel world, a cancelled parent's cascade to its subagent child can
drop entirely — the child's wait-for-cancellation tool rides out its
fallback timeout and the child runs to completion with no turn.cancelled.
Every drop path in cancelDescendantTurnsStep was silent: a missing
pending batch, missing child session ids, and retry exhaustion all
returned as if the cancel had been delivered.

Retry descendant cancels with exponential backoff (~8s budget instead of
a flat 3s), classify no_active_turn results with the error that marked
the target inactive (HookNotFoundError vs EntityConflictError etc.), and
warn on every path that gives up so an uncancelled child is at least
observable in logs while a durable delivery design is worked out.

Signed-off-by: Andrew Barba <barba@hey.com>
@AndrewBarba

Copy link
Copy Markdown
Collaborator Author

Added a second commit after digging into this PR's own e2e failures: the two e2e-vercel (agent-cancellation, *) legs exposed a second cancellation bug, distinct from the settle race this PR fixes.

Failure forensics (cancel-subagent, both model legs): the parent settled turn.cancelled correctly, but the child subagent never received its cascade cancel — its wait-for-cancellation tool rode out the full 90s fallback timeout, then the child ran another model step and completed done with zero turn.cancelled. Since the child's next step was serialized with an unaborted signal, the cancel-hook resume never reached the child run at all — a dropped delivery, not latency.

Every drop path in cancelDescendantTurnsStep was silent: missing pending batch, missing childSessionIds, and retry exhaustion (12×250ms, then return-as-success) all looked identical to a delivered cancel.

The hardening commit:

  • exponential backoff on descendant cancel requests (~8s budget instead of a flat 3s)
  • no_active_turn results now carry the classifying error (HookNotFoundError = target likely finished, EntityConflictError = transient contention, …)
  • warn-level logs on every give-up path, so an uncancelled child is observable instead of invisible

This makes the drop far less likely and diagnosable when it still happens. A fully durable delivery (child consults durable cancel state at settle time, instead of best-effort hook resume) is a design change worth its own research doc — this keeps the fix reviewable.

The remaining e2e failures on the first run (agent-tools web-search legs) are the pre-existing live-model flake that #1387 quarantines.

A wake that replays a journaled cancel alongside a completed turn step
drives both continuations in one microtask drain, payload first. The
abort was chained on consumeMatchingCancel with .then, so it ran one
hop behind the turn loop's continuation: the loop checked
signal.aborted before the abort fired, took the done arm, and disposed
the pending cancel. Fire the abort inside the matching read's own
continuation so the signal is already flipped when any same-drain
continuation checks it.

Observed on the Vercel world as a subagent child completing normally
after its parent's cascade cancel had been accepted.

Signed-off-by: Andrew Barba <barba@hey.com>
@AndrewBarba

Copy link
Copy Markdown
Collaborator Author

The single e2e-vercel (agent-cancellation, anthropic-opus) failure on the previous run (same child-never-cancelled signature, now intermittent — sol passed) pinned down a third defect, fixed in 4e222be1.

The microtask race: when the child run wakes and replays a journaled cancel alongside a completed turn step, both continuations resolve in the same drain, payload first. But the abort was chained on consumeMatchingCancel(...).then(() => controller.abort(...)) — one microtask hop behind the turn loop's own continuation. The loop checked signal.aborted before the abort callback ran, saw false, took the done arm, and dispose() swallowed the pending cancel permanently. The commit-1 settle check could lose by exactly one hop.

Fix: fire the abort inside the matching read's own continuation, so the signal is already flipped when any continuation from the same drain checks it. The new unit test reproduces the drain ordering (payload resolves before the step result) and fails against the chained-.then implementation.

Residual risk, for the record: the run→step abort bridge (the durable-stream side channel workflow-core uses to propagate an abort into an in-flight step invocation) swallows write failures; if that hop is lost, the child's tool still rides out its fallback timeout before the (now-reliable) run-side settle path catches the cancel at the next step boundary — slow, but no longer droppable by eve. Making descendant cancellation fully durable (child consults durable cancel state at settle) is follow-up design work.

The descendant cascade previously lived only inside the parent's
workflow (cancelDescendantTurnsStep), which cannot execute while the
parent run is suspended — including while it is parked awaiting the
very child it needs to cancel. When the workflow world deferred the
parent's cancel wake until the child's completion delivery, the
cascade was dead on arrival: the child ran its full fallback timeout
and completed uncancelled (observed repeatedly in the cancel-subagent
e2e eval).

runtime.cancelTurn now cancels the target and immediately fans out to
every dispatched-but-unresolved descendant, recursively, without any
run having to wake. Descendants are derived from the session's own
durable event stream — subagent.called records the dispatch and
action.result marks it resolved — read up to the tail observed on
entry so a live stream cannot block the request. Remote children are
re-resolved from the registry by dispatch URL and cancelled over the
standard remote cancel route, which fans out on its own deployment.

The settle-time cascade remains as the durable backstop, but no longer
burns its 8-attempt retry budget on children that were already
cancelled at request time: terminal no_active_turn reasons (hook gone,
run gone, run expired) return after a single attempt, and only
hook-claim contention is retried.

Signed-off-by: Andrew Barba <barba@hey.com>
@AndrewBarba

Copy link
Copy Markdown
Collaborator Author

Root cause found and fixed structurally in 66ca1a28 — this is not another patch on the turn loop.

The smoking gun (from the run-3 artifact timestamps): the cancel was accepted at 02:27:52, but the parent run — the only place the descendant cascade could execute — never woke until the child's completion delivery at 02:29:24. The child's wait-for-cancellation tool reported "fallback timeout reached without cancellation": no cancel ever reached it, because the cascade lived exclusively inside the suspended parent's workflow. The 9.9s gap between child completion (02:29:24.301) and the parent's turn.cancelled (02:29:34.220) is exactly the settle-time cascade's 8-attempt retry backoff against the already-completed child's missing hook. Every millisecond accounted for; commits 1–3 were working as designed and are why the parent settled cancelled at all.

The fix: runtime.cancelTurn now fans cancellation out to descendants immediately at request time, with no run wake required. Descendants are derived from the session's durable event stream (subagent.called minus action.result), read up to the tail observed on entry so a live stream can't block the request, and cancelled recursively — local children directly, remote children via the standard remote cancel route (which fans out on its own deployment). The settle-time cascade remains as the durable backstop, but terminal no_active_turn reasons now return after a single attempt instead of burning the 10s retry budget on children already cancelled at request time.

Deliberately avoided: extending the public subagent.called event (it would have bumped all four extension-capability contract epochs); remote nodes are re-resolved from the registry by dispatch URL instead.

Unit (5,670) + integration (568) green, guard/docs checks pass. The dropped/deferred hook-resume wake in the workflow world is still worth an upstream look — with this change eve no longer depends on it for cancellation correctness, but it cost this parent 92 seconds of suspension with an accepted cancel pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant