vein: run control — cancel, pause/resume, durable resume (RUN_CONTROL_SPEC)

[Evanfeenstra]

Implements all three rungs of vein/RUN_CONTROL_SPEC.md — cancel, pause/resume, and durable resume for run trees, including the nested trees the lab produces (harvey-evolve → optimizer.run → harvey-evolve-gen → …).

Rung 1 — Cancel

• RunController (src/run-control.ts): cooperative checkpoint() awaited at every boundary — between DAG steps, loop/foreach iterations, retry attempts. Registered at every launch site (launchDetached, vein.run, authoring's runWorkflow, chat's run_workflow), superseding the old activeRuns set.
• Tree linkage: nested launches attach via parentRunId (meta/run-workflow and the lab optimizer pass ctx.runId), so cancelling a parent cancels its whole subtree via the effective-state walk.
• CancelledError is a distinct outcome: run.cancelled terminal event, status: "cancelled", never retried or diverted into onError, onRunEnd teardown still fires, partial outputs stay journaled. executeFlow now settles every in-flight branch before finalizing, so no step events land after the terminal event.
• POST /workflows/:name/runs/:runId/cancel (404 unknown, 409 terminal) + a topbar Cancel button with a subtree-consequence confirm.

Rung 2 — Pause / resume (in-memory)

• pause() parks every active branch at its next boundary; quiescence is tracked via unit accounting (a parked ctx.control.checkpoint() releases its unit, so a paused agent counts as quiesced). run.paused/run.resumed markers record the gap in the log; the runs listing reports paused.
• The agent step checkpoints between tool calls (prepareStep) — the highest-value boundary in the lab's workloads. evolve-loop and eval/optimize checkpoint between generations via ctx.control.

Rung 3 — Durable resume (journal replay)

• src/journal.ts: step.end outputs keyed by path (last write wins) become the replay journal; runWorkflow({ journal, resume: true }) replays journaled paths as step.replayed (zero cost, no side effects) and executes live from the first missing path. Skip/gate logic reconstructs naturally; foreach re-runs only the failed #i; loop replays iterations and re-evaluates until against replayed $current.
• Crash-hardened: torn trailing JSONL lines are skipped; run.start records the workflow content hash (resume refuses a changed DAG unless forced) plus per-run params/paramOverrides so re-execution uses the original knobs; run.resumed reopens SSE tails past an old terminal event.
• Failure-resume + from invalidation: error/cancelled/stale runs resume as-is; a successful run needs from: <stepPath> — the target, its transitive dependents (computed per flow level, recursing through statically-named subflows), its iteration children, and later iterations of an enclosing loop are dropped from the journal.
• POST …/resume is dual-purpose: live controller → releases a pause; dead run → journal replay. UI: Resume on stale/error/cancelled runs, "Re-run from here" in the step flyout (drill-aware true paths).
• harvey/evolve-loop consumes ctx.journal: journaled generations replay with loop state (best / sinceImprove / stop logic) rebuilt from their synthetic #gen outputs, which now carry the approach summary + digest excerpt so later briefings stay faithful.

Tests

36 new tests in src/run-control.test.ts (controller semantics, cancel/pause mid-DAG/mid-foreach/mid-retry, replay correctness incl. skip/gate reconstruction and ctx.journal, torn tail, tail reopening, and full HTTP round-trips of cancel/pause/resume + the hash guard + re-run-from-here). Full suite: 533/533; vein, web, and mcp all typecheck.

🤖 Generated with Claude Code