Repository navigation
[feat] Publish and resume complete workflow state across machines #1084
Description
Parent: #818
Goal
Allow explicit handoff of a composed workflow run between machines without a server or concurrent multi-machine leasing.
Scope
- Add
teamai workflow publishandresume. - Publish a masked snapshot under
runs/<id>/state.yamlonteamai-reports. - Serialize DAG progress, nested sub-workflow frames, expression-visible results, named ends, loop/fan-out state when present, workspace and branch lineage, cleanup state, and session-continuation lineage.
- Record the names of branches the previous owner pushed, for manual recovery.
resumedoes not merge or delete those branches. - Preserve machine/tool bindings needed to decide whether a session may resume, without publishing private session contents.
- Exclude prompts, secrets, unsafe outputs, and absolute machine paths.
- Record owner, state revision, team revision, and publication base.
- Claim ownership when another machine resumes.
- Reject publication from a machine that no longer owns the latest snapshot.
- Return in-flight leases to pending with new attempts.
- Resume non-portable sessions fresh while preserving their semantic continuation point.
- Reuse reports-branch locking and push-conflict handling.
Out of scope
Run records, concurrent multi-machine leasing, automatic branch recovery, live synchronization, state merging, and event waits.
Dependencies
- Blocked by: [feat] Add DAG conditions and composable sub-workflows #1078, [feat] Add workflow workspaces, branches, and session continuation #1080, [feat] Define the team revision once for run records, labels and snapshots #1163
- Unblocks: none
Acceptance criteria
- A nested DAG run with workspaces and session continuation survives machine handoff.
- Machine B can claim a published run; A’s next publish is rejected.
- In-flight attempts receive new IDs.
- Snapshots contain no secret, prompt, private session content, or absolute machine path.
- The snapshot preserves pushed branch names for manual recovery, and
resumeneither merges nor deletes those branches. - Local-only runs never touch Git.
Verification
Focused serialization, migration, two-clone ownership, pushed-branch snapshot, masking, and attempt-renewal tests; build/typecheck/lint; one real-CLI publish/resume E2E covering a sub-workflow and workspace. Regenerate commands.md and update both guides.
Metadata
Metadata
Assignees
Labels
No labels
Activity
SaulMoro commentedon Oct 13, 2026
#818 says a branch the old driver pushed is noted in the run record for manual recovery, and never merged automatically. Automatic recovery is out of scope here, which matches, but noting the branch isn't in this ticket or in #1083. Maybe a line here?
hsuchifeng commentedon Oct 14, 2026
The snapshot records the branch names the previous owner pushed so they are available for manual recovery.
resumedoes not merge or delete those branches.Automatic recovery stays out of scope. Run records stay in #1083.
SaulMoro commentedon Oct 14, 2026
Notes from #1078 (#1129) for publishing and resuming run state:
start, with onesourceHash, so resuming on another machine does not need that machine's cache to match.inputs, and leases carry renderedenvvalues. Publishing a snapshot has to treat both as possibly sensitive (see [feat] Enforce workflow requirements and execution limits #1082's masking).needsmakesnextreturnrun_unreadableinstead of running that step as a root. A resumed snapshot from an older teamai hits the same check.The engine rules that span files are in
docs/dev/workflow-engine.md, added by #1129.SaulMoro commentedon Oct 15, 2026
For "Record owner, state revision, team revision": the team revision follows #1163. It is the commit the run's cached definitions came from, and the run state already carries it, so the snapshot copies it rather than reading
HEADon the publishing machine. The snapshot already carriessourceHashas the exact identity of what ran.Acceptance criterion to add: a run started at commit A and published after the team repo moves to commit B has a snapshot with
teamRevision: A.SaulMoro commentedon Oct 16, 2026
Two things from #1148 for the snapshot:
run.archiveis part of the state. A loop that runs a sub-workflow orfor_eachagain moves the old children torun.archive, and a new instance numbers its pass after them (firstPass). A snapshot without it reuses pass ids, which collide in [feat] Add workflow scorers and recorded quality labels #1085 labels and [feat] Add workflow workspaces, branches, and session continuation #1080 branch names, and turns reads of a previous pass into null. The archive is never trimmed. The run file of a 1,000-itemfor_eachthat a loop ran 10 times was 4.66 MB, against 0.51 MB for one pass.schemaVersion: v1alpha1. A CLI without loops continues a run that uses them and runs the rest without them, and a CLI with loops drops step keys it does not know when it reads a run. On one machine that takes a downgrade. With snapshots it takes two members on different versions. A snapshot field that names the features the run uses, which a CLI refuses to resume when it does not know one, would make this fail closed.SaulMoro commentedon Oct 20, 2026
Things from #1175 that a resume on another machine has to handle:
run.copies: { [branch]: { owner: DriverIdentity, commit? } }.nextholds back anyneworcontinuestep whose branch is owned by a differentkind/machine/workspace(workspace.ts:260-263). The owner is recorded at lease time (engine.ts:261-265), so a step in flight at handoff is held back too, and on machine B the local driver stops withbusy(local-driver.ts:850). Claiming a run should rewrite everycopies[*].ownerto the claiming driver, andcontinuekeeps starting fromcopies[*].commit, the last commit a step reported, as proposed on [feat] Add workflow workspaces, branches, and session continuation #1080.owner.workspace,lease.driverandrun.driverhold absolute paths (src/workflow/types.ts:185,:206-207; the local driver setsworkspace: path.resolve(cwd),local-driver.ts:54), so the snapshot drops them; resume rewrites them anyway. Criterion: a run published while acontinuestep is in flight resumes on another clone, and that step starts from the last reported commit of its branch.base.commit,start.commitandmerge[].commit, and the local driver never fetches. On machine B, merging a commit B lacks fails, andmergeBranchreports it as a conflict with no files (worktree.ts:123-125): a command failsmerge_conflict(local-driver.ts:383), and a prompt is told to resolve a conflict that does not exist. I'd have the driver fetch fromoriginwhen a commit the lease needs is missing, and fail with its own code.resumeruns the same check forrun.baseand every commit the run still needs, and refuses, naming the commit and its branch, when one cannot be fetched, for example a base the member never pushed. It does not block on an old branch whose commits nothing needs any more. No new report field is needed, since a failed push already fails the step withpush_failed. [feat] Add a Paseo driver adapter #1087's adapter meets the same contract from its own clone, so the fetch rule belongs in the driver contract.{ tool, id }(workspace.ts:33-36), and a settled pass keeps no driver. I'd record the reporting driver'skindandmachinewith each settled pass, leaving outworkspacebecause it is an absolute path, and put them on the lease'ssession. Criterion: a session started on machine A and continued after a resume on machine B is leased with A's machine, and B's driver starts it fresh.fix(workflow): refuse to publish absolute paths of this machine
feat(workflow): publish and resume workflow runs across machines (#1084…