From 46839d38c4f2f649aa241b6e884ada1df4966dbe Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 22 Oct 2026 08:04:13 +0200 Subject: [PATCH 01/10] feat(workflow): publish and resume workflow runs across machines A member can publish a run to teamai-reports and continue it on another machine, or hand it to a teammate, with its workspaces and, when asked, its agent sessions. The session commands now write teamai-reports under its lock with exit codes that say when nothing was written, and the local driver stops a command step when it alone gets SIGTERM. --- docs/dev/agent-sessions.md | 4 +- docs/dev/workflow-engine.md | 12 +- docs/guide/dashboard.md | 6 +- docs/guide/workflow-advanced.md | 104 +- docs/guide/workflow.md | 23 +- docs/guide/zh-CN/dashboard.md | 6 +- docs/guide/zh-CN/workflow-advanced.md | 104 +- docs/guide/zh-CN/workflow.md | 23 +- skill-data/core/references/commands.md | 19 +- src/__tests__/dry-run-load-path.test.ts | 12 + src/__tests__/e2e/cli-error-exit.test.ts | 4 +- .../e2e/workflow-conformance.test.ts | 145 +- .../e2e/workflow-publish-resume.test.ts | 397 +++ src/__tests__/e2e/workflow-signals.test.ts | 40 + src/__tests__/git-kind-reports.test.ts | 75 + src/__tests__/redact.test.ts | 118 +- src/__tests__/save-session.test.ts | 136 +- src/__tests__/session-push.test.ts | 324 +++ src/__tests__/workflow-commands.test.ts | 102 +- src/__tests__/workflow-conformance.test.ts | 1 + src/__tests__/workflow-engine.test.ts | 279 ++- .../workflow-git-conformance.test.ts | 149 ++ src/__tests__/workflow-handoff.test.ts | 2221 +++++++++++++++++ src/__tests__/workflow-local-driver.test.ts | 149 +- src/__tests__/workflow-records.test.ts | 102 +- src/__tests__/workflow-run-controls.test.ts | 5 + src/__tests__/workflow-settle-cost.test.ts | 26 +- src/__tests__/workflow-workspace.test.ts | 262 +- src/dry-run-guard.ts | 1 + src/index.ts | 16 +- src/save-session.ts | 96 +- src/session-flow/archive.ts | 249 ++ src/session-flow/session-cmd-shared.ts | 145 +- src/session-flow/session-cmd.ts | 252 +- src/session-flow/session-migrate-cmd.ts | 212 +- src/session-flow/sync.ts | 15 + src/utils/branch-worktree.ts | 23 +- src/utils/git.ts | 33 +- src/utils/redact.ts | 33 +- src/utils/reports-branch.ts | 52 +- src/workflow-commands.ts | 52 +- src/workflow/cache.ts | 101 + src/workflow/conformance.ts | 248 +- src/workflow/engine.ts | 219 +- src/workflow/handoff.ts | 763 ++++++ src/workflow/local-driver.ts | 241 +- src/workflow/records.ts | 110 +- src/workflow/run-controls.ts | 9 + src/workflow/scope.ts | 2 +- src/workflow/snapshot.ts | 353 +++ src/workflow/step-state.ts | 89 +- src/workflow/store.ts | 89 +- src/workflow/types.ts | 31 +- src/workflow/workspace.ts | 49 +- src/workflow/worktree.ts | 179 +- 55 files changed, 7850 insertions(+), 660 deletions(-) create mode 100644 src/__tests__/e2e/workflow-publish-resume.test.ts create mode 100644 src/__tests__/session-push.test.ts create mode 100644 src/__tests__/workflow-git-conformance.test.ts create mode 100644 src/__tests__/workflow-handoff.test.ts create mode 100644 src/session-flow/archive.ts create mode 100644 src/workflow/handoff.ts create mode 100644 src/workflow/snapshot.ts diff --git a/docs/dev/agent-sessions.md b/docs/dev/agent-sessions.md index 2b5c868e3..6fd868dde 100644 --- a/docs/dev/agent-sessions.md +++ b/docs/dev/agent-sessions.md @@ -13,8 +13,8 @@ Each rule below spans more than one file. The rule names the code that owns it, - **Read the session that was listed.** One id can live in several workspaces (copies an older TeamAI wrote), and Codex also accepts an id suffix. A caller that enumerates sessions passes the `filePath` it listed (`SessionReadRef` in `adapters/base.ts`; a conversation directory on CodeBuddy IDE), so exactly that session is read, and the session id comes from what was opened, never from the request. Reads outside the copy path go through `AgentSession.find`, which redacts unless the caller passes `scrub: false`; `session push` and `session migrate --push` pass their `--scrub` choice. - **List through `AgentSession.list`.** A caller that wants a member's sessions (`session migrate`, `session push`, a scorer, the dashboard) calls `AgentSession.list({ platforms?, cwd?, updatedSince? })` in `agent-session.ts`, never an adapter's `listConversations`. Each listed session is a ref that `find` reads as is. A session whose first lines do not parse is still listed, under a placeholder title, so `find` reports it `unreadable`, not `not_found`. A tool that cannot be listed is one failure value (`no_adapter`, `tool_unavailable`, or `list_failed`) next to the other tools' sessions. OpenCode's CLI lists one project at a time, and inside a git repo that project is the whole repo, so its adapter keeps only the sessions of the workspace asked for. Without `cwd`, OpenCode is `list_failed`, rather than returning the workspace the process runs in as if it were all of them. Its listing has no message count or size, so those fields are absent rather than 0. - **Stores follow the recorded tool root.** Claude Code, Codex and Cursor CLI keep their sessions under the root the member recorded with `teamai init` (`toolRoots`), resolved by `resolveMemberToolRoots` and `resolveToolRootDir`, never from `CLAUDE_CONFIG_DIR`, `CODEX_HOME` or `CURSOR_DATA_DIR` at run time. `AgentSession` and the `session` commands that build an adapter themselves (`resume`, `rollback`, `platforms`) resolve the roots with `recordedToolRoots` in `agent-session.ts` for the workspace at hand and pass them through the adapter registry (`adapters/index.ts`) to `getClaudeCodeProjectsDir`, `getCodexSessionsDir` and `getCursorProjectsDir` in `fs.ts`. A run that reads or copies many sessions passes one `toolRootsLookup()`, so each workspace is resolved once. With nothing recorded they stay `~/.claude/projects`, `~/.codex/sessions` and `~/.cursor/projects`. Cursor is the one tool with two roots, recorded as `cursor-data` and `cursor-config` rather than as `cursor`, because the `toolPaths` entry named `cursor` must not move: its user `mcp.json`, `rules/` and `commands/` stay in `~/.cursor` whatever the variables say. Its co-author write follows the config root through `cursorConfigDir` in `types.ts`, which also carries the `$XDG_CONFIG_HOME/cursor` fallback that `toolRoots` cannot record. The variants (`claude-internal`, `codex-internal`, ...) keep their fixed paths. -- **The driver picks how a session continues.** The driver that runs a workflow step decides: the same tool on the same machine resumes the session by id; a driver with its own way to move a session uses it; otherwise the local driver copies it with `AgentSession.copy`; failing those, the step starts a fresh session. The workflow engine never copies a session. A step's report carries the tool and that tool's native session id. -- **Session index lock.** Every write to a team repo archive (`saveSession`: dedup lookup, name choice, file writes, `_index.json` upsert) and every index rebuild (`rebuildIndex`, `rebuildAllIndexes`) runs inside `withSessionIndexLock(repoRoot, fn)`. The async command callers take it; `SyncManager` stays synchronous and never takes it, so it cannot re-enter. The lock file is `~/.teamai/locks/sessions-.lock`, keyed by the real path of the team repo's `sessions/`. It stays out of the team repo because there the archive commit could stage it and `pull --all`'s directory renames could move it while held. It covers one machine only. Two machines that both wrote `_index.json` diverge there: `SyncManager.gitPull` (also the retry inside `gitPush`) merges and returns the conflicted indexes, and the command caller rebuilds them and commits the merge (`commitRebuiltIndexes`) under the lock, with the network pull and push outside it. When the lock stays busy the merge is left in progress. The next `teamai session pull` finishes it first, then runs its own pull. +- **The driver picks how a session continues.** The driver that runs a workflow step decides: the same tool on the same machine resumes the session by id; a driver with its own way to move a session uses it; otherwise the local driver copies it with `AgentSession.copy`. Failing those, it restores a copy from the team session archive (`openSessionArchive` in `archive.ts`). Failing that too, the step starts a fresh session, with a warning that says why each way failed. `workflow publish --sessions` is what puts a run's sessions in the archive. `sessionArchiveRoot` in `session-cmd-shared.ts` is the one rule for where the archive lives, for the session commands and the driver alike. On a git-kind team, the first restore in a driver run fetches origin's default branch into the team clone's `refs/remotes/origin/` and checks that commit out in a temporary `--shared` clone. Every restore in that run, hit or miss, reads that clone, so a session archived again under the same id restores as origin last archived it, and the member's branch, HEAD, and files never move. `close()` removes the clone after the run's prompt has exited. A failed fetch reads the team clone as it is, and the restore says it may be older than origin. In self mode the claim has just synced the reports checkout, so the driver does not fetch. The workflow engine never copies a session, and a run snapshot carries only the session's `{ tool, id }`. A step's report carries the tool and that tool's native session id. A step that continued the lease's session also echoes it as `resumed`, and the engine records that the reported session continues it. The next step that resumes the conversation then gets that copy, with every turn, on any machine that has it or finds it in the archive; failing that, it tries the earlier sessions the lease lists, newest first, and gets fewer turns. +- **Session index lock.** Every write to a team repo archive (`saveSession`: dedup lookup, name choice, file writes, `_index.json` upsert) and every index rebuild (`rebuildIndex`, `rebuildAllIndexes`) runs inside `withSessionIndexLock(repoRoot, fn)`. The async command callers take it; `SyncManager` stays synchronous and never takes it, so it cannot re-enter. The lock file is `~/.teamai/locks/sessions-.lock`, keyed by the real path of the team repo's `sessions/`. It stays out of the team repo because there the archive commit could stage it and `pull --all`'s directory renames could move it while held. It covers one machine only. Two machines that both wrote `_index.json` diverge there: `SyncManager.gitPull` (also the retry inside `gitPush`) merges and returns the conflicted indexes, and the command caller rebuilds them and commits the merge (`commitRebuiltIndexes`) under the lock, with the network pull and push outside it. When the lock stays busy the merge is left in progress. The next `teamai session pull` finishes it first, then runs its own pull. Every session writer to the teamai-reports checkout (`session save --push`, `session push`, `session migrate --push`, the merge and index rebuild of `session pull`, and `workflow publish --sessions`) goes through `updateReports`, under the reports lock, and the session commands that only read it (`list`, `search`, `resume`) open it through `readableReportsWorktree`. ## Tests diff --git a/docs/dev/workflow-engine.md b/docs/dev/workflow-engine.md index 10301b488..d21cc5835 100644 --- a/docs/dev/workflow-engine.md +++ b/docs/dev/workflow-engine.md @@ -6,12 +6,14 @@ Each rule below spans more than one file. The rule names the code that owns it, ## Rules that span files -- **A run ends only in `settleRun`.** The shared `finishRun` path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. Respond, cancel and retry in `run-controls.ts`, and next in `engine.ts`, share that path. Whole-run cancel settles directly as `cancelled`, without reconciling another round. To run code when a run ends, hook into `finishRun`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `writeRun` then fails, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). +- **A run ends only in `settleRun`.** The shared `finishRun` path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. Respond, cancel and retry in `run-controls.ts`, and next in `engine.ts`, share that path. Whole-run cancel settles directly as `cancelled`, without reconciling another round. To run code when a run ends, hook into `finishRun`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `updateRun` then fails to save the run file, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). - **Reconcile the latest report before applying another or handing out a lease.** `reconcileRun`, shared by `leaseNext` and `reportAttempt`, settles every driverless transition to a fixpoint at the run's current revision, through `settleScope` and `settleItems`, including workflow starts, fan-out expansion, render failures and bound exhaustion. `reportAttempt` advances the revision only after it, so only the latest revision can be unreconciled. A reached end applies before a loop decision. Only after reconciliation does `leaseNext` hand back an omitted open lease or issue the first ready leaf in declaration order that this driver can take. A running workflow occupies its parent's position in that order. Settling returns every ready leaf in that order (`ready`), and `leaseNext` walks it. A ready item does not count against `max_parallel` until it is leased, and one `next` leases at most one step, so a held-back item leaves its room to the next one without exceeding the cap. `canTakeLease: false` still reconciles driverless transitions. Every changed run is saved, including a `wait`. - **A report is checked against the revision of its own lease.** `reportAttempt` and `retryStep` increment `revision`; skips, `if`, `end`, and the creation of child steps inside `next` never do. A lease keeps the run revision it was handed out at, and `reportAttempt` compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: `stale_revision` when its revision is not the run's, otherwise `wrong_attempt`. - **An end or break stops what is still open under its workflow.** `stopWhere` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending and waiting ones. The walk takes a path predicate and failure. End and break construct their own end_reached failure; manual cancel and retry supply theirs. Step descendants use the same `isUnder` predicate as `archiveUnder` in `loop.ts`. Cancel records the ids this walk changed before reconciliation can settle or skip other steps. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. -- **The local driver overlaps steps that do not share a checkout.** `runLocalDriver` runs up to `workflows.max_parallel` steps (local config, default 1) and sends the ones it has started as `running`. A command with no workspace, and a prompt in the current checkout, share that checkout, so only one of them runs there at a time. A prompt earlier in the queue for that checkout blocks a later command there. The driver starts the prompt when the checkout is free and fewer than `max_parallel` steps are in flight; commands in other worktrees keep running. A prompt with its own worktree does not take the checkout. A headless prompt takes one of the slots, and the driver keeps leasing steps for the others while it runs. An interactive prompt uses the terminal, starts only when no command is running, and no step is leased while it runs. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any captured descendant or process in a detached group remains, even after its command exited. Attached commands snapshot their descendant tree and start times before stopping, then probe and send SIGKILL only to descendants whose start times still match. If the identity snapshot fails, only the shell is stopped; detached commands use their process group. It does not send a stopping command as `running` again, so `next` does not cancel it twice. Dying commands still take a slot, and a new attempt of their step waits until they are gone. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. -- **A workspace is a git contract on the lease, and the driver is the only code that touches git.** `planWorkspace` in `workspace.ts` chooses the mode. An explicit `workspace` wins. A command that omits it is read-only, even inside a copy; when it exits 0 and changed the checkout, the driver fails it `read_only_dirty`. That includes a commit that leaves the worktree clean, a switch to another branch or a detach at the same commit, a change that only stages an existing edit, another staged hunk that leaves the status mixed, a permission change on a file that was already dirty, an edit to a quoted non-ASCII path, a symlink whose target changes while the file it names stays the same, and an edit to the destination of a staged rename, and a change inside a submodule that was already dirty. An interactive prompt that sets `workspace: read-only` is checked the same way. An interactive prompt that does not may change the checkout, and those edits stay. Conflict paths, including agent files, are read without Git's quoting, so a non-ASCII name is still checked for leftover markers. A prompt that omits it continues the nearest enclosing `new` or `continue` workflow, and a `for_each` ancestor makes that `new`, so items do not share a worktree. `leaseNext` puts the contract on the lease for `new` and `continue`, and for a read-only step inside a copy (`mode: 'read-only'`, `merge: []`). That reader's branch is the one its declared `needs` wrote or read (`changes` or `read`, through items and children, but not `after` items or inherited needs), or the enclosing copy's when they touched none; needs on several branches fail `workspace_base`. It needs `continue` in `can.workspace` and the copy's owner, as `continue` does. A read-only step outside any copy gets no contract and runs in the active checkout. The local driver stages a reader without merges, takes its snapshot over the base overlay, and afterwards restores the overlay; when the step changed the copy or left its branch, it keeps the state on `refs/teamai/kept/` (`git add -A`, then `write-tree` and `commit-tree` on HEAD; not the stash list, which every worktree shares), and pushes each submodule commit that tree records and the read commit does not into the checkout's git dir as `refs/teamai/kept-modules//` (at any depth, without the user's push hooks or `push.recurseSubmodules`), since a gitlink does not keep its commit reachable once the copy is removed, then runs `git checkout -f `, `git reset --hard` to the commit it read, `git clean -fd`, and `git submodule update --checkout --recursive` on the submodules it changed, so a later writer never commits a reader's edits. It resets nothing when a repository among the changed paths has uncommitted or untracked files, or a nested submodule commit its parent does not record, which the keep cannot hold. The copy counts as put back only when HEAD is `refs/heads/` at that commit and `git status --porcelain --untracked-files=all --ignore-submodules=none` prints the same lines it printed when the reader was staged; ignored files are left in place. Otherwise the reader fails `workspace_base`. `stageLease` writes a `teamai-unrestored` marker, with that status, in the worktree's git dir before the reader's overlay, and the driver removes it once the copy is put back or left untouched, so a driver killed mid-reader still leaves it. `stageLease` refuses later leases in a marked copy until the copy is back, when it removes the marker. A status line does not show content, so a marker whose status was not empty stays until the person who puts the copy back deletes it. `ensureWorktree` also refuses an existing worktree whose HEAD is not on its branch or a directory that is not a worktree, and prunes and recreates one whose directory was deleted. The reader reports `read` (`branch`, `base`, `commit`) and never `changes`. `report` returns `invalid_report` for `read` on any other lease, and for `changes` or a success without `read` on a reader's lease. `run.copies[].commit` and fan-out `branches` follow `changes` only. `cleanup` comes from the lease's contract, so a reader that was the copy's last user removes it too. `leaseNext` holds a step back for this driver, and tries the next ready leaf, when the driver's `can.workspace` lacks that mode, or when a `continue` copy belongs to another driver (`run.copies`). It answers `wait`, with every held step's reason joined by `; `, only when every ready leaf is held back. Driver capabilities stay out of `reconcileRun`, which `reportAttempt` and the run controls also call. It fails `workspace_base` when `continue` has no enclosing `new`, or when the run has no base. `workflow start` in a git checkout records the base (commit, and the branch unless HEAD is detached); otherwise the first `next` that sends `base` sets it, and a later `base` never changes it. The local driver fails a lease whose `workspace` it cannot read with `workspace_base` instead of running it in the active checkout. The branch is `teamai///p` (`branchName`); a `[n]` in the path becomes `-n`. The first branch in `needs` order is where the step starts, and the rest are merged in that order. An item's `after` branches come first. The local driver (`local-driver.ts`, `worktree.ts`) adds the worktree under `/.teamai/workflow-worktrees/`, points `core.hooksPath` at an empty directory, and replaces `AGENT_PATHS` (`AGENTS.md`, `CLAUDE.md`, `.agents`, `.githooks`, and each default `toolPaths` entry's project directory, project MCP file and Copilot `.github` paths) with the run's base commit. An agent path under a symlinked parent fails `base_config` before anything is copied or removed, since both would follow the link. It then gives the worktree the inline `pull` a new worktree gets (`prepareCheckout`), which delivers the team's skills, rules, agents and MCP and records that checkout's team revision. The overlay is taken after that delivery, so those files are not the step's commit. A path the step edits is committed. A step that left HEAD off the leased branch fails `workspace_base` before anything is committed, reported as `changes`, or pushed, and the worktree stays as it left it. A path it leaves untouched is put back from the branch. A merge conflict stops the merge: a command fails `merge_conflict`, and a prompt is told the branches that remain. Snapshots and commits read submodules with `--ignore-submodules=none`, and after `git add -A`, which skips a submodule whose `ignore` is `all`, each submodule is staged by name, so `submodule..ignore` and `diff.ignoreSubmodules` hide no change from a reader's check, its keep, or a writer's commit. The driver commits as `TeamAI ` and reports `changes` (`branch`, `base`, `from`, `to`, `commits`). It pushes when `origin` exists and this step made commits, and also when a later attempt adds no commit but this branch has commits `origin` does not have, including a first push that never landed and an origin with no tracking refs yet. A failed push is `push_failed`, and the worktree stays. `report` sets `cleanup` only when the step succeeded and no step that can still run uses that branch, judged on the run as the next `next` settles it: steps a loop runs again count, and a workflow or `for_each` step that has not started counts by the branch `plannedBranch` gives, the one around it unless it opens its own; only then does the driver run `git worktree remove --force`. It does not delete the branch, and it never resets, cleans, or switches the active checkout. A command that starts its own sandbox, such as `docker`, stays read-only. What it writes does not feed `new`, `branches`, or a later step. `session: continue` and `session.from` are step reads. The engine puts `{ tool, id }` on the lease and never copies a session. The driver resumes Claude and Codex by that id, copies another tool through `AgentSession.copy`, and starts fresh when resume is not available. OpenCode has no resume flag, so it starts fresh. +- **The local driver overlaps steps that do not share a checkout.** `runLocalDriver` runs up to `workflows.max_parallel` steps (local config, default 1) and sends the ones it has started as `running`. A command with no workspace, and a prompt in the current checkout, share that checkout, so only one of them runs there at a time. A prompt earlier in the queue for that checkout blocks a later command there. The driver starts the prompt when the checkout is free and fewer than `max_parallel` steps are in flight; commands in other worktrees keep running. A prompt with its own worktree does not take the checkout. A headless prompt takes one of the slots, and the driver keeps leasing steps for the others while it runs. An interactive prompt uses the terminal, starts only when no command is running, and no step is leased while it runs. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any captured descendant or process in a detached group remains, even after its command exited. Attached commands snapshot their descendant tree and start times before stopping, then probe and send SIGKILL only to descendants whose start times still match. If the identity snapshot fails, only the shell is stopped; detached commands use their process group. It does not send a stopping command as `running` again, so `next` does not cancel it twice. Dying commands still take a slot, and a new attempt of their step waits until they are gone. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. A signal `env exec` would pass on (SIGTERM or SIGHUP, and SIGINT or SIGQUIT when the terminal does not reach the command) makes the driver stop the command, wait for it, and exit with that signal. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. +- **A workspace is a git contract on the lease, and the driver is the only code that touches git.** `planWorkspace` in `workspace.ts` chooses the mode. An explicit `workspace` wins. A command that omits it is read-only, even inside a copy; when it exits 0 and changed the checkout, the driver fails it `read_only_dirty`. That includes a commit that leaves the worktree clean, a switch to another branch or a detach at the same commit, a change that only stages an existing edit, another staged hunk that leaves the status mixed, a permission change on a file that was already dirty, an edit to a quoted non-ASCII path, a symlink whose target changes while the file it names stays the same, and an edit to the destination of a staged rename, and a change inside a submodule that was already dirty. An interactive prompt that sets `workspace: read-only` is checked the same way. An interactive prompt that does not may change the checkout, and those edits stay. Conflict paths, including agent files, are read without Git's quoting, so a non-ASCII name is still checked for leftover markers. A prompt that omits it continues the nearest enclosing `new` or `continue` workflow, and a `for_each` ancestor makes that `new`, so items do not share a worktree. `leaseNext` puts the contract on the lease for `new` and `continue`, and for a read-only step inside a copy (`mode: 'read-only'`, `merge: []`). That reader's branch is the one its declared `needs` wrote or read (`changes` or `read`, through items and children, but not `after` items or inherited needs), or the enclosing copy's when they touched none; needs on several branches fail `workspace_base`. It needs `continue` in `can.workspace` and the copy's owner, as `continue` does. A read-only step outside any copy gets no contract and runs in the active checkout. The local driver stages a reader without merges, takes its snapshot over the base overlay, and afterwards restores the overlay; when the step changed the copy or left its branch, it keeps the state on `refs/teamai/kept/` (`git add -A`, then `write-tree` and `commit-tree` on HEAD; not the stash list, which every worktree shares), and pushes each submodule commit that tree records and the read commit does not into the checkout's git dir as `refs/teamai/kept-modules//` (at any depth, without the user's push hooks or `push.recurseSubmodules`), since a gitlink does not keep its commit reachable once the copy is removed, then runs `git checkout -f `, `git reset --hard` to the commit it read, `git clean -fd`, and `git submodule update --checkout --recursive` on the submodules it changed, so a later writer never commits a reader's edits. It resets nothing when a repository among the changed paths has uncommitted or untracked files, or a nested submodule commit its parent does not record, which the keep cannot hold. The copy counts as put back only when HEAD is `refs/heads/` at that commit and `git status --porcelain --untracked-files=all --ignore-submodules=none` prints the same lines it printed when the reader was staged; ignored files are left in place. Otherwise the reader fails `workspace_base`. `stageLease` writes a `teamai-unrestored` marker, with that status, in the worktree's git dir before the reader's overlay, and the driver removes it once the copy is put back or left untouched, so a driver killed mid-reader still leaves it. `stageLease` refuses later leases in a marked copy until the copy is back, when it removes the marker. A status line does not show content, so a marker whose status was not empty stays until the person who puts the copy back deletes it. `ensureWorktree` also refuses an existing worktree whose HEAD is not on its branch or a directory that is not a worktree, and prunes and recreates one whose directory was deleted. The reader reports `read` (`branch`, `base`, `commit`) and never `changes`. `report` returns `invalid_report` for `read` on any other lease, and for `changes` or a success without `read` on a reader's lease. `run.copies[].commit` and fan-out `branches` follow `changes` only. `cleanup` comes from the lease's contract, so a reader that was the copy's last user removes it too. `leaseNext` holds a step back for this driver, and tries the next ready leaf, when the driver's `can.workspace` lacks that mode, or when a `continue` copy belongs to another driver (`run.copies`). It answers `wait`, with every held step's reason joined by `; `, only when every ready leaf is held back. Driver capabilities stay out of `reconcileRun`, which `reportAttempt` and the run controls also call. It fails `workspace_base` when `continue` has no enclosing `new`, or when the run has no base. `workflow start` in a git checkout records the base (commit, and the branch unless HEAD is detached); otherwise the first `next` that sends `base` sets it, and a later `base` never changes it. The local driver fails a lease whose `workspace` it cannot read with `workspace_base` instead of running it in the active checkout. The branch is `teamai///p` (`branchName`); a `[n]` in the path becomes `-n`. The first branch in `needs` order is where the step starts, and the rest are merged in that order. An item's `after` branches come first. The local driver (`local-driver.ts`, `worktree.ts`) adds the worktree under `/.teamai/workflow-worktrees/`, points `core.hooksPath` at an empty directory, and replaces `AGENT_PATHS` (`AGENTS.md`, `CLAUDE.md`, `.agents`, `.githooks`, and each default `toolPaths` entry's project directory, project MCP file and Copilot `.github` paths) with the run's base commit. An agent path under a symlinked parent fails `base_config` before anything is copied or removed, since both would follow the link. It then gives the worktree the inline `pull` a new worktree gets (`prepareCheckout`), which delivers the team's skills, rules, agents and MCP and records that checkout's team revision. The overlay is taken after that delivery, so those files are not the step's commit. A path the step edits is committed. A step that left HEAD off the leased branch fails `workspace_base` before anything is committed, reported as `changes`, or pushed, and the worktree stays as it left it. A path it leaves untouched is put back from the branch. A merge conflict stops the merge: a command fails `merge_conflict`, and a prompt is told the branches that remain. Snapshots and commits read submodules with `--ignore-submodules=none`, and after `git add -A`, which skips a submodule whose `ignore` is `all`, each submodule is staged by name, so `submodule..ignore` and `diff.ignoreSubmodules` hide no change from a reader's check, its keep, or a writer's commit. The driver commits as `TeamAI ` and reports `changes` (`branch`, `base`, `from`, `to`, `commits`). It pushes when `origin` exists and this step made commits, whether the step succeeded or failed, and also when a later attempt adds no commit but this branch has commits `origin` does not have, including a first push that never landed and an origin with no tracking refs yet. A failed push fails a step that succeeded with `push_failed`, and the worktree stays. `report` sets `cleanup` only when the step succeeded and no step that can still run uses that branch, judged on the run as the next `next` settles it: steps a loop runs again count, and a workflow or `for_each` step that has not started counts by the branch `plannedBranch` gives, the one around it unless it opens its own; only then does the driver run `git worktree remove --force`. It does not delete the branch, and it never resets, cleans, or switches the active checkout. A command that starts its own sandbox, such as `docker`, stays read-only. What it writes does not feed `new`, `branches`, or a later step. `session: continue` and `session.from` are step reads. The engine puts `{ tool, id }` on the lease, with `earlier`, the sessions that one continued, newest first, and never copies a session. The driver resumes Claude and Codex by that id, copies another tool through `AgentSession.copy`, restores a copy from the team session archive when neither works, and tries each `earlier` session the same way before it starts fresh. OpenCode has no resume flag, so it starts fresh. A driver that continued the lease's session or an earlier one, by that id or in a copy, echoes the one it continued as `resumed` in its report. One that started a new session leaves `resumed` out. `reportAttempt` answers `invalid_report` for a `resumed` on a step that resumes no session, and for one that no pass of the run reported and no link continued (`knowsSession`). When the reported `tool` and `session` differ from `resumed`, `reportAttempt` appends `{ from: resumed, to }` to `run.sessionContinuations`. `sessionOf` follows those links from the session a step would resume, taking the latest link from each session, so a later step resumes the copy that holds the last turns, whichever machine or tool made it. The snapshot publishes the links, so `resumableSessions` on this machine and the claim on the next one both see them. A step that resumed by the same id adds no link, so a run on one machine with one tool leases the same sessions as without links. +- **A driver pushes the commits it reports, and fetches the commits it is leased.** Every driver keeps these two git rules, so a run another driver or another machine continues finds each commit on `origin`. Whenever `origin` exists, a driver pushes the commits it reports in `changes` before it reports them, failed steps included; the local driver's `finishWorkspace` pushes for every outcome. A push that fails still reports the commits. A step that succeeded fails `push_failed`. A step that failed keeps its own failure, and the push sentence is added to its message: a retry must fix the step first, and a `push_failed` in its place would hide that cause from the person and from the run record. Before it starts from, merges, or applies a lease commit (`start`, each `merge[]` entry, `configCommit`), a driver checks it with `git cat-file -e ^{commit}`. For a missing one it fetches `+refs/heads/teamai//*:refs/remotes/origin/teamai//*`, and checks again, since a refspec fetch that matches no branch exits 0. A fetch by commit id is not used: a server can refuse it, and a fetched commit no ref holds can be collected later. A commit still missing fails `workspace_base` naming the commit and its branch, before anything merges or runs, so it is never a file-less `merge_conflict` or a prompt told to resolve one. The local driver does this in `stageLease` through `fetchMissingCommits`. `workflowGitConformanceScenarios` in `conformance.ts` checks both rules against a bare origin, and `workflow-git-conformance.test.ts` runs them against the local driver. +- **A run changes machines only through `handoff.ts`.** `publishRun` and the claim behind `claimRun` and `resumeRun` are the only writers of `workflows/runs//state.json` on `teamai-reports`, and they write it inside `updateReports`. A publish writes only while origin shows this install (`installOwner`, from `/workflows/owner.json`) and the state revision this machine last saw (`acknowledged`). An unchanged body writes nothing. A publish checks the business origin without a lock, then answers `state_moved` when, under the locks, the run's base, a copy's commit, or the sessions it archived differ from what it checked (`checkedOutsideLocks`). Anything else that moved meanwhile is published as the locks find it, so a run a driver is still advancing can be published. `resumeRun` continues a run locally when origin has no snapshot and this machine never saw one acknowledged (`stateRevision` 0), since no other install can have claimed it. A claim reads origin under the reports lock and releases it, decides outside the lock (`definitionsOf` can fetch and sync a shared clone, and `checkOrigin` runs `ls-remote`), then writes only while origin still holds the `write` id it read (`replaceSnapshot`). The compare key is the `write` id, not the revision, because two claims of one revision both write `stateRevision + 1`. A moved or missing snapshot answers `state_moved`. Every `next` that answers `done`, and every run control, calls `publishRunEnd`, which runs `publishRun` when the run was published and has ended, so origin never hands out a run its owner finished. A publish drains the run's records first, so none lands after its end. The drain (`publishPendingRecords`) drops the records of a run whose snapshot on origin names another install, so an install that ends its copy of a run it lost never replaces the owner's record. Once the end is on origin, a repeat writes nothing, so an end that a process saved but stopped before publishing reaches origin on the next request. A claim changes steps only through `takeOver` in `step-state.ts`: it releases a lease, and repends a lease in a `new` step so its next pass gets a new branch. `handoff.ts` never imports the engine. The engine calls `claimRun` for the `claim` field of `next`, and `workflow publish --sessions` passes the engine's `resumableSessions` in. What the snapshot carries is a rule table per type in `snapshot.ts` (`publish`, `drop`, or `rewrite`), so a new run or step field does not compile until it has a rule, and `parseSnapshot` refuses a field it would drop. - **The end reached first wins.** A reported step's `settledAt` is the revision its report made. Every terminal transition, including a skip or cancellation, carries `settledAt`. A step the engine settles takes the run's current revision, which is that of the report that caused it, because every earlier revision was reconciled before that report was accepted. An open lease it fails takes the revision it was leased at, and a cancellation the winning end's revision. `reportAttempt` accepts a report only after `reconcileRun` has applied every end reached before it, so an end a later report reaches never competes with an earlier one. Every end `settleScope` has yet to apply was reached at the run's current revision, and it applies the one declared first. The winner therefore does not depend on when `next` runs between reports. - **A break is a nameless succeeded end.** A succeeded step whose `break` holds competes with the steps that reached an end by the same rule. It ends its workflow, or that workflow's loop, as success. Its value cannot change after its step succeeds, because pull rejects `break` inside a step loop's range, where a previous-pass read could change. - **A workflow loop is decided when a round has nothing left.** Once every step of a looping workflow has settled and no end or break was reached, `decideRound` in `loop.ts` evaluates the workflow's `loop.while`. When it is false, the workflow ends with the round's outcome and outputs. Otherwise every step of the workflow repends for the next round, unless the workflow has run `max_iterations` rounds: then it fails `loop_exhausted` (`WorkflowResult.failure`, and `run.failure`, shown in `done`, at the top). A failed step alone does not stop the loop. The round is kept on the step that runs the workflow (`Scope.runner`), so `repend` resets it with that step, or on the run at the top. Pull rejects a step loop in a looping workflow. @@ -35,8 +37,8 @@ Each rule below spans more than one file. The rule names the code that owns it, - **Retry validates every container before resetting any.** `retryContainers` in `run-controls.ts` climbs from the target through scopes and fan-out items until a running runner or the top. Each dependency cone includes needs, reads and loop waits, or item after dependencies. An independent standing end or break rejects the retry. The apply pass archives replaced passes and descendants, reopens settled runners with `start` to keep inputs and round, then bumps revision and reconciles. Only the targeted loop owner's iteration resets; the top-level round does not. An answered refused approve or succeeded choose is retryable too, so a changed answer resets its dependent branches. A HITL request in the cone that is waiting, or answered and not yet run, starts a new pass too, so its answer is asked again in the new context and an answer to the old pass is stale. Unanswered skips in every affected container reset so skips caused by cancellation can run again. Retry's prelude reconciles without finishing a provisional outcome. Reopening clears the run's end, failure and endRecordQueued flag and removes its queued end file through records.ts. These three pieces stay consistent: run state, the pending end record, and published stats. The next end upserts the same record path; stats replace the origin end's outcome and duration contribution with the new end's, without adding another run. Replaying the same endedAt contributes nothing. - **The local driver keeps tools and questions interruptible.** A single question joins command and prompt exits in the driver's race. An interactive tool owns the terminal alone: no question opens while it runs, and starting one aborts and closes an open question. A poll closes the question when its own step is leased or a different step or pass waits; an unrelated lease leaves it open. It polls next every second while asking, every two seconds while tools run, and aborts a replaced HITL request or cancelled prompt. `process-stop.ts` shares process stopping between commands and adapters. Headless tools detach and stop with SIGTERM; interactive tools stay attached and stop with their captured descendant tree. Foreground interactive signals reuse env-exec's terminal detection and signal policy: when the terminal reaches the child, teamai ignores SIGINT and SIGQUIT, while SIGTERM and SIGHUP are forwarded and stop the driver. Interactive adapters inherit stdio and accept an optional result file. Repository edits are kept for interactive steps; only headless prompts run the read-only check. A nonzero interactive exit without a result reports its exit status. `leaseNext` records the identity of the latest driver that called next in `RunState.driver`, without capabilities. Resume uses this machine's open local lease workspace, then run.driver.workspace for a local driver on this machine, then cwd. Controls pass run.driver to finishRun so end records retain the driver kind. -- **One local driver owns a run on this machine.** `runLocalDriver` takes `driverLockPath` once with `acquireLock` and releases it in finally. A live holder returns `driver_running`, including its pid; the existing lock machinery reclaims dead owners. The driver lock spans tool execution and questions, while `withRun` holds a different run lock only during one engine operation. Neither lock's scope is extended by the other. Every HITL decision without a HITL driver and interactive hitl_required failure logs once when the engine changes the step. HITL replies carry one literal `respond` command template, which drivers display unchanged. +- **One local driver owns a run on this machine.** `withDriverLock` in `local-driver.ts` takes `driverLockPath` once with `acquireLock` and releases it in finally. `runLocalDriver` drives inside it, and `workflow resume` also claims inside it, so a claim never lands under a driver already running the run. A live holder returns `driver_running`, including its pid, before anything is claimed or driven; the existing lock machinery reclaims dead owners. The driver lock spans tool execution and questions, while `withRun` holds a different run lock only during one engine operation. Neither lock's scope is extended by the other. Every HITL decision without a HITL driver and interactive hitl_required failure logs once when the engine changes the step. HITL replies carry one literal `respond` command template, which drivers display unchanged. ## Tests -Test parsing through `parseWorkflow` and `syncWorkflowDefinitions`. Test the engine through `startRun`, `leaseNext`, and `reportAttempt`. Every driver must pass the conformance suite in `src/workflow/conformance.ts`. If a driver can see a new behavior, add a scenario there. +Test parsing through `parseWorkflow` and `syncWorkflowDefinitions`. Test the engine through `startRun`, `leaseNext`, and `reportAttempt`. Every driver must pass the protocol scenarios in `src/workflow/conformance.ts`, and a driver that takes workspace leases the git scenarios there too. If a driver can see a new behavior, add a scenario there. diff --git a/docs/guide/dashboard.md b/docs/guide/dashboard.md index 0f955a44d..90a9a01f5 100644 --- a/docs/guide/dashboard.md +++ b/docs/guide/dashboard.md @@ -73,4 +73,8 @@ A migration never overwrites. If the target workspace already has a copy of the For a script, `migrate -s -t --json` prints one line on stdout, the same `{ "data": ... }` or `{ "error": { "code", "message" } }` shape as `teamai workflow`; progress goes to stderr. `data` has the target `sessionId`, `cwd`, `reused`, and the `fidelity` report. `error.code` names the failure, such as `no_adapter` (unknown tool), `tool_unavailable` (the source or target tool is not installed, or its CLI cannot run), `not_found`, or `unreadable`, and the exit code is 1. `--json` never prompts: a missing id, `-s`, or `-t` fails with `prompt_refused`, and `--push` needs `-y` or `--scrub`. -`--dry-run` previews. `--scrub` redacts before an archive; without it, an interactive run asks and a script needs `-y`. `--all` confirms above 10 sessions (push: 5). Archives go to `sessions/repos///` under the session's own git remote, or `_unattributed`. +`--dry-run` previews. `--scrub` redacts before an archive; without it, an interactive run asks and a script needs `-y`. `--all` confirms above 10 sessions (push: 5). Archives go to `sessions/repos///` under the session's own git remote, or `_unattributed`. A session that a workflow step ran in its own worktree goes under the repository of the checkout that the worktree came from. + +Session commands use the team repo clone of this project's install, else of the user install, so you need `--repo-root` only to point at another clone. With no install, they print `teamai is not set up for this directory.`, name `teamai init` and `--repo-root`, and exit 1. + +When this repo is the team repo, archives live on the `teamai-reports` branch. While another teamai command writes that branch, `session push`, `session migrate --push`, and `session pull` print `Error: Another teamai command is writing teamai-reports` and exit 1 without changing the branch. `migrate --push` keeps the copy it made. Run the command again once the other one has finished. diff --git a/docs/guide/workflow-advanced.md b/docs/guide/workflow-advanced.md index cfb100501..ba6697de2 100644 --- a/docs/guide/workflow-advanced.md +++ b/docs/guide/workflow-advanced.md @@ -70,7 +70,7 @@ With `max_parallel` above 1: - Each command runs without a terminal, so it cannot read input from you. - On Ctrl-C, or on a SIGTERM, SIGQUIT, or SIGHUP, teamai stops every command, waits for them to exit, and exits with that signal. -With 1, the driver runs one step at a time, and a command runs in your terminal, where Ctrl-C reaches it. +With 1, the driver runs one step at a time, and a command runs in your terminal, where Ctrl-C reaches it. A SIGTERM or SIGHUP sent to teamai alone stops the command, waits for it, and exits with that signal. When an end cancels a running command, the driver sends SIGTERM to it and the processes it started, then SIGKILL 5 seconds later if any still run. A stopping command keeps its slot until its processes exit. @@ -648,4 +648,104 @@ History stays in `passes` and the archive, so `length(steps.x.passes)` counts fa A single-step `cancel` stops pending, waiting, leased, or running work. Its open descendants stop too. Cancelled steps count as stopped, so dependents skip unless an `if` reads their outcome for recovery. The run normally fails. A whole-run cancel ends immediately with run outcome `cancelled`, without starting another loop round. Cancel does not bump the revision. A late report returns `lease_cancelled`; `next.cancelled` tells drivers which tools to stop. -`resume` drives the run locally after a retry, response, Ctrl-C, or closed terminal. It uses this machine's open local lease workspace first, then the workspace of the latest driver that called `next` when that driver was local on this machine, then the current directory. Resuming after a retry from a subdirectory therefore keeps the original local workspace. Only one local driver can drive a run at a time. A second resume returns `driver_running` with the owning process id; stop that driver or wait. Locks from dead processes are reclaimed automatically. A settled run returns `done`. Failed and cancelled local runs exit 1; successful control commands exit 0 even when their answer describes a failed or cancelled run. +`resume` drives the run locally after a retry, response, Ctrl-C, or closed terminal. `workflow start --driver local` prints the run id on stderr when the run starts, with the `resume` command that continues it. It uses this machine's open local lease workspace first, then the workspace of the latest driver that called `next` when that driver was local on this machine, then the current directory. Resuming after a retry from a subdirectory therefore keeps the original local workspace. Only one local driver can drive a run at a time. A second resume returns `driver_running` with the owning process id; stop that driver or wait. Locks from dead processes are reclaimed automatically. A settled run returns `done`. Failed and cancelled local runs exit 1; successful control commands exit 0 even when their answer describes a failed or cancelled run. + +## Publish and claim + +```sh +teamai workflow publish [--sessions] +teamai workflow resume [--claim] +``` + +[Continue a run on another machine](workflow.md#continue-a-run-on-another-machine) shows the common case. This section covers the rules. + +The first `publish` makes this install the run's owner. The owner is an install id, kept in `workflows/owner.json` under the TeamAI data directory, plus the OS user name. Two installs on one machine show the same user name, and the id tells them apart. `publish` prints one envelope: + +```json +{"data":{"run":"","publication":"published","stateRevision":1,"inFlight":[{"step":"fix","attempt":"","driverKind":"local"}]}} +``` + +`inFlight` lists the leases still open. A claim releases them, so those steps run again. Publishing a run that has not changed since its last publish answers `"publication":"unchanged"` with the same `stateRevision`, and writes nothing. Each publish that changes the run adds the whole state to the history of `teamai-reports`, so publish when you hand a run over, not after every step. Only the owner can publish the run. On another install, `publish` answers `not_owner` and changes nothing. + +When a published run ends, through `next` or through `workflow respond`, `retry`, or `cancel`, teamai publishes how it ended, as `workflow publish` does. When that is refused, the `done` envelope of `next`, or the `data` of the command, carries `notPublished`, a refusal from the table below. `not_owner` means another install claimed the run, so its state and its run record stay as that install has them. For any other refusal, take its next step. The next `next` or `workflow resume` of the run also publishes the end again. + +`resume` decides from what this machine has: + +- A run this machine has and never published continues locally. `resume` does not read `teamai-reports`. +- A run whose first publish never reached `teamai-reports` continues locally too. `resume` reads `teamai-reports` to check, and answers `reports_unreachable` when it cannot. +- A run this machine does not have is claimed from `teamai-reports`, then driven. +- A published run continues only while this install owns it. Otherwise `resume` answers `not_owner` with `owner` (`id`, `user`) and `ownedAt`. +- A claim of a run that ended runs none of its steps again. The claiming install owns the run, so `workflow retry` there runs again only the step it names and the steps that depend on it. +- `--claim` takes a published run over from its owner. It replaces this machine's copy of the run. While a local driver of the run is running on this machine, it answers `driver_running` and claims nothing. + +Run `resume` in a checkout of the run's business repository. A claim checks with `git ls-remote` that the run's branches are on that checkout's `origin` at the commits the run recorded. It never fetches into the checkout. When this machine's workflow cache differs from the workflows the run ran, the claim reads them at the run's team commit, fetching that commit when the team repo lacks it, and prints `[team commit ] Synced N workflows` on stderr. Your cache and your team repo checkout stay as they are. + +The published state is `workflows/runs//state.json` on `teamai-reports`. It carries what the run needs to continue: status, inputs, each step's outcome, outputs, passes and HITL answers with their notes, the base, the commit of each workspace, and the tool and id of each session copy a step continued. The workflows themselves travel only as `sourceHash`, `definitionHash`, and `teamRevision`. It leaves out step and pass `message`, the driver identity of each lease and answer (a lease keeps its driver kind), workspace owners, and the business checkout path. Every failure message becomes `The details stay on the machine that ran this step.`, and the failure `code` stays. A commit appears only as a full id (40 or 64 hex digits). `publish` checks every value and key against TeamAI's secret patterns and this process's environment secrets, and refuses the first match with `secret_detected`. The refusal names the step, the field, and the kind of secret, never the value. Such a run cannot be handed over. Continue it on this machine. + +`--sessions` also puts the latest copy of each agent session the unsettled steps will continue (`session: continue` or `session.from`) in the team session archive, scrubbed as `teamai session push --scrub` scrubs them. That includes a step a loop is about to run again and the steps of a workflow step that has not started. It never includes a settled step's session or an earlier pass's. The envelope then lists them in `sessions` as `{"tool","id"}`, an empty list when there are none. A session this machine cannot read is left out: its chain archives the newest one it can read, and `skipped` lists the newer ones it left out. A chain with none readable refuses the publish. The sessions are archived before the state is written, so a publish refused after that leaves them archived, and an unchanged publish still archives them. On the machine that continues the run, the local driver resumes a step's session in this order: + +1. The session on this machine. The step's tool resumes it by id when it is the session's tool, and gets a copy of it otherwise. +2. A copy of the archived session. The copy is written once, and a later restore reuses it with the turns added to it. The driver reads the archive as the team repo's origin has it, fetched once per driver run, so you do not run `teamai pull` first, and your team repo clone stays as it is. When that fetch fails, it reads your clone and says it may be older than origin. +3. When the session is a copy another step made, and neither of those has it, the same two ways for the session that copy continued, and so on back to the first. The step then misses the turns of the copies it could not get. +4. A new session, with a warning on stderr that starts `⚠ Step "" starts a new session:` and says why each of those failed. + +The step's tool needs its session store on that machine, such as `~/.claude/projects` for Claude Code, as `teamai session restore` does. OpenCode gets the archived session, but its step still starts a new session, because OpenCode has no flag to resume one. + +A lease's `session` lists in `earlier` the sessions it continued, newest first, for a driver that cannot get the session itself. A driver other than `local` that continued the lease's `session`, or one in `earlier`, by its id or in a copy, puts the one it continued in its report as `resumed` (`{"tool","id"}`), next to the `session` it ran. It leaves `resumed` out when it started a new session. Without `resumed`, later steps resume the lease's session, not the copy. A `resumed` without a tool and an id, on a step that resumes no session, or naming a session the lease did not offer (neither its `session` nor one in `earlier`), answers `invalid_report`. + +Every refusal is one JSON envelope with `error.code`, `error.message`, and the extra fields listed below. The command exits 1. + +| Code | What happened | Next step | +|---|---|---| +| `not_owner` | Another install owns the run. Carries `owner` and `ownedAt`. | Ask the owner, or take it over with `resume --claim`. | +| `state_moved` | The run changed while this machine prepared its claim or its publish: on `teamai-reports` for a claim, on this machine for a publish. Nothing was claimed or published. | Claim again, or publish again. | +| `state_conflict` | `teamai-reports` holds a later state than this machine last saw. | Run `resume ` to take the published state. | +| `run_not_published` | Nobody published the run. | Publish it from the machine that has it. | +| `run_not_found` | This machine has no run with that id. | Check the id. | +| `run_unreadable` | The run file on this machine cannot be read. | Check the run file in the TeamAI data directory. | +| `run_mismatch` | A run here has the same id but started at another time. Carries `localCreatedAt` and `remoteCreatedAt`. | Neither run is changed. They are different runs. | +| `invalid_run` | The id is not `r` followed by 8 or 16 hex digits. | Check the id. | +| `unsupported_repo` | The team repo is an `http` repo, which has no `teamai-reports`. | Continue the run on the machine that has it. | +| `busy` | Another teamai process is writing `teamai-reports` or the run. | Try again. | +| `reports_unreachable` | `teamai-reports` could not be read from the team repo's origin. | Check the connection, then try again. | +| `reports_failed` | The write to `teamai-reports` failed. | Try again. | +| `records_pending` | The run's records did not reach `teamai-reports`: git's push error, a busy writer, or a record queued while the publish ran. | Try again. | +| `definition_mismatch` | The claim cannot get the workflows the run ran. May carry `teamRevision`. | Follow the message: run `teamai pull` after the other machine pushed its workflows, fetch the team commit, or use the same teamai version on both machines. | +| `snapshot_unsupported` | The state has a field, named in `field`, that the snapshot format cannot carry or this teamai cannot read. | On a claim, update teamai, then try again. On a publish, fix the driver that wrote the value, such as a commit that is not a full id. | +| `secret_detected` | A value looks like a secret. Carries `step` (when in a step), `field`, and `label`. | Continue the run on this machine. | +| `sessions_not_archived` | No session of a chain the steps continue can be read here (or its tool is not one teamai reads), the archive was busy, or the commit or push failed. | Publish again, or publish without `--sessions` so those steps start new sessions. | +| `origin_unavailable` | The business repository's origin could not be read or updated. | Run the command in the run's business checkout, with access to its origin. | +| `commit_missing` | A recorded branch is not on origin, is behind the recorded commit on a publish, or the base cannot be pinned because this checkout lacks it. Carries `branch` and `commit`. | Push the branch from the machine that has the commit. On a publish, the message gives the `git -C push origin :refs/heads/` to run. Or publish from the checkout the run started in. | +| `branch_diverged` | A branch on origin holds work the run does not know about. Carries `branch`, `expected`, and `actual`. | On a publish, have the driver that pushed it report it, or move the branch on origin back to the recorded commit. On a claim, the member the message names can publish the run again; moving the branch back discards the commits past the recorded one. | + +After a claim, `resume` can also answer `branch_diverged` when a local branch cannot be brought to the commit the run recorded. The message names the workspace and what to do. See [Workspaces in your own driver](#workspaces-in-your-own-driver). + +### Publish from CI + +A CI job that runs a workflow with `--driver local` can publish the run when it fails, so a member resumes it on their own machine. The job publishes from the checkout the run started in. `publish` pushes the run's base to the business repository's origin once, as `teamai//base`, without force, and reads the run's branches there with `git ls-remote`. The job's token therefore needs push access to the business repository, which on GitHub Actions is `permissions: contents: write`. A workflow that runs for a pull request from a fork gets a read-only token and cannot publish. The job also needs write access to the team repo, where `teamai-reports` lives. + +### Claim from your own driver + +A driver other than `local` claims a run by adding `"claim": true` to its first `next` payload: + +```text +teamai workflow next --json '{"claim":true,"driver":{"kind":"script","machine":"runner","workspace":"/work"}}' +``` + +The claim runs before `next` leases anything, and checks the run's branches on the origin of `driver.workspace`. A run this install already owns stays as it is. A claim refusal comes back as the `next` error envelope. On `state_moved`, send the claim again. A value other than a boolean answers `invalid_json` with `claim must be a boolean`. A driver whose runs another machine may continue reports full commit ids in `changes` and `read`. `publish` refuses a run with any other commit value as `snapshot_unsupported`, naming the field. + +## Workspaces in your own driver + +A driver other than `local` that takes workspace leases keeps the run's commits where another driver or another machine can find them. Every driver follows two rules: + +1. Before it starts a step from `workspace.start`, merges an entry of `workspace.merge`, or restores agent files from `workspace.configCommit`, it checks that its checkout has the commit, with `git cat-file -e ^{commit}`. When one is missing, it fetches the run's branches with the refspec `+refs/heads/teamai//*:refs/remotes/origin/teamai//*` and checks again. A fetch whose refspec matches no branch still succeeds, so only the second check decides. A commit that is still missing fails the step with `workspace_base` before anything runs, and the message names the commit and its branch. +2. When the checkout has an `origin`, it pushes the branch of the commits it reports in `changes` before it sends the report, also when the step failed. When that push fails, it still reports the commits. A step that succeeded then fails with `push_failed`. A step that failed keeps its own failure, and its message adds that the branch was not pushed. + +`workflowGitConformanceScenarios` in `src/workflow/conformance.ts` checks both rules against a bare `origin`. Run them against your driver, as `src/__tests__/workflow-git-conformance.test.ts` runs them against `local`. + +A driver that continues a [published run](#publish-and-claim) relies on these rules too: + +- `publish` pins the run's base on `origin` as `teamai//base`, so the fetch in rule 1 also brings in the base commit. +- After a claim, before it continues a workspace it has locally, the driver moves that branch forward to the commit the run recorded. It refuses, and moves nothing, when the local branch is ahead of that commit or has diverged from it, when its worktree has changes or is on another branch, or when a read-only step left it and it is not back. The local driver does this once per claim, before its first lease, and answers `branch_diverged` with what to do. +- A workspace the claim brought in has no owner until a driver leases a step in it. That driver becomes its owner. +- A lease another machine held in a step that opened its own workspace starts again as that step's next pass, on a new branch. The old branch stays on `origin` with whatever was pushed to it. That step asks its HITL question again. A lease in any other step is released and runs again in the same pass. diff --git a/docs/guide/workflow.md b/docs/guide/workflow.md index c97b3cba5..2cd76d51a 100644 --- a/docs/guide/workflow.md +++ b/docs/guide/workflow.md @@ -159,7 +159,7 @@ teamai workflow report --json '{"attempt":"","revision":1,"outcom The run is stored in this scope's TeamAI data directory. The command exits when it finishes. A local run that ends `failed` or `cancelled` exits with status 1. No workflow process keeps running. -Use `workflow respond ` for a waiting HITL request, `workflow retry ` for a failed or stuck step, `workflow cancel [step]` to stop work, and `workflow resume ` to continue locally. `next` answers `lease`, `hitl`, `wait`, or `done`. See [HITL](workflow-advanced.md#hitl-human-in-the-loop) and [Retry, cancel, and resume](workflow-advanced.md#retry-cancel-and-resume). +Use `workflow respond ` for a waiting HITL request, `workflow retry ` for a failed or stuck step, `workflow cancel [step]` to stop work, `workflow resume ` to continue locally, and `workflow publish ` to let another machine continue it ([Continue a run on another machine](#continue-a-run-on-another-machine)). `next` answers `lease`, `hitl`, `wait`, or `done`. See [HITL](workflow-advanced.md#hitl-human-in-the-loop) and [Retry, cancel, and resume](workflow-advanced.md#retry-cancel-and-resume). ## Workspaces @@ -173,9 +173,9 @@ Inside a workspace an enclosing step opened, a read-only step runs in that works The step sees the agent files from the commit the run started at: `AGENTS.md`, `CLAUDE.md`, `.agents`, `.githooks`, every tool's project directory such as `.claude` or `.tclaude`, the project MCP files `.mcp.json` and `opencode.json`, and Copilot's agent paths under `.github`. A hook an earlier step committed does not run in a later step. The team's skills, rules, agents and MCP are then delivered into the worktree, as they are for any new worktree of this checkout. Those delivered files are not part of the step's commit. A file the step itself edits is committed. -When the checkout has an `origin` and the step made commits, the driver pushes that branch. A later attempt that adds no commit still pushes when this branch has commits `origin` does not have. That includes a first push that never landed, and an origin that has no branches yet. A failed push fails the step with `push_failed` and keeps the worktree. With no `origin`, or when `origin` already has that commit, the run still succeeds and reports the branch. +When the checkout has an `origin` and the step made commits, the driver pushes that branch, also when the step failed. A later attempt that adds no commit still pushes when this branch has commits `origin` does not have. That includes a first push that never landed, and an origin that has no branches yet. A failed push fails a step that succeeded with `push_failed` and keeps the worktree. A step that failed keeps its own failure, and its message adds that the push failed. With no `origin`, or when `origin` already has that commit, the run still succeeds and reports the branch. -`session: continue` resumes this step's previous pass with the same tool. `session: { from: }` resumes a step this one needs. A different tool starts a fresh session. OpenCode always starts fresh. +`session: continue` resumes this step's previous pass with the same tool. `session: { from: }` resumes a step this one needs. A different tool resumes a copy of the session. OpenCode gets the copy but always starts fresh, because it has no flag to resume a session. A step resumes a session as the last step that continued it left it, whichever tool or machine that step ran on. A step that started a new session, such as an OpenCode step, does not change what later steps resume. A command that starts its own sandbox, such as `docker`, stays read-only. What it writes does not become a branch a later step can merge. @@ -198,6 +198,23 @@ steps: command: node -e "require('fs').writeFileSync('joined.txt', 'J')" ``` +## Continue a run on another machine + +A run lives in the TeamAI data directory of the machine that started it. To continue it somewhere else, publish it, then resume it on the other machine. Use this when your tool runs out of tokens in the middle of a run, when you hand a run to a teammate, or when a CI run failed and you want to finish it on your own machine. + +```text +teamai workflow publish # on the machine that has the run +teamai workflow resume # on the other machine, in a checkout of the same repository +``` + +`publish` writes the run's state to the team's `teamai-reports` branch and makes this install the run's owner. Stop the driver first, so no step is in flight. A step still leased when you publish runs again on the other machine. A Ctrl-C leaves what the in-flight step wrote in that step's worktree on this machine, and another machine can still claim the run. If you take the run back here later, the claim makes this install the owner, but `resume` refuses with `branch_diverged` and names the worktree. Move those changes elsewhere or remove them, then run `resume` again. On a machine that does not have the run, `resume` takes it from `teamai-reports`, makes that install the owner, and drives it with the local driver. + +Only the owner continues a published run. On another install that has the run, `resume` and `publish` answer `not_owner` with the owner's user name and install id. Two installs on one machine show the same user name, and the install id tells them apart. To take the run over, run `teamai workflow resume --claim`. The claim replaces this machine's copy of the run and releases the leases other machines held. The previous owner can then no longer publish or resume the run, unless it claims it back. + +When a published run ends, teamai publishes how it ended, without `--sessions`. A member who then takes the run over runs none of its steps again. + +A step with `session: continue` or `session.from` starts a new agent session on the other machine, unless you publish with `--sessions`. When a run goes back and forth, every machine that continued a session publishes with `--sessions` before the next claim, so the next step gets every turn. [Publish and claim](workflow-advanced.md#publish-and-claim) covers what the published state carries, every error and its next step, and publishing from CI. + ## Run records With [usage reporting](reporting.md) on, every run publishes a privacy-preserving record to the team's `teamai-reports` branch: once when it starts and once when it ends. With it off, nothing is published and the envelopes carry no `record`. Records live in `workflows/runs/.json`, keyed by run id, so a retried publication updates the same record instead of adding a copy. diff --git a/docs/guide/zh-CN/dashboard.md b/docs/guide/zh-CN/dashboard.md index f8553773b..49a5bba15 100644 --- a/docs/guide/zh-CN/dashboard.md +++ b/docs/guide/zh-CN/dashboard.md @@ -73,4 +73,8 @@ teamai session resume --platform claude-code 脚本可用 `migrate -s <工具> -t <工具> --json`：stdout 只输出一行，格式与 `teamai workflow` 相同，即 `{ "data": ... }` 或 `{ "error": { "code", "message" } }`；进度信息走 stderr。`data` 包含目标 `sessionId`、`cwd`、`reused` 和 `fidelity` 报告。`error.code` 标明失败原因，如 `no_adapter`（未知工具）、`tool_unavailable`（源工具或目标工具未安装，或其 CLI 无法运行）、`not_found`、`unreadable`，退出码为 1。`--json` 从不询问：缺少会话 id、`-s` 或 `-t` 时以 `prompt_refused` 失败，`--push` 需要 `-y` 或 `--scrub`。 -`--dry-run` 只预览。`--scrub` 先脱敏再归档；不加时，交互运行会询问，脚本需要 `-y`。`--all` 超过 10 条要确认（push 为 5 条）。归档在 `sessions/repos///`，按会话自己的 git remote；否则归入 `_unattributed`。 +`--dry-run` 只预览。`--scrub` 先脱敏再归档；不加时，交互运行会询问，脚本需要 `-y`。`--all` 超过 10 条要确认（push 为 5 条）。归档在 `sessions/repos///`，按会话自己的 git remote；否则归入 `_unattributed`。工作流步骤在自己工作树里运行的会话，归到该工作树所属检出目录的仓库下。 + +session 命令使用本项目安装的团队仓库克隆，没有时使用用户级安装的克隆，所以只有要指向另一个克隆时才需要 `--repo-root`。没有任何安装时，它们输出 `teamai is not set up for this directory.`，提示 `teamai init` 和 `--repo-root`，并以退出码 1 退出。 + +业务仓库自己当团队仓库时，归档在 `teamai-reports` 分支上。另一个 teamai 命令正在写这个分支时，`session push`、`session migrate --push` 和 `session pull` 输出 `Error: Another teamai command is writing teamai-reports`，以退出码 1 退出，不改动该分支。`migrate --push` 会保留它生成的副本。等另一个命令结束后再执行一次。 diff --git a/docs/guide/zh-CN/workflow-advanced.md b/docs/guide/zh-CN/workflow-advanced.md index c47e58172..1b884c77c 100644 --- a/docs/guide/zh-CN/workflow-advanced.md +++ b/docs/guide/zh-CN/workflow-advanced.md @@ -70,7 +70,7 @@ workflows: - 每个命令都在没有终端的情况下运行，因此无法读取你的输入。 - 收到 Ctrl-C，或 SIGTERM、SIGQUIT、SIGHUP 时，teamai 停止所有命令，等它们退出，然后以该信号退出。 -为 1 时，驱动一次运行一个步骤，命令在你的终端里运行，Ctrl-C 会直接到达它。 +为 1 时，驱动一次运行一个步骤，命令在你的终端里运行，Ctrl-C 会直接到达它。只发给 teamai 的 SIGTERM 或 SIGHUP 会停止命令，等它退出，然后以该信号退出。 结局取消一个正在运行的命令时，驱动向它及其启动的进程发送 SIGTERM，5 秒后如果其中任何进程仍在运行，再发送 SIGKILL。正在停止的命令仍占用并行槽位，直到它的所有进程退出。 @@ -648,4 +648,104 @@ teamai workflow resume --tools default=claude 指定步骤的 cancel 停止 pending、waiting、leased 或 running 工作，同时停止尚未结束的后代。cancelled 属于 stopped，因此依赖步骤会跳过，除非其 if 读取取消结果来恢复；运行通常失败。取消整个运行立即以 `cancelled` 结束，不启动下一轮。cancel 不增加修订号。迟到的报告返回 `lease_cancelled`，`next.cancelled` 告诉驱动停止哪些工具。 -在重试、回答、Ctrl-C 或关闭终端后，用 resume 继续本地运行。它优先使用本机未结本地租约的工作目录，其次使用最近调用 `next` 的驱动工作目录（该驱动必须是本机本地驱动），最后使用当前目录。因此重试后即使从子目录继续，也会保留原来的本地工作目录。每个运行同时只能有一个本地驱动。重复 resume 返回 driver_running 和持有者进程号，请停止该驱动或等待；死进程留下的锁会自动回收。已结算运行返回 done。本地运行以 failed 或 cancelled 结束时退出码为 1；控制命令成功时退出码为 0，即使返回的运行状态为 failed 或 cancelled。 +在重试、回答、Ctrl-C 或关闭终端后，用 resume 继续本地运行。`workflow start --driver local` 在运行开始时就把运行 id 和继续它的 `resume` 命令输出到 stderr。它优先使用本机未结本地租约的工作目录，其次使用最近调用 `next` 的驱动工作目录（该驱动必须是本机本地驱动），最后使用当前目录。因此重试后即使从子目录继续，也会保留原来的本地工作目录。每个运行同时只能有一个本地驱动。重复 resume 返回 driver_running 和持有者进程号，请停止该驱动或等待；死进程留下的锁会自动回收。已结算运行返回 done。本地运行以 failed 或 cancelled 结束时退出码为 1；控制命令成功时退出码为 0，即使返回的运行状态为 failed 或 cancelled。 + +## 发布和认领 + +```sh +teamai workflow publish [--sessions] +teamai workflow resume [--claim] +``` + +[在另一台机器上继续运行](workflow.md#在另一台机器上继续运行)介绍常见用法。本节说明具体规则。 + +第一次 `publish` 让本安装实例成为该运行的所有者。所有者是一个安装实例 id（保存在 TeamAI 数据目录下的 `workflows/owner.json`）加上操作系统用户名。同一台机器上的两个安装实例显示相同的用户名，用 id 区分它们。`publish` 输出一个信封： + +```json +{"data":{"run":"","publication":"published","stateRevision":1,"inFlight":[{"step":"fix","attempt":"","driverKind":"local"}]}} +``` + +`inFlight` 列出仍未结束的租约。认领会释放它们，所以这些步骤会重新运行。发布一个自上次发布以来没有变化的运行，返回 `"publication":"unchanged"` 和相同的 `stateRevision`，不写入任何内容。每次发布有变化的运行，都会把完整状态加入 `teamai-reports` 的历史，所以在交接运行时发布，而不是每个步骤后都发布。只有所有者能发布该运行。在另一个安装实例上，`publish` 返回 `not_owner`，不做任何改动。 + +已发布的运行结束时（通过 `next`，或通过 `workflow respond`、`retry`、`cancel`），teamai 会像 `workflow publish` 一样发布它的结束状态。被拒绝时，`next` 的 `done` 信封，或该命令的 `data`，会带上 `notPublished`，内容是下表中的一个拒绝。`not_owner` 表示另一个安装实例认领了该运行，所以它的状态和运行记录保持那个安装实例的样子。其他拒绝按其下一步处理。之后对该运行的 `next` 或 `workflow resume` 也会再次发布结束状态。 + +`resume` 根据本机已有的内容决定怎么做： + +- 本机有、且从未发布过的运行在本地继续。`resume` 不读取 `teamai-reports`。 +- 第一次发布没有到达 `teamai-reports` 的运行也在本地继续。`resume` 会读取 `teamai-reports` 来确认，读不到时返回 `reports_unreachable`。 +- 本机没有的运行从 `teamai-reports` 认领，然后运行。 +- 已发布的运行只在本安装实例拥有它时继续。否则 `resume` 返回 `not_owner`，带 `owner`（`id`、`user`）和 `ownedAt`。 +- 认领一个已结束的运行，不会重新运行它的任何步骤。认领的安装实例成为该运行的所有者，所以在那里执行 `workflow retry` 只重新运行指定的步骤和依赖它的步骤。 +- `--claim` 从所有者那里接管已发布的运行，并替换本机的运行副本。本机有该运行的本地驱动正在运行时，它返回 `driver_running`，不认领任何内容。 + +在该运行的业务仓库的检出目录中执行 `resume`。认领会用 `git ls-remote` 检查该运行的分支在这个检出目录的 `origin` 上，并且停在运行记录的提交上。它从不 fetch 到这个检出目录。本机的工作流缓存与运行所用的工作流不同时，认领会在运行的团队提交上读取它们（团队仓库没有该提交时先 fetch），并在 stderr 输出 `[team commit ] Synced N workflows`。你的缓存和团队仓库检出目录保持不变。 + +发布的状态是 `teamai-reports` 上的 `workflows/runs//state.json`。它包含运行继续所需的内容：状态、输入、每个步骤的结果、输出、pass 和带备注的 HITL 回答、基准、每个工作区的提交，以及步骤继续过的每个会话副本的工具和 id。工作流本身只以 `sourceHash`、`definitionHash` 和 `teamRevision` 传递。它不包含步骤和 pass 的 `message`、每个租约和回答的驱动身份（租约保留驱动类型）、工作区所有者，以及业务检出目录的路径。每条失败消息都换成 `The details stay on the machine that ran this step.`，失败的 `code` 保留。提交只以完整 id（40 或 64 位十六进制）出现。`publish` 用 TeamAI 的密钥规则和本进程环境变量里的密钥检查每个值和键，遇到第一个匹配就以 `secret_detected` 拒绝。拒绝信息写明步骤、字段和密钥类型，从不包含值本身。这样的运行不能交接，请在本机继续。 + +`--sessions` 还会把未结算步骤将要继续的每个 agent 会话的最新副本（`session: continue` 或 `session.from`）放进团队会话归档，并像 `teamai session push --scrub` 一样脱敏。这包括循环即将再次运行的步骤，以及尚未开始的工作流步骤里的步骤。已结算步骤的会话和更早 pass 的会话从不包括在内。信封随后在 `sessions` 中以 `{"tool","id"}` 列出它们，没有时为空列表。本机读不到的会话不归档：其所在的链归档本机能读的最新会话，`skipped` 列出被跳过的较新会话。整条链都读不到时，发布被拒绝。会话在写入状态之前归档，所以之后被拒绝的发布仍会留下已归档的会话，没有变化的发布也照样归档。在接着运行的机器上，本地驱动按以下顺序恢复步骤的会话： + +1. 本机上的会话。步骤的工具与会话的工具相同时按 id 恢复，否则得到它的一份副本。 +2. 归档会话的副本。副本只写一次，之后的恢复会复用它，包括后来加入的对话轮次。驱动按团队仓库的 origin 上的样子读取归档，每次驱动运行 fetch 一次，所以你不需要先执行 `teamai pull`，你的团队仓库克隆也保持不变。fetch 失败时，驱动读取你的克隆，并说明它可能比 origin 旧。 +3. 如果该会话是另一个步骤做出的副本，而前两种方式都拿不到它，就对这个副本所继续的会话用同样两种方式，依此类推直到最初的会话。这时步骤会缺少拿不到的那些副本中的对话轮次。 +4. 新会话，并在 stderr 输出以 `⚠ Step "" starts a new session:` 开头的警告，说明上述每种方式为什么失败。 + +步骤的工具需要在那台机器上有自己的会话存储，例如 Claude Code 的 `~/.claude/projects`，和 `teamai session restore` 一样。OpenCode 会得到归档的会话，但它的步骤仍然开一个新会话，因为 OpenCode 没有恢复会话的参数。 + +租约的 `session` 在 `earlier` 中按从新到旧列出它所继续的会话，供拿不到该会话本身的驱动使用。`local` 以外的驱动如果继续了租约的 `session` 或 `earlier` 中的某个会话（按 id 或在副本中），要在上报中把它继续的那个会话作为 `resumed`（`{"tool","id"}`）放在它运行的 `session` 旁边。新开会话时不带 `resumed`。没有 `resumed` 时，后续步骤恢复租约的会话，而不是副本。缺少工具或 id 的 `resumed`、出现在不恢复会话的步骤上的 `resumed`，或指向租约未提供的会话（既不是它的 `session`，也不在 `earlier` 中）的 `resumed`，都返回 `invalid_report`。 + +每个拒绝都是一个 JSON 信封，带 `error.code`、`error.message` 和下表列出的额外字段。命令以退出码 1 结束。 + +| 代码 | 发生了什么 | 下一步 | +|---|---|---| +| `not_owner` | 另一个安装实例拥有该运行。带 `owner` 和 `ownedAt`。 | 联系所有者，或用 `resume --claim` 接管。 | +| `state_moved` | 本机准备认领或发布期间，运行发生了变化：认领时在 `teamai-reports` 上，发布时在本机上。没有认领或发布任何内容。 | 再认领一次，或再发布一次。 | +| `state_conflict` | `teamai-reports` 上的状态比本机上次看到的更新。 | 执行 `resume ` 取得已发布的状态。 | +| `run_not_published` | 没有人发布过该运行。 | 在有这个运行的机器上发布它。 | +| `run_not_found` | 本机没有这个 id 的运行。 | 检查 id。 | +| `run_unreadable` | 本机的运行文件无法读取。 | 检查 TeamAI 数据目录中的运行文件。 | +| `run_mismatch` | 本机有一个 id 相同、但开始时间不同的运行。带 `localCreatedAt` 和 `remoteCreatedAt`。 | 两个运行都不会被改动。它们是不同的运行。 | +| `invalid_run` | id 不是 `r` 加 8 或 16 位十六进制数字。 | 检查 id。 | +| `unsupported_repo` | 团队仓库是 `http` 仓库，没有 `teamai-reports`。 | 在有这个运行的机器上继续。 | +| `busy` | 另一个 teamai 进程正在写 `teamai-reports` 或该运行。 | 重试。 | +| `reports_unreachable` | 无法从团队仓库的 origin 读取 `teamai-reports`。 | 检查网络连接后重试。 | +| `reports_failed` | 写入 `teamai-reports` 失败。 | 重试。 | +| `records_pending` | 运行记录没有到达 `teamai-reports`：git 推送出错、另一个写入方占用，或发布期间有记录排队。 | 重试。 | +| `definition_mismatch` | 认领拿不到运行所用的工作流。可能带 `teamRevision`。 | 按消息操作：等另一台机器推送工作流后执行 `teamai pull`，fetch 该团队提交，或两台机器使用相同的 teamai 版本。 | +| `snapshot_unsupported` | 状态中有一个字段（`field` 给出）快照格式无法承载，或当前 teamai 无法读取。 | 认领时，更新 teamai 后重试。发布时，修复写入该值的驱动，例如不是完整 id 的提交。 | +| `secret_detected` | 某个值看起来像密钥。带 `step`（位于步骤中时）、`field` 和 `label`。 | 在本机继续该运行。 | +| `sessions_not_archived` | 本机读不到某条链的所有会话（或其工具不是 teamai 能读取的）、归档被占用，或提交或推送失败。 | 再发布一次，或不加 `--sessions` 发布，让这些步骤开新会话。 | +| `origin_unavailable` | 无法读取或更新业务仓库的 origin。 | 在该运行的业务检出目录中执行命令，并确保能访问它的 origin。 | +| `commit_missing` | 记录的分支不在 origin 上，发布时落后于记录的提交，或本检出目录缺少基准提交，无法固定基准。带 `branch` 和 `commit`。 | 在有该提交的机器上推送分支。发布时，消息会给出要执行的 `git -C push origin :refs/heads/`。或在运行开始时所在的检出目录中发布。 | +| `branch_diverged` | origin 上的某个分支包含运行不知道的工作。带 `branch`、`expected` 和 `actual`。 | 发布时，让推送它的驱动上报它，或把 origin 上的分支移回记录的提交。认领时，消息里写明的成员可以再发布一次该运行；把分支移回会丢弃记录的提交之后的提交。 | + +认领之后，如果本地分支无法移到运行记录的提交，`resume` 也可能返回 `branch_diverged`。消息里写明工作区和该怎么做。见[在自己的驱动里使用工作区](#在自己的驱动里使用工作区)。 + +### 从 CI 发布 + +用 `--driver local` 运行工作流的 CI 任务可以在运行失败时发布它，让成员在自己的机器上继续。任务在运行开始时所在的检出目录中发布。`publish` 把运行的基准以 `teamai//base` 推送到业务仓库的 origin，只推一次且不强制推送，并用 `git ls-remote` 读取运行在那里的分支。因此任务的 token 需要业务仓库的推送权限：在 GitHub Actions 上是 `permissions: contents: write`。为来自 fork 的 pull request 运行的工作流只拿到只读 token，无法发布。任务还需要团队仓库的写权限，因为 `teamai-reports` 在团队仓库里。 + +### 在自己的驱动里认领 + +`local` 以外的驱动在第一次 `next` 的 payload 中加上 `"claim": true` 来认领运行： + +```text +teamai workflow next --json '{"claim":true,"driver":{"kind":"script","machine":"runner","workspace":"/work"}}' +``` + +认领在 `next` 租出任何步骤之前执行，并在 `driver.workspace` 的 origin 上检查运行的分支。本安装实例已经拥有的运行保持原样。认领被拒绝时，作为 `next` 的错误信封返回。遇到 `state_moved` 时，再发送一次认领。非布尔值返回 `invalid_json`，消息为 `claim must be a boolean`。运行可能被其他机器继续的驱动，要在 `changes` 和 `read` 中上报完整的提交 id。`publish` 遇到其他形式的提交值时以 `snapshot_unsupported` 拒绝，并写明字段。 + +## 在自己的驱动里使用工作区 + +`local` 以外领取工作区租约的驱动，要把运行的提交放在其他驱动或其他机器能找到的地方。每个驱动都遵守两条规则： + +1. 从 `workspace.start` 开始一个步骤、合并 `workspace.merge` 里的一项，或从 `workspace.configCommit` 恢复 agent 文件之前，先用 `git cat-file -e ^{commit}` 检查检出目录里有没有这个提交。缺少时，用 refspec `+refs/heads/teamai//*:refs/remotes/origin/teamai//*` 取回这次运行的分支，再检查一次。refspec 没有匹配到任何分支时，fetch 也会成功，所以只有第二次检查算数。仍然缺少的提交会让步骤在运行任何内容之前以 `workspace_base` 失败，消息里写明该提交和它所在的分支。 +2. 检出目录有 `origin` 时，驱动在发送报告之前推送 `changes` 里报告的提交所在的分支，步骤失败时也一样。推送失败时，这些提交仍然会被报告。成功的步骤随后以 `push_failed` 失败。本身已失败的步骤保留自己的失败，消息里再补充该分支没有推送。 + +`src/workflow/conformance.ts` 里的 `workflowGitConformanceScenarios` 在一个 bare `origin` 上检查这两条规则。像 `src/__tests__/workflow-git-conformance.test.ts` 对 `local` 所做的那样，用它们检查你的驱动。 + +继续一个[已发布的运行](#发布和认领)的驱动同样依赖这些规则： + +- `publish` 把运行的基准以 `teamai//base` 固定在 `origin` 上，所以规则 1 的 fetch 也会取回基准提交。 +- 认领之后，驱动在继续一个本地已有的工作区之前，先把该分支向前移到运行记录的提交。本地分支领先于该提交或与它分叉、worktree 有改动或不在该分支上、或被只读步骤留下且尚未恢复时，驱动拒绝，什么都不移动。本地驱动在每次认领后、第一次租约前做一次这项检查，并以 `branch_diverged` 返回，说明该怎么做。 +- 认领带来的工作区在某个驱动租下其中的步骤之前没有所有者。该驱动成为它的所有者。 +- 另一台机器在一个自己打开工作区的步骤上持有的租约，会作为该步骤的下一个 pass 重新开始，使用新分支。旧分支和推送到它上面的内容留在 `origin` 上。该步骤会重新询问它的 HITL 问题。其他步骤上的租约被释放，并在同一个 pass 中重新运行。 diff --git a/docs/guide/zh-CN/workflow.md b/docs/guide/zh-CN/workflow.md index 8f8bd05cf..f387db24a 100644 --- a/docs/guide/zh-CN/workflow.md +++ b/docs/guide/zh-CN/workflow.md @@ -159,7 +159,7 @@ teamai workflow report --json '{"attempt":"","revision":1,"outcom 运行状态保存在当前 scope 的 TeamAI 数据目录里。命令结束后进程就退出。本地运行以 `failed` 或 `cancelled` 结束时，退出码是 1。不会留下常驻的工作流进程。 -等待 HITL 请求用 `workflow respond ` 回答，失败或卡住的步骤用 `workflow retry ` 重试，`workflow cancel [step]` 停止工作，`workflow resume ` 继续本地执行。next 返回 lease、hitl、wait 或 done。参见[HITL](workflow-advanced.md#hitl人在回路中)和[重试、取消和继续运行](workflow-advanced.md#重试取消和继续运行)。本地运行以 failed 或 cancelled 结束时退出码为 1。结束运行的操作在锁内排队写入结束记录，在释放锁后发布；重试后仍更新同一路径。回答记录只发布 choice、channel 和 agent，备注、回答时间和回答驱动身份只保存在本地。 +等待 HITL 请求用 `workflow respond ` 回答，失败或卡住的步骤用 `workflow retry ` 重试，`workflow cancel [step]` 停止工作，`workflow resume ` 继续本地执行，`workflow publish ` 让另一台机器接着运行（见[在另一台机器上继续运行](#在另一台机器上继续运行)）。next 返回 lease、hitl、wait 或 done。参见[HITL](workflow-advanced.md#hitl人在回路中)和[重试、取消和继续运行](workflow-advanced.md#重试取消和继续运行)。本地运行以 failed 或 cancelled 结束时退出码为 1。结束运行的操作在锁内排队写入结束记录，在释放锁后发布；重试后仍更新同一路径。回答记录只发布 choice、channel 和 agent，备注、回答时间和回答驱动身份只保存在本地。 ## 工作区 @@ -173,9 +173,9 @@ teamai workflow report --json '{"attempt":"","revision":1,"outcom 步骤看到的是运行开始时那个提交上的 agent 文件：`AGENTS.md`、`CLAUDE.md`、`.agents`、`.githooks`，每个工具的项目目录（如 `.claude` 或 `.tclaude`），项目级 MCP 文件 `.mcp.json` 和 `opencode.json`，以及 Copilot 在 `.github` 下的 agent 路径。前一个步骤提交的 hook 不会在后面的步骤里运行。团队的 skills、rules、agents 和 MCP 随后会交付到这个 worktree，和这个检出目录的其他新 worktree 一样。这些交付的文件不属于该步骤的提交。步骤自己改过的文件会被提交。 -检出目录有 `origin`，并且这一步产生了提交时，驱动会推送该分支。后一次尝试即使没有新提交，只要这个分支上有 `origin` 还没有的提交，仍会再推一次。这包括第一次推送根本没成功，以及 `origin` 上还没有任何分支。推送失败时步骤以 `push_failed` 失败，并保留 worktree。没有 `origin`，或 `origin` 已经有这个提交时，运行仍然成功，并报告该分支。 +检出目录有 `origin`，并且这一步产生了提交时，驱动会推送该分支，步骤失败时也一样。后一次尝试即使没有新提交，只要这个分支上有 `origin` 还没有的提交，仍会再推一次。这包括第一次推送根本没成功，以及 `origin` 上还没有任何分支。推送失败时，成功的步骤以 `push_failed` 失败，并保留 worktree。本身已失败的步骤保留自己的失败，消息里再补充推送失败。没有 `origin`，或 `origin` 已经有这个提交时，运行仍然成功，并报告该分支。 -`session: continue` 用同一个工具恢复这一步上一 pass 的会话。`session: { from: <步骤> }` 恢复这一步所依赖的步骤。换成别的工具会开一个新会话。OpenCode 总是新开会话。 +`session: continue` 用同一个工具恢复这一步上一 pass 的会话。`session: { from: <步骤> }` 恢复这一步所依赖的步骤。换成别的工具时，会把会话复制到该工具里，再恢复这份副本。OpenCode 也会拿到副本，但总是新开会话，因为它没有恢复会话的参数。步骤恢复的会话，是最后一个继续它的步骤留下的样子，不管那个步骤在哪个工具或哪台机器上运行。新开会话的步骤（例如 OpenCode 步骤）不会改变后续步骤恢复什么。 自己启动沙箱的命令（例如 `docker`）仍然是只读的。它写出去的内容不会变成后续步骤可以合并的分支。 @@ -198,6 +198,23 @@ steps: command: node -e "require('fs').writeFileSync('joined.txt', 'J')" ``` +## 在另一台机器上继续运行 + +运行保存在启动它的那台机器的 TeamAI 数据目录里。要在别处接着运行，先发布它，再在另一台机器上继续。适用的情况：工具在运行中途用完了 token，把运行交给队友，或者 CI 里的运行失败了，你想在自己的机器上把它跑完。 + +```text +teamai workflow publish # 在有这个运行的机器上 +teamai workflow resume # 在另一台机器上，位于同一个仓库的检出目录中 +``` + +`publish` 把运行状态写到团队的 `teamai-reports` 分支，并让本安装实例成为该运行的所有者。先停止驱动，让没有步骤在执行。发布时仍处于租约中的步骤会在另一台机器上重新运行。按 Ctrl-C 后，正在执行的步骤写下的内容会留在本机该步骤的工作树里，另一台机器仍然可以认领这个运行。之后你在本机把运行接管回来时，认领会让本安装实例成为所有者，但 `resume` 会返回 `branch_diverged` 并指出那个工作树。把这些改动移到别处或删除，再执行 `resume`。在没有这个运行的机器上，`resume` 从 `teamai-reports` 取得它，让该安装实例成为所有者，并用本地驱动运行它。 + +已发布的运行只有所有者能继续。在另一个已有该运行的安装实例上，`resume` 和 `publish` 都返回 `not_owner`，带上所有者的用户名和安装实例 id。同一台机器上的两个安装实例显示相同的用户名，靠安装实例 id 区分。要接管这个运行，执行 `teamai workflow resume --claim`。认领会替换本机的运行副本，并释放其他机器持有的租约。此后原所有者不能再发布或继续这个运行，除非它再认领回来。 + +已发布的运行结束时，teamai 会发布它的结束状态，不带 `--sessions`。之后接手该运行的成员不会重新运行它的任何步骤。 + +带 `session: continue` 或 `session.from` 的步骤在另一台机器上会开一个新的 agent 会话，除非发布时加上 `--sessions`。运行在机器之间来回交接时，每台继续过会话的机器都要在下一次认领前用 `--sessions` 发布，这样下一个步骤能拿到所有对话轮次。[发布和认领](workflow-advanced.md#发布和认领)说明发布的状态包含什么、每个错误及下一步怎么做，以及如何从 CI 发布。 + ## 运行记录 开启[使用统计上报](reporting.md)时，每次运行都会向团队的 `teamai-reports` 分支发布一条保护隐私的记录：开始时一次，结束时一次。关闭后不发布任何记录，返回中也不带 `record`。记录存放在 `workflows/runs/.json`，以 run id 为键，重试的发布会更新同一条记录而不是产生副本。 diff --git a/skill-data/core/references/commands.md b/skill-data/core/references/commands.md index b0277fa9b..7e7fb59b7 100644 --- a/skill-data/core/references/commands.md +++ b/skill-data/core/references/commands.md @@ -227,7 +227,7 @@ Generated: do not edit by hand. Regenerate with - `--auto-approve` — With --driver local, automatically answer optional approve HITL requests and choose defaults in this process - `--inputs ` — Values for the inputs the workflow declares, as a JSON object, for example {"issue":42}. Every declared input is required. - `teamai workflow next ` — Get a lease, hitl, wait, or done JSON envelope. Show hitl.message and hitl.respond to a person, who replaces