diff --git a/docs/dev/workflow-engine.md b/docs/dev/workflow-engine.md index c1510c855..6141d3953 100644 --- a/docs/dev/workflow-engine.md +++ b/docs/dev/workflow-engine.md @@ -6,11 +6,11 @@ Each rule below spans more than one file. The rule names the code that owns it, ## Rules that span files -- **A run ends only in `settleRun`.** `leaseNext` calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. To run code when a run ends, hook into that path in `leaseNext`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `writeRun` then fails, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). +- **A run ends only in `settleRun`.** The shared `finishRun` path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. Respond, cancel and retry in `run-controls.ts`, and next in `engine.ts`, share that path. Whole-run cancel settles directly as `cancelled`, without reconciling another round. To run code when a run ends, hook into `finishRun`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `writeRun` then fails, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). - **Reconcile the latest report before applying another or handing out a lease.** `reconcileRun`, shared by `leaseNext` and `reportAttempt`, settles every driverless transition to a fixpoint at the run's current revision, through `settleScope` and `settleItems`, including workflow starts, fan-out expansion, render failures and bound exhaustion. `reportAttempt` advances the revision only after it, so only the latest revision can be unreconciled. A reached end applies before a loop decision. Only after reconciliation does `leaseNext` hand back an omitted open lease or issue the first ready leaf in declaration order. A running workflow occupies its parent's position in that order. `canTakeLease: false` still reconciles driverless transitions. Every changed run is saved, including a `wait`. -- **A report is checked against the revision of its own lease.** Only `reportAttempt` increments `revision`; skips, `if`, `end`, and the creation of child steps inside `next` never do. A lease keeps the run revision it was handed out at, and `reportAttempt` compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: `stale_revision` when its revision is not the run's, otherwise `wrong_attempt`. -- **An end or break stops what is still open under its workflow.** `stopUnder` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending ones. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. -- **The local driver runs commands in parallel and a prompt alone.** `runLocalDriver` runs up to `workflows.max_parallel` commands (local config, default 1) and sends them as `running`. It holds a leased prompt until no command runs or is still exiting, because the prompt's read-only check compares the whole worktree before and after it. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any process of its group is left, even after the command itself exited. It does not send a stopping command as `running` again, so `next` does not cancel it twice. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. +- **A report is checked against the revision of its own lease.** `reportAttempt` and `retryStep` increment `revision`; skips, `if`, `end`, and the creation of child steps inside `next` never do. A lease keeps the run revision it was handed out at, and `reportAttempt` compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: `stale_revision` when its revision is not the run's, otherwise `wrong_attempt`. +- **An end or break stops what is still open under its workflow.** `stopWhere` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending and waiting ones. The walk takes a path predicate and failure. End and break construct their own end_reached failure; manual cancel and retry supply theirs. Step descendants use the same `isUnder` predicate as `archiveUnder` in `loop.ts`. Cancel records the ids this walk changed before reconciliation can settle or skip other steps. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. +- **The local driver runs commands in parallel and a prompt alone.** `runLocalDriver` runs up to `workflows.max_parallel` commands (local config, default 1) and sends them as `running`. It holds a leased prompt until no command runs or is still exiting, because the prompt's read-only check compares the whole worktree before and after it. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any captured descendant or process in a detached group remains, even after its command exited. Attached commands snapshot their descendant tree and start times before stopping, then probe and send SIGKILL only to descendants whose start times still match. If the identity snapshot fails, only the shell is stopped; detached commands use their process group. It does not send a stopping command as `running` again, so `next` does not cancel it twice. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. - **The end reached first wins.** A reported step's `settledAt` is the revision its report made. Every terminal transition, including a skip or cancellation, carries `settledAt`. A step the engine settles takes the run's current revision, which is that of the report that caused it, because every earlier revision was reconciled before that report was accepted. An open lease it fails takes the revision it was leased at, and a cancellation the winning end's revision. `reportAttempt` accepts a report only after `reconcileRun` has applied every end reached before it, so an end a later report reaches never competes with an earlier one. Every end `settleScope` has yet to apply was reached at the run's current revision, and it applies the one declared first. The winner therefore does not depend on when `next` runs between reports. - **A break is a nameless succeeded end.** A succeeded step whose `break` holds competes with the steps that reached an end by the same rule. It ends its workflow, or that workflow's loop, as success. Its value cannot change after its step succeeds, because pull rejects `break` inside a step loop's range, where a previous-pass read could change. - **A workflow loop is decided when a round has nothing left.** Once every step of a looping workflow has settled and no end or break was reached, `decideRound` in `loop.ts` evaluates the workflow's `loop.while`. When it is false, the workflow ends with the round's outcome and outputs. Otherwise every step of the workflow repends for the next round, unless the workflow has run `max_iterations` rounds: then it fails `loop_exhausted` (`WorkflowResult.failure`, and `run.failure`, shown in `done`, at the top). A failed step alone does not stop the loop. The round is kept on the step that runs the workflow (`Scope.runner`), so `repend` resets it with that step, or on the run at the top. Pull rejects a step loop in a looping workflow. @@ -20,9 +20,9 @@ Each rule below spans more than one file. The rule names the code that owns it, - **A pass id is never reused.** `repend` moves a step's current pass into its `passes` and bumps `pass`. `archiveUnder` moves what a repended workflow or `for_each` step ran into `run.archive`, and `firstPass` starts a new instance after every archived pass of its path. The full record of a run is `run.steps`, with each step's `passes`, plus `run.archive`; a run record, scorer, or budget reads both. `steps..passes` is a step's `passes` plus its current pass once it settled (`passesOf`). The expression projection keeps `results` for a `for_each` pass, including in `previousPass`. Pull checks the same outputs-or-results shape in step reads and workflow output aliases. - **An expression reads its absolute paths, wherever they are.** `expressionPaths` returns every `inputs` and `steps` path, including those inside function arguments and predicates, and never a relative path such as `.size`, which reads the current element of a predicate. `readsOf`, `skipUnreachable` and the pull read checks use it, so a new kind of expression must return its paths there. - **Expression types are checked in two places, through `typeProblems`.** `parseExpression` rejects what an expression shows by itself, such as `x < 'a'` or `any(...) < 1`. `readDiagnostics` checks paths against the types the workflow declares. It checks each path only in the workflow that owns it, so a predicate an inline step runs over an enclosing step is reported once. -- **Pull and the engine share one list of templated fields, one list of step expressions, and one list of actions.** The templated fields are in `templatedTexts` and the expressions, such as `if` and `break`, in `stepExpressions`, both in `template.ts`. The actions are in `STEP_ACTIONS` in `types.ts`. Add a new field or action there, not to a copy in `parse.ts` or `engine.ts`. Pull, the cache's `prompt_file` check, the lease render and `readsOf` all split a template with `parseTemplate`, which decides which `{{ … }}` is a template, and the render evaluates it with `evaluateExpression`, as for an `if`. `parseReads` in `engine.ts` and the step parse in `parse.ts` key a table by each expression, so a new one does not compile until both say how to read it. -- **A new step status must fit `stopped()`, `settled()` and `stopUnder()`.** The first two are in `step-state.ts`, and `stopUnder()` is in `engine.ts`. `stopped()` decides which dependents are skipped, which steps can recover, and the run outcome. `settled()` decides readiness. `stopUnder()` decides what an end or a break does to each step under it. Each switches over every status, so a status added to `StepStateSchema` does not compile until all three place it. -- **Only the transitions in `step-state.ts` write step fields.** `skip`, `lease`, `release`, `start`, `succeed`, `fail`, `cancel`, `repend`, `nextRound`, `endLoop` and `exhaust` change a step in place and drop the fields its new status does not have. `readRun` parses every step through `StepStateSchema`, which gives a run file from an older teamai its defaults: `settledAt` 0, `outcome` from `status`, and no fields its status does not have. +- **Pull and the engine share one list of templated fields, one list of step expressions, and one list of actions.** The templated fields are in `templatedTexts` and the expressions, such as `if` and `break`, in `stepExpressions`, both in `template.ts`. HITL declarations normalize in `parse-hitl.ts`. The actions are in `STEP_ACTIONS` in `types.ts`. Add a new field or action there, not to a copy in `parse.ts` or `engine.ts`. Pull, the cache's `prompt_file` check, the lease render and `readsOf` all split a template with `parseTemplate`, which decides which `{{ … }}` is a template, and the render evaluates it with `evaluateExpression`, as for an `if`. `parseReads` in `engine.ts` and the step parse in `parse.ts` key a table by each expression, so a new one does not compile until both say how to read it. +- **A new step status must fit `stopped()`, `settled()` and `stopWhere()`.** The first two are in `step-state.ts`, and `stopWhere()` is in `engine.ts`. `stopped()` decides which dependents are skipped, which steps can recover, and the run outcome. `settled()` decides readiness. `stopWhere()` decides what an end or a break does to each step under it. Each switches over every status, so a status added to `StepStateSchema` does not compile until all three place it. +- **Only the transitions in `step-state.ts` write step fields.** `waitForAnswer`, `answerHitl`, `resetPending`, `resetIteration`, `skip`, `lease`, `release`, `start`, `succeed`, `fail`, `cancel`, `repend`, `nextRound`, `endLoop` and `exhaust` change a step in place and drop the fields its new status does not have. `readRun` parses every step through `StepStateSchema`, which gives a run file from an older teamai its defaults: `settledAt` 0, `outcome` from `status`, and no fields its status does not have. - **`run.steps` is replaced, never changed in place.** Its type is `readonly`, so `addItems`, `insertChildren` and `archiveUnder` add or remove instances by assigning a new array. An instance's `id` is `readonly` too, so an instance cannot move to another scope's group. `statesIn` in `scope.ts` keeps the instances of every scope per array, in a `WeakMap`, so settling a scope reads its own instances without scanning every step. A new array starts a new index, except that `insertChildren`, which adds the instances of a workflow a step starts, carries the old index over. - **A pull check that changes what a file means carries a `code`.** `dropsKeptVersion` in `cache.ts` removes the cached copy when a diagnostic has a `code`. An older definition then cannot keep running with the new meaning. A diagnostic without a `code` keeps the last valid version, but only after `checkReferences` checks that version again against what the workflows it runs settle to in the same sync. - **The workflow cache holds only what one sync settled.** `checkReferences` checks each file's version against the versions the other files settle to, and `dropUnsettled` removes every other cache entry. `startRun` relies on this: it reads the workflows a run uses from the cache and does not check them against each other again. A new way for a file to keep or lose its version goes through `checkReferences`. @@ -30,6 +30,12 @@ Each rule below spans more than one file. The rule names the code that owns it, - **`item` is bound per instance, in the engine and in pull alike.** `contextOf` gives an item its own value and any other instance its workflow's (`Scope.item`), which `childScope` passes into the inline workflow an item runs. A `for_each` step's own `if` and `for_each` therefore read the enclosing item, never their own. Pull checks the same binding: `ReadSite.item` marks the templates of a `for_each` step, and `ReadScope.item` a workflow that runs per item. A template reads `item` only where it is bound: `parseTemplate` takes the binding, which pull decides from the step and the engine from whether the context holds an item, so elsewhere `{{ item }}` stays text in both. A new place that reads `item` must decide its binding in both. - **Step ids are paths.** A step inside a sub-workflow has an id such as `ship.check.classify`, an item of a `for_each` step one such as `build[2]`, and a step inside the workflow an item runs one such as `build[2].verify`. `parseStepPath` in `step-path.ts` reads them all; don't split ids by hand. A child step exists in the run only after the engine reaches it. To find a step's definition, use `stepDefinitions` in `scope.ts`, which reads the definitions alone, so it also finds the step of an archived pass. `locateStep` in `engine.ts` also gives the step's scope, and needs the instances along the path. Don't look a step up by top-level id. +- **A HITL request waits before any action or end.** `settleScope` checks loop exhaustion first, renders a HITL request message, and moves an unanswered step to `waiting`. Waiting is neither stopped nor settled. An answer moves it to pending without a revision bump. Approve no skips, choose succeeds, approve yes continues. An end-only step with HITL reaches its end only after yes. The shared `templatedTexts` makes HITL reads participate in readiness. Only `driver.can.hitl` declares HITL. `parseNextPayload` reads it into `NextOptions.hitl`; stored `DriverIdentity` and leases contain only kind, machine and workspace. Busy polls never answer waiting HITL requests through the no-HITL rules. Only a driver declaring HITL receives a waiting request; a busy poll without HITL receives wait. Both new leases and hand-backs enforce HITL for interactive steps. A refused hand-back fails at its lease revision and invalidates the attempt. The process-only autoApprove option is never read by the JSON payload parser. `respond --json` uses the same identity schema as next, reports unread fields through ignored, and records a driver UI's declared person answer with channel driver and agent false. Answer driver identity stays local; records publish only choice, channel and agent. +- **Retry validates every container before resetting any.** `retryContainers` in `run-controls.ts` climbs from the target through scopes and fan-out items until a running runner or the top. Each dependency cone includes needs, reads and loop waits, or item after dependencies. An independent standing end or break rejects the retry. The apply pass archives replaced passes and descendants, reopens settled runners with `start` to keep inputs and round, then bumps revision and reconciles. Only the targeted loop owner's iteration resets; the top-level round does not. An answered refused approve or succeeded choose is retryable too, so a changed answer resets its dependent branches. A HITL request in the cone that is waiting, or answered and not yet run, starts a new pass too, so its answer is asked again in the new context and an answer to the old pass is stale. Unanswered skips in every affected container reset so skips caused by cancellation can run again. Retry's prelude reconciles without finishing a provisional outcome. Reopening clears the run's end, failure and endRecordQueued flag and removes its queued end file through records.ts. These three pieces stay consistent: run state, the pending end record, and published stats. The next end upserts the same record path; stats replace the origin end's outcome and duration contribution with the new end's, without adding another run. Replaying the same endedAt contributes nothing. +- **The local driver keeps tools and questions interruptible.** A single question joins command and prompt exits in the driver's race. An interactive tool owns the terminal alone: no question opens while it runs, and starting one aborts and closes an open question. A poll closes the question when its own step is leased or a different step or pass waits; an unrelated lease leaves it open. It polls next every second while asking, every two seconds while tools run, and aborts a replaced HITL request or cancelled prompt. `process-stop.ts` shares process stopping between commands and adapters. Headless tools detach and stop with SIGTERM; interactive tools stay attached and stop with their captured descendant tree. Foreground interactive signals reuse env-exec's terminal detection and signal policy: when the terminal reaches the child, teamai ignores SIGINT and SIGQUIT, while SIGTERM and SIGHUP are forwarded and stop the driver. Interactive adapters inherit stdio and accept an optional result file. Repository edits are kept for interactive steps; only headless prompts run the read-only check. A nonzero interactive exit without a result reports its exit status. `leaseNext` records the identity of the latest driver that called next in `RunState.driver`, without capabilities. Resume uses this machine's open local lease workspace, then run.driver.workspace for a local driver on this machine, then cwd. Controls pass run.driver to finishRun so end records retain the driver kind. + +- **One local driver owns a run on this machine.** `runLocalDriver` takes `driverLockPath` once with `acquireLock` and releases it in finally. A live holder returns `driver_running`, including its pid; the existing lock machinery reclaims dead owners. The driver lock spans tool execution and questions, while `withRun` holds a different run lock only during one engine operation. Neither lock's scope is extended by the other. Every HITL decision without a HITL driver and interactive hitl_required failure logs once when the engine changes the step. HITL replies carry one literal `respond` command template, which drivers display unchanged. + ## Tests Test parsing through `parseWorkflow` and `syncWorkflowDefinitions`. Test the engine through `startRun`, `leaseNext`, and `reportAttempt`. Every driver must pass the conformance suite in `src/workflow/conformance.ts`. If a driver can see a new behavior, add a scenario there. diff --git a/docs/guide/workflow-advanced.md b/docs/guide/workflow-advanced.md index 214863c26..5770f1924 100644 --- a/docs/guide/workflow-advanced.md +++ b/docs/guide/workflow-advanced.md @@ -40,7 +40,7 @@ Each call to `next` does this, in order: A driver that sends no `running` gets its open lease back on every call, so it runs one step at a time. -`"canTakeLease":false` stops after step 2. The answer is `done`, an open lease that `running` leaves out, or `wait`. When nothing is leased, the `wait` reason is `canTakeLease is false`. Use it to learn about an end while the driver has no room for another step. That `wait` does not mean that nothing is left to run, so call `next` again once the driver has room. Step 2 can still hand back an open lease that the driver has no room for, so keep a queue of the leases the driver has not started. +`"canTakeLease":false` issues no new lease but still returns a waiting HITL request to a driver that declares `driver.can.hitl`. The answer is `done`, an open lease that `running` leaves out, `hitl`, or `wait`. When nothing is leased or waiting at a HITL request, the `wait` reason is `canTakeLease is false`. Use it to learn about an end while the driver has no room for another step. That `wait` does not mean that nothing is left to run, so call `next` again once the driver has room. Step 2 can still hand back an open lease that the driver has no room for, so keep a queue of the leases the driver has not started. Every `data` answer of `next` has `cancelled`. It lists the attempts in `running` that are no longer open leases of this driver. Stop those processes. An `error` answer has no `cancelled`. @@ -59,9 +59,10 @@ A wrong `running` or `canTakeLease` returns `invalid_json` with `running must be ```yaml workflows: max_parallel: 4 + auto_approve: false ``` -A prompt step always runs alone. The driver starts it once every command has exited, and starts no command until it finishes. The read-only check compares the whole working tree before and after the prompt, so a command that changes files while the prompt runs would fail it. +A prompt step always runs alone. The driver starts it once every command has exited, and starts no command until it finishes. For a headless prompt, the read-only check compares the whole working tree before and after the prompt, so a command that changes files while the prompt runs would fail it. With `max_parallel` above 1: @@ -71,7 +72,7 @@ With `max_parallel` above 1: With 1, the driver runs one step at a time, and a command runs in your terminal, where Ctrl-C reaches it. -When an end cancels a running command, the driver sends it SIGTERM, and SIGKILL 5 seconds later if it, or with `max_parallel` above 1 a process it started, still runs. Until it exits, it no longer counts toward `max_parallel`. +When an end cancels a running command, the driver sends SIGTERM to it and the processes it started, then SIGKILL 5 seconds later if any still run. A stopping command keeps its slot until its processes exit. ## An end cancels the steps still open @@ -573,3 +574,76 @@ steps: ``` Pass the whole list, as `explain` does in `fan-build`, when the prompt needs every entry. + + +## HITL (human in the loop) + +HITL covers `approve`, `choose`, and `interactive` steps where a person takes part. + +`approve: ` asks before an action. `yes` runs it; `no` skips it. An action-less approve also works. `choose: { message, options, default? }` records a choice without running an action. Options are unique, non-empty strings without whitespace. A choose step cannot carry an action or `required`. Neither HITL request can carry `for_each`; put it on a step the fan-out needs. Messages use the same templates and readiness rules as prompts. + +```yaml +schemaVersion: v1alpha1 +name: hitl-build +description: Choose a path and approve the work. +steps: + - id: decide + choose: + message: Build or stop? + options: [build, stop] + default: build + - id: build + needs: [decide] + if: steps.decide.choice == 'build' + approve: Build it? + command: npm run build + - id: pair + needs: [build] + if: steps.build.outcome == 'failed' + interactive: true + prompt: Fix the build with me. +``` + +Later steps read `steps.decide.choice` and `steps.decide.note`, also through `passes[n]`. A new pass asks again. An approve HITL request on an end-only step must receive `yes` before that end is reached. + +For an approval that must come directly from a person, use `approve: { message: Publish?, required: true }`. Required approvals never use automatic answers. `respond` records the answer time, channel, choice, and optional note. With `--json`, its channel is `driver`, with `agent: false`. Without `--json`, TTY stdin records `tty`, with `agent: false`; otherwise it records `cli`, with `agent: true`. A required approval rejects `cli` answers with `agent_in_path`. A TTY is a channel convention, not protection against someone who can create a PTY. + +```sh +teamai workflow respond decide build --note "Ready" --pass 1 +teamai workflow resume +``` + +`--pass` rejects an answer for a HITL request replaced by a later pass. A second answer in the same pass is `repeated_response`. + +Drivers other than `local` use the same HITL contract. Send `driver.can.hitl` on every `next` request, including busy polls. Set it to `true` when a person can answer HITL requests and work in interactive steps. When no lease is ready, `next` returns `hitl` with `run`, `step`, `pass`, `kind`, rendered `message`, `options`, optional `default`, and `required`, plus a `respond` command template. Show `hitl.message` and `hitl.respond` to that person, who runs `teamai workflow respond` with `