diff --git a/docs/dev/workflow-engine.md b/docs/dev/workflow-engine.md index ea7755fd1..c1510c855 100644 --- a/docs/dev/workflow-engine.md +++ b/docs/dev/workflow-engine.md @@ -1,19 +1,34 @@ # Workflow engine -Maintainer notes on `src/workflow/`. The workflow file format and the driver contract are in the [Workflows](../guide/workflow.md) guide page. +Maintainer notes on `src/workflow/`. The workflow file format and the driver contract are in the [Workflows](../guide/workflow.md) and [Advanced workflows](../guide/workflow-advanced.md) guide pages. Each rule below spans more than one file. The rule names the code that owns it, so you can read the details there. ## Rules that span files -- **A run ends only in `settleRun`.** `leaseNext` calls it before it hands out a lease. `reportAttempt` records the outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. To run code when a run ends, hook into that path in `leaseNext`. Do network I/O only after `withRun` releases the run lock. -- **Settling does not change `revision`.** Only `reportAttempt` increments it. Skips, `if`, `end`, and the creation of child steps happen inside `next`, so the `report` of an open lease still matches its revision. -- **A step waits for every step it reads.** `readsOf` collects the steps that the step's `if`, templates, and `with` read. For a step that runs an inline workflow, it also collects the outer steps that the inline steps read. `skipUnreachable` uses that set to decide readiness. If a new field reads steps, add it to `readsOf`. Otherwise the step can read a value before it exists, or an inline workflow can wait forever. -- **Pull and the engine share one list of templated fields and one list of actions.** The templated fields are in `templatedTexts` in `template.ts`. The actions are in `STEP_ACTIONS` in `types.ts`. Add a new field or action there, not to a copy in `parse.ts` or `engine.ts`. -- **A new step status must fit `stopped()` and `settled()`.** `stopped()` decides which dependents are skipped, which steps can recover, and the run outcome. `settled()` decides readiness. If either function misses the new status, skips, recovery, readiness, and the run outcome change with no error. +- **A run ends only in `settleRun`.** `leaseNext` calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. `reportAttempt` first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next `next` skips the dependents and ends the run. To run code when a run ends, hook into that path in `leaseNext`. Do network I/O only after `withRun` releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's `endRecordQueued` field. When `writeRun` then fails, the queue is one save ahead of the run file, and the next `next` settles the run again and overwrites the queued `.end.json`. A later `next` reports a record state only for a run whose field is true (`runRecordState` in `records.ts`). +- **Reconcile the latest report before applying another or handing out a lease.** `reconcileRun`, shared by `leaseNext` and `reportAttempt`, settles every driverless transition to a fixpoint at the run's current revision, through `settleScope` and `settleItems`, including workflow starts, fan-out expansion, render failures and bound exhaustion. `reportAttempt` advances the revision only after it, so only the latest revision can be unreconciled. A reached end applies before a loop decision. Only after reconciliation does `leaseNext` hand back an omitted open lease or issue the first ready leaf in declaration order. A running workflow occupies its parent's position in that order. `canTakeLease: false` still reconciles driverless transitions. Every changed run is saved, including a `wait`. +- **A report is checked against the revision of its own lease.** Only `reportAttempt` increments `revision`; skips, `if`, `end`, and the creation of child steps inside `next` never do. A lease keeps the run revision it was handed out at, and `reportAttempt` compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: `stale_revision` when its revision is not the run's, otherwise `wrong_attempt`. +- **An end or break stops what is still open under its workflow.** `stopUnder` cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in `cancelledAttempts`. It cancels running workflow steps there and skips pending ones. `reportAttempt` answers a cancelled attempt `lease_cancelled` before it checks the driver; `cancelledAttempts` is never trimmed. Every `next` answer lists in `cancelled` the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, `done` included. +- **The local driver runs commands in parallel and a prompt alone.** `runLocalDriver` runs up to `workflows.max_parallel` commands (local config, default 1) and sends them as `running`. It holds a leased prompt until no command runs or is still exiting, because the prompt's read-only check compares the whole worktree before and after it. On `cancelled`, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any process of its group is left, even after the command itself exited. It does not send a stopping command as `running` again, so `next` does not cancel it twice. The signal policy follows `max_parallel`. At 1, a command runs in teamai's process group, as `env exec` runs one (`src/env-exec.ts`): the terminal's Ctrl-C reaches it, and it can read `/dev/tty`. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses `taskkill /T /F` in place of the group kill. +- **The end reached first wins.** A reported step's `settledAt` is the revision its report made. Every terminal transition, including a skip or cancellation, carries `settledAt`. A step the engine settles takes the run's current revision, which is that of the report that caused it, because every earlier revision was reconciled before that report was accepted. An open lease it fails takes the revision it was leased at, and a cancellation the winning end's revision. `reportAttempt` accepts a report only after `reconcileRun` has applied every end reached before it, so an end a later report reaches never competes with an earlier one. Every end `settleScope` has yet to apply was reached at the run's current revision, and it applies the one declared first. The winner therefore does not depend on when `next` runs between reports. +- **A break is a nameless succeeded end.** A succeeded step whose `break` holds competes with the steps that reached an end by the same rule. It ends its workflow, or that workflow's loop, as success. Its value cannot change after its step succeeds, because pull rejects `break` inside a step loop's range, where a previous-pass read could change. +- **A workflow loop is decided when a round has nothing left.** Once every step of a looping workflow has settled and no end or break was reached, `decideRound` in `loop.ts` evaluates the workflow's `loop.while`. When it is false, the workflow ends with the round's outcome and outputs. Otherwise every step of the workflow repends for the next round, unless the workflow has run `max_iterations` rounds: then it fails `loop_exhausted` (`WorkflowResult.failure`, and `run.failure`, shown in `done`, at the top). A failed step alone does not stop the loop. The round is kept on the step that runs the workflow (`Scope.runner`), so `repend` resets it with that step, or on the run at the top. Pull rejects a step loop in a looping workflow. +- **A step waits for every step it reads.** `readsOf` collects the steps that the step's expressions (`stepExpressions`), templates, and `with` read. For a step that runs an inline workflow, it also collects the outer steps that the inline steps read. `skipUnreachable` uses that set to decide readiness. If a new field reads steps, add it to `stepExpressions` or `templatedTexts`. Otherwise the step can read a value before it exists, or an inline workflow can wait forever. A step outside a loop that needs or reads a step inside it also waits until the loop owner is settled (`waitsFor` in `loop.ts`), and is neither skipped nor ready before. +- **A read of a later step in the same loop is a previous-pass read, and never waits.** `previous` in `loop.ts` classifies it: the target is in the reader's loop range and needs the reader, directly or not. For an inline step, the reader is the step that runs its workflow. Pull allows the read, the engine leaves it out of the reads a step waits for, and `expressionSteps` gives the reader the target's latest archived pass (`previousPass`), which holds no pass fields on pass 1. Classify a new kind of read through `previous` too, or pull and the engine disagree on it. +- **A step loop is decided once per run of its owner.** After checking for an end, `settleScope` calls `decideStepLoop` for each succeeded owner without `loopFinal`. A false `while` sets `loopFinal`. Otherwise the range runs again through `repend`, unless a `while` owner has run `max_iterations` times, which fails it `loop_exhausted`. A loop without `while` is bounded where the owner's action would start, after the skip rule and `if`, for a leaf and a workflow step alike (`exhaustedAtStart`). The owner counts its runs in `iteration`. Pull rejects ranges that share a step and loops that wait for each other (`loopProblems`), so the engine does not check them again. +- **A pass id is never reused.** `repend` moves a step's current pass into its `passes` and bumps `pass`. `archiveUnder` moves what a repended workflow or `for_each` step ran into `run.archive`, and `firstPass` starts a new instance after every archived pass of its path. The full record of a run is `run.steps`, with each step's `passes`, plus `run.archive`; a run record, scorer, or budget reads both. `steps..passes` is a step's `passes` plus its current pass once it settled (`passesOf`). The expression projection keeps `results` for a `for_each` pass, including in `previousPass`. Pull checks the same outputs-or-results shape in step reads and workflow output aliases. +- **An expression reads its absolute paths, wherever they are.** `expressionPaths` returns every `inputs` and `steps` path, including those inside function arguments and predicates, and never a relative path such as `.size`, which reads the current element of a predicate. `readsOf`, `skipUnreachable` and the pull read checks use it, so a new kind of expression must return its paths there. +- **Expression types are checked in two places, through `typeProblems`.** `parseExpression` rejects what an expression shows by itself, such as `x < 'a'` or `any(...) < 1`. `readDiagnostics` checks paths against the types the workflow declares. It checks each path only in the workflow that owns it, so a predicate an inline step runs over an enclosing step is reported once. +- **Pull and the engine share one list of templated fields, one list of step expressions, and one list of actions.** The templated fields are in `templatedTexts` and the expressions, such as `if` and `break`, in `stepExpressions`, both in `template.ts`. The actions are in `STEP_ACTIONS` in `types.ts`. Add a new field or action there, not to a copy in `parse.ts` or `engine.ts`. Pull, the cache's `prompt_file` check, the lease render and `readsOf` all split a template with `parseTemplate`, which decides which `{{ … }}` is a template, and the render evaluates it with `evaluateExpression`, as for an `if`. `parseReads` in `engine.ts` and the step parse in `parse.ts` key a table by each expression, so a new one does not compile until both say how to read it. +- **A new step status must fit `stopped()`, `settled()` and `stopUnder()`.** The first two are in `step-state.ts`, and `stopUnder()` is in `engine.ts`. `stopped()` decides which dependents are skipped, which steps can recover, and the run outcome. `settled()` decides readiness. `stopUnder()` decides what an end or a break does to each step under it. Each switches over every status, so a status added to `StepStateSchema` does not compile until all three place it. +- **Only the transitions in `step-state.ts` write step fields.** `skip`, `lease`, `release`, `start`, `succeed`, `fail`, `cancel`, `repend`, `nextRound`, `endLoop` and `exhaust` change a step in place and drop the fields its new status does not have. `readRun` parses every step through `StepStateSchema`, which gives a run file from an older teamai its defaults: `settledAt` 0, `outcome` from `status`, and no fields its status does not have. +- **`run.steps` is replaced, never changed in place.** Its type is `readonly`, so `addItems`, `insertChildren` and `archiveUnder` add or remove instances by assigning a new array. An instance's `id` is `readonly` too, so an instance cannot move to another scope's group. `statesIn` in `scope.ts` keeps the instances of every scope per array, in a `WeakMap`, so settling a scope reads its own instances without scanning every step. A new array starts a new index, except that `insertChildren`, which adds the instances of a workflow a step starts, carries the old index over. - **A pull check that changes what a file means carries a `code`.** `dropsKeptVersion` in `cache.ts` removes the cached copy when a diagnostic has a `code`. An older definition then cannot keep running with the new meaning. A diagnostic without a `code` keeps the last valid version, but only after `checkReferences` checks that version again against what the workflows it runs settle to in the same sync. - **The workflow cache holds only what one sync settled.** `checkReferences` checks each file's version against the versions the other files settle to, and `dropUnsettled` removes every other cache entry. `startRun` relies on this: it reads the workflows a run uses from the cache and does not check them against each other again. A new way for a file to keep or lose its version goes through `checkReferences`. -- **Step ids are paths.** A step inside a sub-workflow has an id such as `ship.check.classify`, and the grammar leaves room for `[n]`. A child step exists in the run only after the engine reaches it. To find a step's definition, use `locateStep`. Don't look it up by top-level id. +- **The items of a `for_each` step settle like a workflow.** When settling reaches a `for_each` step, `advanceInstance` evaluates its list and `addItems` puts an instance per item right after the step; the step is `running` until every item settled. `settleRunning` settles a running `for_each` step through `settleItems`, the way it settles a workflow step through `settleScope`. An item waits for the items its `after` names and is skipped when one of them stopped (`afterStanding`). It starts only while fewer than `max_parallel` items are leased or running (`startedItems`, counted once per scan). A failed item cancels nothing; the step fails once every item settled (`itemsSettled`). The rules that decide the items are in `fan-out.ts`, the settling in `engine.ts`. Pull rejects `for_each` with `loop`, `end` or `break` (`loopProblems`). +- **`item` is bound per instance, in the engine and in pull alike.** `contextOf` gives an item its own value and any other instance its workflow's (`Scope.item`), which `childScope` passes into the inline workflow an item runs. A `for_each` step's own `if` and `for_each` therefore read the enclosing item, never their own. Pull checks the same binding: `ReadSite.item` marks the templates of a `for_each` step, and `ReadScope.item` a workflow that runs per item. A template reads `item` only where it is bound: `parseTemplate` takes the binding, which pull decides from the step and the engine from whether the context holds an item, so elsewhere `{{ item }}` stays text in both. A new place that reads `item` must decide its binding in both. +- **Step ids are paths.** A step inside a sub-workflow has an id such as `ship.check.classify`, an item of a `for_each` step one such as `build[2]`, and a step inside the workflow an item runs one such as `build[2].verify`. `parseStepPath` in `step-path.ts` reads them all; don't split ids by hand. A child step exists in the run only after the engine reaches it. To find a step's definition, use `stepDefinitions` in `scope.ts`, which reads the definitions alone, so it also finds the step of an archived pass. `locateStep` in `engine.ts` also gives the step's scope, and needs the instances along the path. Don't look a step up by top-level id. ## Tests diff --git a/docs/guide/workflow-advanced.md b/docs/guide/workflow-advanced.md new file mode 100644 index 000000000..214863c26 --- /dev/null +++ b/docs/guide/workflow-advanced.md @@ -0,0 +1,575 @@ +# Advanced workflows + +> Part of the [TeamAI CLI Usage Guide](../usage-guide.md). + +Each section below builds on [Workflows](workflow.md) and stands on its own. + +## Run ready steps in parallel + +A driver can hold several leases at once. In each `next` call it lists, in `running`, the attempts of its open leases that it still runs. `next` then leases the next ready step instead of handing an open lease back. + +```yaml +schemaVersion: v1alpha1 +name: checks +description: Lint and test at the same time, then build. +steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: build + needs: [lint, test] + command: npm run build +``` + +Each call below sends `"driver":{"kind":"script","machine":"host","workspace":"/work"}` and the fields shown: + +| `next` fields besides `driver` | Answer | +| --- | --- | +| none | `{"data":{"lease":{"run":"ra780d018","step":"lint","attempt":"af143e3a7","revision":1,"pass":1,"command":"npm run lint"},"cancelled":[]}}` | +| `"running":["af143e3a7"]` | `{"data":{"lease":{"run":"ra780d018","step":"test","attempt":"a0ce8107f","revision":1,"pass":1,"command":"npm test"},"cancelled":[]}}` | +| `"running":["af143e3a7","a0ce8107f"]` | `{"data":{"wait":{"reason":"lint is leased"},"cancelled":[]}}` | + +Each call to `next` does this, in order: + +1. It settles the run. If the run has ended, it answers `done`. +2. It hands back an open lease of this driver that `running` leaves out. A restarted driver sends `"running":[]` and gets its open leases back, one per call. +3. It leases the first ready step in file order. The steps of a sub-workflow take the place of the step that runs it. +4. When every ready step is leased, it answers `wait`, and `reason` names the first open lease. + +A driver that sends no `running` gets its open lease back on every call, so it runs one step at a time. + +`"canTakeLease":false` stops after step 2. The answer is `done`, an open lease that `running` leaves out, or `wait`. When nothing is leased, the `wait` reason is `canTakeLease is false`. Use it to learn about an end while the driver has no room for another step. That `wait` does not mean that nothing is left to run, so call `next` again once the driver has room. Step 2 can still hand back an open lease that the driver has no room for, so keep a queue of the leases the driver has not started. + +Every `data` answer of `next` has `cancelled`. It lists the attempts in `running` that are no longer open leases of this driver. Stop those processes. An `error` answer has no `cancelled`. + +A wrong `running` or `canTakeLease` returns `invalid_json` with `running must be an array of attempt ids` or `canTakeLease must be a boolean`. While one driver holds a lease, another driver gets `wait`, and `reason` names that lease. + +`report` checks each report against the revision of its own lease, not the run's latest revision. Above, `next` handed out both leases at revision 1. A report for `test` with `"revision":1` moves the run to revision 2. A report for `lint` with `"revision":1` is still accepted and moves it to 3. A report with another revision for an open lease returns `stale_revision`: + +```text +{"error":{"code":"stale_revision","message":"Revision 2 is stale; attempt ae75cb049 was leased at 1"}} +``` + +### Run local commands in parallel + +`--driver local` runs up to `workflows.max_parallel` command steps at once. Set it in the local config. It defaults to 1, and no flag sets it. + +```yaml +workflows: + max_parallel: 4 +``` + +A prompt step always runs alone. The driver starts it once every command has exited, and starts no command until it finishes. The read-only check compares the whole working tree before and after the prompt, so a command that changes files while the prompt runs would fail it. + +With `max_parallel` above 1: + +- Commands share the working tree. Two commands that write the same file can overwrite each other. +- Each command runs without a terminal, so it cannot read input from you. +- On Ctrl-C, or on a SIGTERM, SIGQUIT, or SIGHUP, teamai stops every command, waits for them to exit, and exits with that signal. + +With 1, the driver runs one step at a time, and a command runs in your terminal, where Ctrl-C reaches it. + +When an end cancels a running command, the driver sends it SIGTERM, and SIGKILL 5 seconds later if it, or with `max_parallel` above 1 a process it started, still runs. Until it exits, it no longer counts toward `max_parallel`. + +## An end cancels the steps still open + +When a workflow reaches an end, its steps that have not settled stop. A pending step is skipped. A leased step, and a step whose workflow is running, become `cancelled`. The same applies to the steps of its sub-workflows. An end inside a sub-workflow stops only that sub-workflow, and the parent goes on. + +```yaml +schemaVersion: v1alpha1 +name: fail-fast +description: Lint and test at the same time, and stop at the first failure. +ends: { broken: failed } +steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: stop + needs: [lint] + if: steps.lint.outcome == 'failed' + end: broken +``` + +A driver runs `lint` and `test` together. `lint` fails, so `stop` reaches `broken`. The next `next` that lists `test` in `running` answers: + +```text +{"data":{"done":{"run":"r2c7f2c70","outcome":"failed","end":"broken"},"cancelled":["acc763f50"]}} +``` + +`report` does not record a report for a cancelled attempt. It returns `lease_cancelled`, whatever driver sends it: + +```text +{"error":{"code":"lease_cancelled","message":"Attempt acc763f50 was cancelled; its report is not recorded"}} +``` + +`steps..outcome` reads `cancelled` for a cancelled step. Nothing in a workflow runs after its end, so only the workflow's `outputs` can read it. Here the parent learns that `lint` failed before the tests finished: + +```yaml +schemaVersion: v1alpha1 +name: gate +description: Report which check stopped a gate. +steps: + - id: checks + workflow: + ends: { broken: failed } + outputs: { test: steps.test.outcome } + steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: stop + needs: [lint] + if: steps.lint.outcome == 'failed' + end: broken + - id: notify + needs: [checks] + if: steps.checks.outputs.test == 'cancelled' + command: echo "lint failed before the tests finished" +``` + +## Run a step once per item + +`for_each` runs a step once per item of a list. It takes an expression that gives the list, or a map with that expression in `items` and a limit in `max_parallel`. Each item runs as an instance of its own, named by the step id and the item's index from 0: `build[0]`, `build[1]`, and so on. + +```yaml +schemaVersion: v1alpha1 +name: fan-build +description: Build each package, two at a time, then explain the failures. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: build + for_each: { items: inputs.packages, max_parallel: 2 } + command: npm run build --prefix "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: explain + needs: [build] + if: any(steps.build.results, .outcome == 'failed') + prompt: "Explain each failed build in these results: {{ steps.build.results }}" +``` + +`build` reads `inputs.packages` once, when it becomes ready. If the value is not a list, the step fails with `invalid_items`. If the list is empty, the step succeeds at once, and its results are `[]`. + +Each instance reads its item as `item` in `prompt`, `env`, and `with`, and in every step of the inline workflow it runs. A `with` value that is only `{{ item }}` keeps the item's JSON type. A `command` is not templated, so pass the item through `env`. The step's own `if` and `for_each` are evaluated once, before any item runs, so they cannot read the step's items. Outside a `for_each`, `{{ item }}` stays as written, and pull rejects any other read of `item`, such as in an `if` or in `{{ filter(inputs.xs, . == item) }}`. + +A driver gets each item as an ordinary lease whose `step` is `build[0]`, `build[1]`, and so on. A driver that lists its open leases in `running` gets the next item while the others run. `max_parallel` limits how many items are leased or running at once. At the limit, `next` hands out no other item until one of them settles. + +A failed item does not stop the other items. The step settles when every item has settled, and it fails if any item failed. `output` checks the outputs of each item. + +`steps..results` lists how each item went, in item order. Each entry has `item`, `outcome`, and `outputs` when the item has outputs: + +```json +[{"item":{"id":"core","dir":"packages/core"},"outcome":"success"},{"item":{"id":"cli","dir":"packages/cli"},"outcome":"failed"}] +``` + +A step with `for_each` has no outputs of its own, and a step without `for_each` has no results. Pull rejects a read of either, including in workflow outputs and earlier passes. To read one item, index the results, as in `steps.build.results[1].outcome`. + +To retry failed items, use `for_each: "filter(steps.build.results, .outcome == 'failed')"` and `if: steps.build.outcome == 'failed'` on a step with `needs: [build]`. The `if` reads the failed dependency, so the retry step can run. Each retry item wraps the original value in `item.item`. + +An item that is an object can carry a string `id` and an `after` list of the ids of other items. The item starts only after each of them succeeded. It is skipped when one of them failed, was skipped, or was cancelled. + +```yaml +schemaVersion: v1alpha1 +name: fan-publish +description: Pack and check each package after the packages it depends on. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: publish + for_each: inputs.packages + workflow: + steps: + - id: pack + command: npm pack "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: verify + needs: [pack] + prompt: "Check that the tarball of {{ item.id }} holds only files from {{ item.dir }}." +``` + +```text +teamai workflow start fan-publish --inputs '{"packages":[{"id":"core","dir":"packages/core"},{"id":"cli","dir":"packages/cli","after":["core"]}]}' +``` + +`publish[1]` starts after `publish[0]` succeeded. The steps of each item's workflow have ids such as `publish[0].pack` and `publish[1].verify`. + +An `after` id that no item has, two items with the same id, or an `after` that is not a list of ids fails the step with `invalid_items` before any item runs. Items that wait for each other in a cycle fail it with `item_cycle`, and the message names the cycle: + +```text +step "publish" has items that run after each other in a cycle: core -> cli -> core +``` + +Pull rejects `for_each` together with `loop`, `end`, or `break`. To run the work of an item again, put the loop in the workflow the item runs. + +## The end reached first wins + +Reports can reach two ends before the driver calls `next` again. `next` then applies the end that was reached first, whatever order the steps have in the file. + +- A step that a driver reports reaches its end at the revision of that report. +- A step with only `id`, `needs`, `if`, and `end` reaches its end when the last step it needs or reads settled. +- On a tie, the step declared first wins. + +```yaml +schemaVersion: v1alpha1 +name: race +description: Try two fixes at once and keep the first that works. +ends: { patched: succeeded, rewritten: succeeded } +steps: + - id: patch + command: ./scripts/patch.sh + end: patched + - id: rewrite + needs: [] + command: ./scripts/rewrite.sh + end: rewritten +``` + +If `rewrite` reports success and then `patch` reports success, the next `next` answers `{"data":{"done":{"run":"ra0ca39d4","outcome":"succeeded","end":"rewritten"},"cancelled":[]}}`. The time at which the driver calls `next` does not change the winner. A report after the winning end returns `lease_cancelled`, even if `next` has not run yet. It does not advance the revision, change workflow outputs, or start another loop pass. Skips and failures that need no driver take the revision that caused them, so they can reach an earlier end too. + +## Loop back to an earlier step + +A step with `loop` runs again after it succeeds, together with the steps before it up to `back_to`. Those steps are the loop's range: `back_to`, the step that has the loop, and every step between them. A step is between them when it needs `back_to` and the looping step needs it, directly or through their needs. + +```yaml +schemaVersion: v1alpha1 +name: loop-review-and-fix +description: Review the branch and fix what the review finds, up to three times. +outputs: + reviews: steps.review.passes +steps: + - id: review + prompt: Review the changes on this branch. + output: + verdict: pass | fix + findings: [string] + - id: fix + needs: [review] + if: steps.review.outputs.verdict == 'fix' + command: ./scripts/fix.sh + env: + FINDINGS: "{{ steps.review.outputs.findings }}" + loop: { back_to: review, max_iterations: 3 } +``` + +`loop` has these fields: + +- `back_to` is the first step of the range. It is the looping step itself or a step it needs, directly or through its needs. +- `max_iterations` is the most times the looping step runs, a whole number from 1. +- `while` is optional. It is an expression, evaluated each time the looping step succeeds. + +When the looping step succeeds and `while` is false, the loop ends. Otherwise the range runs again from `back_to`. The loop also ends when the looping step is skipped or fails. Above, `fix` is skipped once a review passes, so the loop ends and the run succeeds. + +`max_iterations` limits the looping step: + +- With `while`, the step fails `loop_exhausted` when `while` is still true after its last run. +- Without `while`, the range runs once more, and the step fails `loop_exhausted` when its `if` lets it run past the limit. Above, a fourth review runs after the third fix. If it passes, `fix` is skipped and the run succeeds. Otherwise `fix` fails `loop_exhausted`. + +A `loop_exhausted` step keeps the outputs of its last run. + +A step outside the range that needs or reads a step inside it waits until the loop ends. It is not skipped or run before. Here `done` needs only `e2e`, and it still waits for the `fix` loop: + +```yaml +schemaVersion: v1alpha1 +name: loop-implement-task +description: Implement a task, then fix it until the end-to-end tests pass, up to three times. +ends: { done: succeeded, stuck: failed } +steps: + - id: implement + command: ./scripts/implement.sh + - id: e2e + needs: [implement] + command: npm run test:e2e + - id: fix + needs: [e2e] + if: steps.e2e.outcome == 'failed' + command: ./scripts/fix-e2e.sh + loop: { back_to: e2e, max_iterations: 3 } + - id: done + needs: [e2e] + if: steps.e2e.outcome == 'success' + end: done + - id: give-up + needs: [fix] + if: steps.fix.outcome == 'failed' + end: stuck +``` + +With `--driver local`, when `e2e` passes after one fix, the run ends: + +```text +{"data":{"done":{"run":"re4b12bbc","outcome":"succeeded","end":"done"},"cancelled":[]}} +``` + +When `e2e` still fails after three fixes, `fix` fails `loop_exhausted` instead of running a fourth time, and `give-up` reaches `stuck`: + +```text +{"data":{"done":{"run":"r3319969c","outcome":"failed","end":"stuck"},"cancelled":[]}} +``` + +### Read every pass + +Each run of a step is a pass, numbered from 1. A lease carries its `pass`. In a run of `loop-implement-task`, the second lease of `e2e` is: + +```text +{"data":{"lease":{"run":"rd50c3422","step":"e2e","attempt":"a54df2231","revision":4,"pass":2,"command":"npm run test:e2e"},"cancelled":[]}} +``` + +`steps..outcome`, `steps..end`, and `steps..outputs` read the latest pass. `steps..passes` lists every pass of the step, the current one included once it has settled. Each element has `pass` and `outcome`, and `outputs` and `end` when that pass has them. A pass of a `for_each` step has `results` instead of `outputs`, including in a previous-pass read. In `loop-review-and-fix`, the workflow output `reviews` returns every review. + +A step can read a later step of its own range, one that needs it directly or through its needs. That read gets the later step's previous pass, `null` on pass 1, and does not wait for it. In that read, `passes` lists only the earlier passes. Here `plan` reads the note of the last decision: + +```yaml +schemaVersion: v1alpha1 +name: loop-plan-change +description: Plan a change, and plan again while the review asks for a revision. +inputs: { issue: number } +ends: { planned: succeeded, dropped: succeeded } +steps: + - id: plan + prompt: "Plan a fix for issue #{{ inputs.issue }}. Address this note on the last plan, if there is one: {{ steps.decide.outputs.note }}" + output: + summary: string + - id: decide + needs: [plan] + prompt: "Decide whether to build, revise, or drop this plan: {{ steps.plan.outputs.summary }}" + output: + verdict: build | revise | drop + note: string + loop: { back_to: plan, while: "steps.decide.outputs.verdict == 'revise'", max_iterations: 3 } + - id: build + needs: [decide] + if: steps.decide.outputs.verdict == 'build' + end: planned + - id: drop + needs: [decide] + if: steps.decide.outputs.verdict == 'drop' + end: dropped +``` + +On pass 1, `{{ steps.decide.outputs.note }}` is empty text. On pass 2, it is the note that `decide` returned on pass 1. + +### Loops that pull rejects + +`teamai pull` rejects a step loop when: + +- `back_to` is not the looping step or a step it needs, directly or through its needs +- the looping step also has `end` +- its range shares a step with the range of another loop in the same workflow +- its workflow has a `loop` of its own +- a step in its range has `break` +- loops wait for each other in a cycle, through the steps that need or read a step inside a loop + +Then the file has nothing to run, even when an older version was valid. Loops nest only through a sub-workflow, in a file or inline. For example, a second loop on `e2e` in `loop-implement-task` shares `e2e` with the `fix` loop: + +```yaml + - id: e2e + needs: [implement] + command: npm run test:e2e + loop: { back_to: implement, max_iterations: 2 } +``` + +`teamai pull` reports: + +```text +⚠ [workflows] workflows/loop-implement-task.yaml:16: field "steps[2].loop": step "fix" loop runs "e2e" again, which the loop of step "e2e" also runs; loops in one workflow share no step, so put the inner one in a sub-workflow +⚠ [workflows] workflows/loop-implement-task.yaml has no runnable version +``` + +## Run a whole workflow again, and stop it with `break` + +A workflow with `loop` runs all of its steps again, round after round. Its `loop` has `max_iterations`, the most rounds, and an optional `while`. It has no `back_to`. An inline workflow can have `loop` too. + +```yaml +schemaVersion: v1alpha1 +name: loop-wait-for-ci +description: Check CI every minute until it passes or fails, for at most 30 minutes. +ends: { red: failed } +loop: { max_iterations: 30 } +steps: + - id: check + command: ./scripts/ci-state.sh + output: + state: pending | passed | failed + break: steps.check.outputs.state == 'passed' + - id: red + needs: [check] + if: steps.check.outputs.state == 'failed' + end: red + - id: wait + needs: [check] + if: steps.check.outputs.state == 'pending' + command: sleep 60 +``` + +A step with `break` evaluates it when the step succeeds. When it is true, the step's workflow ends as success, and its loop stops. Like an end, it skips the steps that have not run and cancels the open leases. A `break` and an end compete by the order of [The end reached first wins](#the-end-reached-first-wins). A `break` works in a workflow without `loop` too. + +Each round runs until it reaches an end or a `break`, or has nothing left to run: + +- An end stops the loop, and the workflow takes that end's outcome. Above, `red` stops it as `failed`. +- A `break` stops the loop as success. Above, `wait` is skipped in that round. +- Otherwise teamai evaluates `while`. When it is false, the workflow ends with the outcome and outputs of that round. When it is true, or there is no `while`, every step runs again in the next round. +- After `max_iterations` rounds, the workflow fails `loop_exhausted` instead of running another round. + +A failed step alone does not stop the loop. The next round runs, and the workflow ends with the outcome of its last round. To retry a flaky step, write `loop: { while: "steps.e2e.outcome == 'failed'", max_iterations: 3 }`. + +A workflow loop without `while` succeeds only through an end or a `break`. Running out of rounds fails it with `loop_exhausted`. + +Each round is a new pass of every step, so `steps..passes` lists the earlier rounds. A `while` reads only the inputs and the steps of its own workflow. + +Three runs of `loop-wait-for-ci` with `--driver local` end like this: + +| CI states | Answer | +| --- | --- | +| `pending`, `pending`, `passed` | `{"data":{"done":{"run":"re0ac0c1e","outcome":"succeeded"},"cancelled":[]}}` | +| `pending`, `failed` | `{"data":{"done":{"run":"r7d3d9b06","outcome":"failed","end":"red"},"cancelled":[]}}` | +| `pending` 30 times | `{"data":{"done":{"run":"r7ac9a988","outcome":"failed","failure":{"code":"loop_exhausted","message":"Loop of workflow \"loop-wait-for-ci\" is exhausted: it ran 30 rounds, its max_iterations"}},"cancelled":[]}}` | + +`done` carries `failure` when a run fails for a reason that no end names, such as a top-level `loop_exhausted`. A step that runs a looping workflow fails `loop_exhausted` the same way, and keeps the outputs of the last round. + +A workflow with `loop` cannot contain a step with `loop`. Put one of the two loops in a sub-workflow. + +## Expression reference + +An `if` is an expression. The rules for which steps it can read, and for what counts as true, are in [Workflows](workflow.md). + +```yaml +schemaVersion: v1alpha1 +name: review-files +description: Review the changed files, then fix the ones with findings. +steps: + - id: review + prompt: List each changed file and how many findings it has. + output: + files: [{ path: string, findings: number }] + - id: fix + needs: [review] + if: any(steps.review.outputs.files, .findings > 0) + prompt: "Fix the findings in {{ steps.review.outputs.files[0].path }} first." + - id: label + needs: [review] + if: length(filter(steps.review.outputs.files, .findings >= 10)) > 2 + command: gh pr edit --add-label large-change +``` + +### Values + +| Value | Example | +| --- | --- | +| An input | `inputs.issue`, `inputs.issue.labels` | +| A step's outcome, end, or output | `steps.check.outcome`, `steps.check.end`, `steps.review.outputs.files` | +| Every pass of a step | `steps.review.passes`, `steps.review.passes[0].outputs.verdict`. See [Read every pass](#read-every-pass). | +| The results of a `for_each` step | `steps.build.results`, `steps.build.results[0].outcome` | +| The item of a `for_each` instance | `item`, `item.dir` | +| A string | `'failed'`. Use single quotes. Write a quote inside a string twice: `'it''s'`. | +| A number | `3`, `-1`, `1.5`. Write a digit before the decimal point: `0.5`, not `.5`. | +| A boolean or null | `true`, `false`, `null` | +| An empty array | `[]`. It is the only array literal. | + +A missing value reads as `null`. A step that has not settled reads as `null` too. + +### Indexes + +`[n]` reads element `n` of an array, counting from 0: `steps.review.outputs.files[0].path`, `inputs.tags[1]`, or `steps.grid.outputs.rows[1][0]`. An index past the end, or an index into a value that is not an array, reads `null`. + +An index is a whole number from 0. `[-1]`, `[01]`, and `[inputs.n]` do not parse. An index reads an array inside `inputs`, `outputs`, `passes`, `results`, or `item`. It cannot follow a step id, `outcome`, or `end`, so `steps.build[2].outcome` does not parse. + +### Operators and precedence + +From the loosest binding to the tightest: + +| Operators | Result | +| --- | --- | +| `\|\|` | `true` when either side is true | +| `&&` | `true` when both sides are true | +| `!` | `true` when its operand is false | +| `==`, `!=` | whether both sides are the same value, compared strictly | +| `<`, `<=`, `>`, `>=` | the comparison of two numbers | +| a value, a function call, or `( … )` | the value | + +`!` binds looser than `==`. `!steps.check.outcome == 'success'` means `!(steps.check.outcome == 'success')`. + +`==`, `!=`, and the comparisons do not chain. `1 < 2 < 3` and `1 == 1 == true` do not parse. Add parentheses, as in `(1 == 1) == true`. + +A comparison is true only when both sides are numbers. With any other value, such as a string or `null`, it is false. + +### Functions + +| Call | Result | +| --- | --- | +| `length(list)` | the number of elements | +| `any(list, predicate)` | `true` when the predicate is true for at least one element. `false` for an empty list. | +| `all(list, predicate)` | `true` when the predicate is true for every element. `true` for an empty list. | +| `filter(list, predicate)` | an array of the elements for which the predicate is true | + +When `list` is not an array at run time, each function returns `null`. + +A predicate reads the current element with a path that starts with `.`, as in jq: + +- `.` is the element itself. +- `.findings` is a field of the element. +- `.files[0]` and `.[0]` read by index. + +Inside a nested call, `.` is the element of the innermost call. A predicate can also read `inputs` and `steps`. A path that starts with `.` does not parse outside a predicate. + +Any other function name does not parse: `"contains" is not a function; the functions are length, any, all and filter`. A call with the wrong number of arguments does not parse either: `any is called as any(list, predicate)`. + +### Type checks at pull + +`teamai pull` checks the types in each `if`. It knows the type of a literal, of each operator and function result, and of every path the workflow declares in `inputs` or `output`. Each of these makes the file invalid: + +- A comparison with a side that is not a number. Such a side is a string, a boolean, `null`, a condition, an `any`, `all`, or `filter` call, or a path whose declared type is not `number`. +- A `list` argument that is not an array. Such an argument is a literal other than `[]`, a condition, a `length`, `any`, or `all` call, or a path whose declared type is not an array. +- A predicate that reads a field the declared item type does not have. + +For example, if `review` above also declared `title: string`, this `fix` step would make the file invalid: + +```yaml + - id: fix + needs: [review] + if: steps.review.outputs.title > 3 || any(steps.review.outputs.files, .finding > 0) || all(steps.review.outputs.title, .done) + prompt: Fix the findings. +``` + +`teamai pull` reports: + +```text +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if compares steps.review.outputs.title with >, but it is not a number +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if reads .finding, which is not in an item of steps.review.outputs.files +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if passes steps.review.outputs.title to all as its list, but it is not an array +``` + +The outputs of a step that runs a workflow have no declared type, so pull does not check them. At run time, a comparison with a value that is not a number is false, and a function whose `list` is not an array returns `null`. + +### Templates + +A template reads one path, indexes included, such as `{{ steps.review.outputs.files[0].path }}` or `{{ with.refs[1] }}`. It also evaluates an expression as an `if` does, such as `{{ length(steps.build.results) }}` or `{{ steps.build.outcome == 'failed' }}`, when the expression starts with a path, after any `!` or `(`, or calls `length`, `any`, `all` or `filter` on one. An expression does not read `with`. A `with` value that is only one expression keeps its JSON type. `teamai pull` checks the expression as it checks an `if`, and says what to fix when it has a mistake in it, a misspelled function on a step such as `{{ lenght(steps.build.results) }}` included. Other text, such as `{{ filter(items) }}` in a Vue sample, stays as written. + +To put a count or a filtered list into a prompt, evaluate it in the template: + +```yaml +schemaVersion: v1alpha1 +name: fan-build-failures +description: Build each package, then look for what the failed builds share. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: build + for_each: inputs.packages + command: npm run build --prefix "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: explain + needs: [build] + if: steps.build.outcome == 'failed' + prompt: "{{ length(filter(steps.build.results, .outcome == 'failed')) }} packages failed to build: {{ filter(steps.build.results, .outcome == 'failed') }}. Find what they share." +``` + +Pass the whole list, as `explain` does in `fan-build`, when the prompt needs every entry. diff --git a/docs/guide/workflow.md b/docs/guide/workflow.md index e20779a0c..9b0581d49 100644 --- a/docs/guide/workflow.md +++ b/docs/guide/workflow.md @@ -2,7 +2,7 @@ > Part of the [TeamAI CLI Usage Guide](../usage-guide.md). -A workflow is a YAML file at `workflows/.yaml` in the team repo. It runs command and read-only prompt steps, each after the steps it needs. +A workflow is a YAML file at `workflows/.yaml` in the team repo. It runs command and read-only prompt steps, each after the steps it needs. [Advanced workflows](workflow-advanced.md) covers parallel steps, loops, `for_each`, what an end does to steps that still run, and the full expression language. ```yaml schemaVersion: v1alpha1 @@ -24,9 +24,9 @@ steps: `schemaVersion` must be `v1alpha1`. `name` must match the file name. `description` is required. Each step has a unique `id` and exactly one of `command`, `prompt`, `prompt_file`, or `workflow`. `agent` is optional on prompt steps. `output` uses the existing shorthand: `string`, `number`, `boolean`, a literal union, an object, or an array with one item type. TeamAI validates successful command stdout or a prompt tool's `output` against it. -Every step after the first lists the steps it waits for in `needs`. `needs: []` starts another branch with no dependency. One step runs at a time: the first step in the file whose needs are done. A step is skipped when a step it needs failed, or when every step it needs was skipped. A skipped step never runs. A failure does not stop other branches. When nothing is left to run, the run ends `failed` if any step failed, otherwise `succeeded`. A missing `needs`, a `needs` id that is not a step, or a cycle makes the file invalid. A missing `needs` also leaves the workflow with nothing to run, even when an older version was valid. +Every step after the first lists the steps it waits for in `needs`. `needs: []` starts another branch with no dependency. Each `next` leases the first step in the file whose needs are done. `--driver local` runs one step at a time unless `workflows.max_parallel` is above 1. A step is skipped when a step it needs failed, or when every step it needs was skipped. A skipped step never runs. A failure does not stop other branches. When nothing is left to run, the run ends `failed` if any step failed, otherwise `succeeded`. A missing `needs`, a `needs` id that is not a step, or a cycle makes the file invalid. A missing `needs` also leaves the workflow with nothing to run, even when an older version was valid. -A step can carry an `if` expression. When it is false, the step is skipped, and that does not fail the run. An expression reads `steps..outcome` (`success`, `failed`, or `skipped`) and `steps..outputs.`, only for steps it needs, directly or through their needs. It has `==`, `!=`, `&&`, `||`, `!`, parentheses, and the literals `'text'`, numbers, `true`, `false`, `null`, and `[]`. A missing value reads as `null`. `false`, `null`, `0`, and `''` are false; everything else is true. `==` is strict and compares whole values: `'1' == 1` and `'Build' == 'build'` are false. +A step can carry an `if` expression. When it is false, the step is skipped, and that does not fail the run. An expression reads `steps..outcome` (`success`, `failed`, or `skipped`) and `steps..outputs.`, only for steps it needs, directly or through their needs. It has `==`, `!=`, `<`, `<=`, `>`, `>=`, `&&`, `||`, `!`, parentheses, and the literals `'text'`, numbers, `true`, `false`, `null`, and `[]`. A path can index an array, as in `steps.review.outputs.files[0]`. The functions `length`, `any`, `all`, and `filter` read arrays. A missing value reads as `null`. `false`, `null`, `0`, and `''` are false; everything else is true. `==` is strict and compares whole values: `'1' == 1` and `'Build' == 'build'` are false. A step whose dependency failed still runs when its `if` reads every failed dependency. Use this to recover: @@ -45,7 +45,7 @@ A workflow can declare named `ends`, each `succeeded` or `failed`, such as `ends A workflow can declare `inputs` with the same shorthand as `output`, such as `inputs: { issue: number, title: string }`. Every input is required. Pass them as a JSON object: `teamai workflow start fix --driver local --inputs '{"issue":42,"title":"Crash"}'`. A missing, extra, or mistyped input returns `invalid_inputs` naming the field, and no run is created. An `if` can read `inputs.`. -`{{ inputs. }}` and `{{ steps..outputs. }}` (or `.outcome`) fill in a `prompt`, the text of a `prompt_file`, and `env` values when the step starts. Other `{{ ... }}` text, such as `${{ secrets.TOKEN }}`, stays as written. So does any `{{` right after a `$`, such as `${{ inputs.version }}` in a prompt about GitHub Actions. To put a value right after a `$`, add a space: `$ {{ inputs.price }}`. Strings appear as they are, numbers and booleans as text, objects and arrays as JSON, and `null` or a missing value as empty text. A template reads only what an `if` on that step could read. +`{{ inputs. }}` and `{{ steps..outputs. }}` (or `.outcome`) fill in a `prompt`, the text of a `prompt_file`, and `env` values when the step starts. A template can also evaluate an expression as an `if` does, such as `{{ length(steps.review.outputs.files) }}`. Other `{{ ... }}` text, such as `${{ secrets.TOKEN }}`, stays as written. So does any `{{` right after a `$`, such as `${{ inputs.version }}` in a prompt about GitHub Actions. To put a value right after a `$`, add a space: `$ {{ inputs.price }}`. Strings appear as they are, numbers and booleans as text, objects and arrays as JSON, and `null` or a missing value as empty text. A template reads only what an `if` on that step could read. A `command` is never filled in. Pass values to it through `env`, which only a command step can have: @@ -112,7 +112,7 @@ An agent-specific entry wins over `default`. `claude`, `codex`, and `opencode` a teamai workflow start build --driver local ``` -`--driver local` runs one step at a time in the current directory. Each attempt uses `result.json` under the scope's TeamAI data directory, outside the checkout. Prompt steps are read-only: if a tool changes the repository, the step fails and leaves those files in place for inspection. This includes changes to files inside an existing untracked directory; leaving that directory unchanged does not fail the step. The driver calls the same `next` and `report` operations shown below: +`--driver local` runs steps in the current directory, one at a time unless [`workflows.max_parallel`](workflow-advanced.md#run-local-commands-in-parallel) is above 1. Each attempt uses `result.json` under the scope's TeamAI data directory, outside the checkout. Prompt steps are read-only: if a tool changes the repository, the step fails and leaves those files in place for inspection. This includes changes to files inside an existing untracked directory; leaving that directory unchanged does not fail the step. The driver calls the same `next` and `report` operations shown below: ```text teamai workflow start build @@ -128,7 +128,7 @@ The run is stored in this scope's TeamAI data directory. The command exits when With [usage reporting](reporting.md) on, every run publishes a privacy-preserving record to the team's `teamai-reports` branch: once when it starts and once when it ends. With it off, nothing is published and the envelopes carry no `record`. Records live in `workflows/runs/.json`, keyed by run id, so a retried publication updates the same record instead of adding a copy. -A record carries only: the run id, workflow name, phase (`start` or `end`), driver kind, status, the end the run reached (when it reached one), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens — a mixed-agent run records each step's own fields), summary step counts (succeeded, failed, skipped), the cumulative settled-attempt count, and the publishing username. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state. +A record carries only: the run id, workflow name, phase (`start` or `end`), driver kind, status, the end the run reached (when it reached one), the failure code of a run that failed without reaching an end, when it is a code teamai sets, such as `loop_exhausted` for a loop that ran out or `tool_unavailable` when the local driver cannot find the tool (never its message), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens — a mixed-agent run records each step's own fields; `tool` is kept only when it is one of the local driver's tools, and `model` is the name the driver reported, kept only when it is shaped like a model name), summary step counts (succeeded, failed, skipped, cancelled), the cumulative settled-attempt count, and the publishing username. Every entry has its `pass` number, and a step that a loop ran more than once has one entry per pass. An item of a `for_each` counts the times that item ran, so its `pass` can be lower than its step's when an earlier pass had fewer items. A cancelled step's entry has the outcome `cancelled`, and a skipped step's has none. The counts cover each step once, as the run ended. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state. The end record is published by the `next` that settles the run. A publication that fails — no remote, a busy reports lock, a rejected push — is queued in this scope's data directory and retried on the next `teamai pull`. The `start` and `next` envelopes then carry `record: pending` instead of `record: published`. The run itself is unaffected: a machine that dies after starting a run leaves it visible as started but not ended. diff --git a/docs/guide/zh-CN/workflow-advanced.md b/docs/guide/zh-CN/workflow-advanced.md new file mode 100644 index 000000000..95adba68f --- /dev/null +++ b/docs/guide/zh-CN/workflow-advanced.md @@ -0,0 +1,575 @@ +# 高级工作流 + +> 本文是 [TeamAI CLI 使用指南](../../usage-guide.zh-CN.md) 的一部分。 + +下面每一节都建立在 [工作流](workflow.md) 之上,各节相互独立。 + +## 并行运行就绪的步骤 + +一个驱动可以同时持有多个租约。每次调用 `next` 时,它在 `running` 里列出自己仍在运行的未结租约的 attempt。这样 `next` 会租出下一个就绪的步骤,而不是把未结租约交还给它。 + +```yaml +schemaVersion: v1alpha1 +name: checks +description: Lint and test at the same time, then build. +steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: build + needs: [lint, test] + command: npm run build +``` + +下面每次调用都发送 `"driver":{"kind":"script","machine":"host","workspace":"/work"}` 和表中的字段: + +| `driver` 之外的 `next` 字段 | 返回 | +| --- | --- | +| 无 | `{"data":{"lease":{"run":"ra780d018","step":"lint","attempt":"af143e3a7","revision":1,"pass":1,"command":"npm run lint"},"cancelled":[]}}` | +| `"running":["af143e3a7"]` | `{"data":{"lease":{"run":"ra780d018","step":"test","attempt":"a0ce8107f","revision":1,"pass":1,"command":"npm test"},"cancelled":[]}}` | +| `"running":["af143e3a7","a0ce8107f"]` | `{"data":{"wait":{"reason":"lint is leased"},"cancelled":[]}}` | + +每次调用 `next` 都按顺序执行以下几步: + +1. 结算运行。如果运行已经结束,返回 `done`。 +2. 把本驱动的、`running` 里没有列出的未结租约交还给它。重启后的驱动发送 `"running":[]`,每次调用拿回一个未结租约。 +3. 按文件顺序租出第一个就绪的步骤。子工作流的步骤位于运行它的那个步骤的位置。 +4. 每个就绪的步骤都已租出时,返回 `wait`,`reason` 指出第一个未结租约。 + +不发送 `running` 的驱动每次调用都会拿回自己的未结租约,因此一次只运行一个步骤。 + +`"canTakeLease":false` 执行到第 2 步就停止。返回的是 `done`、一个 `running` 没有列出的未结租约,或 `wait`。没有任何租约时,`wait` 的 reason 是 `canTakeLease is false`。驱动没有空位运行新步骤时,可以用它来得知是否已到达结局。这个 `wait` 并不表示没有剩余的步骤可运行,所以驱动有空位后要再次调用 `next`。第 2 步仍可能交还一个驱动没有空位运行的未结租约,所以要为驱动尚未启动的租约维护一个队列。 + +`next` 的每个 `data` 返回都带 `cancelled`。它列出 `running` 中已经不再是本驱动未结租约的 attempt。请停止这些进程。`error` 返回不带 `cancelled`。 + +`running` 或 `canTakeLease` 格式不对时,返回 `invalid_json`,消息为 `running must be an array of attempt ids` 或 `canTakeLease must be a boolean`。一个驱动持有租约时,其他驱动会收到 `wait`,`reason` 指出那个租约。 + +`report` 按每次上报自己租约的 revision 检查,而不是按运行最新的 revision。上例中 `next` 在 revision 1 租出了两个租约。`test` 以 `"revision":1` 上报后,运行变为 revision 2。`lint` 以 `"revision":1` 上报仍会被接受,运行变为 revision 3。未结租约以其他 revision 上报时返回 `stale_revision`: + +```text +{"error":{"code":"stale_revision","message":"Revision 2 is stale; attempt ae75cb049 was leased at 1"}} +``` + +### 并行运行本地命令 + +`--driver local` 最多同时运行 `workflows.max_parallel` 个命令步骤。在本地配置中设置它。默认值是 1,没有对应的命令行参数。 + +```yaml +workflows: + max_parallel: 4 +``` + +提示词步骤总是单独运行。驱动等所有命令都退出后才启动它,它结束前不会启动任何命令。只读检查会比较提示词运行前后的整个工作树,所以提示词运行期间修改文件的命令会让它失败。 + +`max_parallel` 大于 1 时: + +- 命令共用工作树。两个命令写同一个文件时可能互相覆盖。 +- 每个命令都在没有终端的情况下运行,因此无法读取你的输入。 +- 收到 Ctrl-C,或 SIGTERM、SIGQUIT、SIGHUP 时,teamai 停止所有命令,等它们退出,然后以该信号退出。 + +为 1 时,驱动一次运行一个步骤,命令在你的终端里运行,Ctrl-C 会直接到达它。 + +结局取消一个正在运行的命令时,驱动向它发送 SIGTERM,5 秒后如果它仍在运行(`max_parallel` 大于 1 时也包括它启动的进程),再发送 SIGKILL。它在退出前已不计入 `max_parallel`。 + +## 到达结局时取消仍未结束的步骤 + +工作流到达结局时,它尚未完成的步骤都会停止。pending 的步骤被跳过。已租出的步骤,以及其工作流正在运行的步骤,变为 `cancelled`。它的子工作流中的步骤也一样。子工作流里的结局只停止该子工作流,父工作流继续运行。 + +```yaml +schemaVersion: v1alpha1 +name: fail-fast +description: Lint and test at the same time, and stop at the first failure. +ends: { broken: failed } +steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: stop + needs: [lint] + if: steps.lint.outcome == 'failed' + end: broken +``` + +驱动同时运行 `lint` 和 `test`。`lint` 失败,于是 `stop` 到达 `broken`。之后在 `running` 中列出 `test` 的那次 `next` 返回: + +```text +{"data":{"done":{"run":"r2c7f2c70","outcome":"failed","end":"broken"},"cancelled":["acc763f50"]}} +``` + +`report` 不记录已取消 attempt 的上报。无论哪个驱动上报,都返回 `lease_cancelled`: + +```text +{"error":{"code":"lease_cancelled","message":"Attempt acc763f50 was cancelled; its report is not recorded"}} +``` + +被取消的步骤,`steps..outcome` 读作 `cancelled`。工作流到达结局后不再运行任何步骤,所以只有该工作流的 `outputs` 能读到它。下例中,父工作流得知 `lint` 在测试完成前就失败了: + +```yaml +schemaVersion: v1alpha1 +name: gate +description: Report which check stopped a gate. +steps: + - id: checks + workflow: + ends: { broken: failed } + outputs: { test: steps.test.outcome } + steps: + - id: lint + command: npm run lint + - id: test + needs: [] + command: npm test + - id: stop + needs: [lint] + if: steps.lint.outcome == 'failed' + end: broken + - id: notify + needs: [checks] + if: steps.checks.outputs.test == 'cancelled' + command: echo "lint failed before the tests finished" +``` + +## 对每一项运行一次步骤 + +`for_each` 让一个步骤对列表中的每一项各运行一次。它的值是给出列表的表达式,或一个映射,`items` 里写这个表达式,`max_parallel` 里写上限。每一项作为单独的实例运行,以步骤 id 和该项从 0 开始的下标命名:`build[0]`、`build[1]`,依此类推。 + +```yaml +schemaVersion: v1alpha1 +name: fan-build +description: Build each package, two at a time, then explain the failures. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: build + for_each: { items: inputs.packages, max_parallel: 2 } + command: npm run build --prefix "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: explain + needs: [build] + if: any(steps.build.results, .outcome == 'failed') + prompt: "Explain each failed build in these results: {{ steps.build.results }}" +``` + +`build` 在就绪时读取一次 `inputs.packages`。值不是列表时,该步骤以 `invalid_items` 失败。列表为空时,该步骤立即成功,其 results 为 `[]`。 + +每个实例在 `prompt`、`env` 和 `with` 中,以及它运行的内联工作流的每个步骤中,以 `item` 读取自己的那一项。只有 `{{ item }}` 的 `with` 值保留该项的 JSON 类型。`command` 不会被填充,所以请通过 `env` 传入该项。步骤自己的 `if` 和 `for_each` 在任何一项运行之前求值一次,因此读不到该步骤的项。在 `for_each` 之外,`{{ item }}` 保持原样,其他读取 `item` 的地方都会被 pull 拒绝,例如 `if` 中或 `{{ filter(inputs.xs, . == item) }}` 中。 + +驱动拿到的每一项都是普通租约,其 `step` 为 `build[0]`、`build[1]` 等。在 `running` 中列出未结租约的驱动,会在其他项运行时拿到下一项。`max_parallel` 限制同时处于已租出或运行中的项数。达到上限时,`next` 不再租出其他项,直到其中一项完成。 + +某一项失败不会停止其他项。所有项都完成后,该步骤才完成,任一项失败时该步骤失败。`output` 检查每一项的输出。 + +`steps..results` 按项的顺序列出每一项的结果。每个条目有 `item`、`outcome`,该项有输出时还有 `outputs`: + +```json +[{"item":{"id":"core","dir":"packages/core"},"outcome":"success"},{"item":{"id":"cli","dir":"packages/cli"},"outcome":"failed"}] +``` + +带 `for_each` 的步骤没有自己的输出,不带 `for_each` 的步骤没有 results。读取其中任何一个,pull 都会拒绝,包括在工作流 outputs 和之前的 pass 中的读取。要读取某一项,请对 results 使用下标,例如 `steps.build.results[1].outcome`。 + +要重试失败的项,在带 `needs: [build]` 的步骤上同时设置 `for_each: "filter(steps.build.results, .outcome == 'failed')"` 和 `if: steps.build.outcome == 'failed'`。`if` 读取失败的依赖,所以重试步骤可以运行。每个重试项将原始值保存在 `item.item` 中。 + +是对象的项可以带一个字符串 `id`,以及一个 `after` 列表,列出其他项的 id。该项要等这些项都成功后才开始。其中任一项失败、被跳过或被取消时,该项被跳过。 + +```yaml +schemaVersion: v1alpha1 +name: fan-publish +description: Pack and check each package after the packages it depends on. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: publish + for_each: inputs.packages + workflow: + steps: + - id: pack + command: npm pack "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: verify + needs: [pack] + prompt: "Check that the tarball of {{ item.id }} holds only files from {{ item.dir }}." +``` + +```text +teamai workflow start fan-publish --inputs '{"packages":[{"id":"core","dir":"packages/core"},{"id":"cli","dir":"packages/cli","after":["core"]}]}' +``` + +`publish[1]` 在 `publish[0]` 成功后开始。每一项的工作流中的步骤,id 形如 `publish[0].pack` 和 `publish[1].verify`。 + +`after` 中的 id 没有对应的项、两项 id 相同,或 `after` 不是 id 列表时,该步骤在任何一项运行之前以 `invalid_items` 失败。项之间循环等待时,该步骤以 `item_cycle` 失败,消息会指出这个循环: + +```text +step "publish" has items that run after each other in a cycle: core -> cli -> core +``` + +`for_each` 与 `loop`、`end` 或 `break` 一起使用时,pull 会拒绝。要再次运行某一项的工作,请把循环放进该项运行的工作流里。 + +## 先到达的结局生效 + +驱动再次调用 `next` 之前,多个上报可能到达两个结局。此时 `next` 采用先到达的那个结局,与步骤在文件中的顺序无关。 + +- 驱动上报的步骤,在该次上报产生的 revision 到达它的结局。 +- 只有 `id`、`needs`、`if` 和 `end` 的步骤,在它依赖或读取的最后一个步骤完成时到达它的结局。 +- 同时到达时,先声明的步骤生效。 + +```yaml +schemaVersion: v1alpha1 +name: race +description: Try two fixes at once and keep the first that works. +ends: { patched: succeeded, rewritten: succeeded } +steps: + - id: patch + command: ./scripts/patch.sh + end: patched + - id: rewrite + needs: [] + command: ./scripts/rewrite.sh + end: rewritten +``` + +如果 `rewrite` 先上报成功,`patch` 随后上报成功,下一次 `next` 返回 `{"data":{"done":{"run":"ra0ca39d4","outcome":"succeeded","end":"rewritten"},"cancelled":[]}}`。驱动何时调用 `next` 不会改变生效的结局。在胜出的结局之后才上报时,即使还没调用 `next`,也会返回 `lease_cancelled`。这次上报不会增加修订号、改变工作流的输出,也不会开始新的循环 pass。无需驱动即可发生的跳过和失败,使用导致该转换的修订号,因此也可能先到达另一个结局。 + +## 回到前面的步骤再运行 + +带 `loop` 的步骤成功后会再次运行,连同它前面直到 `back_to` 的步骤一起。这些步骤是循环的范围:`back_to`、带循环的步骤,以及两者之间的每个步骤。一个步骤依赖 `back_to`,而带循环的步骤又依赖它,无论直接依赖还是通过依赖的依赖,它就位于两者之间。 + +```yaml +schemaVersion: v1alpha1 +name: loop-review-and-fix +description: Review the branch and fix what the review finds, up to three times. +outputs: + reviews: steps.review.passes +steps: + - id: review + prompt: Review the changes on this branch. + output: + verdict: pass | fix + findings: [string] + - id: fix + needs: [review] + if: steps.review.outputs.verdict == 'fix' + command: ./scripts/fix.sh + env: + FINDINGS: "{{ steps.review.outputs.findings }}" + loop: { back_to: review, max_iterations: 3 } +``` + +`loop` 有以下字段: + +- `back_to` 是范围的第一个步骤。它是带循环的步骤本身,或该步骤直接或间接依赖的步骤。 +- `max_iterations` 是带循环的步骤最多运行的次数,是从 1 开始的整数。 +- `while` 可选。它是一个表达式,带循环的步骤每次成功后求值。 + +带循环的步骤成功且 `while` 为假时,循环结束。否则范围从 `back_to` 起再次运行。带循环的步骤被跳过或失败时,循环也会结束。上例中,审查通过后 `fix` 被跳过,于是循环结束,运行成功。 + +`max_iterations` 限制带循环的步骤: + +- 有 `while` 时,如果最后一次运行后 `while` 仍为真,该步骤以 `loop_exhausted` 失败。 +- 没有 `while` 时,范围会再运行一次。如果该步骤的 `if` 让它超出上限运行,它以 `loop_exhausted` 失败。上例中,第三次修复之后会运行第四次审查。审查通过时 `fix` 被跳过,运行成功。否则 `fix` 以 `loop_exhausted` 失败。 + +以 `loop_exhausted` 失败的步骤保留它最后一次运行的输出。 + +范围外依赖或读取范围内步骤的步骤,会等到循环结束。在此之前它既不会被跳过,也不会运行。这里 `done` 只依赖 `e2e`,但仍要等 `fix` 循环: + +```yaml +schemaVersion: v1alpha1 +name: loop-implement-task +description: Implement a task, then fix it until the end-to-end tests pass, up to three times. +ends: { done: succeeded, stuck: failed } +steps: + - id: implement + command: ./scripts/implement.sh + - id: e2e + needs: [implement] + command: npm run test:e2e + - id: fix + needs: [e2e] + if: steps.e2e.outcome == 'failed' + command: ./scripts/fix-e2e.sh + loop: { back_to: e2e, max_iterations: 3 } + - id: done + needs: [e2e] + if: steps.e2e.outcome == 'success' + end: done + - id: give-up + needs: [fix] + if: steps.fix.outcome == 'failed' + end: stuck +``` + +使用 `--driver local` 时,如果 `e2e` 在一次修复后通过,运行结束: + +```text +{"data":{"done":{"run":"re4b12bbc","outcome":"succeeded","end":"done"},"cancelled":[]}} +``` + +如果三次修复后 `e2e` 仍然失败,`fix` 不会运行第四次,而是以 `loop_exhausted` 失败,`give-up` 到达 `stuck`: + +```text +{"data":{"done":{"run":"r3319969c","outcome":"failed","end":"stuck"},"cancelled":[]}} +``` + +### 读取每一个 pass + +步骤的每次运行是一个 pass,从 1 开始编号。租约带有它的 `pass`。在 `loop-implement-task` 的一次运行中,`e2e` 的第二个租约是: + +```text +{"data":{"lease":{"run":"rd50c3422","step":"e2e","attempt":"a54df2231","revision":4,"pass":2,"command":"npm run test:e2e"},"cancelled":[]}} +``` + +`steps..outcome`、`steps..end` 和 `steps..outputs` 读取最新的 pass。`steps..passes` 列出该步骤的每个 pass,当前 pass 完成后也包括在内。每个元素都有 `pass` 和 `outcome`,该 pass 有 `outputs` 和 `end` 时也带上它们。`for_each` 步骤的 pass 带 `results`,没有 `outputs`,读取上一个 pass 时也如此。在 `loop-review-and-fix` 中,工作流输出 `reviews` 返回每一次审查。 + +步骤可以读取它所在范围内后面的步骤,即直接或间接依赖它的步骤。这种读取得到后面那个步骤的上一个 pass,在第 1 个 pass 上为 `null`,并且不会等待它。在这种读取中,`passes` 只列出之前的 pass。这里 `plan` 读取上一次决定的备注: + +```yaml +schemaVersion: v1alpha1 +name: loop-plan-change +description: Plan a change, and plan again while the review asks for a revision. +inputs: { issue: number } +ends: { planned: succeeded, dropped: succeeded } +steps: + - id: plan + prompt: "Plan a fix for issue #{{ inputs.issue }}. Address this note on the last plan, if there is one: {{ steps.decide.outputs.note }}" + output: + summary: string + - id: decide + needs: [plan] + prompt: "Decide whether to build, revise, or drop this plan: {{ steps.plan.outputs.summary }}" + output: + verdict: build | revise | drop + note: string + loop: { back_to: plan, while: "steps.decide.outputs.verdict == 'revise'", max_iterations: 3 } + - id: build + needs: [decide] + if: steps.decide.outputs.verdict == 'build' + end: planned + - id: drop + needs: [decide] + if: steps.decide.outputs.verdict == 'drop' + end: dropped +``` + +在第 1 个 pass 上,`{{ steps.decide.outputs.note }}` 是空文本。在第 2 个 pass 上,它是 `decide` 在第 1 个 pass 返回的备注。 + +### pull 拒绝的循环 + +以下情况下,`teamai pull` 拒绝步骤循环: + +- `back_to` 既不是带循环的步骤,也不是它直接或间接依赖的步骤 +- 带循环的步骤还有 `end` +- 它的范围与同一工作流中另一个循环的范围有共同的步骤 +- 它所在的工作流自身有 `loop` +- 它范围内的某个步骤有 `break` +- 多个循环通过依赖或读取循环内步骤的步骤互相等待,形成环 + +此时该文件没有可运行版本,即使之前有有效版本也一样。循环只能通过子工作流嵌套,子工作流可以是文件,也可以内联。例如,在 `loop-implement-task` 中给 `e2e` 再加一个循环,它与 `fix` 循环共用 `e2e`: + +```yaml + - id: e2e + needs: [implement] + command: npm run test:e2e + loop: { back_to: implement, max_iterations: 2 } +``` + +`teamai pull` 报告: + +```text +⚠ [workflows] workflows/loop-implement-task.yaml:16: field "steps[2].loop": step "fix" loop runs "e2e" again, which the loop of step "e2e" also runs; loops in one workflow share no step, so put the inner one in a sub-workflow +⚠ [workflows] workflows/loop-implement-task.yaml has no runnable version +``` + +## 整个工作流再次运行,并用 `break` 停止 + +带 `loop` 的工作流一轮接一轮地重新运行它的所有步骤。它的 `loop` 有 `max_iterations`(最多运行的轮数)和可选的 `while`。它没有 `back_to`。内联工作流也可以有 `loop`。 + +```yaml +schemaVersion: v1alpha1 +name: loop-wait-for-ci +description: Check CI every minute until it passes or fails, for at most 30 minutes. +ends: { red: failed } +loop: { max_iterations: 30 } +steps: + - id: check + command: ./scripts/ci-state.sh + output: + state: pending | passed | failed + break: steps.check.outputs.state == 'passed' + - id: red + needs: [check] + if: steps.check.outputs.state == 'failed' + end: red + - id: wait + needs: [check] + if: steps.check.outputs.state == 'pending' + command: sleep 60 +``` + +带 `break` 的步骤成功时求值 `break`。值为真时,该步骤所在的工作流以成功结束,它的循环也停止。和结局一样,它会跳过尚未运行的步骤,并取消未结租约。`break` 和结局按 [先到达的结局生效](#先到达的结局生效) 的顺序竞争。没有 `loop` 的工作流也可以用 `break`。 + +每一轮运行到它到达结局或 `break`,或没有可运行的步骤为止: + +- 结局停止循环,工作流取该结局的 outcome。上例中,`red` 让它以 `failed` 停止。 +- `break` 让循环以成功停止。上例中,那一轮的 `wait` 被跳过。 +- 否则 teamai 求值 `while`。为假时,工作流以这一轮的 outcome 和输出结束。为真,或没有 `while` 时,每个步骤都在下一轮再次运行。 +- 运行 `max_iterations` 轮之后,工作流以 `loop_exhausted` 失败,不再运行下一轮。 + +仅有步骤失败不会停止循环。下一轮照常运行,工作流以最后一轮的 outcome 结束。要重试一个不稳定的步骤,可以写 `loop: { while: "steps.e2e.outcome == 'failed'", max_iterations: 3 }`。 + +没有 `while` 的工作流循环只能通过结局或 `break` 成功。轮数用完时,它以 `loop_exhausted` 失败。 + +每一轮都是每个步骤的一个新 pass,所以 `steps..passes` 列出之前的各轮。`while` 只读取它所在工作流的输入和步骤。 + +`loop-wait-for-ci` 用 `--driver local` 运行三次,结果如下: + +| CI 状态 | 返回 | +| --- | --- | +| `pending`、`pending`、`passed` | `{"data":{"done":{"run":"re0ac0c1e","outcome":"succeeded"},"cancelled":[]}}` | +| `pending`、`failed` | `{"data":{"done":{"run":"r7d3d9b06","outcome":"failed","end":"red"},"cancelled":[]}}` | +| `pending` 30 次 | `{"data":{"done":{"run":"r7ac9a988","outcome":"failed","failure":{"code":"loop_exhausted","message":"Loop of workflow \"loop-wait-for-ci\" is exhausted: it ran 30 rounds, its max_iterations"}},"cancelled":[]}}` | + +运行失败的原因不是某个结局时,`done` 带有 `failure`,例如顶层的 `loop_exhausted`。运行循环工作流的步骤也以同样的方式以 `loop_exhausted` 失败,并保留最后一轮的输出。 + +带 `loop` 的工作流不能包含带 `loop` 的步骤。请把其中一个循环放进子工作流。 + +## 表达式参考 + +`if` 是一个表达式。它能读取哪些步骤、哪些值为真,见 [工作流](workflow.md)。 + +```yaml +schemaVersion: v1alpha1 +name: review-files +description: Review the changed files, then fix the ones with findings. +steps: + - id: review + prompt: List each changed file and how many findings it has. + output: + files: [{ path: string, findings: number }] + - id: fix + needs: [review] + if: any(steps.review.outputs.files, .findings > 0) + prompt: "Fix the findings in {{ steps.review.outputs.files[0].path }} first." + - id: label + needs: [review] + if: length(filter(steps.review.outputs.files, .findings >= 10)) > 2 + command: gh pr edit --add-label large-change +``` + +### 值 + +| 值 | 示例 | +| --- | --- | +| 输入 | `inputs.issue`、`inputs.issue.labels` | +| 步骤的 outcome、结局或输出 | `steps.check.outcome`、`steps.check.end`、`steps.review.outputs.files` | +| 步骤的每个 pass | `steps.review.passes`、`steps.review.passes[0].outputs.verdict`。见[读取每一个 pass](#读取每一个-pass)。 | +| `for_each` 步骤的 results | `steps.build.results`、`steps.build.results[0].outcome` | +| `for_each` 实例的项 | `item`、`item.dir` | +| 字符串 | `'failed'`。使用单引号。字符串里的单引号写两次:`'it''s'`。 | +| 数字 | `3`、`-1`、`1.5`。小数点前要有数字:写 `0.5`,不写 `.5`。 | +| 布尔值或 null | `true`、`false`、`null` | +| 空数组 | `[]`。这是唯一的数组字面量。 | + +不存在的值读作 `null`。尚未完成的步骤也读作 `null`。 + +### 下标 + +`[n]` 读取数组的第 `n` 个元素,从 0 开始计数:`steps.review.outputs.files[0].path`、`inputs.tags[1]` 或 `steps.grid.outputs.rows[1][0]`。超出末尾的下标,或对非数组值使用的下标,读作 `null`。 + +下标是从 0 开始的整数。`[-1]`、`[01]` 和 `[inputs.n]` 无法解析。下标读取 `inputs`、`outputs`、`passes`、`results` 或 `item` 里的数组。它不能跟在步骤 id、`outcome` 或 `end` 后面,所以 `steps.build[2].outcome` 无法解析。 + +### 运算符和优先级 + +从结合最松到最紧: + +| 运算符 | 结果 | +| --- | --- | +| `\|\|` | 任一侧为真时为 `true` | +| `&&` | 两侧都为真时为 `true` | +| `!` | 操作数为假时为 `true` | +| `==`、`!=` | 两侧是否为同一个值,严格比较 | +| `<`、`<=`、`>`、`>=` | 比较两个数字 | +| 值、函数调用或 `( … )` | 该值 | + +`!` 的结合比 `==` 松。`!steps.check.outcome == 'success'` 表示 `!(steps.check.outcome == 'success')`。 + +`==`、`!=` 和比较运算符不能连写。`1 < 2 < 3` 和 `1 == 1 == true` 无法解析。请加括号,例如 `(1 == 1) == true`。 + +只有两侧都是数字时,比较才可能为真。任一侧是其他值,例如字符串或 `null`,结果都为假。 + +### 函数 + +| 调用 | 结果 | +| --- | --- | +| `length(list)` | 元素个数 | +| `any(list, predicate)` | 至少一个元素使 predicate 为真时为 `true`。空数组为 `false`。 | +| `all(list, predicate)` | 每个元素都使 predicate 为真时为 `true`。空数组为 `true`。 | +| `filter(list, predicate)` | 使 predicate 为真的元素组成的数组 | + +运行时 `list` 不是数组时,每个函数都返回 `null`。 + +predicate 用以 `.` 开头的路径读取当前元素,和 jq 一样: + +- `.` 是元素本身。 +- `.findings` 是元素的一个字段。 +- `.files[0]` 和 `.[0]` 按下标读取。 + +在嵌套调用里,`.` 是最内层调用的元素。predicate 也可以读取 `inputs` 和 `steps`。以 `.` 开头的路径在 predicate 之外无法解析。 + +其他函数名无法解析:`"contains" is not a function; the functions are length, any, all and filter`。参数个数不对的调用也无法解析:`any is called as any(list, predicate)`。 + +### pull 时的类型检查 + +`teamai pull` 检查每个 `if` 中的类型。它知道字面量、每个运算符和函数结果的类型,以及工作流在 `inputs` 或 `output` 中声明的每个路径的类型。以下任一情况都会让文件无效: + +- 比较的某一侧不是数字。这样的一侧可以是字符串、布尔值、`null`、条件、`any`、`all` 或 `filter` 调用,或声明类型不是 `number` 的路径。 +- `list` 参数不是数组。这样的参数可以是 `[]` 以外的字面量、条件、`length`、`any` 或 `all` 调用,或声明类型不是数组的路径。 +- predicate 读取了声明的元素类型中没有的字段。 + +例如,如果上面的 `review` 还声明了 `title: string`,下面这个 `fix` 步骤会让文件无效: + +```yaml + - id: fix + needs: [review] + if: steps.review.outputs.title > 3 || any(steps.review.outputs.files, .finding > 0) || all(steps.review.outputs.title, .done) + prompt: Fix the findings. +``` + +`teamai pull` 报告: + +```text +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if compares steps.review.outputs.title with >, but it is not a number +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if reads .finding, which is not in an item of steps.review.outputs.files +⚠ [workflows] workflows/review-files.yaml:12: field "steps[1].if": step "fix" if passes steps.review.outputs.title to all as its list, but it is not an array +``` + +运行工作流的步骤,其输出没有声明类型,因此 pull 不检查它们。运行时,与非数字值的比较为假,`list` 不是数组的函数返回 `null`。 + +### 模板 + +模板读取一个路径,可以带下标,例如 `{{ steps.review.outputs.files[0].path }}` 或 `{{ with.refs[1] }}`。表达式以路径开头(前面可以有 `!` 或 `(`),或对路径调用 `length`、`any`、`all` 或 `filter` 时,模板也会像 `if` 一样对它求值,例如 `{{ length(steps.build.results) }}` 或 `{{ steps.build.outcome == 'failed' }}`。表达式不读取 `with`。只包含一个表达式的 `with` 值保留其 JSON 类型。`teamai pull` 像检查 `if` 一样检查该表达式,写错时还会指出要改什么,包括对步骤拼错的函数名,例如 `{{ lenght(steps.build.results) }}`。其他文本保持原样,例如 Vue 示例中的 `{{ filter(items) }}`。 + +要把计数或过滤后的列表放进提示词,请在模板中对它求值: + +```yaml +schemaVersion: v1alpha1 +name: fan-build-failures +description: Build each package, then look for what the failed builds share. +inputs: + packages: [{ id: string, dir: string }] +steps: + - id: build + for_each: inputs.packages + command: npm run build --prefix "$PACKAGE_DIR" + env: + PACKAGE_DIR: "{{ item.dir }}" + - id: explain + needs: [build] + if: steps.build.outcome == 'failed' + prompt: "{{ length(filter(steps.build.results, .outcome == 'failed')) }} packages failed to build: {{ filter(steps.build.results, .outcome == 'failed') }}. Find what they share." +``` + +提示词需要每一项时,像 `fan-build` 中的 `explain` 那样传入整个列表。 diff --git a/docs/guide/zh-CN/workflow.md b/docs/guide/zh-CN/workflow.md index 87473f8fd..f7a7e2c14 100644 --- a/docs/guide/zh-CN/workflow.md +++ b/docs/guide/zh-CN/workflow.md @@ -2,7 +2,7 @@ > 本文是 [TeamAI CLI 使用指南](../../usage-guide.zh-CN.md) 的一部分。 -工作流是团队仓库里的 `workflows/.yaml`。它运行命令步骤和只读提示词步骤,每个步骤在它依赖的步骤之后运行。 +工作流是团队仓库里的 `workflows/.yaml`。它运行命令步骤和只读提示词步骤,每个步骤在它依赖的步骤之后运行。[高级工作流](workflow-advanced.md) 介绍并行步骤、循环、`for_each`、到达结局时仍在运行的步骤会怎样,以及完整的表达式语言。 ```yaml schemaVersion: v1alpha1 @@ -24,9 +24,9 @@ steps: `schemaVersion` 必须是 `v1alpha1`。`name` 必须和文件名一致。`description` 必填。每个步骤有唯一的 `id`,并且必须且只能使用 `command`、`prompt`、`prompt_file`、`workflow` 之一。提示词步骤可选填 `agent`。`output` 沿用已有简写:`string`、`number`、`boolean`、字面量联合、对象,或只含一种元素类型的数组。TeamAI 会用它校验成功命令的 stdout 或提示词工具返回的 `output`。 -第一个步骤之后的每个步骤都要用 `needs` 列出它等待的步骤。`needs: []` 表示另起一个没有依赖的分支。每次只运行一个步骤:文件中第一个依赖都已完成的步骤。某个依赖失败,或所有依赖都被跳过时,该步骤会被跳过。被跳过的步骤不会运行。一个步骤失败不会停止其他分支。没有可运行的步骤时,只要有步骤失败,运行结果就是 `failed`,否则是 `succeeded`。缺少 `needs`、`needs` 中的 id 不是本工作流的步骤,或依赖成环,都会让该文件无效。缺少 `needs` 时,即使之前有有效版本,该工作流也没有可运行版本。 +第一个步骤之后的每个步骤都要用 `needs` 列出它等待的步骤。`needs: []` 表示另起一个没有依赖的分支。每次 `next` 租出文件中第一个依赖都已完成的步骤。`--driver local` 一次运行一个步骤,除非 `workflows.max_parallel` 大于 1。某个依赖失败,或所有依赖都被跳过时,该步骤会被跳过。被跳过的步骤不会运行。一个步骤失败不会停止其他分支。没有可运行的步骤时,只要有步骤失败,运行结果就是 `failed`,否则是 `succeeded`。缺少 `needs`、`needs` 中的 id 不是本工作流的步骤,或依赖成环,都会让该文件无效。缺少 `needs` 时,即使之前有有效版本,该工作流也没有可运行版本。 -步骤可以带 `if` 表达式。表达式为假时跳过该步骤,这不会让运行失败。表达式可以读取 `steps..outcome`(`success`、`failed` 或 `skipped`)和 `steps..outputs.`,且只能读取该步骤直接或间接依赖的步骤。可用 `==`、`!=`、`&&`、`||`、`!`、括号,以及字面量 `'text'`、数字、`true`、`false`、`null` 和 `[]`。不存在的值读作 `null`。`false`、`null`、`0` 和 `''` 为假,其他值都为真。`==` 是严格比较,并比较整个值:`'1' == 1` 和 `'Build' == 'build'` 都为假。 +步骤可以带 `if` 表达式。表达式为假时跳过该步骤,这不会让运行失败。表达式可以读取 `steps..outcome`(`success`、`failed` 或 `skipped`)和 `steps..outputs.`,且只能读取该步骤直接或间接依赖的步骤。可用 `==`、`!=`、`<`、`<=`、`>`、`>=`、`&&`、`||`、`!`、括号,以及字面量 `'text'`、数字、`true`、`false`、`null` 和 `[]`。路径可以用下标读取数组元素,例如 `steps.review.outputs.files[0]`。函数 `length`、`any`、`all` 和 `filter` 读取数组。不存在的值读作 `null`。`false`、`null`、`0` 和 `''` 为假,其他值都为真。`==` 是严格比较,并比较整个值:`'1' == 1` 和 `'Build' == 'build'` 都为假。 依赖失败的步骤,如果 `if` 读取了每个失败的依赖,仍会运行。可以用它做补救: @@ -45,7 +45,7 @@ steps: 工作流可以用和 `output` 相同的简写声明 `inputs`,例如 `inputs: { issue: number, title: string }`。每个输入都必填。用 JSON 对象传入:`teamai workflow start fix --driver local --inputs '{"issue":42,"title":"Crash"}'`。缺少、多出或类型不对的输入会返回 `invalid_inputs` 并指出字段,且不会创建运行。`if` 可以读取 `inputs.`。 -`{{ inputs. }}` 和 `{{ steps..outputs. }}`(或 `.outcome`)会在步骤开始时填入 `prompt`、`prompt_file` 的文本和 `env` 的值。其他 `{{ ... }}` 文本,例如 `${{ secrets.TOKEN }}`,保持原样。紧跟在 `$` 后面的 `{{` 也保持原样,例如讲解 GitHub Actions 的提示词中的 `${{ inputs.version }}`。要在 `$` 后面填入值,请加一个空格:`$ {{ inputs.price }}`。字符串原样填入,数字和布尔值填入其文本,对象和数组填入 JSON,`null` 或不存在的值填入空文本。模板只能读取该步骤的 `if` 能读取的内容。 +`{{ inputs. }}` 和 `{{ steps..outputs. }}`(或 `.outcome`)会在步骤开始时填入 `prompt`、`prompt_file` 的文本和 `env` 的值。模板也可以像 `if` 一样对表达式求值,例如 `{{ length(steps.review.outputs.files) }}`。其他 `{{ ... }}` 文本,例如 `${{ secrets.TOKEN }}`,保持原样。紧跟在 `$` 后面的 `{{` 也保持原样,例如讲解 GitHub Actions 的提示词中的 `${{ inputs.version }}`。要在 `$` 后面填入值,请加一个空格:`$ {{ inputs.price }}`。字符串原样填入,数字和布尔值填入其文本,对象和数组填入 JSON,`null` 或不存在的值填入空文本。模板只能读取该步骤的 `if` 能读取的内容。 `command` 永远不会被填充。通过 `env` 把值传给命令,只有命令步骤可以有 `env`: @@ -112,7 +112,7 @@ workflows: teamai workflow start build --driver local ``` -`--driver local` 在当前目录里一次运行一个步骤。每次 attempt 的 `result.json` 位于当前 scope 的 TeamAI 数据目录中,不在产品检出目录内。提示词步骤只读;如果工具修改了仓库,步骤会失败,同时保留文件供检查。这也包括已有未跟踪目录内的文件变化;该目录保持不变时,步骤不会因此失败。驱动调用的 `next` 和 `report` 与下面相同: +`--driver local` 在当前目录里运行步骤,一次运行一个,除非 [`workflows.max_parallel`](workflow-advanced.md#并行运行本地命令) 大于 1。每次 attempt 的 `result.json` 位于当前 scope 的 TeamAI 数据目录中,不在产品检出目录内。提示词步骤只读;如果工具修改了仓库,步骤会失败,同时保留文件供检查。这也包括已有未跟踪目录内的文件变化;该目录保持不变时,步骤不会因此失败。驱动调用的 `next` 和 `report` 与下面相同: ```text teamai workflow start build @@ -128,7 +128,7 @@ teamai workflow report --json '{"attempt":"","revision":1,"outcom 开启[使用统计上报](reporting.md)时,每次运行都会向团队的 `teamai-reports` 分支发布一条保护隐私的记录:开始时一次,结束时一次。关闭后不发布任何记录,返回中也不带 `record`。记录存放在 `workflows/runs/.json`,以 run id 为键,重试的发布会更新同一条记录而不是产生副本。 -记录只包含:run id、工作流名、阶段(`start` 或 `end`)、驱动的 kind、状态、运行到达的 end(到达时才有)、开始和结束时间、时长、逐步条目(step 路径如 `check.classify`、agent、tool、model、outcome、tokens——混合 agent 的运行会分别记录每步自己的字段)、步骤汇总计数(succeeded、failed、skipped)、累计已结算 attempt 数,以及发布者用户名。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建,而不是从运行状态里过滤。 +记录只包含:run id、工作流名、阶段(`start` 或 `end`)、驱动的 kind、状态、运行到达的 end(到达时才有)、未到达 end 就失败的运行的失败代码(仅限 teamai 设定的代码,如循环用完次数时的 `loop_exhausted`,或本地驱动找不到工具时的 `tool_unavailable`,不含失败消息)、开始和结束时间、时长、逐步条目(step 路径如 `check.classify`、agent、tool、model、outcome、tokens——混合 agent 的运行会分别记录每步自己的字段;`tool` 仅在是本地驱动的工具之一时保留,`model` 是驱动报告的名称,仅在形如模型名时保留)、步骤汇总计数(succeeded、failed、skipped、cancelled)、累计已结算 attempt 数,以及发布者用户名。每条条目都带 `pass` 编号,循环运行了多次的步骤每个 pass 各有一条条目。`for_each` 的项按该项自己运行的次数计 `pass`,所以前面某个 pass 的项较少时,项的 `pass` 可能比所属步骤的小。被取消步骤的条目 outcome 为 `cancelled`,被跳过的步骤没有 outcome。汇总计数按运行结束时的状态,每个步骤只计一次。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建,而不是从运行状态里过滤。 结束记录由结算该运行的那次 `next` 发布。发布失败时——没有远端、reports 锁被占用、push 被拒——记录会进入当前 scope 数据目录的本地队列,在下一次 `teamai pull` 时重试。此时 `start`、`next` 的返回会带 `record: pending` 而不是 `record: published`。运行本身不受影响:一台机器在运行开始后崩溃,该运行会以"已开始未结束"的状态保持可见。 diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 498b2513c..63f4b2c2e 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -12,6 +12,7 @@ Start with [Getting Started](guide/getting-started.md): set up a team, join one, - [Distribution mechanism](guide/distribution.md) — scope, repo, roles, projects, tags, and subscriptions - [Core Harness](guide/sharing.md) — skills, rules, env, agents, hooks, MCP, models, and packages - [Workflows](guide/workflow.md) — command steps in `workflows/*.yaml`: pull, start, next, and report +- [Advanced workflows](guide/workflow-advanced.md) — parallel steps, loops, `for_each`, ends that cancel open steps, and the expression reference - [Team Context](guide/knowledge.md) — learnings, the codebase graph, teamwiki, and CI - [Team Culture](guide/team-culture.md) — Company and team culture injected into every agent - [Dashboard](guide/dashboard.md) — Local dashboard and session save diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 358472493..7783e8dd4 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -12,6 +12,7 @@ - [分发管理机制](guide/zh-CN/distribution.md) — 安装范围、仓库、角色、项目、标签、订阅 - [基础 Harness](guide/zh-CN/sharing.md) — skills、rules、env、agents、hooks、MCP、models、团队包 - [工作流](guide/zh-CN/workflow.md) — `workflows/*.yaml` 里的 command 步骤:pull、start、next、report +- [高级工作流](guide/zh-CN/workflow-advanced.md) — 并行步骤、循环、`for_each`、到达结局时取消未完成的步骤、表达式参考 - [团队上下文](guide/zh-CN/knowledge.md) — learnings、代码知识图谱、teamwiki、CI - [团队文化](guide/zh-CN/team-culture.md) — 注入到每个 Agent 的公司与团队文化 - [Dashboard](guide/zh-CN/dashboard.md) — 本地看板与会话存档 diff --git a/skill-data/core/references/commands.md b/skill-data/core/references/commands.md index 5b05e7359..42358abde 100644 --- a/skill-data/core/references/commands.md +++ b/skill-data/core/references/commands.md @@ -225,7 +225,7 @@ Generated: do not edit by hand. Regenerate with - `--tools ` — Override local prompt tools for this run, as comma-separated name=tool pairs - `--inputs ` — Values for the inputs the workflow declares, as a JSON object, for example {"issue":42}. Every declared input is required. - `teamai workflow next ` — Lease the next pending step of a run. Prints a JSON envelope. - - `--json ` — Driver identity, for example {"driver":{"kind":"local","machine":"host","workspace":"/work"}} + - `--json ` — Driver identity, for example {"driver":{"kind":"local","machine":"host","workspace":"/work"}}. Optional: running, the attempt ids of its open leases the driver still runs (an open lease left out is handed back), and canTakeLease, false to settle without taking a new lease. Every answer lists in cancelled the running attempts to stop. - `teamai workflow report ` — Report the outcome of a leased step. Prints a JSON envelope. - `--json ` — attempt, revision, outcome (success, failed, or released), and driver diff --git a/src/__tests__/e2e/env-exec-signals.test.ts b/src/__tests__/e2e/env-exec-signals.test.ts index 39da5a3b4..82a561010 100644 --- a/src/__tests__/e2e/env-exec-signals.test.ts +++ b/src/__tests__/e2e/env-exec-signals.test.ts @@ -8,11 +8,12 @@ * number. */ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; -import { spawn, spawnSync } from 'node:child_process'; +import { spawn } from 'node:child_process'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; +import { hasPython, IN_TERMINAL } from '../helpers/in-terminal.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const CLI = path.resolve(__dirname, '..', '..', '..', 'dist', 'index.js'); @@ -25,24 +26,6 @@ const COUNTER = [ 'setInterval(() => {}, 1000);', ].join('\n'); -/** - * Runs argv[2:] as the foreground job of a new terminal (os.forkpty), writes - * its pid to argv[1], and exits as it did. - */ -const IN_TERMINAL = [ - 'import os, sys', - 'pid, fd = os.forkpty()', - 'if pid == 0:', - ' os.execvp(sys.argv[2], sys.argv[2:])', - 'open(sys.argv[1], "w").write(str(pid))', - 'while True:', - ' try:', - ' if not os.read(fd, 4096): break', - ' except OSError:', - ' break', - 'sys.exit(os.waitstatus_to_exitcode(os.waitpid(pid, 0)[1]))', -].join('\n'); -const hasPython = spawnSync('python3', ['-c', 'import os; os.forkpty'], { stdio: 'ignore' }).status === 0; type Ended = { code: number | null; signal: NodeJS.Signals | null; stderr: string }; diff --git a/src/__tests__/e2e/workflow-conformance.test.ts b/src/__tests__/e2e/workflow-conformance.test.ts index b6898c816..d4271c14b 100644 --- a/src/__tests__/e2e/workflow-conformance.test.ts +++ b/src/__tests__/e2e/workflow-conformance.test.ts @@ -65,11 +65,17 @@ describe('workflow protocol conformance through the CLI', () => { if ('error' in started) throw new Error(started.error.message); const runId = String(started.data.run); const leases = new Map(); + const attemptOf = (name: string): string | undefined => leases.get(name)?.attempt; for (const operation of scenario.operations) { if (operation.kind === 'next') { - const result = await runCLI(['workflow', 'next', runId, '--json', JSON.stringify({ driver })], repo); + const result = await runCLI(['workflow', 'next', runId, '--json', JSON.stringify({ + driver, + ...(operation.running ? { running: operation.running.map(attemptOf) } : {}), + ...(operation.canTakeLease !== undefined ? { canTakeLease: operation.canTakeLease } : {}), + })], repo); const value = envelope(result.stdout); expect(value).toMatchObject(operation.expect); + if (operation.cancelled) expect('data' in value && value.data.cancelled).toEqual(operation.cancelled.map(attemptOf)); if ('data' in value && value.data.lease) { leases.set(operation.saveAs, value.data.lease as { attempt: string; revision: number }); } diff --git a/src/__tests__/e2e/workflow-inputs.test.ts b/src/__tests__/e2e/workflow-inputs.test.ts index e909ed085..0037a873b 100644 --- a/src/__tests__/e2e/workflow-inputs.test.ts +++ b/src/__tests__/e2e/workflow-inputs.test.ts @@ -90,4 +90,36 @@ steps: expect(fs.readFileSync(path.join(repo, 'title.txt'), 'utf8')).toBe('$(touch pwned)'); expect(fs.existsSync(path.join(repo, 'pwned'))).toBe(false); }); + + it('runs a step whose if reads an array element, compares numbers and calls any, and skips one that calls length', async () => { + const init = await runCLI(['init', '--self', '--agent', 'claude', '--force'], repo); + expect(init.code, `${init.stdout}\n${init.stderr}`).toBe(0); + + const workflows = path.join(repo, '.teamai', 'workflows'); + fs.mkdirSync(workflows, { recursive: true }); + fs.writeFileSync(path.join(workflows, 'triage.yaml'), `schemaVersion: v1alpha1 +name: triage +description: Label the first issue when it is small and another issue is large. +inputs: + issues: [{ title: string, size: number }] +steps: + - id: label + if: inputs.issues[0].size < 10 && any(inputs.issues, .size >= 100) + command: node -e "require('fs').writeFileSync('label.txt', process.env.TITLE)" + env: + TITLE: "{{ inputs.issues[0].title }}" + - id: many + needs: [] + if: length(inputs.issues) > 2 + command: node -e "require('fs').writeFileSync('many.txt', 'ran')" +`); + const pulled = await runCLI(['pull'], repo); + expect(pulled.code, `${pulled.stdout}\n${pulled.stderr}`).toBe(0); + + const started = await runCLI(['workflow', 'start', 'triage', '--driver', 'local', '--inputs', '{"issues":[{"title":"typo","size":3},{"title":"rewrite","size":400}]}'], repo); + expect(started.code, `${started.stdout}\n${started.stderr}`).toBe(0); + expect(envelope(started.stdout)).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(fs.readFileSync(path.join(repo, 'label.txt'), 'utf8')).toBe('typo'); + expect(fs.existsSync(path.join(repo, 'many.txt'))).toBe(false); + }); }); diff --git a/src/__tests__/e2e/workflow-loop-fan-out.test.ts b/src/__tests__/e2e/workflow-loop-fan-out.test.ts new file mode 100644 index 000000000..beb5a9eed --- /dev/null +++ b/src/__tests__/e2e/workflow-loop-fan-out.test.ts @@ -0,0 +1,161 @@ +/** + * Real CLI: a step loop runs a set number of passes, then a `for_each` of two + * items meets at a barrier, and a last step records what it reads of both. + * With `workflows.max_parallel: 2` the items overlap and the run succeeds. + * With 1 the first item times out alone, and its failure cancels no sibling. + */ +import { execFileSync, spawn } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import YAML from 'yaml'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { resolveProjectDataHome } from '../../config.js'; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..', '..'); +const CLI = path.join(ROOT, 'dist', 'index.js'); + +const TICK = `const fs = require('fs'); +const count = (fs.existsSync('ticks.txt') ? Number(fs.readFileSync('ticks.txt', 'utf8')) : 0) + 1; +fs.writeFileSync('ticks.txt', String(count)); +console.log(JSON.stringify({ count })); +`; + +const BARRIER = `const fs = require('fs'); +const { ME: me, OTHER: other } = process.env; +fs.writeFileSync(me + '.ready', ''); +const deadline = Date.now() + 3000; +setInterval(() => { + if (fs.existsSync(other + '.ready')) process.exit(0); + if (Date.now() > deadline) { console.error(me + ' never saw ' + other); process.exit(1); } +}, 20); +`; + +const RECORD = `require('fs').writeFileSync('record.json', JSON.stringify({ + passes: JSON.parse(process.env.PASSES), + results: JSON.parse(process.env.RESULTS), +})); +`; + +const WORKFLOW = `schemaVersion: v1alpha1 +name: loop-and-fan +description: Count to three in a loop, meet two items at a barrier, then record both. +inputs: + pairs: [{ me: string, other: string }] +steps: + - id: tick + command: node tick.js + output: + count: number + loop: { back_to: tick, while: "steps.tick.outputs.count < 3", max_iterations: 5 } + - id: meet + needs: [tick] + for_each: inputs.pairs + command: node barrier.js + env: + ME: "{{ item.me }}" + OTHER: "{{ item.other }}" + - id: record + needs: [tick, meet] + if: length(steps.tick.passes) == 3 && length(steps.meet.results) == 2 + command: node record.js + env: + PASSES: "{{ steps.tick.passes }}" + RESULTS: "{{ steps.meet.results }}" +`; + +const PAIRS = [{ me: 'left', other: 'right' }, { me: 'right', other: 'left' }]; + +function runCLI(args: string[], cwd: string): Promise<{ code: number | null; stdout: string; stderr: string }> { + return new Promise((resolve) => { + const child = spawn(process.execPath, [CLI, ...args], { cwd, env: { ...process.env, FORCE_COLOR: '0' }, stdio: ['ignore', 'pipe', 'pipe'] }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (chunk: Buffer) => { stdout += chunk.toString(); }); + child.stderr.on('data', (chunk: Buffer) => { stderr += chunk.toString(); }); + child.on('close', (code) => resolve({ code, stdout, stderr })); + }); +} + +function git(args: string[], cwd: string): void { + execFileSync('git', args, { cwd, env: process.env, stdio: 'pipe' }); +} + +function envelope(stdout: string): { data?: Record } { + const line = stdout.trim().split('\n').filter((item) => item.startsWith('{')).at(-1); + if (!line) throw new Error(`no JSON envelope in stdout:\n${stdout}`); + return JSON.parse(line) as { data?: Record }; +} + +describe.skipIf(process.platform === 'win32')('workflow step loop and for_each', () => { + let repo: string; + let configPath: string; + + function setMaxParallel(max: number): void { + const config = YAML.parse(fs.readFileSync(configPath, 'utf8')) as Record; + fs.writeFileSync(configPath, YAML.stringify({ ...config, workflows: { max_parallel: max } })); + } + + async function start(): Promise<{ code: number | null; stdout: string; stderr: string }> { + for (const file of ['ticks.txt', 'left.ready', 'right.ready', 'record.json']) fs.rmSync(path.join(repo, file), { force: true }); + return runCLI(['workflow', 'start', 'loop-and-fan', '--driver', 'local', '--inputs', JSON.stringify({ pairs: PAIRS })], repo); + } + + function record(): { passes: unknown; results: unknown } { + return JSON.parse(fs.readFileSync(path.join(repo, 'record.json'), 'utf8')) as { passes: unknown; results: unknown }; + } + + const passes = [1, 2, 3].map((count) => ({ pass: count, outcome: 'success', outputs: { count } })); + + beforeAll(async () => { + repo = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-loop-fan-out-e2e-'))); + fs.writeFileSync(path.join(repo, 'README.md'), '# app\n'); + git(['init', '-q', '-b', 'main'], repo); + git(['add', 'README.md'], repo); + git(['-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '-m', 'app'], repo); + git(['remote', 'add', 'origin', 'http://127.0.0.1:9/team/workflow.git'], repo); + const init = await runCLI(['init', '--self', '--agent', 'claude', '--force'], repo); + expect(init.code, `${init.stdout}\n${init.stderr}`).toBe(0); + configPath = path.join(await resolveProjectDataHome(repo), 'config.yaml'); + fs.writeFileSync(path.join(repo, 'tick.js'), TICK); + fs.writeFileSync(path.join(repo, 'barrier.js'), BARRIER); + fs.writeFileSync(path.join(repo, 'record.js'), RECORD); + const workflows = path.join(repo, '.teamai', 'workflows'); + fs.mkdirSync(workflows, { recursive: true }); + fs.writeFileSync(path.join(workflows, 'loop-and-fan.yaml'), WORKFLOW); + const pulled = await runCLI(['pull'], repo); + expect(pulled.code, `${pulled.stdout}\n${pulled.stderr}`).toBe(0); + }); + + afterAll(() => { + if (repo) fs.rmSync(repo, { recursive: true, force: true }); + }); + + it('loops three times, then runs both items at once with max_parallel 2', async () => { + setMaxParallel(2); + + const run = await start(); + + expect(run.code, `${run.stdout}\n${run.stderr}`).toBe(0); + expect(envelope(run.stdout)).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(fs.readFileSync(path.join(repo, 'ticks.txt'), 'utf8')).toBe('3'); + expect(record()).toEqual({ + passes, + results: PAIRS.map((item) => ({ item, outcome: 'success' })), + }); + }); + + it('fails the barrier with max_parallel 1, and the failed item cancels no sibling', async () => { + setMaxParallel(1); + + const run = await start(); + + expect(run.code, `${run.stdout}\n${run.stderr}`).toBe(1); + expect(envelope(run.stdout)).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(record()).toEqual({ + passes, + results: [{ item: PAIRS[0], outcome: 'failed' }, { item: PAIRS[1], outcome: 'success' }], + }); + }); +}); diff --git a/src/__tests__/e2e/workflow-records-reports-branch-1083.test.ts b/src/__tests__/e2e/workflow-records-reports-branch-1083.test.ts index a1c1f4367..94f224c36 100644 --- a/src/__tests__/e2e/workflow-records-reports-branch-1083.test.ts +++ b/src/__tests__/e2e/workflow-records-reports-branch-1083.test.ts @@ -92,10 +92,10 @@ describe('#1083 workflow run records reach the reports branch', () => { const endRecord = JSON.parse(endRaw) as Record; expect(endRecord.phase).toBe('end'); expect(endRecord.status).toBe('succeeded'); - expect(endRecord.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0 }); + expect(endRecord.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0, cancelled: 0 }); expect(endRecord.steps).toEqual([ - { step: 'one', outcome: 'success' }, - { step: 'two', outcome: 'success' }, + { step: 'one', pass: 1, outcome: 'success' }, + { step: 'two', pass: 1, outcome: 'success' }, ]); expect(endRecord.startedAt).toBe(startRecord.startedAt); diff --git a/src/__tests__/e2e/workflow-signals.test.ts b/src/__tests__/e2e/workflow-signals.test.ts new file mode 100644 index 000000000..6ada2a477 --- /dev/null +++ b/src/__tests__/e2e/workflow-signals.test.ts @@ -0,0 +1,156 @@ +/** + * Real CLI: with `workflows.max_parallel: 2`, the local driver runs commands in + * process groups of their own. A SIGINT to the driver stops every group, + * children included, and the driver exits with the signal. A terminal's Ctrl-C + * goes to the foreground group: at 2 only the driver is in it and stops the + * groups; at 1 the command is in it and gets the signal directly, as before. + */ +import { execFileSync, spawn } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import YAML from 'yaml'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { resolveProjectDataHome } from '../../config.js'; +import { hasPython, IN_TERMINAL } from '../helpers/in-terminal.js'; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..', '..'); +const CLI = path.join(ROOT, 'dist', 'index.js'); + +/** Starts a child that holds none of its output, so only a group kill stops it, then writes both pids to `.pids` and waits. */ +const HOLD = `const { spawn } = require('child_process'); +const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); +require('fs').writeFileSync(process.argv[2] + '.pids', process.pid + ' ' + child.pid); +setInterval(() => {}, 1000); +`; + +function runCLI(args: string[], cwd: string): Promise<{ code: number | null; stdout: string; stderr: string }> { + return new Promise((resolve) => { + const child = spawn(process.execPath, [CLI, ...args], { cwd, env: { ...process.env, FORCE_COLOR: '0' }, stdio: ['ignore', 'pipe', 'pipe'] }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (chunk: Buffer) => { stdout += chunk.toString(); }); + child.stderr.on('data', (chunk: Buffer) => { stderr += chunk.toString(); }); + child.on('close', (code) => resolve({ code, stdout, stderr })); + }); +} + +function git(args: string[], cwd: string): void { + execFileSync('git', args, { cwd, env: process.env, stdio: 'pipe' }); +} + +function alive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +async function waitFor(check: () => boolean, ms: number): Promise { + for (let waited = 0; !check(); waited += 50) { + if (waited >= ms) return false; + await new Promise((resolve) => setTimeout(resolve, 50)); + } + return true; +} + +describe.skipIf(process.platform === 'win32')('workflow local driver signals', () => { + let repo: string; + let configPath: string; + + function setMaxParallel(max: number): void { + const config = YAML.parse(fs.readFileSync(configPath, 'utf8')) as Record; + fs.writeFileSync(configPath, YAML.stringify({ ...config, workflows: { max_parallel: max } })); + } + + beforeAll(async () => { + repo = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-signals-e2e-'))); + fs.writeFileSync(path.join(repo, 'README.md'), '# app\n'); + git(['init', '-q', '-b', 'main'], repo); + git(['add', 'README.md'], repo); + git(['-c', 'user.name=Test', '-c', 'user.email=test@example.com', 'commit', '-q', '-m', 'app'], repo); + git(['remote', 'add', 'origin', 'http://127.0.0.1:9/team/workflow.git'], repo); + const init = await runCLI(['init', '--self', '--agent', 'claude', '--force'], repo); + expect(init.code, `${init.stdout}\n${init.stderr}`).toBe(0); + configPath = path.join(await resolveProjectDataHome(repo), 'config.yaml'); + const workflows = path.join(repo, '.teamai', 'workflows'); + fs.mkdirSync(workflows, { recursive: true }); + fs.writeFileSync(path.join(repo, 'hold.js'), HOLD); + fs.writeFileSync(path.join(workflows, 'hold.yaml'), `schemaVersion: v1alpha1 +name: hold +description: Two commands that run until they are stopped. +steps: + - id: one + needs: [] + command: node hold.js one + - id: two + needs: [] + command: node hold.js two +`); + const pulled = await runCLI(['pull'], repo); + expect(pulled.code, `${pulled.stdout}\n${pulled.stderr}`).toBe(0); + }); + + afterAll(() => { + if (repo) fs.rmSync(repo, { recursive: true, force: true }); + }); + + it('stops every command and its children on SIGINT, then exits with the signal', async () => { + setMaxParallel(2); + const pidFiles = ['one.pids', 'two.pids'].map((file) => path.join(repo, file)); + for (const file of pidFiles) fs.rmSync(file, { force: true }); + const driver = spawn(process.execPath, [CLI, 'workflow', 'start', 'hold', '--driver', 'local'], { + cwd: repo, env: { ...process.env, FORCE_COLOR: '0' }, stdio: ['ignore', 'pipe', 'pipe'], + }); + let stderr = ''; + driver.stderr.on('data', (chunk: Buffer) => { stderr += chunk.toString(); }); + const ended = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => { + driver.on('close', (code, signal) => resolve({ code, signal })); + }); + let pids: number[] = []; + try { + expect(await waitFor(() => pidFiles.every((file) => fs.existsSync(file)), 20_000), stderr).toBe(true); + pids = pidFiles.flatMap((file) => fs.readFileSync(file, 'utf8').trim().split(' ').map(Number)); + expect(pids.every(alive)).toBe(true); + + driver.kill('SIGINT'); + + expect(await ended).toEqual({ code: null, signal: 'SIGINT' }); + expect(await waitFor(() => !pids.some(alive), 2000), `still running: ${pids.filter(alive).join(' ')}`).toBe(true); + } finally { + for (const pid of pids.filter(alive)) process.kill(pid, 'SIGKILL'); + if (driver.exitCode === null && driver.signalCode === null) driver.kill('SIGKILL'); + } + }); + + it.skipIf(!hasPython).each([2, 1])("stops every command and its children on a terminal's Ctrl-C with max_parallel %i", async (max) => { + setMaxParallel(max); + const pidFiles = ['one.pids', 'two.pids'].map((file) => path.join(repo, file)); + for (const file of pidFiles) fs.rmSync(file, { force: true }); + const terminalPid = path.join(repo, 'terminal.pid'); + fs.rmSync(terminalPid, { force: true }); + const wrapper = spawn('python3', ['-c', IN_TERMINAL, terminalPid, process.execPath, CLI, 'workflow', 'start', 'hold', '--driver', 'local'], { + cwd: repo, env: { ...process.env, FORCE_COLOR: '0' }, stdio: 'ignore', + }); + const ended = new Promise((resolve) => { wrapper.on('close', () => resolve()); }); + const started = max === 2 ? pidFiles : pidFiles.slice(0, 1); + let pids: number[] = []; + try { + expect(await waitFor(() => fs.existsSync(terminalPid) && started.every((file) => fs.existsSync(file)), 20_000)).toBe(true); + pids = started.flatMap((file) => fs.readFileSync(file, 'utf8').trim().split(' ').map(Number)); + expect(pids.every(alive)).toBe(true); + + process.kill(-Number(fs.readFileSync(terminalPid, 'utf8')), 'SIGINT'); + + await ended; + expect(await waitFor(() => !pids.some(alive), 2000), `still running: ${pids.filter(alive).join(' ')}`).toBe(true); + expect(max === 1 && fs.existsSync(pidFiles[1]!), 'the second command started after Ctrl-C').toBe(false); + } finally { + for (const pid of pids.filter(alive)) process.kill(pid, 'SIGKILL'); + if (wrapper.exitCode === null && wrapper.signalCode === null) wrapper.kill('SIGKILL'); + } + }); +}); diff --git a/src/__tests__/fixtures/workflow/run-before-loops.json b/src/__tests__/fixtures/workflow/run-before-loops.json new file mode 100644 index 000000000..dae9342c3 --- /dev/null +++ b/src/__tests__/fixtures/workflow/run-before-loops.json @@ -0,0 +1,119 @@ +{ + "schemaVersion": "v1alpha1", + "id": "rdc7b5bd3", + "name": "legacy", + "revision": 4, + "status": "running", + "sourceHash": "e1e478e5f06c5d7de569117b10e9b8a0b1d0a589c29d4e3a5db95186c80fa201", + "definition": { + "schemaVersion": "v1alpha1", + "name": "legacy", + "description": "A run file written before loops and for_each.", + "steps": [ + { + "id": "build", + "command": "node -e \"process.stdout.write(JSON.stringify({artifact:'app.tgz'}))\"", + "output": { + "type": "object", + "fields": { + "artifact": { + "type": "string" + } + } + } + }, + { + "id": "check", + "needs": [ + "build" + ], + "workflow": { + "ends": { + "ready": "succeeded" + }, + "outputs": { + "verdict": "steps.classify.outputs.verdict" + }, + "steps": [ + { + "id": "classify", + "command": "node -e \"process.stdout.write(JSON.stringify({verdict:'ship'}))\"", + "output": { + "type": "object", + "fields": { + "verdict": { + "type": "string" + } + } + } + }, + { + "id": "done", + "needs": [ + "classify" + ], + "end": "ready" + } + ] + } + }, + { + "id": "ship", + "needs": [ + "check" + ], + "command": "node -e \"require('fs').writeFileSync('shipped.txt', process.env.ARTIFACT)\"", + "env": { + "ARTIFACT": "{{ steps.build.outputs.artifact }}" + } + } + ] + }, + "steps": [ + { + "id": "build", + "status": "succeeded", + "exitCode": 0, + "outcome": "success", + "outputs": { + "artifact": "app.tgz" + } + }, + { + "id": "check", + "status": "succeeded", + "outcome": "success", + "end": "ready", + "outputs": { + "verdict": "ship" + } + }, + { + "id": "check.classify", + "status": "succeeded", + "exitCode": 0, + "outcome": "success", + "outputs": { + "verdict": "ship" + } + }, + { + "id": "check.done", + "status": "succeeded" + }, + { + "id": "ship", + "status": "pending", + "exitCode": 130, + "message": "driver stopped before the step finished", + "outcome": "released" + } + ], + "settledAttempts": [ + "adee20d55", + "a1baa570f", + "a466320c2" + ], + "createdAt": "2026-10-15T12:10:53.819Z", + "updatedAt": "2026-10-15T12:10:54.766Z" +} diff --git a/src/__tests__/helpers/in-terminal.ts b/src/__tests__/helpers/in-terminal.ts new file mode 100644 index 000000000..8f5b29e5d --- /dev/null +++ b/src/__tests__/helpers/in-terminal.ts @@ -0,0 +1,22 @@ +import { spawnSync } from 'node:child_process'; + +/** + * Runs argv[2:] as the foreground job of a new terminal (os.forkpty), writes + * its pid to argv[1], and exits as it did. + */ +export const IN_TERMINAL = [ + 'import os, sys', + 'pid, fd = os.forkpty()', + 'if pid == 0:', + ' os.execvp(sys.argv[2], sys.argv[2:])', + 'open(sys.argv[1], "w").write(str(pid))', + 'while True:', + ' try:', + ' if not os.read(fd, 4096): break', + ' except OSError:', + ' break', + 'sys.exit(os.waitstatus_to_exitcode(os.waitpid(pid, 0)[1]))', +].join('\n'); + +/** Whether python3 with os.forkpty is available to run IN_TERMINAL. */ +export const hasPython = spawnSync('python3', ['-c', 'import os; os.forkpty'], { stdio: 'ignore' }).status === 0; diff --git a/src/__tests__/workflow-cache.test.ts b/src/__tests__/workflow-cache.test.ts index ebf3e29f0..35668e3db 100644 --- a/src/__tests__/workflow-cache.test.ts +++ b/src/__tests__/workflow-cache.test.ts @@ -62,6 +62,128 @@ afterEach(() => { }); describe('workflow cache', () => { + it.each([ + ['fan-out pass outputs', "steps.build.passes[0].outputs.ok", ' output: { ok: boolean }\n'], + ['fan-out pass outputs without an output schema', "steps.build.passes[0].outputs.ok", ''], + ['undeclared item output in a pass', "steps.build.passes[0].results[0].outputs.missing", ' output: { ok: boolean }\n'], + ['ordinary pass results', "steps.leaf.passes[0].results", ''], + ['typed pass result outcome', "steps.build.passes[0].results[0].outcome > 1", ''], + ['typed pass result predicate', "any(steps.build.passes, .results[0].outcome > 1)", ''], + ])('rejects %s during workflow sync', async (_name, read, output) => { + const config = testConfig(); + writeWorkflow(config, 'build', `schemaVersion: v1alpha1 +name: build +description: Validate pass fields. +steps: + - id: build + for_each: "[]" + command: echo build +${output} - id: leaf + needs: [build] + command: echo leaf + - id: summary + needs: [leaf] + if: ${JSON.stringify(read)} + command: echo summary +`); + const synced = await syncWorkflowDefinitions(config); + expect(synced.invalid).not.toEqual([]); + expect(synced.cached).toEqual([]); + }); + + it.each([ + ['an output', 'steps.build.outputs.ok', 'output "value" reads steps.build.outputs, but step "build" has for_each'], + ['an output', 'steps.leaf.results', 'output "value" reads steps.leaf.results, but step "leaf" has no for_each'], + ['an output', 'steps.build.passes[0].outputs.ok', 'output "value" reads steps.build.passes[0].outputs, but step "build" has for_each'], + ['an output', 'steps.leaf.passes[0].results', 'output "value" reads steps.leaf.passes[0].results, but step "leaf" has no for_each'], + ['an if', 'steps.build.outputs.ok', 'step "summary" if reads steps.build.outputs, but step "build" has for_each'], + ['an if', 'steps.build.passes[0].outputs.ok', 'step "summary" if reads steps.build.passes[0].outputs, but step "build" has for_each'], + ['an if', 'any(steps.build.passes, .outputs.ok)', 'step "summary" if reads .outputs.ok, which is not in an item of steps.build.passes'], + ['an if', 'steps.leaf.passes[0].results', 'step "summary" if reads steps.leaf.passes[0].results, but step "leaf" has no for_each'], + ])('keeps the last valid version when %s reads the unavailable %s', async (site, read, message) => { + const config = testConfig(); + const body = `schemaVersion: v1alpha1 +name: build +description: Read a field a step no longer has. +steps: + - id: group + workflow: +${site === 'an output' ? ` outputs:\n value: ${read}\n` : ''} steps: + - id: build + for_each: "[]" + command: echo build + output: { ok: boolean } + - id: leaf + needs: [build] + command: echo leaf + - id: summary + needs: [leaf] +${site === 'an if' ? ` if: ${JSON.stringify(read)}\n` : ''} command: echo summary +`; + writeWorkflow(config, 'build', body.replace(' for_each: "[]"\n', '').replace(' - id: leaf\n', ' - id: leaf\n for_each: "[]"\n')); + expect((await syncWorkflowDefinitions(config)).cached).toEqual(['build']); + const valid = await loadCachedWorkflow(config, 'build'); + expect(valid).not.toBeNull(); + writeWorkflow(config, 'build', body); + const synced = await syncWorkflowDefinitions(config); + expect(synced.invalid).toEqual([expect.objectContaining({ message: expect.stringContaining(message) })]); + expect(synced.invalid[0]).not.toHaveProperty('code'); + expect(synced.kept).toEqual(['build']); + expect(await loadCachedWorkflow(config, 'build')).toEqual(valid); + }); + + it('accepts declared item outputs in fan-out pass results and workflow outputs', async () => { + const config = testConfig(); + writeWorkflow(config, 'build', `schemaVersion: v1alpha1 +name: build +description: Read item outputs in passes. +outputs: + ok: steps.build.passes[0].results[0].outputs.ok + results: steps.build.results +steps: + - id: build + for_each: "[]" + command: echo build + output: { ok: boolean } + - id: summary + needs: [build] + if: steps.build.passes[0].results[0].outputs.ok && all(steps.build.passes, .results[0].outputs.ok) + command: echo summary +`); + const synced = await syncWorkflowDefinitions(config); + expect(synced.invalid).toEqual([]); + expect(synced.cached).toEqual(['build']); + }); + + it.each(['ok', 'missing'])('checks the child output %s inside a fan-out pass result', async (field) => { + const config = testConfig(); + writeWorkflow(config, 'build', `schemaVersion: v1alpha1 +name: build +description: Read workflow item outputs in passes. +steps: + - id: build + for_each: "[]" + workflow: + outputs: { ok: steps.run.outputs.ok } + steps: + - id: run + command: echo run + output: { ok: boolean } + - id: summary + needs: [build] + if: steps.build.passes[0].results[0].outputs.${field} + command: echo summary +`); + const synced = await syncWorkflowDefinitions(config); + if (field === 'ok') { + expect(synced.invalid).toEqual([]); + expect(synced.cached).toEqual(['build']); + } else { + expect(synced.invalid).toEqual([expect.objectContaining({ code: 'undeclared_output' })]); + expect(synced.cached).toEqual([]); + } + }); + it('caches prompt_file bytes and snapshots them into a run', async () => { const config = testConfig(); fs.mkdirSync(path.join(config.repo.localPath, 'prompts')); @@ -250,7 +372,7 @@ steps: await syncWorkflowDefinitions(config); expect(await loadCachedWorkflow(config, 'build')).not.toBeNull(); - writeWorkflow(config, 'build', VALID.replace('command: node compile.js', 'command: node compile.js\n for_each: items')); + writeWorkflow(config, 'build', VALID.replace('command: node compile.js', 'command: node compile.js\n timeout: 10m')); const synced = await syncWorkflowDefinitions(config); expect(synced.unrunnable).toContain('workflows/build.yaml'); @@ -799,14 +921,14 @@ steps: env: TASK: "{{ steps.plan.outputs.task }}" FREE: "{{ steps.free.outputs.y }}" - SAME: "{{ inputs.title == 'x' }}" + SAME: "{{ inputs.title = 'x' }}" `); const result = await syncWorkflowDefinitions(config); expect(result.unrunnable).toEqual(['workflows/build.yaml']); expect(result.invalid.map(formatDiagnostic)).toEqual([ - 'workflows/build.yaml:23: field "steps[3].env.SAME": step "label" env.SAME template {{ inputs.title == \'x\' }}: a template reads one path', + 'workflows/build.yaml:23: field "steps[3].env.SAME": step "label" env.SAME template {{ inputs.title = \'x\' }}: unexpected "="', 'workflows/build.yaml:16: field "steps[2].prompt": step "fix" prompt reads inputs.titel, which is not an input the workflow declares', 'workflows/build.yaml:16: field "steps[2].prompt": step "fix" prompt reads steps.plan.outputs.tsk, which is not in the output step "plan" declares', 'workflows/build.yaml:16: field "steps[2].prompt": step "fix" prompt reads steps.free, which is not a step it needs, directly or through its needs', @@ -815,6 +937,79 @@ steps: expect(result.invalid.map((diagnostic) => diagnostic.code)).toEqual(Array(5).fill('bad_template')); }); + it('accepts a function call in a template, drops the last valid version for a call with a mistake in it, and leaves other braces as text', async () => { + const config = testConfig(); + fs.mkdirSync(path.join(config.repo.localPath, 'prompts')); + const promptFile = path.join(config.repo.localPath, 'prompts', 'review.md'); + const body = (env: string, value: string, prompt: string): string => `schemaVersion: v1alpha1 +name: build +description: Report the failed builds. +inputs: + targets: [string] +steps: + - id: build + for_each: inputs.targets + command: node build.js + - id: notify + needs: [build] + command: node notify.js + env: + FAILED: "${env}" + - id: fix + needs: [build] + with: + failed: "${value}" + prompt: "${prompt} {{ with.failed }}" +`; + fs.writeFileSync(promptFile, 'Review the builds.'); + writeWorkflow(config, 'build', body('{{ steps.build.results }}', '{{ steps.build.outcome }}', 'Fix the failed builds.')); + writeWorkflow(config, 'review', `schemaVersion: v1alpha1 +name: review +description: Review the builds. +inputs: + targets: [string] +steps: + - id: build + for_each: inputs.targets + command: node build.js + - id: review + needs: [build] + prompt_file: prompts/review.md +`); + expect((await syncWorkflowDefinitions(config)).cached).toEqual(['build', 'review']); + fs.writeFileSync(promptFile, 'Review {{ length(steps.build.results) }} builds.'); + writeWorkflow(config, 'build', body( + "{{ filter(steps.build.results, .outcome == 'failed') }}", + "{{ any(steps.build.results, .outcome == 'failed') }}", + 'Fix {{ length(steps.build.results) }} builds.', + )); + writeWorkflow(config, 'jinja', `schemaVersion: v1alpha1 +name: jinja +description: Explain a Jinja template. +steps: + - id: explain + prompt: "Explain {{ items | length }} and {{ name }} in this Jinja template." +`); + + const accepted = await syncWorkflowDefinitions(config); + + expect(accepted.invalid.map(formatDiagnostic)).toEqual([]); + expect(accepted.cached).toEqual(['build', 'jinja', 'review']); + expect((await loadCachedWorkflow(config, 'build'))?.definition.steps[1]?.env).toEqual({ FAILED: "{{ filter(steps.build.results, .outcome == 'failed') }}" }); + expect((await loadCachedWorkflow(config, 'jinja'))?.definition.steps[0]?.prompt).toBe('Explain {{ items | length }} and {{ name }} in this Jinja template.'); + + fs.writeFileSync(promptFile, 'Review {{ length(steps.build.result) }} builds.'); + const result = await syncWorkflowDefinitions(config); + + expect(result.invalid.map(formatDiagnostic)).toEqual([ + 'workflows/review.yaml:12: field "steps[1].prompt_file": step "review" prompt_file template {{ length(steps.build.result) }}: "steps.build.result" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results', + ]); + expect(result.invalid.map((diagnostic) => diagnostic.code)).toEqual(['bad_template']); + expect(result.unrunnable).toEqual(['workflows/review.yaml']); + expect(result.kept).toEqual([]); + expect(await startRun(config, 'review', { targets: [] })).toMatchObject({ error: { code: 'workflow_not_found' } }); + }); + it('reports a bad template in prompt_file text and drops the version that rendered it as text', async () => { const config = testConfig(); fs.mkdirSync(path.join(config.repo.localPath, 'prompts')); @@ -846,6 +1041,40 @@ steps: expect(await startRun(config, 'build')).toMatchObject({ error: { code: 'workflow_not_found' } }); }); + it('lets a prompt_file read item where a for_each binds it, and keeps item as written elsewhere', async () => { + const config = testConfig(); + fs.mkdirSync(path.join(config.repo.localPath, 'prompts')); + fs.writeFileSync(path.join(config.repo.localPath, 'prompts', 'review.md'), 'Review {{ item.path }}.'); + const quote = 'Quote {{ item }}, {{ item.path }}, {{ item[0] }} and {{ length(item.xs) }} from a playbook.'; + fs.writeFileSync(path.join(config.repo.localPath, 'prompts', 'quote.md'), quote); + writeWorkflow(config, 'fan', `schemaVersion: v1alpha1 +name: fan +description: Review each file. +inputs: + files: [{ path: string }] +steps: + - id: review + for_each: inputs.files + prompt_file: prompts/review.md + - id: check + needs: [review] + for_each: inputs.files + workflow: + steps: + - id: recheck + prompt_file: prompts/review.md + - id: summary + needs: [check] + prompt_file: prompts/quote.md +`); + + const result = await syncWorkflowDefinitions(config); + + expect(result.invalid.map(formatDiagnostic)).toEqual([]); + expect(result.cached).toEqual(['fan']); + expect((await loadCachedWorkflow(config, 'fan'))?.definition.steps[2]?.prompt).toBe(quote); + }); + it('reports a with read outside the prompt of a step whose with declares it', async () => { const config = testConfig(); writeWorkflow(config, 'build', `schemaVersion: v1alpha1 @@ -1253,7 +1482,7 @@ steps: expect(result.invalid.map((diagnostic) => [formatDiagnostic(diagnostic), diagnostic.code])).toEqual([ ['workflows/build.yaml:17: field "steps[2].workflow.inputs": step "check" runs an inline workflow, which declares no inputs; its steps read the enclosing workflow\'s inputs', undefined], - ['workflows/build.yaml:19: field "steps[2].workflow.name": step "check" runs an inline workflow, which carries only steps, outputs, and ends', undefined], + ['workflows/build.yaml:19: field "steps[2].workflow.name": step "check" runs an inline workflow, which carries only steps, outputs, ends, and loop', undefined], ['workflows/build.yaml:14: field "steps[2].with": step "check" with is not allowed on an inline workflow; its steps read the enclosing inputs and steps', undefined], ['workflows/build.yaml:22: field "steps[2].workflow.steps[0].prompt": step "classify" prompt reads steps.free, which is not a step it needs, or one the step that runs its workflow needs, directly or through their needs', 'bad_template'], ['workflows/build.yaml:25: field "steps[2].workflow.steps[1].if": step "comment" if reads steps.free, which is not a step it needs, or one the step that runs its workflow needs, directly or through their needs', undefined], diff --git a/src/__tests__/workflow-commands.test.ts b/src/__tests__/workflow-commands.test.ts index 613503832..53f8cd4a1 100644 --- a/src/__tests__/workflow-commands.test.ts +++ b/src/__tests__/workflow-commands.test.ts @@ -122,6 +122,28 @@ describe('workflow json payloads', () => { const missing = await capture(() => workflowNext('r00000000', { json: ' ' })); expect(missing).toMatchObject({ error: { code: 'invalid_json', message: 'Pass --json with a driver object' } }); }); + + it.each([ + ['running that is not an array', { running: 'a00000000' }, 'running must be an array of attempt ids'], + ['running that holds a number', { running: [1] }, 'running must be an array of attempt ids'], + ['canTakeLease that is not a boolean', { canTakeLease: 'false' }, 'canTakeLease must be a boolean'], + ])('rejects a next payload with %s', async (_case, fields, message) => { + const { config, runId } = await startBuild(); + state.config = config; + + const rejected = await capture(() => workflowNext(runId, { json: JSON.stringify({ driver, ...fields }) })); + + expect(rejected).toEqual({ error: { code: 'invalid_json', message } }); + }); + + it('passes running and canTakeLease from the next payload to the run', async () => { + const { config, runId } = await startBuild(); + state.config = config; + + const reconciled = await capture(() => workflowNext(runId, { json: JSON.stringify({ driver, running: ['a00000000'], canTakeLease: false }) })); + + expect(reconciled).toEqual({ data: { wait: { reason: 'canTakeLease is false' }, cancelled: ['a00000000'] } }); + }); }); describe('workflow start --inputs', () => { diff --git a/src/__tests__/workflow-conformance.test.ts b/src/__tests__/workflow-conformance.test.ts index 45775b995..7f32ba49f 100644 --- a/src/__tests__/workflow-conformance.test.ts +++ b/src/__tests__/workflow-conformance.test.ts @@ -5,7 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'; import type { LocalConfig } from '../types.js'; import { syncWorkflowDefinitions } from '../workflow/cache.js'; import { workflowConformanceScenarios, type WorkflowClient } from '../workflow/conformance.js'; -import { leaseNext, parseReport, reportAttempt, startRun } from '../workflow/engine.js'; +import { leaseNext, parseNextPayload, parseReport, reportAttempt, startRun } from '../workflow/engine.js'; import { readRun } from '../workflow/store.js'; import type { DriverIdentity } from '../workflow/types.js'; @@ -38,7 +38,12 @@ describe('workflow protocol conformance in process', () => { await syncWorkflowDefinitions(config); const client: WorkflowClient = { start: (name) => startRun(config, name), - next: (runId, identity) => leaseNext(config, runId, identity), + next: async (runId, body) => { + const parsed = parseNextPayload(body); + if (typeof parsed === 'string') return { error: { code: 'invalid_json', message: parsed } }; + const { driver: identity, ...options } = parsed; + return leaseNext(config, runId, identity, options); + }, report: async (runId, body) => { const parsed = parseReport(body); return typeof parsed === 'string' @@ -50,10 +55,16 @@ describe('workflow protocol conformance in process', () => { if ('error' in started) throw new Error(started.error.message); const runId = String(started.data.run); const leases = new Map(); + const attemptOf = (name: string): string | undefined => leases.get(name)?.attempt; for (const operation of scenario.operations) { if (operation.kind === 'next') { - const envelope = await client.next(runId, driver); + const envelope = await client.next(runId, { + driver, + ...(operation.running ? { running: operation.running.map(attemptOf) } : {}), + ...(operation.canTakeLease !== undefined ? { canTakeLease: operation.canTakeLease } : {}), + }); expect(envelope).toMatchObject(operation.expect); + if (operation.cancelled) expect('data' in envelope && envelope.data.cancelled).toEqual(operation.cancelled.map(attemptOf)); if ('data' in envelope && envelope.data.lease) { leases.set(operation.saveAs, envelope.data.lease as { attempt: string; revision: number }); } diff --git a/src/__tests__/workflow-docs-examples.test.ts b/src/__tests__/workflow-docs-examples.test.ts new file mode 100644 index 000000000..01fe35136 --- /dev/null +++ b/src/__tests__/workflow-docs-examples.test.ts @@ -0,0 +1,289 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterEach, describe, expect, it } from 'vitest'; +import { parseDocument } from 'yaml'; +import type { LocalConfig } from '../types.js'; +import { syncWorkflowDefinitions } from '../workflow/cache.js'; +import { leaseNext, reportAttempt, startRun } from '../workflow/engine.js'; +import { formatDiagnostic } from '../workflow/parse.js'; +import type { DriverIdentity, WorkflowEnvelope } from '../workflow/types.js'; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); +const GUIDE = path.join(ROOT, 'docs', 'guide'); + +// docs/guide/**/workflow*.md, so a new page or language is checked by default. fs.globSync needs Node 22; CI runs Node 20. +const PAGES = fs.readdirSync(GUIDE, { recursive: true, encoding: 'utf8' }) + .filter((file) => /^workflow.*\.md$/.test(path.basename(file))) + .map((file) => path.posix.join('docs/guide', file.split(path.sep).join('/'))) + .sort(); + +interface Example { + label: string; + source: string; +} + +function examplesOf(page: string): Example[] { + const text = fs.readFileSync(path.join(ROOT, page), 'utf8'); + return [...text.matchAll(/^```yaml\n([\s\S]*?)^```/gm)].map((match) => { + const line = text.slice(0, match.index).split('\n').length + 1; + const source = match[1] ?? ''; + return { label: `${page}:${line} (${source.split('\n')[0]?.trim()})`, source }; + }); +} + +function isWorkflow(value: unknown): value is { name?: unknown } { + return !!value && typeof value === 'object' && !Array.isArray(value) && 'steps' in value; +} + +function promptFiles(value: unknown): string[] { + if (Array.isArray(value)) return value.flatMap(promptFiles); + if (!value || typeof value !== 'object') return []; + return Object.entries(value).flatMap(([key, field]) => (key === 'prompt_file' && typeof field === 'string' ? [field] : promptFiles(field))); +} + +/** Workflows a start line names that its page declares only in prose, as `inputs: { … }` in the same paragraph. */ +const DECLARED_IN_PROSE = new Set(['fix']); + +interface Start { + label: string; + line: number; + name: string; + json: string; + paragraph: string; +} + +/** Each `teamai workflow start … --inputs ''` on `page`, and each `--inputs` with a value it could not read as one. */ +function startsOf(page: string): { starts: Start[]; unread: string[] } { + const text = fs.readFileSync(path.join(ROOT, page), 'utf8'); + const lineOf = (index: number): number => text.slice(0, index).split('\n').length; + const starts = [...text.matchAll(/teamai workflow start ([A-Za-z0-9][A-Za-z0-9._-]*)[^\n`']*--inputs '([^'\n]*)'/g)].map((match) => ({ + label: `${page}:${lineOf(match.index)} (${match[1]})`, + line: lineOf(match.index), + name: match[1] ?? '', + json: match[2] ?? '', + paragraph: text.split(/\n\s*\n/).find((block) => block.includes(match[0])) ?? '', + })); + const unread = [...text.matchAll(/--inputs[ =]['"{]/g)] + .map((match) => lineOf(match.index)) + .filter((line) => !starts.some((start) => start.line === line)) + .map((line) => `${page}:${line}: --inputs is not part of a teamai workflow start line`); + return { starts, unread }; +} + +const dirs: string[] = []; + +function teamRepo(): LocalConfig { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-docs-')); + dirs.push(root); + const knowledge = path.join(root, 'knowledge'); + fs.mkdirSync(path.join(knowledge, 'workflows'), { recursive: true }); + return { + repo: { localPath: knowledge, remote: 'local', kind: 'self' }, + username: 'tester', + scope: 'project', + additionalRoles: [], + projectRoot: root, + dataHome: path.join(root, 'data'), + usageReportEnabled: false, + }; +} + +const driver: DriverIdentity = { kind: 'local', machine: 'docs', workspace: '/work' }; + +/** Writes each complete workflow example on `page` into the team repo, and the prompt files they name. */ +function writeExamples(page: string, config: LocalConfig): { byFile: Map; failures: string[] } { + const failures: string[] = []; + const byFile = new Map(); + for (const example of examplesOf(page)) { + const document = parseDocument(example.source); + if (document.errors.length > 0) { + failures.push(`${example.label}: ${document.errors[0]?.message}`); + continue; + } + const value: unknown = document.toJS(); + if (!isWorkflow(value)) continue; + const file = `workflows/${String(value.name)}.yaml`; + const twin = byFile.get(file); + if (twin) failures.push(`${example.label}: ${file} is also ${twin}`); + byFile.set(file, example.label); + fs.writeFileSync(path.join(config.repo.localPath, file), example.source); + for (const prompt of promptFiles(value).filter((file) => file.startsWith('prompts/'))) { + fs.mkdirSync(path.dirname(path.join(config.repo.localPath, prompt)), { recursive: true }); + fs.writeFileSync(path.join(config.repo.localPath, prompt), 'Review the change.\n'); + } + } + return { byFile, failures }; +} + +afterEach(() => { + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +const ADVANCED = PAGES.filter((page) => path.posix.basename(page) === 'workflow-advanced.md'); +/** The driver the advanced page's `next` table sends. */ +const scriptDriver: DriverIdentity = { kind: 'script', machine: 'host', workspace: '/work' }; + +interface Lease { + step: string; + attempt: string; + revision: number; + pass: number; + prompt?: string; +} + +/** Each first group of `pattern` on `page`, parsed as JSON. */ +function jsonOn(page: string, pattern: RegExp): unknown[] { + return [...fs.readFileSync(path.join(ROOT, page), 'utf8').matchAll(pattern)].map((match) => JSON.parse(match[1] ?? '') as unknown); +} + +/** `value` with its run and attempt ids masked, which differ in every run. */ +function withoutIds(value: unknown): unknown { + return JSON.parse(JSON.stringify(value).replace(/"run":"r[0-9a-f]+"/g, '"run":""').replace(/"attempt":"a[0-9a-f]+"/g, '"attempt":""')); +} + +function leaseOf(answer: WorkflowEnvelope): Lease | undefined { + return 'data' in answer ? answer.data.lease as Lease | undefined : undefined; +} + +/** Starts the workflow `name` that `page` shows. */ +async function startExample(page: string, name: string, inputs?: unknown): Promise<{ config: LocalConfig; runId: string }> { + const config = teamRepo(); + writeExamples(page, config); + await syncWorkflowDefinitions(config); + const started = await startRun(config, name, inputs); + if ('error' in started) throw new Error(`${page}: ${started.error.message}`); + return { config, runId: String(started.data.run) }; +} + +/** Reports each lease with `outcome` until `next` leases nothing more, and returns every answer that leased a step. */ +async function driveExample(config: LocalConfig, runId: string, outcome: (lease: Lease) => 'success' | 'failed'): Promise { + const answers: WorkflowEnvelope[] = []; + for (let turn = 0; turn < 50; turn += 1) { + const answer = await leaseNext(config, runId, scriptDriver); + const lease = leaseOf(answer); + if (!lease) return answers; + answers.push(answer); + await reportAttempt(config, runId, { attempt: lease.attempt, revision: lease.revision, outcome: outcome(lease), driver: scriptDriver }); + } + throw new Error(`run ${runId} still leases steps after 50 reports`); +} + +describe('workflow guide examples', () => { + it('finds the English and Chinese workflow pages', () => { + expect(PAGES).toContain('docs/guide/workflow.md'); + expect(PAGES).toContain('docs/guide/zh-CN/workflow.md'); + expect(ADVANCED).toEqual(['docs/guide/workflow-advanced.md', 'docs/guide/zh-CN/workflow-advanced.md']); + }); + + it.each(PAGES)('%s: each yaml block parses, and its complete workflows pass pull together', async (page) => { + const config = teamRepo(); + const { byFile, failures } = writeExamples(page, config); + expect(byFile.size, `${page} has no complete workflow example`).toBeGreaterThan(0); + + const result = await syncWorkflowDefinitions(config, { dryRun: true }); + for (const diagnostic of result.invalid) { + failures.push(`${byFile.get(diagnostic.file) ?? page}: ${formatDiagnostic(diagnostic)}`); + } + for (const file of result.unrunnable) failures.push(`${byFile.get(file) ?? page}: ${file} has no runnable version`); + expect(failures).toEqual([]); + }); + + it('finds the --inputs lines of workflow start', () => { + expect(PAGES.flatMap((page) => startsOf(page).starts.map((start) => start.name))).toEqual(expect.arrayContaining(['fan-publish', 'fix'])); + }); + + it('declares in prose only workflows that a page starts without an example', () => { + const inProse = PAGES.flatMap((page) => { + const { byFile } = writeExamples(page, teamRepo()); + return startsOf(page).starts.filter((start) => !byFile.has(`workflows/${start.name}.yaml`)).map((start) => start.name); + }); + expect([...DECLARED_IN_PROSE].filter((name) => !inProse.includes(name))).toEqual([]); + }); + + it.each(PAGES)('%s: each workflow start --inputs fits the inputs its workflow declares on the page', async (page) => { + const config = teamRepo(); + const { byFile } = writeExamples(page, config); + const { starts, unread } = startsOf(page); + const failures = [...unread]; + const declared: Start[] = []; + for (const start of starts) { + if (!byFile.has(`workflows/${start.name}.yaml`)) { + const inputs = DECLARED_IN_PROSE.has(start.name) ? /`inputs: (\{[^`]*\})`/.exec(start.paragraph)?.[1] : undefined; + if (inputs === undefined) { + failures.push(`${start.label}: workflow "${start.name}" is not an example on the page`); + continue; + } + fs.writeFileSync(path.join(config.repo.localPath, 'workflows', `${start.name}.yaml`), + `schemaVersion: v1alpha1\nname: ${start.name}\ndescription: Declared in prose.\ninputs: ${inputs}\nsteps:\n - id: run\n command: node run.js\n`); + } + declared.push(start); + } + await syncWorkflowDefinitions(config); + for (const start of declared) { + let inputs: unknown; + try { + inputs = JSON.parse(start.json); + } catch { + failures.push(`${start.label}: --inputs is not JSON`); + continue; + } + const started = await startRun(config, start.name, inputs); + if ('error' in started) { + failures.push(`${start.label}: ${started.error.code}: ${started.error.message}`); + continue; + } + // Some inputs, such as the `after` of a for_each item, are read only when the first step is leased. + const next = await leaseNext(config, String(started.data.run), driver); + if (!('data' in next) || !next.data.lease) failures.push(`${start.label}: the first next leases nothing: ${JSON.stringify(next)}`); + } + expect(failures).toEqual([]); + }); + + it.each(ADVANCED)('%s: the next table answers what the engine answers', async (page) => { + const { config, runId } = await startExample(page, 'checks'); + const rows = [...fs.readFileSync(path.join(ROOT, page), 'utf8').matchAll(/^\| (.*) \| `(\{"data":\{"(?:lease|wait)".*\})` \|$/gm)]; + expect(rows).toHaveLength(3); + // The table's own attempt ids, in the `running` it sends, stand for the ones this run leased. + const attempts = new Map(); + for (const [, fields = '', shown = ''] of rows) { + const running = /`"running":(\[[^`]*\])`/.exec(fields)?.[1]; + const options = running === undefined ? {} : { running: (JSON.parse(running) as string[]).map((attempt) => attempts.get(attempt) ?? attempt) }; + const answer = await leaseNext(config, runId, scriptDriver, options); + const expected = JSON.parse(shown) as WorkflowEnvelope; + const [shownLease, lease] = [leaseOf(expected), leaseOf(answer)]; + if (shownLease && lease) attempts.set(shownLease.attempt, lease.attempt); + expect(withoutIds(answer), fields).toEqual(withoutIds(expected)); + } + }); + + it.each(ADVANCED)('%s: the second lease of e2e in loop-implement-task is what the engine leases', async (page) => { + const { config, runId } = await startExample(page, 'loop-implement-task'); + const shown = jsonOn(page, /^```text\n(\{"data":\{"lease".*\})\n```/gm); + expect(shown).toHaveLength(1); + + const answers = await driveExample(config, runId, (lease) => (lease.step === 'e2e' && lease.pass === 1 ? 'failed' : 'success')); + + expect(withoutIds(answers.filter((answer) => leaseOf(answer)?.step === 'e2e')[1])).toEqual(withoutIds(shown[0])); + }); + + it.each(ADVANCED)('%s: the for_each results are what fan-build renders, in item order', async (page) => { + const [shown] = jsonOn(page, /^```json\n(\[.*\])\n```/gm) as Array>; + if (!shown) throw new Error(`${page} shows no results`); + const { config, runId } = await startExample(page, 'fan-build', { packages: shown.map((entry) => entry.item) }); + + const answers = await driveExample(config, runId, (lease) => shown[Number(/\[(\d+)\]$/.exec(lease.step)?.[1])]?.outcome ?? 'success'); + + const prompt = answers.map(leaseOf).find((lease) => lease?.step === 'explain')?.prompt ?? ''; + expect(JSON.parse(prompt.slice(prompt.indexOf('[')))).toEqual(shown); + }); + + it.each(PAGES)('%s: every relative link to a .md file resolves', (page) => { + const text = fs.readFileSync(path.join(ROOT, page), 'utf8'); + const missing = [...text.matchAll(/\]\(([^)#\s:]+\.md)(?:#[^)]*)?\)/g)] + .map((match) => match[1]!) + .filter((target) => !fs.existsSync(path.join(ROOT, path.dirname(page), target))); + expect(missing).toEqual([]); + }); +}); diff --git a/src/__tests__/workflow-engine.test.ts b/src/__tests__/workflow-engine.test.ts index a4fc75729..ebd98b8fb 100644 --- a/src/__tests__/workflow-engine.test.ts +++ b/src/__tests__/workflow-engine.test.ts @@ -4,13 +4,14 @@ import path from 'node:path'; import { afterEach, describe, expect, it, vi } from 'vitest'; import type { LocalConfig } from '../types.js'; import { acquireLock, releaseLock } from '../update.js'; +import { log } from '../utils/logger.js'; import { syncWorkflowDefinitions } from '../workflow/cache.js'; import { leaseNext, parseReport, reportAttempt, SETTLED_ATTEMPTS_LIMIT, startRun, type ReportInput } from '../workflow/engine.js'; import { runLocalDriver } from '../workflow/local-driver.js'; import { outputMismatch } from '../workflow/output.js'; import { pendingRecords, type WorkflowRunRecord } from '../workflow/records.js'; -import { readRun, runLockPath, writeRun } from '../workflow/store.js'; -import type { DriverIdentity } from '../workflow/types.js'; +import { readRun, runFilePath, runLockPath, writeRun } from '../workflow/store.js'; +import type { DriverIdentity, RunState, TokenUsage } from '../workflow/types.js'; const dirs: string[] = []; @@ -145,10 +146,14 @@ describe('workflow engine', () => { const leased = await leaseNext(config, runId, driverA); if ('error' in leased) throw new Error(leased.error.message); const attempt = String((leased.data.lease as { attempt: string }).attempt); - const released = await reportAttempt(config, runId, { attempt, revision: 1, outcome: 'released', driver: driverA }); + const released = await reportAttempt(config, runId, { + attempt, revision: 1, outcome: 'released', driver: driverA, exitCode: 130, message: 'stopped', + }); expect(released).toMatchObject({ data: { accepted: true, step: { status: 'pending', outcome: 'released' } } }); const stored = await readRun(config, runId); - expect(stored).toMatchObject({ status: 'running', revision: 2, steps: [{ status: 'pending', outcome: 'released' }, { status: 'pending' }] }); + expect(stored).toMatchObject({ status: 'running', revision: 2, steps: [{ id: 'one', status: 'pending' }, { id: 'two', status: 'pending' }] }); + if (typeof stored !== 'object') throw new Error('run missing'); + expect(stored.steps[0]).toEqual({ id: 'one', status: 'pending' }); const again = await leaseNext(config, runId, driverA); expect(again).toMatchObject({ data: { lease: { step: 'one', revision: 2 } } }); @@ -198,7 +203,7 @@ describe('workflow engine', () => { const finishing = await startBuild(TWO_STEPS); const stored = await readRun(finishing.config, finishing.runId); if (typeof stored !== 'object') throw new Error('run missing'); - stored.steps = stored.steps.map((step) => ({ ...step, status: 'succeeded', outcome: 'success' })); + stored.steps = stored.steps.map((step) => ({ ...step, status: 'succeeded', outcome: 'success', settledAt: 0 })); await writeRun(finishing.config, stored); vi.setSystemTime(new Date('2026-01-01T00:02:00.000Z')); await leaseNext(finishing.config, finishing.runId, driverA); @@ -269,8 +274,8 @@ describe('workflow engine', () => { const stored = await readRun(config, runId); if (typeof stored !== 'object') throw new Error('run missing'); stored.steps = [ - { id: 'one', status: 'failed', outcome: 'failed' }, - { id: 'two', status: 'succeeded', outcome: 'success' }, + { id: 'one', status: 'failed', outcome: 'failed', settledAt: 0 }, + { id: 'two', status: 'succeeded', outcome: 'success', settledAt: 0 }, ]; await writeRun(config, stored); @@ -365,7 +370,7 @@ describe('workflow engine', () => { const stored = await readRun(config, runId); if (typeof stored !== 'object') throw new Error('run missing'); expect(stored.steps.filter((step) => step.status === 'leased')).toHaveLength(1); - expect(new Set(stored.steps.map((step) => step.lease?.attempt).filter(Boolean)).size).toBe(1); + expect(new Set(stored.steps.map((step) => (step.status === 'leased' ? step.lease.attempt : undefined)).filter(Boolean)).size).toBe(1); }); it('runs one command at a time through the same next and report operations', async () => { @@ -429,7 +434,7 @@ steps: const end = (await pendingRecords(config)).find((record) => record.phase === 'end'); expect(end).toBeDefined(); expect(end!.status).toBe('failed'); - expect(end!.stepCounts).toEqual({ total: 1, succeeded: 0, failed: 1, skipped: 0 }); + expect(end!.stepCounts).toEqual({ total: 1, succeeded: 0, failed: 1, skipped: 0, cancelled: 0 }); }); it('queues an end record when the run settles', async () => { @@ -443,7 +448,7 @@ steps: const end = queued.find((record) => record.phase === 'end'); expect(end).toBeDefined(); expect(end!.status).toBe('succeeded'); - expect(end!.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0 }); + expect(end!.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0, cancelled: 0 }); }); it.each([ @@ -463,9 +468,54 @@ steps: if ('error' in started) throw new Error(started.error.message); expect(started.data).not.toHaveProperty('record'); const runId = String(started.data.run); - expect(await runLocalDriver(config, runId, work)).toEqual({ data: { done: { run: runId, outcome: 'succeeded' } } }); - expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, outcome: 'succeeded' } } }); + expect(await runLocalDriver(config, runId, work)).toEqual({ data: { done: { run: runId, outcome: 'succeeded' }, cancelled: [] } }); + expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, outcome: 'succeeded' }, cancelled: [] } }); + expect(await pendingRecords(config)).toEqual([]); + }); + + it('reports no record for a run that settled with usage reporting off, after reporting is turned back on', async () => { + const { config, work } = testConfig(); + config.usageReportEnabled = false; + fs.writeFileSync(path.join(config.repo.localPath, 'workflows', 'build.yaml'), TWO_STEPS); + fs.writeFileSync(path.join(work, 'one.js'), "process.stdout.write(JSON.stringify({ ok: true }))\n"); + fs.writeFileSync(path.join(work, 'two.js'), "process.stdout.write(JSON.stringify({ ok: true }))\n"); + await syncWorkflowDefinitions(config); + const started = await startRun(config, 'build'); + if ('error' in started) throw new Error(started.error.message); + const runId = String(started.data.run); + await runLocalDriver(config, runId, work); + + config.usageReportEnabled = true; + expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, outcome: 'succeeded' }, cancelled: [] } }); + }); + + it('reports no record for a run whose records could not be queued', async () => { + const { config, work } = testConfig(); + fs.writeFileSync(path.join(config.repo.localPath, 'workflows', 'build.yaml'), TWO_STEPS); + fs.writeFileSync(path.join(work, 'one.js'), "process.stdout.write(JSON.stringify({ ok: true }))\n"); + fs.writeFileSync(path.join(work, 'two.js'), "process.stdout.write(JSON.stringify({ ok: true }))\n"); + await syncWorkflowDefinitions(config); + const queue = path.join(config.dataHome!, 'workflows', 'pending'); + fs.mkdirSync(path.dirname(queue), { recursive: true }); + fs.writeFileSync(queue, ''); + const warn = vi.spyOn(log, 'warn').mockImplementation(() => {}); + + const started = await startRun(config, 'build'); + if ('error' in started) throw new Error(started.error.message); + const runId = String(started.data.run); + const settling = await runLocalDriver(config, runId, work); + const later = await leaseNext(config, runId, driverA); + const warnings = warn.mock.calls.map(([message]) => message); + warn.mockRestore(); + + expect(later).toEqual({ data: { done: { run: runId, outcome: 'succeeded' }, cancelled: [] } }); + expect(settling).toEqual(later); + expect(started.data).not.toHaveProperty('record'); expect(await pendingRecords(config)).toEqual([]); + expect(warnings).toEqual([ + expect.stringContaining(`[workflows] Could not queue the start record of run ${runId}, so the team report will not include it: `), + expect.stringContaining(`[workflows] Could not queue the end record of run ${runId}, so the team report will not include it: `), + ]); }); it('reports the record state of the requested run, not of another queued run', async () => { @@ -480,7 +530,7 @@ steps: if ('error' in other) throw new Error(other.error.message); expect(other.data.record).toBe('pending'); - expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, outcome: 'succeeded', record: 'published' } } }); + expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, outcome: 'succeeded', record: 'published' }, cancelled: [] } }); }); it('counts the attempts a run settled before the iteration counter existed', async () => { @@ -692,7 +742,7 @@ steps: expect(leased).toEqual(order); expect(done).toEqual({ run: runId, record: 'pending', ...expected }); - expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, record: 'pending', ...expected } } }); + expect(await leaseNext(config, runId, driverA)).toEqual({ data: { done: { run: runId, record: 'pending', ...expected }, cancelled: [] } }); }); it('reaches a ready end before it leases a step declared earlier', async () => { @@ -787,6 +837,32 @@ steps: }); }); + it('reads an array element by index in an if, and an index out of range as null', async () => { + const { config, runId } = await startBuild(graph(` - id: review + command: node review.js + output: + files: [string] + - id: fix + needs: [review] + if: steps.review.outputs.files[1] == 'b.ts' + command: node fix.js + - id: third + needs: [review] + if: steps.review.outputs.files[2] != null + command: node third.js +`)); + const first = await leaseNext(config, runId, driverA); + if ('error' in first) throw new Error(first.error.message); + const lease = first.data.lease as { attempt: string; revision: number }; + await reportAttempt(config, runId, { ...lease, outcome: 'success', outputs: { files: ['a.ts', 'b.ts'] }, driver: driverA }); + + const { leased, done } = await drive(config, runId); + + expect(leased).toEqual(['fix']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [{}, { id: 'fix', status: 'succeeded' }, { id: 'third', status: 'skipped' }] }); + }); + it('evaluates an if only after every step it reads has settled', async () => { // gate is skipped as soon as build fails, while late is still pending; deploy reads late through gate. const { config, runId } = await startBuild(graph(` - id: build @@ -811,6 +887,40 @@ steps: expect(leased).toEqual(['build', 'docs', 'late', 'deploy']); }); + it('compares numbers and runs a predicate over a list, and waits for every step a predicate reads', async () => { + // gate is skipped as soon as build fails, while late is still pending; deploy reads late only inside its predicate. + const { config, runId } = await startBuild(graph(` - id: build + command: node build.js + - id: docs + needs: [] + command: node docs.js + output: + pages: [{ size: number }] + - id: gate + needs: [build, late] + command: node gate.js + - id: deploy + needs: [gate, docs] + if: length(steps.docs.outputs.pages) == 2 && steps.docs.outputs.pages[0].size < 10 && any(steps.docs.outputs.pages, .size > 10 && steps.late.outcome == 'success') + command: node deploy.js + - id: late + needs: [] + command: node late.js +`)); + const leased: string[] = []; + for (;;) { + const next = await leaseNext(config, runId, driverA); + if ('error' in next) throw new Error(next.error.message); + if (next.data.done) break; + const lease = next.data.lease as { step: string; attempt: string; revision: number }; + leased.push(lease.step); + const outputs = lease.step === 'docs' ? { pages: [{ size: 5 }, { size: 50 }] } : undefined; + await reportAttempt(config, runId, { attempt: lease.attempt, revision: lease.revision, outcome: lease.step === 'build' ? 'failed' : 'success', ...(outputs ? { outputs } : {}), driver: driverA }); + } + + expect(leased).toEqual(['build', 'docs', 'late', 'deploy']); + }); + it('evaluates an if against the run inputs', async () => { const fixture = testConfig(); fs.writeFileSync(path.join(fixture.config.repo.localPath, 'workflows', 'build.yaml'), graph(` - id: build @@ -1018,6 +1128,14 @@ steps: error: { code: 'run_unreadable', message: expect.stringContaining('step "label" has an if that does not parse') }, }); }); + + it('types the id of a step instance as read-only', () => { + const rename = (run: RunState): void => { + // @ts-expect-error The engine indexes instances by id, so an instance never changes its id. + run.steps[0]!.id = 'x'; + }; + expect(rename).toBeTypeOf('function'); + }); }); /** Writes each workflow file, syncs, and starts `name`. */ @@ -1096,7 +1214,7 @@ describe('file sub-workflows', () => { fs.writeFileSync(path.join(workflows, 'check.yaml'), CHECK); await syncWorkflowDefinitions(fixture.config); const restoreShip = saveCachedCopy(fixture.config, 'ship'); - fs.writeFileSync(path.join(workflows, 'check.yaml'), `${CHECK} for_each: [1]\n`); + fs.writeFileSync(path.join(workflows, 'check.yaml'), `${CHECK} timeout: 10m\n`); await syncWorkflowDefinitions(fixture.config); restoreShip(); @@ -1200,7 +1318,7 @@ describe('file sub-workflows', () => { const { leased } = await drive(config, runId, ['build']); expect(leased).toEqual(['build', 'docs', 'late', 'deploy.classify']); - expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([{ id: 'deploy', status: 'succeeded', outcome: 'success', inputs: { late: 'success' } }]) }); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([{ id: 'deploy', status: 'succeeded', outcome: 'success', inputs: { late: 'success' }, settledAt: expect.any(Number) }]) }); }); it('fails a workflow step with invalid_inputs when with does not fit the child inputs, and keeps settling', async () => { @@ -1265,7 +1383,7 @@ describe('file sub-workflows', () => { const done = await leaseNext(config, runId, driverA); expect(close).toMatchObject({ data: { lease: { step: 'close', env: { VERDICT: 'drop' } } } }); - expect(done).toEqual({ data: { done: { run: runId, record: 'pending', outcome: 'succeeded' } } }); + expect(done).toEqual({ data: { done: { run: runId, record: 'pending', outcome: 'succeeded' }, cancelled: [] } }); expect(await readRun(config, runId)).toMatchObject({ steps: [ { id: 'check', status: 'succeeded', end: 'dropped', outputs: { verdict: 'drop' } }, { id: 'check.classify', status: 'succeeded' }, @@ -1459,7 +1577,7 @@ describe('with on prompt steps', () => { `)); const run = await readRun(config, runId); if (typeof run !== 'object' || !run.definition.steps[0]?.with) throw new Error('run missing'); - run.definition.steps[0].with = { task: '{{ inputs.task == 1 }}' }; + run.definition.steps[0].with = { task: '{{ inputs.task = 1 }}' }; await writeRun(config, run); const { leased, done } = await drive(config, runId); @@ -1467,7 +1585,7 @@ describe('with on prompt steps', () => { expect(leased).toEqual(['label']); expect(done).toEqual({ run: runId, record: 'pending', outcome: 'failed' }); expect(await readRun(config, runId)).toMatchObject({ steps: [ - { id: 'fix', status: 'failed', failure: { code: 'invalid_template', message: 'step "fix" with template {{ inputs.task == 1 }}: a template reads one path' } }, + { id: 'fix', status: 'failed', failure: { code: 'invalid_template', message: 'step "fix" with template {{ inputs.task = 1 }}: unexpected "="' } }, { id: 'label', status: 'succeeded' }, ] }); }); @@ -1507,6 +1625,10 @@ describe('inline workflows', () => { expect(classify).toMatchObject({ data: { lease: { step: 'check.classify', prompt: 'Classify Bug for fix.' } } }); expect(leased).toEqual(['check.comment', 'ship']); expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error('run missing'); + // The inline workflow reads the enclosing inputs; its step does not store them. + expect(stored.steps.find((step) => step.id === 'check')).toEqual({ id: 'check', status: 'succeeded', outcome: 'success', settledAt: 4 }); }); it('finishes only the inline workflow at its end, and the parent reads its end and outputs', async () => { @@ -1634,6 +1756,29 @@ describe('inline workflows', () => { } }); }); + it('settles a workflow step whose only child its if skips, then leases the step after it, in serial order', async () => { + const { config, runId } = await startTree({ ship: workflow('ship', ` - id: w + workflow: + steps: + - id: only + if: inputs.skip + command: node only.js + - id: after + needs: [w] + command: node after.js +`, 'inputs:\n skip: boolean\n') }, 'ship', { skip: false }); + + const { leased, done } = await drive(config, runId); + + expect(leased).toEqual(['after']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'w', status: 'succeeded' }, + { id: 'w.only', status: 'skipped' }, + { id: 'after', status: 'succeeded' }, + ] }); + }); + it('reads its own step where an enclosing step has the same id', async () => { const { config, runId } = await startTree({ ship: workflow('ship', ` - id: plan command: node plan.js @@ -1705,10 +1850,10 @@ describe('run records of graph runs', () => { }); expect((await endRecord(config))?.steps).toEqual([ - { step: 'check', outcome: 'success' }, - { step: 'check.classify', agent: 'classifier', tool: 'claude', model: 'claude-sonnet', outcome: 'success', tokens: { input: 10, output: 5 } }, - { step: 'check.comment', outcome: 'success' }, - { step: 'fix', agent: 'implementer', tool: 'codex', model: 'gpt-codex', outcome: 'success', tokens: { input: 20, output: 8 } }, + { step: 'check', pass: 1, outcome: 'success' }, + { step: 'check.classify', pass: 1, agent: 'classifier', tool: 'claude', model: 'claude-sonnet', outcome: 'success', tokens: { input: 10, output: 5 } }, + { step: 'check.comment', pass: 1, outcome: 'success' }, + { step: 'fix', pass: 1, agent: 'implementer', tool: 'codex', model: 'gpt-codex', outcome: 'success', tokens: { input: 20, output: 8 } }, ]); }); @@ -1762,9 +1907,9 @@ describe('run records of graph runs', () => { await driveWith(config, runId); expect((await endRecord(config))?.steps).toEqual([ - { step: 'build', outcome: 'success' }, - { step: 'retry' }, - { step: 'done', outcome: 'success' }, + { step: 'build', pass: 1, outcome: 'success' }, + { step: 'retry', pass: 1 }, + { step: 'done', pass: 1, outcome: 'success' }, ]); }); @@ -1784,11 +1929,11 @@ describe('run records of graph runs', () => { const record = await endRecord(config); expect(record?.steps).toEqual([ - { step: 'review', outcome: 'success' }, - { step: 'review.read', agent: 'reviewer', outcome: 'success' }, - { step: 'check', outcome: 'failed' }, - { step: 'check.classify', outcome: 'success' }, - { step: 'check.comment', outcome: 'failed' }, + { step: 'review', pass: 1, outcome: 'success' }, + { step: 'review.read', pass: 1, agent: 'reviewer', outcome: 'success' }, + { step: 'check', pass: 1, outcome: 'failed' }, + { step: 'check.classify', pass: 1, outcome: 'success' }, + { step: 'check.comment', pass: 1, outcome: 'failed' }, ]); expect(JSON.stringify(record)).not.toContain('undefined'); expect(JSON.stringify(record)).not.toContain('comment failed'); @@ -1837,4 +1982,1920 @@ describe('run records of graph runs', () => { if (typeof stored !== 'object') throw new Error('run missing'); expect((await endRecord(config))?.endedAt).toBe(stored.updatedAt); }); + + it('publishes no failure code a driver reported', async () => { + const { config, runId } = await startBuild(TWO_STEPS); + + await driveWith(config, runId, { one: { outcome: 'failed', failure: { code: 'secret-x', message: 'secret-x happened' } } }); + + const record = await endRecord(config); + expect(record).toMatchObject({ status: 'failed' }); + expect(JSON.stringify(record)).not.toContain('secret-x'); + }); + + it('counts and marks cancelled steps, and publishes every pass a loop ran with its own tokens', async () => { + const fan = workflow('fan', ` - id: build + for_each: inputs.targets + command: node build.js + - id: check + needs: [build] + command: node check.js + loop: { back_to: build, while: "length(steps.check.passes) < 3", max_iterations: 3 } + - id: deploy + needs: [check] + for_each: { items: inputs.targets, max_parallel: 1 } + command: node deploy.js + - id: stop + needs: [check] + command: node stop.js + end: done +`, 'inputs:\n targets: [string]\nends:\n done: succeeded\n'); + const { config, runId } = await startTree({ fan }, 'fan', { targets: ['SENTINEL-web', 'SENTINEL-api'] }); + const reported: TokenUsage[] = []; + const running: string[] = []; + for (;;) { + const next = await leaseNext(config, runId, driverA, { running }); + if ('error' in next) throw new Error(next.error.message); + if (next.data.done) break; + const lease = leaseIn(next); + // A deploy item is still running when stop ends the workflow, which cancels it. + if (lease.step.startsWith('deploy')) { + running.push(lease.attempt); + continue; + } + const tokens = { input: 100 + reported.length, output: reported.length + 1 }; + reported.push(tokens); + await reportAttempt(config, runId, { ...lease, outcome: 'success', outputs: { note: 'SENTINEL-output' }, tokens, driver: driverA }); + } + const sum = (usages: Array): TokenUsage => usages.reduce( + (total, usage) => ({ input: (total.input ?? 0) + (usage?.input ?? 0), output: (total.output ?? 0) + (usage?.output ?? 0) }), + { input: 0, output: 0 }, + ); + + const record = await endRecord(config); + if (!record?.stepCounts || !record.steps) throw new Error('no end record'); + const { total, ...byStatus } = record.stepCounts; + const entries = record.steps; + + expect(reported).toHaveLength(10); + expect.soft(byStatus).toEqual({ succeeded: 5, failed: 0, skipped: 1, cancelled: 2 }); + expect.soft(total).toBe(Object.values(byStatus).reduce((all, count) => all + count, 0)); + expect.soft(sum(entries.map((entry) => entry.tokens))).toEqual(sum(reported)); + const passes = (step: string): typeof entries => entries.filter((entry) => entry.step === step).sort((a, b) => (a.pass ?? 0) - (b.pass ?? 0)); + expect(passes('check')).toEqual([2, 5, 8].map((index, pass) => ({ step: 'check', pass: pass + 1, outcome: 'success', tokens: reported[index] }))); + expect(passes('build[0]')).toEqual([0, 3, 6].map((index, pass) => ({ step: 'build[0]', pass: pass + 1, outcome: 'success', tokens: reported[index] }))); + expect(passes('stop')).toEqual([{ step: 'stop', pass: 1, outcome: 'success', tokens: reported[9] }]); + expect(passes('deploy')).toEqual([{ step: 'deploy', pass: 1, outcome: 'cancelled' }]); + expect(passes('deploy[0]')).toEqual([{ step: 'deploy[0]', pass: 1, outcome: 'cancelled' }]); + expect(passes('deploy[1]')).toEqual([{ step: 'deploy[1]', pass: 1 }]); + expect(JSON.stringify(record)).not.toContain('SENTINEL'); + }); + + it('gives its agent to the pass of an item that a later pass of the loop no longer has', async () => { + const fan = workflow('fan', ` - id: plan + command: node plan.js + output: + targets: [string] + - id: build + needs: [plan] + for_each: steps.plan.outputs.targets + workflow: + steps: + - id: review + prompt: "Review {{ item }}." + agent: reviewer + - id: check + needs: [build] + command: node check.js + loop: { back_to: plan, while: "length(steps.check.passes) < 2", max_iterations: 2 } +`); + const { config, runId } = await startTree({ fan }, 'fan'); + + const { leased } = await driveLoop(config, runId, (lease, earlier) => (lease.step === 'plan' ? { outputs: { targets: earlier === 0 ? ['web'] : [] } } : {})); + + expect(leased).toEqual(['plan#1', 'build[0].review#1', 'check#1', 'plan#2', 'check#2']); + expect((await endRecord(config))?.steps?.filter((entry) => entry.step === 'build[0].review')).toEqual([ + { step: 'build[0].review', pass: 1, agent: 'reviewer', outcome: 'success' }, + ]); + }); +}); + +/** The lease a `next` envelope hands out; throws on anything else. */ +function leaseIn(envelope: Awaited>): { step: string; attempt: string; revision: number } { + if ('error' in envelope || !envelope.data.lease) throw new Error(`expected a lease, got ${JSON.stringify(envelope)}`); + return envelope.data.lease as { step: string; attempt: string; revision: number }; +} + +describe('parallel leases', () => { + const ROOTS = graph(` - id: a + command: node a.js + - id: b + needs: [] + command: node b.js + - id: join + needs: [a, b] + command: node join.js +`); + + it('leases a second root while the driver lists the first as running', async () => { + const { config, runId } = await startBuild(ROOTS); + const a = leaseIn(await leaseNext(config, runId, driverA)); + + const b = await leaseNext(config, runId, driverA, { running: [a.attempt] }); + + expect(a.step).toBe('a'); + expect(b).toMatchObject({ data: { lease: { step: 'b', revision: 1 }, cancelled: [] } }); + expect(leaseIn(b).attempt).not.toBe(a.attempt); + }); + + it('hands its open lease back to a driver that lists no running attempts', async () => { + const { config, runId } = await startBuild(ROOTS); + const first = await leaseNext(config, runId, driverA); + + const again = await leaseNext(config, runId, driverA); + + expect(leaseIn(first).step).toBe('a'); + expect(again).toEqual(first); + }); + + it('lists as cancelled a running attempt that is not an open lease of the driver', async () => { + const { config, runId } = await startBuild(ROOTS); + + const next = await leaseNext(config, runId, driverA, { running: ['a00000000'] }); + + expect(next).toMatchObject({ data: { lease: { step: 'a' }, cancelled: ['a00000000'] } }); + }); + + it('hands an unlisted lease back to a restarted driver before a step declared earlier that became ready', async () => { + const { config, runId } = await startBuild(graph(` - id: gate + command: node gate.js + - id: step1 + needs: [gate] + command: node step1.js + - id: step2 + needs: [] + command: node step2.js +`)); + const gate = leaseIn(await leaseNext(config, runId, driverA)); + const step2 = await leaseNext(config, runId, driverA, { running: [gate.attempt] }); + await reportAttempt(config, runId, { ...gate, outcome: 'success', driver: driverA }); + + const restarted = await leaseNext(config, runId, driverA); + + expect(leaseIn(step2).step).toBe('step2'); + expect(restarted).toEqual(step2); + }); + + it('settles a workflow step whose only child its if skips, and leases what needs it while another lease runs', async () => { + const fixture = testConfig(); + fs.writeFileSync(path.join(fixture.config.repo.localPath, 'workflows', 'build.yaml'), graph(` - id: a + command: node a.js + - id: w + needs: [] + workflow: + steps: + - id: only + if: inputs.skip + command: node only.js + - id: after + needs: [w] + command: node after.js +`, 'inputs:\n skip: boolean\n')); + await syncWorkflowDefinitions(fixture.config); + const started = await startRun(fixture.config, 'build', { skip: false }); + if ('error' in started) throw new Error(started.error.message); + const runId = String(started.data.run); + const a = leaseIn(await leaseNext(fixture.config, runId, driverA)); + + const after = await leaseNext(fixture.config, runId, driverA, { running: [a.attempt] }); + + expect(after).toMatchObject({ data: { lease: { step: 'after' } } }); + expect(await readRun(fixture.config, runId)).toMatchObject({ steps: [ + { id: 'a', status: 'leased' }, + { id: 'w', status: 'succeeded' }, + { id: 'w.only', status: 'skipped' }, + { id: 'after', status: 'leased' }, + ] }); + }); + + it('checks a report against the revision of its own lease, not the run revision a sibling report made', async () => { + const { config, runId } = await startBuild(ROOTS); + const a = leaseIn(await leaseNext(config, runId, driverA)); + const b = leaseIn(await leaseNext(config, runId, driverA, { running: [a.attempt] })); + await reportAttempt(config, runId, { ...a, outcome: 'success', driver: driverA }); + + const stale = await reportAttempt(config, runId, { ...b, revision: 2, outcome: 'success', driver: driverA }); + + expect(stale).toEqual({ error: { code: 'stale_revision', message: `Revision 2 is stale; attempt ${b.attempt} was leased at 1` } }); + expect(await reportAttempt(config, runId, { ...b, outcome: 'success', driver: driverA })).toMatchObject({ data: { accepted: true, revision: 3 } }); + }); + + const W = ` - id: w + needs: [] + workflow: + steps: + - id: c1 + command: node c1.js + end: a +`; + const C2 = ` - id: c2 + needs: [] + command: node c2.js + end: b +`; + const C3 = ` - id: c3 + needs: [] + command: node c3.js +`; + it.each([ + ['the workflow step is declared first', `${W}${C2}${C3}`], + ['the other end is declared first', `${C2}${W}${C3}`], + ])('applies the end reported first when three reports arrive with no next between them and %s', async (_order, steps) => { + const { config, runId } = await startBuild(graph(steps, 'ends:\n a: succeeded\n b: failed\n')); + const leases = new Map(); + for (let i = 0; i < 3; i += 1) { + const lease = leaseIn(await leaseNext(config, runId, driverA, { running: [...leases.values()].map((item) => item.attempt) })); + leases.set(lease.step, lease); + } + for (const step of ['w.c1', 'c2', 'c3']) { + expect(await reportAttempt(config, runId, { ...leases.get(step)!, outcome: 'success', driver: driverA })).toMatchObject( + step === 'w.c1' ? { data: { accepted: true } } : { error: { code: 'lease_cancelled' } }, + ); + } + + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { run: runId, outcome: 'succeeded', end: 'a' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + expect.objectContaining({ id: 'w', status: 'succeeded', settledAt: 2 }), + expect.objectContaining({ id: 'c2', status: 'cancelled', settledAt: 2 }), + ]) }); + }); + + it('stamps a step it cannot render with when it became ready, not with when next ran', async () => { + const { config, runId } = await startBuild(graph(` - id: check + command: node check.js + - id: other + needs: [] + command: node other.js + - id: label + needs: [check] + command: node label.js + env: + TITLE: fixed +`)); + const run = await readRun(config, runId); + if (typeof run !== 'object') throw new Error('run missing'); + run.definition.steps[2]!.env = { TITLE: '{{ inputs.a b }}' }; + await writeRun(config, run); + const check = leaseIn(await leaseNext(config, runId, driverA)); + const other = leaseIn(await leaseNext(config, runId, driverA, { running: [check.attempt] })); + await reportAttempt(config, runId, { ...check, outcome: 'success', driver: driverA }); + await reportAttempt(config, runId, { ...other, outcome: 'success', driver: driverA }); + + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'check', settledAt: 2 }, + { id: 'other', settledAt: 3 }, + { id: 'label', status: 'failed', settledAt: 2, failure: { code: 'invalid_template' } }, + ] }); + }); + + it('stamps an open lease it cannot render again with the revision it was leased at', async () => { + const { config, runId } = await startBuild(ROOTS); + const a = leaseIn(await leaseNext(config, runId, driverA)); + const b = leaseIn(await leaseNext(config, runId, driverA, { running: [a.attempt] })); + await reportAttempt(config, runId, { ...b, outcome: 'success', driver: driverA }); + const run = await readRun(config, runId); + if (typeof run !== 'object') throw new Error('run missing'); + run.definition.steps[0]!.env = { TITLE: '{{ inputs.a b }}' }; + await writeRun(config, run); + + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'a', status: 'failed', settledAt: 1, failure: { code: 'invalid_template' } }, + { id: 'b', settledAt: 2 }, + { id: 'join', status: 'skipped' }, + ] }); + }); + + it('orders an end-only step by the report that reached it, not by when next settled it', async () => { + const { config, runId } = await startBuild(graph(` - id: check + command: node check.js + - id: other + needs: [] + command: node other.js + end: late + - id: handle + needs: [check] + end: early +`, 'ends:\n early: succeeded\n late: failed\n')); + const check = leaseIn(await leaseNext(config, runId, driverA)); + const other = leaseIn(await leaseNext(config, runId, driverA, { running: [check.attempt] })); + await reportAttempt(config, runId, { ...check, outcome: 'success', driver: driverA }); + await reportAttempt(config, runId, { ...other, outcome: 'success', driver: driverA }); + + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { outcome: 'succeeded', end: 'early' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'check', settledAt: 2 }, + { id: 'other', status: 'cancelled', settledAt: 2 }, + { id: 'handle', status: 'succeeded', settledAt: 2 }, + ] }); + }); + + it('reaches a finished workflow end past a ready step when the driver takes no lease, and cancels what still runs', async () => { + const { config, runId } = await startBuild(graph(` - id: trigger + command: node trigger.js + - id: spare + needs: [trigger] + command: node spare.js + - id: w + needs: [] + workflow: + steps: + - id: finish + command: node finish.js + end: done + - id: slow + needs: [] + command: node slow.js + - id: ask + needs: [] + prompt: Ask. +`, 'ends:\n done: succeeded\n')); + const leases = new Map(); + for (let i = 0; i < 4; i += 1) { + const lease = leaseIn(await leaseNext(config, runId, driverA, { running: [...leases.values()].map((item) => item.attempt) })); + leases.set(lease.step, lease); + } + expect([...leases.keys()]).toEqual(['trigger', 'w.finish', 'slow', 'ask']); + await reportAttempt(config, runId, { ...leases.get('trigger')!, outcome: 'success', driver: driverA }); + await reportAttempt(config, runId, { ...leases.get('w.finish')!, outcome: 'success', driver: driverA }); + const slow = leases.get('slow')!; + const ask = leases.get('ask')!; + + const reconciled = await leaseNext(config, runId, driverA, { running: [slow.attempt, ask.attempt], canTakeLease: false }); + + expect(reconciled).toEqual({ data: { done: { run: runId, record: 'pending', outcome: 'succeeded', end: 'done' }, cancelled: [slow.attempt, ask.attempt] } }); + expect(await readRun(config, runId)).toMatchObject({ + cancelledAttempts: [slow.attempt, ask.attempt], + steps: [ + { id: 'trigger', status: 'succeeded' }, + { id: 'spare', status: 'skipped' }, + { id: 'w', status: 'succeeded' }, + { id: 'w.finish', status: 'succeeded' }, + { id: 'slow', status: 'cancelled', attempt: slow.attempt, failure: { code: 'end_reached', message: 'Cancelled: workflow "build" reached end "done"' } }, + { id: 'ask', status: 'cancelled', attempt: ask.attempt }, + ], + }); + expect(await reportAttempt(config, runId, { ...slow, outcome: 'success', driver: driverB })).toEqual({ error: { + code: 'lease_cancelled', message: `Attempt ${slow.attempt} was cancelled; its report is not recorded`, + } }); + }); + + it('cancels only what runs under a sub-workflow that reaches its end, and the run goes on', async () => { + const { config, runId } = await startBuild(graph(` - id: w + workflow: + ends: + stopped: succeeded + outputs: + long: steps.long.outcome + steps: + - id: stop + command: node stop.js + end: stopped + - id: long + needs: [] + command: node long.js + - id: other + needs: [] + command: node other.js + - id: after + needs: [w] + command: node after.js + env: + LONG: "{{ steps.w.outputs.long }}" +`)); + const stop = leaseIn(await leaseNext(config, runId, driverA)); + const long = leaseIn(await leaseNext(config, runId, driverA, { running: [stop.attempt] })); + const other = leaseIn(await leaseNext(config, runId, driverA, { running: [stop.attempt, long.attempt] })); + await reportAttempt(config, runId, { ...stop, outcome: 'success', driver: driverA }); + + const after = await leaseNext(config, runId, driverA, { running: [long.attempt, other.attempt] }); + + expect([stop.step, long.step, other.step]).toEqual(['w.stop', 'w.long', 'other']); + expect(after).toMatchObject({ data: { lease: { step: 'after', env: { LONG: 'cancelled' } }, cancelled: [long.attempt] } }); + expect(await readRun(config, runId)).toMatchObject({ cancelledAttempts: [long.attempt], steps: [ + { id: 'w', status: 'succeeded', end: 'stopped', outputs: { long: 'cancelled' } }, + { id: 'w.stop', status: 'succeeded' }, + { id: 'w.long', status: 'cancelled', failure: { code: 'end_reached', message: 'Cancelled: step "w" inline workflow reached end "stopped"' } }, + { id: 'other', status: 'leased' }, + { id: 'after', status: 'leased' }, + ] }); + }); + + it('waits while every ready step is leased and listed as running', async () => { + const { config, runId } = await startBuild(ROOTS); + const a = leaseIn(await leaseNext(config, runId, driverA)); + const b = leaseIn(await leaseNext(config, runId, driverA, { running: [a.attempt] })); + + const next = await leaseNext(config, runId, driverA, { running: [a.attempt, b.attempt] }); + + expect(next).toEqual({ data: { wait: { reason: 'a is leased' }, cancelled: [] } }); + }); + + it('leases a later root while a workflow step waits for its leased child', async () => { + const { config, runId } = await startBuild(graph(` - id: w + workflow: + steps: + - id: x + command: node x.js + - id: c + needs: [] + command: node c.js +`)); + const x = leaseIn(await leaseNext(config, runId, driverA)); + + const c = await leaseNext(config, runId, driverA, { running: [x.attempt] }); + + expect(x.step).toBe('w.x'); + expect(c).toMatchObject({ data: { lease: { step: 'c' }, cancelled: [] } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'w', status: 'running' }, + { id: 'w.x', status: 'leased' }, + { id: 'c', status: 'leased' }, + ] }); + }); + + it('ends the run on a reported end before it hands an unlisted lease back to a restarted driver', async () => { + const { config, runId } = await startBuild(graph(` - id: slow + command: node slow.js + - id: finish + needs: [] + command: node finish.js + end: done +`, 'ends:\n done: succeeded\n')); + const slow = leaseIn(await leaseNext(config, runId, driverA)); + const finish = leaseIn(await leaseNext(config, runId, driverA, { running: [slow.attempt] })); + await reportAttempt(config, runId, { ...finish, outcome: 'success', driver: driverA }); + + const restarted = await leaseNext(config, runId, driverA); + + expect(restarted).toMatchObject({ data: { done: { run: runId, outcome: 'succeeded', end: 'done' } } }); + expect(await readRun(config, runId)).toMatchObject({ cancelledAttempts: [slow.attempt], steps: [{ id: 'slow', status: 'cancelled' }, { id: 'finish' }] }); + expect(await reportAttempt(config, runId, { ...slow, outcome: 'success', driver: driverA })).toMatchObject({ error: { code: 'lease_cancelled' } }); + }); + + it('settles and saves without leasing when the driver takes no lease', async () => { + const { config, runId } = await startBuild(graph(` - id: a + command: node a.js + - id: w + needs: [] + workflow: + steps: + - id: x + command: node x.js + - id: after + needs: [w] + command: node after.js +`)); + const a = leaseIn(await leaseNext(config, runId, driverA)); + const x = leaseIn(await leaseNext(config, runId, driverA, { running: [a.attempt] })); + await reportAttempt(config, runId, { ...x, outcome: 'success', driver: driverA }); + + const reconciled = await leaseNext(config, runId, driverA, { running: [a.attempt], canTakeLease: false }); + + expect(reconciled).toEqual({ data: { wait: { reason: 'a is leased' }, cancelled: [] } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'a', status: 'leased' }, + { id: 'w', status: 'succeeded' }, + { id: 'w.x', status: 'succeeded' }, + { id: 'after', status: 'pending' }, + ] }); + expect(await leaseNext(config, runId, driverA, { running: [a.attempt] })).toMatchObject({ data: { lease: { step: 'after' } } }); + }); +}); + +describe('causal settlement order', () => { + it.each(['workflow', 'for_each'])('checks an end-only condition after a %s step settles', async (action) => { + const build = action === 'workflow' + ? ' workflow:\n steps:\n - id: run\n command: echo build\n' + : ' for_each: inputs.items\n command: echo build\n'; + const body = graph(` - id: build +${build} - id: stop + needs: [build] + if: steps.build.outcome == 'failed' + end: broke + - id: after + needs: [build] + command: echo after +`, 'ends: { broke: failed }\ninputs: { items: [number] }\n'); + const { config } = testConfig(); + fs.writeFileSync(path.join(config.repo.localPath, 'workflows', 'build.yaml'), body); + expect((await syncWorkflowDefinitions(config)).invalid).toEqual([]); + const started = await startRun(config, 'build', { items: [1] }); + if ('error' in started) throw new Error(started.error.message); + const runId = String(started.data.run); + const built = leaseIn(await leaseNext(config, runId, driverA)); + expect(built.step).toBe(action === 'workflow' ? 'build.run' : 'build[0]'); + await reportAttempt(config, runId, { ...built, outcome: 'success', driver: driverA }); + const after = leaseIn(await leaseNext(config, runId, driverA)); + expect(after.step).toBe('after'); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + expect.objectContaining({ id: 'stop', status: 'skipped' }), + ]) }); + await reportAttempt(config, runId, { ...after, outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + }); + + it('stamps a capped item with the report that freed its slot', async () => { + const build = workflow('build', ` - id: plan + command: echo plan + - id: build + needs: [plan] + for_each: { items: steps.plan.outputs.items, max_parallel: 1 } + workflow: child + with: { task: "{{ item.task }}" } +`); + const child = workflow('child', ' - id: run\n command: echo child\n', 'inputs: { task: string }\n'); + const { config, runId } = await startTree({ build, child }, 'build'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { items: [{ task: 'valid' }, { task: 1 }] }, driver: driverA }); + const first = leaseIn(await leaseNext(config, runId, driverA)); + expect(first.step).toBe('build[0].run'); + await reportAttempt(config, runId, { ...first, outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA, { canTakeLease: false })).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + expect.objectContaining({ id: 'build[1]', status: 'failed', settledAt: 3, failure: expect.objectContaining({ code: 'invalid_inputs' }) }), + ]) }); + }); + + it('keeps downstream reports identical when next reconciles a nested end between reports', async () => { + const run = async (between: boolean) => { + const { config, runId } = await startBuild(graph(` - id: inner + workflow: + ends: { stop: succeeded } + outputs: { review: steps.review.outputs.v, fix: steps.fix.outcome } + steps: + - id: review + command: echo review + - id: fix + needs: [review] + command: echo fix + loop: { back_to: review, max_iterations: 3 } + - id: guard + needs: [] + command: echo guard + end: stop + - id: after + needs: [inner] + command: echo after + env: { VALUE: "{{ steps.inner.outputs }}" } +`)); + const review = leaseIn(await leaseNext(config, runId, driverA)); + const guard = leaseIn(await leaseNext(config, runId, driverA, { running: [review.attempt] })); + await reportAttempt(config, runId, { ...review, outcome: 'success', outputs: { v: 'one' }, driver: driverA }); + const fix = leaseIn(await leaseNext(config, runId, driverA, { running: [guard.attempt] })); + await reportAttempt(config, runId, { ...guard, outcome: 'success', driver: driverA }); + if (between) await leaseNext(config, runId, driverA, { running: [fix.attempt], canTakeLease: false }); + await reportAttempt(config, runId, { ...fix, outcome: 'success', driver: driverA }); + const after = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...after, outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error(`run ${stored}`); + const names = new Map([review, guard, fix, after].map((lease) => [lease.attempt, lease.step])); + return JSON.parse(JSON.stringify(stored.steps).replace(/a[0-9a-f]{8}/g, (attempt) => names.get(attempt) ?? attempt)); + }; + expect(await run(false)).toEqual(await run(true)); + }); + + it.each([ + ['invalid_items', ' for_each: steps.plan.outputs.items\n command: echo build\n', { items: 1 }], + ['item_cycle', ' for_each: steps.plan.outputs.items\n command: echo build\n', { items: [{ id: 'a', after: ['b'] }, { id: 'b', after: ['a'] }] }], + ['invalid_template', ' command: echo build\n env: { TITLE: fixed }\n', {}], + ['invalid_inputs', ' workflow: child\n with: { task: "{{ steps.plan.outputs.task }}" }\n', { task: 1 }], + ])('reconciles %s before a later end even without capacity for a lease', async (code, action, outputs) => { + const body = workflow('build', ` - id: plan + command: echo plan + - id: build + needs: [plan] +${action} - id: failed + needs: [build] + if: steps.build.outcome == 'failed' + end: failed + - id: fast + needs: [] + command: echo fast + end: fast +`, 'ends: { failed: failed, fast: succeeded }\n'); + const child = workflow('child', ' - id: run\n command: echo child\n', 'inputs: { task: string }\n'); + const { config, runId } = await startTree({ build: body, child }, 'build'); + if (code === 'invalid_template') { + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error(`run ${stored}`); + stored.definition.steps[1]!.env = { TITLE: '{{ inputs.bad path }}' }; + await writeRun(config, stored); + } + const plan = leaseIn(await leaseNext(config, runId, driverA)); + const fast = leaseIn(await leaseNext(config, runId, driverA, { running: [plan.attempt] })); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs, driver: driverA }); + expect(await reportAttempt(config, runId, { ...fast, revision: fast.revision + 1, outcome: 'success', driver: driverB })) + .toMatchObject({ error: { code: 'lease_cancelled' } }); + await reportAttempt(config, runId, { ...fast, outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA, { running: [], canTakeLease: false })).toMatchObject({ data: { done: { outcome: 'failed', end: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + expect.objectContaining({ id: 'build', status: 'failed', settledAt: 2, failure: { code, message: expect.any(String) } }), + expect.objectContaining({ id: 'fast', status: 'cancelled', settledAt: 2 }), + ]) }); + expect(await reportAttempt(config, runId, { ...fast, outcome: 'success', driver: driverA })).toMatchObject({ error: { code: 'lease_cancelled' } }); + }); + + it('carries the loop report revision through a driverless exhausted pass', async () => { + const { config, runId } = await startBuild(graph(` - id: review + command: echo review + - id: fix + needs: [review] + command: echo fix + loop: { back_to: review, max_iterations: 1 } + - id: failed + needs: [fix] + if: steps.fix.outcome == 'failed' + end: failed + - id: fast + needs: [] + command: echo fast + end: fast +`, 'ends: { failed: failed, fast: succeeded }\n')); + const review = leaseIn(await leaseNext(config, runId, driverA)); + const fast = leaseIn(await leaseNext(config, runId, driverA, { running: [review.attempt] })); + await reportAttempt(config, runId, { ...review, outcome: 'success', driver: driverA }); + const fix = leaseIn(await leaseNext(config, runId, driverA, { running: [fast.attempt] })); + await reportAttempt(config, runId, { ...fix, outcome: 'success', driver: driverA }); + const secondReview = leaseIn(await leaseNext(config, runId, driverA, { running: [fast.attempt] })); + await reportAttempt(config, runId, { ...secondReview, outcome: 'success', driver: driverA }); + await reportAttempt(config, runId, { ...fast, outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA, { running: [], canTakeLease: false })).toMatchObject({ data: { done: { outcome: 'failed', end: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + expect.objectContaining({ id: 'fix', status: 'failed', settledAt: 4, failure: expect.objectContaining({ code: 'loop_exhausted' }) }), + expect.objectContaining({ id: 'fast', status: 'cancelled', settledAt: 4 }), + ]) }); + }); + + it.each([ + { + name: 'an engine failure reaches an end before a later report', + body: graph(` - id: plan + command: echo plan + - id: build + needs: [plan] + for_each: steps.plan.outputs.items + command: echo build + - id: stop + needs: [build] + if: steps.build.outcome == 'failed' + end: broke + - id: fast + needs: [] + command: echo fast + end: won +`, 'ends: { won: succeeded, broke: failed }\n'), + leases: ['plan', 'fast'], + reports: [ + { step: 'plan', outcome: 'success', outputs: { items: 'not a list' } }, + { step: 'fast', outcome: 'success' }, + ], + done: { outcome: 'failed', end: 'broke' }, + }, + { + name: 'a late skip cannot reach an end before an earlier report', + body: graph(` - id: a + command: echo a + - id: direct + needs: [] + command: echo direct + end: direct + - id: b + needs: [] + command: echo b + - id: skipped + needs: [b] + command: echo skip + - id: derived + needs: [a, skipped] + if: steps.skipped.outcome == 'skipped' + end: after_skip +`, 'ends: { direct: succeeded, after_skip: failed }\n'), + leases: ['a', 'direct', 'b'], + reports: [ + { step: 'a', outcome: 'success' }, + { step: 'direct', outcome: 'success' }, + { step: 'b', outcome: 'failed' }, + ], + done: { outcome: 'succeeded', end: 'direct' }, + }, + { + name: 'a later loop report preserves the results before the end', + body: graph(` - id: review + command: echo review + - id: fix + needs: [review] + command: echo fix + loop: { back_to: review, max_iterations: 3 } + - id: guard + needs: [] + command: echo guard + end: stop +`, 'ends: { stop: succeeded }\noutputs: { review: steps.review.outputs.v, fix: steps.fix.outcome }\n'), + leases: ['review', 'guard', 'fix'], + reports: [ + { step: 'review', outcome: 'success', outputs: { v: 'one' } }, + { step: 'guard', outcome: 'success' }, + { step: 'fix', outcome: 'success' }, + ], + done: { outcome: 'succeeded', end: 'stop' }, + }, + ] satisfies Array<{ + name: string; + body: string; + leases: string[]; + reports: Array<{ step: string; outcome: 'success' | 'failed'; outputs?: unknown }>; + done: { outcome: string; end: string }; + }>)('gives the same done and stored steps with next between reports when $name', async ({ body, leases, reports, done }) => { + const run = async (between: boolean) => { + const { config, runId } = await startBuild(body); + const held = new Map>(); + const running = () => [...held.values()].map((lease) => lease.attempt); + let reported = 0; + for (const step of leases) { + if (step === 'fix') { + const first = reports[reported++]!; + await reportAttempt(config, runId, { ...held.get(first.step)!, ...first, driver: driverA }); + } + const next = leaseIn(await leaseNext(config, runId, driverA, { running: running() })); + expect(next.step).toBe(step); + held.set(step, next); + } + for (const report of reports.slice(reported)) { + await reportAttempt(config, runId, { ...held.get(report.step)!, ...report, driver: driverA }); + if (between) await leaseNext(config, runId, driverA, { running: running(), canTakeLease: false }); + } + const next = await leaseNext(config, runId, driverA, { running: running(), canTakeLease: false }); + expect(next).toMatchObject({ data: { done } }); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error(`run ${stored}`); + const normalize = (value: unknown) => JSON.parse(JSON.stringify(value).replaceAll(runId, '').replace(/a[0-9a-f]{8}/g, (attempt) => + [...held.values()].find((lease) => lease.attempt === attempt)?.step ?? attempt)); + if ('error' in next) throw new Error(next.error.message); + return { done: normalize(next.data.done), steps: normalize(stored.steps) }; + }; + const interleaved = await run(true); + const batched = await run(false); + expect(batched).toEqual(interleaved); + }); +}); + +describe('a run file written before loops and for_each', () => { + // Written through `workflow next` and `report`: build succeeded, check ran an inline workflow + // whose end-only step the engine settled, and ship was leased, then released. + const FIXTURE = path.join(__dirname, 'fixtures', 'workflow', 'run-before-loops.json'); + + it('reads the run and continues it', async () => { + const { config } = testConfig(); + const runId = 'rdc7b5bd3'; + fs.mkdirSync(path.dirname(runFilePath(config, runId)), { recursive: true }); + fs.copyFileSync(FIXTURE, runFilePath(config, runId)); + + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error(`run ${stored}`); + expect(stored.steps).toEqual([ + { id: 'build', status: 'succeeded', outcome: 'success', exitCode: 0, outputs: { artifact: 'app.tgz' }, settledAt: 0 }, + { id: 'check', status: 'succeeded', outcome: 'success', end: 'ready', outputs: { verdict: 'ship' }, settledAt: 0 }, + { id: 'check.classify', status: 'succeeded', outcome: 'success', exitCode: 0, outputs: { verdict: 'ship' }, settledAt: 0 }, + { id: 'check.done', status: 'succeeded', outcome: 'success', settledAt: 0 }, + { id: 'ship', status: 'pending' }, + ]); + expect(stored).toMatchObject({ revision: 4, cancelledAttempts: [], archive: [] }); + + const next = await leaseNext(config, runId, driverA); + expect(next).toMatchObject({ data: { lease: { step: 'ship', revision: 4, env: { ARTIFACT: 'app.tgz' } } } }); + if ('error' in next) return; + const lease = next.data.lease as { attempt: string; revision: number }; + const reported = await reportAttempt(config, runId, { ...lease, outcome: 'success', driver: driverA, exitCode: 0 }); + expect(reported).toMatchObject({ data: { accepted: true, revision: 5, step: { id: 'ship', status: 'succeeded' } } }); + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { done: { run: runId, outcome: 'succeeded' } } }); + }); +}); + +interface LoopLease { + step: string; + attempt: string; + revision: number; + pass: number; + command?: string; + prompt?: string; + env?: Record; +} + +/** + * Drives a run serially. `answer` reports each lease, given how many times its step has been leased before. + * Returns each lease as `#`, and the done envelope. + */ +async function driveLoop( + config: LocalConfig, + runId: string, + answer: (lease: LoopLease, earlier: number) => { outcome?: 'success' | 'failed'; outputs?: unknown } = () => ({}), +): Promise<{ leased: string[]; done: unknown }> { + const leased: string[] = []; + const counts = new Map(); + for (;;) { + const next = await leaseNext(config, runId, driverA); + if ('error' in next) throw new Error(next.error.message); + if (next.data.done) return { leased, done: next.data.done }; + const lease = next.data.lease as LoopLease; + leased.push(`${lease.step}#${lease.pass}`); + const earlier = counts.get(lease.step) ?? 0; + counts.set(lease.step, earlier + 1); + const { outcome = 'success', outputs } = answer(lease, earlier); + const reported = await reportAttempt(config, runId, { attempt: lease.attempt, revision: lease.revision, outcome, outputs, driver: driverA }); + if ('error' in reported) throw new Error(reported.error.message); + } +} + +const REVIEW_AND_FIX = workflow('review-and-fix', ` - id: review + command: node review.js + output: + verdict: pass | fix + findings: [string] + - id: fix + needs: [review] + if: steps.review.outputs.verdict == 'fix' + command: node fix.js + output: + patch: string + loop: { back_to: review, max_iterations: 3 } +`, 'outputs:\n findings: steps.review.passes\n'); + +function reviews(...verdicts: Array<'pass' | 'fix'>) { + return (lease: LoopLease, earlier: number) => (lease.step.endsWith('review') + ? { outputs: { verdict: verdicts[earlier], findings: [`finding ${earlier + 1}`] } } + : { outputs: { patch: `patch ${earlier + 1}` } }); +} + +const IMPLEMENT_TASK = workflow('implement-task', ` - id: implement + command: node implement.js + - id: e2e + needs: [implement] + command: npm run test:e2e + - id: fix + needs: [e2e] + if: steps.e2e.outcome == 'failed' + command: node fix.js + loop: { back_to: e2e, max_iterations: 3 } + - id: done + needs: [e2e] + if: steps.e2e.outcome == 'success' + end: done + - id: give-up + needs: [fix] + if: steps.fix.outcome == 'failed' + end: stuck +`, 'ends:\n done: succeeded\n stuck: failed\n'); + +const PLAN_CHANGE = workflow('plan-change', ` - id: plan + command: node plan.js + output: + summary: string + - id: decide + needs: [plan] + command: node decide.js + output: + choice: build | revise | drop + note: string + loop: { back_to: plan, while: "steps.decide.outputs.choice == 'revise'", max_iterations: 3 } + - id: accept + needs: [decide] + if: steps.decide.outputs.choice == 'build' + end: planned + - id: drop + needs: [decide] + if: steps.decide.outputs.choice == 'drop' + end: dropped +`, 'ends:\n planned: succeeded\n dropped: succeeded\n'); + +function decisions(...choices: Array<'build' | 'revise' | 'drop'>) { + return (lease: LoopLease, earlier: number) => (lease.step === 'plan' + ? { outputs: { summary: `plan ${earlier + 1}` } } + : { outputs: { choice: choices[earlier], note: `note ${earlier + 1}` } }); +} + +describe('step loops', () => { + it('plans again while the person asks for a revision, and builds the revised plan', async () => { + const { config, runId } = await startTree({ 'plan-change': PLAN_CHANGE }, 'plan-change'); + + const { leased, done } = await driveLoop(config, runId, decisions('revise', 'build')); + + expect(leased).toEqual(['plan#1', 'decide#1', 'plan#2', 'decide#2']); + expect(done).toMatchObject({ outcome: 'succeeded', end: 'planned' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', pass: 2, outputs: { summary: 'plan 2' } }, + { id: 'decide', status: 'succeeded', pass: 2, iteration: 2, loopFinal: true, outputs: { choice: 'build' } }, + { id: 'accept', status: 'succeeded' }, + { id: 'drop', status: 'skipped' }, + ] }); + }); + + it('hands plan the note of the previous decision, and nothing on the first pass', async () => { + const planChange = PLAN_CHANGE.replace(' command: node plan.js\n', ' command: node plan.js\n env:\n NOTE: "{{ steps.decide.outputs.note }}"\n'); + const { config, runId } = await startTree({ 'plan-change': planChange }, 'plan-change'); + const notes: Array = []; + const answer = decisions('revise', 'revise', 'build'); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => { + if (lease.step === 'plan') notes.push(lease.env?.NOTE); + return answer(lease, earlier); + }); + + expect(leased).toEqual(['plan#1', 'decide#1', 'plan#2', 'decide#2', 'plan#3', 'decide#3']); + expect(notes).toEqual(['', 'note 1', 'note 2']); + expect(done).toMatchObject({ outcome: 'succeeded', end: 'planned' }); + }); + + it('keeps an inner loop closed when the step it reads in the previous pass settles later, and reads that pass next time', async () => { + const reopen = workflow('reopen', ` - id: A + command: node a.js + - id: W + needs: [A] + workflow: + steps: + - id: p + command: node p.js + - id: q + needs: [p] + command: node q.js + loop: { back_to: p, while: "steps.C.outputs.again && length(steps.q.passes) < 2", max_iterations: 3 } + - id: C + needs: [W] + command: node c.js + output: + again: boolean + - id: D + needs: [C] + command: node d.js + output: + more: boolean + loop: { back_to: A, while: steps.D.outputs.more, max_iterations: 2 } +`); + const { config, runId } = await startTree({ reopen }, 'reopen'); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => (lease.step === 'C' + ? { outputs: { again: earlier === 0 } } + : lease.step === 'D' ? { outputs: { more: earlier === 0 } } : {})); + + expect(leased).toEqual(['A#1', 'W.p#1', 'W.q#1', 'C#1', 'D#1', 'A#2', 'W.p#2', 'W.q#2', 'W.p#3', 'W.q#3', 'C#2', 'D#2']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + }); + + it('fails decide with loop_exhausted when the last pass still asks for a revision, keeping that answer', async () => { + const { config, runId } = await startTree({ 'plan-change': PLAN_CHANGE }, 'plan-change'); + + const { leased, done } = await driveLoop(config, runId, decisions('revise', 'revise', 'revise')); + + expect(leased).toEqual(['plan#1', 'decide#1', 'plan#2', 'decide#2', 'plan#3', 'decide#3']); + expect(done).toMatchObject({ outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', status: 'succeeded', pass: 3 }, + { + id: 'decide', status: 'failed', pass: 3, iteration: 3, attempt: expect.any(String), outputs: { choice: 'revise', note: 'note 3' }, + failure: { code: 'loop_exhausted' }, + }, + { id: 'accept', status: 'skipped' }, + { id: 'drop', status: 'skipped' }, + ] }); + }); + + it('decides the loop of a workflow step once its workflow finishes, then runs what needs it', async () => { + const { config, runId } = await startBuild(graph(` - id: check + workflow: + outputs: + again: steps.run.outputs.again + steps: + - id: run + command: node check.js + output: + again: boolean + loop: { back_to: check, while: steps.check.outputs.again, max_iterations: 3 } + - id: ship + needs: [check] + command: node ship.js +`)); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => (lease.step === 'check.run' ? { outputs: { again: earlier === 0 } } : {})); + + expect(leased).toEqual(['check.run#1', 'check.run#2', 'ship#1']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ + steps: [ + { id: 'check', status: 'succeeded', pass: 2, iteration: 2, loopFinal: true, outputs: { again: false }, passes: [{ pass: 1, outcome: 'success', outputs: { again: true } }] }, + { id: 'check.run', status: 'succeeded', pass: 2, outputs: { again: false } }, + { id: 'ship', status: 'succeeded' }, + ], + archive: [{ step: 'check.run', pass: 1, outcome: 'success', attempt: expect.any(String), outputs: { again: true } }], + }); + }); + + it('leases the child of a workflow step loop without while once when max_iterations is 1, and fails the step', async () => { + const { config, runId } = await startBuild(graph(` - id: check + workflow: + steps: + - id: run + command: node check.js + loop: { back_to: check, max_iterations: 1 } + - id: ship + needs: [check] + command: node ship.js +`)); + + const { leased, done } = await driveLoop(config, runId); + + expect(leased).toEqual(['check.run#1']); + expect(done).toMatchObject({ outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ + steps: [ + { id: 'check', status: 'failed', pass: 2, iteration: 1, failure: { code: 'loop_exhausted' }, passes: [{ pass: 1, outcome: 'success' }] }, + { id: 'ship', status: 'skipped' }, + ], + archive: [{ step: 'check.run', pass: 1, outcome: 'success' }], + }); + }); + + it.each([ + ['passes on the first try', 1, ['implement#1', 'e2e#1'], { outcome: 'succeeded', end: 'done' }], + ['passes after one fix', 2, ['implement#1', 'e2e#1', 'fix#1', 'e2e#2'], { outcome: 'succeeded', end: 'done' }], + ['still fails after three fixes', 5, ['implement#1', 'e2e#1', 'fix#1', 'e2e#2', 'fix#2', 'e2e#3', 'fix#3', 'e2e#4'], { outcome: 'failed', end: 'stuck' }], + ] as const)('ends implement-task when e2e %s, and nothing outside the loop reads e2e before the loop settles', async (_case, passingRun, leases, outcome) => { + const { config, runId } = await startTree({ 'implement-task': IMPLEMENT_TASK }, 'implement-task'); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => ({ + outcome: lease.step === 'e2e' && earlier + 1 < passingRun ? 'failed' : 'success', + })); + + expect(leased).toEqual(leases); + expect(done).toMatchObject(outcome); + }); + + it('runs review again after a fix, and leases each pass with its number', async () => { + const { config, runId } = await startTree({ 'review-and-fix': REVIEW_AND_FIX }, 'review-and-fix'); + + const { leased, done } = await driveLoop(config, runId, reviews('fix', 'pass')); + + expect(leased).toEqual(['review#1', 'fix#1', 'review#2']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error('run missing'); + expect(stored.steps).toMatchObject([ + { + id: 'review', status: 'succeeded', pass: 2, outputs: { verdict: 'pass' }, + passes: [{ pass: 1, outcome: 'success', attempt: expect.any(String), outputs: { verdict: 'fix', findings: ['finding 1'] } }], + }, + { id: 'fix', status: 'skipped', pass: 2, iteration: 1, passes: [{ pass: 1, outcome: 'success', outputs: { patch: 'patch 1' } }] }, + ]); + }); + + it('succeeds when the fourth review passes after three fixes', async () => { + const { config, runId } = await startTree({ 'review-and-fix': REVIEW_AND_FIX }, 'review-and-fix'); + + const { leased, done } = await driveLoop(config, runId, reviews('fix', 'fix', 'fix', 'pass')); + + expect(leased).toEqual(['review#1', 'fix#1', 'review#2', 'fix#2', 'review#3', 'fix#3', 'review#4']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + }); + + it('fails the loop owner with loop_exhausted instead of a fourth fix, and keeps the outputs of the third', async () => { + const { config, runId } = await startTree({ 'review-and-fix': REVIEW_AND_FIX }, 'review-and-fix'); + + const { leased, done } = await driveLoop(config, runId, reviews('fix', 'fix', 'fix', 'fix')); + + expect(leased).toEqual(['review#1', 'fix#1', 'review#2', 'fix#2', 'review#3', 'fix#3', 'review#4']); + expect(done).toMatchObject({ outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'review', status: 'succeeded', pass: 4 }, + { + id: 'fix', status: 'failed', pass: 4, iteration: 3, outputs: { patch: 'patch 3' }, + failure: { code: 'loop_exhausted', message: 'Loop of step "fix" is exhausted: it ran 3 times, its max_iterations' }, + }, + ] }); + }); + + it.each([ + ['the first review passes', ['pass'], [{ pass: 1, outcome: 'success', outputs: { verdict: 'pass', findings: ['finding 1'] } }]], + ['a fix comes before the review that passes', ['fix', 'pass'], [ + { pass: 1, outcome: 'success', outputs: { verdict: 'fix', findings: ['finding 1'] } }, + { pass: 2, outcome: 'success', outputs: { verdict: 'pass', findings: ['finding 2'] } }, + ]], + ] as const)('returns every review in findings, once each, when %s', async (_case, verdicts, findings) => { + const { config, runId } = await startTree({ + 'review-and-fix': REVIEW_AND_FIX, + ship: workflow('ship', ` - id: review + workflow: review-and-fix +`), + }, 'ship'); + + const { done } = await driveLoop(config, runId, reviews(...verdicts)); + + expect(done).toMatchObject({ outcome: 'succeeded' }); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error('run missing'); + expect(stored.steps.find((step) => step.id === 'review')).toMatchObject({ status: 'succeeded', outputs: { findings } }); + }); +}); + +const ROUNDS = workflow('rounds', ` - id: check + workflow: + steps: + - id: review + command: node review.js + output: + verdict: string + - id: decide + needs: [check] + command: node decide.js + output: + again: boolean +`, 'loop: { while: steps.decide.outputs.again, max_iterations: 3 }\n'); + +function rounds(...again: boolean[]) { + return (lease: LoopLease, earlier: number) => (lease.step === 'decide' + ? { outputs: { again: again[earlier] } } + : { outputs: { verdict: `R${earlier + 1}` } }); +} + +describe('workflow loops', () => { + it('runs the workflow again while its loop holds, and keeps every earlier pass in the run file', async () => { + const { config, runId } = await startTree({ rounds: ROUNDS }, 'rounds'); + + const { leased, done } = await driveLoop(config, runId, rounds(true, false)); + + expect(leased).toEqual(['check.review#1', 'decide#1', 'check.review#2', 'decide#2']); + expect(done).toMatchObject({ outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ + round: 2, + steps: [ + { id: 'check', status: 'succeeded', pass: 2, passes: [{ pass: 1, outcome: 'success' }] }, + { id: 'check.review', status: 'succeeded', pass: 2, outputs: { verdict: 'R2' } }, + { id: 'decide', status: 'succeeded', pass: 2, outputs: { again: false }, passes: [{ pass: 1, outcome: 'success', outputs: { again: true } }] }, + ], + archive: [{ step: 'check.review', pass: 1, outcome: 'success', outputs: { verdict: 'R1' } }], + }); + }); + + it('fails the run with loop_exhausted when the loop still holds after its last round', async () => { + const { config, runId } = await startTree({ rounds: ROUNDS }, 'rounds'); + + const { leased, done } = await driveLoop(config, runId, rounds(true, true, true)); + + expect(leased).toEqual(['check.review#1', 'decide#1', 'check.review#2', 'decide#2', 'check.review#3', 'decide#3']); + expect(done).toEqual({ + run: runId, + record: 'pending', + outcome: 'failed', + failure: { code: 'loop_exhausted', message: 'Loop of workflow "rounds" is exhausted: it ran 3 rounds, its max_iterations' }, + }); + }); + + it('publishes the code of the run failure in the end record, without its message', async () => { + const { config, runId } = await startTree({ rounds: ROUNDS }, 'rounds'); + + const { done } = await driveLoop(config, runId, rounds(true, true, true)); + + const { message } = (done as { failure: { message: string } }).failure; + const record = await endRecord(config); + expect(record?.failure).toEqual({ code: 'loop_exhausted' }); + expect(JSON.stringify(record)).not.toContain(message); + }); + + it.each([ + ['its step loop ran out', { 'review-and-fix': REVIEW_AND_FIX }, 'review-and-fix', reviews('fix', 'fix', 'fix', 'fix')], + ['the step loop of a sub-workflow ran out', { + 'review-and-fix': REVIEW_AND_FIX, + ship: workflow('ship', ` - id: review + workflow: review-and-fix +`), + }, 'ship', reviews('fix', 'fix', 'fix', 'fix')], + ['the looping workflow of a step ran out', { retry: workflow('retry', ` - id: attempt + workflow: + loop: { while: "steps.try.outcome == 'success'", max_iterations: 2 } + steps: + - id: try + command: node try.js +`) }, 'retry', () => ({})], + ])('publishes loop_exhausted without its message when a run fails because %s', async (_case, files, name, answer) => { + const { config, runId } = await startTree(files, name); + + const { done } = await driveLoop(config, runId, answer); + + expect(done).toMatchObject({ outcome: 'failed' }); + const record = await endRecord(config); + expect(record?.failure).toEqual({ code: 'loop_exhausted' }); + expect(JSON.stringify(record)).not.toContain('is exhausted'); + }); + + it('runs another round after a round with a failed step, and ends with the outcome of the last round', async () => { + const again = workflow('again', ` - id: work + command: node work.js +`, "loop: { while: \"steps.work.outcome == 'failed'\", max_iterations: 3 }\n"); + const { config, runId } = await startTree({ again }, 'again'); + + const { leased, done } = await driveLoop(config, runId, (_lease, earlier) => ({ outcome: earlier === 0 ? 'failed' : 'success' })); + + expect(leased).toEqual(['work#1', 'work#2']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + }); + + it('stops the loop at a succeeded end, with that end', async () => { + const shipLoop = workflow('ship-loop', ` - id: build + command: node build.js + output: + ready: boolean + - id: ship + needs: [build] + if: steps.build.outputs.ready + end: shipped +`, 'ends:\n shipped: succeeded\nloop: { max_iterations: 3 }\n'); + const { config, runId } = await startTree({ 'ship-loop': shipLoop }, 'ship-loop'); + + const { leased, done } = await driveLoop(config, runId, (_lease, earlier) => ({ outputs: { ready: earlier === 1 } })); + + expect(leased).toEqual(['build#1', 'build#2']); + expect(done).toMatchObject({ outcome: 'succeeded', end: 'shipped' }); + }); + + it('ends the loop as success when a break holds after its step succeeds, and skips what has not run', async () => { + const poll = workflow('poll', ` - id: probe + command: node probe.js + output: + status: string + break: steps.probe.outputs.status == 'ready' + - id: notify + needs: [probe] + command: node notify.js +`, 'loop: { max_iterations: 5 }\n'); + const { config, runId } = await startTree({ poll }, 'poll'); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => (lease.step === 'probe' + ? { outputs: { status: earlier === 1 ? 'ready' : 'waiting' } } + : {})); + + expect(leased).toEqual(['probe#1', 'notify#1', 'probe#2']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'probe', status: 'succeeded', pass: 2, outputs: { status: 'ready' } }, + { id: 'notify', status: 'skipped', pass: 2 }, + ] }); + }); + + it('fails the step that runs a looping workflow with loop_exhausted after its last round, keeping that round\'s outputs', async () => { + const retry = workflow('retry', ` - id: attempt + workflow: + outputs: + result: steps.try.outputs.result + loop: { while: "steps.try.outputs.result != 'ok'", max_iterations: 2 } + steps: + - id: try + command: node try.js + output: + result: string + - id: after + needs: [attempt] + command: node after.js +`); + const { config, runId } = await startTree({ retry }, 'retry'); + + const { leased, done } = await driveLoop(config, runId, (_lease, earlier) => ({ outputs: { result: `flaky ${earlier + 1}` } })); + + expect(leased).toEqual(['attempt.try#1', 'attempt.try#2']); + expect(done).toMatchObject({ outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { + id: 'attempt', status: 'failed', round: 2, outputs: { result: 'flaky 2' }, + failure: { code: 'loop_exhausted', message: 'Loop of step "attempt" inline workflow is exhausted: it ran 2 rounds, its max_iterations' }, + }, + { id: 'attempt.try', status: 'succeeded', pass: 2 }, + { id: 'after', status: 'skipped' }, + ] }); + }); +}); + +const FAN_PLAN = workflow('fan', ` - id: plan + command: node plan.js + output: + targets: [string] + - id: build + needs: [plan] + for_each: steps.plan.outputs.targets + command: node build.js + env: + TARGET: "{{ item }}" + - id: ship + needs: [build] + command: node ship.js +`); + +const FAN_TASKS = workflow('fan', ` - id: plan + command: node plan.js + - id: build + needs: [plan] + for_each: steps.plan.outputs.tasks + command: node build.js + env: + TASK: "{{ item.id }}" + - id: ship + needs: [build] + command: node ship.js +`); + +describe('for_each', () => { + it('keeps fan-out results in previous-pass reads and in every exposed pass', async () => { + const fan = workflow('fan', ` - id: plan + command: echo plan + env: + PRIOR: "{{ steps.build.results }}" + PASSES: "{{ steps.build.passes }}" + - id: build + needs: [plan] + for_each: steps.plan.outputs.targets + command: echo build + output: { ok: boolean } + - id: decide + needs: [build] + command: echo decide + loop: { back_to: plan, while: "length(steps.decide.passes) < 2", max_iterations: 2 } + - id: summary + needs: [decide] + command: echo summary + env: + PASSES: "{{ steps.build.passes }}" +`); + const { config, runId } = await startTree({ fan }, 'fan'); + const plans: Array = []; + let summary: LoopLease['env']; + const result = [{ item: 'a', outcome: 'success', outputs: { ok: true } }]; + await driveLoop(config, runId, (lease) => { + if (lease.step === 'plan') plans.push(lease.env); + if (lease.step === 'summary') summary = lease.env; + return lease.step === 'plan' ? { outputs: { targets: ['a'] } } + : lease.step === 'build[0]' ? { outputs: { ok: true } } : {}; + }); + expect(plans).toEqual([ + { PRIOR: '', PASSES: '[]' }, + { PRIOR: JSON.stringify(result), PASSES: JSON.stringify([{ pass: 1, outcome: 'success', results: result }]) }, + ]); + expect(JSON.parse(summary?.PASSES ?? 'null')).toEqual([ + { pass: 1, outcome: 'success', results: result }, + { pass: 2, outcome: 'success', results: result }, + ]); + }); + + it('runs a command once per item, as build[0] and build[1] with the item in env, and keeps results in item order', async () => { + const fan = workflow('fan', ` - id: build + for_each: inputs.targets + command: node build.js + env: + TARGET: "{{ item }}" +`, 'inputs:\n targets: [string]\n'); + const { config, runId } = await startTree({ fan }, 'fan', { targets: ['web', 'api'] }); + const targets: Array = []; + + const { leased, done } = await driveLoop(config, runId, (lease) => { + targets.push(lease.env?.TARGET); + return {}; + }); + + expect(leased).toEqual(['build[0]#1', 'build[1]#1']); + expect(targets).toEqual(['web', 'api']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'build', status: 'succeeded', results: [{ item: 'web', outcome: 'success' }, { item: 'api', outcome: 'success' }] }, + { id: 'build[0]', status: 'succeeded', item: 'web' }, + { id: 'build[1]', status: 'succeeded', item: 'api' }, + ] }); + }); + + it('leases item as written from a step without for_each, and fills it in a for_each step of the same workflow', async () => { + const fixture = testConfig(); + const knowledge = fixture.config.repo.localPath; + fs.mkdirSync(path.join(knowledge, 'prompts')); + const quote = 'Quote {{ item }}, {{ item.name }}, {{ item[0] }} and {{ length(item.xs) }} from a playbook.'; + fs.writeFileSync(path.join(knowledge, 'prompts', 'quote.md'), quote); + fs.writeFileSync(path.join(knowledge, 'workflows', 'fan.yaml'), workflow('fan', ` - id: explain + prompt: "Explain {{ item }}, {{ item.name }}, {{ item[0] }} and {{ length(item.xs) }} in a Jinja loop: {{ with.loop }}." + with: + loop: ["{{ item }}", "{{ length(item.xs) }}"] + - id: quote + needs: [explain] + prompt_file: prompts/quote.md + - id: play + needs: [quote] + command: ansible localhost -m debug -a 'msg={{ item }}' + env: + ITEM: "{{ item }}" + COUNT: "{{ length(item.xs) }}" + - id: build + needs: [play] + for_each: inputs.packages + prompt: "Build {{ item.dir }}." +`, 'inputs:\n packages: [{ dir: string }]\n')); + await syncWorkflowDefinitions(fixture.config); + const started = await startRun(fixture.config, 'fan', { packages: [{ dir: 'packages/core' }] }); + if ('error' in started) throw new Error(started.error.message); + const leases: Array> = []; + + await driveLoop(fixture.config, String(started.data.run), ({ step, command, prompt, env }) => { + leases.push({ step, ...(command ? { command } : {}), ...(prompt ? { prompt } : {}), ...(env ? { env } : {}) }); + return {}; + }); + + expect(leases).toEqual([ + { step: 'explain', prompt: 'Explain {{ item }}, {{ item.name }}, {{ item[0] }} and {{ length(item.xs) }} in a Jinja loop: ["{{ item }}","{{ length(item.xs) }}"].' }, + { step: 'quote', prompt: quote }, + { step: 'play', command: "ansible localhost -m debug -a 'msg={{ item }}'", env: { ITEM: '{{ item }}', COUNT: '{{ length(item.xs) }}' } }, + { step: 'build[0]', prompt: 'Build packages/core.' }, + ]); + }); + + it('evaluates an expression in a prompt, a prompt_file, env and with, and leaves other braces as written', async () => { + const fixture = testConfig(); + const knowledge = fixture.config.repo.localPath; + fs.mkdirSync(path.join(knowledge, 'prompts')); + fs.writeFileSync(path.join(knowledge, 'prompts', 'review.md'), "Review {{ length(steps.build.results) }} builds; {{ filter(steps.build.results, .outcome == 'failed') }} failed."); + const kept = "Keep {{ filter(items) }}, {{ all(products) }}, {{ items | length }}, ${{ length(steps.build.results) }}, {{#each items}}{{this}}{{/each}}, {{{ x }}} and {{ item }} as written."; + fs.writeFileSync(path.join(knowledge, 'workflows', 'triage.yaml'), workflow('triage', ` - id: build + for_each: inputs.packages + command: node build.js + env: + PACKAGE: "{{ item.id }}" + LAST: "{{ item.id == 'cli' }}" + - id: notify + needs: [build] + if: steps.build.outcome == 'failed' + command: node notify.js + env: + FAILED: "{{ filter(steps.build.results, .outcome == 'failed') }}" + COUNT: "{{ length(steps.build.results) }}" + ANY: "{{ any(steps.build.results, .outcome == 'failed') }}" + MANY: "{{ length(steps.build.results) > 3 }}" + BROKE: "{{ steps.build.outcome == 'failed' }}" + - id: fix + needs: [notify] + with: + failed: "{{ filter(steps.build.results, .outcome == 'failed') }}" + summary: "{{ length(filter(steps.build.results, .outcome == 'failed')) }} of {{ length(steps.build.results) }}" + prompt: "Fix {{ with.failed[0].item.id }} ({{ with.summary }} failed): {{ with.failed }}" + - id: review + needs: [fix] + prompt_file: prompts/review.md + - id: quote + needs: [review] + prompt: "${kept}" +`, 'inputs:\n packages: [{ id: string, dir: string }]\n')); + const synced = await syncWorkflowDefinitions(fixture.config); + expect(synced.invalid.map((diagnostic) => diagnostic.message)).toEqual([]); + const started = await startRun(fixture.config, 'triage', { packages: [{ id: 'core', dir: 'packages/core' }, { id: 'cli', dir: 'packages/cli' }] }); + if ('error' in started) throw new Error(started.error.message); + const leases: Array> = []; + + const { done } = await driveLoop(fixture.config, String(started.data.run), ({ step, prompt, env }) => { + leases.push({ step, ...(prompt ? { prompt } : {}), ...(env ? { env } : {}) }); + return step === 'build[1]' ? { outcome: 'failed' } : {}; + }); + + const failed = '[{"item":{"id":"cli","dir":"packages/cli"},"outcome":"failed"}]'; + expect(leases).toEqual([ + { step: 'build[0]', env: { PACKAGE: 'core', LAST: 'false' } }, + { step: 'build[1]', env: { PACKAGE: 'cli', LAST: 'true' } }, + { step: 'notify', env: { FAILED: failed, COUNT: '2', ANY: 'true', MANY: 'false', BROKE: 'true' } }, + { step: 'fix', prompt: `Fix cli (1 of 2 failed): ${failed}` }, + { step: 'review', prompt: `Review 2 builds; ${failed} failed.` }, + { step: 'quote', prompt: kept }, + ]); + expect(done).toEqual({ run: String(started.data.run), record: 'pending', outcome: 'failed' }); + }); + + it('renders a read of a skipped step as an if evaluates it: a list it does not have is null, so a count is empty and a comparison false', async () => { + const { config, runId } = await startTree({ lint: workflow('lint', ` - id: lint + if: inputs.lint + for_each: inputs.packages + command: node lint.js + - id: plan + needs: [] + command: node plan.js + - id: report + needs: [plan, lint] + prompt: "Lint {{ steps.lint.outcome }}: [{{ length(steps.lint.results) }}] checked, any failed [{{ any(steps.lint.results, .outcome == 'failed') }}], over one {{ length(steps.lint.results) > 1 }}, first [{{ steps.lint.results[0].item.id }}], failed [{{ filter(steps.lint.results, .outcome == 'failed') }}]." +`, 'inputs:\n packages: [{ id: string }]\n lint: boolean\n') }, 'lint', { packages: [{ id: 'core' }], lint: false }); + const prompts: string[] = []; + + const { leased } = await driveLoop(config, runId, ({ prompt }) => { + if (prompt) prompts.push(prompt); + return {}; + }); + + expect(leased).toEqual(['plan#1', 'report#1']); + expect(prompts).toEqual(['Lint skipped: [] checked, any failed [], over one false, first [], failed [].']); + }); + + it('checks each item against output, keeps its outputs in results, and a later step reads them by index', async () => { + const sized = workflow('sized', ` - id: build + for_each: inputs.targets + command: node build.js + output: + size: number + - id: report + needs: [build] + if: steps.build.results[1].outcome == 'failed' + command: node report.js + env: + FIRST: "{{ steps.build.results[0].outputs.size }}" +`, 'inputs:\n targets: [string]\n'); + const { config, runId } = await startTree({ sized }, 'sized', { targets: ['web', 'api'] }); + const env: Array | undefined> = []; + + const { leased, done } = await driveLoop(config, runId, (lease) => { + env.push(lease.env); + return { outputs: { size: lease.step === 'build[0]' ? 3 : 'big' } }; + }); + + expect(leased).toEqual(['build[0]#1', 'build[1]#1', 'report#1']); + expect(env.at(-1)).toEqual({ FIRST: '3' }); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'build', status: 'failed', results: [ + { item: 'web', outcome: 'success', outputs: { size: 3 } }, + { item: 'api', outcome: 'failed' }, + ] }, + { id: 'build[0]', status: 'succeeded', outputs: { size: 3 } }, + { id: 'build[1]', status: 'failed', failure: { code: 'invalid_output', message: 'stdout JSON does not match output: output.size must be a number' } }, + { id: 'report', status: 'succeeded' }, + ] }); + }); + + it('runs again only the items that failed, through a for_each that filters the results of an earlier step', async () => { + const retry = workflow('retry', ` - id: build + for_each: inputs.targets + command: node build.js + env: + TARGET: "{{ item }}" + - id: again + needs: [build] + if: steps.build.outcome == 'failed' + for_each: filter(steps.build.results, .outcome == 'failed') + command: node build.js + env: + TARGET: "{{ item.item }}" +`, 'inputs:\n targets: [string]\n'); + const { config, runId } = await startTree({ retry }, 'retry', { targets: ['web', 'api', 'cli'] }); + const targets: Array = []; + + const { leased } = await driveLoop(config, runId, (lease) => { + targets.push(lease.env?.TARGET); + return { outcome: lease.step === 'build[1]' ? 'failed' : 'success' }; + }); + + expect(leased).toEqual(['build[0]#1', 'build[1]#1', 'build[2]#1', 'again[0]#1']); + expect(targets).toEqual(['web', 'api', 'cli', 'api']); + expect(await readRun(config, runId)).toMatchObject({ steps: expect.arrayContaining([ + { id: 'again', status: 'succeeded', outcome: 'success', settledAt: 5, results: [{ item: { item: 'api', outcome: 'failed' }, outcome: 'success' }] }, + ]) }); + }); + + it('runs an inline workflow per item under ids such as build[2].verify, and its steps read the item', async () => { + const inline = workflow('inline', ` - id: build + for_each: inputs.targets + workflow: + outputs: + size: steps.verify.outputs.size + steps: + - id: compile + command: node compile.js + env: + TARGET: "{{ item.name }}" + - id: verify + needs: [compile] + if: item.check + command: node verify.js + env: + TARGET: "{{ item.name }}" + output: + size: number +`, 'inputs:\n targets: [{ name: string, check: boolean }]\n'); + const targets = [{ name: 'web', check: true }, { name: 'api', check: false }, { name: 'cli', check: true }]; + const { config, runId } = await startTree({ inline }, 'inline', { targets }); + const leases: string[] = []; + + const { leased, done } = await driveLoop(config, runId, (lease) => { + leases.push(`${lease.step}:${lease.env?.TARGET}`); + return lease.step.endsWith('verify') ? { outputs: { size: leases.length } } : {}; + }); + + expect(leased).toEqual(['build[0].compile#1', 'build[0].verify#1', 'build[1].compile#1', 'build[2].compile#1', 'build[2].verify#1']); + expect(leases).toEqual(['build[0].compile:web', 'build[0].verify:web', 'build[1].compile:api', 'build[2].compile:cli', 'build[2].verify:cli']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + const run = await readRun(config, runId) as RunState; + expect(run.steps.map((step) => `${step.id}:${step.status}`)).toEqual([ + 'build:succeeded', + 'build[0]:succeeded', 'build[0].compile:succeeded', 'build[0].verify:succeeded', + 'build[1]:succeeded', 'build[1].compile:succeeded', 'build[1].verify:skipped', + 'build[2]:succeeded', 'build[2].compile:succeeded', 'build[2].verify:succeeded', + ]); + expect(run.steps[0]).toMatchObject({ results: [ + { item: targets[0], outcome: 'success', outputs: { size: 2 } }, + { item: targets[1], outcome: 'success', outputs: { size: null } }, + { item: targets[2], outcome: 'success', outputs: { size: 5 } }, + ] }); + }); + + it('passes a whole item to a workflow file through with, keeping its JSON type', async () => { + const ship = workflow('ship', ` - id: build + for_each: inputs.tasks + workflow: implement + with: + task: "{{ item }}" +`, 'inputs:\n tasks: [{ id: string, title: string }]\n'); + const implement = workflow('implement', ` - id: work + command: node work.js + env: + TITLE: "{{ inputs.task.title }}" +`, 'inputs:\n task: { id: string, title: string }\n'); + const tasks = [{ id: 'a', title: 'Parse' }, { id: 'b', title: 'Render' }]; + const { config, runId } = await startTree({ ship, implement }, 'ship', { tasks }); + const titles: Array = []; + + const { leased, done } = await driveLoop(config, runId, (lease) => { + titles.push(lease.env?.TITLE); + return {}; + }); + + expect(leased).toEqual(['build[0].work#1', 'build[1].work#1']); + expect(titles).toEqual(['Parse', 'Render']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'build', status: 'succeeded' }, + { id: 'build[0]', status: 'succeeded', item: tasks[0], inputs: { task: tasks[0] } }, + { id: 'build[0].work', status: 'succeeded' }, + { id: 'build[1]', status: 'succeeded', item: tasks[1], inputs: { task: tasks[1] } }, + { id: 'build[1].work', status: 'succeeded' }, + ] }); + }); + + it('archives the items of a round when its workflow loops, and numbers the next round\'s items after them', async () => { + const rounds = workflow('rounds', ` - id: build + for_each: inputs.targets + command: node build.js + output: + round: number + - id: decide + needs: [build] + command: node decide.js + output: + again: boolean +`, 'inputs:\n targets: [string]\nloop: { while: steps.decide.outputs.again, max_iterations: 2 }\n'); + const { config, runId } = await startTree({ rounds }, 'rounds', { targets: ['web', 'api'] }); + + const { leased, done } = await driveLoop(config, runId, (lease, earlier) => (lease.step === 'decide' + ? { outputs: { again: earlier === 0 } } + : { outputs: { round: earlier + 1 } })); + + expect(leased).toEqual(['build[0]#1', 'build[1]#1', 'decide#1', 'build[0]#2', 'build[1]#2', 'decide#2']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + const run = await readRun(config, runId) as RunState; + expect(run.steps).toMatchObject([ + { id: 'build', pass: 2, results: [{ item: 'web', outputs: { round: 2 } }, { item: 'api', outputs: { round: 2 } }], passes: [ + { pass: 1, outcome: 'success', results: [{ item: 'web', outcome: 'success', outputs: { round: 1 } }, { item: 'api', outcome: 'success', outputs: { round: 1 } }] }, + ] }, + { id: 'build[0]', pass: 2, item: 'web', outputs: { round: 2 } }, + { id: 'build[1]', pass: 2, item: 'api', outputs: { round: 2 } }, + { id: 'decide', pass: 2 }, + ]); + expect(run.archive).toEqual([ + expect.objectContaining({ step: 'build[0]', pass: 1, outcome: 'success', outputs: { round: 1 } }), + expect.objectContaining({ step: 'build[1]', pass: 1, outcome: 'success', outputs: { round: 1 } }), + ]); + }); + + it('succeeds an empty fan-out with no results in the same next, and runs what needs it', async () => { + const { config, runId } = await startTree({ fan: FAN_PLAN }, 'fan'); + + const { leased, done } = await driveLoop(config, runId, (lease) => (lease.step === 'plan' ? { outputs: { targets: [] } } : {})); + + expect(leased).toEqual(['plan#1', 'ship#1']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'succeeded' }); + expect((await readRun(config, runId) as RunState).steps).toEqual([ + { id: 'plan', status: 'succeeded', outcome: 'success', attempt: expect.any(String), outputs: { targets: [] }, settledAt: 2 }, + { id: 'build', status: 'succeeded', outcome: 'success', results: [], settledAt: 2 }, + { id: 'ship', status: 'succeeded', outcome: 'success', attempt: expect.any(String), settledAt: 3 }, + ]); + }); + + it('fails the step with invalid_items when for_each gives no list, and skips what needs it', async () => { + const { config, runId } = await startTree({ fan: FAN_PLAN.replace(' output:\n targets: [string]\n', '') }, 'fan'); + + const { leased, done } = await driveLoop(config, runId, () => ({ outputs: { targets: 'web' } })); + + expect(leased).toEqual(['plan#1']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', status: 'succeeded' }, + { id: 'build', status: 'failed', failure: { code: 'invalid_items', message: 'step "build" for_each gives a string, not a list of items' } }, + { id: 'ship', status: 'skipped' }, + ] }); + }); + + it.each([ + ['an unknown id', [{ id: 'a', after: [] }, { id: 'b', after: ['x'] }], 'step "build" item 1 runs after "x", which is not the id of an item'], + ['a duplicate id', [{ id: 'a' }, { id: 'b' }, { id: 'a' }], 'step "build" items 0 and 2 have the same id "a"'], + ['after that is not a list of ids', [{ id: 'a', after: 'b' }, { id: 'b' }], 'step "build" item 0 after must be a list of item ids'], + ])('fails the step with invalid_items for %s, before any item runs', async (_case, tasks, message) => { + const { config, runId } = await startTree({ fan: FAN_TASKS }, 'fan'); + + const { leased, done } = await driveLoop(config, runId, () => ({ outputs: { tasks } })); + + expect(leased).toEqual(['plan#1']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', status: 'succeeded' }, + { id: 'build', status: 'failed', failure: { code: 'invalid_items', message } }, + { id: 'ship', status: 'skipped' }, + ] }); + }); + + it('starts at most max_parallel items at once, and gives the next one to a driver that lists the ones it runs', async () => { + const capped = FAN_PLAN.replace('for_each: steps.plan.outputs.targets', 'for_each: { items: steps.plan.outputs.targets, max_parallel: 2 }'); + const { config, runId } = await startTree({ fan: capped }, 'fan'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { targets: ['web', 'api', 'cli'] }, driver: driverA }); + + const web = leaseIn(await leaseNext(config, runId, driverA)); + const api = leaseIn(await leaseNext(config, runId, driverA, { running: [web.attempt] })); + const full = await leaseNext(config, runId, driverA, { running: [web.attempt, api.attempt] }); + await reportAttempt(config, runId, { ...web, outcome: 'success', driver: driverA }); + const cli = await leaseNext(config, runId, driverA, { running: [api.attempt] }); + + expect([web.step, api.step]).toEqual(['build[0]', 'build[1]']); + expect(full).toEqual({ data: { wait: { reason: 'build[0] is leased' }, cancelled: [] } }); + expect(cli).toMatchObject({ data: { lease: { step: 'build[2]', env: { TARGET: 'cli' } }, cancelled: [] } }); + }); + + it('hands a restarted driver its item leases back with the same ids, attempts and env, one per call', async () => { + const { config, runId } = await startTree({ fan: FAN_PLAN }, 'fan'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { targets: ['web', 'api'] }, driver: driverA }); + const web = await leaseNext(config, runId, driverA); + const api = await leaseNext(config, runId, driverA, { running: [leaseIn(web).attempt] }); + + const webAgain = await leaseNext(config, runId, driverA, { running: [] }); + const apiAgain = await leaseNext(config, runId, driverA, { running: [leaseIn(webAgain).attempt] }); + + expect(webAgain).toEqual(web); + expect(apiAgain).toEqual(api); + expect(apiAgain).toMatchObject({ data: { lease: { step: 'build[1]', env: { TARGET: 'api' } } } }); + await reportAttempt(config, runId, { ...leaseIn(webAgain), outcome: 'success', driver: driverA }); + await reportAttempt(config, runId, { ...leaseIn(apiAgain), outcome: 'success', driver: driverA }); + expect(await leaseNext(config, runId, driverA)).toMatchObject({ data: { lease: { step: 'ship' } } }); + }); + + it('accepts the report of an item leased beside a sibling whose report came first', async () => { + const { config, runId } = await startTree({ fan: FAN_PLAN }, 'fan'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { targets: ['web', 'api'] }, driver: driverA }); + const web = leaseIn(await leaseNext(config, runId, driverA)); + const api = leaseIn(await leaseNext(config, runId, driverA, { running: [web.attempt] })); + + const apiReport = await reportAttempt(config, runId, { ...api, outcome: 'success', driver: driverA }); + const webReport = await reportAttempt(config, runId, { ...web, outcome: 'success', driver: driverA }); + + expect([web.revision, api.revision]).toEqual([2, 2]); + expect(apiReport).toMatchObject({ data: { accepted: true, revision: 3, step: { id: 'build[1]', status: 'succeeded' } } }); + expect(webReport).toMatchObject({ data: { accepted: true, revision: 4, step: { id: 'build[0]', status: 'succeeded' } } }); + }); + + it('keeps a sibling running when an item fails, and fails the step once every item settled', async () => { + const { config, runId } = await startTree({ fan: FAN_PLAN }, 'fan'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { targets: ['web', 'api'] }, driver: driverA }); + const web = leaseIn(await leaseNext(config, runId, driverA)); + const api = leaseIn(await leaseNext(config, runId, driverA, { running: [web.attempt] })); + + await reportAttempt(config, runId, { ...web, outcome: 'failed', driver: driverA }); + const meanwhile = await leaseNext(config, runId, driverA, { running: [api.attempt] }); + const stored = await readRun(config, runId); + const reported = await reportAttempt(config, runId, { ...api, outcome: 'success', driver: driverA }); + const done = await leaseNext(config, runId, driverA); + + expect(meanwhile).toEqual({ data: { wait: { reason: 'build[1] is leased' }, cancelled: [] } }); + expect(stored).toMatchObject({ steps: [ + { id: 'plan' }, { id: 'build', status: 'running' }, { id: 'build[0]', status: 'failed' }, { id: 'build[1]', status: 'leased' }, { id: 'ship', status: 'pending' }, + ] }); + expect(reported).toMatchObject({ data: { accepted: true, step: { id: 'build[1]', status: 'succeeded' } } }); + expect(done).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan' }, + { id: 'build', status: 'failed', results: [{ item: 'web', outcome: 'failed' }, { item: 'api', outcome: 'success' }] }, + { id: 'build[0]' }, { id: 'build[1]' }, + { id: 'ship', status: 'skipped' }, + ] }); + }); + + it('starts an item only after the items its after names, and skips it when one of them failed or was skipped', async () => { + const { config, runId } = await startTree({ fan: FAN_TASKS }, 'fan'); + const plan = leaseIn(await leaseNext(config, runId, driverA)); + const tasks = [{ id: 'a' }, { id: 'b', after: ['a'] }, { id: 'c' }, { id: 'd', after: ['b'] }]; + await reportAttempt(config, runId, { ...plan, outcome: 'success', outputs: { tasks }, driver: driverA }); + + const a = leaseIn(await leaseNext(config, runId, driverA)); + const c = leaseIn(await leaseNext(config, runId, driverA, { running: [a.attempt] })); + const full = await leaseNext(config, runId, driverA, { running: [a.attempt, c.attempt] }); + await reportAttempt(config, runId, { ...a, outcome: 'failed', driver: driverA }); + await reportAttempt(config, runId, { ...c, outcome: 'success', driver: driverA }); + const done = await leaseNext(config, runId, driverA); + + expect([a.step, c.step]).toEqual(['build[0]', 'build[2]']); + expect(full).toMatchObject({ data: { wait: { reason: 'build[0] is leased' }, cancelled: [] } }); + expect(done).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', status: 'succeeded' }, + { id: 'build', status: 'failed', results: [ + { item: tasks[0], outcome: 'failed' }, + { item: tasks[1], outcome: 'skipped' }, + { item: tasks[2], outcome: 'success' }, + { item: tasks[3], outcome: 'skipped' }, + ] }, + { id: 'build[0]', status: 'failed' }, + { id: 'build[1]', status: 'skipped' }, + { id: 'build[2]', status: 'succeeded' }, + { id: 'build[3]', status: 'skipped' }, + { id: 'ship', status: 'skipped' }, + ] }); + }); + + it('fails the step with item_cycle naming the cycle, before any item runs', async () => { + const { config, runId } = await startTree({ fan: FAN_TASKS }, 'fan'); + const tasks = [{ id: 'd' }, { id: 'a', after: ['c'] }, { id: 'b', after: ['a'] }, { id: 'c', after: ['b'] }]; + + const { leased, done } = await driveLoop(config, runId, () => ({ outputs: { tasks } })); + + expect(leased).toEqual(['plan#1']); + expect(done).toEqual({ run: runId, record: 'pending', outcome: 'failed' }); + expect(await readRun(config, runId)).toMatchObject({ steps: [ + { id: 'plan', status: 'succeeded' }, + { id: 'build', status: 'failed', failure: { code: 'item_cycle', message: 'step "build" has items that run after each other in a cycle: a -> c -> b -> a' } }, + { id: 'ship', status: 'skipped' }, + ] }); + }); +}); + +describe('where an item is bound', () => { + const header = 'inputs:\n items: [{ name: string, xs: [string] }]\n'; + const pair = (prompt: string, indent = ''): string => `${indent} - id: s +${indent} prompt: "${prompt}" +${indent} - id: p +${indent} needs: [] +${indent} command: node p.js +`; + const positions: Array<[string, (prompt: string) => Record, string, boolean]> = [ + ['a step outside a for_each', (prompt) => ({ top: workflow('top', pair(prompt), header) }), 's', false], + ['a for_each step', (prompt) => ({ top: workflow('top', ` - id: s + for_each: inputs.items + prompt: "${prompt}" + - id: p + needs: [] + command: node p.js +`, header) }), 's[0]', true], + ["a step of an item's inline workflow", (prompt) => ({ top: workflow('top', ` - id: each + for_each: inputs.items + workflow: + steps: +${pair(prompt, ' ')}`, header) }), 'each[0].s', true], + ['a step of a named workflow an item runs', (prompt) => ({ + top: workflow('top', ' - id: each\n for_each: inputs.items\n workflow: inner\n', header), + inner: workflow('inner', pair(prompt)), + }), 'each[0].s', false], + ]; + const inputs = { items: [{ name: 'I', xs: ['success'] }] }; + + it.each(positions)('renders item in %s only where pull, the lease and what the step waits for all bind it', async (_case, files, step, bound) => { + const reads = await startTree(files('{{ item.name }}|{{ length(item.xs) }}'), 'top', inputs); + expect(leaseIn(await leaseNext(reads.config, reads.runId, driverA))).toMatchObject({ + step, prompt: bound ? 'I|1' : '{{ item.name }}|{{ length(item.xs) }}', + }); + + const fixture = testConfig(); + for (const [file, body] of Object.entries(files('{{ any(item.xs, . == steps.p.outcome) }}'))) { + fs.writeFileSync(path.join(fixture.config.repo.localPath, 'workflows', `${file}.yaml`), body); + } + const pulled = await syncWorkflowDefinitions(fixture.config); + expect(pulled.invalid.map((diagnostic) => diagnostic.message).filter((message) => message.includes('steps.p'))).toHaveLength(bound ? 1 : 0); + if (bound) return; + // Where item is not bound, the step reads no step, so it does not wait for p. + const started = await startRun(fixture.config, 'top', inputs); + if ('error' in started) throw new Error(started.error.message); + expect(leaseIn(await leaseNext(fixture.config, String(started.data.run), driverA)).step).toBe(step); + }); }); diff --git a/src/__tests__/workflow-expression.test.ts b/src/__tests__/workflow-expression.test.ts index 6bea8e34f..cc255bc18 100644 --- a/src/__tests__/workflow-expression.test.ts +++ b/src/__tests__/workflow-expression.test.ts @@ -4,7 +4,10 @@ import { evaluateExpression, expressionPaths, isTruthy, parseExpression, type Ex const context: ExpressionContext = { steps: { check: { outcome: 'failed' }, - review: { outcome: 'success', outputs: { verdict: 'fix', count: 0, notes: '', tags: ['a', 'b'], meta: { level: 1, kind: 'x' } } }, + review: { + outcome: 'success', + outputs: { verdict: 'fix', count: 0, notes: '', tags: ['a', 'b'], meta: { level: 1, kind: 'x' }, matrix: [[1, 2], [3, 4]], files: [{ path: 'a.ts' }] }, + }, copy: { outcome: 'success', outputs: { tags: ['a', 'b'], reversed: ['b', 'a'], meta: { kind: 'x', level: 1 }, other: { level: 1, kind: 'x', extra: null } } }, }, }; @@ -61,6 +64,39 @@ describe('workflow expressions', () => { ['true || false && false', true], ['(true || false) && false', false], ["!(steps.check.outcome == 'success') && steps.review.outputs.verdict == 'fix'", true], + ["steps.review.outputs.tags[1] == 'b'", true], + ['steps.review.outputs.tags[2] == null', true], + ['steps.review.outputs.verdict[0] == null', true], + ['steps.review.outputs.meta[0] == null', true], + ['steps.review.outputs.matrix[1][0] == 3', true], + ["steps.review.outputs.files[0].path == 'a.ts'", true], + ['steps.review.outputs.count < 1', true], + ['steps.review.outputs.count <= 0', true], + ['steps.review.outputs.count > 0', false], + ['steps.review.outputs.count >= 0', true], + ['1.5 > 1', true], + ['steps.review.outputs.verdict < 1', false], + ['steps.review.outputs.verdict >= 1', false], + ['inputs.missing < 1', false], + ['1 < 2 == true', true], + ["!steps.check.outcome == 'success'", true], + ['length(steps.review.outputs.tags) == 2', true], + ["any(steps.review.outputs.files, .path == 'a.ts')", true], + ["all(steps.review.outputs.tags, . != 'c')", true], + ["filter(steps.review.outputs.tags, . == 'z') == []", true], + ["length(filter(steps.review.outputs.tags, . != 'a')) == 1", true], + ['any([], . == 1)', false], + ['all([], . == 1)', true], + ['filter([], true) == []', true], + ['filter(steps.review.outputs.tags, 1) == steps.copy.outputs.tags', true], + ['length(steps.review.outputs.verdict) == null', true], + ['any(steps.review.outputs.meta, true) == null', true], + ['all(inputs.missing, true) == null', true], + ['filter(steps.review.outputs.count, true) == null', true], + ['any(steps.review.outputs.matrix, any(., . == 4))', true], + ['all(steps.review.outputs.matrix, length(.) == 2)', true], + ['any(steps.review.outputs.matrix, .[1] == 4)', true], + ['any(steps.review.outputs.tags, . == steps.copy.outputs.tags[1])', true], ])('evaluates %s to %s', (source, expected) => { expect(evaluate(source)).toBe(expected); }); @@ -74,19 +110,67 @@ describe('workflow expressions', () => { ['(true', 6, 'expected ")", found the end of the expression'], ['true false', 6, 'unexpected "false"'], ["1 == 1 == 1", 8, 'unexpected "=="'], + ['1 < 2 < 3', 7, 'unexpected "<"'], + ["steps.review.outputs.count < '3'", 30, "compares '3' with <, but it is not a number"], + ['inputs.n > null', 12, 'compares null with >, but it is not a number'], + ['true >= 1', 1, 'compares true with >=, but it is not a number'], + ["'it''s' < 3", 1, "compares 'it''s' with <, but it is not a number"], + ["true 'it''s'", 6, "unexpected 'it''s'"], + ['inputs.n < .5', 12, 'numbers need a leading digit, as in 0.5'], + ['any(inputs.xs, . < .5)', 20, 'numbers need a leading digit, as in 0.5'], + ['all(inputs.xs, any(.ys, . == .25))', 30, 'numbers need a leading digit, as in 0.5'], ["['a']", 2, 'the only array literal is []'], - ['env.HOME', 1, '"env" is not readable; an expression reads only inputs and steps'], - ['github.ref', 1, '"github" is not readable; an expression reads only inputs and steps'], - ["contains(steps.review.outputs.tags, 'a')", 1, '"contains" is not readable; an expression reads only inputs and steps'], - ['process.env.SECRET', 1, '"process" is not readable; an expression reads only inputs and steps'], - ['steps.check', 1, '"steps.check" must read steps..outcome, steps..end or steps..outputs'], - ['steps.check.status', 1, '"steps.check.status" must read steps..outcome, steps..end or steps..outputs'], - ['steps.check.outcome.text', 1, '"steps.check.outcome.text" must read steps..outcome, steps..end or steps..outputs'], + ['env.HOME', 1, '"env" is not readable; an expression reads only inputs, steps and item'], + ['github.ref', 1, '"github" is not readable; an expression reads only inputs, steps and item'], + ["contains(steps.review.outputs.tags, 'a')", 1, '"contains" is not a function; the functions are length, any, all and filter'], + ['inputs.list(1)', 12, 'unexpected "("'], + ['any(steps.review.outputs.tags, true) < 1', 1, 'compares any(...) with <, but it is not a number'], + ['filter(steps.review.outputs.tags, true) > 0', 1, 'compares filter(...) with >, but it is not a number'], + ['(1 == 1) < 2', 4, 'compares a condition with <, but it is not a number'], + ["length('abc') == 3", 8, "passes 'abc' to length as its list, but it is not an array"], + ['any(length(steps.review.outputs.tags), true)', 5, 'passes length(...) to any as its list, but it is not an array'], + ['all(null, true)', 5, 'passes null to all as its list, but it is not an array'], + ['length(steps.review.outputs.tags, 1)', 1, 'length is called as length(list)'], + ['length()', 1, 'length is called as length(list)'], + ['any(steps.review.outputs.tags)', 1, 'any is called as any(list, predicate)'], + ['all(steps.review.outputs.tags, true, false)', 1, 'all is called as all(list, predicate)'], + [".path == 'a.ts'", 1, '".path" reads the current element, which exists only inside the predicate of any, all or filter'], + ['length(.) == 1', 8, '"." reads the current element, which exists only inside the predicate of any, all or filter'], + ['process.env.SECRET', 1, '"process" is not readable; an expression reads only inputs, steps and item'], + ['steps.check', 1, '"steps.check" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], + ['steps.check.status', 1, '"steps.check.status" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], + ['steps.check.outcome.text', 1, '"steps.check.outcome.text" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], ['inputs.', 7, 'unexpected "."'], + ['inputs.tags[-1]', 12, '"[-1]" is not an index; write a whole number from 0, such as [0]'], + ['inputs.tags[inputs.n] == 1', 12, '"[inputs.n]" is not an index; write a whole number from 0, such as [0]'], + ['inputs.tags[01]', 12, '"[01]" is not an index; write a whole number from 0, such as [0]'], + ['inputs.tags[0', 12, 'an index is not closed with ]'], + ['steps.check.outcome[0]', 1, '"steps.check.outcome[0]" indexes outcome, which is a name; compare it whole, as in steps.check.outcome == \'\''], + ['steps.check.end[0]', 1, '"steps.check.end[0]" indexes end, which is a name; compare it whole, as in steps.check.end == \'\''], + ['steps.build[2].outcome', 1, '"steps.build[2].outcome" indexes a step; an index reads an array in its outputs, as in steps.build.outputs.[2]'], ])('rejects %j at column %i', (source, column, message) => { expect(parseExpression(source)).toEqual({ ok: false, column, message }); }); + it('reads a digit-named field below the element, which only a lone . cannot be', () => { + const parsed = parseExpression('any(inputs.xs, .5.x == 1 || .a.5 == 1)'); + if (!parsed.ok) throw new Error(parsed.message); + + expect(evaluateExpression(parsed.expression, { inputs: { xs: [{ 5: { x: 1 } }] }, steps: {} })).toBe(true); + }); + + it('compares values nested 4,000 levels deep without overflowing the stack', () => { + const nested = (bottom: unknown): unknown => { + let value = bottom; + for (let level = 0; level < 4000; level += 1) value = level % 2 === 0 ? [value] : { a: value }; + return value; + }; + const parsed = parseExpression('inputs.a == inputs.b && inputs.a != inputs.c'); + if (!parsed.ok) throw new Error(parsed.message); + + expect(evaluateExpression(parsed.expression, { inputs: { a: nested(1), b: nested(1), c: nested(2) }, steps: {} })).toBe(true); + }); + it('lists the paths an expression reads, with their columns', () => { const parsed = parseExpression("inputs.issue.number == 1 || !(steps.check.outcome == 'failed' && steps.review.outputs.verdict.text)"); if (!parsed.ok) throw new Error(parsed.message); @@ -97,4 +181,41 @@ describe('workflow expressions', () => { { root: 'steps', step: 'review', field: 'outputs', fields: ['verdict', 'text'], column: 66 }, ]); }); + + it('lists the paths inside list arguments and predicates, and no relative path', () => { + const parsed = parseExpression("any(steps.review.outputs.files, .path == inputs.file) && length(steps.copy.outputs.tags) > 0"); + if (!parsed.ok) throw new Error(parsed.message); + + expect(expressionPaths(parsed.expression)).toEqual([ + { root: 'steps', step: 'review', field: 'outputs', fields: ['files'], column: 5 }, + { root: 'inputs', fields: ['file'], column: 42 }, + { root: 'steps', step: 'copy', field: 'outputs', fields: ['tags'], column: 65 }, + ]); + }); + + it('lists an index as a number among the fields a path reads', () => { + const parsed = parseExpression('steps.review.outputs.matrix[0][1] == inputs.tags[2]'); + if (!parsed.ok) throw new Error(parsed.message); + + expect(expressionPaths(parsed.expression)).toEqual([ + { root: 'steps', step: 'review', field: 'outputs', fields: ['matrix', 0, 1], column: 1 }, + { root: 'inputs', fields: ['tags', 2], column: 38 }, + ]); + }); + + it.each([ + ["steps.loop.passes[0].outputs.verdict == 'fix'", true], + ["steps.loop.passes[1].outcome == 'success'", true], + ['length(steps.loop.passes) == 2', true], + ["any(steps.loop.passes, .outputs.verdict == 'pass')", true], + ['steps.loop.passes[2] == null', true], + ['steps.check.passes == null', true], + ])('reads the passes of a step: %s is %s', (source, value) => { + const parsed = parseExpression(source); + if (!parsed.ok) throw new Error(parsed.message); + expect(evaluateExpression(parsed.expression, { steps: { ...context.steps, loop: { passes: [ + { pass: 1, outcome: 'success', outputs: { verdict: 'fix' } }, + { pass: 2, outcome: 'success', outputs: { verdict: 'pass' } }, + ] } } })).toBe(value); + }); }); diff --git a/src/__tests__/workflow-local-driver.test.ts b/src/__tests__/workflow-local-driver.test.ts index 4cd07d824..9a4bfda1f 100644 --- a/src/__tests__/workflow-local-driver.test.ts +++ b/src/__tests__/workflow-local-driver.test.ts @@ -1,4 +1,4 @@ -import { execFileSync } from 'node:child_process'; +import { execFileSync, spawn } from 'node:child_process'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; @@ -6,14 +6,27 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import type { LocalConfig, TeamaiConfig } from '../types.js'; import { log } from '../utils/logger.js'; import { syncWorkflowDefinitions } from '../workflow/cache.js'; -import { startRun } from '../workflow/engine.js'; -import type { LocalToolAdapter, LocalToolRun } from '../workflow/local-tool-adapter.js'; -import { runLocalDriver } from '../workflow/local-driver.js'; +import { leaseNext, reportAttempt, startRun } from '../workflow/engine.js'; +import { createLocalToolAdapter, type LocalToolAdapter, type LocalToolRun, type ToolSpawn } from '../workflow/local-tool-adapter.js'; +import { localDriver, runLocalDriver } from '../workflow/local-driver.js'; +import { pendingRecords, type WorkflowRunRecord } from '../workflow/records.js'; import { readRun } from '../workflow/store.js'; const dirs: string[] = []; async function fixture(agent?: string): Promise<{ config: LocalConfig; workspace: string; runId: string }> { + return workflowFixture(`schemaVersion: v1alpha1 +name: build +description: Review the project. +steps: + - id: review + prompt: Review this project. +${agent ? ` agent: ${agent}\n` : ''} output: + ok: boolean +`); +} + +async function workflowFixture(yaml: string): Promise<{ config: LocalConfig; workspace: string; runId: string }> { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-driver-')); dirs.push(root); const knowledge = path.join(root, 'knowledge'); @@ -33,15 +46,7 @@ async function fixture(agent?: string): Promise<{ config: LocalConfig; workspace dataHome: path.join(root, 'data'), workflows: { tools: { default: 'claude' } }, }; - fs.writeFileSync(path.join(knowledge, 'workflows', 'build.yaml'), `schemaVersion: v1alpha1 -name: build -description: Review the project. -steps: - - id: review - prompt: Review this project. -${agent ? ` agent: ${agent}\n` : ''} output: - ok: boolean -`); + fs.writeFileSync(path.join(knowledge, 'workflows', 'build.yaml'), yaml); await syncWorkflowDefinitions(config); const started = await startRun(config, 'build'); if ('error' in started) throw new Error(started.error.message); @@ -362,4 +367,430 @@ sandbox_mode = "read-only" steps: [{ status: 'failed', failure: { code: 'tool_failed' } }], }); }); + + it('publishes the failure code it sets when the tool is not on PATH', async () => { + const { config, workspace, runId } = await fixture(); + + await runLocalDriver(config, runId, workspace, { adapter: createLocalToolAdapter(undefined, () => null), toolPaths }); + + expect(await endRecord(config)).toMatchObject({ status: 'failed', failure: { code: 'tool_unavailable' } }); + }); + + it('publishes no failure code the tool wrote into its result', async () => { + const { config, workspace, runId } = await fixture(); + const writesFailure: ToolSpawn = (_command, _args, options) => spawn(process.execPath, [ + '-e', + 'require("fs").writeFileSync(process.env.TEAMAI_WORKFLOW_RESULT, JSON.stringify({ failure: { code: "secret-x", message: "secret-x happened" } }))', + ], options); + + await runLocalDriver(config, runId, workspace, { adapter: createLocalToolAdapter(writesFailure, () => process.execPath), toolPaths }); + + expect(await readRun(config, runId)).toMatchObject({ steps: [{ status: 'failed', failure: { code: 'secret-x' } }] }); + const record = await endRecord(config); + expect(record).toMatchObject({ status: 'failed' }); + expect(JSON.stringify(record)).not.toContain('secret-x'); + }); +}); + +/** The end record the run queued; the fixtures have no reports remote, so it stays queued. */ +async function endRecord(config: LocalConfig): Promise { + return (await pendingRecords(config)).find((record) => record.phase === 'end'); +} + +const BARRIER = `const fs = require('fs'); +const [me, other] = process.argv.slice(2); +fs.writeFileSync(me + '.ready', ''); +const deadline = Date.now() + 2000; +setInterval(() => { + if (fs.existsSync(other + '.ready')) process.exit(0); + if (Date.now() > deadline) process.exit(1); +}, 20); +`; + +const LOG = `const fs = require('fs'); +fs.appendFileSync('log.txt', 'start ' + process.argv[2] + '\\n'); +setTimeout(() => fs.appendFileSync('log.txt', 'end ' + process.argv[2] + '\\n'), 100); +`; + +const TWO_BARRIERS = `schemaVersion: v1alpha1 +name: build +description: Two roots that each wait for the other. +steps: + - id: left + needs: [] + command: node barrier.js left right + - id: right + needs: [] + command: node barrier.js right left +`; + +async function parallelFixture(yaml: string, maxParallel: number): Promise<{ config: LocalConfig; workspace: string; runId: string }> { + const built = await workflowFixture(yaml); + built.config.workflows = { tools: { default: 'claude' }, max_parallel: maxParallel }; + fs.writeFileSync(path.join(built.workspace, 'barrier.js'), BARRIER); + return built; +} + +function alive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +async function exitsWithin(pid: number, ms: number): Promise { + for (let waited = 0; alive(pid); waited += 20) { + if (waited >= ms) return false; + await new Promise((resolve) => setTimeout(resolve, 20)); + } + return true; +} + +describe('local workflow driver concurrency', () => { + it('runs two command roots at the same time with max_parallel 2', async () => { + const { config, workspace, runId } = await parallelFixture(TWO_BARRIERS, 2); + + const done = await runLocalDriver(config, runId, workspace); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + }); + + it('holds a prompt leased while a command runs, and starts it once the command has exited', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: A command and a prompt with no needs. +steps: + - id: slow + needs: [] + command: node -e "setTimeout(() => require('fs').writeFileSync('slow.done', ''), 300)" + - id: ask + needs: [] + prompt: Review this project. + output: + ok: boolean +`, 2); + const commandExitedFirst: boolean[] = []; + const adapter: LocalToolAdapter = { + run: vi.fn(async () => { + commandExitedFirst.push(fs.existsSync(path.join(workspace, 'slow.done'))); + return { exitCode: 0, output: { ok: true } }; + }), + }; + + const done = await runLocalDriver(config, runId, workspace, { adapter, toolPaths }); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(commandExitedFirst).toEqual([true]); + }); + + it('drops a held prompt that an end cancels before it starts, and the run goes on', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: An inline workflow ends while its prompt is held. +steps: + - id: w + needs: [] + workflow: + ends: + stopped: succeeded + steps: + - id: finish + needs: [] + command: node -e "setTimeout(() => {}, 200)" + end: stopped + - id: ask + needs: [] + prompt: Review this project. + output: + ok: boolean + - id: after + needs: [w] + command: node -e "process.exit(0)" +`, 2); + const adapter: LocalToolAdapter = { run: vi.fn(async () => ({ exitCode: 0, output: { ok: true } })) }; + + const done = await runLocalDriver(config, runId, workspace, { adapter, toolPaths }); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(adapter.run).not.toHaveBeenCalled(); + expect(await readRun(config, runId)).toMatchObject({ + steps: expect.arrayContaining([ + expect.objectContaining({ id: 'w.ask', status: 'cancelled' }), + expect.objectContaining({ id: 'after', status: 'succeeded' }), + ]), + }); + }); + + it('kills a cancelled command that ignores SIGTERM after the grace period, and finishes', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: An end cancels a command that ignores SIGTERM. +ends: + done: succeeded +steps: + - id: stubborn + needs: [] + command: node stubborn.js + - id: finish + needs: [] + command: node finish.js + end: done +`, 2); + fs.writeFileSync(path.join(workspace, 'stubborn.js'), `process.on('SIGTERM', () => {}); +require('fs').writeFileSync('stubborn.pid', String(process.pid)); +setInterval(() => {}, 1000); +`); + fs.writeFileSync(path.join(workspace, 'finish.js'), `const timer = setInterval(() => { + if (require('fs').existsSync('stubborn.pid')) clearInterval(timer); +}, 20); +`); + let pid: number | undefined; + try { + const done = await runLocalDriver(config, runId, workspace); + pid = Number(fs.readFileSync(path.join(workspace, 'stubborn.pid'), 'utf8')); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded', end: 'done' } } }); + expect(await exitsWithin(pid, 1000)).toBe(true); + expect(await readRun(config, runId)).toMatchObject({ + steps: [{ id: 'stubborn', status: 'cancelled' }, { id: 'finish', status: 'succeeded' }], + }); + } finally { + if (pid !== undefined && alive(pid)) process.kill(pid, 'SIGKILL'); + } + }); + + it('kills a descendant that ignores SIGTERM after the cancelled command itself exited, and finishes', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: An end cancels a command whose child ignores SIGTERM and outlives it. +ends: + done: succeeded +steps: + - id: parent + needs: [] + command: node parent.js + - id: finish + needs: [] + command: node finish.js + end: done +`, 2); + fs.writeFileSync(path.join(workspace, 'parent.js'), `require('child_process').spawn(process.execPath, ['-e', \`process.on('SIGTERM', () => {}); +require('fs').writeFileSync('child.pid', String(process.pid)); +setInterval(() => {}, 1000);\`], { stdio: 'ignore' }); +setInterval(() => {}, 1000); +`); + fs.writeFileSync(path.join(workspace, 'finish.js'), `const timer = setInterval(() => { + if (require('fs').existsSync('child.pid')) clearInterval(timer); +}, 20); +`); + let pid: number | undefined; + try { + const done = await runLocalDriver(config, runId, workspace); + pid = Number(fs.readFileSync(path.join(workspace, 'child.pid'), 'utf8')); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded', end: 'done' } } }); + expect(await exitsWithin(pid, 1000)).toBe(true); + } finally { + if (pid !== undefined && alive(pid)) process.kill(pid, 'SIGKILL'); + } + }); + + it.each([ + ['leased with the answer that cancels the command', 2, ` - id: ask + needs: [w] +`], + ['held when the command is cancelled', 3, ` - id: ask + needs: [] +`], + ])('starts a prompt %s only once the command exits in its grace period', async (_case, maxParallel, ask) => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: An end cancels a command that takes a moment to exit, and a prompt waits for it. +steps: + - id: w + needs: [] + workflow: + ends: + stopped: succeeded + steps: + - id: finish + needs: [] + command: node finish.js + end: stopped + - id: lingering + needs: [] + command: node lingering.js +${ask} prompt: Review this project. + output: + ok: boolean +`, maxParallel); + fs.writeFileSync(path.join(workspace, 'lingering.js'), `const fs = require('fs'); +process.on('SIGTERM', () => setTimeout(() => { fs.writeFileSync('lingering.exited', ''); process.exit(0); }, 300)); +fs.writeFileSync('lingering.started', ''); +setInterval(() => {}, 1000); +`); + fs.writeFileSync(path.join(workspace, 'finish.js'), `const timer = setInterval(() => { + if (require('fs').existsSync('lingering.started')) clearInterval(timer); +}, 20); +`); + const commandExitedFirst: boolean[] = []; + const adapter: LocalToolAdapter = { + run: vi.fn(async () => { + commandExitedFirst.push(fs.existsSync(path.join(workspace, 'lingering.exited'))); + return { exitCode: 0, output: { ok: true } }; + }), + }; + + const done = await runLocalDriver(config, runId, workspace, { adapter, toolPaths }); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(commandExitedFirst).toEqual([true]); + expect(await readRun(config, runId)).toMatchObject({ + steps: expect.arrayContaining([ + expect.objectContaining({ id: 'w.lingering', status: 'cancelled' }), + expect.objectContaining({ id: 'ask', status: 'succeeded' }), + ]), + }); + }); + + it('discards the lease_cancelled answer for a command cancelled while it ran, and finishes', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: Another caller with the same identity reaches an end while slow runs. +ends: + done: succeeded +steps: + - id: slow + needs: [] + command: node slow.js + - id: finish + needs: [] + command: node -e "process.exit(0)" + end: done +`, 1); + fs.writeFileSync(path.join(workspace, 'slow.js'), `const fs = require('fs'); +fs.writeFileSync('slow.started', ''); +const timer = setInterval(() => { if (fs.existsSync('go')) clearInterval(timer); }, 20); +`); + const driving = runLocalDriver(config, runId, workspace); + while (!fs.existsSync(path.join(workspace, 'slow.started'))) await new Promise((resolve) => setTimeout(resolve, 20)); + const run = await readRun(config, runId); + const slow = typeof run === 'object' ? run.steps.find((step) => step.id === 'slow') : undefined; + if (slow?.status !== 'leased') throw new Error('slow is not leased'); + const same = localDriver(workspace); + const leased = await leaseNext(config, runId, same, { running: [slow.attempt] }); + const finish = 'data' in leased ? leased.data.lease as { attempt: string; revision: number } : undefined; + if (!finish) throw new Error('finish was not leased'); + await reportAttempt(config, runId, { attempt: finish.attempt, revision: finish.revision, outcome: 'success', driver: same }); + expect(await leaseNext(config, runId, same, { running: [slow.attempt], canTakeLease: false })).toMatchObject({ + data: { cancelled: [slow.attempt] }, + }); + fs.writeFileSync(path.join(workspace, 'go'), ''); + + expect(await driving).toMatchObject({ data: { done: { outcome: 'succeeded', end: 'done' } } }); + expect(await readRun(config, runId)).toMatchObject({ steps: [{ id: 'slow', status: 'cancelled' }, { id: 'finish', status: 'succeeded' }] }); + }); + + it.skipIf(process.platform === 'win32').each([ + [1, 'the driver\'s process group', true], + [2, 'a process group of its own', false], + ] as const)('runs a command with max_parallel %i in %s', async (maxParallel, _group, shared) => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: Report the process group a command runs in. +steps: + - id: group + command: node pgid.js + output: + pgid: number +`, maxParallel); + fs.writeFileSync(path.join(workspace, 'pgid.js'), `const pgid = require('child_process').execFileSync('ps', ['-o', 'pgid=', '-p', String(process.pid)]); +process.stdout.write(JSON.stringify({ pgid: Number(String(pgid).trim()) })); +`); + const driverGroup = Number(execFileSync('ps', ['-o', 'pgid=', '-p', String(process.pid)], { encoding: 'utf8' }).trim()); + + await runLocalDriver(config, runId, workspace); + + const run = await readRun(config, runId); + const outputs = typeof run === 'object' && run.steps[0]?.status === 'succeeded' ? run.steps[0].outputs as { pgid: number } : undefined; + expect(outputs?.pgid).toEqual(expect.any(Number)); + expect(outputs?.pgid === driverGroup).toBe(shared); + }); + + it('gets the leases of an earlier driver back one at a time, and runs them one at a time with max_parallel 1', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: Two roots an earlier driver leased and never reported. +steps: + - id: a + needs: [] + command: node log.js a + - id: b + needs: [] + command: node log.js b +`, 1); + fs.writeFileSync(path.join(workspace, 'log.js'), LOG); + const earlier = localDriver(workspace); + const a = await leaseNext(config, runId, earlier); + const attempt = 'data' in a ? (a.data.lease as { attempt: string }).attempt : ''; + expect(await leaseNext(config, runId, earlier, { running: [attempt] })).toMatchObject({ data: { lease: { step: 'b' } } }); + + const done = await runLocalDriver(config, runId, workspace); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(fs.readFileSync(path.join(workspace, 'log.txt'), 'utf8')).toBe('start a\nend a\nstart b\nend b\n'); + }); + + it('runs a prompt an earlier driver leased alone, between the commands leased before and after it', async () => { + const { config, workspace, runId } = await parallelFixture(`schemaVersion: v1alpha1 +name: build +description: A command, a prompt and a command an earlier driver leased. +steps: + - id: c1 + needs: [] + command: node log.js c1 + - id: ask + needs: [] + prompt: Review this project. + output: + ok: boolean + - id: c2 + needs: [] + command: node log.js c2 +`, 2); + fs.writeFileSync(path.join(workspace, 'log.js'), LOG); + const earlier = localDriver(workspace); + const running: string[] = []; + for (const step of ['c1', 'ask', 'c2']) { + const leased = await leaseNext(config, runId, earlier, { running }); + expect(leased).toMatchObject({ data: { lease: { step } } }); + if ('data' in leased) running.push((leased.data.lease as { attempt: string }).attempt); + } + const logAtPrompt: string[] = []; + const adapter: LocalToolAdapter = { + run: vi.fn(async () => { + logAtPrompt.push(fs.readFileSync(path.join(workspace, 'log.txt'), 'utf8')); + return { exitCode: 0, output: { ok: true } }; + }), + }; + + const done = await runLocalDriver(config, runId, workspace, { adapter, toolPaths }); + + expect(done).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + expect(logAtPrompt).toEqual(['start c1\nend c1\n']); + expect(fs.readFileSync(path.join(workspace, 'log.txt'), 'utf8')).toBe('start c1\nend c1\nstart c2\nend c2\n'); + }); + + it('runs one command at a time with max_parallel 1, so the barrier fails', async () => { + const { config, workspace, runId } = await parallelFixture(TWO_BARRIERS, 1); + + const done = await runLocalDriver(config, runId, workspace); + + expect(done).toMatchObject({ data: { done: { outcome: 'failed' } } }); + expect(await readRun(config, runId)).toMatchObject({ + steps: [{ id: 'left', status: 'failed', exitCode: 1 }, { id: 'right', status: 'succeeded' }], + }); + }); }); diff --git a/src/__tests__/workflow-parse.test.ts b/src/__tests__/workflow-parse.test.ts index 8d8af6c98..cf677ca10 100644 --- a/src/__tests__/workflow-parse.test.ts +++ b/src/__tests__/workflow-parse.test.ts @@ -137,15 +137,15 @@ description: Build it. steps: - id: one command: npm test - for_each: items + timeout: 10m `); expect(result.ok).toBe(false); if (result.ok) return; expect(result.diagnostics).toEqual(expect.arrayContaining([ expect.objectContaining({ code: 'unknown_key', - field: 'steps[0].for_each', - message: 'workflows/build.yaml uses `for_each`, which this teamai does not know. Update teamai.', + field: 'steps[0].timeout', + message: 'workflows/build.yaml uses `timeout`, which this teamai does not know. Update teamai.', }), ])); }); @@ -167,6 +167,217 @@ steps: ])); }); + it('reads an element of a declared array by index', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Fix the first file. +inputs: + labels: [string] +outputs: + first: steps.review.outputs.files[0].path +steps: + - id: review + command: node review.js + output: + files: [{ path: string }] + - id: fix + needs: [review] + if: steps.review.outputs.files[0].path == 'a.ts' && inputs.labels[0] == 'bug' + prompt: Fix {{ steps.review.outputs.files[0].path }} +`); + expect(result).toMatchObject({ ok: true }); + }); + + it('rejects an index into a value that is not declared as an array, and names the path as written', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Fix the first file. +inputs: + title: string +outputs: + first: steps.review.outputs.verdict[0] +steps: + - id: review + command: node review.js + output: + verdict: pass | fix + - id: fix + needs: [review] + if: inputs.title[0] == 'a' || steps.review.outputs.verdict[0] == 'p' + prompt: Fix {{ with.nope[0] }} + with: + refs: [a] +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics.map(formatDiagnostic).sort()).toEqual([ + 'workflows/build.yaml:15: field "steps[1].if": step "fix" if reads inputs.title[0], which is not an input the workflow declares', + 'workflows/build.yaml:15: field "steps[1].if": step "fix" if reads steps.review.outputs.verdict[0], which is not in the output step "review" declares', + 'workflows/build.yaml:16: field "steps[1].prompt": step "fix" prompt reads with.nope[0], which is not a name its with declares', + 'workflows/build.yaml:7: field "outputs.first": output "first" reads steps.review.outputs.verdict[0], which is not in the output step "review" declares', + ]); + }); + + it('rejects a comparison of a value declared as something other than a number', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +inputs: + title: string +steps: + - id: review + command: node review.js + output: + verdict: pass | fix + - id: fix + needs: [review] + if: steps.review.outputs.verdict < 1 || inputs.title > 0 || steps.review.outcome >= 1 + command: node fix.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics.map(formatDiagnostic)).toEqual([ + 'workflows/build.yaml:13: field "steps[1].if": step "fix" if compares steps.review.outputs.verdict with <, but it is not a number', + 'workflows/build.yaml:13: field "steps[1].if": step "fix" if compares inputs.title with >, but it is not a number', + 'workflows/build.yaml:13: field "steps[1].if": step "fix" if compares steps.review.outcome with >=, but it is not a number', + ]); + }); + + it('rejects a list argument declared as something other than an array', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +steps: + - id: review + command: node review.js + output: + verdict: pass | fix + - id: fix + needs: [review] + if: any(steps.review.outputs.verdict, true) + command: node fix.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics.map(formatDiagnostic)).toEqual([ + 'workflows/build.yaml:11: field "steps[1].if": step "fix" if passes steps.review.outputs.verdict to any as its list, but it is not an array', + ]); + }); + + it('checks what a predicate reads of each element against the items the list declares', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +steps: + - id: review + command: node review.js + output: + files: [{ path: string, lines: number, tags: [string] }] + - id: fix + needs: [review] + if: any(steps.review.outputs.files, .pth == 'a') || all(steps.review.outputs.files, .path < 1) || any(steps.review.outputs.files, any(.path, true)) + command: node fix.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics.map(formatDiagnostic)).toEqual([ + 'workflows/build.yaml:11: field "steps[1].if": step "fix" if reads .pth, which is not in an item of steps.review.outputs.files', + 'workflows/build.yaml:11: field "steps[1].if": step "fix" if compares .path with <, but it is not a number', + 'workflows/build.yaml:11: field "steps[1].if": step "fix" if passes .path to any as its list, but it is not an array', + ]); + }); + + it('accepts comparisons, functions and predicates that fit the declared types', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +inputs: + limit: number +steps: + - id: review + command: node review.js + output: + count: number + files: [{ path: string, lines: number, tags: [string] }] + - id: fix + needs: [review] + if: steps.review.outputs.count < inputs.limit && length(filter(steps.review.outputs.files, .lines >= 10)) > 0 && any(steps.review.outputs.files, any(.tags, . == 'bug') && .path != '') && all(steps.review.outputs.files, length(.tags) > 0) && steps.review.outputs.files[0].lines > 0 + command: node fix.js +`); + expect(result).toMatchObject({ ok: true }); + }); + + it('reports an if that calls a function on literals and is always false', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Never runs. +steps: + - id: one + if: any([], . == 1) || length([]) > 0 + command: node one.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics).toEqual([expect.objectContaining({ code: 'false_if', message: 'step "one" if is always false, so the step never runs' })]); + }); + + it('checks a predicate an inline step runs over an enclosing step once, in the enclosing workflow', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +steps: + - id: review + command: node review.js + output: + files: [{ path: string }] + - id: grp + needs: [review] + workflow: + steps: + - id: local + command: node local.js + output: + n: number + - id: use + needs: [local] + if: any(steps.review.outputs.files, .pth == 'a') && steps.local.outputs.n < length(steps.review.outputs.files) + command: node use.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics.map(formatDiagnostic)).toEqual([ + 'workflows/build.yaml:19: field "steps[1].workflow.steps[1].if": step "use" if reads .pth, which is not in an item of steps.review.outputs.files', + ]); + }); + + it('reports an end compared inside a predicate that the workflow a step runs does not declare', () => { + const result = parse(`schemaVersion: v1alpha1 +name: build +description: Check the review. +inputs: + tags: [string] +steps: + - id: check + workflow: + ends: + done: succeeded + steps: + - id: one + command: node one.js + end: done + - id: fix + needs: [check] + if: any(inputs.tags, steps.check.end == 'nope') + command: node fix.js +`); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.diagnostics).toEqual([expect.objectContaining({ + code: 'undeclared_end', + message: 'step "fix" if compares steps.check.end with \'nope\', which is not an end the inline workflow of step "check" declares', + })]); + }); + it('rejects a YAML syntax error with the file and line', () => { const result = parse('schemaVersion: [\n'); expect(result.ok).toBe(false); @@ -175,4 +386,459 @@ steps: expect(result.diagnostics[0]?.line).toBeGreaterThan(0); expect(formatDiagnostic(result.diagnostics[0]!)).toContain(`${FILE}:`); }); + + it('reports an expression nested 5,000 levels deep in an if and in a prompt template', () => { + const deep = `${'('.repeat(5000)}length(inputs.xs)${')'.repeat(5000)}`; + expect(rejected(`schemaVersion: v1alpha1 +name: build +description: Nest deep. +inputs: + xs: [string] +steps: + - id: explain + if: ${deep} > 0 + prompt: "Explain {{ ${deep} }} and {{ ${'('.repeat(5000)} }}" +`)).toEqual([ + { message: `${FILE}:8: field "steps[0].if": step "explain" if does not parse at column 101: an expression nests at most 100 levels deep, counting each "(", "!", call, && and ||` }, + { message: `${FILE}:9: field "steps[0].prompt": step "explain" prompt template {{ ${deep} }}: an expression nests at most 100 levels deep, counting each "(", "!", call, && and ||`, code: 'bad_template' }, + ]); + }); +}); + +function rejected(content: string): Array<{ message: string; code?: string }> { + const result = parse(content); + if (result.ok) throw new Error('expected the file to be rejected'); + return result.diagnostics.map((diagnostic) => ({ message: formatDiagnostic(diagnostic), ...(diagnostic.code ? { code: diagnostic.code } : {}) })); +} + +const REVIEW_AND_FIX = `schemaVersion: v1alpha1 +name: build +description: Review and fix, up to three times. +steps: + - id: review + command: node review.js + output: + verdict: pass | fix + findings: [string] + - id: fix + needs: [review] + if: steps.review.outputs.verdict == 'fix' + command: node fix.js + loop: { back_to: review, max_iterations: 3 } +`; + +describe('step loops', () => { + it('parses a loop back to a step the owner needs, with an optional while', () => { + const result = parse(`${REVIEW_AND_FIX} - id: decide + needs: [fix] + command: node decide.js + output: + choice: build | revise + loop: + back_to: decide + while: steps.decide.outputs.choice == 'revise' + max_iterations: 2 +`); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.definition.steps.map((step) => step.loop)).toEqual([ + undefined, + { backTo: 'review', maxIterations: 3 }, + { backTo: 'decide', while: "steps.decide.outputs.choice == 'revise'", maxIterations: 2 }, + ]); + }); + + it('rejects a loop that is not a map of back_to, max_iterations and while', () => { + const loop = (value: string): string => REVIEW_AND_FIX.replace('loop: { back_to: review, max_iterations: 3 }', `loop: ${value}`); + expect(rejected(loop('3'))).toEqual([ + { message: `${FILE}:14: field "steps[1].loop": step "fix" loop must map back_to, max_iterations, and optionally while`, code: 'bad_loop' }, + ]); + expect(rejected(loop("{ back_to: '', max_iterations: 0, while: 'steps.review.outputs.verdict ==', until: x }"))).toEqual([ + { message: `${FILE}:14: field "steps[1].loop.until": workflows/build.yaml uses \`until\`, which this teamai does not know. Update teamai.`, code: 'unknown_key' }, + { message: `${FILE}:14: field "steps[1].loop.back_to": step "fix" loop.back_to must name a step`, code: 'bad_loop' }, + { message: `${FILE}:14: field "steps[1].loop.max_iterations": step "fix" loop.max_iterations must be a whole number from 1`, code: 'bad_loop' }, + { message: `${FILE}:14: field "steps[1].loop.while": step "fix" loop.while does not parse at column 32: expected a value, found the end of the expression`, code: 'bad_loop' }, + ]); + }); + + it('rejects a loop back to a step its owner does not need, a loop on a step with end, and loops that share a step', () => { + expect(rejected(REVIEW_AND_FIX.replace('back_to: review', 'back_to: docs') + ` - id: docs + needs: [] + command: node docs.js +`)).toEqual([{ + message: `${FILE}:14: field "steps[1].loop.back_to": step "fix" loop.back_to is "docs", which is not "fix" or a step it needs, directly or through its needs`, + code: 'bad_loop', + }]); + expect(rejected(`${REVIEW_AND_FIX.replace('description:', 'ends:\n done: succeeded\ndescription:')} end: done +`)).toEqual([{ + message: `${FILE}:16: field "steps[1].loop": step "fix" has loop and end; a step that ends its workflow cannot run again`, + code: 'bad_loop', + }]); + expect(rejected(`${REVIEW_AND_FIX} - id: verify + needs: [fix] + command: node verify.js + loop: { back_to: fix, max_iterations: 2 } +`)).toEqual([{ + message: `${FILE}:18: field "steps[2].loop": step "verify" loop runs "fix" again, which the loop of step "fix" also runs; loops in one workflow share no step, so put the inner one in a sub-workflow`, + code: 'bad_loop', + }]); + }); + + it('checks what loop.while reads: its own step and the steps that step needs', () => { + expect(rejected(REVIEW_AND_FIX.replace('max_iterations: 3 }', "max_iterations: 3, while: \"steps.fix.outputs.again && steps.docs.outcome == 'success' || steps.review.outputs.verdcit\" }") + ` - id: docs + needs: [] + command: node docs.js +`)).toEqual([ + { message: `${FILE}:14: field "steps[1].loop.while": step "fix" loop.while reads steps.docs, which is not a step it needs, directly or through its needs`, code: 'bad_loop' }, + { message: `${FILE}:14: field "steps[1].loop.while": step "fix" loop.while reads steps.review.outputs.verdcit, which is not in the output step "review" declares`, code: 'bad_loop' }, + ]); + }); + + it('types steps..passes from the output its step declares', () => { + const reads = (condition: string): string => `${REVIEW_AND_FIX} - id: report + needs: [fix] + if: ${JSON.stringify(condition)} + command: node report.js +`; + expect(parse(reads("length(steps.review.passes) > 1 && steps.review.passes[0].outputs.verdict == 'fix' && any(steps.review.passes, .outputs.findings != [] && .pass > 1)")).ok).toBe(true); + expect(rejected(reads("steps.review.passes[0].outputs.verdcit == 'fix' || steps.review.passes[0].round == 1 || any(steps.review.passes, .outcome > 1)"))).toEqual([ + { message: `${FILE}:17: field "steps[2].if": step "report" if reads steps.review.passes[0].outputs.verdcit, which is not in a pass of step "review": pass, outcome, end, and outputs as its output declares` }, + { message: `${FILE}:17: field "steps[2].if": step "report" if reads steps.review.passes[0].round, which is not in a pass of step "review": pass, outcome, end, and outputs as its output declares` }, + { message: `${FILE}:17: field "steps[2].if": step "report" if compares .outcome with >, but it is not a number` }, + ]); + }); + + it('lets a step read a step after it in the same loop, in the previous pass, and no other step after it', () => { + expect(parse(REVIEW_AND_FIX.replace(" command: node review.js\n", " if: steps.fix.outcome != 'failed'\n command: node review.js\n")).ok).toBe(true); + expect(rejected(`schemaVersion: v1alpha1 +name: build +description: Two checks between a build and a gate that loops. +steps: + - id: build + command: node build.js + - id: unit + needs: [build] + if: steps.lint.outcome == 'success' + command: node unit.js + - id: lint + needs: [build] + command: node lint.js + - id: gate + needs: [unit, lint] + command: node gate.js + loop: { back_to: build, max_iterations: 2 } + - id: docs + needs: [] + if: steps.gate.outcome == 'success' + command: node docs.js +`)).toEqual([ + { message: `${FILE}:9: field "steps[1].if": step "unit" if reads steps.lint, which is not a step it needs, directly or through its needs` }, + { message: `${FILE}:20: field "steps[4].if": step "docs" if reads steps.gate, which is not a step it needs, directly or through its needs` }, + ]); + }); + + it('rejects two loops that wait for each other through the steps they need', () => { + expect(rejected(`schemaVersion: v1alpha1 +name: build +description: Two loops, each needed by the other's owner. +steps: + - id: build + command: node build.js + - id: fetch + needs: [] + command: node fetch.js + - id: test + needs: [build, fetch] + command: node test.js + loop: { back_to: build, max_iterations: 2 } + - id: lint + needs: [build, fetch] + command: node lint.js + loop: { back_to: fetch, max_iterations: 2 } +`)).toEqual([{ + message: `${FILE}:11: field "steps[2].needs": step "test" waits in a cycle: test -> lint -> test; a step that needs or reads a step inside a loop waits until the loop ends`, + code: 'bad_loop', + }]); + }); +}); + +describe('workflow loops', () => { + const POLL = `schemaVersion: v1alpha1 +name: build +description: Probe until ready. +inputs: + target: string +loop: + while: "steps.probe.outputs.status != 'ready' && inputs.target != ''" + max_iterations: 5 +steps: + - id: probe + command: node probe.js + output: + status: string + - id: report + needs: [probe] + command: node report.js + break: steps.probe.outputs.status == 'gone' + - id: retry + needs: [] + workflow: + loop: { max_iterations: 2 } + steps: + - id: try + command: node try.js +`; + + it('parses a loop on a workflow file and on an inline workflow, and a break on a step', () => { + const result = parse(POLL); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.definition.loop).toEqual({ while: "steps.probe.outputs.status != 'ready' && inputs.target != ''", maxIterations: 5 }); + expect(result.definition.steps[1]?.break).toBe("steps.probe.outputs.status == 'gone'"); + const retry = result.definition.steps[2]?.workflow; + expect(typeof retry === 'object' ? retry.loop : undefined).toEqual({ maxIterations: 2 }); + }); + + it('rejects a break inside the range of a step loop', () => { + expect(rejected(REVIEW_AND_FIX.replace(" command: node review.js\n", " command: node review.js\n break: steps.review.outputs.verdict == 'pass'\n"))).toEqual([{ + message: `${FILE}:7: field "steps[0].break": step "review" has break inside the loop of step "fix", where its if and while already end the loop`, + code: 'bad_loop', + }]); + }); + + it('rejects a step loop in a workflow that loops', () => { + expect(rejected(REVIEW_AND_FIX.replace('steps:\n', 'loop: { max_iterations: 2 }\nsteps:\n'))).toEqual([{ + message: `${FILE}:15: field "steps[1].loop": step "fix" has loop in a workflow that loops; put one of the two loops in a sub-workflow`, + code: 'bad_loop', + }]); + }); + + it('checks what a workflow loop.while reads: its own steps, the outputs they declare, and the inputs', () => { + expect(rejected(POLL + .replace("inputs.target != ''", "inputs.taget != '' || steps.probe.outputs.stauts == 'x' || steps.gone.outcome == 'success'") + .replace('loop: { max_iterations: 2 }', "loop: { while: \"steps.probe.outcome == 'failed'\", max_iterations: 2 }"))).toEqual([ + { message: `${FILE}:21: field "steps[2].workflow.loop.while": step "retry" workflow loop.while reads steps.probe, which is not one of its steps`, code: 'bad_loop' }, + { message: `${FILE}:7: field "loop.while": workflow loop.while reads inputs.taget, which is not an input the workflow declares`, code: 'bad_loop' }, + { message: `${FILE}:7: field "loop.while": workflow loop.while reads steps.probe.outputs.stauts, which is not in the output step "probe" declares`, code: 'bad_loop' }, + { message: `${FILE}:7: field "loop.while": workflow loop.while reads steps.gone, which is not one of its steps`, code: 'bad_loop' }, + ]); + }); + + it('rejects a workflow loop that is not a map of max_iterations and while, and a break that is not an expression', () => { + const header = (value: string): string => POLL.replace(/loop:\n {2}while: .*\n {2}max_iterations: 5\n/, `loop: ${value}\n`); + expect(rejected(header('3'))).toEqual([ + { message: `${FILE}:6: field "loop": workflow loop must map max_iterations, and optionally while`, code: 'bad_loop' }, + ]); + expect(rejected(header("{ back_to: probe, max_iterations: 0, while: 'steps.probe.outcome ==' }"))).toEqual([ + { message: `${FILE}:6: field "loop.back_to": workflow loop has no back_to; each round runs every step of the workflow`, code: 'bad_loop' }, + { message: `${FILE}:6: field "loop.max_iterations": workflow loop.max_iterations must be a whole number from 1`, code: 'bad_loop' }, + { message: `${FILE}:6: field "loop.while": workflow loop.while does not parse at column 23: expected a value, found the end of the expression`, code: 'bad_loop' }, + ]); + expect(rejected(POLL.replace("break: steps.probe.outputs.status == 'gone'", "break: \"steps.probe.outputs.status ==\""))).toEqual([ + { message: `${FILE}:17: field "steps[1].break": step "report" break does not parse at column 30: expected a value, found the end of the expression`, code: 'bad_loop' }, + ]); + expect(rejected(POLL.replace("break: steps.probe.outputs.status == 'gone'", 'break: [gone]'))).toEqual([ + { message: `${FILE}:17: field "steps[1].break": step "report" break must be an expression`, code: 'bad_loop' }, + ]); + }); +}); + +const FAN = `schemaVersion: v1alpha1 +name: build +description: Build each target. +inputs: + targets: [string] +steps: + - id: build + for_each: inputs.targets + command: node build.js + env: + TARGET: "{{ item }}" +`; + +const ITEM_TEXT = `schemaVersion: v1alpha1 +name: build +description: Explain loops that other tools run. +steps: + - id: explain + prompt: "Explain {{ item }}, {{ item.name }}, {{ item[0] }} and {{ length(item.xs) }} in a Jinja loop: {{ with.loop }}." + with: + loop: ["{{ item }}", "{{ item.name }}", "{{ length(item.xs) }}"] + - id: play + needs: [explain] + command: ansible localhost -m debug -a 'msg={{ item }}' + env: + ITEM: "{{ item }}" + NAME: "{{ item.name }}" + COUNT: "{{ length(item.xs) }}" +`; + +describe('for_each', () => { + it.each([ + ['loop', ' loop: { back_to: build, max_iterations: 2 }\n', '', 'step "build" has for_each and loop; put the loop in a sub-workflow'], + ['end', ' end: built\n', 'ends: { built: succeeded }\n', 'step "build" has for_each and end; a step that runs once per item cannot end its workflow'], + ['break', ' break: "true"\n', '', 'step "build" has for_each and break; a step that runs once per item cannot end its workflow'], + ])('rejects for_each with %s', (_key, line, header, message) => { + expect(rejected(FAN.replace('steps:\n', `${header}steps:\n`).replace(' command: node build.js\n', ` command: node build.js\n${line}`))).toEqual([ + { message: `${FILE}:${header ? 9 : 8}: field "steps[0].for_each": ${message}`, code: 'bad_loop' }, + ]); + }); + + it('keeps item as written in the templates and command of a step without for_each, where no item is bound', () => { + const result = parse(ITEM_TEXT); + + expect(result.ok ? result.definition.steps : result.diagnostics.map(formatDiagnostic)).toEqual([ + { + id: 'explain', + prompt: 'Explain {{ item }}, {{ item.name }}, {{ item[0] }} and {{ length(item.xs) }} in a Jinja loop: {{ with.loop }}.', + with: { loop: ['{{ item }}', '{{ item.name }}', '{{ length(item.xs) }}'] }, + }, + { + id: 'play', + needs: ['explain'], + command: "ansible localhost -m debug -a 'msg={{ item }}'", + env: { ITEM: '{{ item }}', NAME: '{{ item.name }}', COUNT: '{{ length(item.xs) }}' }, + }, + ]); + }); + + it('accepts a call on item in a template of a for_each step, where item is bound', () => { + const result = parse(FAN.replace('"{{ item }}"', '"{{ length(item.xs) }}"')); + + expect(result.ok ? result.definition.steps[0]?.env : result.diagnostics.map(formatDiagnostic)).toEqual({ TARGET: '{{ length(item.xs) }}' }); + }); + + it('rejects an if that reads item in a step without for_each', () => { + expect(rejected(ITEM_TEXT.replace(' command: ansible', ' if: item.x == 1\n command: ansible'))).toEqual([ + { message: `${FILE}:11: field "steps[1].if": step "play" if reads item, which is bound only in the templates of a for_each step and in the workflow each item runs` }, + ]); + }); + + it.each([ + ['if', " if: item != 'docs'\n", 'if'], + ['for_each', '', 'for_each'], + ])('rejects item in the %s of a for_each step, which is evaluated once before any item', (_what, line, field) => { + const content = FAN.replace(' for_each: inputs.targets\n', `${line} for_each: ${field === 'for_each' ? 'item.targets' : 'inputs.targets'}\n`); + expect(rejected(content)).toEqual([ + { message: `${FILE}:8: field "steps[0].${field}": step "build" ${field} reads item, which is bound only in the templates of a for_each step and in the workflow each item runs` }, + ]); + }); + + it.each([ + ['a list', '[a, b]', 'steps[0].for_each', 'step "build" for_each must be an expression, or a map of items and max_parallel'], + ['a map without items', '{ max_parallel: 2 }', 'steps[0].for_each.items', 'step "build" for_each.items must be an expression'], + ['max_parallel 0', '{ items: inputs.targets, max_parallel: 0 }', 'steps[0].for_each.max_parallel', 'step "build" for_each.max_parallel must be a whole number from 1'], + ])('rejects a for_each that is %s, so a kept copy cannot run the step once', (_what, value, field, message) => { + expect(rejected(FAN.replace('for_each: inputs.targets', `for_each: ${value}`))).toEqual([ + { message: `${FILE}:8: field "${field}": ${message}`, code: 'bad_loop' }, + ]); + }); + + it('rejects a read of the outputs of a for_each step, whose outputs are per item', () => { + const content = `${FAN.replace(' env:\n', ' output: { ok: boolean }\n env:\n')} - id: report + needs: [build] + if: steps.build.outputs.ok + command: node report.js +`; + expect(rejected(content)).toEqual([ + { message: `${FILE}:15: field "steps[1].if": step "report" if reads steps.build.outputs, but step "build" has for_each, so its outputs are in steps.build.results` }, + ]); + }); + + it('rejects a read of the results of a step without for_each', () => { + expect(rejected(`schemaVersion: v1alpha1 +name: build +description: Build. +steps: + - id: build + command: node build.js + - id: report + needs: [build] + if: any(steps.build.results, .outcome == 'failed') + command: node report.js +`)).toEqual([ + { message: `${FILE}:9: field "steps[1].if": step "report" if reads steps.build.results, but step "build" has no for_each, so it has no results` }, + ]); + }); + + it('reports a rejected for_each step once, not again at each read of its results', () => { + const content = `${FAN.replace('node build.js', 'node build.js {{ item }}')} - id: report + needs: [build] + if: any(steps.build.results, .outcome == 'failed') + command: node report.js +`; + expect(rejected(content)).toEqual([ + { message: `${FILE}:9: field "steps[0].command": step "build" command contains {{ item }}; command strings are not templated; pass the value through env:` }, + ]); + }); + + it('rejects item in a command, which is never rendered', () => { + expect(rejected(FAN.replace('node build.js', 'node build.js {{ item.name }}'))).toEqual([ + { message: `${FILE}:9: field "steps[0].command": step "build" command contains {{ item.name }}; command strings are not templated; pass the value through env:` }, + ]); + }); +}); + +const TRIAGE = `schemaVersion: v1alpha1 +name: build +description: Triage the builds. +inputs: + packages: [{ id: string }] +steps: + - id: plan + command: node plan.js + output: + title: string + files: [{ path: string }] + - id: build + needs: [plan] + for_each: inputs.packages + command: node build.js + - id: other + needs: [] + command: node other.js + - id: fix + needs: [build] + prompt: "Fix {{ length(filter(steps.build.results, .outcome == 'failed')) }} builds: {{ filter(steps.build.results, .outcome == 'failed') }}." +`; +const FIX_PROMPT = `"Fix {{ length(filter(steps.build.results, .outcome == 'failed')) }} builds: {{ filter(steps.build.results, .outcome == 'failed') }}."`; + +describe('expressions in templates', () => { + it('accepts an expression in a template and keeps the text to render when the step starts', () => { + const result = parse(TRIAGE); + + expect(result.ok ? result.definition.steps[3]?.prompt : result.diagnostics.map(formatDiagnostic)) + .toBe("Fix {{ length(filter(steps.build.results, .outcome == 'failed')) }} builds: {{ filter(steps.build.results, .outcome == 'failed') }}."); + }); + + it.each([ + ['a double-quoted string', '{{ filter(steps.build.results, .outcome == "failed") }}', "strings use single quotes, as in 'failed'"], + ['= for ==', "{{ any(steps.build.results, .outcome = 'failed') }}", 'any is called as any(list, predicate)'], + ['a field a step does not have', '{{ length(steps.build.result) }}', + '"steps.build.result" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], + ['a string compared with <', "{{ length(steps.build.results) < 'x' }}", "compares 'x' with <, but it is not a number"], + ])('rejects a call with %s, with the parse error', (_case, text, message) => { + expect(rejected(TRIAGE.replace(FIX_PROMPT, `'Fix ${text.replaceAll("'", "''")}.'`))).toEqual([ + { message: `${FILE}:21: field "steps[3].prompt": step "fix" prompt template ${text}: ${message}`, code: 'bad_template' }, + ]); + }); + + it('rejects an expression in a template that reads a step its step does not need, as in an if', () => { + expect(rejected(TRIAGE.replace(FIX_PROMPT, `"Fix it if {{ steps.other.outcome == 'success' }}."`))).toEqual([ + { message: `${FILE}:21: field "steps[3].prompt": step "fix" prompt reads steps.other, which is not a step it needs, directly or through its needs`, code: 'bad_template' }, + ]); + }); + + it('checks the types an expression in a template reads, as in an if', () => { + expect(rejected(TRIAGE.replace(FIX_PROMPT, '"Fix {{ length(steps.plan.outputs.title) }} and {{ filter(steps.plan.outputs.files, .finding) }}."'))).toEqual([ + { message: `${FILE}:21: field "steps[3].prompt": step "fix" prompt passes steps.plan.outputs.title to length as its list, but it is not an array`, code: 'bad_template' }, + { message: `${FILE}:21: field "steps[3].prompt": step "fix" prompt reads .finding, which is not in an item of steps.plan.outputs.files`, code: 'bad_template' }, + ]); + }); + + it('rejects a call in a command, which is never rendered', () => { + expect(rejected(TRIAGE.replace('node other.js', 'node other.js {{ length(inputs.packages) }}'))).toEqual([ + { message: `${FILE}:18: field "steps[2].command": step "other" command contains {{ length(inputs.packages) }}; command strings are not templated; pass the value through env:` }, + ]); + }); }); diff --git a/src/__tests__/workflow-reconciliation.test.ts b/src/__tests__/workflow-reconciliation.test.ts new file mode 100644 index 000000000..bc0524af3 --- /dev/null +++ b/src/__tests__/workflow-reconciliation.test.ts @@ -0,0 +1,415 @@ +import { createHash } from 'node:crypto'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { z } from 'zod'; +import type { LocalConfig } from '../types.js'; +import { syncWorkflowDefinitions } from '../workflow/cache.js'; +import { leaseNext, reportAttempt, startRun } from '../workflow/engine.js'; +import { readRun } from '../workflow/store.js'; +import type { DriverIdentity } from '../workflow/types.js'; + +const dirs: string[] = []; +afterEach(() => { + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +function testConfig(): LocalConfig { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-reconciliation-')); + dirs.push(root); + const knowledge = path.join(root, 'knowledge'); + fs.mkdirSync(path.join(knowledge, 'workflows'), { recursive: true }); + return { + repo: { localPath: knowledge, remote: 'local', kind: 'self' }, username: 'tester', scope: 'project', + additionalRoles: [], projectRoot: root, dataHome: path.join(root, 'data'), + }; +} + +const driver: DriverIdentity = { kind: 'local', machine: 'a', workspace: '/work' }; +const LeaseSchema = z.object({ step: z.string(), attempt: z.string(), revision: z.number() }); + +function random(seed: number): () => number { + return () => { + seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; + return seed / 4294967296; + }; +} + +const scenarios: Record; name: string; outputs: (step: string, r: () => number) => unknown }> = { + nested: { + name: 'top', + files: { top: `schemaVersion: v1alpha1 +name: top +description: d +ends: { fast: succeeded, slow: failed } +steps: + - id: plan + command: echo plan + - id: inner + needs: [plan] + workflow: + ends: { stop: succeeded, bad: failed } + outputs: { r: steps.review.outcome, f: steps.fix.outcome } + steps: + - id: review + command: echo review + - id: fix + needs: [review] + command: echo fix + loop: { back_to: review, max_iterations: 3 } + - id: guard + needs: [] + command: echo guard + end: stop + - id: bad + needs: [fix] + if: steps.fix.outcome == 'failed' + end: bad + - id: fast + needs: [] + command: echo fast + end: fast + - id: rec + needs: [inner] + if: steps.inner.outcome == 'failed' + end: slow + - id: after + needs: [inner] + command: echo after +` }, + outputs: () => undefined, + }, + fan: { + name: 'fan', + files: { + item: `schemaVersion: v1alpha1 +name: item +description: d +inputs: { n: number } +ends: { early: succeeded } +steps: + - id: a + command: echo a + - id: e + needs: [a] + if: inputs.n > 1 + end: early + - id: b + needs: [a] + command: echo b +`, + fan: `schemaVersion: v1alpha1 +name: fan +description: d +ends: { won: succeeded, broke: failed } +steps: + - id: plan + command: echo plan + output: { items: [number] } + - id: build + needs: [plan] + for_each: { items: steps.plan.outputs.items, max_parallel: 2 } + workflow: item + with: { n: "{{ item }}" } + - id: stop + needs: [build] + if: steps.build.outcome == 'failed' + end: broke + - id: fast + needs: [] + command: echo fast + end: won + - id: after + needs: [build] + command: echo after +`, + }, + outputs: (step, r) => (step === 'plan' ? { items: Array.from({ length: 1 + Math.floor(r() * 4) }, () => Math.floor(r() * 3)) } : undefined), + }, + fanafter: { + name: 'fa', + files: { fa: `schemaVersion: v1alpha1 +name: fa +description: d +ends: { won: succeeded, broke: failed } +steps: + - id: plan + command: echo plan + - id: build + needs: [plan] + for_each: { items: steps.plan.outputs.items, max_parallel: 2 } + command: echo build + - id: stop + needs: [build] + if: steps.build.outcome == 'failed' + end: broke + - id: fast + needs: [] + command: echo fast + end: won +` }, + outputs: (step, r) => (step === 'plan' ? { items: r() < 0.15 ? 'bad' : [{ id: 'a' }, { id: 'b', after: ['a'] }, { id: 'c' }, { id: 'd', after: ['c', 'b'] }] } : undefined), + }, + rounds: { + name: 'rounds', + files: { rounds: `schemaVersion: v1alpha1 +name: rounds +description: d +ends: { out: failed } +steps: + - id: w + workflow: + loop: { max_iterations: 3 } + ends: { red: failed } + steps: + - id: check + command: echo check + break: steps.check.outcome == 'success' && steps.check.outputs.ok == true + - id: red + needs: [check] + if: steps.check.outcome == 'failed' + end: red + - id: side + needs: [] + command: echo side + - id: fast + needs: [] + command: echo fast + end: out + - id: last + needs: [w] + command: echo last +` }, + outputs: (step, r) => (step.endsWith('check') ? { ok: r() < 0.3 } : undefined), + }, + loopwhile: { + name: 'lw', + files: { lw: `schemaVersion: v1alpha1 +name: lw +description: d +ends: { done: succeeded, stuck: failed, quick: succeeded } +steps: + - id: implement + command: echo i + - id: e2e + needs: [implement] + command: echo e2e + - id: fix + needs: [e2e] + if: steps.e2e.outcome == 'failed' + command: echo fix + loop: { back_to: e2e, max_iterations: 2 } + - id: done + needs: [e2e] + if: steps.e2e.outcome == 'success' + end: done + - id: give-up + needs: [fix] + if: steps.fix.outcome == 'failed' + end: stuck + - id: quick + needs: [] + command: echo q + end: quick + - id: tail + needs: [fix] + command: echo tail +` }, + outputs: () => undefined, + }, + itemloop: { + name: 'il', + files: { + fixer: `schemaVersion: v1alpha1 +name: fixer +description: d +inputs: { n: number } +ends: { gaveup: failed } +steps: + - id: review + command: echo review + - id: fix + needs: [review] + if: steps.review.outcome == 'success' + command: echo fix + loop: { back_to: review, max_iterations: 2 } + - id: give + needs: [fix] + if: steps.fix.outcome == 'failed' + end: gaveup +`, + il: `schemaVersion: v1alpha1 +name: il +description: d +ends: { won: succeeded, broke: failed } +steps: + - id: plan + command: echo plan + - id: build + needs: [plan] + for_each: { items: steps.plan.outputs.items, max_parallel: 1 } + workflow: fixer + with: { n: "{{ item }}" } + - id: retry + needs: [build] + if: steps.build.outcome == 'failed' + for_each: "filter(steps.build.results, .outcome == 'failed')" + command: echo retry + - id: stop + needs: [retry] + if: steps.retry.outcome == 'failed' + end: broke + - id: fast + needs: [] + command: echo fast + end: won +`, + }, + outputs: (step, r) => (step === 'plan' ? { items: Array.from({ length: 1 + Math.floor(r() * 3) }, () => Math.floor(r() * 3)) } : undefined), + }, + deep: { + name: 'deep', + files: { deep: `schemaVersion: v1alpha1 +name: deep +description: d +ends: { top: failed } +steps: + - id: a + command: echo a + - id: outer + needs: [] + workflow: + ends: { mid: succeeded } + steps: + - id: m + command: echo m + - id: inner + needs: [] + workflow: + ends: { low: failed } + steps: + - id: x + command: echo x + - id: y + needs: [] + command: echo y + - id: lowend + needs: [x, y] + if: steps.x.outcome == 'failed' || steps.y.outcome == 'skipped' + end: low + - id: z + needs: [x] + command: echo z + - id: midend + needs: [m] + if: steps.m.outcome == 'success' + end: mid + - id: after + needs: [inner] + command: echo after + - id: topend + needs: [a] + if: steps.a.outcome == 'failed' + end: top + - id: tail + needs: [outer, a] + command: echo tail +` }, + outputs: () => undefined, + }, +}; + +async function replay(config: LocalConfig, key: string, seed: number, between: boolean): Promise { + const scenario = scenarios[key]!; + const started = await startRun(config, scenario.name); + if ('error' in started) throw new Error(started.error.message); + const runId = String(started.data.run); + const nextRandom = random(seed); + const held = new Map>(); + /** Every lease handed out and not reported, those a restart dropped included. */ + const unreported = new Set(); + const names = new Map(); + const normalize = (value: unknown): unknown => JSON.parse(JSON.stringify(value).replaceAll(runId, '') + .replace(/a[0-9a-f]{8}/g, (attempt) => names.get(attempt) ?? attempt)); + const trace: unknown[] = []; + let done = false; + for (let turn = 0; turn < 120; turn += 1) { + const restart = held.size > 0 && nextRandom() < 0.3; + if (restart || held.size === 0 || nextRandom() < 0.4) { + const next = await leaseNext(config, runId, driver, { running: restart ? [] : [...held.keys()] }); + if ('error' in next) throw new Error(next.error.message); + // A restarted driver runs nothing, so it holds only the lease handed back to it. + if (restart) held.clear(); + if (next.data.lease) { + const lease = LeaseSchema.parse(next.data.lease); + if (!names.has(lease.attempt)) names.set(lease.attempt, `<${lease.step}#${names.size}>`); + held.set(lease.attempt, lease); + unreported.add(lease.attempt); + } + trace.push({ [restart ? 'restart' : 'next']: normalize({ ...next.data, cancelled: undefined }) }); + if (next.data.done) { + done = true; + break; + } + expect(next.data.lease || held.size > 0).toBeTruthy(); + } else { + const leases = [...held.values()]; + const lease = leases[Math.floor(nextRandom() * leases.length)]!; + const outcomeRoll = nextRandom(); + const outcome = outcomeRoll < 0.5 ? 'success' : outcomeRoll < 0.85 ? 'failed' : 'released'; + const outputs = scenario.outputs(lease.step, nextRandom); + const reported = await reportAttempt(config, runId, { + attempt: lease.attempt, revision: lease.revision, outcome, driver, + ...(outputs !== undefined ? { outputs } : {}), + }); + held.delete(lease.attempt); + unreported.delete(lease.attempt); + trace.push({ report: names.get(lease.attempt), outcome, response: normalize(reported) }); + // It lists the leases a restart dropped too, so it only applies what the report settled and hands nothing back. + if (between) await leaseNext(config, runId, driver, { running: [...unreported], canTakeLease: false }); + } + } + expect(done).toBe(true); + const stored = await readRun(config, runId); + if (typeof stored !== 'object') throw new Error(`run ${stored}`); + trace.push(normalize({ + steps: stored.steps, archive: stored.archive, revision: stored.revision, status: stored.status, + end: stored.end, failure: stored.failure, round: stored.round, + settledAttempts: stored.settledAttempts, cancelledAttempts: stored.cancelledAttempts, + })); + return trace; +} + +/** + * The sha1 of each scenario's traces, seeds 1 to 12, without and then with the extra next. Comparing the two replays + * catches what an extra next changes; the digest also catches a change to both, such as which lease a restart hands + * back. After a change that is meant to move leases, take the new digest from the failure. + */ +const TRACE_DIGESTS: Record = { + nested: '51c333ea87245bf7062dd75eedfd60e3a49bd40a', + fan: '8409e84163f667f26db9cb8fedd7033755c2377b', + fanafter: '837bae14a541473ba6915d84239732e9111ffe36', + rounds: '497799353921a38f7d2a2e6f63cbc63ee03bda81', + loopwhile: '63fdc60ac3cf158ababaa29b204edf2d099c2785', + itemloop: 'd0afb7c749c9a84c0e5141633ac02d7f38a16b06', + deep: '76c90bd6071acb9eb73f44b7d4587d80e3a313fc', +}; + +describe('workflow report reconciliation', () => { + it.each(Object.keys(scenarios))('keeps %s identical with an extra next between reports, and as it was', async (key) => { + const config = testConfig(); + for (const [file, body] of Object.entries(scenarios[key]!.files)) { + fs.writeFileSync(path.join(config.repo.localPath, 'workflows', `${file}.yaml`), body); + } + expect((await syncWorkflowDefinitions(config)).invalid).toEqual([]); + const digest = createHash('sha1'); + for (let seed = 1; seed <= 12; seed += 1) { + const plain = await replay(config, key, seed, false); + const extra = await replay(config, key, seed, true); + expect(plain, `${key}, seed ${seed}`).toEqual(extra); + digest.update(`${JSON.stringify({ key, seed, between: false, trace: plain })}\n${JSON.stringify({ key, seed, between: true, trace: extra })}\n`); + } + expect(digest.digest('hex')).toBe(TRACE_DIGESTS[key]); + }); +}); diff --git a/src/__tests__/workflow-records.test.ts b/src/__tests__/workflow-records.test.ts index 0580062ae..286861b13 100644 --- a/src/__tests__/workflow-records.test.ts +++ b/src/__tests__/workflow-records.test.ts @@ -1,6 +1,7 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; +import { fileURLToPath } from 'node:url'; import { simpleGit } from 'simple-git'; import { afterEach, describe, expect, it } from 'vitest'; import YAML from 'yaml'; @@ -15,7 +16,7 @@ import { recordPath, } from '../workflow/records.js'; import { generateRunId, isRunId } from '../workflow/store.js'; -import type { DriverIdentity, RunState, WorkflowDefinition } from '../workflow/types.js'; +import type { DriverIdentity, RunState, WorkflowDefinition, WorkflowDiagnostic } from '../workflow/types.js'; const dirs: string[] = []; @@ -66,8 +67,8 @@ function runState(overrides: Partial = {}): RunState { sourceHash: 'abc', definition, steps: [ - { id: 'one', status: 'succeeded', attempt: 'a1', outcome: 'success', exitCode: 0 }, - { id: 'two', status: 'succeeded', attempt: 'a2', outcome: 'success', exitCode: 0 }, + { id: 'one', status: 'succeeded', attempt: 'a1', settledAt: 2, outcome: 'success', exitCode: 0 }, + { id: 'two', status: 'succeeded', attempt: 'a2', settledAt: 3, outcome: 'success', exitCode: 0 }, ], settledAttempts: ['a1', 'a2', 'a1'], createdAt: '2026-10-14T10:00:00.000Z', @@ -76,6 +77,36 @@ function runState(overrides: Partial = {}): RunState { }; } +/** A settled run of a for_each over `items` items, each of which runs an inline workflow with a reviewer prompt. */ +function settledFan(items: number): RunState { + const done = { status: 'succeeded', outcome: 'success', settledAt: 2 } as const; + return runState({ + definition: { + ...definition, + steps: [ + { + id: 'build', + forEach: { items: 'inputs.xs' }, + workflow: { steps: [ + { id: 'compile', command: 'node compile.js' }, + { id: 'review', needs: ['compile'], prompt: 'Review it.', agent: 'reviewer' }, + ] }, + }, + { id: 'after', needs: ['build'], command: 'node after.js' }, + ], + }, + steps: [ + { id: 'build', ...done }, + ...Array.from({ length: items }, (_, index) => [ + { id: `build[${index}]`, item: index, ...done }, + { id: `build[${index}].compile`, ...done }, + { id: `build[${index}].review`, ...done }, + ]).flat(), + { id: 'after', ...done }, + ], + }); +} + describe('#1083 workflow run records', () => { it('builds a start record with exactly the allowlisted fields', () => { const record = buildStartRecord(runState({ status: 'running' }), driver, 'tester'); @@ -102,6 +133,7 @@ describe('#1083 workflow run records', () => { id: 'one', status: 'failed', attempt: 'a1', + settledAt: 2, outcome: 'failed', exitCode: 1, // Step state may carry failure text quoting the command and paths. @@ -119,20 +151,60 @@ describe('#1083 workflow run records', () => { } }); + it('publishes the failure code of a run only when teamai sets it', () => { + const failed = (code: string): RunState => runState({ status: 'failed', failure: { code, message: `the ${code} message` } }); + + expect(buildEndRecord(failed('loop_exhausted'), driver, 'tester').failure).toEqual({ code: 'loop_exhausted' }); + expect(JSON.stringify(buildEndRecord(failed('secret-x'), driver, 'tester'))).not.toContain('secret-x'); + }); + + it('publishes the failure code of every step failure teamai writes, and only codes it can name', () => { + // Codes a `code: '…'` literal in src/workflow writes that never become a run's failure. + const diagnostics = ['unknown_key', 'missing_needs', 'false_if', 'bad_template', 'dotted_name', 'undeclared_output', 'undeclared_end', 'bad_loop'] satisfies Array; + const answers = ['end_reached', 'repeated_report', 'wrong_attempt', 'lease_cancelled', 'interrupted', 'run_unreadable', 'driver_stuck', 'busy']; + const source = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', 'workflow'); + const written = new Set(fs.readdirSync(source).filter((file) => file.endsWith('.ts')) + .flatMap((file) => [...fs.readFileSync(path.join(source, file), 'utf8').matchAll(/code: '([^']+)'/g)].map((match) => match[1]!))); + const failedWith = (code: string): RunState => runState({ + status: 'failed', + steps: [{ id: 'one', status: 'failed', outcome: 'failed', attempt: 'a1', settledAt: 2, failure: { code, message: 'm' } }], + }); + + expect([...diagnostics, ...answers].filter((code) => !written.has(code))).toEqual([]); + expect([...written].filter((code) => ![...diagnostics, ...answers].includes(code) && buildEndRecord(failedWith(code), driver, 'tester').failure?.code !== code)).toEqual([]); + }); + + it('publishes the first failure teamai can name, under a top-level step whose own code it cannot', () => { + const run = runState({ + status: 'failed', + definition: { ...definition, steps: [{ id: 'build', workflow: { steps: [{ id: 'compile', command: 'node compile.js' }] } }] }, + steps: [ + { id: 'build', status: 'failed', outcome: 'failed', settledAt: 3, failure: { code: 'my secret code', message: 'm' } }, + { id: 'build.compile', status: 'failed', outcome: 'failed', attempt: 'a1', settledAt: 2, failure: { code: 'tool_failed', message: 'm' } }, + ], + }); + + expect(buildEndRecord(run, driver, 'tester').failure).toEqual({ code: 'tool_failed' }); + }); + it('builds an end record with outcome, duration, iterations, and step counts', () => { const record = buildEndRecord(runState(), driver, 'tester'); expect(record.phase).toBe('end'); expect(record.status).toBe('succeeded'); expect(record.durationMs).toBe(150000); expect(record.iterations).toBe(3); - expect(record.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0 }); + expect(record.stepCounts).toEqual({ total: 2, succeeded: 2, failed: 0, skipped: 0, cancelled: 0 }); expect(record.steps).toEqual([ - { step: 'one', outcome: 'success' }, - { step: 'two', outcome: 'success' }, + { step: 'one', pass: 1, outcome: 'success' }, + { step: 'two', pass: 1, outcome: 'success' }, ]); expect(record.endedAt).toBe('2026-10-14T10:02:30.000Z'); }); + it('gives the entry of a step that ran once pass 1, as its lease did', () => { + expect(buildEndRecord(runState(), driver, 'tester').steps?.map((entry) => entry.pass)).toEqual([1, 1]); + }); + it('records per-step agent fields: a Codex implementer beside a Claude reviewer', () => { // SaulMoro's P1: run-level tool/model cannot express a mixed-agent run. const mixed = runState({ @@ -144,17 +216,57 @@ describe('#1083 workflow run records', () => { ], }, steps: [ - { id: 'one', status: 'succeeded', attempt: 'a1', outcome: 'success', exitCode: 0, tool: 'codex', model: 'gpt-5.4-codex', tokens: { input: 100, output: 50 } }, - { id: 'two', status: 'succeeded', attempt: 'a2', outcome: 'success', exitCode: 0, tool: 'claude', model: 'claude-sonnet-4.6' }, + { id: 'one', status: 'succeeded', attempt: 'a1', settledAt: 2, outcome: 'success', exitCode: 0, tool: 'codex', model: 'gpt-5.4-codex', tokens: { input: 100, output: 50 } }, + { id: 'two', status: 'succeeded', attempt: 'a2', settledAt: 3, outcome: 'success', exitCode: 0, tool: 'claude', model: 'claude-sonnet-4.6' }, ], }); const record = buildEndRecord(mixed, driver, 'tester'); expect(record.steps).toEqual([ - { step: 'one', agent: 'implementer', tool: 'codex', model: 'gpt-5.4-codex', outcome: 'success', tokens: { input: 100, output: 50 } }, - { step: 'two', agent: 'reviewer', tool: 'claude', model: 'claude-sonnet-4.6', outcome: 'success' }, + { step: 'one', pass: 1, agent: 'implementer', tool: 'codex', model: 'gpt-5.4-codex', outcome: 'success', tokens: { input: 100, output: 50 } }, + { step: 'two', pass: 1, agent: 'reviewer', tool: 'claude', model: 'claude-sonnet-4.6', outcome: 'success' }, ]); }); + it.each([ + ['a local tool and a model name', 'claude-internal', 'anthropic/claude-sonnet-4.6@2026', { tool: 'claude-internal', model: 'anthropic/claude-sonnet-4.6@2026' }], + ['a secret as the tool', 'sk-ant-SECRET-token', 'gpt-5.4-codex', { model: 'gpt-5.4-codex' }], + ['an absolute path as the model', 'codex', '/Users/me/private/model', { tool: 'codex' }], + ['a relative path that climbs as the model', 'codex', 'models/../../home/me', { tool: 'codex' }], + ['a Windows absolute path as the model', 'codex', 'C:/Users/alice/private/model.gguf', { tool: 'codex' }], + ['a file URL as the model', 'codex', 'file:/Users/me/private/model', { tool: 'codex' }], + ['a tagged model name', 'codex', 'ollama:llama3.1:8b', { tool: 'codex', model: 'ollama:llama3.1:8b' }], + ['a model with a space', 'codex', 'my private model', { tool: 'codex' }], + ['a model longer than 80 characters', 'codex', `m${'x'.repeat(80)}`, { tool: 'codex' }], + ])('records the tool and model of a report with %s only when they are a local tool name and a model name', (_case, tool, model, fields) => { + const run = runState({ + definition: { ...definition, steps: [{ id: 'one', command: 'node one.js' }] }, + steps: [{ id: 'one', status: 'succeeded', attempt: 'a1', settledAt: 2, outcome: 'success', exitCode: 0, tool, model }], + }); + + expect(buildEndRecord(run, driver, 'tester').steps).toEqual([{ step: 'one', pass: 1, ...fields, outcome: 'success' }]); + }); + + it('builds the end record of a for_each with work linear in the item count', () => { + // Every lookup over the run's step instances reads each one's path, so the number of `id` reads counts how many it looks at. + const work = (items: number): number => { + let read = 0; + const run = settledFan(items); + run.steps = run.steps.map((state) => new Proxy(state, { + get: (target, key, receiver) => { + if (key === 'id') read += 1; + return Reflect.get(target, key, receiver); + }, + })); + expect(buildEndRecord(run, driver, 'tester').steps?.filter((entry) => entry.agent === 'reviewer')).toHaveLength(items); + return read; + }; + const small = work(100); + const large = work(200); + expect(small).toBeGreaterThan(100); + // Linear work doubles with the items; a scan of every instance per entry grows about four times. + expect(large / small).toBeLessThan(3); + }); + it('reports the cumulative iteration count, not the bounded history length', () => { // settledAttempts is a 32-entry dedup history; a run with 40 settled // attempts still has 32 entries. The record must say 40. diff --git a/src/__tests__/workflow-settle-cost.test.ts b/src/__tests__/workflow-settle-cost.test.ts new file mode 100644 index 000000000..6af0c68f8 --- /dev/null +++ b/src/__tests__/workflow-settle-cost.test.ts @@ -0,0 +1,236 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import type { LocalConfig } from '../types.js'; +import { syncWorkflowDefinitions } from '../workflow/cache.js'; +import { leaseNext, reportAttempt, startRun } from '../workflow/engine.js'; +import type { DriverIdentity } from '../workflow/types.js'; + +// Settling reads a step instance's scope from its path, so the number of paths parsed counts settle work without a clock. +const counter = vi.hoisted(() => ({ parsed: 0, read: 0, status: 0 })); +vi.mock('../workflow/step-path.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + parseStepPath: (value: string) => { + counter.parsed += 1; + return actual.parseStepPath(value); + }, + }; +}); + +// Every scan over a run's step instances reads each one's path, so the number of `id` reads counts how many it looks at. +// Counting the instances started against `max_parallel` reads only their `status`, which `status` reads count. +vi.mock('../workflow/store.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + readRun: async (...args: Parameters) => { + const run = await actual.readRun(...args); + if (typeof run === 'string') return run; + run.steps = run.steps.map((state) => new Proxy(state, { + get: (target, key, receiver) => { + if (key === 'id') counter.read += 1; + if (key === 'status') counter.status += 1; + return Reflect.get(target, key, receiver); + }, + })); + return run; + }, + }; +}); + +const dirs: string[] = []; +afterEach(() => { + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +const driver: DriverIdentity = { kind: 'local', machine: 'a', workspace: '/work' }; + +const FAN = `schemaVersion: v1alpha1 +name: fan +description: An uncapped for_each over an inline workflow. +inputs: + xs: [number] +steps: + - id: build + for_each: inputs.xs + workflow: + steps: + - id: compile + command: node compile.js + - id: test + needs: [compile] + command: node test.js + - id: after + needs: [build] + command: node after.js +`; + +/** The fan workflow with a workflow inside each item's workflow, so starting an item adds instances twice. */ +const NESTED_FAN = `schemaVersion: v1alpha1 +name: fan +description: An uncapped for_each over an inline workflow that runs another. +inputs: + xs: [number] +steps: + - id: build + for_each: inputs.xs + workflow: + steps: + - id: inner + workflow: + steps: + - id: compile + command: node compile.js + - id: test + needs: [inner] + command: node test.js + - id: after + needs: [build] + command: node after.js +`; + +/** An uncapped for_each whose items each run after the one before them. */ +const CHAIN = `schemaVersion: v1alpha1 +name: fan +description: An uncapped for_each over a chain of items. +inputs: + xs: [{ id: string }] +steps: + - id: build + for_each: inputs.xs + command: node build.js +`; + +/** A for_each that runs at most two items at a time. */ +const CAPPED = `schemaVersion: v1alpha1 +name: fan +description: A capped for_each. +inputs: + xs: [number] +steps: + - id: build + for_each: { items: inputs.xs, max_parallel: 2 } + command: node build.js +`; + +/** A run of a fan workflow over `items` items that no `next` has advanced yet. */ +async function startFan(items: number, body = FAN, item = (index: number): unknown => index): Promise<{ config: LocalConfig; run: string }> { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-workflow-settle-cost-')); + dirs.push(root); + const knowledge = path.join(root, 'knowledge'); + fs.mkdirSync(path.join(knowledge, 'workflows'), { recursive: true }); + fs.writeFileSync(path.join(knowledge, 'workflows', 'fan.yaml'), body); + const config: LocalConfig = { + repo: { localPath: knowledge, remote: 'local', kind: 'self' }, username: 'tester', scope: 'project', + additionalRoles: [], projectRoot: root, dataHome: path.join(root, 'data'), usageReportEnabled: false, + }; + await syncWorkflowDefinitions(config); + const started = await startRun(config, 'fan', { xs: Array.from({ length: items }, (_, index) => item(index)) }); + if ('error' in started) throw new Error(started.error.message); + return { config, run: String(started.data.run) }; +} + +/** The paths parsed by the first `next` of a run over `items` items, which starts every item and leases `first`. */ +async function firstNextWork(items: number, body: string, first: string): Promise { + const { config, run } = await startFan(items, body); + counter.parsed = 0; + const next = await leaseNext(config, run, driver); + expect(next).toMatchObject({ data: { lease: { step: first } } }); + return counter.parsed; +} + +/** The step instances looked at per `next` and `report` while one driver runs every step of a run over `items` items. */ +async function wholeRunWork(items: number): Promise { + const { config, run } = await startFan(items); + counter.read = 0; + let calls = 0; + for (;;) { + const next = await leaseNext(config, run, driver); + calls += 1; + const lease = 'data' in next ? next.data.lease as { attempt: string; revision: number } | undefined : undefined; + if (!lease) { + expect(next).toMatchObject({ data: { done: { outcome: 'succeeded' } } }); + break; + } + const report = await reportAttempt(config, run, { attempt: lease.attempt, revision: lease.revision, outcome: 'success', driver }); + expect(report).toMatchObject({ data: { accepted: true } }); + calls += 1; + } + // A lease and a report for each item's two steps and for `after`, then the `next` that answers done. + expect(calls).toBe(2 * (2 * items + 1) + 1); + return counter.read / calls; +} + +/** + * The step instances looked at by the `next` that skips a chain of `items` items behind the failed item at its head. + * Forward, each item runs after the one before it; in reverse, after the one after it. + */ +async function skippedChainWork(items: number, order: 'forward' | 'reverse'): Promise { + const head = order === 'forward' ? 0 : items - 1; + const { config, run } = await startFan(items, CHAIN, (index) => (index === head + ? { id: `i${index}` } + : { id: `i${index}`, after: [`i${order === 'forward' ? index - 1 : index + 1}`] })); + const first = await leaseNext(config, run, driver); + const lease = 'data' in first ? first.data.lease as { step: string; attempt: string; revision: number } : undefined; + expect(lease?.step).toBe(`build[${head}]`); + await reportAttempt(config, run, { attempt: lease!.attempt, revision: lease!.revision, outcome: 'failed', driver }); + counter.read = 0; + expect(await leaseNext(config, run, driver)).toMatchObject({ data: { done: { outcome: 'failed' } } }); + return counter.read; +} + +/** The step instance statuses read by a `next` of a capped for_each over `items` items while both its leases run. */ +async function cappedWork(items: number): Promise { + const { config, run } = await startFan(items, CAPPED); + const running: string[] = []; + for (const step of ['build[0]', 'build[1]']) { + const next = await leaseNext(config, run, driver, { running }); + const lease = 'data' in next ? next.data.lease as { step: string; attempt: string } : undefined; + expect(lease?.step).toBe(step); + running.push(lease!.attempt); + } + counter.status = 0; + expect(await leaseNext(config, run, driver, { running })).toMatchObject({ data: { wait: { reason: 'build[0] is leased' } } }); + return counter.status; +} + +describe('settle cost', () => { + it.each([ + ['an inline workflow', FAN, 'build[0].compile'], + ['an inline workflow that runs another', NESTED_FAN, 'build[0].inner.compile'], + ])('starts every item of an uncapped for_each over %s with work linear in the item count', async (_case, body, first) => { + const small = await firstNextWork(100, body, first); + const large = await firstNextWork(200, body, first); + expect(small).toBeGreaterThan(100); + // Linear work doubles with the items; quadratic work grows about four times. + expect(large / small).toBeLessThan(3); + }); + + // The quadratic form runs several times longer, so this test has its own timeout and fails on its ratio, not on time. + it('runs an uncapped for_each to the end with work per call linear in the item count', async () => { + const small = await wholeRunWork(100); + const large = await wholeRunWork(200); + expect(small).toBeGreaterThan(100); + // Each call settles every running item; a scan of all instances per item makes that quadratic. + expect(large / small).toBeLessThan(3); + }, 60_000); + + it.each(['forward', 'reverse'] as const)('skips a %s chain of items behind a failed item with work linear in the item count', async (order) => { + const small = await skippedChainWork(100, order); + const large = await skippedChainWork(200, order); + expect(small).toBeGreaterThan(100); + // Each skip makes the next item skippable; scanning every item again after each skip makes that quadratic. + expect(large / small).toBeLessThan(3); + }); + + it('waits on a full max_parallel with work linear in the item count', async () => { + const small = await cappedWork(100); + const large = await cappedWork(200); + expect(small).toBeGreaterThan(100); + // Counting the started items again for each waiting item makes that quadratic. + expect(large / small).toBeLessThan(3); + }); +}); diff --git a/src/__tests__/workflow-step-path.test.ts b/src/__tests__/workflow-step-path.test.ts new file mode 100644 index 000000000..66174d59f --- /dev/null +++ b/src/__tests__/workflow-step-path.test.ts @@ -0,0 +1,18 @@ +import { describe, expect, it } from 'vitest'; +import { parseStepPath } from '../workflow/step-path.js'; + +describe('workflow step paths', () => { + it.each([ + ['build', [{ id: 'build' }]], + ['ship.check.classify', [{ id: 'ship' }, { id: 'check' }, { id: 'classify' }]], + ['build[0]', [{ id: 'build', index: 0 }]], + ['a.b[2].c', [{ id: 'a' }, { id: 'b', index: 2 }, { id: 'c' }]], + ['review-v2_x.fan[12]', [{ id: 'review-v2_x' }, { id: 'fan', index: 12 }]], + ])('parses %s', (path, segments) => { + expect(parseStepPath(path)).toEqual(segments); + }); + + it.each(['b[x]', 'b[]', 'b[01]', 'b[-1]', 'b[1]x', 'b[1][2]'])('reads %s as a step id with no item index', (path) => { + expect(parseStepPath(path)).toEqual([{ id: path }]); + }); +}); diff --git a/src/__tests__/workflow-template.test.ts b/src/__tests__/workflow-template.test.ts index 0843727cb..372796032 100644 --- a/src/__tests__/workflow-template.test.ts +++ b/src/__tests__/workflow-template.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import type { ExpressionContext } from '../workflow/expression.js'; +import { evaluateExpression, parseExpression, type ExpressionContext } from '../workflow/expression.js'; import { findTemplate, parseTemplate, renderText, renderValue, type Template } from '../workflow/template.js'; const context: ExpressionContext = { @@ -7,8 +7,56 @@ const context: ExpressionContext = { steps: { plan: { outcome: 'success', outputs: { task: { files: ['a.ts'] }, count: 0 } }, lint: { outcome: 'skipped' } }, }; -function template(text: string): Template { - const parsed = parseTemplate(text); +const WITH = 'with is read only as one path, such as with.; an expression reads only inputs, steps and item'; +const DEEP = 'an expression nests at most 100 levels deep, counting each "(", "!", call, && and ||'; + +/** mulberry32: a seeded generator, so a failing case reruns from its seed. */ +function seeded(seed: number): () => number { + let state = seed; + return () => { + state = (state + 0x6d2b79f5) | 0; + let t = Math.imul(state ^ (state >>> 15), 1 | state); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +const OPENERS = [['(', ')'], ['!', ''], ['any(steps.build.results, ', ')'], ['all(inputs.labels, ', ')'], ['filter(item.xs, ', ')']] as const; +const VALUES = ["steps.build.outcome == 'it''s'", 'inputs.labels[0]', 'item.xs[0].ok[0]', 'length(inputs.labels) < 2', 'null', '[]', 'true']; +const JOINS = [' && ', ' || ']; +const STRAYS = ['.ok[0]', ')', '(', '!', '[0]', '}}', '{{ ', "'", "'}}'", '"', '.5', ', ', 'length(']; + +/** + * About `tokens` tokens: an expression that parses, nested in calls, "(" and "!", and chained with && and ||. A quarter + * stay whole, and the rest get stray tokens, are cut short, or open thousands of levels before the expression. + */ +function generated(seed: number, tokens: number): string { + const next = seeded(seed); + const below = (limit: number): number => Math.floor(next() * limit); + const pick = (list: readonly T[]): T => list[below(list.length)] as T; + const width = Math.ceil(Math.cbrt(tokens / 4)); + const chain = (part: () => string[]): string[] => [part(), ...Array.from({ length: below(width) }, () => [pick(JOINS), ...part()])].flat(); + const group = (): string[] => { + const openers = Array.from({ length: below(width) }, () => pick(OPENERS)); + return [...openers.map(([open]) => open), ...chain(() => [pick(VALUES)]), ...openers.reverse().map(([, close]) => close)]; + }; + const parts = chain(() => ['(', ...chain(group), ')']); + switch (below(4)) { + case 1: + for (let count = 1 + below(3); count > 0; count -= 1) parts.splice(below(parts.length + 1), 0, pick(STRAYS)); + break; + case 2: + parts.length = below(parts.length); + break; + case 3: + parts.unshift(...Array.from({ length: below(tokens) }, () => pick(OPENERS)[0])); + break; + } + return parts.join(''); +} + +function template(text: string, item = false): Template { + const parsed = parseTemplate(text, item); if (!parsed.ok) throw new Error(`${text}: ${parsed.errors[0]?.message}`); return parsed.template; } @@ -24,11 +72,30 @@ describe('workflow templates', () => { ['null as empty text', '[{{ inputs.none }}]', '[]'], ['a missing path as empty text', '[{{ steps.lint.outputs.report }}]', '[]'], ['an outcome', '{{ steps.lint.outcome }}', 'skipped'], + ['an array element', '{{ inputs.labels[1] }} {{ steps.plan.outputs.task.files[0] }}', 'p1 a.ts'], ['other braces as written', '${{ secrets.TOKEN }} {{ name }} {{ env.HOME }} {{inputs}}', '${{ secrets.TOKEN }} {{ name }} {{ env.HOME }} {{inputs}}'], ])('renders %s', (_case, text, rendered) => { expect(renderText(template(text), context)).toBe(rendered); }); + it.each([ + ['a count', '{{ length(inputs.labels) }}', '2'], + ['a filtered list', "{{ filter(inputs.labels, . == 'p1') }}", '["p1"]'], + ['a check on a list', "{{ any(inputs.labels, . == 'bug') }} {{ all(inputs.labels, . == 'bug') }}", 'true false'], + ['a compared count', '{{ length(inputs.labels) > 3 }}', 'false'], + ['a comparison that starts with a path', '{{ inputs.issue == 42 }} {{ inputs.issue < 1 }}', 'true false'], + ['an outcome compared', "{{ steps.lint.outcome == 'skipped' }}", 'true'], + ['a negated call', "{{ !any(inputs.labels, . == 'bug') }}", 'false'], + ['a call in parentheses', '{{ (length(inputs.labels)) }}', '2'], + ['a call after &&', "{{ true && any(inputs.labels, . == 'p1') }}", 'true'], + ['a call with a brace in its arguments', "{{ filter(inputs.labels, . == '}') }}", '[]'], + ['a nested call', "{{ length(filter(inputs.labels, . != 'bug')) }}", '1'], + ['a call on a missing list as empty text, the null it is in an if', '[{{ length(steps.lint.results) }}] [{{ filter(steps.lint.results, .ok) }}]', '[] []'], + ['a missing list compared as false, as in an if', '{{ length(steps.lint.results) > 0 }}', 'false'], + ])('evaluates %s', (_case, text, rendered) => { + expect(renderText(template(text), context)).toBe(rendered); + }); + it.each([ ['an object', '{{ steps.plan.outputs.task }}', { files: ['a.ts'] }], ['a number', '{{inputs.issue}}', 42], @@ -36,25 +103,42 @@ describe('workflow templates', () => { ['text around a path as a string', '#{{ inputs.issue }}', '#42'], ['two paths as a string', '{{ inputs.issue }}{{ inputs.draft }}', '42false'], ['plain text as it is', 'v1', 'v1'], + ['a filtered list as an array', "{{ filter(inputs.labels, . == 'p1') }}", ['p1']], + ['a count as a number', '{{ length(inputs.labels) }}', 2], + ['a check as a boolean', "{{ any(inputs.labels, . == 'bug') }}", true], + ['text around an expression as a string', 'n={{ length(inputs.labels) }}', 'n=2'], ])('renders a whole value: %s', (_case, text, value) => { expect(renderValue(template(text), context)).toEqual(value); }); it.each([ - ['{{ inputs.issue == 1 }}', 'a template reads one path'], - ['{{ steps.plan }}', '"steps.plan" must read steps..outcome, steps..end or steps..outputs'], + ['{{ steps.plan }}', '"steps.plan" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], ['{{ inputs.a b }}', 'unexpected "b"'], + ['{{ inputs.labels[-1] }}', '"[-1]" is not an index; write a whole number from 0, such as [0]'], + ['{{ with.refs[x] }}', '"[x]" is not an index; write a whole number from 0, such as [0]'], ])('rejects %s', (text, message) => { - expect(parseTemplate(`before ${text} after`)).toEqual({ ok: false, errors: [{ source: text, message }] }); + expect(parseTemplate(`before ${text} after`, false)).toEqual({ ok: false, errors: [{ source: text, message }] }); }); it('finds a template in text that is never rendered', () => { - expect(findTemplate('echo "{{inputs.title}}" "{{ steps.plan.outputs.task }}"')).toBe('{{inputs.title}}'); - expect(findTemplate('echo "${{ secrets.TOKEN }} {{ name }}"')).toBeUndefined(); + expect(findTemplate('echo "{{inputs.title}}" "{{ steps.plan.outputs.task }}"', false)).toBe('{{inputs.title}}'); + expect(findTemplate('echo "${{ secrets.TOKEN }} {{ name }}"', false)).toBeUndefined(); }); it('finds a ${{ … }} template in a command too, since the command would receive the braces as written', () => { - expect(findTemplate('echo "${{ inputs.title }}"')).toBe('{{ inputs.title }}'); + expect(findTemplate('echo "${{ inputs.title }}"', false)).toBe('{{ inputs.title }}'); + expect(findTemplate('echo "${{ length(inputs.files) }}"', false)).toBe('{{ length(inputs.files) }}'); + expect(findTemplate('echo "${{ !inputs.draft }}"', false)).toBe('{{ !inputs.draft }}'); + }); + + it('reads item only where an item is bound, and leaves it as written elsewhere', () => { + const text = 'Explain {{ item }}, {{ item.name }} and {{ item[0] }}.'; + const itemContext = { ...context, item: { name: 'web' } }; + + expect(renderText(template(text, true), itemContext)).toBe('Explain {"name":"web"}, web and .'); + expect(renderText(template(text, false), itemContext)).toBe(text); + expect(findTemplate('echo {{ item.name }}', true)).toBe('{{ item.name }}'); + expect(findTemplate('echo {{ item.name }}', false)).toBeUndefined(); }); it('renders a step\'s own with values, and leaves a template right after a $ as written', () => { @@ -62,13 +146,140 @@ describe('workflow templates', () => { expect(renderText(template('{{ with.task.title }} on {{with.ref}}, {{ with.none }}'), withContext)).toBe('Crash on main, '); expect(renderValue(template('{{ with.task }}'), withContext)).toEqual({ title: 'Crash' }); + expect(renderValue(template('{{ with.refs[1] }}'), { ...withContext, with: { refs: ['main', 'dev'] } })).toBe('dev'); expect(renderText(template('${{ inputs.issue }} ${{with.ref}} $ {{ inputs.issue }}'), withContext)).toBe('${{ inputs.issue }} ${{with.ref}} $ 42'); }); it.each([ ['{{ with.task == 1 }}'], ['{{ with. }}'], - ])('rejects %s as more than one with path', (text) => { - expect(parseTemplate(text)).toEqual({ ok: false, errors: [{ source: text, message: 'a template reads one path, such as with.' }] }); + ])('rejects %s, since with is read only as one path', (text) => { + expect(parseTemplate(text, false)).toEqual({ ok: false, errors: [{ source: text, message: WITH }] }); + }); + + it.each([ + ['a double-quoted string', '{{ filter(steps.build.results, .outcome == "failed") }}', "strings use single quotes, as in 'failed'"], + ['= for ==', "{{ any(steps.build.results, .outcome = 'failed') }}", 'any is called as any(list, predicate)'], + ['a field a step does not have', '{{ length(steps.build.result) }}', + '"steps.build.result" must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'], + ['a string compared with <', "{{ length(inputs.xs) < 'x' }}", "compares 'x' with <, but it is not a number"], + ['a call on a with value', '{{ length(with.xs) }}', '"with" is not readable; an expression reads only inputs, steps and item'], + ['a call after a third brace', '{{{ length(steps.build.results) }}', 'expected a value, found "{"'], + ['a call in 5,000 parentheses', `{{ ${'('.repeat(5000)}length(steps.build.results)${')'.repeat(5000)} }}`, DEEP], + ['a call after 20,000 !', `{{ ${'!'.repeat(20000)}any(steps.build.results, .ok) }}`, DEEP], + ['a call in a chain of 5,000 &&', `{{ length(steps.build.results)${' && true'.repeat(5000)} }}`, DEEP], + ])('rejects a call with %s, with the parse error', (_case, text, message) => { + expect(parseTemplate(`Fix ${text} first.`, false)).toEqual({ ok: false, errors: [{ source: text, message }] }); + }); + + it('renders a comparison of values nested 4,000 levels deep', () => { + const nested = (bottom: unknown): unknown => JSON.parse(`${'['.repeat(4000)}${JSON.stringify(bottom)}${']'.repeat(4000)}`); + const deep = { inputs: { a: nested(1), b: nested(1), c: nested(2) }, steps: {} }; + + expect(renderText(template('{{ inputs.a == inputs.b }} {{ inputs.a == inputs.c }}'), deep)).toBe('true false'); + }); + + it('evaluates a call on item only where an item is bound', () => { + const text = 'Count {{ length(item.xs) }}.'; + + expect(renderText(template(text, true), { ...context, item: { xs: [1, 2] } })).toBe('Count 2.'); + expect(renderText(template(text, false), { ...context, item: { xs: [1, 2] } })).toBe(text); + }); + + it('rejects a call that a }} inside its arguments cuts short', () => { + expect(parseTemplate("Fix {{ filter(steps.build.results, .name == '}}') }} first.", false)).toEqual({ + ok: false, errors: [{ source: "{{ filter(steps.build.results, .name == '}}", message: "a string is not closed with '" }], + }); + }); + + it.each([ + ['a Vue or Angular call', '{{ filter(items) }}'], + ['a Liquid call', '{{ all(products) }}'], + ['a Jinja filter', '{{ items | length }}'], + ['a Jinja call', '{{ range(10) }}'], + ['a GitHub Actions expression', '${{ length(steps.build.results) }}'], + ['a Handlebars block', '{{#each items}}{{this}}{{/each}}'], + ['a Handlebars triple-stash', '{{{ x }}}'], + ['a Go template', '{{ len .Items }} {{ .Name }}'], + ['a call on a root no expression reads, such as a typo of inputs', '{{ length(input.xs) }}'], + ['a call on literals alone', "{{ length('abc') }} {{ length(5) }} {{ any(3, .x) }}"], + ['5,000 parentheses', `{{ ${'('.repeat(5000)} }}`], + ])('leaves %s as text', (_case, text) => { + expect(renderText(template(`Explain ${text} here.`), context)).toBe(`Explain ${text} here.`); + }); + + it.each([10, 100, 1_000, 10_000])('parses, scans and evaluates generated text of about %i tokens without throwing', (tokens) => { + const bound = { ...context, item: { xs: [{ ok: [true] }] } }; + for (let seed = 1; seed <= 50; seed += 1) { + const text = generated(seed * tokens, tokens); + expect(() => { + const parsed = parseExpression(text); + if (parsed.ok) evaluateExpression(parsed.expression, bound); + for (const item of [false, true]) { + for (const wrapped of [text, `Fix {{ ${text} }}.`, `Fix {{ steps.build.results && ${text} }}.`, `Fix {{ length(${text}) }}.`]) { + findTemplate(wrapped, item); + const scanned = parseTemplate(wrapped, item); + if (scanned.ok) { + renderText(scanned.template, bound); + renderValue(scanned.template, bound); + } + } + } + }, `seed ${seed * tokens}`).not.toThrow(); + } + }); + + const tableContext = { + inputs: { name: 'N', title: 'T', x: true, l: [1, 2, 3] }, + steps: { a: { outcome: 'success', outputs: { l: [{ k: 'x' }, { k: 'y' }], n: 2 } }, build: { outcome: 'failed', results: [] } }, + item: { name: 'I' }, + } satisfies ExpressionContext; + const AS_WRITTEN = Symbol('as written'); + const STEP_FIELDS = 'must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'; + + it.each<[string, string, boolean, typeof AS_WRITTEN | { renders: string } | { error: string }]>([ + ['a Jinja filter', '{{ items | length }}', false, AS_WRITTEN], + ['a Jinja call', '{{ range(10) }}', false, AS_WRITTEN], + ['a Jinja filter on a root', '{{ inputs.name | upper }}', false, { error: 'unexpected "|"' }], + ['a Jinja variable and loop', '{{ name }} {% for x in xs %}{{ x.name }}{% endfor %}', false, AS_WRITTEN], + ['Liquid', "{{ all(products) }} {{ product.title | upcase }} {{ product.title | append: 'x' }}", false, AS_WRITTEN], + ['a Vue call', '{{ filter(items) }} {{ filter(items, fn) }}', false, AS_WRITTEN], + ['a Vue call that names inputs later', '{{ filter(items).map(i => inputs[i]) }}', false, AS_WRITTEN], + ['a Vue call on a bare steps or inputs', '{{ length(steps) }} of the wizard, {{ all(inputs, isValid) }}', false, AS_WRITTEN], + ['a Vue item outside a for_each', '{{ item.name }}', false, AS_WRITTEN], + ['a Vue item in a for_each', '{{ item.name }}', true, { renders: 'I' }], + ['a Vue method on a root', '{{ inputs.items.filter(x => x) }}', false, { error: 'unexpected "("' }], + ['an Angular pipe outside a for_each', '{{ item.title | uppercase }}', false, AS_WRITTEN], + ['an Angular pipe in a for_each', '{{ item.title | uppercase }}', true, { error: 'unexpected "|"' }], + ['Handlebars blocks', '{{#each inputs.items}}x{{/each}} {{#if (length inputs.items)}}y{{/if}} {{#each steps}}{{this}}{{/each}}', false, AS_WRITTEN], + ['a Handlebars triple-stash on a root', '{{{ inputs.title }}}', false, { renders: '{T}' }], + ['Go templates', '{{ .Values.x }} {{ len .Items }} {{ range .Items }}{{ .Name }}{{ end }} {{ index .Map "steps" }}', false, AS_WRITTEN], + ['GitHub Actions', '${{ inputs.x }} ${{ length(steps.a.outputs.l) > 0 }} ${{ steps.x.outputs.y }}', false, AS_WRITTEN], + ['an Ansible item outside a for_each', 'msg={{ item }}', false, AS_WRITTEN], + ['an Ansible item in a for_each', 'msg={{ item }}', true, { renders: 'msg={"name":"I"}' }], + ['prose quoting a teamai expression', '`{{ length(steps.a.outputs.l) }}`', false, { renders: '`2`' }], + ['a call on a typo of inputs', '{{ length(input.tags) }} {{ lenth(inputs.tags) }}', false, AS_WRITTEN], + ['a misspelled function name on a step', '{{ lenght(steps.build.results) }}', false, { error: '"lenght" is not a function; the functions are length, any, all and filter' }], + ['a misspelled steps root', '{{ length(step.build.results) }}', false, { error: '"step" is not readable; an expression reads only inputs, steps and item' }], + ['a negated path', '{{ !steps.build.outcome }} {{ !inputs.x }}', false, { renders: 'false false' }], + ['a path in parentheses', '{{ (steps.build.outcome) }} {{ (inputs.x) }}', false, { renders: 'failed true' }], + ['an unclosed call', '{{ length(steps.a.outputs.l }}', false, { error: 'length is called as length(list)' }], + ['a double-quoted string', '{{ any(steps.a.outputs.l, .k == "x") }}', false, { error: "strings use single quotes, as in 'failed'" }], + ['= for ==', '{{ inputs.x = 1 }}', false, { error: 'unexpected "="' }], + ['a field a step does not have', '{{ steps.build.result }}', false, { error: `"steps.build.result" ${STEP_FIELDS}` }], + ['a comparison that starts with a path', '{{ steps.a.outputs.n < 3 }}', false, { renders: 'true' }], + ['a call on with', '{{ length(with.items) }}', false, { error: '"with" is not readable; an expression reads only inputs, steps and item' }], + ['with in an expression', '{{ with.x == 1 }}', false, { error: WITH }], + ])('classifies %s', (_case, text, item, expected) => { + const parsed = parseTemplate(text, item); + const rendered = parsed.ok ? renderText(parsed.template, tableContext) : undefined; + const actual = !parsed.ok ? { error: parsed.errors[0]?.message } : rendered === text ? AS_WRITTEN : { renders: rendered }; + expect(actual).toEqual(expected); + }); + + it('finds a call in a command, which would receive its braces as written, and leaves other text', () => { + expect(findTemplate("echo {{ length(steps.build.results) }} {{ filter(items) }}", false)).toBe('{{ length(steps.build.results) }}'); + expect(findTemplate('echo {{ filter(items) }} {{ length(item.xs) }}', false)).toBeUndefined(); + expect(findTemplate('echo {{ length(item.xs) }}', true)).toBe('{{ length(item.xs) }}'); }); }); diff --git a/src/index.ts b/src/index.ts index 7886bdaa2..61e2a46f9 100644 --- a/src/index.ts +++ b/src/index.ts @@ -772,7 +772,7 @@ workflowCmd workflowCmd .command('next ') .description('Lease the next pending step of a run. Prints a JSON envelope.') - .option('--json ', 'Driver identity, for example {"driver":{"kind":"local","machine":"host","workspace":"/work"}}') + .option('--json ', 'Driver identity, for example {"driver":{"kind":"local","machine":"host","workspace":"/work"}}. Optional: running, the attempt ids of its open leases the driver still runs (an open lease left out is handed back), and canTakeLease, false to settle without taking a new lease. Every answer lists in cancelled the running attempts to stop.') .action(async (run: string, cmdOpts: { json?: string }) => { const { workflowNext } = await import('./workflow-commands.js'); await workflowNext(run, cmdOpts); diff --git a/src/types.ts b/src/types.ts index b1c0c30e9..bdbf2d310 100644 --- a/src/types.ts +++ b/src/types.ts @@ -858,6 +858,8 @@ export const LocalConfigSchema = z.object({ /** Per-machine tool selection for locally executed workflow prompt steps. */ workflows: z.object({ tools: z.record(z.string(), z.string().min(1)).default({}), + /** How many command steps the local driver runs at once. Unset means 1. */ + max_parallel: z.number().int().positive().optional(), }).optional(), /** * Per-machine map from a gateway/proxy model alias to a known Claude model diff --git a/src/workflow-commands.ts b/src/workflow-commands.ts index dada2e67a..dd68cb5ec 100644 --- a/src/workflow-commands.ts +++ b/src/workflow-commands.ts @@ -1,6 +1,6 @@ import { autoDetectInit } from './config.js'; import { setStderrOnly } from './utils/logger.js'; -import { leaseNext, parseDriverPayload, parseReport, reportAttempt, startRun } from './workflow/engine.js'; +import { leaseNext, parseNextPayload, parseReport, reportAttempt, startRun } from './workflow/engine.js'; import { runLocalDriver } from './workflow/local-driver.js'; import { scopedToolPaths, type LocalConfig, type TeamaiConfig } from './types.js'; import type { WorkflowEnvelope } from './workflow/types.js'; @@ -114,12 +114,13 @@ export async function workflowNext(runId: string, options: { json?: string }): P writeEnvelope(payload.envelope); return; } - const driver = parseDriverPayload(payload.value); - if (typeof driver === 'string') { - writeEnvelope({ error: { code: 'invalid_json', message: driver } }); + const request = parseNextPayload(payload.value); + if (typeof request === 'string') { + writeEnvelope({ error: { code: 'invalid_json', message: request } }); return; } - writeEnvelope(await leaseNext(config, runId, driver)); + const { driver, ...next } = request; + writeEnvelope(await leaseNext(config, runId, driver, next)); } export async function workflowReport(runId: string, options: { json?: string }): Promise { diff --git a/src/workflow/cache.ts b/src/workflow/cache.ts index 71f2be874..f658c5fae 100644 --- a/src/workflow/cache.ts +++ b/src/workflow/cache.ts @@ -11,7 +11,8 @@ import { managedMcpWorkspaceId, type LocalConfig, } from '../types.js'; -import { formatDiagnostic, parseWorkflow, promptFileDiagnostics, stepField, type WorkflowFiles } from './parse.js'; +import { formatDiagnostic, parseWorkflow, promptFileDiagnostics, stepField } from './parse.js'; +import type { WorkflowFiles } from './parse-context.js'; import type { CachedWorkflow, WorkflowDefinition, WorkflowDiagnostic, WorkflowStep } from './types.js'; import { assertSafePath, assertWithinRoot } from '../utils/path-safety.js'; diff --git a/src/workflow/conformance.ts b/src/workflow/conformance.ts index 9e27e0177..2cad08f6c 100644 --- a/src/workflow/conformance.ts +++ b/src/workflow/conformance.ts @@ -1,8 +1,8 @@ -import type { DriverIdentity, WorkflowEnvelope } from './types.js'; +import type { WorkflowEnvelope } from './types.js'; export interface WorkflowClient { start(name: string): Promise; - next(runId: string, driver: DriverIdentity): Promise; + next(runId: string, body: unknown): Promise; report(runId: string, body: unknown): Promise; } @@ -15,8 +15,9 @@ export interface ConformanceScenario { operations: ConformanceOperation[]; } +/** `running` and `cancelled` name leases that earlier operations saved; `cancelled` must match exactly. */ export type ConformanceOperation = - | { kind: 'next'; saveAs: string; expect: Record } + | { kind: 'next'; saveAs: string; running?: string[]; canTakeLease?: boolean; cancelled?: string[]; expect: Record } | { kind: 'report'; lease?: string; @@ -120,6 +121,80 @@ steps: task: "{{ steps.plan.outputs.task }}" `; +const PARALLEL_ROOTS = `schemaVersion: v1alpha1 +name: build +description: Two roots a driver runs in parallel, then a join. +steps: + - id: a + command: echo a + - id: b + needs: [] + command: echo b + - id: join + needs: [a, b] + command: echo join +`; + +const END_WHILE_RUNNING = `schemaVersion: v1alpha1 +name: build +description: A step reaches the end while another one runs. +ends: + done: succeeded +steps: + - id: slow + command: echo slow + - id: finish + needs: [] + command: echo finish + end: done +`; + +const STEP_LOOP = `schemaVersion: v1alpha1 +name: build +description: A fix sends the work back to review. +steps: + - id: review + command: echo review + output: + verdict: pass | fix + - id: fix + needs: [review] + if: steps.review.outputs.verdict == 'fix' + command: echo fix + loop: { back_to: review, max_iterations: 2 } +`; + +const BREAK_LOOP = `schemaVersion: v1alpha1 +name: build +description: Probe until ready while a slow check runs beside it. +loop: { max_iterations: 3 } +steps: + - id: slow + command: echo slow + - id: probe + needs: [] + command: echo probe + output: + ready: boolean + break: steps.probe.outputs.ready +`; + +const FAN_OUT = `schemaVersion: v1alpha1 +name: build +description: Build each target the plan lists, two at a time. +steps: + - id: plan + command: echo plan + output: + targets: [string] + - id: build + needs: [plan] + for_each: { items: steps.plan.outputs.targets, max_parallel: 2 } + command: echo build + env: + TARGET: "{{ item }}" +`; + export const workflowConformanceScenarios: ConformanceScenario[] = [ { name: 'two-step command success', @@ -237,4 +312,102 @@ export const workflowConformanceScenarios: ConformanceScenario[] = [ { kind: 'next', saveAs: 'fix', expect: { data: { lease: { step: 'fix', prompt: 'Fix the crash; keep ${{ inputs.version }}.' } } } }, ], }, + { + name: 'a driver that lists its running attempts gets parallel leases', + workflowName: 'build', + yaml: PARALLEL_ROOTS, + operations: [ + { kind: 'next', saveAs: 'a', expect: { data: { lease: { step: 'a' }, cancelled: [] } } }, + { kind: 'next', saveAs: 'b', running: ['a'], expect: { data: { lease: { step: 'b' }, cancelled: [] } } }, + { kind: 'next', saveAs: 'wait', running: ['a', 'b'], expect: { data: { wait: { reason: 'a is leased' }, cancelled: [] } } }, + { kind: 'next', saveAs: 'reconcile', running: ['a', 'b'], canTakeLease: false, expect: { data: { wait: { reason: 'a is leased' } } } }, + { kind: 'next', saveAs: 'again', running: ['b'], expect: { data: { lease: { step: 'a' } } } }, + { kind: 'report', lease: 'a', body: { outcome: 'success' }, expect: { data: { step: { id: 'a', status: 'succeeded' } } } }, + ], + }, + { + name: 'a report of a sibling does not make a parallel attempt stale', + workflowName: 'build', + yaml: PARALLEL_ROOTS, + operations: [ + { kind: 'next', saveAs: 'a', expect: { data: { lease: { step: 'a', revision: 1 } } } }, + { kind: 'next', saveAs: 'b', running: ['a'], expect: { data: { lease: { step: 'b', revision: 1 } } } }, + { kind: 'report', lease: 'a', body: { outcome: 'success' }, expect: { data: { revision: 2, step: { id: 'a', status: 'succeeded' } } } }, + { kind: 'report', lease: 'b', body: { outcome: 'success' }, expect: { data: { revision: 3, step: { id: 'b', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'join', expect: { data: { lease: { step: 'join', revision: 3 } } } }, + ], + }, + { + name: 'an end cancels a running lease, and its report is lease_cancelled', + workflowName: 'build', + yaml: END_WHILE_RUNNING, + operations: [ + { kind: 'next', saveAs: 'slow', expect: { data: { lease: { step: 'slow' } } } }, + { kind: 'next', saveAs: 'finish', running: ['slow'], expect: { data: { lease: { step: 'finish' } } } }, + { kind: 'report', lease: 'finish', body: { outcome: 'success' }, expect: { data: { step: { id: 'finish', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'done', running: ['slow'], cancelled: ['slow'], expect: { data: { done: { outcome: 'succeeded', end: 'done' } } } }, + { kind: 'next', saveAs: 'again', running: ['slow'], cancelled: ['slow'], expect: { data: { done: { outcome: 'succeeded', end: 'done' } } } }, + { kind: 'report', lease: 'slow', body: { outcome: 'success' }, expect: { error: { code: 'lease_cancelled' } } }, + ], + }, + { + name: 'a report after an earlier end is lease_cancelled before next', + workflowName: 'build', + yaml: END_WHILE_RUNNING, + operations: [ + { kind: 'next', saveAs: 'slow', expect: { data: { lease: { step: 'slow' } } } }, + { kind: 'next', saveAs: 'finish', running: ['slow'], expect: { data: { lease: { step: 'finish' } } } }, + { kind: 'report', lease: 'finish', body: { outcome: 'success' }, expect: { data: { accepted: true } } }, + { kind: 'report', lease: 'slow', body: { outcome: 'success' }, expect: { error: { code: 'lease_cancelled' } } }, + { kind: 'next', saveAs: 'done', expect: { data: { done: { outcome: 'succeeded', end: 'done' } } } }, + { kind: 'report', lease: 'slow', body: { outcome: 'success' }, expect: { error: { code: 'lease_cancelled' } } }, + ], + }, + { + name: 'a step loop leases its range again, and each lease names its pass', + workflowName: 'build', + yaml: STEP_LOOP, + operations: [ + { kind: 'next', saveAs: 'review1', cancelled: [], expect: { data: { lease: { step: 'review', pass: 1 } } } }, + { kind: 'report', lease: 'review1', body: { outcome: 'success', outputs: { verdict: 'fix' } }, expect: { data: { step: { id: 'review', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'fix', cancelled: [], expect: { data: { lease: { step: 'fix', pass: 1 } } } }, + { kind: 'report', lease: 'fix', body: { outcome: 'success' }, expect: { data: { step: { id: 'fix', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'review2', cancelled: [], expect: { data: { lease: { step: 'review', pass: 2 } } } }, + { kind: 'report', lease: 'review2', body: { outcome: 'success', outputs: { verdict: 'pass' } }, expect: { data: { step: { id: 'review', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'done', cancelled: [], expect: { data: { done: { outcome: 'succeeded' } } } }, + ], + }, + { + name: 'a looping workflow runs every step again each round, and a break ends it and cancels what still runs', + workflowName: 'build', + yaml: BREAK_LOOP, + operations: [ + { kind: 'next', saveAs: 'slow1', cancelled: [], expect: { data: { lease: { step: 'slow', pass: 1 } } } }, + { kind: 'next', saveAs: 'probe1', running: ['slow1'], cancelled: [], expect: { data: { lease: { step: 'probe', pass: 1 } } } }, + { kind: 'report', lease: 'probe1', body: { outcome: 'success', outputs: { ready: false } }, expect: { data: { step: { id: 'probe', status: 'succeeded' } } } }, + { kind: 'report', lease: 'slow1', body: { outcome: 'success' }, expect: { data: { step: { id: 'slow', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'slow2', cancelled: [], expect: { data: { lease: { step: 'slow', pass: 2 } } } }, + { kind: 'next', saveAs: 'probe2', running: ['slow2'], cancelled: [], expect: { data: { lease: { step: 'probe', pass: 2 } } } }, + { kind: 'report', lease: 'probe2', body: { outcome: 'success', outputs: { ready: true } }, expect: { data: { step: { id: 'probe', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'done', running: ['slow2'], cancelled: ['slow2'], expect: { data: { done: { outcome: 'succeeded' } } } }, + { kind: 'report', lease: 'slow2', body: { outcome: 'success' }, expect: { error: { code: 'lease_cancelled' } } }, + ], + }, + { + name: 'a for_each leases its items in parallel up to max_parallel, and a failed item cancels none of them', + workflowName: 'build', + yaml: FAN_OUT, + operations: [ + { kind: 'next', saveAs: 'plan', expect: { data: { lease: { step: 'plan' } } } }, + { kind: 'report', lease: 'plan', body: { outcome: 'success', outputs: { targets: ['a', 'b', 'c'] } }, expect: { data: { step: { id: 'plan', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'a', cancelled: [], expect: { data: { lease: { step: 'build[0]', env: { TARGET: 'a' } } } } }, + { kind: 'next', saveAs: 'b', running: ['a'], cancelled: [], expect: { data: { lease: { step: 'build[1]', env: { TARGET: 'b' } } } } }, + { kind: 'next', saveAs: 'full', running: ['a', 'b'], cancelled: [], expect: { data: { wait: { reason: 'build[0] is leased' } } } }, + { kind: 'report', lease: 'a', body: { outcome: 'failed' }, expect: { data: { step: { id: 'build[0]', status: 'failed' } } } }, + { kind: 'next', saveAs: 'c', running: ['b'], cancelled: [], expect: { data: { lease: { step: 'build[2]', env: { TARGET: 'c' } } } } }, + { kind: 'report', lease: 'b', body: { outcome: 'success' }, expect: { data: { step: { id: 'build[1]', status: 'succeeded' } } } }, + { kind: 'report', lease: 'c', body: { outcome: 'success' }, expect: { data: { step: { id: 'build[2]', status: 'succeeded' } } } }, + { kind: 'next', saveAs: 'done', cancelled: [], expect: { data: { done: { outcome: 'failed' } } } }, + ], + }, ]; diff --git a/src/workflow/engine.ts b/src/workflow/engine.ts index e8c0f4668..aeacb30ed 100644 --- a/src/workflow/engine.ts +++ b/src/workflow/engine.ts @@ -4,21 +4,27 @@ import type { LocalConfig } from '../types.js'; import { loadCachedWorkflow, workflowsRunBy } from './cache.js'; import { evaluateExpression, expressionPaths, isRecord, isTruthy, parseExpression, resolvePath, - type Expression, type ExpressionPath, type ExpressionStep, type VisibleOutcome, + type Expression, type ExpressionPath, } from './expression.js'; import { outputMismatch } from './output.js'; -import { noteRunRecord, runRecordState } from './records.js'; -import { parseTemplate, renderStrings, renderText, templatedTexts, templatePaths, type TemplateContext, type TemplateError } from './template.js'; +import { publishRunRecords, queueRunRecord, runRecordState } from './records.js'; +import { + parseTemplate, renderStrings, renderText, stepExpressions, templatedTexts, templatePaths, + type StepExpression, type TemplateError, +} from './template.js'; +import { parseStepPath } from './step-path.js'; +import { cancel, fail, lease, release, settled, skip, start, stopped, succeed, type ItemResult, type LeasedStep, type StepState } from './step-state.js'; +import { addItems, afterStanding, expandItems, isItem, itemsAfter, itemsOf, itemsSettled, startedItems } from './fan-out.js'; +import { decideRound, decideStepLoop, exhaustedAtStart, firstPass, loopSettled } from './loop.js'; +import { childScope, contextOf, insertChildren, localId, statesIn, topScope, type Scope } from './scope.js'; import { generateRunId, isRunId, readRun, runLockPath, writeRun } from './store.js'; import type { CachedWorkflow, DriverIdentity, - InlineWorkflow, OutputSchema, RunState, RunStatus, StepOutcome, - StepState, TokenUsage, WorkflowDefinition, WorkflowEnvelope, @@ -102,51 +108,20 @@ interface RenderedLease { env?: Record; } -/** A workflow inside a run: the top-level one, or the sub-workflow a step runs. */ -interface Scope { - /** What every step path inside it starts with: '' at the top, `.` inside a sub-workflow. */ - prefix: string; - definition: WorkflowDefinition | InlineWorkflow; - inputs: Record | undefined; - /** For an inline workflow: the enclosing workflows' steps, by id, which its own steps hide. */ - outer?: Record; -} - -function topScope(run: RunState): Scope { - return { prefix: '', definition: run.definition, inputs: run.inputs }; -} - -/** - * The scope of the workflow a step runs, or undefined when the run did not keep that workflow. - * An inline workflow reads the inputs and the steps of the workflow `scope` around it. - */ -function childScope(run: RunState, scope: Scope, states: StepState[], state: StepState, step: WorkflowStep): Scope | undefined { - if (typeof step.workflow === 'object') { - return { prefix: `${state.id}.`, definition: step.workflow, inputs: scope.inputs, outer: expressionSteps(scope, states) }; - } - const definition = step.workflow === undefined ? undefined : run.workflows?.[step.workflow]; - return definition && { prefix: `${state.id}.`, definition, inputs: state.inputs }; -} - -/** The step instances directly inside a scope, in declaration order. */ -function statesIn(run: RunState, scope: Scope): StepState[] { - return run.steps.filter((state) => state.id.startsWith(scope.prefix) && !state.id.slice(scope.prefix.length).includes('.')); -} - -/** The step id an instance has inside its own workflow. */ -function localId(scope: Scope, state: StepState): string { - return state.id.slice(scope.prefix.length); +function workflowLabel(scope: Scope): string { + return 'name' in scope.definition ? `workflow "${scope.definition.name}"` : `step "${scope.prefix.slice(0, -1)}" inline workflow`; } /** The scope and definition of the step instance at `path`, or undefined when the run file does not describe it. */ -export function locateStep(run: RunState, path: string): { scope: Scope; step: WorkflowStep } | undefined { +function locateStep(run: RunState, path: string): { scope: Scope; step: WorkflowStep } | undefined { let scope = topScope(run); - const ids = path.split('.'); - for (const [index, id] of ids.entries()) { - const step = scope.definition.steps.find((item) => item.id === id); + const segments = parseStepPath(path); + for (const [index, segment] of segments.entries()) { + const step = scope.definition.steps.find((item) => item.id === segment.id); if (!step) return undefined; - if (index === ids.length - 1) return { scope, step }; - const state = run.steps.find((item) => item.id === `${scope.prefix}${id}`); + if (index === segments.length - 1) return { scope, step }; + const instance = `${scope.prefix}${segment.id}${segment.index === undefined ? '' : `[${segment.index}]`}`; + const state = run.steps.find((item) => item.id === instance); const child = state && childScope(run, scope, statesIn(run, scope), state, step); if (!child) return undefined; scope = child; @@ -165,7 +140,7 @@ function withTemplateFailure(state: StepState, error: TemplateError): WorkflowFa * Pull rejects a template that does not parse, so a failure here means the run file was changed. */ function renderLease(scope: Scope, states: StepState[], state: StepState, definition: WorkflowStep): RenderedLease | WorkflowFailure { - const context: TemplateContext = { inputs: scope.inputs, steps: expressionSteps(scope, states) }; + const context = contextOf(scope, states, state); // A prompt step's `with` renders first, by the same rules as a workflow step's; its prompt reads it as `with.`. if (definition.with !== undefined) { const values = renderStrings(definition.with, context); @@ -175,7 +150,7 @@ function renderLease(scope: Scope, states: StepState[], state: StepState, defini const rendered = new Map(); for (const { what, text } of templatedTexts(definition)) { if (what.startsWith('with.')) continue; - const parsed = parseTemplate(text); + const parsed = parseTemplate(text, Object.hasOwn(context, 'item')); if (!parsed.ok) { const [problem] = parsed.errors; return { code: 'invalid_template', message: `step "${state.id}" ${what} template ${problem?.source}: ${problem?.message}` }; @@ -189,53 +164,62 @@ function renderLease(scope: Scope, states: StepState[], state: StepState, defini }; } -function leaseEnvelope(run: RunState, state: StepState, definition: WorkflowStep, rendered: RenderedLease): WorkflowEnvelope { - return ok({ lease: { +function leaseData(run: RunState, state: LeasedStep, definition: WorkflowStep, rendered: RenderedLease): Record { + return { lease: { run: run.id, step: state.id, - attempt: state.lease?.attempt, - revision: state.lease?.revision, + attempt: state.lease.attempt, + revision: state.lease.revision, + pass: state.pass ?? 1, ...(definition.command !== undefined ? { command: definition.command } : {}), ...(rendered.env ? { env: rendered.env } : {}), ...(rendered.prompt !== undefined ? { prompt: rendered.prompt } : {}), ...(definition.agent !== undefined ? { agent: definition.agent } : {}), ...(definition.output ? { output: definition.output } : {}), - } }); + } }; } -/** Fails a step the engine cannot hand out, so the rest of the graph keeps settling. Like a skip, it does not bump `revision`. */ -function failStep(step: StepState, failure: WorkflowFailure): void { - step.status = 'failed'; - step.outcome = 'failed'; - step.lease = undefined; - step.attempt = undefined; - step.message = failure.message; - step.failure = failure; +/** + * Fails a step the engine cannot hand out, so the rest of the graph keeps settling. Like a skip, it does not bump + * `revision`. `settledAt` is the revision the failure is stamped with: the run's current one, or that of an open lease. + */ +function failStep(step: StepState, failure: WorkflowFailure, settledAt: number): void { + fail(step, { settledAt, message: failure.message, failure }); } /** The status of a run, or of a workflow inside it, once it has settled. */ type FinalRunStatus = Exclude; -/** How a workflow of the run settled: its outcome, the end it reached, and the outputs it returns. */ +/** + * How a workflow of the run settled: its outcome, the end it reached, and the outputs it returns. `settledAt` is + * the run's current revision. + */ interface WorkflowResult { outcome: FinalRunStatus; end?: string; outputs?: Record; + settledAt: number; + /** Why a workflow that reached no failed end failed, such as `loop_exhausted`. */ + failure?: WorkflowFailure; + /** For the items of a `for_each` step, which settle like a workflow: how each one settled. */ + results?: ItemResult[]; } -/** - * Ends the run. Every path that settles a run goes through here, so anything that reacts to a settled run hooks in here. - * Returns a copy of the settled run for the end record, which publishes after the run lock is released (#1126). - */ -function settleRun(run: RunState, now: string, outcome: FinalRunStatus, end?: string): RunState { +/** The revision a step settled at; 0 before it settles, or in a run file from an older teamai. */ +function settledAtOf(step: StepState | undefined): number { + return step && 'settledAt' in step ? step.settledAt : 0; +} + +/** Ends the run. Every path that settles a run goes through here, so anything that reacts to a settled run hooks in here. */ +function settleRun(run: RunState, now: string, { outcome, end, failure }: WorkflowResult): void { run.status = outcome; if (end !== undefined) run.end = end; + if (failure) run.failure = failure; run.updatedAt = now; - return { ...run, steps: run.steps.map((step) => ({ ...step })) }; } -function doneEnvelope(run: RunState): WorkflowEnvelope { - return ok({ done: { run: run.id, outcome: run.status, ...(run.end !== undefined ? { end: run.end } : {}) } }); +function doneData(run: RunState): Record { + return { done: { run: run.id, outcome: run.status, ...(run.end !== undefined ? { end: run.end } : {}), ...(run.failure ? { failure: run.failure } : {}) } }; } /** A step with `end` and no action only ends the workflow; the engine settles it without a lease. */ @@ -243,110 +227,124 @@ function endsOnly(step: WorkflowStep | undefined): boolean { return step?.end !== undefined && !hasAction(step); } -/** A step that ran and did not succeed. Later issues add `timeout` and `cancelled`. */ -function stopped(step: StepState): boolean { - return step.status === 'failed'; -} - -function settled(step: StepState | undefined): boolean { - return step?.status === 'succeeded' || step?.status === 'failed' || step?.status === 'skipped'; -} - /** The step ids these paths read. */ function stepsRead(paths: ExpressionPath[]): Set { return new Set(paths.flatMap((path) => (path.root === 'steps' ? [path.step] : []))); } /** - * The step ids a step reads in its `if` and templates, and, when it runs an inline workflow, the ids that workflow's - * steps read outside it, at any depth. An `if` or template that does not parse reads nothing here; settling its own - * workflow reports it. + * The step ids a step reads in its expressions and templates, and, when it runs an inline workflow, the ids that + * workflow's steps read outside it, at any depth. A `loop.while` or `break` that reads its own step reads it after it + * ran, so the step is left out. An expression or template that does not parse reads nothing here; settling its own + * workflow reports it. `item` is whether the step's workflow runs per item, as pull decided which templates read it. */ -function readsOf(step: WorkflowStep): Set { - const parsed = step.if === undefined ? undefined : parseExpression(step.if); +function readsOf(step: WorkflowStep, item: boolean): Set { + const bindsItem = item || step.forEach !== undefined; const paths = [ - ...(parsed?.ok ? expressionPaths(parsed.expression) : []), + ...stepExpressions(step).flatMap(({ source }) => { + const parsed = parseExpression(source); + return parsed.ok ? expressionPaths(parsed.expression) : []; + }), ...templatedTexts(step).flatMap(({ text }) => { - const template = parseTemplate(text); + const template = parseTemplate(text, bindsItem); return template.ok ? templatePaths(template.template) : []; }), ]; const ids = stepsRead(paths); if (typeof step.workflow === 'object') { const own = new Set(step.workflow.steps.map((item) => item.id)); - for (const item of step.workflow.steps) for (const id of readsOf(item)) if (!own.has(id)) ids.add(id); + for (const inner of step.workflow.steps) for (const id of readsOf(inner, bindsItem)) if (!own.has(id)) ids.add(id); } + ids.delete(step.id); return ids; } interface RunReads { /** Each step's parsed `if`. */ conditions: Map; + /** Each loop owner's parsed `loop.while`. */ + whiles: Map; + /** Each step's parsed `break`. */ + breaks: Map; + /** Each `for_each` step's parsed list of items. */ + items: Map; /** The step ids each step's `if` and templates read. */ reads: Map>; + /** The loop owners each step waits for besides what it needs and reads. */ + waits: Map; /** The path each declared output reads. */ outputs: Map; + /** The workflow's own parsed `loop.while`. */ + loopWhile?: Expression; } /** Parses every `if` and template in a workflow of the run. A run file whose `if` does not parse cannot settle; the message names the step. */ function parseReads(run: RunState, scope: Scope): RunReads | string { const conditions = new Map(); + const whiles = new Map(); + const breaks = new Map(); + const items = new Map(); const reads = new Map>(); + const waits = new Map(); + const parsedInto: Record> = { if: conditions, 'loop.while': whiles, break: breaks, for_each: items }; const own = new Set(scope.definition.steps.map((step) => step.id)); for (const step of scope.definition.steps) { - if (step.if !== undefined) { - const parsed = parseExpression(step.if); - if (!parsed.ok) return `Run ${run.id} step "${scope.prefix}${step.id}" has an if that does not parse: ${parsed.message}`; - conditions.set(step.id, parsed.expression); + for (const { what, source } of stepExpressions(step)) { + const parsed = parseExpression(source); + if (!parsed.ok) return `Run ${run.id} step "${scope.prefix}${step.id}" has ${what === 'if' ? 'an' : 'a'} ${what} that does not parse: ${parsed.message}`; + parsedInto[what].set(step.id, parsed.expression); } // A template that does not parse reads nothing; the step fails when it would be leased. // An inline workflow's reads of enclosing steps settled before it started, so only its own steps are waited for. - reads.set(step.id, new Set([...readsOf(step)].filter((id) => !scope.outer || own.has(id)))); + // A read of a step's previous pass never waits. + const previous = scope.loops.previous(step.id); + const read = new Set([...readsOf(step, Object.hasOwn(scope, 'item'))].filter((id) => (!scope.outer || own.has(id)) && !previous.has(id))); + reads.set(step.id, read); + waits.set(step.id, scope.loops.waitsFor(step.id, read)); + } + let loopWhile: Expression | undefined; + if (scope.definition.loop?.while !== undefined) { + const parsed = parseExpression(scope.definition.loop.while); + if (!parsed.ok) return `Run ${run.id} ${workflowLabel(scope)} has a loop.while that does not parse: ${parsed.message}`; + loopWhile = parsed.expression; } const outputs = new Map(); for (const [name, source] of Object.entries(scope.definition.outputs ?? {})) { const parsed = parseExpression(source); - const workflow = 'name' in scope.definition ? `workflow "${scope.definition.name}"` : `step "${scope.prefix.slice(0, -1)}" inline workflow`; - if (!parsed.ok || parsed.expression.kind !== 'path') return `Run ${run.id} ${workflow} output "${name}" is not a path`; + if (!parsed.ok || parsed.expression.kind !== 'path') return `Run ${run.id} ${workflowLabel(scope)} output "${name}" is not a path`; outputs.set(name, parsed.expression.path); } - return { conditions, reads, outputs }; -} - -const VISIBLE_OUTCOMES: Partial> ={ succeeded: 'success', failed: 'failed', skipped: 'skipped' }; - -/** - * What expressions in a workflow read as `steps.`: its own step instances, by their id in that workflow, - * and, in an inline workflow, the enclosing steps its own steps do not hide. - */ -function expressionSteps(scope: Scope, states: StepState[]): Record { - return { ...scope.outer, ...Object.fromEntries(states.map((step) => [localId(scope, step), { - ...(VISIBLE_OUTCOMES[step.status] ? { outcome: VISIBLE_OUTCOMES[step.status] } : {}), - ...(step.end !== undefined ? { end: step.end } : {}), - ...(step.outputs !== undefined ? { outputs: step.outputs } : {}), - }])) }; + return { conditions, whiles, breaks, items, reads, waits, outputs, ...(loopWhile ? { loopWhile } : {}) }; } /** * Marks every pending step of a workflow that can no longer run as `skipped`, until nothing changes: - * the skip rule in `mustSkip`, and a ready step whose `if` is false. + * a stopped need its `if` does not read, all needs skipped, or a ready step whose `if` is false. A step that waits for a loop is neither skipped + * nor ready until that loop is settled. * Returns whether a step is ready: everything it needs and everything its `if` and templates read has settled. */ -function skipUnreachable(scope: Scope, states: StepState[], { conditions, reads }: RunReads): (step: StepState) => boolean { +function skipUnreachable(run: RunState, scope: Scope, states: StepState[], { conditions, reads, waits }: RunReads): (step: StepState) => boolean { const byId = new Map(states.map((step) => [localId(scope, step), step])); const declared = new Map(scope.definition.steps.map((step) => [step.id, step.needs ?? []])); const needs = (step: StepState): Array => (declared.get(localId(scope, step)) ?? []).map((id) => byId.get(id)); - const ready = (step: StepState): boolean => needs(step).every(settled) + const waiting = (step: StepState): boolean => (waits.get(localId(scope, step)) ?? []).some((owner) => !loopSettled(byId.get(owner))); + const ready = (step: StepState): boolean => !waiting(step) && needs(step).every(settled) && [...reads.get(localId(scope, step)) ?? []].every((id) => settled(byId.get(id))); + const conditionHolds = (step: StepState): boolean => { + const condition = conditions.get(localId(scope, step)); + return !condition || isTruthy(evaluateExpression(condition, contextOf(scope, states, step))); + }; for (let changed = true; changed;) { changed = false; for (const step of states) { - if (step.status !== 'pending') continue; + if (step.status !== 'pending' || waiting(step)) continue; const condition = conditions.get(localId(scope, step)); - const conditionFalse = (): boolean => !!condition && ready(step) - && !isTruthy(evaluateExpression(condition, { inputs: scope.inputs, steps: expressionSteps(scope, states) })); - if (!mustSkip(needs(step), condition, (need) => localId(scope, need)) && !conditionFalse()) continue; - step.status = 'skipped'; + const read = stepsRead(condition ? expressionPaths(condition) : []); + const stoppedNeed = needs(step).some((need) => need && stopped(need) && !read.has(localId(scope, need))); + const allSkipped = needs(step).length > 0 && needs(step).every((need) => need?.status === 'skipped'); + const conditionFalse = ready(step) && !conditionHolds(step); + if (!stoppedNeed && !allSkipped && !conditionFalse) continue; + skip(step, run.revision); changed = true; } } @@ -354,19 +352,13 @@ function skipUnreachable(scope: Scope, states: StepState[], { conditions, reads } /** - * A step is skipped when a step it needs stopped and its `if` does not read that step, - * or when every step it needs was skipped. An `if` that reads every stopped dependency is a recovery step. + * How settling a workflow of the run ended: a step to lease, the workflow's outcome, nothing to do until an open + * lease is reported, or a run file that cannot settle. */ -function mustSkip(needs: Array, condition: Expression | undefined, id: (need: StepState) => string): boolean { - const read = stepsRead(condition ? expressionPaths(condition) : []); - if (needs.some((need) => need && stopped(need) && !read.has(id(need)))) return true; - return needs.length > 0 && needs.every((need) => need?.status === 'skipped'); -} - -/** How settling a workflow of the run ended: a step to lease, the workflow's outcome, or a run file that cannot settle. */ type Settled = | { lease: StepState; definition: WorkflowStep; rendered: RenderedLease } | WorkflowResult + | { inProgress: true } | { unreadable: string }; /** @@ -375,41 +367,181 @@ type Settled = */ function startWorkflowStep(run: RunState, scope: Scope, states: StepState[], state: StepState, step: WorkflowStep, child: Scope): WorkflowFailure | undefined { // An inline workflow takes no `with`: it already reads the enclosing inputs. + let inputs: Record | undefined; if ('name' in child.definition) { - const rendered = renderStrings(step.with ?? {}, { inputs: scope.inputs, steps: expressionSteps(scope, states) }); + const rendered = renderStrings(step.with ?? {}, contextOf(scope, states, state)); if (!rendered.ok) return withTemplateFailure(state, rendered.error); const mismatch = inputsMismatch(child.definition.name, child.definition.inputs, rendered.value); if (mismatch) return { code: 'invalid_inputs', message: `step "${state.id}" with does not fit workflow "${child.definition.name}": ${mismatch}` }; if (child.definition.inputs) { - state.inputs = rendered.value as Record; - child.inputs = state.inputs; + inputs = rendered.value as Record; + child.inputs = inputs; + } + } + start(state, inputs); + const steps = child.definition.steps.map((step): StepState => { + const id = `${child.prefix}${step.id}`; + const pass = firstPass(run, id); + return { id, status: 'pending', ...(pass > 1 ? { pass } : {}) }; + }); + insertChildren(run, state, steps); + return undefined; +} + +/** + * An end or break reached in a workflow stops everything under it that has not settled, in its sub-workflows too: a + * leased step is cancelled and its attempt kept in `cancelledAttempts`, a running workflow step is cancelled, and a + * pending step is skipped. `reached` names the end or break. + */ +function stopUnder(run: RunState, scope: Scope, reached: string, settledAt: number): void { + const failure = { code: 'end_reached', message: `Cancelled: ${workflowLabel(scope)} reached ${reached}` }; + for (const state of run.steps) { + if (!state.id.startsWith(scope.prefix)) continue; + switch (state.status) { + case 'pending': + skip(state, settledAt); + break; + case 'running': + cancel(state, failure, settledAt); + break; + case 'leased': + (run.cancelledAttempts ??= []).push(state.attempt); + cancel(state, failure, settledAt); + break; + case 'succeeded': + case 'failed': + case 'skipped': + case 'cancelled': + break; + default: { + const unknown: never = state; + return unknown; + } + } + } +} + +/** + * A step takes how what it ran settled: a workflow step, the outcome, the end, and the outputs of its workflow; a + * `for_each` step, the outcome and the results of its items. + */ +function finishStep(state: StepState, result: WorkflowResult): void { + const settlement = { + settledAt: result.settledAt, + ...(result.end !== undefined ? { end: result.end } : {}), + ...(result.outputs !== undefined ? { outputs: result.outputs } : {}), + ...(result.results ? { results: result.results } : {}), + }; + if (result.outcome === 'succeeded') succeed(state, settlement); + else fail(state, { ...settlement, ...(result.failure ? { message: result.failure.message, failure: result.failure } : {}) }); +} + +function childOrUnreadable(run: RunState, scope: Scope, states: StepState[], state: StepState, definition: WorkflowStep): Scope | { unreadable: string } { + return childScope(run, scope, states, state, definition) + ?? { unreadable: `Run ${run.id} step "${state.id}" runs workflow "${definition.workflow}", which the run did not keep` }; +} + +function settleRunning(run: RunState, scope: Scope, states: StepState[], state: StepState, definition: WorkflowStep): Settled { + if (definition.forEach && !isItem(state)) return settleItems(run, scope, states, state, definition); + const child = childOrUnreadable(run, scope, states, state, definition); + return 'unreadable' in child ? child : settleScope(run, child); +} + +/** + * Advances one ready instance. A running one settles what it runs. A pending leaf is returned to lease, a pending + * workflow step starts its workflow, and a pending `for_each` step adds an instance per item of `items`. What the + * engine cannot hand out or start fails at the current revision. Returns undefined when it changed the run. + */ +function advanceInstance( + run: RunState, scope: Scope, states: StepState[], state: StepState, definition: WorkflowStep, items?: Expression, +): Settled | undefined { + if (state.status === 'running') { + const result = settleRunning(run, scope, states, state, definition); + if (!('outcome' in result)) return result; + finishStep(state, result); + return undefined; + } + if (definition.forEach && !isItem(state)) { + const values = expandItems(state.id, items && evaluateExpression(items, contextOf(scope, states, state))); + if (!Array.isArray(values)) { + failStep(state, values, run.revision); + return undefined; } + start(state); + addItems(run, state, values); + return undefined; } - state.status = 'running'; - const steps = child.definition.steps.map((step): StepState => ({ id: `${child.prefix}${step.id}`, status: 'pending' })); - run.steps.splice(run.steps.indexOf(state) + 1, 0, ...steps); + if (definition.workflow === undefined) { + const rendered = renderLease(scope, states, state, definition); + if (!('code' in rendered)) return { lease: state, definition, rendered }; + failStep(state, rendered, run.revision); + return undefined; + } + const child = childOrUnreadable(run, scope, states, state, definition); + if ('unreadable' in child) return child; + const failure = startWorkflowStep(run, scope, states, state, definition, child); + if (failure) failStep(state, failure, run.revision); return undefined; } -/** A workflow step takes the outcome, the end, and the outputs of the workflow it ran. */ -function finishWorkflowStep(state: StepState, result: WorkflowResult): void { - state.status = result.outcome; - state.outcome = result.outcome === 'succeeded' ? 'success' : 'failed'; - if (result.end !== undefined) state.end = result.end; - if (result.outputs !== undefined) state.outputs = result.outputs; +/** + * Settles items at the current revision and returns the first leaf available to lease. An item waits for its + * `after` dependencies and for room under `max_parallel`. + */ +function settleItems( + run: RunState, scope: Scope, states: StepState[], state: StepState, definition: WorkflowStep, +): Settled { + for (;;) { + const items = itemsOf(run, state); + const after = itemsAfter(state.id, items.map((item) => item.item)); + if (!Array.isArray(after)) return { unreadable: `Run ${run.id} ${after.message}` }; + const dependencies = (index: number): StepState[] => after[index]?.map((other) => items[other]!) ?? []; + const standing = (index: number): ReturnType => afterStanding(dependencies(index)); + const dependents: number[][] = items.map(() => []); + after.forEach((others, index) => others.forEach((other) => dependents[other]!.push(index))); + const maxParallel = definition.forEach?.maxParallel; + let started = startedItems(items); + let changed = false; + let candidate: Settled | undefined; + for (const [index, item] of items.entries()) { + if (item.status === 'pending' && standing(index) === 'skip') { + // A skip makes the items after it skippable wherever they are in item order, so it reaches them all now. + const skippable = [index]; + for (const next of skippable) { + if (items[next]!.status !== 'pending' || standing(next) !== 'skip') continue; + skip(items[next]!, run.revision); + skippable.push(...dependents[next]!); + } + changed = true; + continue; + } + if (item.status !== 'running' && (item.status !== 'pending' || standing(index) !== 'ready' || (maxParallel !== undefined && started >= maxParallel))) continue; + const advanced = advanceInstance(run, scope, states, item, definition); + if (advanced && 'unreadable' in advanced) return advanced; + if (advanced && 'lease' in advanced) candidate ??= advanced; + if (advanced) continue; + changed = true; + // Starting an item changes no other item's standing, so the scan goes on. Settling one can make an earlier item + // ready, so the scan starts over. + if (item.status !== 'running') break; + started += 1; + } + if (changed) continue; + if (items.every(settled)) return itemsSettled(items, run.revision); + return candidate ?? { inProgress: true }; + } } /** The values a workflow returns, read from its own steps; a skipped step reads null. */ function workflowOutputs(scope: Scope, states: StepState[], paths: Map): Record | undefined { if (!scope.definition.outputs) return undefined; - const context = { inputs: scope.inputs, steps: expressionSteps(scope, states) }; + const context = contextOf(scope, states); return Object.fromEntries([...paths].map(([name, path]) => [name, resolvePath(path, context)])); } /** - * Settles one workflow of the run while no lease is open: skips, end-only steps, and the workflows its steps run. - * Returns the first step to lease in declaration order, where a running workflow step stands for its own steps, - * or the workflow's outcome once nothing is left to run. + * Settles driverless transitions before applying an end or deciding a loop. A lease candidate is returned only + * after this fixpoint, before another report can advance the revision. */ function settleScope(run: RunState, scope: Scope): Settled { const reads = parseReads(run, scope); @@ -417,49 +549,74 @@ function settleScope(run: RunState, scope: Scope): Settled { const states = statesIn(run, scope); const defined = new Map(scope.definition.steps.map((item) => [item.id, item])); const definitionOf = (state: StepState): WorkflowStep | undefined => defined.get(localId(scope, state)); - // A step that cannot render, or a workflow step that finishes, changes the graph, and settling starts again. for (;;) { - const canRun = skipUnreachable(scope, states, reads); - const ready = (item: StepState): boolean => item.status === 'pending' && canRun(item); - const ending = states.find((item) => ready(item) && endsOnly(definitionOf(item))); - if (ending) ending.status = 'succeeded'; - const reached = states.find((item) => item.status === 'succeeded' && definitionOf(item)?.end !== undefined); - const end = reached ? definitionOf(reached)?.end : undefined; - if (end !== undefined) { - for (const item of states) if (item.status === 'pending') item.status = 'skipped'; - return { outcome: scope.definition.ends?.[end] ?? 'failed', end, outputs: workflowOutputs(scope, states, reads.outputs) }; - } - const state = states.find((item) => item.status === 'running' || ready(item)); - if (!state) { - if (states.some((item) => item.status === 'pending')) { - return { unreadable: `Run ${run.id} has pending steps whose needs cannot be met` }; + let changed = false; + let candidate: Settled | undefined; + const canRun = skipUnreachable(run, scope, states, reads); + for (const state of states) { + if (state.status !== 'running' && (state.status !== 'pending' || !canRun(state))) continue; + const definition = definitionOf(state); + if (!definition) return { unreadable: `Run ${run.id} has no definition for step "${state.id}"` }; + if (state.status === 'pending' && endsOnly(definition)) { + succeed(state, { settledAt: run.revision }); + } else if (!(state.status === 'pending' && definition.loop && exhaustedAtStart(state, definition.loop, run.revision))) { + const advanced = advanceInstance(run, scope, states, state, definition, reads.items.get(localId(scope, state))); + if (advanced && 'unreadable' in advanced) return advanced; + if (advanced && 'lease' in advanced) candidate ??= advanced; + if (advanced) continue; } - return { outcome: states.some(stopped) ? 'failed' : 'succeeded', outputs: workflowOutputs(scope, states, reads.outputs) }; + changed = true; + break; } - const definition = definitionOf(state); - if (!definition) return { unreadable: `Run ${run.id} has no definition for step "${state.id}"` }; - if (definition.workflow !== undefined) { - const child = childScope(run, scope, states, state, definition); - if (!child) return { unreadable: `Run ${run.id} step "${state.id}" runs workflow "${definition.workflow}", which the run did not keep` }; - const failure = state.status === 'pending' ? startWorkflowStep(run, scope, states, state, definition, child) : undefined; - if (failure) { - failStep(state, failure); - continue; - } - const result = settleScope(run, child); - if (!('outcome' in result)) return result; - finishWorkflowStep(state, result); - continue; + if (changed) continue; + const breaks = (state: StepState): boolean => { + const condition = reads.breaks.get(localId(scope, state)); + return !!condition && isTruthy(evaluateExpression(condition, contextOf(scope, states, state))); + }; + const reached = states + .filter((item) => item.status === 'succeeded' && (definitionOf(item)?.end !== undefined || breaks(item))) + .sort((left, right) => settledAtOf(left) - settledAtOf(right))[0]; + if (reached) { + const end = definitionOf(reached)?.end; + stopUnder(run, scope, end === undefined ? `the break of step "${reached.id}"` : `end "${end}"`, settledAtOf(reached)); + const result = { outputs: workflowOutputs(scope, states, reads.outputs), settledAt: settledAtOf(reached) }; + return end === undefined ? { outcome: 'succeeded', ...result } : { outcome: scope.definition.ends?.[end] ?? 'failed', end, ...result }; } - const rendered = renderLease(scope, states, state, definition); - if ('code' in rendered) { - failStep(state, rendered); - continue; + for (const state of states) { + const loop = definitionOf(state)?.loop; + if (!loop || state.status !== 'succeeded' || state.loopFinal) continue; + const condition = reads.whiles.get(localId(scope, state)); + const holds = !condition || isTruthy(evaluateExpression(condition, contextOf(scope, states, state))); + const range = scope.loops.ranges.get(localId(scope, state)) ?? new Set(); + decideStepLoop(run, state, loop, holds, states.filter((item) => range.has(localId(scope, item)))); + changed = true; + break; + } + if (changed) continue; + if (states.every(settled)) { + const result: WorkflowResult = { + outcome: states.some(stopped) ? 'failed' : 'succeeded', + outputs: workflowOutputs(scope, states, reads.outputs), + settledAt: run.revision, + }; + const { loop } = scope.definition; + if (!loop) return result; + const holds = !reads.loopWhile || isTruthy(evaluateExpression(reads.loopWhile, contextOf(scope, states))); + const decided = decideRound(run, scope.runner, loop, holds, workflowLabel(scope), states); + if (decided === 'again') continue; + return decided === 'ended' ? result : { ...result, outcome: 'failed', failure: decided }; } - return { lease: state, definition, rendered }; + if (candidate) return candidate; + if (states.some((item) => item.status === 'leased' || item.status === 'running')) return { inProgress: true }; + return { unreadable: `Run ${run.id} has pending steps whose needs cannot be met` }; } } +/** Every stored report is reconciled before a later protocol operation can apply another transition. */ +function reconcileRun(run: RunState): Settled { + return settleScope(run, topScope(run)); +} + /** Why `inputs` do not fit what the workflow declares, naming the field, or null when they fit. Every declared input is required. */ function inputsMismatch(name: string, declared: Record | undefined, inputs: unknown): string | null { if (!isRecord(inputs)) return 'inputs must be a JSON object'; @@ -539,7 +696,7 @@ export async function startRun( await writeRun(config, run); // #1083: publish the start record; a failed publication is queued locally // and retried on the next pull — it never fails the run itself. - const record = await noteRunRecord(config, run, options.driver, 'start'); + const record = await queueRunRecord(config, run, options.driver, 'start') ? await publishRunRecords(config) : undefined; return ok({ run: run.id, name: run.name, status: run.status, revision: run.revision, ...(record ? { record } : {}) }); } @@ -552,54 +709,78 @@ function missingNeeds(run: RunState): string | undefined { return undefined; } -export async function leaseNext(config: LocalConfig, runId: string, driver: DriverIdentity): Promise { - // The end record publishes AFTER withRun releases the run lock (#1126 P3): - // publication does a reports-branch push, and no network I/O may hold the - // lock other engine operations wait on. - let settledRun: RunState | undefined; - let alreadySettled = false; +/** What a driver tells `next` besides who it is. */ +export interface NextOptions { + /** The attempts of its open leases the driver is still running. An open lease it does not list is handed back to it. */ + running?: string[]; + /** False to reconcile without issuing a new lease: `next` answers `done`, an unlisted lease, or `wait`. */ + canTakeLease?: boolean; +} + +export async function leaseNext(config: LocalConfig, runId: string, driver: DriverIdentity, options: NextOptions = {}): Promise { + const running = new Set(options.running); + // The end record is queued under the run lock, so the run stores whether it + // was in the same save. It publishes AFTER withRun releases the lock (#1126 + // P3): publication does a reports-branch push, and no network I/O may hold + // the lock other engine operations wait on. + let queued = false; + let alreadySettled: RunState | undefined; const envelope = await withRun(config, runId, async (run, now) => { + const before = JSON.stringify(run); + // Whatever settling changed is saved. `cancelled` is derived after settling: the attempts the driver runs that + // are no longer its open leases. + const answer = (data: Record): { save: boolean; envelope: WorkflowEnvelope } => ({ + save: JSON.stringify(run) !== before, + envelope: ok({ + ...data, + cancelled: [...running].filter((attempt) => !run.steps.some((step) => step.status === 'leased' + && step.lease.attempt === attempt && sameDriver(step.lease.driver, driver))), + }), + }); + const done = async (result: WorkflowResult): Promise<{ save: boolean; envelope: WorkflowEnvelope }> => { + settleRun(run, now, result); + queued = await queueRunRecord(config, run, driver, 'end'); + run.endRecordQueued = queued; + return answer(doneData(run)); + }; + const wait = (open = run.steps.find((step) => step.status === 'leased')): { save: boolean; envelope: WorkflowEnvelope } => ( + answer({ wait: { reason: open ? `${open.id} is leased` : 'canTakeLease is false' } }) + ); if (run.status !== 'running') { - // Settled by an earlier next, which published the end record; the - // done envelope still reports the queue-derived record state. - alreadySettled = true; - return { save: false, envelope: doneEnvelope(run) }; + // Settled by an earlier next, which queued the end record unless usage + // reporting was off; the done envelope reports that record's state. + alreadySettled = run; + return answer(doneData(run)); } const older = missingNeeds(run); if (older) return { save: false, envelope: error('run_unreadable', older) }; - const leased = run.steps.find((step) => step.status === 'leased'); - if (leased?.lease) { - if (!sameDriver(leased.lease.driver, driver)) { - return { save: false, envelope: ok({ wait: { reason: `${leased.id} is leased` } }) }; + const other = run.steps.find((step): step is LeasedStep => step.status === 'leased' && !sameDriver(step.lease.driver, driver)); + if (other) return wait(other); + for (;;) { + // Ends reached through reports apply before an open lease is handed back, so a restarted driver does not run + // again what an end already stopped. + const result = reconcileRun(run); + if ('unreadable' in result) return { save: false, envelope: error('run_unreadable', result.unreadable) }; + if ('outcome' in result) return done(result); + // An open lease the driver does not list is handed back before another lease is issued, as a restarted driver expects. + const leased = run.steps.find((step): step is LeasedStep => step.status === 'leased' && !running.has(step.lease.attempt)); + if (!leased) { + if (options.canTakeLease === false || 'inProgress' in result) return wait(); + const step = lease(result.lease, { attempt: `a${randomBytes(4).toString('hex')}`, revision: run.revision, driver }); + return answer(leaseData(run, step, result.definition, result.rendered)); } const located = locateStep(run, leased.id); if (!located) return { save: false, envelope: error('run_unreadable', `Run ${run.id} has no definition for step "${leased.id}"`) }; const rendered = renderLease(located.scope, statesIn(run, located.scope), leased, located.step); - if (!('code' in rendered)) return { save: false, envelope: leaseEnvelope(run, leased, located.step, rendered) }; - failStep(leased, rendered); - } - // Settle the graph here, while no lease is open, so a later report still matches its lease revision. - const result = settleScope(run, topScope(run)); - if ('unreadable' in result) return { save: false, envelope: error('run_unreadable', result.unreadable) }; - if ('outcome' in result) { - settledRun = settleRun(run, now, result.outcome, result.end); - return { save: true, envelope: doneEnvelope(run) }; + if (!('code' in rendered)) return answer(leaseData(run, leased, located.step, rendered)); + failStep(leased, rendered, leased.lease.revision); } - const step = result.lease; - const attempt = `a${randomBytes(4).toString('hex')}`; - step.status = 'leased'; - step.attempt = attempt; - step.outcome = undefined; - step.lease = { attempt, revision: run.revision, driver }; - return { save: true, envelope: leaseEnvelope(run, step, result.definition, result.rendered) }; }); - if (settledRun || alreadySettled) { - const record = settledRun - ? await noteRunRecord(config, settledRun, driver, 'end') - : await runRecordState(config, runId); - const done = (envelope as { data?: { done?: Record } }).data?.done; - if (done && record) done.record = record; - } + const record = queued + ? await publishRunRecords(config) + : alreadySettled && await runRecordState(config, alreadySettled); + const done = (envelope as { data?: { done?: Record } }).data?.done; + if (done && record) done.record = record; return envelope; } @@ -658,10 +839,28 @@ export function parseReport(value: unknown): ReportInput | string { return report; } -export function parseDriverPayload(value: unknown): DriverIdentity | string { +/** A `next` payload: who the driver is, and what else it tells `next`. */ +export interface NextRequest extends NextOptions { + driver: DriverIdentity; +} + +export function parseNextPayload(value: unknown): NextRequest | string { if (!value || typeof value !== 'object' || Array.isArray(value)) return 'next payload must be a JSON object'; - const driver = driverFrom((value as { driver?: unknown }).driver); - return driver ?? 'driver must include kind, machine, and workspace'; + const record = value as Record; + const driver = driverFrom(record.driver); + if (!driver) return 'driver must include kind, machine, and workspace'; + const request: NextRequest = { driver }; + if (record.running !== undefined) { + if (!Array.isArray(record.running) || !record.running.every((attempt): attempt is string => typeof attempt === 'string')) { + return 'running must be an array of attempt ids'; + } + request.running = record.running; + } + if (record.canTakeLease !== undefined) { + if (typeof record.canTakeLease !== 'boolean') return 'canTakeLease must be a boolean'; + request.canTakeLease = record.canTakeLease; + } + return request; } function rememberSettledAttempt(run: RunState, attempt: string): void { @@ -680,13 +879,27 @@ export async function reportAttempt(config: LocalConfig, runId: string, report: if (run.settledAttempts.includes(report.attempt)) { return { save: false, envelope: error('repeated_report', `Attempt ${report.attempt} was already reported`) }; } - if (report.revision !== run.revision) { - return { save: false, envelope: error('stale_revision', `Revision ${report.revision} is stale; the run is at ${run.revision}`) }; + // Before the driver check, so any driver learns that the attempt was cancelled. + if (run.cancelledAttempts?.includes(report.attempt)) { + return { save: false, envelope: error('lease_cancelled', `Attempt ${report.attempt} was cancelled; its report is not recorded`) }; } - const step = run.steps.find((item) => item.status === 'leased' && item.lease?.attempt === report.attempt); - if (!step?.lease) { + const step = run.steps.find((item): item is LeasedStep => item.status === 'leased' && item.lease.attempt === report.attempt); + if (!step) { + if (report.revision !== run.revision) { + return { save: false, envelope: error('stale_revision', `Revision ${report.revision} is stale; the run is at ${run.revision}`) }; + } return { save: false, envelope: error('wrong_attempt', `Attempt ${report.attempt} is not the leased attempt`) }; } + const reconciled = reconcileRun(run); + if ('unreadable' in reconciled) return { save: false, envelope: error('run_unreadable', reconciled.unreadable) }; + if (run.cancelledAttempts?.includes(report.attempt)) { + return { save: true, envelope: error('lease_cancelled', `Attempt ${report.attempt} was cancelled; its report is not recorded`) }; + } + + // A lease is checked against the revision it was handed out at, so a report of a parallel lease does not make it stale. + if (report.revision !== step.lease.revision) { + return { save: false, envelope: error('stale_revision', `Revision ${report.revision} is stale; attempt ${report.attempt} was leased at ${step.lease.revision}`) }; + } if (!sameDriver(step.lease.driver, report.driver)) { return { save: false, envelope: error('wrong_driver', 'This lease belongs to a different driver') }; } @@ -704,32 +917,27 @@ export async function reportAttempt(config: LocalConfig, runId: string, report: } } - step.lease = undefined; - step.attempt = undefined; - step.exitCode = report.exitCode; - step.message = message; - step.tool = report.tool; - step.model = report.model; - step.session = report.session; - step.tokens = report.tokens; - step.failure = failure; - step.outcome = outcome; - if (outcome === 'released') { - step.status = 'pending'; - } else if (outcome === 'success') { - step.status = 'succeeded'; - if (report.outputs !== undefined) step.outputs = report.outputs; - } else { - step.status = 'failed'; - } rememberSettledAttempt(run, report.attempt); run.revision += 1; + const settlement = { + settledAt: run.revision, + attempt: report.attempt, + exitCode: report.exitCode, + message, + tool: report.tool, + model: report.model, + session: report.session, + tokens: report.tokens, + }; + const settledStep = outcome === 'released' ? release(step) + : outcome === 'success' ? succeed(step, { ...settlement, ...(report.outputs !== undefined ? { outputs: report.outputs } : {}) }) + : fail(step, { ...settlement, failure }); return { save: true, envelope: ok({ accepted: true, revision: run.revision, - step: { id: step.id, status: step.status, outcome }, + step: { id: settledStep.id, status: settledStep.status, outcome }, }), }; }); diff --git a/src/workflow/expression.ts b/src/workflow/expression.ts index 705dfc139..0bf7a4513 100644 --- a/src/workflow/expression.ts +++ b/src/workflow/expression.ts @@ -1,32 +1,78 @@ /** - * The workflow expression language: `if` conditions, and the paths that templates read. - * It reads only `inputs` and `steps`, has no functions, and its evaluation never throws. + * The workflow expression language: `if` conditions, and what templates read and evaluate. + * It reads `inputs`, `steps` and the `item` of a `for_each`, compares numbers, and calls `length`, `any`, `all` and + * `filter`, whose predicate reads each element of the list as `.`. Its evaluation never throws. */ +import type { VisibleOutcome } from './step-state.js'; +import { INDEX } from './step-path.js'; +import type { OutputSchema } from './types.js'; + +/** A field name, or an array index such as the 0 in `tags[0]`. */ +export type PathField = string | number; /** A value read by a path. A missing value is `null`. */ export type ExpressionPath = - | { root: 'inputs'; fields: string[]; column: number } - | { root: 'steps'; step: string; field: 'outcome' | 'end' | 'outputs'; fields: string[]; column: number }; + | { root: 'inputs'; fields: PathField[]; column: number } + /** The item a `for_each` step instance runs for. */ + | { root: 'item'; fields: PathField[]; column: number } + | { root: 'steps'; step: string; field: 'outcome' | 'end' | 'outputs' | 'passes' | 'results'; fields: PathField[]; column: number }; + +/** `<`, `<=`, `>` and `>=` compare two numbers; with anything else they are false. */ +const COMPARISONS = { + '<': (left: number, right: number) => left < right, + '<=': (left: number, right: number) => left <= right, + '>': (left: number, right: number) => left > right, + '>=': (left: number, right: number) => left >= right, +}; +type Comparison = keyof typeof COMPARISONS; + +/** Each function, as a call to it is written. */ +const FUNCTIONS = { + length: 'length(list)', + any: 'any(list, predicate)', + all: 'all(list, predicate)', + filter: 'filter(list, predicate)', +}; +type FunctionName = keyof typeof FUNCTIONS; export type Expression = | { kind: 'literal'; value: null | boolean | number | string | []; column: number } | { kind: 'path'; path: ExpressionPath; column: number } | { kind: 'not'; operand: Expression; column: number } - | { kind: 'binary'; operator: '==' | '!=' | '&&' | '||'; left: Expression; right: Expression; column: number }; - -/** The outcome an expression reads of a settled step. */ -export type VisibleOutcome = 'success' | 'failed' | 'skipped'; + | { kind: 'binary'; operator: '==' | '!=' | Comparison | '&&' | '||'; left: Expression; right: Expression; column: number } + | { kind: 'call'; name: 'length'; list: Expression; column: number } + /** `predicate` reads each element of `list` through `element` expressions. */ + | { kind: 'call'; name: Exclude; list: Expression; predicate: Expression; column: number } + /** `.`, `.a.b` or `.a[0]`: the current element of a predicate, or a value inside it. */ + | { kind: 'element'; fields: PathField[]; column: number }; /** A step as an expression sees it. `outcome` is set once the step settled. */ export interface ExpressionStep { outcome?: VisibleOutcome; end?: string; outputs?: unknown; + /** Every settled pass of the step, earliest first: those a loop ran again, then the current one once it settled. */ + passes?: ExpressionPass[]; + /** For a settled `for_each` step: how each item settled, in item order. */ + results?: Array<{ item?: unknown; outcome: VisibleOutcome; outputs?: unknown }>; +} + +/** One pass of a step in `steps..passes`. */ +export interface ExpressionPass { + pass: number; + outcome: VisibleOutcome; + outputs?: unknown; + end?: string; + results?: ExpressionStep['results']; } export interface ExpressionContext { inputs?: unknown; steps: Record; + /** The item of the `for_each` the expression is evaluated for. */ + item?: unknown; + /** The element a predicate is evaluated for. */ + element?: unknown; } export type ExpressionParse = @@ -38,6 +84,7 @@ type Token = | { kind: 'number'; value: number; column: number } | { kind: 'string'; value: string; column: number } | { kind: 'symbol'; text: string; column: number } + | { kind: 'element'; text: string; column: number } | { kind: 'end'; column: number }; class ExpressionError extends Error { @@ -46,9 +93,16 @@ class ExpressionError extends Error { } } -const WORD = /[A-Za-z_][A-Za-z0-9_-]*(?:\.[A-Za-z0-9_-]+)*/y; +const WORD = /[A-Za-z_][A-Za-z0-9_-]*(?:\.[A-Za-z0-9_-]+|\[[^\]]*\]?)*/y; +/** One name or `[index]` of a path, in order. */ +const FIELD = /\.?([A-Za-z0-9_-]+)|\[([^\]]*)(\]?)/y; +const WHOLE_INDEX = new RegExp(`^(?:${INDEX})$`); +/** `.`, then any names and indexes, as in `.a.b[0]` or `.[0]`. */ +const ELEMENT = /\.(?:(?:[A-Za-z0-9_-]+|\[[^\]]*\]?)(?:\.[A-Za-z0-9_-]+|\[[^\]]*\]?)*)?/y; +/** A word with no "." or "[", which a "(" makes a function name. */ +const NAME = /^[A-Za-z_][A-Za-z0-9_-]*$/; const NUMBER = /-?[0-9]+(?:\.[0-9]+)?/y; -const SYMBOLS = ['==', '!=', '&&', '||', '!', '(', ')', '[', ']']; +const SYMBOLS = ['==', '!=', '<=', '>=', '&&', '||', '!', '<', '>', '(', ')', '[', ']', ',']; /** Reads the token that starts at or after `index`. Lexing is lazy, so errors come in source order. */ function lexAt(source: string, from: number): { token: Token; next: number } { @@ -80,38 +134,118 @@ function lexAt(source: string, from: number): { token: Token; next: number } { WORD.lastIndex = index; const word = WORD.exec(source); if (word) return { token: { kind: 'word', text: word[0], column }, next: index + word[0].length }; + ELEMENT.lastIndex = index; + const element = ELEMENT.exec(source); + if (element) return { token: { kind: 'element', text: element[0], column }, next: index + element[0].length }; const symbol = SYMBOLS.find((candidate) => source.startsWith(candidate, index)) ?? char; return { token: { kind: 'symbol', text: symbol, column }, next: index + symbol.length }; } +/** A string literal as the expression writes it, with each ' doubled. */ +function quoted(value: string): string { + return `'${value.replaceAll("'", "''")}'`; +} + function describe(token: Token): string { switch (token.kind) { case 'end': return 'the end of the expression'; - case 'word': return `"${token.text}"`; + case 'word': + case 'element': + return `"${token.text}"`; case 'number': return String(token.value); - case 'string': return `'${token.value}'`; + case 'string': return quoted(token.value); case 'symbol': return `"${token.text}"`; } } +/** The names and indexes of a path written at `column`. The WORD token rule has already checked its shape. */ +function fieldsOf(text: string, column: number): PathField[] { + const fields: PathField[] = []; + for (FIELD.lastIndex = 0; FIELD.lastIndex < text.length;) { + const at = column + FIELD.lastIndex; + const match = FIELD.exec(text); + if (!match) break; + if (match[1] !== undefined) { + fields.push(match[1]); + } else if (match[3] === '') { + throw new ExpressionError(at, 'an index is not closed with ]'); + } else if (WHOLE_INDEX.test(match[2] ?? '')) { + fields.push(Number(match[2])); + } else { + throw new ExpressionError(at, `"${match[0]}" is not an index; write a whole number from 0, such as [0]`); + } + } + return fields; +} + +/** The names and indexes of a path, root included, such as a template's `with.refs[1]`, or why an index is malformed. */ +export function parseFields(text: string): { ok: true; fields: PathField[] } | { ok: false; message: string } { + try { + return { ok: true, fields: fieldsOf(text, 1) }; + } catch (error) { + if (error instanceof ExpressionError) return { ok: false, message: error.message }; + throw error; + } +} + function parsePath(token: { text: string; column: number }): ExpressionPath { - const [root, ...segments] = token.text.split('.'); - if (root === 'inputs') return { root, fields: segments, column: token.column }; + const [root, ...segments] = fieldsOf(token.text, token.column); + if (root === 'inputs' || root === 'item') return { root, fields: segments, column: token.column }; if (root !== 'steps') { - throw new ExpressionError(token.column, `"${root}" is not readable; an expression reads only inputs and steps`); + throw new ExpressionError(token.column, `"${root}" is not readable; an expression reads only inputs, steps and item`); } const [step, field, ...fields] = segments; - if (!step || !field) { - throw new ExpressionError(token.column, `"${token.text}" must read steps..outcome, steps..end or steps..outputs`); + if (typeof step === 'string' && typeof field === 'number') { + throw new ExpressionError(token.column, `"${token.text}" indexes a step; an index reads an array in its outputs, as in steps.${step}.outputs.[${field}]`); } - if (field === 'outputs') return { root, step, field, fields, column: token.column }; + if (typeof step !== 'string' || !field) throw new ExpressionError(token.column, `"${token.text}" ${STEP_FIELDS}`); + if (field === 'outputs' || field === 'passes' || field === 'results') return { root, step, field, fields, column: token.column }; if ((field === 'outcome' || field === 'end') && fields.length === 0) return { root, step, field, fields, column: token.column }; - throw new ExpressionError(token.column, `"${token.text}" must read steps..outcome, steps..end or steps..outputs`); + if ((field === 'outcome' || field === 'end') && typeof fields[0] === 'number') { + throw new ExpressionError(token.column, `"${token.text}" indexes ${field}, which is a name; compare it whole, as in steps.${step}.${field} == ''`); + } + throw new ExpressionError(token.column, `"${token.text}" ${STEP_FIELDS}`); +} + +const STEP_FIELDS = 'must read steps..outcome, steps..end, steps..outputs, steps..passes or steps..results'; + +/** + * A type pull knows for a value before the run: a declared shorthand, or `unknown`, which any value fits, such as the + * item of a `for_each` result. Every field and index inside `unknown` reads `unknown` too. + */ +export type DeclaredType = + | OutputSchema + | { type: 'unknown' } + | { type: 'array'; items: DeclaredType } + | { type: 'object'; fields: Record }; + +const UNKNOWN: DeclaredType = { type: 'unknown' }; + +/** + * The type the `output` a step declares gives `steps..`, or undefined when it gives none: outcome and end + * are names, each item of `passes` holds the outputs of one pass, and each item of `results` the item and outputs of one + * `for_each` item. + */ +export function stepFieldSchema(field: Extract['field'], output: OutputSchema | undefined, forEach: boolean): DeclaredType | undefined { + const results: DeclaredType = { type: 'array', items: { type: 'object', fields: { item: UNKNOWN, outcome: { type: 'string' }, outputs: output ?? UNKNOWN } } }; + switch (field) { + case 'outcome': + case 'end': + return { type: 'string' }; + case 'outputs': return forEach ? undefined : output; + case 'passes': + return { type: 'array', items: { type: 'object', fields: { pass: { type: 'number' }, outcome: { type: 'string' }, end: { type: 'string' }, + ...(forEach ? { results } : { outputs: output ?? UNKNOWN }), + } } }; + case 'results': return forEach ? results : undefined; + } } class Parser { private lookahead: { token: Token; next: number } | undefined; private index = 0; + private depth = 0; + private predicateDepth = 0; constructor(private readonly source: string) {} @@ -140,33 +274,73 @@ class Parser { return this.take(); } + /** One level deeper in the tree. Text pasted in a prompt could otherwise nest or chain deep enough to overflow the stack. */ + private deeper(column: number): void { + if (this.depth === 100) throw new ExpressionError(column, 'an expression nests at most 100 levels deep, counting each "(", "!", call, && and ||'); + this.depth += 1; + } + private or(): Expression { + const depth = this.depth; let left = this.and(); for (let operator = this.symbol('||'); operator; operator = this.symbol('||')) { + this.deeper(operator.column); left = { kind: 'binary', operator: '||', left, right: this.and(), column: operator.column }; } + this.depth = depth; return left; } private and(): Expression { + const depth = this.depth; let left = this.not(); for (let operator = this.symbol('&&'); operator; operator = this.symbol('&&')) { + this.deeper(operator.column); left = { kind: 'binary', operator: '&&', left, right: this.not(), column: operator.column }; } + this.depth = depth; return left; } private not(): Expression { + const depth = this.depth; + this.deeper(this.peek().column); const operator = this.symbol('!'); - if (operator) return { kind: 'not', operand: this.not(), column: operator.column }; - return this.compare(); + const expression: Expression = operator ? { kind: 'not', operand: this.not(), column: operator.column } : this.equality(); + this.depth = depth; + return expression; } - private compare(): Expression { - const left = this.primary(); + private equality(): Expression { + const left = this.comparison(); const operator = this.symbol('==') ?? this.symbol('!='); if (!operator || operator.kind !== 'symbol') return left; - return { kind: 'binary', operator: operator.text as '==' | '!=', left, right: this.primary(), column: operator.column }; + return { kind: 'binary', operator: operator.text as '==' | '!=', left, right: this.comparison(), column: operator.column }; + } + + private comparison(): Expression { + const left = this.primary(); + const operator = this.peek(); + if (operator.kind !== 'symbol' || !Object.hasOwn(COMPARISONS, operator.text)) return left; + this.take(); + return { kind: 'binary', operator: operator.text as Comparison, left, right: this.primary(), column: operator.column }; + } + + /** The arguments of a call, after its "(". */ + private call(name: FunctionName, column: number): Expression { + const usage = (): ExpressionError => new ExpressionError(column, `${name} is called as ${FUNCTIONS[name]}`); + if (this.symbol(')')) throw usage(); + const list = this.or(); + if (name === 'length') { + if (!this.symbol(')')) throw usage(); + return { kind: 'call', name, list, column }; + } + if (!this.symbol(',')) throw usage(); + this.predicateDepth += 1; + const predicate = this.or(); + this.predicateDepth -= 1; + if (!this.symbol(')')) throw usage(); + return { kind: 'call', name, list, predicate, column }; } private primary(): Expression { @@ -179,6 +353,10 @@ class Parser { case 'word': if (token.text === 'true' || token.text === 'false') return { kind: 'literal', value: token.text === 'true', column }; if (token.text === 'null') return { kind: 'literal', value: null, column }; + if (NAME.test(token.text) && this.symbol('(')) { + if (!Object.hasOwn(FUNCTIONS, token.text)) throw new ExpressionError(column, `"${token.text}" is not a function; the functions are length, any, all and filter`); + return this.call(token.text as FunctionName, column); + } return { kind: 'path', path: parsePath(token), column }; case 'symbol': if (token.text === '(') { @@ -193,39 +371,212 @@ class Parser { return { kind: 'literal', value: [], column }; } throw new ExpressionError(column, `expected a value, found ${describe(token)}`); + case 'element': + if (/^\.[0-9]+$/.test(token.text) || (this.predicateDepth === 0 && /^\.[0-9]/.test(token.text))) { + throw new ExpressionError(column, 'numbers need a leading digit, as in 0.5'); + } + if (this.predicateDepth === 0) { + throw new ExpressionError(column, `"${token.text}" reads the current element, which exists only inside the predicate of any, all or filter`); + } + return { kind: 'element', fields: fieldsOf(token.text.slice(1), column + 1), column }; case 'end': throw new ExpressionError(column, 'expected a value, found the end of the expression'); } } } +/** An operand or argument whose type does not fit where it is used, and the column it starts at. */ +export interface TypeProblem { + column: number; + message: string; +} + +function textOf(expression: Expression): string { + switch (expression.kind) { + case 'literal': return typeof expression.value === 'string' ? quoted(expression.value) : JSON.stringify(expression.value); + case 'path': { + const { path } = expression; + return path.root === 'steps' ? `steps.${path.step}.${path.field}${fieldsText(path.fields)}` : `${path.root}${fieldsText(path.fields)}`; + } + case 'element': { + const text = fieldsText(expression.fields); + return text.startsWith('.') ? text : `.${text}`; + } + case 'call': return `${expression.name}(...)`; + default: return 'a condition'; + } +} + +interface ElementType { + items: DeclaredType; + list: Expression; +} + +/** The type a workflow declares for what a path reads, or undefined when it is not known before the run. */ +export type SchemaOf = (path: ExpressionPath) => DeclaredType | undefined; + +type StaticType = Exclude | { type: 'array'; items?: undefined } | { type: 'null' }; + +function known(type: DeclaredType | undefined): StaticType | undefined { + return type?.type === 'unknown' ? undefined : type; +} + +function typeOf(expression: Expression, schemaOf: SchemaOf, element: ElementType | undefined): StaticType | undefined { + switch (expression.kind) { + case 'literal': { + const { value } = expression; + if (value === null) return { type: 'null' }; + if (Array.isArray(value)) return { type: 'array' }; + return typeof value === 'number' ? { type: 'number' } : typeof value === 'string' ? { type: 'string' } : { type: 'boolean' }; + } + case 'path': return known(schemaOf(expression.path)); + case 'element': return element && known(schemaAt(element.items, expression.fields)); + case 'not': + case 'binary': + return { type: 'boolean' }; + case 'call': { + if (expression.name === 'length') return { type: 'number' }; + if (expression.name !== 'filter') return { type: 'boolean' }; + const list = typeOf(expression.list, schemaOf, element); + return list?.type === 'array' ? list : { type: 'array' }; + } + } +} + +/** + * The comparisons whose operands are not numbers, the list arguments that are not arrays, and what predicates read + * that the items of their list do not declare. Parsing passes no `schemaOf`, so it finds only what the expression itself + * shows, such as `x < 'a'`; pull passes the declared types. + */ +export function typeProblems(expression: Expression, schemaOf: SchemaOf = () => undefined): TypeProblem[] { + return problemsIn(expression, schemaOf, undefined); +} + +function problemsIn(expression: Expression, schemaOf: SchemaOf, element: ElementType | undefined): TypeProblem[] { + switch (expression.kind) { + case 'literal': + case 'path': + return []; + case 'element': + return element && !schemaAt(element.items, expression.fields) + ? [{ column: expression.column, message: `reads ${textOf(expression)}, which is not in an item of ${textOf(element.list)}` }] + : []; + case 'not': return problemsIn(expression.operand, schemaOf, element); + case 'call': { + const problems = problemsIn(expression.list, schemaOf, element); + const list = typeOf(expression.list, schemaOf, element); + if (list && list.type !== 'array') { + problems.push({ column: expression.list.column, message: `passes ${textOf(expression.list)} to ${expression.name} as its list, but it is not an array` }); + } + if (expression.name === 'length') return problems; + const items = list?.type === 'array' && list.items ? { items: list.items, list: expression.list } : undefined; + return [...problems, ...problemsIn(expression.predicate, schemaOf, items)]; + } + case 'binary': { + const problems = [...problemsIn(expression.left, schemaOf, element), ...problemsIn(expression.right, schemaOf, element)]; + if (!Object.hasOwn(COMPARISONS, expression.operator)) return problems; + for (const operand of [expression.left, expression.right]) { + const type = typeOf(operand, schemaOf, element); + if (type && type.type !== 'number') { + problems.push({ column: operand.column, message: `compares ${textOf(operand)} with ${expression.operator}, but it is not a number` }); + } + } + return problems; + } + } +} + /** Parses an expression. Columns are 1-based offsets into `source`. */ export function parseExpression(source: string): ExpressionParse { try { - return { ok: true, expression: new Parser(source).parse() }; + const expression = new Parser(source).parse(); + const [problem] = typeProblems(expression); + if (problem) return { ok: false, ...problem }; + return { ok: true, expression }; } catch (error) { if (error instanceof ExpressionError) return { ok: false, column: error.column, message: error.message }; throw error; } } +/** A step field written as a path, `steps` misspelled `step` included, as in `steps.build.results`. */ +const STEP_FIELD = /^steps?\.[A-Za-z0-9_-]+\.(?:outcome|end|outputs|passes|results)(?![A-Za-z0-9_-])/; + +/** + * Whether `source` reads like an expression, judged by its tokens alone and past any that do not lex: its first value, + * or the first argument of a call in it, after any "!" and "(", is a path `path` matches, or a step field. Only a step + * field counts after a name that is not a function, since other text, such as a Vue sample, calls its own functions on + * its own names. Unlike parseExpression, it also finds an expression with a mistake in it, such as a misspelled + * function or a double-quoted string. + */ +export function readsLikeExpression(source: string, path: RegExp): boolean { + let first: 'value' | 'unknown call' | undefined = 'value'; + let previous: Token | undefined; + for (let index = 0; ;) { + let lexed: { token: Token; next: number }; + try { + lexed = lexAt(source, index); + } catch (error) { + if (!(error instanceof ExpressionError)) throw error; + index = error.column; + first = undefined; + previous = undefined; + continue; + } + const { token, next } = lexed; + if (token.kind === 'end') return false; + if (token.kind === 'symbol' && token.text === '(' && previous?.kind === 'word' && NAME.test(previous.text)) { + first = Object.hasOwn(FUNCTIONS, previous.text) ? 'value' : 'unknown call'; + } else if (first !== undefined && token.kind === 'word' && (STEP_FIELD.test(token.text) || (first === 'value' && path.test(token.text)))) { + return true; + } else if (!(token.kind === 'symbol' && (token.text === '!' || token.text === '('))) { + first = undefined; + } + previous = token; + index = next; + } +} + export function isRecord(value: unknown): value is Record { return !!value && typeof value === 'object' && !Array.isArray(value); } +/** The names and indexes after a path's root, as a path writes them, such as `.files[0].path`. */ +export function fieldsText(fields: PathField[]): string { + return fields.map((field) => (typeof field === 'number' ? `[${field}]` : `.${field}`)).join(''); +} + /** Reads a path. Anything missing along the way, including a step that has not settled, is `null`. */ export function resolvePath(path: ExpressionPath, context: ExpressionContext): unknown { - let value: unknown; - if (path.root === 'inputs') { - value = context.inputs; - } else { - const step = Object.hasOwn(context.steps, path.step) ? context.steps[path.step] : undefined; - value = step?.[path.field]; + if (path.root === 'inputs') return readFields(context.inputs, path.fields); + if (path.root === 'item') return readFields(context.item, path.fields); + const step = Object.hasOwn(context.steps, path.step) ? context.steps[path.step] : undefined; + return readFields(step?.[path.field], path.fields); +} + +/** The type `schema` declares for the value `fields` names inside it, or undefined when it declares none. */ +export function schemaAt(schema: DeclaredType, fields: PathField[]): DeclaredType | undefined { + let current = schema; + for (const field of fields) { + if (current.type === 'unknown') return current; + if (typeof field === 'number') { + if (current.type !== 'array') return undefined; + current = current.items; + } else { + if (current.type !== 'object' || !Object.hasOwn(current.fields, field)) return undefined; + current = current.fields[field]!; + } } - for (const field of path.fields) { - value = isRecord(value) && Object.hasOwn(value, field) ? value[field] : undefined; + return current; +} + +function readFields(value: unknown, fields: PathField[]): unknown { + let current = value; + for (const field of fields) { + if (typeof field === 'number') current = Array.isArray(current) ? current[field] : undefined; + else current = isRecord(current) && Object.hasOwn(current, field) ? current[field] : undefined; } - return value ?? null; + return current ?? null; } /** The falsy values are `false`, `null`, `0` and `''`. Everything else, `[]` and `{}` included, is truthy. */ @@ -233,17 +584,26 @@ export function isTruthy(value: unknown): boolean { return !(value === false || value === null || value === undefined || value === 0 || value === ''); } -/** Strict JSON equality: no type coercion, case-sensitive strings, arrays in order, objects by their keys. */ +/** + * Strict JSON equality: no type coercion, case-sensitive strings, arrays in order, objects by their keys. It walks the + * pairs with a stack of its own, since a value a step reports can nest deeper than the call stack goes. + */ function deepEqual(left: unknown, right: unknown): boolean { - if (left === right) return true; - if (Array.isArray(left) || Array.isArray(right)) { - return Array.isArray(left) && Array.isArray(right) && left.length === right.length - && left.every((item, index) => deepEqual(item, right[index])); + const pairs: Array<[unknown, unknown]> = [[left, right]]; + for (let pair = pairs.pop(); pair; pair = pairs.pop()) { + const [a, b] = pair; + if (a === b) continue; + if (Array.isArray(a) || Array.isArray(b)) { + if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length) return false; + a.forEach((item, index) => pairs.push([item, b[index]])); + continue; + } + if (!isRecord(a) || !isRecord(b)) return false; + const keys = Object.keys(a); + if (keys.length !== Object.keys(b).length || !keys.every((key) => Object.hasOwn(b, key))) return false; + for (const key of keys) pairs.push([a[key], b[key]]); } - if (!isRecord(left) || !isRecord(right)) return false; - const keys = Object.keys(left); - return keys.length === Object.keys(right).length - && keys.every((key) => Object.hasOwn(right, key) && deepEqual(left[key], right[key])); + return true; } /** Evaluates an expression. It never throws. */ @@ -259,17 +619,36 @@ export function evaluateExpression(expression: Expression, context: ExpressionCo case '||': return isTruthy(left) || isTruthy(evaluateExpression(expression.right, context)); case '==': return deepEqual(left, evaluateExpression(expression.right, context)); case '!=': return !deepEqual(left, evaluateExpression(expression.right, context)); + default: { + const right = evaluateExpression(expression.right, context); + return typeof left === 'number' && typeof right === 'number' && COMPARISONS[expression.operator](left, right); + } + } + } + case 'call': { + const list = evaluateExpression(expression.list, context); + if (!Array.isArray(list)) return null; + if (expression.name === 'length') return list.length; + const { predicate } = expression; + const holds = (element: unknown): boolean => isTruthy(evaluateExpression(predicate, { ...context, element })); + switch (expression.name) { + case 'any': return list.some(holds); + case 'all': return list.every(holds); + case 'filter': return list.filter(holds); } } + case 'element': return readFields(context.element, expression.fields); } } -/** Every path an expression reads, in source order. */ +/** Every `inputs`, `steps` and `item` path an expression reads, in source order. */ export function expressionPaths(expression: Expression): ExpressionPath[] { switch (expression.kind) { case 'literal': return []; case 'path': return [expression.path]; case 'not': return expressionPaths(expression.operand); case 'binary': return [...expressionPaths(expression.left), ...expressionPaths(expression.right)]; + case 'call': return [...expressionPaths(expression.list), ...(expression.name === 'length' ? [] : expressionPaths(expression.predicate))]; + case 'element': return []; } } diff --git a/src/workflow/fan-out.ts b/src/workflow/fan-out.ts new file mode 100644 index 000000000..b9687b074 --- /dev/null +++ b/src/workflow/fan-out.ts @@ -0,0 +1,115 @@ +/** + * Fan-out. A step with `for_each` runs its action once per item of a list, each as an instance such as `build[2]`, + * and settles once every item has, keeping how each one went in `results`. These rules decide the items and the + * step's result; the engine settles and leases the items. + */ +import { isRecord } from './expression.js'; +import { firstCycle, firstPass } from './loop.js'; +import { parseStepPath } from './step-path.js'; +import { passOf, stopped, type ItemResult, type StepState } from './step-state.js'; +import type { RunState, WorkflowFailure } from './types.js'; + +/** Whether an instance is one item of a `for_each` step, such as `build[2]`. */ +export function isItem(state: StepState): boolean { + return parseStepPath(state.id).at(-1)?.index !== undefined; +} + +/** The items of a `for_each` step, in item order. */ +export function itemsOf(run: RunState, state: StepState): StepState[] { + return run.steps.filter((item) => item.id.startsWith(`${state.id}[`) && !item.id.slice(state.id.length).includes('.')); +} + +/** Adds an instance per item right after the `for_each` step, each after every archived pass of its id. */ +export function addItems(run: RunState, state: StepState, items: unknown[]): void { + const at = run.steps.indexOf(state) + 1; + run.steps = [...run.steps.slice(0, at), ...items.map((item, index): StepState => { + const id = `${state.id}[${index}]`; + const pass = firstPass(run, id); + return { id, status: 'pending', item, ...(pass > 1 ? { pass } : {}) }; + }), ...run.steps.slice(at)]; +} + +function kindOf(value: unknown): string { + if (value === null || value === undefined) return 'null'; + return typeof value === 'object' ? 'an object' : `a ${typeof value}`; +} + +/** The items the `for_each` of step `id` gives, or why the step cannot run them. */ +export function expandItems(id: string, value: unknown): unknown[] | WorkflowFailure { + if (!Array.isArray(value)) return { code: 'invalid_items', message: `step "${id}" for_each gives ${kindOf(value)}, not a list of items` }; + const after = itemsAfter(id, value); + if (!Array.isArray(after)) return after; + // `after` names only items with an id, so only they can be in a cycle. + const edges = new Map(after.flatMap((indexes, index) => { + const name = idOf(value[index]); + return name === undefined ? [] : [[name, indexes.map((other) => idOf(value[other]) ?? '')] as const]; + })); + const cycle = firstCycle([...edges.keys()], edges); + if (cycle) return { code: 'item_cycle', message: `step "${id}" has items that run after each other in a cycle: ${cycle.join(' -> ')}` }; + return value; +} + +function idOf(item: unknown): string | undefined { + return isRecord(item) && typeof item.id === 'string' ? item.id : undefined; +} + +/** + * For each item of step `id`, the indexes of the items its `after` names, or why they cannot be read: two items with + * the same id, an `after` that is not a list of ids, or an id no item has. + */ +export function itemsAfter(id: string, items: unknown[]): number[][] | WorkflowFailure { + const invalid = (message: string): WorkflowFailure => ({ code: 'invalid_items', message: `step "${id}" ${message}` }); + const indexes = new Map(); + for (const [index, item] of items.entries()) { + const own = idOf(item); + if (own === undefined) continue; + const earlier = indexes.get(own); + if (earlier !== undefined) return invalid(`items ${earlier} and ${index} have the same id "${own}"`); + indexes.set(own, index); + } + const after: number[][] = []; + for (const [index, item] of items.entries()) { + const names: unknown = isRecord(item) ? item.after : undefined; + if (names === undefined) { + after.push([]); + continue; + } + if (!Array.isArray(names) || !names.every((name): name is string => typeof name === 'string')) { + return invalid(`item ${index} after must be a list of item ids`); + } + const unknown = names.find((name) => !indexes.has(name)); + if (unknown !== undefined) return invalid(`item ${index} runs after "${unknown}", which is not the id of an item`); + after.push(names.map((name) => indexes.get(name)!)); + } + return after; +} + +/** + * Where a pending item stands on the items its `after` names: it can start once all of them succeeded, and it is + * skipped once one of them failed, was skipped, or was cancelled. + */ +export function afterStanding(after: StepState[]): 'ready' | 'waiting' | 'skip' { + if (after.some((item) => stopped(item) || item.status === 'skipped')) return 'skip'; + return after.every((item) => item.status === 'succeeded') ? 'ready' : 'waiting'; +} + +/** How many items are started and not yet settled, which `max_parallel` bounds. */ +export function startedItems(items: StepState[]): number { + return items.filter((item) => item.status === 'leased' || item.status === 'running').length; +} + +/** + * How a `for_each` step settles at `settledAt` once every item has: failed when an item stopped, with each item's + * outcome and outputs. With no items, it succeeds. + */ +export function itemsSettled(items: StepState[], settledAt: number): { outcome: 'succeeded' | 'failed'; settledAt: number; results: ItemResult[] } { + const results = items.map((state): ItemResult => { + const pass = passOf(state); + return { item: state.item, outcome: pass?.outcome ?? 'skipped', ...(pass?.outputs !== undefined ? { outputs: pass.outputs } : {}) }; + }); + return { + outcome: items.some(stopped) ? 'failed' : 'succeeded', + settledAt, + results, + }; +} diff --git a/src/workflow/local-driver.ts b/src/workflow/local-driver.ts index 35946f6d1..5ebbd108e 100644 --- a/src/workflow/local-driver.ts +++ b/src/workflow/local-driver.ts @@ -11,6 +11,7 @@ import { type AgentSpec, type ToolName, } from '../resources/agent-format.js'; +import { exitLike } from '../env-exec.js'; import { getDataHome, resolveToolBaseDir, type LocalConfig, type TeamaiConfig } from '../types.js'; import { readFileSafe, writeJsonAtomic } from '../utils/fs.js'; import { log } from '../utils/logger.js'; @@ -30,10 +31,34 @@ export function localDriver(cwd: string): DriverIdentity { return { kind: 'local', machine, workspace: path.resolve(cwd) }; } -/** Runs a leased command. The lease `env` is set over the inherited environment, so values reach the shell only as variables. */ -function runCommand(command: string, cwd: string, env: Record): Promise<{ exitCode: number; stdout: string; stderr: string }> { - return new Promise((resolve) => { - const child = spawn(command, { cwd, shell: true, env: { ...process.env, ...env }, windowsHide: true }); +interface CommandResult { + exitCode: number; + stdout: string; + stderr: string; +} + +interface CommandProcess { + /** Settles when the command has exited and closed its output. */ + exit: Promise; + /** + * SIGTERM, then SIGKILL after the grace period. Settles once the command's process group is gone, which can be after + * `exit` when a process it started outlives it; a second call changes nothing. + */ + stop(): Promise; +} + +const STOP_GRACE_MS = 5000; + +const STOP_SIGNALS: readonly NodeJS.Signals[] = ['SIGINT', 'SIGQUIT', 'SIGTERM', 'SIGHUP']; + +/** + * Starts a leased command. The lease `env` is set over the inherited environment, so values reach the shell only as + * variables. A detached command leads a process group of its own, so `stop` reaches everything it started; it has no + * controlling terminal, and a terminal's Ctrl-C does not reach it. + */ +function startCommand(command: string, cwd: string, env: Record, detached: boolean): CommandProcess { + const child = spawn(command, { cwd, shell: true, env: { ...process.env, ...env }, windowsHide: true, detached }); + const exit = new Promise((resolve) => { let stdout = ''; let stderr = ''; child.stdout?.on('data', (chunk: Buffer) => { stdout += chunk.toString(); }); @@ -41,6 +66,39 @@ function runCommand(command: string, cwd: string, env: Record): child.on('error', (error) => resolve({ exitCode: 1, stdout, stderr: error.message })); child.on('close', (code) => resolve({ exitCode: code ?? 1, stdout, stderr })); }); + /** Whether the signal reached a process: the command, or for a detached command any process left in its group. */ + const signal = (name: NodeJS.Signals | 0): boolean => { + if (!detached || child.pid === undefined) return name === 0 ? false : child.kill(name); + try { + process.kill(-child.pid, name); + return true; + } catch { + // The whole group has already exited. + return false; + } + }; + let stopping: Promise | undefined; + const stop = async (): Promise => { + if (process.platform === 'win32' && child.pid !== undefined) { + execFile('taskkill', ['/pid', String(child.pid), '/T', '/F'], () => {}); + return exit; + } + signal('SIGTERM'); + let timer: NodeJS.Timeout | undefined; + const killed = new Promise((resolve) => { + timer = setTimeout(() => { + signal('SIGKILL'); + resolve(); + }, STOP_GRACE_MS); + }); + const result = await exit; + // The shell closing its output does not mean the group is gone: a process it started with its own output can + // ignore SIGTERM and outlive it. Keep the SIGKILL for as long as any process in the group is left. + if (signal(0)) await killed; + else clearTimeout(timer); + return result; + }; + return { exit, stop: () => (stopping ??= stop()) }; } function jsonStdout(stdout: string): unknown | undefined { @@ -162,9 +220,111 @@ function failureReport(failure: WorkflowFailure): { outcome: 'failed'; message: return { outcome: 'failed', message: failure.message, failure }; } +type ReportBody = Parameters[2]; + +function parseLease(raw: unknown): Lease | undefined { + const rawLease = raw as Record | undefined; + if (!rawLease || typeof rawLease.attempt !== 'string' || typeof rawLease.revision !== 'number') return undefined; + const env = stringMap(rawLease.env); + const lease: Lease = { + attempt: rawLease.attempt, + revision: rawLease.revision, + ...(typeof rawLease.command === 'string' ? { command: rawLease.command } : {}), + ...(env ? { env } : {}), + ...(typeof rawLease.prompt === 'string' ? { prompt: rawLease.prompt } : {}), + ...(typeof rawLease.agent === 'string' ? { agent: rawLease.agent } : {}), + }; + return (lease.command === undefined) === (lease.prompt === undefined) ? undefined : lease; +} + +async function commandReport(attemptResultPath: string, driver: DriverIdentity, lease: Lease, result: CommandResult): Promise { + const output = jsonStdout(result.stdout); + await fs.mkdir(path.dirname(attemptResultPath), { recursive: true }); + await writeJsonAtomic(attemptResultPath, output !== undefined ? { output } : {}); + const snippet = result.stderr.trim().slice(0, 400); + return { + attempt: lease.attempt, + revision: lease.revision, + outcome: result.exitCode === 0 ? 'success' : 'failed', + driver, + ...(output !== undefined ? { outputs: output } : {}), + exitCode: result.exitCode, + ...(result.exitCode !== 0 + ? { message: snippet === '' ? `command exited with status ${result.exitCode}` : `command exited with status ${result.exitCode}: ${snippet}` } + : {}), + }; +} + +async function promptReport( + config: LocalConfig, + attemptResultPath: string, + driver: DriverIdentity, + lease: Lease, + tools: Record, + adapter: LocalToolAdapter, + options: LocalDriverOptions, +): Promise { + const selected = lease.agent && tools[lease.agent] ? tools[lease.agent] : tools.default; + if (!selected) { + const failure = { code: 'tool_unconfigured', message: 'No local workflow tool is configured for this prompt step' }; + return { attempt: lease.attempt, revision: lease.revision, driver, ...failureReport(failure) }; + } + if (!supportedTool(selected)) { + const failure = { code: 'tool_unsupported', message: `The local workflow tool "${selected}" is not supported` }; + return { attempt: lease.attempt, revision: lease.revision, driver, tool: selected, ...failureReport(failure) }; + } + const agent = lease.agent ? await loadAgent(selected, lease.agent, config, options.toolPaths) : undefined; + if (lease.agent && !agent) { + const failure = { + code: 'agent_unavailable', + message: `Agent "${lease.agent}" is not available for ${selected}. Run teamai pull and try again.`, + }; + return { attempt: lease.attempt, revision: lease.revision, driver, tool: selected, ...failureReport(failure) }; + } + if (agent && localToolFamily(selected) === 'codex') { + const ignored = ignoredCodexSettings(agent); + if (ignored.length > 0) { + log.warn(`Agent "${agent.name}" on ${selected} does not apply ${ignored.join(', ')} from the deployed agent file. codex exec still has a shell. The step is not blocked.`); + } + } + const before = await snapshotWorktree(driver.workspace); + let result; + try { + result = await adapter.run(selected, { + prompt: lease.prompt!, + cwd: driver.workspace, + resultPath: attemptResultPath, + ...(agent ? { agent: { name: lease.agent!, instructions: agent.instructions, ...(agent.model ? { model: agent.model } : {}) } } : {}), + }); + } catch { + result = { + exitCode: 1, + failure: { code: 'tool_failed', message: `${selected} could not complete the workflow step` }, + }; + } + const after = await snapshotWorktree(driver.workspace); + const paths = changedPaths(before, after); + const dirtyFailure = paths.length > 0 + ? { code: 'read_only_dirty', message: `Read-only prompt step changed repository paths: ${paths.join(', ')}` } + : undefined; + const failure = dirtyFailure ?? result.failure; + return { + attempt: lease.attempt, + revision: lease.revision, + driver, + tool: selected, + ...(result.model ?? agent?.model ? { model: result.model ?? agent?.model } : {}), + ...(result.session ? { session: result.session } : {}), + ...(result.tokens ? { tokens: result.tokens } : {}), + ...(result.output !== undefined ? { outputs: result.output } : {}), + exitCode: result.exitCode, + ...(failure ? failureReport(failure) : { outcome: result.exitCode === 0 ? 'success' : 'failed' }), + }; +} + /** - * Run leased commands one at a time. Each lease and report goes through the - * same engine operations an external driver calls. + * Run leased commands, up to `workflows.max_parallel` at once. Each lease and + * report goes through the same engine operations an external driver calls. */ export async function runLocalDriver( config: LocalConfig, @@ -175,118 +335,104 @@ export async function runLocalDriver( const driver = localDriver(cwd); const tools = { ...config.workflows?.tools, ...options.tools }; const adapter = options.adapter ?? createLocalToolAdapter(); + const maxParallel = config.workflows?.max_parallel ?? 1; + // One command at a time keeps the terminal: Ctrl-C reaches it, and it can prompt on /dev/tty. Parallel commands get + // process groups of their own, which `stop` and the signal handlers below can end as a whole. + const detached = maxParallel > 1 && process.platform !== 'win32'; const reported = new Set(); - for (;;) { - const next = await leaseNext(config, runId, driver); - if (isWorkflowError(next)) return next; - if ('done' in next.data) return next; - if ('wait' in next.data) { - const reason = typeof (next.data.wait as { reason?: unknown }).reason === 'string' - ? (next.data.wait as { reason: string }).reason - : 'the next step is not available'; - return { error: { code: 'busy', message: reason } }; - } - const rawLease = next.data.lease as Record | undefined; - if (!rawLease || typeof rawLease.attempt !== 'string' || typeof rawLease.revision !== 'number') { - return { error: { code: 'run_unreadable', message: `Run ${runId} returned a lease the local driver cannot run` } }; - } - const env = stringMap(rawLease.env); - const lease: Lease = { - attempt: rawLease.attempt, - revision: rawLease.revision, - ...(typeof rawLease.command === 'string' ? { command: rawLease.command } : {}), - ...(env ? { env } : {}), - ...(typeof rawLease.prompt === 'string' ? { prompt: rawLease.prompt } : {}), - ...(typeof rawLease.agent === 'string' ? { agent: rawLease.agent } : {}), - }; - if ((lease.command === undefined) === (lease.prompt === undefined)) { - return { error: { code: 'run_unreadable', message: `Run ${runId} returned a lease the local driver cannot run` } }; - } - if (reported.has(lease.attempt)) { - return { error: { code: 'driver_stuck', message: `Attempt ${lease.attempt} was leased again before it could be reported` } }; - } - let body: Parameters[2]; - const attemptResultPath = resultPath(config, runId, lease.attempt); - if (lease.command !== undefined) { - const result = await runCommand(lease.command, driver.workspace, lease.env ?? {}); - const output = jsonStdout(result.stdout); - await fs.mkdir(path.dirname(attemptResultPath), { recursive: true }); - await writeJsonAtomic(attemptResultPath, output !== undefined ? { output } : {}); - const snippet = result.stderr.trim().slice(0, 400); - body = { - attempt: lease.attempt, - revision: lease.revision, - outcome: result.exitCode === 0 ? 'success' : 'failed', - driver, - ...(output !== undefined ? { outputs: output } : {}), - exitCode: result.exitCode, - ...(result.exitCode !== 0 - ? { message: snippet === '' ? `command exited with status ${result.exitCode}` : `command exited with status ${result.exitCode}: ${snippet}` } - : {}), - }; - } else { - const selected = lease.agent && tools[lease.agent] ? tools[lease.agent] : tools.default; - if (!selected) { - const failure = { code: 'tool_unconfigured', message: 'No local workflow tool is configured for this prompt step' }; - body = { attempt: lease.attempt, revision: lease.revision, driver, ...failureReport(failure) }; - } else if (!supportedTool(selected)) { - const failure = { code: 'tool_unsupported', message: `The local workflow tool "${selected}" is not supported` }; - body = { attempt: lease.attempt, revision: lease.revision, driver, tool: selected, ...failureReport(failure) }; - } else { - const agent = lease.agent ? await loadAgent(selected, lease.agent, config, options.toolPaths) : undefined; - if (lease.agent && !agent) { - const failure = { - code: 'agent_unavailable', - message: `Agent "${lease.agent}" is not available for ${selected}. Run teamai pull and try again.`, - }; - body = { attempt: lease.attempt, revision: lease.revision, driver, tool: selected, ...failureReport(failure) }; - } else { - if (agent && localToolFamily(selected) === 'codex') { - const ignored = ignoredCodexSettings(agent); - if (ignored.length > 0) { - log.warn(`Agent "${agent.name}" on ${selected} does not apply ${ignored.join(', ')} from the deployed agent file. codex exec still has a shell. The step is not blocked.`); - } - } - const before = await snapshotWorktree(driver.workspace); - let result; - try { - result = await adapter.run(selected, { - prompt: lease.prompt!, - cwd: driver.workspace, - resultPath: attemptResultPath, - ...(agent ? { agent: { name: lease.agent!, instructions: agent.instructions, ...(agent.model ? { model: agent.model } : {}) } } : {}), - }); - } catch { - result = { - exitCode: 1, - failure: { code: 'tool_failed', message: `${selected} could not complete the workflow step` }, - }; - } - const after = await snapshotWorktree(driver.workspace); - const paths = changedPaths(before, after); - const dirtyFailure = paths.length > 0 - ? { code: 'read_only_dirty', message: `Read-only prompt step changed repository paths: ${paths.join(', ')}` } - : undefined; - const failure = dirtyFailure ?? result.failure; - body = { - attempt: lease.attempt, - revision: lease.revision, - driver, - tool: selected, - ...(result.model ?? agent?.model ? { model: result.model ?? agent?.model } : {}), - ...(result.session ? { session: result.session } : {}), - ...(result.tokens ? { tokens: result.tokens } : {}), - ...(result.output !== undefined ? { outputs: result.output } : {}), - exitCode: result.exitCode, - ...(failure ? failureReport(failure) : { outcome: result.exitCode === 0 ? 'success' : 'failed' }), - }; + const inFlight = new Map(); + // Leases not started yet, in lease order. A prompt starts only when no command runs, because its read-only check + // compares the whole worktree before and after it. + const held: Lease[] = []; + // Cancelled commands whose process group is not gone yet. They are not sent as running, so `next` does not cancel + // them again. + const dying = new Map(); + const room = (): boolean => inFlight.size + held.length < maxParallel && !held.some((lease) => lease.prompt !== undefined); + const cancel = (attempt: string): void => { + const index = held.findIndex((lease) => lease.attempt === attempt); + if (index !== -1) held.splice(index, 1); + const running = inFlight.get(attempt); + if (!running) return; + inFlight.delete(attempt); + dying.set(attempt, running.command); + void running.command.stop(); + }; + let signalled: NodeJS.Signals | undefined; + let wake = (): void => {}; + const exitOn = (signal: NodeJS.Signals): void => { + for (const name of STOP_SIGNALS) process.off(name, onSignal); + exitLike({ kind: 'signaled', signal }); + }; + // Parallel commands do not get the terminal's Ctrl-C, so the driver stops them all before it exits the way the + // signal would. With none running, as during a prompt, the signal ends teamai at once, as it would without a handler. + const onSignal = (signal: NodeJS.Signals): void => { + if (inFlight.size === 0 && dying.size === 0) return exitOn(signal); + signalled ??= signal; + wake(); + }; + if (maxParallel > 1) for (const name of STOP_SIGNALS) process.on(name, onSignal); + const report = async (body: ReportBody): Promise => { + const answer = await reportAttempt(config, runId, body); + // Cancelled while it ran: the next `next` lists it in `cancelled`, and the run goes on without it. + if (isWorkflowError(answer)) return answer.error.code === 'lease_cancelled' ? undefined : answer; + reported.add(body.attempt); + return undefined; + }; + try { + for (;;) { + if (signalled) return { error: { code: 'interrupted', message: `Stopped by ${signalled} after every command it started exited` } }; + const running = [...inFlight.keys(), ...held.map((lease) => lease.attempt)]; + const canTakeLease = room(); + const next = await leaseNext(config, runId, driver, { running, canTakeLease }); + if (signalled) continue; + if (isWorkflowError(next)) return next; + for (const attempt of Array.isArray(next.data.cancelled) ? next.data.cancelled as string[] : []) cancel(attempt); + if ('done' in next.data) return next; + if ('lease' in next.data) { + const lease = parseLease(next.data.lease); + if (!lease) return { error: { code: 'run_unreadable', message: `Run ${runId} returned a lease the local driver cannot run` } }; + if (reported.has(lease.attempt)) { + return { error: { code: 'driver_stuck', message: `Attempt ${lease.attempt} was leased again before it could be reported` } }; } + held.push(lease); } + while (held[0]?.command !== undefined && inFlight.size < maxParallel) { + const lease = held.shift()!; + inFlight.set(lease.attempt, { lease, command: startCommand(lease.command!, driver.workspace, lease.env ?? {}, detached) }); + } + if (held[0]?.prompt !== undefined && inFlight.size === 0 && dying.size === 0) { + const lease = held.shift()!; + const failed = await report(await promptReport(config, resultPath(config, runId, lease.attempt), driver, lease, tools, adapter, options)); + if (failed) return failed; + continue; + } + if ('lease' in next.data && room()) continue; + if (inFlight.size === 0 && dying.size === 0) { + // A wait for a call that could not take a lease says nothing about what is left; ask again with room. + if (!canTakeLease) continue; + const reason = typeof (next.data.wait as { reason?: unknown }).reason === 'string' + ? (next.data.wait as { reason: string }).reason + : 'the next step is not available'; + return { error: { code: 'busy', message: reason } }; + } + const exited = await Promise.race([ + ...[...inFlight.values()].map(({ lease, command }) => command.exit.then((result) => ({ attempt: lease.attempt, lease, result }))), + ...[...dying].map(([attempt, command]) => command.stop().then(() => ({ attempt }))), + new Promise((resolve) => { wake = () => resolve(undefined); }), + ]); + if (!exited) continue; + if (!('lease' in exited)) { + dying.delete(exited.attempt); + continue; + } + inFlight.delete(exited.attempt); + const failed = await report(await commandReport(resultPath(config, runId, exited.attempt), driver, exited.lease, exited.result)); + if (failed) return failed; } - const reportedEnvelope = await reportAttempt(config, runId, { - ...body, - }); - if (isWorkflowError(reportedEnvelope)) return reportedEnvelope; - reported.add(lease.attempt); + } finally { + for (const attempt of inFlight.keys()) cancel(attempt); + await Promise.all([...dying.values()].map((command) => command.stop())); + if (signalled) exitOn(signalled); + for (const name of STOP_SIGNALS) process.off(name, onSignal); } } diff --git a/src/workflow/local-tool-adapter.ts b/src/workflow/local-tool-adapter.ts index f59a11426..78216b786 100644 --- a/src/workflow/local-tool-adapter.ts +++ b/src/workflow/local-tool-adapter.ts @@ -6,21 +6,12 @@ import { resolveCliPath } from '../utils/cli-path.js'; import { writeJsonAtomic } from '../utils/fs.js'; import type { TokenUsage, WorkflowFailure } from './types.js'; -export type LocalWorkflowTool = - | 'claude' - | 'claude-internal' - | 'tclaude' - | 'codex' - | 'codex-internal' - | 'tcodex' - | 'opencode'; +const LOCAL_WORKFLOW_TOOLS = ['claude', 'claude-internal', 'tclaude', 'codex', 'codex-internal', 'tcodex', 'opencode'] as const; -const LOCAL_WORKFLOW_TOOLS = new Set([ - 'claude', 'claude-internal', 'tclaude', 'codex', 'codex-internal', 'tcodex', 'opencode', -]); +export type LocalWorkflowTool = (typeof LOCAL_WORKFLOW_TOOLS)[number]; export function isLocalWorkflowTool(value: string): value is LocalWorkflowTool { - return LOCAL_WORKFLOW_TOOLS.has(value); + return (LOCAL_WORKFLOW_TOOLS as readonly string[]).includes(value); } export function localToolFamily(tool: LocalWorkflowTool): 'claude' | 'codex' | 'opencode' { diff --git a/src/workflow/loop.ts b/src/workflow/loop.ts new file mode 100644 index 000000000..3b74cef93 --- /dev/null +++ b/src/workflow/loop.ts @@ -0,0 +1,251 @@ +/** + * Loops. A step with `loop` runs its range again after it succeeds: `back_to`, the step, and every step between them. + * A workflow with `loop` runs all of its steps again, round after round. These rules are shared by pull, which rejects + * loops a run could not settle, and by the engine, which settles them. + */ +import type { ExpressionPass, ExpressionStep } from './expression.js'; +import { endLoop, exhaust, nextRound, passOf, repend, type StepPass, type StepState } from './step-state.js'; +import type { RunState, StepLoop, WorkflowFailure, WorkflowLoop, WorkflowStep } from './types.js'; + +type LoopStep = Pick; + +/** The steps of one workflow that loops run again, and what waits for those loops. */ +export interface StepLoops { + /** Each loop's range, by the id of the step that owns it. */ + ranges: Map>; + /** The loop owners a step waits for besides what it needs and reads: the loops outside whose range it needs or reads a step inside it. */ + waitsFor(id: string, reads: Iterable): string[]; + /** + * The steps a step reads in their previous pass: those in its own range that need it, directly or not. Such a read + * sees the target's latest archived pass, nothing on pass 1, and never waits. + */ + previous(id: string): Set; +} + +/** A step loop pull rejects, at the key of the step it is reported on. */ +export interface LoopProblem { + id: string; + key: 'loop' | 'loop.back_to' | 'needs' | 'break' | 'for_each'; + message: string; +} + +/** The ids `id` needs, directly or through their needs. */ +export function ancestors(needs: Map, id: string): Set { + const found = new Set(); + const pending = [...needs.get(id) ?? []]; + for (let need = pending.pop(); need !== undefined; need = pending.pop()) { + if (found.has(need)) continue; + found.add(need); + pending.push(...needs.get(need) ?? []); + } + return found; +} + +/** R(X): `back_to`, the owner X, and every step that needs `back_to` and that X needs, directly or not. */ +function rangeOf(needs: Map, owner: string, backTo: string): Set { + const before = ancestors(needs, owner); + return new Set([backTo, ...[...before].filter((id) => ancestors(needs, id).has(backTo)), owner]); +} + +export function stepLoops(steps: LoopStep[]): StepLoops { + const needs = new Map(steps.map((step) => [step.id, step.needs ?? []])); + const ranges = new Map(steps.flatMap((step) => (step.loop ? [[step.id, rangeOf(needs, step.id, step.loop.backTo)] as const] : []))); + return { + ranges, + waitsFor: (id, reads) => { + const used = [...needs.get(id) ?? [], ...reads]; + return [...ranges].filter(([, range]) => !range.has(id) && used.some((read) => range.has(read))).map(([owner]) => owner); + }, + previous: (id) => { + const range = [...ranges.values()].find((members) => members.has(id)) ?? []; + return new Set([...range].filter((member) => ancestors(needs, member).has(id))); + }, + }; +} + +/** The first cycle through `edges`, starting at the id listed first, or undefined when there is none. */ +export function firstCycle(ids: string[], edges: Map): string[] | undefined { + const state = new Map(); + const path: string[] = []; + const visit = (id: string): string[] | undefined => { + state.set(id, 'open'); + path.push(id); + for (const next of edges.get(id) ?? []) { + if (state.get(next) === 'open') return [...path.slice(path.indexOf(next)), next]; + const found = state.has(next) || !edges.has(next) ? undefined : visit(next); + if (found) return found; + } + path.pop(); + state.set(id, 'closed'); + return undefined; + }; + for (const id of ids) { + const found = state.has(id) ? undefined : visit(id); + if (found) return found; + } + return undefined; +} + +/** + * The step loops and breaks of one workflow that pull rejects. `reads` holds the steps of this workflow each step reads + * in the current pass, and `looping` is whether the workflow has a loop of its own. A needs cycle is reported elsewhere, + * so a wait cycle is reported only when `needs` alone has none. + */ +export function loopProblems(steps: LoopStep[], reads: Map>, looping: boolean): LoopProblem[] { + const problems: LoopProblem[] = []; + const needs = new Map(steps.map((step) => [step.id, step.needs ?? []])); + const { ranges, waitsFor } = stepLoops(steps); + const claimed = new Map(); + for (const step of steps) { + const other = step.forEach && (['loop', 'end', 'break'] as const).find((key) => step[key] !== undefined); + if (!other) continue; + const why = other === 'loop' ? 'put the loop in a sub-workflow' : 'a step that runs once per item cannot end its workflow'; + problems.push({ id: step.id, key: 'for_each', message: `step "${step.id}" has for_each and ${other}; ${why}` }); + } + for (const step of steps) { + if (!step.loop || step.forEach) continue; + if (looping) { + problems.push({ id: step.id, key: 'loop', message: `step "${step.id}" has loop in a workflow that loops; put one of the two loops in a sub-workflow` }); + continue; + } + const { backTo } = step.loop; + if (backTo !== step.id && !ancestors(needs, step.id).has(backTo)) { + problems.push({ id: step.id, key: 'loop.back_to', message: `step "${step.id}" loop.back_to is "${backTo}", which is not "${step.id}" or a step it needs, directly or through its needs` }); + continue; + } + if (step.end !== undefined) { + problems.push({ id: step.id, key: 'loop', message: `step "${step.id}" has loop and end; a step that ends its workflow cannot run again` }); + continue; + } + const range = ranges.get(step.id) ?? new Set(); + const shared = [...range].find((id) => claimed.has(id)); + if (shared !== undefined) { + problems.push({ + id: step.id, + key: 'loop', + message: `step "${step.id}" loop runs "${shared}" again, which the loop of step "${claimed.get(shared)}" also runs; loops in one workflow share no step, so put the inner one in a sub-workflow`, + }); + continue; + } + for (const id of range) claimed.set(id, step.id); + } + for (const step of steps) { + const owner = claimed.get(step.id); + if (step.break === undefined || step.forEach || owner === undefined) continue; + problems.push({ id: step.id, key: 'break', message: `step "${step.id}" has break inside the loop of step "${owner}", where its if and while already end the loop` }); + } + if (problems.length > 0 || firstCycle(steps.map((step) => step.id), needs)) return problems; + const waits = new Map(steps.map((step) => [step.id, [...needs.get(step.id) ?? [], ...waitsFor(step.id, reads.get(step.id) ?? [])]])); + const cycle = firstCycle(steps.map((step) => step.id), waits); + if (cycle) { + problems.push({ + id: cycle[0]!, + key: 'needs', + message: `step "${cycle[0]}" waits in a cycle: ${cycle.join(' -> ')}; a step that needs or reads a step inside a loop waits until the loop ends`, + }); + } + return problems; +} + +/** A loop owner whose loop runs no more in this activation: skipped, failed or cancelled, or succeeded and decided final. */ +export function loopSettled(owner: StepState | undefined): boolean { + if (!owner) return false; + if (owner.status === 'succeeded') return owner.loopFinal === true; + return owner.status === 'skipped' || owner.status === 'failed' || owner.status === 'cancelled'; +} + +function exhausted(owner: StepState, loop: StepLoop): WorkflowFailure { + return { code: 'loop_exhausted', message: `Loop of step "${owner.id}" is exhausted: it ran ${loop.maxIterations} times, its max_iterations` }; +} + +/** + * Decides, once per run of a succeeded loop owner, what follows it. `holds` is its `while`, or true when it has none. + * The loop ends when `while` is false. While it holds, the range runs again, unless the owner has already run + * `max_iterations` times: then it fails `loop_exhausted` and keeps its outputs. A loop without `while` always runs + * again, and `exhaustedAtStart` stops the run that would go past the limit. + */ +export function decideStepLoop(run: RunState, owner: Extract, loop: StepLoop, holds: boolean, range: StepState[]): void { + const runs = (owner.iteration ?? 0) + 1; + if (!holds) { + endLoop(owner, runs); + } else if (loop.while !== undefined && runs >= loop.maxIterations) { + exhaust(owner, exhausted(owner, loop), owner.settledAt, runs); + } else { + for (const state of range) { + archiveUnder(run, state.id); + repend(state, state === owner ? runs : undefined); + } + } +} + +/** + * A loop owner about to start an action it has already run `max_iterations` times fails instead, with the outputs of + * its latest pass. Returns whether it did. + */ +export function exhaustedAtStart(owner: Extract, loop: StepLoop, settledAt: number): boolean { + if ((owner.iteration ?? 0) < loop.maxIterations) return false; + exhaust(owner, exhausted(owner, loop), settledAt); + return true; +} + +/** + * Decides what follows a round of a looping workflow that reached no end and no break. `holds` is its `while`, or true + * when it has none. The workflow ends when `while` is false. While it holds, every step of the workflow runs again in + * the next round, kept on `runner`, the step that runs the workflow, or on the run at the top. After `max_iterations` + * rounds, the workflow fails `loop_exhausted` instead, with what its last round returned. + */ +export function decideRound( + run: RunState, + runner: StepState | undefined, + loop: WorkflowLoop, + holds: boolean, + label: string, + states: StepState[], +): 'ended' | 'again' | WorkflowFailure { + const round = (runner ? runner.round : run.round) ?? 1; + if (!holds) return 'ended'; + if (round >= loop.maxIterations) { + return { code: 'loop_exhausted', message: `Loop of ${label} is exhausted: it ran ${loop.maxIterations} rounds, its max_iterations` }; + } + for (const state of states) { + archiveUnder(run, state.id); + repend(state); + } + if (!runner) run.round = round + 1; + else if (runner.status === 'running') nextRound(runner); + return 'again'; +} + +/** Moves the instances under a step, those of the workflow it ran, from the run's steps to its archive, with every pass they had. */ +export function archiveUnder(run: RunState, path: string): void { + const isUnder = (state: StepState): boolean => state.id.startsWith(`${path}.`) || state.id.startsWith(`${path}[`); + for (const state of run.steps.filter(isUnder)) { + const current = passOf(state); + for (const pass of [...state.passes ?? [], ...(current ? [current] : [])]) (run.archive ??= []).push({ step: state.id, ...pass }); + } + run.steps = run.steps.filter((state) => !isUnder(state)); +} + +/** The pass a new instance at `path` starts at: after every pass the archive holds for that path, so a pass id is never reused. */ +export function firstPass(run: RunState, path: string): number { + return 1 + Math.max(0, ...(run.archive ?? []).filter((entry) => entry.step === path).map((entry) => entry.pass)); +} + +function expressionPasses(passes: StepPass[]): ExpressionPass[] { + return passes.map(({ pass, outcome, outputs, end, results }) => ({ + pass, outcome, ...(outputs !== undefined ? { outputs } : {}), ...(end !== undefined ? { end } : {}), ...(results !== undefined ? { results } : {}), + })); +} + +/** `steps..passes`: the passes a loop archived on the instance, then its current pass once it settled. */ +export function passesOf(state: StepState): ExpressionPass[] { + const current = passOf(state); + return expressionPasses([...state.passes ?? [], ...(current ? [current] : [])]); +} + +/** What a previous-pass read of a step sees: its latest archived pass, nothing of a pass on pass 1, and the archived passes. */ +export function previousPass(state: StepState): ExpressionStep { + const passes = expressionPasses(state.passes ?? []); + const { pass: _pass, ...latest } = passes.at(-1) ?? {}; + return { ...latest, passes }; +} diff --git a/src/workflow/parse-context.ts b/src/workflow/parse-context.ts new file mode 100644 index 000000000..5e488e4f1 --- /dev/null +++ b/src/workflow/parse-context.ts @@ -0,0 +1,98 @@ +/** The YAML and diagnostic helpers parse.ts and reads.ts share, kept apart so neither imports the other. */ + +import { isScalar, type LineCounter, type Node } from 'yaml'; +import { fieldsText, schemaAt, stepFieldSchema, type ExpressionPath } from './expression.js'; +import type { EndOutcome, OutputSchema, WorkflowDefinition, WorkflowDiagnostic } from './types.js'; + +export function isYamlNode(value: unknown): value is Node { + return !!value && typeof value === 'object' && 'range' in value; +} + +export function locate(file: string, lineCounter: LineCounter, node: Node | null | undefined, field: string, message: string): WorkflowDiagnostic { + let line = 1; + let column = 1; + const offset = node?.range?.[0]; + if (typeof offset === 'number') { + const pos = lineCounter.linePos(offset); + line = pos.line; + column = pos.col; + } + return { file, line, column, field, message }; +} + +export function scalarString(node: Node | null | undefined): string | undefined { + if (!isScalar(node) || typeof node.value !== 'string') return undefined; + return node.value; +} + +/** What the checks of one workflow file share: where a diagnostic points, where it goes, and what a sync knows of the other files. */ +export interface ParseContext { + file: string; + lineCounter: LineCounter; + diagnostics: WorkflowDiagnostic[]; + promptFileLocations: Record; + files: WorkflowFiles | undefined; +} + +/** What a sync knows about the other workflow files, so a step that runs one can be checked against it. */ +export interface WorkflowFiles { + /** The version of `name` a step would run after this sync, or null when it has none. */ + runnable(name: string): WorkflowDefinition | null; + /** The workflows running `name` from this file passes through until it runs this file again, such as ship -> check -> ship. */ + cycle(name: string): string[] | undefined; +} + +/** What a reader may read of the workflow a step runs, and how a message names that workflow. */ +export interface Child { + label: string; + outputs?: Record; + ends?: Record; +} + +/** Why `path` reads an output the workflow its step runs does not declare, or undefined. A pass and a result have the same outputs. */ +export function undeclaredOutput(children: Map, path: ExpressionPath): string | undefined { + if (path.root !== 'steps') return undefined; + const [index, field, name, nestedField, nestedName] = path.fields; + const inPass = (path.field === 'passes' || path.field === 'results') && typeof index === 'number' && field === 'outputs'; + const inPassResult = path.field === 'passes' && typeof index === 'number' && field === 'results' && typeof name === 'number' && nestedField === 'outputs'; + const output = path.field === 'outputs' ? index : inPass ? name : inPassResult ? nestedName : undefined; + if (output === undefined) return undefined; + const child = children.get(path.step); + if (!child || Object.hasOwn(child.outputs ?? {}, output)) return undefined; + return `reads steps.${path.step}.${path.field}${fieldsText(path.fields)}, which is not an output ${child.label} declares`; +} + +/** Why a step does not expose the field a path reads, shared by step reads and workflow outputs. */ +export function unavailableField(path: ExpressionPath, forEach: boolean): string | undefined { + if (path.root !== 'steps') return undefined; + const inPass = path.field === 'passes' && typeof path.fields[0] === 'number'; + const field = inPass ? path.fields[1] : path.field; + const step = `steps.${path.step}${inPass ? `.passes[${path.fields[0]}]` : ''}`; + if (forEach && field === 'outputs') return `reads ${step}.outputs, but step "${path.step}" has for_each, so its outputs are in ${step}.results`; + if (!forEach && field === 'results') return `reads ${step}.results, but step "${path.step}" has no for_each, so it has no results`; + return undefined; +} + +/** Why `path` reads what the `output` its step declares does not have, or undefined. */ +export function undeclaredField(path: ExpressionPath, output: OutputSchema | undefined, forEach: boolean): string | undefined { + if (path.root !== 'steps' || (path.field !== 'outputs' && path.field !== 'passes' && path.field !== 'results')) return undefined; + const schema = stepFieldSchema(path.field, output, forEach); + if (!schema || schemaAt(schema, path.fields)) return undefined; + const read = `steps.${path.step}.${path.field}${fieldsText(path.fields)}`; + if (path.field === 'outputs') return `reads ${read}, which is not in the output step "${path.step}" declares`; + return path.field === 'passes' + ? `reads ${read}, which is not in a pass of step "${path.step}": pass, outcome, end, and ${forEach ? 'results with outputs as its output declares' : 'outputs as its output declares'}` + : `reads ${read}, which is not in a result of step "${path.step}": item, outcome, and outputs as its output declares`; +} + +export interface GraphNode { + id: string; + field: string; + needs: string[]; + node: Node; +} + +export function position(lineCounter: LineCounter, node: Node | null | undefined): { line: number; column: number } { + const { line, column } = locate('', lineCounter, node, '', ''); + return { line, column }; +} diff --git a/src/workflow/parse.ts b/src/workflow/parse.ts index c5f31a4b6..2d381da16 100644 --- a/src/workflow/parse.ts +++ b/src/workflow/parse.ts @@ -1,16 +1,27 @@ import { isMap, isScalar, isSeq, LineCounter, parseDocument, type Node, type YAMLMap } from 'yaml'; - -function isYamlNode(value: unknown): value is Node { - return !!value && typeof value === 'object' && 'range' in value; -} -import { evaluateExpression, expressionPaths, isTruthy, parseExpression, type Expression, type ExpressionPath } from './expression.js'; -import { findTemplate, parseTemplate, templatedTexts, templatePaths, templateWithPaths, type Template } from './template.js'; -import type { EndOutcome, InlineWorkflow, OutputSchema, WorkflowDefinition, WorkflowDiagnostic, WorkflowStep } from './types.js'; +import { fieldsText, parseExpression, type ExpressionPath } from './expression.js'; +import { + findTemplate, parseTemplate, stepExpressions, templatedTexts, templateExpressions, templateWithPaths, type StepExpression, type Template, +} from './template.js'; +import type { + EndOutcome, ForEach, InlineWorkflow, OutputSchema, StepLoop, WorkflowDefinition, WorkflowDiagnostic, WorkflowLoop, WorkflowStep, +} from './types.js'; import { STEP_ACTIONS } from './types.js'; - -const HEADER_KEYS = new Set(['schemaVersion', 'name', 'description', 'inputs', 'ends', 'outputs', 'steps']); +import { + isYamlNode, locate, position, scalarString, unavailableField, undeclaredField, undeclaredOutput, + type Child, type GraphNode, type ParseContext, type WorkflowFiles, +} from './parse-context.js'; +import { + conditionDiagnostics, endDiagnostics, expressionSite, readDiagnostics, readScope, splitReads, type ConditionNode, type Ending, type ReadSite, +} from './reads.js'; +import { loopProblems } from './loop.js'; + +const HEADER_KEYS = new Set(['schemaVersion', 'name', 'description', 'inputs', 'ends', 'outputs', 'loop', 'steps']); const ACTION_KEYS = Object.keys(STEP_ACTIONS); -const STEP_KEYS = new Set(['id', 'needs', 'if', ...ACTION_KEYS, 'env', 'with', 'agent', 'output', 'end']); +const STEP_KEYS = new Set(['id', 'needs', 'if', ...ACTION_KEYS, 'env', 'with', 'agent', 'output', 'end', 'loop', 'break', 'for_each']); +const STEP_LOOP_KEYS = new Set(['back_to', 'while', 'max_iterations']); +const FOR_EACH_KEYS = new Set(['items', 'max_parallel']); +const WORKFLOW_LOOP_KEYS = new Set(['while', 'max_iterations']); /** Keys a step without an action may carry. */ const END_ONLY_KEYS = new Set(['id', 'needs', 'if', 'end']); /** The actions as a sentence: `command, prompt, prompt_file, or workflow`. */ @@ -18,6 +29,15 @@ const ACTION_LIST = `${ACTION_KEYS.slice(0, -1).join(', ')}, or ${ACTION_KEYS.at const TYPE_NAMES = new Set(['string', 'number', 'boolean']); const TOKEN = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; const ENV_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/; +/** + * How pull checks what each step expression other than `if` reads; an `if` is also checked for being always false. + * `loop.while` and `break` are evaluated after their step succeeds, so they may read it. + */ +const EXPRESSION_SITES: Record, { code: WorkflowDiagnostic['code'] } | undefined> = { + 'loop.while': { code: 'bad_loop' }, + break: { code: 'bad_loop' }, + for_each: undefined, +}; export function formatDiagnostic(diagnostic: WorkflowDiagnostic): string { return `${diagnostic.file}:${diagnostic.line}: field "${diagnostic.field}": ${diagnostic.message}`; @@ -35,38 +55,12 @@ function dottedName(file: string, lineCounter: LineCounter, node: Node | null | return { ...locate(file, lineCounter, node, field, `${name} cannot contain "."; a path reads "." as a separator`), code: 'dotted_name' }; } -function locate(file: string, lineCounter: LineCounter, node: Node | null | undefined, field: string, message: string): WorkflowDiagnostic { - let line = 1; - let column = 1; - const offset = node?.range?.[0]; - if (typeof offset === 'number') { - const pos = lineCounter.linePos(offset); - line = pos.line; - column = pos.col; - } - return { file, line, column, field, message }; -} - -function scalarString(node: Node | null | undefined): string | undefined { - if (!isScalar(node) || typeof node.value !== 'string') return undefined; - return node.value; -} - /** Where a map's `key` is written: its value, or the key when the value is empty. */ function valueNode(map: unknown, key: string): Node | undefined { const pair = isMap(map) ? map.items.find((item) => isScalar(item.key) && String(item.key.value) === key) : undefined; return isYamlNode(pair?.value) ? pair.value : isYamlNode(pair?.key) ? pair.key : undefined; } -/** What the checks of one workflow file share: where a diagnostic points, where it goes, and what a sync knows of the other files. */ -interface ParseContext { - file: string; - lineCounter: LineCounter; - diagnostics: WorkflowDiagnostic[]; - promptFileLocations: Record; - files: WorkflowFiles | undefined; -} - function parseOutput(node: Node, field: string, ctx: ParseContext): OutputSchema | undefined { const { file, lineCounter, diagnostics } = ctx; if (isScalar(node)) { @@ -124,14 +118,6 @@ function parseOutput(node: Node, field: string, ctx: ParseContext): OutputSchema return undefined; } -/** What a sync knows about the other workflow files, so a step that runs one can be checked against it. */ -export interface WorkflowFiles { - /** The version of `name` a step would run after this sync, or null when it has none. */ - runnable(name: string): WorkflowDefinition | null; - /** The workflows running `name` from this file passes through until it runs this file again, such as ship -> check -> ship. */ - cycle(name: string): string[] | undefined; -} - /** * Parse one workflow file. A failure is a list of diagnostics; it does not throw, * so one bad file cannot stop the rest of a sync. Without `files`, the file is checked on its own. @@ -207,25 +193,31 @@ export function parseWorkflow(file: string, content: string, files?: WorkflowFil } // A malformed inputs header is already reported; checking reads against it would only add noise. - const { workflow: { ends, outputs, steps } } = parseBody(root, '', ctx, inputsPair && !inputs ? undefined : inputs ?? {}); + const { workflow: { ends, outputs, loop, steps } } = parseBody(root, '', 'workflow', ctx, inputsPair && !inputs ? undefined : inputs ?? {}, false); if (diagnostics.length > 0 || !name || !description) return { ok: false, diagnostics }; return { ok: true, - definition: { schemaVersion: 'v1alpha1', name, description, ...(inputs ? { inputs } : {}), ...(ends ? { ends } : {}), ...(outputs ? { outputs } : {}), steps }, + definition: { + schemaVersion: 'v1alpha1', name, description, ...(inputs ? { inputs } : {}), ...(ends ? { ends } : {}), ...(outputs ? { outputs } : {}), + ...(loop ? { loop } : {}), steps, + }, promptFileLocations, }; } /** - * The `ends`, `outputs`, and `steps` of a workflow file, or of an inline workflow whose fields start with `at`. + * The `ends`, `outputs`, `loop`, and `steps` of a workflow file, or of an inline workflow whose fields start with `at`. * An inline workflow returns, in `outer`, what its steps read of enclosing steps, for the enclosing workflow to check. + * `itemBound` is whether the workflow runs once per item of a `for_each` step, directly or inside one, so its steps may read `item`. */ function parseBody( root: YAMLMap.Parsed, at: string, + label: string, ctx: ParseContext, inputs: Record | undefined, + itemBound: boolean, ): { workflow: InlineWorkflow; outer: ReadSite[] } { const { file, lineCounter, diagnostics, promptFileLocations, files } = ctx; const endsPair = root.items.find((pair) => scalarString(pair.key) === 'ends'); @@ -235,6 +227,15 @@ function parseBody( const outputEntries = outputsPair ? parseOutputs(outputsPair.value ?? outputsPair.key, at, ctx) : undefined; const outputs = outputEntries && Object.fromEntries(outputEntries.map((entry) => [entry.name, entry.source])); + const loopPair = root.items.find((pair) => scalarString(pair.key) === 'loop'); + const loop = loopPair ? parseLoop(loopPair.value ?? loopPair.key, `${at}loop`, label, ctx, false) : undefined; + const loopWhile = loop?.while === undefined ? undefined : parseExpression(loop.while); + const loopSite: ReadSite | undefined = loopWhile?.ok ? { + ...expressionSite('', `${at}loop.while`, position(lineCounter, valueNode(loopPair?.value, 'while')), 'loop.while', [loopWhile.expression]), + workflow: label, + code: 'bad_loop', + } : undefined; + const stepsPair = root.items.find((pair) => scalarString(pair.key) === 'steps'); const stepsNode = stepsPair?.value; const steps: WorkflowStep[] = []; @@ -248,6 +249,7 @@ function parseBody( const reads: ReadSite[] = []; const endings: Ending[] = []; const references: Array<{ id: string; workflow: string }> = []; + const placed = new Map(); stepsNode.items.forEach((item, index) => { const field = `${at}steps[${index}]`; if (!isMap(item)) { @@ -268,6 +270,7 @@ function parseBody( diagnostics.push(locate(file, lineCounter, idPair?.value ?? item, `${field}.id`, `step id "${id}" is duplicated`)); } else { seen.add(id); + placed.set(id, { field, item }); if (id.includes('.')) diagnostics.push(dottedName(file, lineCounter, idPair?.value, `${field}.id`, `step id "${id}"`)); } const label = id ? `step "${id}"` : field; @@ -298,8 +301,6 @@ function parseBody( diagnostics.push(locate(file, lineCounter, at, `${field}.if`, `${label} if must be an expression`)); } else if (!parsed.ok) { diagnostics.push(locate(file, lineCounter, at, `${field}.if`, `${label} if does not parse at column ${parsed.column}: ${parsed.message}`)); - } else if (inGraph) { - conditions.push({ id, field: `${field}.if`, node: at as Node, expression: parsed.expression }); } } const commandPair = item.items.find((pair) => scalarString(pair.key) === 'command'); @@ -310,8 +311,11 @@ function parseBody( const promptFile = scalarString(promptFilePair?.value); const workflowPair = item.items.find((pair) => scalarString(pair.key) === 'workflow'); const workflow = scalarString(workflowPair?.value)?.trim(); + const forEachPair = item.items.find((pair) => scalarString(pair.key) === 'for_each'); + // The step's templates and the inline workflow it runs render once per item, so they read item as a template. + const bindsItem = itemBound || forEachPair !== undefined; const inline = isMap(workflowPair?.value) - ? parseInline(workflowPair.value, `${field}.workflow`, label, ctx, inputs) + ? parseInline(workflowPair.value, `${field}.workflow`, label, ctx, inputs, bindsItem) : undefined; const actions = ACTION_KEYS.map((key) => item.items.find((pair) => scalarString(pair.key) === key)).filter(Boolean); const endPair = item.items.find((pair) => scalarString(pair.key) === 'end'); @@ -337,7 +341,7 @@ function parseBody( if (commandPair && (command === undefined || command.trim() === '')) { diagnostics.push(locate(file, lineCounter, commandPair?.value ?? commandPair?.key ?? item, `${field}.command`, 'command must be a non-empty string')); } - const commandTemplate = command === undefined ? undefined : findTemplate(command); + const commandTemplate = command === undefined ? undefined : findTemplate(command, bindsItem); if (commandTemplate !== undefined) { diagnostics.push(locate(file, lineCounter, commandPair?.value as Node, `${field}.command`, `${label} command contains ${commandTemplate}; command strings are not templated; pass the value through env:`)); @@ -394,7 +398,7 @@ function parseBody( return { what, field: `${field}.${what}`, node: node as Node, text }; }); for (const template of templates) { - const parsed = parseTemplate(template.text); + const parsed = parseTemplate(template.text, bindsItem); if (!parsed.ok) { for (const problem of parsed.errors) { diagnostics.push({ @@ -410,7 +414,10 @@ function parseBody( } } if (inGraph && parsed.ok) { - reads.push({ id, field: template.field, at: position(lineCounter, template.node), what: template.what, paths: templatePaths(parsed.template), code: 'bad_template' }); + reads.push({ + ...expressionSite(id, template.field, position(lineCounter, template.node), template.what, templateExpressions(parsed.template)), + ...(forEachPair ? { item: true } : {}), code: 'bad_template', + }); } } // What the inline workflow reads of this workflow's steps must be steps this step needs, directly or not. @@ -421,6 +428,35 @@ function parseBody( if (envPair && !commandPair && actions.length > 0) { diagnostics.push(locate(file, lineCounter, envPair.key as Node, `${field}.env`, `${label} env is only allowed on a command step`)); } + const loopPair = item.items.find((pair) => scalarString(pair.key) === 'loop'); + const loop = loopPair ? parseLoop(loopPair.value ?? loopPair.key, `${field}.loop`, label, ctx, true) : undefined; + const breakPair = item.items.find((pair) => scalarString(pair.key) === 'break'); + const breakSource = breakPair ? conditionSource(breakPair.value) : undefined; + if (breakPair) { + const parsed = breakSource === undefined ? undefined : parseExpression(breakSource); + const node = (breakPair.value ?? breakPair.key ?? item) as Node; + const reject = (message: string): void => { + diagnostics.push({ ...locate(file, lineCounter, node, `${field}.break`, `${label} break ${message}`), code: 'bad_loop' }); + }; + if (!parsed) reject('must be an expression'); + else if (!parsed.ok) reject(`does not parse at column ${parsed.column}: ${parsed.message}`); + } + const forEach = forEachPair ? parseForEach(forEachPair.value ?? forEachPair.key, `${field}.for_each`, label, ctx) : undefined; + if (inGraph) { + const nodes: Record = { + if: ifPair?.value, + 'loop.while': valueNode(loopPair?.value, 'while'), + break: breakPair?.value, + for_each: isMap(forEachPair?.value) ? valueNode(forEachPair.value, 'items') : forEachPair?.value, + }; + for (const { what, source } of stepExpressions({ if: condition, loop, break: breakSource, forEach })) { + const parsed = parseExpression(source); + if (!parsed.ok) continue; + const node = (nodes[what] ?? item) as Node; + if (what === 'if') conditions.push({ id, field: `${field}.if`, node, expression: parsed.expression }); + else reads.push(expressionSite(id, `${field}.${what}`, position(lineCounter, node), what, [parsed.expression], EXPRESSION_SITES[what])); + } + } let output: OutputSchema | undefined; const outputPair = item.items.find((pair) => scalarString(pair.key) === 'output'); if (outputPair && workflowPair) { @@ -445,6 +481,9 @@ function parseBody( ...(agent ? { agent } : {}), ...(output ? { output } : {}), ...(end ? { end } : {}), + ...(loop ? { loop } : {}), + ...(breakSource !== undefined ? { break: breakSource } : {}), + ...(forEach ? { forEach } : {}), }); } }); @@ -457,12 +496,27 @@ function parseBody( // An inline workflow's reads of steps that are not its own are checked by the workflow around it. const split = splitReads(reads, at === '' ? undefined : seen); outer = split.outer; - diagnostics.push(...readDiagnostics(file, split.local, readScope(needsById, steps, inputs, children))); + diagnostics.push(...readDiagnostics(file, loopSite ? [...split.local, loopSite] : split.local, { ...readScope(needsById, steps, inputs, children), item: itemBound })); + for (const problem of loopProblems(steps, stepReads(split.local), loop !== undefined)) { + const at = placed.get(problem.id); + const node = problem.key.split('.').reduce((map, key) => valueNode(map, key), at?.item); + diagnostics.push({ ...locate(file, lineCounter, (node ?? at?.item) as Node, `${at?.field}.${problem.key}`, problem.message), code: 'bad_loop' }); + } // A malformed ends header is already reported; checking steps against it would only add noise. if (!endsPair || ends) diagnostics.push(...endDiagnostics(ctx, ends, endsPair?.value, at, endings, graph)); diagnostics.push(...outputDiagnostics(ctx, outputEntries ?? [], at, seen, steps, children)); } - return { workflow: { ...(ends ? { ends } : {}), ...(outputs ? { outputs } : {}), steps }, outer }; + return { workflow: { ...(ends ? { ends } : {}), ...(outputs ? { outputs } : {}), ...(loop ? { loop } : {}), steps }, outer }; +} + +function stepReads(sites: ReadSite[]): Map> { + const reads = new Map>(); + for (const site of sites) { + const ids = reads.get(site.id) ?? new Set(); + for (const path of site.paths) if (path.root === 'steps') ids.add(path.step); + reads.set(site.id, ids); + } + return reads; } /** An inline `workflow: { steps, outputs?, ends? }`. It reads the enclosing inputs and steps, so it declares no inputs. */ @@ -472,36 +526,21 @@ function parseInline( label: string, ctx: ParseContext, inputs: Record | undefined, + itemBound: boolean, ): { workflow: InlineWorkflow; outer: ReadSite[] } { const { file, lineCounter, diagnostics } = ctx; for (const pair of node.items) { const key = scalarString(pair.key); - if (key === 'steps' || key === 'outputs' || key === 'ends') continue; + if (key === 'steps' || key === 'outputs' || key === 'ends' || key === 'loop') continue; if (!key || !HEADER_KEYS.has(key)) { diagnostics.push(unknownKey(file, lineCounter, pair.key ?? node, `${field}.${key ?? 'key'}`, key ?? 'key')); } else if (key === 'inputs') { diagnostics.push(locate(file, lineCounter, pair.key, `${field}.inputs`, `${label} runs an inline workflow, which declares no inputs; its steps read the enclosing workflow's inputs`)); } else { - diagnostics.push(locate(file, lineCounter, pair.key, `${field}.${key}`, `${label} runs an inline workflow, which carries only steps, outputs, and ends`)); + diagnostics.push(locate(file, lineCounter, pair.key, `${field}.${key}`, `${label} runs an inline workflow, which carries only steps, outputs, ends, and loop`)); } } - return parseBody(node, `${field}.`, ctx, inputs); -} - -/** - * Splits what steps read into reads of their own workflow, and reads of steps that are not in it. - * `ids` is undefined for a workflow file, whose steps can read nothing outside it. - */ -function splitReads(sites: ReadSite[], ids: Set | undefined): { local: ReadSite[]; outer: ReadSite[] } { - const inside = (path: ExpressionPath): boolean => !ids || path.root === 'inputs' || ids.has(path.step); - const endInside = ({ step }: { step: string }): boolean => !ids || ids.has(step); - return { - local: sites.map((site) => ({ ...site, paths: site.paths.filter(inside), ends: site.ends?.filter(endInside) })), - outer: sites.flatMap((site) => { - const paths = site.paths.filter((path) => !inside(path)); - return paths.length > 0 ? [{ ...site, paths, ends: site.ends?.filter((end) => !endInside(end)) }] : []; - }), - }; + return parseBody(node, `${field}.`, `${label} workflow`, ctx, inputs, itemBound); } /** A step that runs a workflow file by name. */ @@ -535,13 +574,6 @@ function referenceDiagnostics({ file, lineCounter }: ParseContext, files: Workfl ]; } -/** What a reader may read of the workflow a step runs, and how a message names that workflow. */ -interface Child { - label: string; - outputs?: Record; - ends?: Record; -} - /** * The workflow each step runs: an inline one as written, a file as a sync knows it. * A step whose reference is reported has none, so its reads are not checked. @@ -560,12 +592,77 @@ function childrenOf(steps: Array<{ id: string; workflow?: string | InlineWorkflo return children; } -/** Why `path` reads an output the workflow its step runs does not declare, or undefined. */ -function undeclaredOutput(children: Map, path: ExpressionPath): string | undefined { - if (path.root !== 'steps' || path.field !== 'outputs' || path.fields.length === 0) return undefined; - const child = children.get(path.step); - if (!child || Object.hasOwn(child.outputs ?? {}, path.fields[0]!)) return undefined; - return `reads steps.${path.step}.outputs.${path.fields.join('.')}, which is not an output ${child.label} declares`; +/** + * A step's `loop`, or, with `step` false, a workflow's, which has no `back_to`. Where `back_to` sits in the graph is + * checked once every step is known. + */ +function parseLoop(node: Node | null | undefined, field: string, label: string, ctx: ParseContext, step: true): StepLoop | undefined; +function parseLoop(node: Node | null | undefined, field: string, label: string, ctx: ParseContext, step: false): WorkflowLoop | undefined; +function parseLoop(node: Node | null | undefined, field: string, label: string, ctx: ParseContext, step: boolean): StepLoop | WorkflowLoop | undefined { + const { file, lineCounter, diagnostics } = ctx; + const reject = (at: Node | null | undefined, where: string, message: string): void => { + diagnostics.push({ ...locate(file, lineCounter, at, where, message), code: 'bad_loop' }); + }; + if (!isMap(node)) { + reject(node, field, `${label} loop must map ${step ? 'back_to, ' : ''}max_iterations, and optionally while`); + return undefined; + } + const before = diagnostics.length; + const keys = step ? STEP_LOOP_KEYS : WORKFLOW_LOOP_KEYS; + for (const pair of node.items) { + const keyNode = isYamlNode(pair.key) ? pair.key : undefined; + const key = scalarString(keyNode); + if (!step && key === 'back_to') reject(keyNode, `${field}.back_to`, `${label} loop has no back_to; each round runs every step of the workflow`); + else if (!key || !keys.has(key)) diagnostics.push(unknownKey(file, lineCounter, keyNode ?? node, `${field}.${key ?? 'key'}`, key ?? 'key')); + } + const at = (key: string): Node | undefined => valueNode(node, key); + const backTo = step ? scalarString(at('back_to'))?.trim() : undefined; + if (step && (!backTo || !TOKEN.test(backTo))) reject(at('back_to') ?? node, `${field}.back_to`, `${label} loop.back_to must name a step`); + const maxNode = at('max_iterations'); + const max: unknown = isScalar(maxNode) ? maxNode.value : undefined; + if (typeof max !== 'number' || !Number.isInteger(max) || max < 1) { + reject(at('max_iterations') ?? node, `${field}.max_iterations`, `${label} loop.max_iterations must be a whole number from 1`); + } + const whileNode = at('while'); + const condition = whileNode ? conditionSource(whileNode) : undefined; + if (whileNode) { + const parsed = condition === undefined ? undefined : parseExpression(condition); + if (!parsed) reject(whileNode, `${field}.while`, `${label} loop.while must be an expression`); + else if (!parsed.ok) reject(whileNode, `${field}.while`, `${label} loop.while does not parse at column ${parsed.column}: ${parsed.message}`); + } + if (diagnostics.length > before) return undefined; + const loop = { ...(condition !== undefined ? { while: condition } : {}), maxIterations: max as number }; + return backTo ? { backTo, ...loop } : loop; +} + +function parseForEach(node: Node | null | undefined, field: string, label: string, ctx: ParseContext): ForEach | undefined { + const { file, lineCounter, diagnostics } = ctx; + const reject = (at: Node | null | undefined, where: string, message: string): void => { + diagnostics.push({ ...locate(file, lineCounter, at, where, message), code: 'bad_loop' }); + }; + const map = isMap(node) ? node : undefined; + const before = diagnostics.length; + for (const pair of map?.items ?? []) { + const keyNode = isYamlNode(pair.key) ? pair.key : undefined; + const key = scalarString(keyNode); + if (!key || !FOR_EACH_KEYS.has(key)) diagnostics.push(unknownKey(file, lineCounter, keyNode ?? node, `${field}.${key ?? 'key'}`, key ?? 'key')); + } + const itemsNode = map ? valueNode(map, 'items') : node; + const items = conditionSource(itemsNode); + const parsed = items === undefined ? undefined : parseExpression(items); + if (!parsed) { + reject(itemsNode ?? node, map ? `${field}.items` : field, + map ? `${label} for_each.items must be an expression` : `${label} for_each must be an expression, or a map of items and max_parallel`); + } else if (!parsed.ok) { + reject(itemsNode, map ? `${field}.items` : field, `${label} for_each does not parse at column ${parsed.column}: ${parsed.message}`); + } + const maxNode = map ? valueNode(map, 'max_parallel') : undefined; + const max: unknown = isScalar(maxNode) ? maxNode.value : undefined; + if (maxNode && (typeof max !== 'number' || !Number.isInteger(max) || max < 1)) { + reject(maxNode, `${field}.max_parallel`, `${label} for_each.max_parallel must be a whole number from 1`); + } + if (diagnostics.length > before || items === undefined) return undefined; + return { items, ...(maxNode ? { maxParallel: max as number } : {}) }; } /** `env` maps variable names to text. A YAML number or boolean is read as its text. */ @@ -672,26 +769,23 @@ function outputDiagnostics( diagnostics.push(at(`reads steps.${path.step}, which is not a step in this workflow`)); continue; } + const step = steps.find((step) => step.id === path.step); + const unavailable = step && unavailableField(path, !!step.forEach); + if (unavailable) { + diagnostics.push(at(unavailable)); + continue; + } const undeclared = undeclaredOutput(children, path); if (undeclared) { diagnostics.push({ ...at(undeclared), code: 'undeclared_output' }); continue; } - const output = steps.find((step) => step.id === path.step)?.output; - if (path.field === 'outputs' && output && !declaresField(output, path.fields)) { - diagnostics.push(at(`reads steps.${path.step}.outputs.${path.fields.join('.')}, which is not in the output step "${path.step}" declares`)); - } + const field = undeclaredField(path, step?.output, !!step?.forEach); + if (field) diagnostics.push(at(field)); } return diagnostics; } -interface GraphNode { - id: string; - field: string; - needs: string[]; - node: Node; -} - /** Diagnostics for the `needs` graph: ids that name no step, and cycles reported at their first declared step. */ function graphDiagnostics({ file, lineCounter }: ParseContext, graph: GraphNode[], ids: Set): WorkflowDiagnostic[] { const diagnostics: WorkflowDiagnostic[] = []; @@ -738,203 +832,10 @@ function conditionSource(node: unknown): string | undefined { return typeof value === 'string' || typeof value === 'boolean' || typeof value === 'number' ? String(value) : undefined; } -interface ConditionNode { - id: string; - field: string; - node: Node; - expression: Expression; -} - -/** Whether `fields` names a value inside `schema`. Only object fields can be read further. */ -function declaresField(schema: OutputSchema, fields: string[]): boolean { - let current = schema; - for (const field of fields) { - if (current.type !== 'object' || !Object.hasOwn(current.fields, field)) return false; - current = current.fields[field]!; - } - return true; -} - -/** Reports an `if` that is always false, and queues what every other `if` reads for `readDiagnostics`. */ -function conditionDiagnostics({ file, lineCounter }: ParseContext, conditions: ConditionNode[], reads: ReadSite[]): WorkflowDiagnostic[] { - const diagnostics: WorkflowDiagnostic[] = []; - for (const condition of conditions) { - const paths = expressionPaths(condition.expression); - if (paths.length === 0 && !isTruthy(evaluateExpression(condition.expression, { steps: {} }))) { - diagnostics.push({ - ...locate(file, lineCounter, condition.node, condition.field, `step "${condition.id}" if is always false, so the step never runs`), - code: 'false_if', - }); - continue; - } - reads.push({ id: condition.id, field: condition.field, at: position(lineCounter, condition.node), what: 'if', paths, ends: endComparisons(condition.expression) }); - } - return diagnostics; -} - -function position(lineCounter: LineCounter, node: Node | null | undefined): { line: number; column: number } { - const { line, column } = locate('', lineCounter, node, '', ''); - return { line, column }; -} - -/** Where a step reads paths: its `if`, its prompt, or one `env` value. */ -interface ReadSite { - /** The step whose ancestors the site may read: the reader, or the step that runs the inline workflow the reader is in. */ - id: string; - /** The step that reads, when it is inside an inline workflow that `id` runs. */ - reader?: string; - field: string; - at: { line: number; column: number }; - /** `if`, `prompt`, `prompt_file` or `env.`. */ - what: string; - paths: ExpressionPath[]; - /** The names an `if` compares a step's end with. */ - ends?: Array<{ step: string; end: string }>; - code?: WorkflowDiagnostic['code']; -} - -/** What a workflow lets a step read: its ancestors through `needs`, the output each step declares, and the declared inputs. */ -interface ReadScope { - ancestors: (id: string) => Set; - outputs: Map; - /** Undefined when the inputs header is malformed, so `inputs.…` reads are not checked. */ - inputs?: OutputSchema; - /** The workflow each workflow step runs: an inline one, or a file when a sync knows it. */ - children: Map; -} - -function readScope( - needs: Map, - steps: WorkflowStep[], - inputs: Record | undefined, - children: Map, -): ReadScope { - return { - children, - ancestors: (id) => { - const found = new Set(); - const pending = [...(needs.get(id) ?? [])]; - for (let need = pending.pop(); need !== undefined; need = pending.pop()) { - if (found.has(need)) continue; - found.add(need); - pending.push(...(needs.get(need) ?? [])); - } - return found; - }, - outputs: new Map(steps.map((step) => [step.id, step.output])), - ...(inputs ? { inputs: { type: 'object', fields: inputs } } : {}), - }; -} - -/** - * Diagnostics for what each `if` and template reads: only declared inputs, only steps it needs, - * directly or through their needs, and only output fields that a step with `output` declares. - */ -function readDiagnostics(file: string, sites: ReadSite[], scope: ReadScope): WorkflowDiagnostic[] { - const diagnostics: WorkflowDiagnostic[] = []; - for (const site of sites) { - const at = (message: string): WorkflowDiagnostic => ({ - file, ...site.at, field: site.field, message, ...(site.code ? { code: site.code } : {}), - }); - const readable = scope.ancestors(site.id); - const reader = site.reader ?? site.id; - const reported = new Set(); - for (const path of site.paths) { - if (path.root === 'inputs') { - if (scope.inputs && !declaresField(scope.inputs, path.fields)) { - diagnostics.push(at(`step "${reader}" ${site.what} reads inputs.${path.fields.join('.')}, which is not an input the workflow declares`)); - } - continue; - } - if (!readable.has(path.step)) { - if (!reported.has(path.step)) { - diagnostics.push(at(site.reader === undefined - ? `step "${reader}" ${site.what} reads steps.${path.step}, which is not a step it needs, directly or through its needs` - : `step "${reader}" ${site.what} reads steps.${path.step}, which is not a step it needs, or one the step that runs its workflow needs, directly or through their needs`)); - } - reported.add(path.step); - continue; - } - const undeclared = undeclaredOutput(scope.children, path); - if (undeclared) { - diagnostics.push({ ...at(`step "${reader}" ${site.what} ${undeclared}`), code: 'undeclared_output' }); - continue; - } - const output = scope.outputs.get(path.step); - if (path.field === 'outputs' && output && !declaresField(output, path.fields)) { - diagnostics.push(at(`step "${reader}" ${site.what} reads steps.${path.step}.outputs.${path.fields.join('.')}, which is not in the output step "${path.step}" declares`)); - } - } - for (const { step, end } of site.ends ?? []) { - const child = readable.has(step) ? scope.children.get(step) : undefined; - if (!child || Object.hasOwn(child.ends ?? {}, end)) continue; - diagnostics.push({ - ...at(`step "${reader}" ${site.what} compares steps.${step}.end with '${end}', which is not an end ${child.label} declares`), - code: 'undeclared_end', - }); - } - } - return diagnostics; -} - -/** The names an expression compares a step's end with, as in `steps.check.end == 'ready'`. */ -function endComparisons(expression: Expression): Array<{ step: string; end: string }> { - if (expression.kind === 'not') return endComparisons(expression.operand); - if (expression.kind !== 'binary') return []; - if (expression.operator === '&&' || expression.operator === '||') { - return [...endComparisons(expression.left), ...endComparisons(expression.right)]; - } - const [read, value] = expression.left.kind === 'path' ? [expression.left, expression.right] : [expression.right, expression.left]; - if (read.kind !== 'path' || read.path.root !== 'steps' || read.path.field !== 'end') return []; - return value.kind === 'literal' && typeof value.value === 'string' ? [{ step: read.path.step, end: value.value }] : []; -} - -interface Ending { - id: string; - field: string; - end: string; - node: Node; - /** The step has no action and only ends the workflow. */ - only: boolean; -} - -/** Diagnostics that keep `end` on steps consistent with the declared `ends`. */ -function endDiagnostics( - { file, lineCounter }: ParseContext, - ends: Record | undefined, - endsNode: unknown, - prefix: string, - endings: Ending[], - graph: GraphNode[], -): WorkflowDiagnostic[] { - const diagnostics: WorkflowDiagnostic[] = []; - const endOnly = new Set(endings.filter((step) => step.only).map((step) => step.id)); - for (const step of graph) { - for (const need of step.needs.filter((id) => endOnly.has(id))) { - diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" needs "${need}", which only ends the workflow, so "${step.id}" can never run`)); - } - } - for (const step of endings) { - if (!ends) { - diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" ends with "${step.end}", but the workflow declares no ends`)); - } else if (!Object.hasOwn(ends, step.end)) { - diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" ends with "${step.end}", which is not declared in ends`)); - } - } - if (isMap(endsNode)) { - for (const pair of endsNode.items) { - const name = scalarString(isYamlNode(pair.key) ? pair.key : undefined); - if (!name || endings.some((step) => step.end === name)) continue; - diagnostics.push(locate(file, lineCounter, isYamlNode(pair.key) ? pair.key : endsNode, `${prefix}ends.${name}`, `end "${name}" is declared in ends, but no step ends with it`)); - } - } - return diagnostics; -} - /** `{{ with. }}` reads a value its step's own `with` declares, and only from the step's prompt or prompt_file text. */ function withReadProblems(label: string, what: string, template: Template, declared: Record | undefined): string[] { return templateWithPaths(template).flatMap((path) => { - const read = `${label} ${what} reads with.${path.fields.join('.')}`; + const read = `${label} ${what} reads with${fieldsText(path.fields)}`; if (what !== 'prompt' && what !== 'prompt_file') return [`${read}; only a prompt reads with`]; if (!declared) return [`${read}, but the step has no with`]; return Object.hasOwn(declared, path.fields[0] ?? '') ? [] : [`${read}, which is not a name its with declares`]; @@ -954,29 +855,35 @@ export function promptFileDiagnostics( files?: WorkflowFiles, ): WorkflowDiagnostic[] { const levels: WorkflowStep[][] = [definition.steps]; + const runners: WorkflowStep[] = []; for (const index of trail.slice(0, -1)) { - const workflow = levels[levels.length - 1]?.[index]?.workflow; - if (typeof workflow !== 'object') return []; - levels.push(workflow.steps); + const runner = levels[levels.length - 1]?.[index]; + if (typeof runner?.workflow !== 'object') return []; + runners.push(runner); + levels.push(runner.workflow.steps); } const step = levels[levels.length - 1]?.[trail[trail.length - 1] ?? -1]; if (!step) return []; const field = `${stepField(trail)}.prompt_file`; - const parsed = parseTemplate(text); + const bindsItem = step.forEach !== undefined || runners.some((runner) => runner.forEach !== undefined); + const parsed = parseTemplate(text, bindsItem); if (!parsed.ok) { - return parsed.errors.map((problem) => ({ + return parsed.errors.map((problem): WorkflowDiagnostic => ({ file, ...at, field, message: `step "${step.id}" prompt_file template ${problem.source}: ${problem.message}`, code: 'bad_template', })); } const diagnostics: WorkflowDiagnostic[] = withReadProblems(`step "${step.id}"`, 'prompt_file', parsed.template, step.with) .map((message): WorkflowDiagnostic => ({ file, ...at, field, message, code: 'bad_template' })); - let sites: ReadSite[] = [{ id: step.id, field, at, what: 'prompt_file', paths: templatePaths(parsed.template), code: 'bad_template' }]; + let sites: ReadSite[] = [{ + ...expressionSite(step.id, field, at, 'prompt_file', templateExpressions(parsed.template)), ...(step.forEach ? { item: true } : {}), code: 'bad_template', + }]; // Check the reads in the step's own workflow, then hand what it reads of enclosing steps out, one workflow at a time. for (let level = levels.length - 1; level >= 0 && sites.length > 0; level -= 1) { const steps = levels[level] ?? []; const split = splitReads(sites, level === 0 ? undefined : new Set(steps.map((item) => item.id))); const needs = new Map(steps.map((item) => [item.id, item.needs ?? []])); - diagnostics.push(...readDiagnostics(file, split.local, readScope(needs, steps, definition.inputs ?? {}, childrenOf(steps, files)))); + const item = runners.slice(0, level).some((runner) => runner.forEach !== undefined); + diagnostics.push(...readDiagnostics(file, split.local, { ...readScope(needs, steps, definition.inputs ?? {}, childrenOf(steps, files)), item })); const enclosing = levels[level - 1]?.[trail[level - 1] ?? -1]; sites = enclosing ? split.outer.map((site) => ({ ...site, id: enclosing.id, reader: site.reader ?? site.id })) : []; } diff --git a/src/workflow/reads.ts b/src/workflow/reads.ts new file mode 100644 index 000000000..363ddab24 --- /dev/null +++ b/src/workflow/reads.ts @@ -0,0 +1,261 @@ +import { isMap, type Node } from 'yaml'; +import { + evaluateExpression, expressionPaths, fieldsText, isTruthy, schemaAt, stepFieldSchema, typeProblems, type DeclaredType, type Expression, type ExpressionPath, +} from './expression.js'; +import { + isYamlNode, locate, position, scalarString, unavailableField, undeclaredField, undeclaredOutput, type Child, type GraphNode, type ParseContext, +} from './parse-context.js'; +import { ancestors, stepLoops } from './loop.js'; +import type { EndOutcome, OutputSchema, WorkflowDiagnostic, WorkflowStep } from './types.js'; + +export interface ConditionNode { + id: string; + field: string; + node: Node; + expression: Expression; +} + +/** Reports an `if` that is always false, and queues what every other `if` reads for `readDiagnostics`. */ +export function conditionDiagnostics({ file, lineCounter }: ParseContext, conditions: ConditionNode[], reads: ReadSite[]): WorkflowDiagnostic[] { + const diagnostics: WorkflowDiagnostic[] = []; + for (const condition of conditions) { + const paths = expressionPaths(condition.expression); + if (paths.length === 0 && !isTruthy(evaluateExpression(condition.expression, { steps: {} }))) { + diagnostics.push({ + ...locate(file, lineCounter, condition.node, condition.field, `step "${condition.id}" if is always false, so the step never runs`), + code: 'false_if', + }); + continue; + } + reads.push(expressionSite(condition.id, condition.field, position(lineCounter, condition.node), 'if', [condition.expression])); + } + return diagnostics; +} + +/** What an expression reads, or the expressions of one template, all checked as one `if` is. */ +export function expressionSite( + id: string, + field: string, + at: { line: number; column: number }, + what: string, + expressions: Expression[], + afterRun?: { code: WorkflowDiagnostic['code'] }, +): ReadSite { + return { + id, field, at, what, paths: expressions.flatMap(expressionPaths), ends: expressions.flatMap(endComparisons), expressions, + ...(afterRun ? { afterRun: true, code: afterRun.code } : {}), + }; +} + +/** Where a step reads paths, in one of the fields `what` names. */ +export interface ReadSite { + /** The step whose ancestors the site may read: the reader, or the step that runs the inline workflow the reader is in. */ + id: string; + /** The step that reads, when it is inside an inline workflow that `id` runs. */ + reader?: string; + field: string; + at: { line: number; column: number }; + /** `if`, `loop.while`, `break`, `for_each`, `prompt`, `prompt_file`, `with.` or `env.`. */ + what: string; + paths: ExpressionPath[]; + /** The names an `if` compares a step's end with. */ + ends?: Array<{ step: string; end: string }>; + /** The `if` itself, or each expression of a template, whose comparisons and lists are checked against the types its paths declare. */ + expressions: Expression[]; + /** The site is evaluated after its step succeeds, so it may read that step too. */ + afterRun?: boolean; + /** + * For a workflow's own `loop.while`, how messages name the workflow. It is evaluated after a round, so it may read + * every step of the workflow, and `id` names no step. + */ + workflow?: string; + /** A template of a `for_each` step, rendered once per item, so it may read `item`. */ + item?: boolean; + code?: WorkflowDiagnostic['code']; +} + +/** + * What a workflow lets a step read: its ancestors through `needs`, the steps after it in its own loop's range in their + * previous pass, the output each step declares, and the declared inputs. + */ +export interface ReadScope { + ids: Set; + ancestors: (id: string) => Set; + previous: (id: string) => Set; + outputs: Map; + /** The steps with `for_each`, which have results in place of outputs. */ + forEach: Set; + /** Undefined when the inputs header is malformed, so `inputs.…` reads are not checked. */ + inputs?: OutputSchema; + /** The workflow each workflow step runs: an inline one, or a file when a sync knows it. */ + children: Map; + /** The workflow runs once per item of a `for_each` step, so every site in it may read `item`. */ + item?: boolean; +} + +export function readScope( + needs: Map, + steps: WorkflowStep[], + inputs: Record | undefined, + children: Map, +): ReadScope { + return { + children, + ids: new Set([...needs.keys(), ...steps.map((step) => step.id)]), + ancestors: (id) => ancestors(needs, id), + previous: stepLoops(steps).previous, + outputs: new Map(steps.map((step) => [step.id, step.output])), + forEach: new Set(steps.filter((step) => step.forEach).map((step) => step.id)), + ...(inputs ? { inputs: { type: 'object', fields: inputs } } : {}), + }; +} + +/** + * Diagnostics for what each `if` and template reads: only declared inputs, only steps it needs, + * directly or through their needs, and only output fields that a step with `output` declares. + */ +export function readDiagnostics(file: string, sites: ReadSite[], scope: ReadScope): WorkflowDiagnostic[] { + const diagnostics: WorkflowDiagnostic[] = []; + for (const site of sites) { + const at = (message: string): WorkflowDiagnostic => ({ + file, ...site.at, field: site.field, message, ...(site.code ? { code: site.code } : {}), + }); + const readable = site.workflow !== undefined ? scope.ids : new Set([ + ...scope.ancestors(site.id), ...scope.previous(site.id), ...(site.afterRun && site.reader === undefined ? [site.id] : []), + ]); + const subject = site.workflow ?? `step "${site.reader ?? site.id}"`; + const reported = new Set(); + let itemReported = false; + for (const path of site.paths) { + if (path.root === 'inputs') { + if (scope.inputs && !schemaAt(scope.inputs, path.fields)) { + diagnostics.push(at(`${subject} ${site.what} reads inputs${fieldsText(path.fields)}, which is not an input the workflow declares`)); + } + continue; + } + if (path.root === 'item') { + if (!site.item && !scope.item && !itemReported) { + diagnostics.push(at(`${subject} ${site.what} reads item, which is bound only in the templates of a for_each step and in the workflow each item runs`)); + } + itemReported = true; + continue; + } + if (!readable.has(path.step)) { + if (!reported.has(path.step)) { + diagnostics.push(at(site.workflow !== undefined + ? `${subject} ${site.what} reads steps.${path.step}, which is not one of its steps` + : site.reader === undefined + ? `${subject} ${site.what} reads steps.${path.step}, which is not a step it needs, directly or through its needs` + : `${subject} ${site.what} reads steps.${path.step}, which is not a step it needs, or one the step that runs its workflow needs, directly or through their needs`)); + } + reported.add(path.step); + continue; + } + // A step pull rejected has its own diagnostic already. + const unavailable = scope.outputs.has(path.step) && unavailableField(path, scope.forEach.has(path.step)); + if (unavailable) { + diagnostics.push(at(`${subject} ${site.what} ${unavailable}`)); + continue; + } + const undeclared = undeclaredOutput(scope.children, path); + if (undeclared) { + diagnostics.push({ ...at(`${subject} ${site.what} ${undeclared}`), code: 'undeclared_output' }); + continue; + } + const field = undeclaredField(path, scope.outputs.get(path.step), scope.forEach.has(path.step)); + if (field) diagnostics.push(at(`${subject} ${site.what} ${field}`)); + } + // A path this site does not list reads an enclosing workflow, which checks its type. + const schemaOf = (path: ExpressionPath): DeclaredType | undefined => (site.paths.includes(path) ? declaredSchema(scope, readable, path) : undefined); + for (const problem of site.expressions.flatMap((expression) => typeProblems(expression, schemaOf))) { + diagnostics.push(at(`${subject} ${site.what} ${problem.message}`)); + } + for (const { step, end } of site.ends ?? []) { + const child = readable.has(step) ? scope.children.get(step) : undefined; + if (!child || Object.hasOwn(child.ends ?? {}, end)) continue; + diagnostics.push({ + ...at(`${subject} ${site.what} compares steps.${step}.end with '${end}', which is not an end ${child.label} declares`), + code: 'undeclared_end', + }); + } + } + return diagnostics; +} + +function declaredSchema(scope: ReadScope, readable: Set, path: ExpressionPath): DeclaredType | undefined { + if (path.root === 'inputs') return scope.inputs && schemaAt(scope.inputs, path.fields); + if (path.root === 'item' || !readable.has(path.step)) return undefined; + const schema = stepFieldSchema(path.field, scope.outputs.get(path.step), scope.forEach.has(path.step)); + return schema && schemaAt(schema, path.fields); +} + +/** + * Splits what steps read into reads of their own workflow, and reads of steps that are not in it. + * `ids` is undefined for a workflow file, whose steps can read nothing outside it. + */ +export function splitReads(sites: ReadSite[], ids: Set | undefined): { local: ReadSite[]; outer: ReadSite[] } { + const inside = (path: ExpressionPath): boolean => !ids || path.root !== 'steps' || ids.has(path.step); + const endInside = ({ step }: { step: string }): boolean => !ids || ids.has(step); + return { + local: sites.map((site) => ({ ...site, paths: site.paths.filter(inside), ends: site.ends?.filter(endInside) })), + outer: sites.flatMap((site) => { + const paths = site.paths.filter((path) => !inside(path)); + return paths.length > 0 ? [{ ...site, paths, ends: site.ends?.filter((end) => !endInside(end)) }] : []; + }), + }; +} + +/** The names an expression compares a step's end with, as in `steps.check.end == 'ready'`. */ +function endComparisons(expression: Expression): Array<{ step: string; end: string }> { + if (expression.kind === 'not') return endComparisons(expression.operand); + if (expression.kind === 'call') return [...endComparisons(expression.list), ...(expression.name === 'length' ? [] : endComparisons(expression.predicate))]; + if (expression.kind !== 'binary') return []; + if (expression.operator === '&&' || expression.operator === '||') { + return [...endComparisons(expression.left), ...endComparisons(expression.right)]; + } + const [read, value] = expression.left.kind === 'path' ? [expression.left, expression.right] : [expression.right, expression.left]; + if (read.kind !== 'path' || read.path.root !== 'steps' || read.path.field !== 'end') return []; + return value.kind === 'literal' && typeof value.value === 'string' ? [{ step: read.path.step, end: value.value }] : []; +} + +export interface Ending { + id: string; + field: string; + end: string; + node: Node; + /** The step has no action and only ends the workflow. */ + only: boolean; +} + +/** Diagnostics that keep `end` on steps consistent with the declared `ends`. */ +export function endDiagnostics( + { file, lineCounter }: ParseContext, + ends: Record | undefined, + endsNode: unknown, + prefix: string, + endings: Ending[], + graph: GraphNode[], +): WorkflowDiagnostic[] { + const diagnostics: WorkflowDiagnostic[] = []; + const endOnly = new Set(endings.filter((step) => step.only).map((step) => step.id)); + for (const step of graph) { + for (const need of step.needs.filter((id) => endOnly.has(id))) { + diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" needs "${need}", which only ends the workflow, so "${step.id}" can never run`)); + } + } + for (const step of endings) { + if (!ends) { + diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" ends with "${step.end}", but the workflow declares no ends`)); + } else if (!Object.hasOwn(ends, step.end)) { + diagnostics.push(locate(file, lineCounter, step.node, step.field, `step "${step.id}" ends with "${step.end}", which is not declared in ends`)); + } + } + if (isMap(endsNode)) { + for (const pair of endsNode.items) { + const name = scalarString(isYamlNode(pair.key) ? pair.key : undefined); + if (!name || endings.some((step) => step.end === name)) continue; + diagnostics.push(locate(file, lineCounter, isYamlNode(pair.key) ? pair.key : endsNode, `${prefix}ends.${name}`, `end "${name}" is declared in ends, but no step ends with it`)); + } + } + return diagnostics; +} diff --git a/src/workflow/records.ts b/src/workflow/records.ts index 994cbf2bd..34909151a 100644 --- a/src/workflow/records.ts +++ b/src/workflow/records.ts @@ -17,8 +17,32 @@ import path from 'node:path'; import { loadTeamConfig } from '../config.js'; import type { LocalConfig } from '../types.js'; import { getDataHome, isUsageReportEnabled } from '../types.js'; -import { locateStep } from './engine.js'; -import type { DriverIdentity, RunState, RunStatus, StepOutcome, TokenUsage } from './types.js'; +import { log } from '../utils/logger.js'; +import { isLocalWorkflowTool } from './local-tool-adapter.js'; +import { stepDefinitions } from './scope.js'; +import { settled, VISIBLE_OUTCOMES, type StepPass, type StepState, type VisibleOutcome } from './step-state.js'; +import type { DriverIdentity, RunState, RunStatus, TokenUsage, WorkflowFailure } from './types.js'; + +/** + * The failure codes the engine writes, then those the local driver sets. A code a model writes into its result, or + * another driver reports, can be any text, so a record carries only these. + */ +const RECORDED_FAILURES = [ + 'loop_exhausted', 'invalid_output', 'invalid_template', 'invalid_inputs', 'invalid_items', 'item_cycle', + 'tool_failed', 'tool_unavailable', 'agent_unavailable', 'read_only_dirty', 'tool_unconfigured', 'tool_unsupported', +] as const; +type RecordedFailure = (typeof RECORDED_FAILURES)[number]; + +function isRecordedFailure(failure: WorkflowFailure | undefined): failure is WorkflowFailure & { code: RecordedFailure } { + return failure !== undefined && (RECORDED_FAILURES as readonly string[]).includes(failure.code); +} + +/** + * A model name as a driver reports it, such as `claude-sonnet-4.6` or `anthropic/claude-sonnet-4.6@2026`. A report is + * free text, so a record carries a model only when it is shaped like a name: no leading "/", no drive such as "C:", no + * ":/" as in a URL, no "..", no spaces, and at most 80 characters. + */ +const MODEL_NAME = /^(?![A-Za-z]:)(?!.*:\/)(?!.*\.\.)[A-Za-z0-9][A-Za-z0-9._:@/-]{0,79}$/; /** The record's schema tag; see #1083 for the field contract. */ export type WorkflowRunRecord = { @@ -36,30 +60,44 @@ export type WorkflowRunRecord = { status: RunStatus; /** The named end the run reached, when it reached one. */ end?: string; + /** Why a run that reached no end failed, such as `loop_exhausted`. Code only: the message is free text. */ + failure?: { code: RecordedFailure }; startedAt: string; endedAt?: string; durationMs?: number; /** * Per-step entries (#1083: step, agent, tool/model, outcome, tokens) — a * run whose implementer runs on Codex and reviewer on Claude records each - * step's own fields. Unsettled steps carry no outcome. + * step's own fields. A skipped step, which never ran, carries no outcome. + * Every entry has its `pass`, 1 for a step that ran once. A step a loop ran + * more than once has an entry per pass, so the entries hold the tokens of + * every pass once. */ steps?: Array<{ step: string; + pass: number; agent?: string; tool?: string; model?: string; - outcome?: StepOutcome; + outcome?: Exclude; tokens?: TokenUsage; }>; - /** Summary counts derived from the per-step entries. */ - stepCounts?: { total: number; succeeded: number; failed: number; skipped: number }; + /** + * Counts of the step instances the run ended with, by status. A step a loop + * ran again counts once, as its last pass ended. + */ + stepCounts?: { total: number; succeeded: number; failed: number; skipped: number; cancelled: number }; /** Settled attempts across the run — the cumulative iteration count. */ iterations?: number; username: string; publishedAt: string; }; +type StepEntry = NonNullable[number]; + +/** What one pass of a step reported. */ +type Reported = Pick; + /** A record's path on the reports branch, keyed by run id. */ export function recordPath(runId: string): string { return `workflows/runs/${runId}.json`; @@ -97,6 +135,23 @@ export function buildStartRecord( }; } +/** + * Why a run that reached no end failed, as the first failure with a code teamai sets: the run's own, from a looping + * workflow that ran out, else that of the top-level step that failed first, else the first one under that step. + */ +function failureOf(run: RunState): RecordedFailure | undefined { + if (run.status !== 'failed' || run.end !== undefined) return undefined; + if (isRecordedFailure(run.failure)) return run.failure.code; + const failed = run.steps + .filter((step): step is Extract => step.status === 'failed') + .sort((left, right) => left.settledAt - right.settledAt); + const top = new Set(run.definition.steps.map((step) => step.id)); + const first = failed.find((step) => top.has(step.id)); + if (!first) return undefined; + const under = (step: StepState): boolean => step.id.startsWith(`${first.id}.`) || step.id.startsWith(`${first.id}[`); + return [first, ...failed.filter(under)].map((step) => step.failure).find(isRecordedFailure)?.code; +} + /** * Build the end-phase record: outcome, duration, iterations, and step counts, * all derived as counts so no step payload crosses into the record. @@ -109,35 +164,52 @@ export function buildEndRecord( ): WorkflowRunRecord { const started = Date.parse(run.createdAt); const ended = Date.parse(run.updatedAt); - const succeeded = run.steps.filter((step) => step.status === 'succeeded').length; - const failed = run.steps.filter((step) => step.status === 'failed').length; - const skipped = run.steps.filter((step) => step.status === 'skipped').length; - // Per-step entries: path + agent from the definition, the rest from the - // settled step state. Message, outputs, and session ids never cross. - const steps = run.steps.map((step) => { - const defined = locateStep(run, step.id)?.step; - // An end-only step settles without a report, so it has a status but no outcome. - const outcome = step.outcome ?? (step.status === 'succeeded' ? 'success' : undefined); + const count = (status: StepState['status']): number => run.steps.filter((step) => step.status === status).length; + const failure = failureOf(run); + // An entry per pass: path + agent from the definition, the rest from what + // that pass reported. Message, outputs, results, and session ids never cross. + const definitionOf = stepDefinitions(run); + const entry = (step: string, pass: number | undefined, reported: Reported | undefined): StepEntry => { + const agent = definitionOf(step)?.agent; + const outcome = reported?.outcome === 'skipped' ? undefined : reported?.outcome; return { - step: step.id, - ...(defined?.agent ? { agent: defined.agent } : {}), - ...(step.tool ? { tool: step.tool } : {}), - ...(step.model ? { model: step.model } : {}), + step, + pass: pass ?? 1, + ...(agent ? { agent } : {}), + ...(reported?.tool && isLocalWorkflowTool(reported.tool) ? { tool: reported.tool } : {}), + ...(reported?.model && MODEL_NAME.test(reported.model) ? { model: reported.model } : {}), ...(outcome ? { outcome } : {}), - ...(step.tokens ? { tokens: step.tokens } : {}), + ...(reported?.tokens ? { tokens: reported.tokens } : {}), }; - }); + }; + // A loop keeps a step's earlier passes in `passes`, and the passes of the + // instances it removed from the run, such as the items of a for_each it ran + // again, in `archive`. + const steps = [ + ...run.steps.flatMap((step) => [ + ...(step.passes ?? []).map((earlier) => entry(step.id, earlier.pass, earlier)), + entry(step.id, step.pass, settled(step) ? { ...step, outcome: VISIBLE_OUTCOMES[step.status] } : undefined), + ]), + ...(run.archive ?? []).map((archived) => entry(archived.step, archived.pass, archived)), + ]; return { ...buildStartRecord(run, driver, username, now), phase: 'end', status: run.status, ...(run.end !== undefined ? { end: run.end } : {}), + ...(failure ? { failure: { code: failure } } : {}), endedAt: run.updatedAt, ...(Number.isFinite(started) && Number.isFinite(ended) && ended >= started ? { durationMs: ended - started } : {}), steps, - stepCounts: { total: run.steps.length, succeeded, failed, skipped }, + stepCounts: { + total: run.steps.length, + succeeded: count('succeeded'), + failed: count('failed'), + skipped: count('skipped'), + cancelled: count('cancelled'), + }, // Cumulative when present; runs from before the counter fall back to the // bounded history length (their true count is unknowable now). iterations: run.settledCount ?? run.settledAttempts.length, @@ -167,37 +239,49 @@ async function isRecordingEnabled(config: LocalConfig): Promise { } /** - * Build, queue, and try to publish a run's phase record (#1083). - * - * The enqueue happens before the publish attempt, so a crash mid-publish - * leaves the record queued for the next pull. Publication failures — no - * remote, a busy reports lock, a rejected push — surface as `'pending'` and - * never fail the run operation that produced the record. With usage reporting - * off nothing is queued and there is no state to report. + * Build and queue a run's phase record (#1083). True once the record is in + * the queue; false with usage reporting off, or when it could not be queued, + * which logs a warning and never fails the run operation that produced it. + * Queueing writes only local files, so it may run under the run lock. */ -export async function noteRunRecord( +export async function queueRunRecord( config: LocalConfig, run: RunState, driver: DriverIdentity | undefined, phase: 'start' | 'end', -): Promise { +): Promise { try { - if (!await isRecordingEnabled(config)) return undefined; + if (!await isRecordingEnabled(config)) return false; const username = config.username ?? 'unknown'; const record = phase === 'start' ? buildStartRecord(run, driver, username) : buildEndRecord(run, driver, username); await enqueueRecord(config, record); + return true; + } catch (error) { + log.warn(`[workflows] Could not queue the ${phase} record of run ${run.id}, so the team report will not include it: ${error instanceof Error ? error.message : String(error)}`); + return false; + } +} + +/** + * Try to publish the queued records. A record is queued before this attempt, + * so a crash mid-publish leaves it for the next pull. Publication failures — + * no remote, a busy reports lock, a rejected push — surface as `'pending'` + * and never fail the run operation that queued the record. + */ +export async function publishRunRecords(config: LocalConfig): Promise { + try { return await publishPendingRecords(config); } catch { return 'pending'; } } -/** The record state of a run that settled earlier: pending while any of its records waits in the queue. */ -export async function runRecordState(config: LocalConfig, runId: string): Promise { - if (!await isRecordingEnabled(config)) return undefined; - return (await pendingRecords(config)).some((record) => record.run === runId) ? 'pending' : 'published'; +/** The record state of a run that settled earlier: none when it queued no end record, pending while any of its records waits in the queue. */ +export async function runRecordState(config: LocalConfig, run: RunState): Promise { + if (!run.endRecordQueued || !await isRecordingEnabled(config)) return undefined; + return (await pendingRecords(config)).some((record) => record.run === run.id) ? 'pending' : 'published'; } // ─── Local pending queue ──────────────────────────────── diff --git a/src/workflow/scope.ts b/src/workflow/scope.ts new file mode 100644 index 000000000..2a63e706b --- /dev/null +++ b/src/workflow/scope.ts @@ -0,0 +1,138 @@ +/** + * Scopes: the workflows inside a run, and the step instances directly in each. The instances of a scope are read from an + * index kept per `run.steps` array, so settling one scope does not scan every step of the run. + */ +import type { ExpressionStep } from './expression.js'; +import { isItem } from './fan-out.js'; +import { passesOf, previousPass, stepLoops, type StepLoops } from './loop.js'; +import { parseStepPath } from './step-path.js'; +import { settled, VISIBLE_OUTCOMES, type StepState } from './step-state.js'; +import type { TemplateContext } from './template.js'; +import type { InlineWorkflow, RunState, WorkflowDefinition, WorkflowStep } from './types.js'; + +/** A workflow inside a run: the top-level one, or the sub-workflow a step runs. */ +export interface Scope { + /** What every step path inside it starts with: '' at the top, `.` inside a sub-workflow. */ + prefix: string; + definition: WorkflowDefinition | InlineWorkflow; + inputs: Record | undefined; + /** For an inline workflow: the enclosing workflows' steps, by id, which its own steps hide. */ + outer?: Record; + loops: StepLoops; + /** The step that runs this workflow, which keeps its round. The top-level workflow keeps its round on the run. */ + runner?: StepState; + /** In the inline workflow of a `for_each` item, at any depth: that item, which its steps read as `item`. */ + item?: unknown; +} + +export function topScope(run: RunState): Scope { + return { prefix: '', definition: run.definition, inputs: run.inputs, loops: stepLoops(run.definition.steps) }; +} + +/** + * The scope of the workflow a step runs, or undefined when the run did not keep that workflow. + * An inline workflow reads the inputs and the steps of the workflow `scope` around it. + */ +export function childScope(run: RunState, scope: Scope, states: StepState[], state: StepState, step: WorkflowStep): Scope | undefined { + if (typeof step.workflow === 'object') { + return { + prefix: `${state.id}.`, definition: step.workflow, inputs: scope.inputs, outer: expressionSteps(scope, states, state), + loops: stepLoops(step.workflow.steps), runner: state, ...itemOf(scope, state), + }; + } + const definition = step.workflow === undefined ? undefined : run.workflows?.[step.workflow]; + return definition && { prefix: `${state.id}.`, definition, inputs: state.inputs, loops: stepLoops(definition.steps), runner: state }; +} + +/** + * Finds the definition of the step at a path in the run's workflow definitions alone, so a path still resolves after + * a loop archived the instances along it. Each workflow's steps are indexed by id once, when a path first reaches it. + */ +export function stepDefinitions(run: RunState): (path: string) => WorkflowStep | undefined { + const indexes = new Map>(); + return (path) => { + let workflow: WorkflowDefinition | InlineWorkflow | undefined = run.definition; + let step: WorkflowStep | undefined; + for (const { id } of parseStepPath(path)) { + if (!workflow) return undefined; + const index: Map = indexes.get(workflow) ?? new Map(workflow.steps.map((item) => [item.id, item])); + indexes.set(workflow, index); + step = index.get(id); + workflow = typeof step?.workflow === 'string' ? run.workflows?.[step.workflow] : step?.workflow; + } + return step; + }; +} + +/** For each `run.steps` array, its instances by the prefix of the scope they are directly inside. */ +const scopeIndex = new WeakMap>(); + +/** `states` by the prefix of the scope each is directly inside, in their order. A `for_each` item such as `build[0]` is in none. */ +function byScope(states: readonly StepState[]): Map { + const index = new Map(); + for (const state of states) { + const prefix = state.id.slice(0, state.id.lastIndexOf('.') + 1); + if (parseStepPath(state.id.slice(prefix.length))[0]?.index !== undefined) continue; + const group = index.get(prefix); + if (group) group.push(state); + else index.set(prefix, [state]); + } + return index; +} + +/** The step instances directly inside a scope, in declaration order. A `for_each` item such as `build[0]` is not one. */ +export function statesIn(run: RunState, scope: Scope): StepState[] { + let index = scopeIndex.get(run.steps); + if (!index) { + index = byScope(run.steps); + scopeIndex.set(run.steps, index); + } + return [...index.get(scope.prefix) ?? []]; +} + +/** + * Puts the instances of the workflow `runner` runs right after it. The old array's index carries over to the new one + * with them as the group of their scope, so starting a workflow inside each item does not index every step again. + */ +export function insertChildren(run: RunState, runner: StepState, children: StepState[]): void { + const old = run.steps; + const at = old.indexOf(runner) + 1; + run.steps = [...old.slice(0, at), ...children, ...old.slice(at)]; + const index = scopeIndex.get(old); + if (!index) return; + scopeIndex.delete(old); + for (const [prefix, group] of byScope(children)) index.set(prefix, [...group, ...index.get(prefix) ?? []]); + scopeIndex.set(run.steps, index); +} + +/** The step id an instance has inside its own workflow. */ +export function localId(scope: Scope, state: StepState): string { + const [segment] = parseStepPath(state.id.slice(scope.prefix.length)); + return segment?.id ?? state.id; +} + +function itemOf(scope: Scope, state: StepState | undefined): { item?: unknown } { + if (state && isItem(state)) return { item: state.item }; + return Object.hasOwn(scope, 'item') ? { item: scope.item } : {}; +} + +/** What an expression or template in a scope reads; for `reader`, what that step instance reads. */ +export function contextOf(scope: Scope, states: StepState[], reader?: StepState): TemplateContext { + return { inputs: scope.inputs, steps: expressionSteps(scope, states, reader), ...itemOf(scope, reader) }; +} + +/** + * What expressions in a workflow read as `steps.`: its own step instances, by their id in that workflow, + * and, in an inline workflow, the enclosing steps its own steps do not hide. For `reader`, a step it reads in the + * previous pass is that pass. + */ +function expressionSteps(scope: Scope, states: StepState[], reader?: StepState): Record { + const previous = reader ? scope.loops.previous(localId(scope, reader)) : new Set(); + return { ...scope.outer, ...Object.fromEntries(states.map((step) => [localId(scope, step), previous.has(localId(scope, step)) ? previousPass(step) : { + ...(settled(step) ? { outcome: VISIBLE_OUTCOMES[step.status] } : {}), + ...('end' in step && step.end !== undefined ? { end: step.end } : {}), + ...('outputs' in step && step.outputs !== undefined ? { outputs: step.outputs } : {}), + ...('results' in step && step.results !== undefined ? { results: step.results } : {}), + passes: passesOf(step), + }])) }; +} diff --git a/src/workflow/step-path.ts b/src/workflow/step-path.ts new file mode 100644 index 000000000..1c9a982a0 --- /dev/null +++ b/src/workflow/step-path.ts @@ -0,0 +1,20 @@ +/** One segment of a step path: a step id, and the item index when the instance is one item of a `for_each` step. */ +export interface StepPathSegment { + id: string; + index?: number; +} + +/** An index in a path, such as the 2 in `build[2]`: a whole number from 0 with no leading zero. */ +export const INDEX = '0|[1-9][0-9]*'; +const ITEM = new RegExp(`^([^[\\]]+)\\[(${INDEX})\\]$`); + +/** + * The segments of a step instance's path, such as `a.b[2].c`. Step ids contain neither "." nor "[", + * so a segment that is not `[]` is read whole as an id, which names no step. + */ +export function parseStepPath(path: string): StepPathSegment[] { + return path.split('.').map((segment) => { + const item = ITEM.exec(segment); + return item ? { id: item[1]!, index: Number(item[2]) } : { id: segment }; + }); +} diff --git a/src/workflow/step-state.ts b/src/workflow/step-state.ts new file mode 100644 index 000000000..670fe49eb --- /dev/null +++ b/src/workflow/step-state.ts @@ -0,0 +1,244 @@ +import { z } from 'zod'; +import type { DriverIdentity, StepLease, TokenUsage, WorkflowFailure } from './types.js'; + +const TokenUsageSchema: z.ZodType = z.object({ + input: z.number().optional(), + output: z.number().optional(), + cachedInput: z.number().optional(), +}); + +const FailureSchema: z.ZodType = z.object({ code: z.string(), message: z.string() }); + +const DriverSchema: z.ZodType = z.object({ kind: z.string(), machine: z.string(), workspace: z.string() }); + +const LeaseSchema: z.ZodType = z.object({ attempt: z.string(), revision: z.number(), driver: DriverSchema }); + +/** What a driver reports about the attempt that settled a step. */ +const report = { + exitCode: z.number().optional(), + message: z.string().optional(), + tool: z.string().optional(), + model: z.string().optional(), + session: z.string().optional(), + tokens: TokenUsageSchema.optional(), +}; + +export const OutcomeSchema = z.enum(['success', 'failed', 'skipped', 'cancelled']); +/** The outcome an expression reads of a settled step. */ +export type VisibleOutcome = z.infer; + +const ResultsSchema = z.array(z.object({ + item: z.unknown(), + outcome: OutcomeSchema, + outputs: z.unknown().optional(), +})); +export type ItemResult = z.infer[number]; + +/** One earlier pass of a step instance that a loop ran again. */ +export const StepPassSchema = z.object({ + pass: z.number().int(), + attempt: z.string().optional(), + outcome: OutcomeSchema, + outputs: z.unknown().optional(), + end: z.string().optional(), + results: ResultsSchema.optional(), + failure: FailureSchema.optional(), + ...report, +}); +export type StepPass = z.infer; + +/** A pass of a child or item instance that a loop removed from the run; `step` is its full path. */ +export const ArchivedPassSchema = StepPassSchema.extend({ step: z.string() }); +export type ArchivedPass = z.infer; + +/** + * What every step instance has. `id` is its path: a top-level step id, `.` + * for a step inside a sub-workflow, such as `check.classify`, or `[]` for an item of a `for_each` step. + */ +const StepBaseSchema = z.object({ + id: z.string(), + pass: z.number().int().optional(), + passes: z.array(StepPassSchema).optional(), + item: z.unknown().optional(), + /** The values `with` passed to the workflow this step runs, kept when that workflow declares inputs. */ + inputs: z.record(z.unknown()).optional(), + iteration: z.number().int().optional(), + round: z.number().int().optional(), +}); +type StepBase = z.infer; + +/** + * What a step that ran keeps. A file written before `settledAt` existed reads it as 0. `end` is the end the workflow + * this step runs reached. + */ +const settledFields = { + outputs: z.unknown().optional(), + end: z.string().optional(), + results: ResultsSchema.optional(), + settledAt: z.number().int().default(0), + attempt: z.string().optional(), + ...report, +}; + +/** + * One step instance. Unknown keys are dropped when a run file is read, so a step a 0.22.0 report released + * reads as a clean `pending` step. That version wrote no `outcome` on a step the engine settled; it reads from `status`. + * `running`: a workflow step whose child steps are in the run and not all settled. + */ +export const StepStateSchema = z.discriminatedUnion('status', [ + StepBaseSchema.extend({ status: z.literal('pending') }), + StepBaseSchema.extend({ status: z.literal('leased'), attempt: z.string(), lease: LeaseSchema }), + StepBaseSchema.extend({ status: z.literal('running') }), + StepBaseSchema.extend({ + status: z.literal('succeeded'), outcome: z.literal('success').default('success'), ...settledFields, loopFinal: z.boolean().optional(), + }), + StepBaseSchema.extend({ status: z.literal('failed'), outcome: z.literal('failed').default('failed'), ...settledFields, failure: FailureSchema.optional() }), + StepBaseSchema.extend({ status: z.literal('skipped'), settledAt: z.number().int().default(0) }), + StepBaseSchema.extend({ status: z.literal('cancelled'), settledAt: z.number().int().default(0), attempt: z.string().optional(), failure: FailureSchema }), +]); +type ReadonlyId = T extends unknown ? Omit & { readonly id: string } : never; +/** A step instance's `id` is read-only: the engine indexes instances by it, and every transition keeps it. */ +export type StepState = ReadonlyId>; +type StepStatus = StepState['status']; + +type StepIn = Extract; +type PendingStep = StepIn<'pending'>; +export type LeasedStep = StepIn<'leased'>; +type RunningStep = StepIn<'running'>; +type SucceededStep = StepIn<'succeeded'>; +type FailedStep = StepIn<'failed'>; +type SkippedStep = StepIn<'skipped'>; +type CancelledStep = StepIn<'cancelled'>; + +type Settlement = Omit; + +const BASE_KEYS = Object.keys(StepBaseSchema.shape) as Array; +const SETTLED_KEYS = Object.keys(settledFields) as Array; +export const VISIBLE_OUTCOMES = { succeeded: 'success', failed: 'failed', skipped: 'skipped', cancelled: 'cancelled' } as const; + +/** + * Replaces a step's fields in place with `next`, keeping the fields every step has, and returns it as its new variant. + * In place, because `run.steps` holds the instance and the engine finds positions in it by identity. + */ +function become(state: StepState, next: Omit): T { + const fields = state as Record; + const kept = Object.fromEntries(BASE_KEYS.filter((key) => fields[key] !== undefined).map((key) => [key, fields[key]])); + for (const key of Object.keys(fields)) delete fields[key]; + return Object.assign(fields, kept, next) as T; +} + +export function skip(state: StepState, settledAt: number): SkippedStep { + return become(state, { status: 'skipped', settledAt }); +} + +/** A released lease returns the step to `pending` with nothing from the attempt. */ +export function release(state: LeasedStep): PendingStep { + return become(state, { status: 'pending' }); +} + +export function lease(state: StepState, held: StepLease): LeasedStep { + return become(state, { status: 'leased', attempt: held.attempt, lease: held }); +} + +/** A workflow step starts; `inputs` are the values its `with` passed, when that workflow declares inputs. */ +export function start(state: StepState, inputs?: Record): RunningStep { + const running = become(state, { status: 'running' }); + if (inputs) running.inputs = inputs; + return running; +} + +export function succeed(state: StepState, settlement: Settlement): SucceededStep { + return become(state, { status: 'succeeded', outcome: 'success', ...settlement }); +} + +export function fail(state: StepState, settlement: Settlement): FailedStep { + return become(state, { status: 'failed', outcome: 'failed', ...settlement }); +} + +/** A leased or running step stops before it settles. A leased one keeps its attempt, which can no longer be reported. */ +export function cancel(state: LeasedStep | RunningStep, failure: WorkflowFailure, settledAt: number): CancelledStep { + return become(state, { status: 'cancelled', settledAt, ...(state.status === 'leased' ? { attempt: state.attempt } : {}), failure }); +} + +/** The current pass of a settled step, as `passes` keeps it once a loop runs the step again; undefined while it has not settled. */ +export function passOf(state: StepState): StepPass | undefined { + if (!settled(state)) return undefined; + const outcome = VISIBLE_OUTCOMES[state.status]; + return StepPassSchema.parse({ ...state, pass: state.pass ?? 1, outcome }); +} + +/** + * A loop runs a settled step again: its current pass joins `passes`, and it waits as the next pass, with nothing of + * the workflow it ran. `iteration` is the runs a loop owner has had so far. + */ +export function repend(state: StepState, iteration?: number): PendingStep { + const current = passOf(state); + const passes = [...state.passes ?? [], ...(current ? [current] : [])]; + const pass = (state.pass ?? 1) + 1; + const pending = become(state, { status: 'pending' }); + delete pending.inputs; + delete pending.round; + return Object.assign(pending, { pass, passes }, iteration !== undefined ? { iteration } : {}); +} + +/** A step whose looping workflow runs its next round. */ +export function nextRound(state: RunningStep): RunningStep { + return Object.assign(state, { round: (state.round ?? 1) + 1 }); +} + +/** A loop owner whose loop will not run again after `iteration` runs. */ +export function endLoop(owner: SucceededStep, iteration: number): SucceededStep { + return Object.assign(owner, { loopFinal: true, iteration }); +} + +/** + * A loop owner that would run more than `max_iterations` times fails instead. One that just ran keeps what that run + * reported; one that has not started takes the outputs of its latest pass. + */ +export function exhaust(state: SucceededStep | PendingStep, failure: WorkflowFailure, settledAt: number, iteration?: number): FailedStep { + const latest = state.status === 'succeeded' ? state : state.passes?.at(-1); + const ran = state.status === 'succeeded' + ? Object.fromEntries(SETTLED_KEYS.filter((key) => state[key] !== undefined).map((key) => [key, state[key]])) + : latest?.outputs !== undefined ? { outputs: latest.outputs } : {}; + const failed = fail(state, { ...ran, settledAt, message: failure.message, failure }); + return iteration === undefined ? failed : Object.assign(failed, { iteration }); +} + +/** A step that ran and did not succeed. */ +export function stopped(step: StepState): boolean { + switch (step.status) { + case 'failed': + case 'cancelled': + return true; + case 'pending': + case 'leased': + case 'running': + case 'succeeded': + case 'skipped': + return false; + default: { + const unknown: never = step; + return unknown; + } + } +} + +/** A step that will not run again in this pass, so steps that need or read it can go on. */ +export function settled(step: StepState | undefined): step is Extract { + if (!step) return false; + switch (step.status) { + case 'succeeded': + case 'failed': + case 'skipped': + case 'cancelled': + return true; + case 'pending': + case 'leased': + case 'running': + return false; + default: { + const unknown: never = step; + return unknown; + } + } +} diff --git a/src/workflow/store.ts b/src/workflow/store.ts index 3d15f0f35..5933bd0c7 100644 --- a/src/workflow/store.ts +++ b/src/workflow/store.ts @@ -1,7 +1,9 @@ import { randomBytes } from 'node:crypto'; import path from 'node:path'; +import { z } from 'zod'; import { readFileSafe, writeJsonAtomic } from '../utils/fs.js'; import { getDataHome, type LocalConfig } from '../types.js'; +import { ArchivedPassSchema, StepStateSchema } from './step-state.js'; import type { RunState, WorkflowDefinition } from './types.js'; /** @@ -51,12 +53,22 @@ export function isRunState(value: unknown): value is RunState { && Array.isArray(run.settledAttempts); } +/** The parts of a run file the step union governs, with what a file written before them reads as. */ +const RunProgressSchema = z.object({ + steps: z.array(StepStateSchema), + cancelledAttempts: z.array(z.string()).default([]), + archive: z.array(ArchivedPassSchema).default([]), + round: z.number().int().optional(), +}); + export async function readRun(config: LocalConfig, runId: string): Promise { const raw = await readFileSafe(runFilePath(config, runId)); if (raw === null) return 'missing'; try { const value: unknown = JSON.parse(raw); - return isRunState(value) ? value : 'corrupt'; + if (!isRunState(value)) return 'corrupt'; + const progress = RunProgressSchema.safeParse(value); + return progress.success ? { ...value, ...progress.data } : 'corrupt'; } catch { return 'corrupt'; } diff --git a/src/workflow/template.ts b/src/workflow/template.ts index fe123e1a2..cd40cdbcc 100644 --- a/src/workflow/template.ts +++ b/src/workflow/template.ts @@ -1,15 +1,21 @@ /** - * Workflow templates: `{{ inputs.… }}`, `{{ steps.… }}` and `{{ with.… }}` inside prompt text, `env` values and - * every string inside a `with` value. Any other `{{ … }}` is plain text, so prompts about GitHub Actions or Jinja stay as written. + * Workflow templates inside prompt text, `env` values and every string inside a `with` value: a `{{ … }}` that starts + * with `inputs.`, `steps.`, `with.` or, where an item is bound, `item`, after any "!" or "(", or that calls `length`, + * `any`, `all` or `filter` on one of those, or any function on a step field. It reads one path, or evaluates an + * expression as an `if` does. Any other `{{ … }}` is plain text, so prompts about GitHub Actions, Jinja, Vue or an + * Ansible loop stay as written. */ -import { parseExpression, resolvePath, type ExpressionContext, type ExpressionPath } from './expression.js'; +import { + evaluateExpression, expressionPaths, parseExpression, parseFields, resolvePath, + readsLikeExpression, type Expression, type ExpressionContext, type ExpressionPath, type PathField, +} from './expression.js'; import type { WorkflowStep } from './types.js'; /** A `{{ with.… }}` read: a value the step's own `with` names. Only templates read it; an `if` cannot. */ export interface WithPath { root: 'with'; - /** The `with` name, then the fields read inside its value. */ - fields: string[]; + /** The `with` name, then the names and indexes read inside its value. */ + fields: PathField[]; } /** What a template reads: an expression path, or a value from the step's own `with`. */ @@ -17,17 +23,16 @@ export interface TemplateContext extends ExpressionContext { with?: unknown; } -/** One `{{ path }}` and where it sits in the source text. */ -export interface TemplateRead { - /** The whole `{{ … }}` as written. */ - source: string; - path: ExpressionPath | WithPath; -} +/** + * One `{{ … }}` as written, and what it reads: an expression, which may be one path, or a value from the step's own + * `with`, which no expression reads. + */ +export type TemplateRead = { source: string; expression: Expression } | { source: string; path: WithPath }; /** Literal text and reads, in source order. */ export type Template = Array; -/** A recognized template that is not one path: the `{{ … }}` as written and why. */ +/** A template that does not parse: the `{{ … }}` as written and why. */ export interface TemplateError { source: string; message: string; @@ -35,66 +40,120 @@ export interface TemplateError { export type TemplateParse = { ok: true; template: Template } | { ok: false; errors: TemplateError[] }; -/** A `{{ … }}` whose content starts with `inputs.`, `steps.` or `with.`. */ -const TEMPLATE = /\{\{\s*((?:inputs|steps|with)\.[^{}]*?)\s*\}\}/; -/** A template not preceded by `$`: `${{ inputs.x }}` is GitHub Actions text and stays as written. */ -const RECOGNIZED = new RegExp(`(?`, then any fields inside its value. */ -const WITH_PATH = /^with(?:\.[A-Za-z0-9_-]+)+$/; +/** What templates read, where no item is bound and where one is. */ +interface Syntax { + /** A template, `${{ … }}` included. */ + template: RegExp; + /** A template not preceded by `$`: `${{ inputs.x }}` is GitHub Actions text and stays as written. */ + recognized: RegExp; + /** The start of a path a template reads, which an expression must start with or call a function on. */ + path: RegExp; +} + +/** `paths` is the regex source of what a template that reads one path holds between its braces. */ +function syntax(paths: string): Syntax { + const template = new RegExp(`\\{\\{\\s*(${paths})\\s*\\}\\}`); + return { template, recognized: new RegExp(`(?`, then any names and indexes inside its value. */ +const WITH_PATH = /^with\.[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+|\[[^\]]*\]?)*$/; +/** A `{{ … }}`, `${{ … }}` included, up to the first `}}`, that holds no other `{{`. */ +const ANY_BRACES = /\{\{((?:(?!\{\{)[\s\S])*?)\}\}/g; +/** Those not preceded by `$`. */ +const BRACES = new RegExp(`(?; an expression reads only inputs, steps and item'; /** - * The first template-like `{{ … }}` in `text` as written, if any; used to reject one where text is never rendered, - * such as `command`. It also finds `${{ … }}`, since a command would receive those braces as written too. + * A template-like `{{ … }}` in `text` as written, if any; used to reject one where text is never rendered, such as + * `command`. Unlike parseTemplate, it also finds `${{ … }}`, path or expression, since a command would receive those + * braces as written too. */ -export function findTemplate(text: string): string | undefined { - return TEMPLATE.exec(text)?.[0]; +export function findTemplate(text: string, item: boolean): string | undefined { + const { template, path } = item ? SYNTAX.bound : SYNTAX.unbound; + return template.exec(text)?.[0] ?? [...text.matchAll(ANY_BRACES)].find((match) => readsLikeExpression(match[1] ?? '', path))?.[0]; } -/** The one path a recognized template reads, or why it does not read exactly one. */ -function readOf(inner: string): ExpressionPath | WithPath | string { +/** What a template reads, or why it does not parse. */ +function readOf(inner: string): { expression: Expression } | { path: WithPath } | string { if (inner.startsWith('with.')) { - return WITH_PATH.test(inner) ? { root: 'with', fields: inner.split('.').slice(1) } : 'a template reads one path, such as with.'; + if (!WITH_PATH.test(inner)) return WITH_MESSAGE; + const parsed = parseFields(inner); + return parsed.ok ? { path: { root: 'with', fields: parsed.fields.slice(1) } } : parsed.message; } const parsed = parseExpression(inner); - if (!parsed.ok) return parsed.message; - return parsed.expression.kind === 'path' ? parsed.expression.path : 'a template reads one path'; + return parsed.ok ? { expression: parsed.expression } : parsed.message; } -/** Splits `text` into literal text and reads. Every recognized template must be exactly one path. */ -export function parseTemplate(text: string): TemplateParse { +/** + * Splits `text` into literal text and reads. A `{{ … }}` that starts with a path is a template, and so is one between + * them that reads like an expression, as written or with a mistake in it; every template must parse. `item` is + * whether an item is bound where the text renders; where none is, `{{ item }}` is literal text. + */ +export function parseTemplate(text: string, item: boolean): TemplateParse { + const { recognized, path } = item ? SYNTAX.bound : SYNTAX.unbound; const template: Template = []; const errors: TemplateError[] = []; - let index = 0; - for (const match of text.matchAll(RECOGNIZED)) { - const source = match[0]; - const path = readOf(match[1] ?? ''); - if (typeof path === 'string') { - errors.push({ source, message: path }); - } else { - if (match.index > index) template.push(text.slice(index, match.index)); - template.push({ source, path }); + const read = (source: string, inner: string): void => { + const parsed = readOf(inner); + if (typeof parsed === 'string') errors.push({ source, message: parsed }); + else template.push({ source, ...parsed }); + }; + const literal = (segment: string): void => { + let index = 0; + for (const match of segment.matchAll(BRACES)) { + if (!readsLikeExpression(match[1] ?? '', path)) continue; + if (match.index > index) template.push(segment.slice(index, match.index)); + read(match[0], match[1] ?? ''); + index = match.index + match[0].length; } - index = match.index + source.length; + if (index < segment.length) template.push(segment.slice(index)); + }; + let index = 0; + for (const match of text.matchAll(recognized)) { + literal(text.slice(index, match.index)); + read(match[0], match[1] ?? ''); + index = match.index + match[0].length; } - if (errors.length > 0) return { ok: false, errors }; - if (index < text.length) template.push(text.slice(index)); - return { ok: true, template }; + literal(text.slice(index)); + return errors.length > 0 ? { ok: false, errors } : { ok: true, template }; +} + +/** Every expression a template evaluates, one path included, in source order. */ +export function templateExpressions(template: Template): Expression[] { + return template.flatMap((part) => (typeof part === 'string' || !('expression' in part) ? [] : [part.expression])); } -/** Every `inputs` and `steps` path a template reads, in source order. */ +/** Every `inputs`, `steps` and `item` path a template reads, in source order. */ export function templatePaths(template: Template): ExpressionPath[] { - return template.flatMap((part) => (typeof part === 'string' || part.path.root === 'with' ? [] : [part.path])); + return templateExpressions(template).flatMap(expressionPaths); } /** Every `with` value a template reads, in source order. */ export function templateWithPaths(template: Template): WithPath[] { - return template.flatMap((part) => (typeof part === 'string' || part.path.root !== 'with' ? [] : [part.path])); + return template.flatMap((part) => (typeof part === 'string' || !('path' in part) ? [] : [part.path])); } -/** Reads a path. A `with` path walks the step's `with` values the way an `inputs` path walks the inputs. */ -function resolve(path: ExpressionPath | WithPath, context: TemplateContext): unknown { - if (path.root !== 'with') return resolvePath(path, context); - return resolvePath({ root: 'inputs', fields: path.fields, column: 1 }, { inputs: context.with, steps: {} }); +/** + * The value a read renders: its expression, evaluated as an `if` is, or its `with` path, which walks the step's `with` + * values the way an `inputs` path walks the inputs. + */ +function valueOf(read: TemplateRead, context: TemplateContext): unknown { + if ('expression' in read) return evaluateExpression(read.expression, context); + return resolvePath({ root: 'inputs', fields: read.path.fields, column: 1 }, { inputs: context.with, steps: {} }); } /** A value as text: strings as they are, numbers and booleans as text, objects and arrays as JSON, null as ''. */ @@ -105,15 +164,15 @@ function textOf(value: unknown): string { return String(value); } -/** Renders a template to text. A missing path renders as ''. */ +/** Renders a template to text. A missing value renders as ''. */ export function renderText(template: Template, context: TemplateContext): string { - return template.map((part) => (typeof part === 'string' ? part : textOf(resolve(part.path, context)))).join(''); + return template.map((part) => (typeof part === 'string' ? part : textOf(valueOf(part, context)))).join(''); } -/** Renders a whole value: a template that is exactly one `{{ path }}` keeps the JSON value it reads; anything else is text. */ +/** Renders a whole value: a template that is exactly one `{{ … }}` keeps the JSON value it reads; anything else is text. */ export function renderValue(template: Template, context: TemplateContext): unknown { const only = template.length === 1 ? template[0] : undefined; - if (only !== undefined && typeof only !== 'string') return resolve(only.path, context); + if (only !== undefined && typeof only !== 'string') return valueOf(only, context); return renderText(template, context); } @@ -136,13 +195,33 @@ export function templatedTexts(step: Pick): StepExpression[] { + return [ + ...(step.if !== undefined ? [{ what: 'if' as const, source: step.if }] : []), + ...(step.loop?.while !== undefined ? [{ what: 'loop.while' as const, source: step.loop.while }] : []), + ...(step.break !== undefined ? [{ what: 'break' as const, source: step.break }] : []), + ...(step.forEach !== undefined ? [{ what: 'for_each' as const, source: step.forEach.items }] : []), + ]; +} + /** * Renders every string inside a value, at any depth, with `renderValue`, so a string that is exactly one - * `{{ path }}` keeps the JSON value it reads. Other values stay as they are. Fails at the first template that does not parse. + * `{{ … }}` keeps the JSON value it reads. Other values stay as they are. Fails at the first template that does not parse. + * `item` is a template only when `context` binds one. */ export function renderStrings(value: unknown, context: TemplateContext): { ok: true; value: unknown } | { ok: false; error: TemplateError } { if (typeof value === 'string') { - const parsed = parseTemplate(value); + const parsed = parseTemplate(value, Object.hasOwn(context, 'item')); return parsed.ok ? { ok: true, value: renderValue(parsed.template, context) } : { ok: false, error: parsed.errors[0]! }; } if (!value || typeof value !== 'object') return { ok: true, value }; diff --git a/src/workflow/types.ts b/src/workflow/types.ts index 1835b33fb..56a976a55 100644 --- a/src/workflow/types.ts +++ b/src/workflow/types.ts @@ -1,5 +1,7 @@ /** Workflow types for the v1alpha1 local-driver subset. */ +import type { ArchivedPass, StepState } from './step-state.js'; + export interface DriverIdentity { kind: string; machine: string; @@ -38,6 +40,42 @@ export interface WorkflowStep { with?: Record; /** Name in `ends` this step finishes the workflow with when it succeeds. A step with no action only ends the workflow and is never leased. */ end?: string; + /** Runs the steps from `backTo` to this step again after this step succeeds. */ + loop?: StepLoop; + /** Expression source, evaluated when this step succeeds; when true, its workflow, or that workflow's loop, ends as success. */ + break?: string; + /** Runs the step's action once per item of a list, each as an instance such as `build[2]`. */ + forEach?: ForEach; +} + +/** A `for_each`: the list of items, and how many of them may run at once. */ +export interface ForEach { + /** Expression source that gives the list. */ + items: string; + /** At most this many items are started and not yet settled. No limit when absent. */ + maxParallel?: number; +} + +/** + * A workflow loop. When a round reaches no end, every step of the workflow runs again, while `while` is true, or + * always when there is none. The workflow runs at most `maxIterations` rounds; one more fails it with `loop_exhausted`. + */ +export interface WorkflowLoop { + /** Expression source, evaluated when a round has nothing left to run. */ + while?: string; + maxIterations: number; +} + +/** + * A step loop. After its step succeeds, every step from `backTo` to it runs again, while `while` is true, or always + * when there is none. The step runs at most `maxIterations` times; a run past that fails it with `loop_exhausted`. + */ +export interface StepLoop { + /** The step itself, or a step it needs, directly or not. */ + backTo: string; + /** Expression source, evaluated when the step succeeds. */ + while?: string; + maxIterations: number; } /** The keys that give a step its action, as YAML writes them, each with the definition field it fills. */ @@ -57,6 +95,7 @@ export type EndOutcome = 'succeeded' | 'failed'; export interface InlineWorkflow { ends?: Record; outputs?: Record; + loop?: WorkflowLoop; steps: WorkflowStep[]; } @@ -70,6 +109,8 @@ export interface WorkflowDefinition { ends?: Record; /** Values this workflow returns to a step that runs it, each a path such as `steps..outputs.`. */ outputs?: Record; + /** Runs the workflow's steps again, round after round. */ + loop?: WorkflowLoop; steps: WorkflowStep[]; } @@ -88,16 +129,16 @@ export interface WorkflowDiagnostic { * templates existed would now render it as empty text instead of the text it was written as. * `dotted_name`: a step id or an `inputs` or `output` field name contains "."; no path can read it, and a * step id would clash with the dotted paths that address steps inside a sub-workflow. - * `undeclared_output`: a read of `steps..outputs.` that the workflow step `` runs does not declare; + * `undeclared_output`: a read names an output the workflow a step runs does not declare; * the file may be unchanged while that workflow changed, and the kept copy would read null where it expects a value. * `undeclared_end`: an `if` compares `steps..end` with a name the workflow step `` runs does not declare in `ends`; * for the same reason, the kept copy would take a branch that can never match. + * `bad_loop`: a `loop`, `break` or `for_each` pull rejects; the kept copy would run once the steps the author wrote + * to repeat or to run per item, or run on past the point the author wrote to stop at. */ - code?: 'unknown_key' | 'missing_needs' | 'false_if' | 'bad_template' | 'dotted_name' | 'undeclared_output' | 'undeclared_end'; + code?: 'unknown_key' | 'missing_needs' | 'false_if' | 'bad_template' | 'dotted_name' | 'undeclared_output' | 'undeclared_end' | 'bad_loop'; } -/** `running`: a workflow step whose child steps are in the run and not all settled. */ -export type StepStatus = 'pending' | 'leased' | 'running' | 'succeeded' | 'failed' | 'skipped'; export type StepOutcome = 'success' | 'failed' | 'released'; export type RunStatus = 'running' | 'succeeded' | 'failed'; @@ -107,30 +148,6 @@ export interface StepLease { driver: DriverIdentity; } -/** - * One step instance. `id` is its path: a top-level step id, or `.` - * for a step inside a sub-workflow, such as `check.classify`. - */ -export interface StepState { - id: string; - status: StepStatus; - attempt?: string; - lease?: StepLease; - outcome?: StepOutcome; - outputs?: unknown; - /** The values `with` passed to the workflow this step runs, kept when that workflow declares inputs. */ - inputs?: Record; - /** The end the workflow this step runs reached. */ - end?: string; - exitCode?: number; - message?: string; - tool?: string; - model?: string; - session?: string; - tokens?: TokenUsage; - failure?: WorkflowFailure; -} - export interface TokenUsage { input?: number; output?: number; @@ -151,6 +168,8 @@ export interface RunState { status: RunStatus; /** The end this run reached, set when it settles. */ end?: string; + /** Why the run failed when no end says it, such as `loop_exhausted` on a looping top-level workflow. */ + failure?: WorkflowFailure; /** The validated `--inputs`, stored when the workflow declares inputs. */ inputs?: Record; /** Covers `definition` and every workflow in `workflows`. */ @@ -158,11 +177,22 @@ export interface RunState { definition: WorkflowDefinition; /** The sub-workflows the run uses, by name, as they were cached when it started. */ workflows?: Record; - /** Step instances in declaration order; a sub-workflow's steps follow the step that runs it. */ - steps: StepState[]; + /** + * Step instances in declaration order; a sub-workflow's steps follow the step that runs it. Never changed in place: + * adding or removing an instance assigns a new array, so an index the engine keeps per array never goes stale. + */ + steps: readonly StepState[]; settledAttempts: string[]; /** Cumulative settled-attempt count; settledAttempts is bounded at 32 for dedup. */ settledCount?: number; + /** Whether the run queued its end record when it settled. A run settled with usage reporting off, or before this field, queued none. */ + endRecordQueued?: boolean; + /** Attempts whose lease the engine cancelled; a report of one is answered `lease_cancelled`. */ + cancelledAttempts?: string[]; + /** Passes of child and item instances a loop removed from `steps`. */ + archive?: ArchivedPass[]; + /** The round of a looping top-level workflow. */ + round?: number; createdAt: string; updatedAt: string; }