From 3687794830373d23e2a6200822fd8ceef9235e50 Mon Sep 17 00:00:00 2001 From: Proactive Runtime Bot Date: Sun, 20 Sep 2026 02:57:30 -0700 Subject: [PATCH 01/12] feat(skills): record Relayflows v2 authoring friction found in a real run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both skills were written against an earlier v2 surface and neither warned about the two things that each cost a full run while authoring a 58-step campaign flow for AgentWorkforce/relay. writing-relayflows (1.0.0 -> 1.1.0): - `--local-agent` is REQUIRED for any local run with an agent step. Also the three sharp edges around it: the remedy is only named for `flows run` on a spec path, `flows resume --local-agent` accepts and ignores the flag on a spec run, and `flows schedule` has no such flag at all. - A failing `subprocess_gate` reports exit=1 with empty stdout AND stderr tails, nothing in relayflowd.log, nothing in the CLI output. Prefer `artifact_exists`/`regex_match`, or put enforcement in a deterministic step after the agent. - Eight gate types, not three, plus the TypeScript `.gate()` postfix form. - The three-argument `flow(name, header, body)` and where a TS budget lives. - `f.run`'s default lease is 30s, maximum 15m — invisible in the old docs and long enough to bite anything touching the network. - There is no `failOnError: false`; document both workarounds. - `permissions` is journaled and NOT enforced; `options.cwd` is declared by the packages and rejected by the runtime at the same version. - A green `flows check` says nothing about whether the daemon will accept the spec, so budget a shakedown run. relay-80-100-workflow (1.0.9 -> 1.1.0): - Every code example is the v1 builder. Say so up front, and add the v1-to-v2 translation table plus the evidence-recorder pattern that replaces failOnError + captureOutput + {{steps.X.output}}. - Don't gate agent steps in v2, for the reason above. Claims are cited to source in the flows tree (e73b315e, CLI 2.0.22) or to an observed run. Items credited to the Relayflows v2 Field Report (2026-09-18, Julian Fann) were corroborated against the same tree. Upstream issues filed as AgentWorkforce/flows#502-#511. Co-Authored-By: Claude Opus 5 --- prpm.json | 8 +- skills/relay-80-100-workflow/SKILL.md | 66 +++++++++++++++- skills/writing-relayflows/SKILL.md | 104 +++++++++++++++++++++++++- 3 files changed, 169 insertions(+), 9 deletions(-) diff --git a/prpm.json b/prpm.json index 55cf53e5..b47651ed 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.1.6", + "version": "1.2.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -95,7 +95,7 @@ }, { "name": "writing-relayflows", - "version": "1.0.0", + "version": "1.1.0", "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) \u2014 see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", "format": "claude", "subtype": "skill", @@ -154,8 +154,8 @@ }, { "name": "relay-80-100-workflow", - "version": "1.0.9", - "description": "Use when writing agent-relay workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", + "version": "1.1.0", + "description": "Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, the v1-to-v2 translation (no failOnError/captureOutput/{{steps.X.output}}) and its evidence-recorder replacement, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", "format": "claude", "subtype": "skill", "tags": [ diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index f115941a..cc25b867 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -1,6 +1,6 @@ --- name: relay-80-100-workflow -description: Use when writing agent-relay workflows that must fully validate features end-to-end before merging. Covers the 80-to-100 pattern - going beyond "code compiles" to "feature works, tested E2E locally." Includes repair-before-failure validation gates, review-depth fresh-eyes review/fix loops with test hardening, PGlite for in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates after every edit, and the full lifecycle from implementation through passing tests to commit. +description: Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging. Covers the 80-to-100 pattern - going beyond "code compiles" to "feature works, tested E2E locally." Includes the v1-to-v2 translation (v2 has no failOnError, captureOutput or {{steps.X.output}}) and the evidence-recorder pattern that replaces them, repair-before-failure validation gates, review-depth fresh-eyes review/fix loops with test hardening, PGlite for in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates after every edit, and the full lifecycle from implementation through passing tests to commit. --- # Writing 80-to-100 Validated Workflows @@ -16,6 +16,68 @@ Most agent workflows get features to ~80%: code written, types check, maybe a bu - Any workflow where "it compiles" is not sufficient proof of correctness - When you want confidence that the commit actually works before deploying +## Which engine are you writing for? + +**Every code example below is the v1 builder** — `@agent-relay/sdk/workflows`'s `workflow(...).step(...)`, with `failOnError`, `captureOutput`, `{{steps.X.output}}` and `.onError()`. The patterns are engine-independent; those four knobs are not. If you are writing a **Relayflows v2** flow (`@relayflows/surface` / `@relayflows/sdk`, CLI `flows`), read the next section before copying anything — a v2 spec has none of them and following these examples literally produces something that will not compile. + +See `writing-relayflows` for the v2 authoring surface itself. + +## The v2 translation + +v2 has three step verbs (`deterministic` / `llm` / `agent`) and gates every deterministic step on exit code **with no opt-out**. That single fact is what every translation below works around. + +| v1 | v2 | +| --- | --- | +| `.timeout(ms)` | `budget.maxWallclockMs`, or `{ budget: { wallclock: '110m' } }` in a TS `FlowHeader` | +| `retries: n` | `maxIterations: n + 1` | +| `failOnError: false` | **nothing** — see the recorder below | +| `captureOutput: true` | **nothing** — output is journaled, but not addressable from a later step's command | +| `{{steps.X.output}}` | **nothing** — no template interpolation between steps | +| `verification: { type: 'file_exists' }` | the `artifact_exists` named gate | +| `verification: { type: 'exit_code' }` on an agent step | no equivalent — see "don't gate agent steps" | +| `.pattern('dag')`, `.maxConcurrency(n)` | nothing; `dependsOn` is the only ordering | +| `.onError('retry')` | nothing; a failed step fails the run (it is resumable with `flows resume`) | +| `.channel(...)`, `.idleNudge(...)`, `preset`/`role` | nothing | + +### The evidence recorder replaces `failOnError` + `captureOutput` + `{{steps.X.output}}` + +All three v1 knobs existed to serve one pattern: run a check, let it be red, hand its output to the agent built to answer it. In v2 you rebuild that with a small script of your own: + +```js +// gates.mjs record --name --command-base64 +// Runs the command, writes {name, command, exitCode, verdict, tail} to +// evidence/.json, and ALWAYS exits 0. +// +// gates.mjs require-green --names a,b,c +// Reads those files back. This is the only thing that says "green". +``` + +The flow then reads: + +```ts +det('unit-tests', record('unit-tests', 'npx vitest run')); // always exits 0 +agentStep({ id: 'repair-ts', agent: 'fixer', dependsOn: ['unit-tests'], + task: ['Read evidence/unit-tests.json. Green means do nothing.', + 'Red means fix it and rerun until the recorder writes green.'] }); +det('ts-final', record('unit-tests', 'npx vitest run'), ['repair-ts']); +det('ts-assert', gate('require-green', '--names unit-tests'), ['ts-final']); // this one may fail +``` + +It is more machinery than `failOnError: false`, and it buys something v1 did not have: the repair agent reads a **file**, not an interpolated string, so it sees the whole tail; and final acceptance recomputes the verdict from recordings rather than trusting any agent's report. Base64-encode the command — it will contain `&&`, pipes and quotes, and the calling shell must not reinterpret them. + +`|| true` is the other option. It works and it throws the exit code away, which means a later gate has nothing to read and a forgotten gate ships red work silently. Tracked upstream at [flows#509](https://github.com/AgentWorkforce/flows/issues/509). + +### Don't gate agent steps in v2 + +v1's `verification: { type: 'exit_code' }` on an agent step has no v2 equivalent, and the closest thing — `subprocess_gate` — is a trap: the lowering runs the command under `stdio: 'inherit'` and the daemon's stdio is captured nowhere, so a failure arrives as `exit=1` with empty `stdout_tail` **and** empty `stderr_tail`, nothing in `relayflowd.log`, and nothing in the CLI output ([flows#511](https://github.com/AgentWorkforce/flows/issues/511)). A 58-step campaign flow lost a full run to exactly this: the agent succeeded, its gate failed three times, and the verdict its gate command printed on every path reached nowhere. + +Use `artifact_exists` (journal-based, replay-stable) where a file is the deliverable, and otherwise **no gate on the agent step at all** — put the enforcement in the deterministic recorded gate that follows it. That is the same advice as *Keep Repairable Gates On The Critical Path* below, and v2 makes it mandatory rather than merely wise. + +### Two more v2 facts that will cost you a run each + +- **`flows run` parks at the first agent step without `--local-agent`.** The message reads as a missing worker, not a missing flag. +- **`permissions` on an agent step is journaled and not enforced.** Declare it for reviewability; do not treat a `flows run` as sandboxed. Your implementation agents share one checkout, so partition their lanes by path *and* sequence them, and make an `edit-gate` that rejects out-of-scope changes. + ## Core Principle: Test In The Workflow The key insight: **run tests as deterministic steps inside the workflow itself**. Don't just write test files — execute them, verify they pass, fix failures, and re-run. The workflow doesn't commit until tests are green. @@ -551,3 +613,5 @@ Output: | No verify gate after agent edits | Agent exits 0 without writing anything | Add `git diff --quiet` check after every edit, then route failures to a repair step | | `git diff --quiet` for new package/test directories | Untracked files are invisible, so valid new artifacts can look like "no changes" | Use `git status --short -- ` and a repairable capture → fix → final gate pattern | | Committing after `failOnError: false` without checking exits | Broken work can be committed because the shell step returned successfully | In `commit-if-green`, record each exit code and skip commit unless all are zero | +| Copying a v1 `.step({ failOnError, captureOutput })` example into a v2 flow | Those fields do not exist in v2; the spec will not compile | Use the evidence recorder; see **The v2 translation** | +| Gating a v2 agent step on `subprocess_gate` | A failure reports `exit=1` with empty stdout and stderr, and the verdict is unrecoverable | `artifact_exists`, or no gate plus a deterministic gate after the step | diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 55dba6d7..76109fb7 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -23,7 +23,7 @@ Relayflows turns a coding-agent task into steps a journal can inspect, verify, a Three step verbs, one per rung — never more (`packages/sdk/src/spec.ts`, `export type StepType = 'deterministic' | 'llm' | 'agent';`): -1. **`run` / `deterministic`** — a shell command. No model. Implicit gate is `exit_code == 0`. +1. **`run` / `deterministic`** — a shell command. No model. Implicit gate is `exit_code == 0`. It takes options: `f.run(cmd, { timeout: '5m' })`. **The default lease is 30 seconds and the maximum is 15 minutes** (`packages/surface/src/context.ts:60`) — invisible in most documentation and long enough to bite any command that touches the network. `git fetch`, `cargo build` and `gh pr create` all exceed it in their bad case, which is exactly when you least want an intermittent failure. 2. **`llm` / `llm`** — a bare model call. Prompt in, verified output out. No workspace, no tool use. 3. **`agent` / `agent`** — a harnessed coding agent in a workspace. Returns `{ summary, artifacts }`, not raw text. @@ -110,6 +110,22 @@ Do not add fields to `AgentOptions`/`Ctx` that aren't in this list — they don' `done(reason)` takes exactly one of the closed run-completion set (`packages/surface/src/completion.ts`): `'success' | 'step_failed' | 'canceled' | 'budget_exceeded'`. There's no fifth option — don't invent one (e.g. `'partial'`, `'skipped'`). +### `flow()` takes a header, and that is where a TypeScript budget goes + +`flow()` has three overloads (`packages/surface/src/flow.ts:57-61`), and the three-argument one is the one you want for anything long-running: + +```ts +export default flow( + 'relay.ci.pr-proof', + { budget: { wallclock: '110m' } }, // FlowHeader + async (f, input) => { ... } +); +``` + +`FlowHeader` carries `budget`, `workspace`, `tools` and `use?: string[]`. It does **not** carry `agents` — that throws `TypeError: flow header has unknown fields: agents` at authoring time. + +The header budget (`wallclock: '110m'`) and the spec budget (`budget: { maxWallclockMs: 6_600_000 }`) are different shapes for the same thing: `BudgetSpec | HeaderBudget` in `packages/sdk/src/spec.ts`. Write the duration form in TypeScript and the millisecond form in a generated spec. + ## The real step shapes (YAML/JSON, `packages/sdk/src/spec.ts`) ```ts @@ -162,9 +178,28 @@ interface FlowSpec { Verification is control flow, not decoration — a gate decides whether a step actually completed, not just whether the process exited cleanly (`packages/sdk/src/spec.ts`, `VerificationGateType`): +There are **eight**, not three (`packages/sdk/src/spec.ts`; the full list is also in `validate.ts`'s `unknown_gate_kind` refusal): + - `exit_code` — implicit default for `deterministic` steps. `exit_code == 0`. Not configurable in v0. - `output_contains` — step output (stdout tail or LLM value, stringified) contains `value`. - `json_schema` — step output validates against a JSON Schema (`boolean | Record`). Used for structured LLM/agent output. +- `regex_match` — `{ pattern, in_output_at?, flags? }`. RE2-safe (no catastrophic backtracking), inspectable before execution. The one to reach for when an agent must emit a specific receipt line. +- `artifact_exists` — `{ path }`, a working-directory-relative POSIX path. Passes when the step's **journaled** `artifacts` list names the path. Nothing on disk is consulted, so replay and resume see the recorded verdict. This is the right gate for "the agent must have written X". +- `subprocess_gate` — `{ command, from_output? }`. Runs a shell command; see the warning below before using it. +- `references_input` — `{ input_key, in_output_at? }`. The output must quote a value from the run input. +- `word_count_bounds` — `{ min?, max? }`. + +In TypeScript these attach **postfix**, with `.gate()` on the returned `Step` (`packages/surface/src/step.ts`) — one per step: + +```ts +await f + .agent('prover', { task: '...', cli: 'claude' }) + .gate({ type: 'regex_match', pattern: 'PROOF_COMPLETE arm=base', in_output_at: ['summary'] }); + +await f.agent('writer', { task: '...' }).gate({ type: 'artifact_exists', path: 'review/security.md' }); +``` + +`.gate()` also takes a predicate — `.gate(v => v.length > 0, 'must not be empty')` — but the closure cannot be inspected by `flows check` before the run. Prefer a `NamedGate` when the check should be visible up front. ```yaml - id: classify @@ -181,6 +216,34 @@ Verified for real: `flows check` on exactly this step passed (`CHECK PASSED`, ex An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. Always give an `agent`/`llm` step a real `verification`, not just the default. +### A failing `subprocess_gate` tells you nothing + +`subprocess_gate` looks like the general-purpose escape hatch. It has a trap worth knowing before you build a flow around it. + +The lowering (`packages/sdk/src/named-gate-lowering.ts`) spawns your command with `stdio: 'inherit'`, and the daemon's stdio is captured nowhere. When the gate fails you get, in the journal: + +```json +{"completionReason":"retries_exhausted","output":{"exit_code":1,"stderr_tail":"","stdout_tail":""}} +``` + +`relayflowd.log` is empty too, and nothing reaches the `flows run` output. A gate command that prints a detailed diagnosis on every path prints it into the void. Tracked at [flows#511](https://github.com/AgentWorkforce/flows/issues/511). + +Two consequences for how you author: + +- **Prefer `artifact_exists` or `regex_match` on agent steps.** Both decide in-process from the journal, so their verdicts are legible and replay-stable. +- **If you must use `subprocess_gate`, make the command write its own verdict to a file** under your artifact directory. That file is the only record you will get. + +More generally: put enforcement in a deterministic step *after* the agent rather than in a gate *on* the agent. A dropped agent transport then reads as "nothing was written" — a repairable fact — instead of as a crashed run. + +### There is no `failOnError: false` + +v2 gates every `deterministic` step on exit code with no opt-out. If you are porting a v1 flow, or writing a repair-before-failure flow where a red check is *work for an agent* rather than the end of the run, you need a pattern for it. The two in use: + +- `{ } || true` — the group braces are load-bearing, because `||` cannot begin a line, so appending `\n|| true` to a multi-line command is a shell syntax error rather than a fallback. This discards the exit code, so a later gate has nothing to read. +- **An evidence recorder** — a small script that runs the command, writes `{command, exitCode, verdict, tail}` to a file, and always exits 0; a later deterministic step reads those files back and decides. Verbose, but it is the only shape where a repair agent can see *why* a check was red and a final gate can recompute green from recordings rather than from an agent's report. + +Tracked at [flows#509](https://github.com/AgentWorkforce/flows/issues/509). + ## `cli` / `model`: what a step actually runs on Both YAML and TypeScript agent/llm steps can set `cli` and `model` directly (TypeScript since flows#310, `AgentOptions.cli?`/`.model?`). Resolution order for `cli` — checked once per step by `preflight.ts`'s `resolveCli` (`packages/sdk/src/preflight.ts:265-282`), identical regardless of authoring language because both compile to the same `StepSpec`: @@ -252,8 +315,31 @@ flows run [--json] [--no-spawn] [--no-observer-link] [--data-dir ] [--local flows resume [--json] [--no-spawn] [--no-observer-link] [--data-dir ] ``` +### A flow with an agent step needs `--local-agent` + +`flows run ` on a flow containing **any** `agent` step parks immediately at that step unless you pass `--local-agent`: + +``` +PARKED [run_parked] Run "01M2Z..." parked at step "implement-rust" (agent): no worker is attached for step type "agent". +RUN 01M2Z... parked (3 steps) # exit 3 +``` + +The flag is in the usage string but nothing says it is *required*; the message reads as "your infrastructure is missing a worker", not "pass this flag". Budget a wasted run for it the first time, or just always pass it for a local run with agents. + +Three details worth having in advance: + +- **The remedy is only named sometimes.** The hint `To start a new run with a local agent worker: flows run --local-agent ''` is appended only for `flows run` on a YAML/JSON spec path. A `.flow.ts` gets the bare message with no remedy, and `flows resume` never gets one (`packages/sdk/src/cli/run.ts:627-640`). +- **`flows resume --local-agent ` is a trap on a spec run.** The flag is accepted and then ignored: `resumeFlow` attaches a worker only on the authored-TS path, so a YAML/JSON run parks again with the identical message. Start a new run instead. ([flows#504](https://github.com/AgentWorkforce/flows/issues/504)) +- **`flows schedule` has no `--local-agent` at all.** A scheduled flow with an agent step parks forever, silently, on a timer. Drive it from cron with `flows run --local-agent` until that changes. ([flows#503](https://github.com/AgentWorkforce/flows/issues/503)) + +### `flows check` passing does not mean the flow will run + `check` is a pure compile-and-preflight — no daemon, no socket, no data dir. It's the fast, safe way to validate a flow before ever running it. `--json` works on `check`/`run`/`resume`; it does **not** exist on `tick start`, `hn-monitor start`, or `observer`. The printed usage above only lists `` for `check`, but it accepts `.flow.ts` too — verified in this skill's own **Verified against** section (`flows check hello.flow.ts` passes); the tool's own `--help` text is just incomplete on this point. +That is what makes it fast and worth running constantly. It also means **the one validation tool the product ships structurally cannot catch spec/daemon skew**: the npm packages and the runtime binary can carry the same version number and disagree about the schema, and you learn it as a mid-run `protocol_error` after the flow has already done work ([flows#489](https://github.com/AgentWorkforce/flows/issues/489), [flows#502](https://github.com/AgentWorkforce/flows/issues/502)). + +Practical consequence for authoring: **budget a shakedown run against a throwaway target, and treat green unit tests plus `CHECK PASSED` as necessary but not remotely sufficient.** The failures that actually block a flow live in the seams between the packages, the daemon, the CLI flags and the forge, and none of those are reachable without a real run. + A `.flow.ts` run via `flows run` requires `--input ` even when the flow body ignores its input argument — `flows run hello.flow.ts` alone refuses `REFUSED [input_missing]`. ### Exit codes and refusal shapes @@ -272,12 +358,17 @@ Both are real, both are exit 2 — just from different code paths, so don't be s - **Assuming `flows.json`'s `models` sets a default model.** It only validates models already declared elsewhere; it never selects one. - **Not awaiting a step, or manually `.then()`-chaining one.** Both are refused (`unawaited_step` / `unsupported_verb`) rather than silently ignored — the executor closes every root operation's lifecycle explicitly. - **Running a `.flow.ts` without `--input`.** Required even for flows that don't use their input argument. -- **Expecting a fifth `done()` reason.** The set is closed: `success | step_failed | canceled | budget_exceeded`. Don't invent `partial` or `skipped`. +- **Expecting a fifth `done()` reason.** The set is closed in this skill's examples as `success | step_failed | canceled | budget_exceeded`; the installed runtime has since added `needs_human` and `declined`. Check `RunCompletionReason` in the version you have rather than trusting either list. +- **Running a flow with agent steps without `--local-agent`.** It parks at the first agent step. See above. +- **Gating an agent step on `subprocess_gate`.** When it fails you get `exit=1` and nothing else. See above. +- **Assuming `f.run` has no timeout.** It leases for 30 seconds by default. ## What this skill does NOT cover - **Named-agent maps in TypeScript** (`agents: { reviewer: { cli, model } }` + reuse across steps by name) — YAML/JSON only today. Tracked for TS composition via `use:` at [flows#300](https://github.com/AgentWorkforce/flows/issues/300). -- **`recoveryMode`, `permissions`, `surfaces`, `budget`, `memory`** on agent steps — real YAML/JSON fields with no TypeScript equivalent. Author that step in YAML and reach it from TypeScript with `f.dispatch` if you need them. +- **`recoveryMode`, `surfaces`, `memory`** on agent steps — real YAML/JSON fields with no TypeScript equivalent. Author that step in YAML and reach it from TypeScript with `f.dispatch` if you need them. (`budget` *does* have a TypeScript home — the `FlowHeader` — and `AgentOptions` has grown `permissions`, `cwd` and `transport` since this skill was first written.) +- **Whether `permissions` does anything.** It is validated and journaled and **not enforced** — the kernel says so itself: *"Carried as data in gate 1; enforcement lands with agent dispatch."* Declare it so the intent is reviewable, but do not treat a `flows run` as sandboxed; an agent step can read and write the whole checkout. ([flows#487](https://github.com/AgentWorkforce/flows/issues/487), [flows#442](https://github.com/AgentWorkforce/flows/issues/442)) +- **`options.cwd` on an agent step.** Declared by `@relayflows/surface` and `@relayflows/sdk`, rejected by the runtime binary at the same version, with `invalid_spec: unknown field "cwd"`. Do not plan a per-step worktree around it yet. ([flows#489](https://github.com/AgentWorkforce/flows/issues/489)) - **Cloud execution** (`flows run --cloud`), **triggers/webhooks**, **memory retrieval**, and the **`f.mcp`**/**`f.slack`** helper namespaces — each is its own surface with its own gotchas; see the [Relayflows product docs](https://agentrelay.com/docs/relayflows) for what's shipped versus designed-but-not-yet-implemented. - The **older `@relayflows/core` `WorkflowBuilder`** engine — see `writing-agent-relay-workflows` and `migrating-persona-to-relayflow` in this repo. @@ -297,10 +388,15 @@ Both are real, both are exit 2 — just from different code paths, so don't be s | `flows check ` | CLI | pure validate + preflight, no daemon | | `flows run [--input ...]` | CLI | actually executes; `.flow.ts` needs `--input` | | `flows resume ` | CLI | resume a parked/crashed run | +| `.gate(config)` on a `Step` | TS | postfix named gate; one per step | +| `--local-agent` | CLI | **required** for any local run with an `agent` step | +| `f.run(cmd, { timeout })` | TS | **default lease 30s, max 15m** — set it for anything touching the network | ## Verified against -`AgentWorkforce/flows@86a2ec2` (origin/main). Built `packages/surface` and `packages/sdk` from source in a clean worktree (published npm `@relayflows/surface@2.0.8` is stale — it predates flows#310 and lacks `cli`/`model` on `AgentOptions`; local build was symlinked in instead), then ran the real CLI: +**Updated 2026-09-20** against `AgentWorkforce/flows@e73b315e` (origin/main, CLI 2.0.22) while authoring a 58-step campaign flow for `AgentWorkforce/relay` (`flows/migrate/native-delivery.spec.ts`). The `--local-agent` park, the silent `subprocess_gate` failure, the eight gate types, the `.gate()` postfix form, the three-argument `flow()` and the `f.run` 30-second lease were each read from the source cited beside them or observed in a real run — the park and the gate silence cost one run each. The `f.gitlab`/`f.github` writeback asymmetry, the `cwd` skew and the `retries_exhausted` behaviour come from the *Relayflows v2 Field Report* (2026-09-18, Julian Fann), corroborated against the same tree. + +Original verification, `AgentWorkforce/flows@86a2ec2` (origin/main). Built `packages/surface` and `packages/sdk` from source in a clean worktree (published npm `@relayflows/surface@2.0.8` is stale — it predates flows#310 and lacks `cli`/`model` on `AgentOptions`; local build was symlinked in instead), then ran the real CLI: ``` $ flows check hello.flow.yaml # this skill's YAML example, cli/model added, flows.json models allowlist set From f57606fae0ee6efd2474b4296c3b26a3cf0f6a74 Mon Sep 17 00:00:00 2001 From: Proactive Runtime Bot Date: Sun, 20 Sep 2026 09:28:53 -0700 Subject: [PATCH 02/12] feat(skills): artifact_exists dot-directory trap, and how to watch a run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit More friction from the same campaign, found after the PR was opened. writing-relayflows: - `artifact_exists` cannot see a dot-directory. It reads the worker's journaled `artifacts` list, which omits those paths. Measured: 6,837 artifacts journaled ({target: 6832, crates: 5}), zero under `.workflow-artifacts/`, while the agent had demonstrably written there. A run then died gating on a 25 KB file sitting on disk. Not a .gitignore effect — target/ is ignored too and is included. (AgentWorkforce/flows#513) With both named gates unusable on an agent step, the shape that works is the one the 80-to-100 skill already prescribes: a deterministic step after the agent, checking the disk. - How to watch a run: `flows status` is the surface and it is good; there is no observer link for a local run and no channel (#341, #264); `flows logs` is documented but refused in 2.0.22; `LEASE OVERDUE` is usually a display artifact; and the reported step count exceeds the authored one because the kernel adds a lowered `.gate` step per gated step. relay-80-100-workflow: - same artifact_exists note, and: gate the hazard, not a tool's mode. A gate asserting "the selector said targeted" rather than "no runtime path is unmapped" was unsatisfiable by construction, because registering a feature required editing the manifest and that forced full-smoke unconditionally. It failed correct work and read as the agent's fault. Co-Authored-By: Claude Opus 5 --- prpm.json | 6 +++--- skills/relay-80-100-workflow/SKILL.md | 6 +++++- skills/writing-relayflows/SKILL.md | 20 ++++++++++++++++++++ 3 files changed, 28 insertions(+), 4 deletions(-) diff --git a/prpm.json b/prpm.json index b47651ed..bd4c1e45 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.2.0", + "version": "1.2.1", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -95,7 +95,7 @@ }, { "name": "writing-relayflows", - "version": "1.1.0", + "version": "1.1.1", "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) \u2014 see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", "format": "claude", "subtype": "skill", @@ -154,7 +154,7 @@ }, { "name": "relay-80-100-workflow", - "version": "1.1.0", + "version": "1.1.1", "description": "Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, the v1-to-v2 translation (no failOnError/captureOutput/{{steps.X.output}}) and its evidence-recorder replacement, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", "format": "claude", "subtype": "skill", diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index cc25b867..952376ec 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -71,7 +71,9 @@ It is more machinery than `failOnError: false`, and it buys something v1 did not v1's `verification: { type: 'exit_code' }` on an agent step has no v2 equivalent, and the closest thing — `subprocess_gate` — is a trap: the lowering runs the command under `stdio: 'inherit'` and the daemon's stdio is captured nowhere, so a failure arrives as `exit=1` with empty `stdout_tail` **and** empty `stderr_tail`, nothing in `relayflowd.log`, and nothing in the CLI output ([flows#511](https://github.com/AgentWorkforce/flows/issues/511)). A 58-step campaign flow lost a full run to exactly this: the agent succeeded, its gate failed three times, and the verdict its gate command printed on every path reached nowhere. -Use `artifact_exists` (journal-based, replay-stable) where a file is the deliverable, and otherwise **no gate on the agent step at all** — put the enforcement in the deterministic recorded gate that follows it. That is the same advice as *Keep Repairable Gates On The Critical Path* below, and v2 makes it mandatory rather than merely wise. +`artifact_exists` is not the escape hatch either: it reads the worker's journaled `artifacts` list, which **omits dot-directories** — so it can never pass for a file under `.workflow-artifacts/` or similar ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)). Measured in one run: 6,837 artifacts journaled, zero under the artifact directory the agent had demonstrably written to. + +With both named gates unusable on an agent step, the answer is **no gate on the agent step at all** — put enforcement in the deterministic recorded step that follows it, checking the disk. That is the same advice as *Keep Repairable Gates On The Critical Path* below, and v2 makes it mandatory rather than merely wise. ### Two more v2 facts that will cost you a run each @@ -304,6 +306,8 @@ grep "my_new_table" packages/web/lib/db/schema.ts >/dev/null && echo "OK" || (ec edits: use `git status --short -- `, because `git diff --quiet` ignores untracked files +**Gate the hazard, not a tool's mode.** A gate that asserts *"the selector reported `targeted`"* rather than *"no runtime path is unmapped"* can be unsatisfiable by construction — in our case registering a feature required editing the manifest, and editing the manifest forced the selector into `full-smoke` unconditionally. The gate failed correct work, and it would have read as the agent's fault. Assert the condition you actually care about, from the tool's structured output, not its summary verdict. + **What NOT to verify:** - Exact content (too brittle — agents format differently) - Line counts or byte sizes (meaningless) diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 76109fb7..113367f9 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -235,6 +235,14 @@ Two consequences for how you author: More generally: put enforcement in a deterministic step *after* the agent rather than in a gate *on* the agent. A dropped agent transport then reads as "nothing was written" — a repairable fact — instead of as a crashed run. +### `artifact_exists` cannot see a dot-directory + +The other named gate you reach for on an agent step has its own blind spot. It reads the worker's journaled `artifacts` list, and that list **omits paths under dot-directories** — which is where artifact conventions usually put things. + +Measured in one run: an agent journaled 6,837 artifacts, `{target: 6832, crates: 5}`, and **zero** under `.workflow-artifacts/`, while provably having written a file there. A later step then died gating on a 25 KB review sitting on disk. It is not a `.gitignore` effect — `target/` is gitignored too and is included in full. ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)) + +So with both named gates unusable on an agent step, the shape that works is the same one: **a deterministic step after the agent that checks the disk and records its verdict.** Which is where enforcement belongs anyway. + ### There is no `failOnError: false` v2 gates every `deterministic` step on exit code with no opt-out. If you are porting a v1 flow, or writing a repair-before-failure flow where a red check is *work for an agent* rather than the end of the run, you need a pattern for it. The two in use: @@ -332,6 +340,18 @@ Three details worth having in advance: - **`flows resume --local-agent ` is a trap on a spec run.** The flag is accepted and then ignored: `resumeFlow` attaches a worker only on the authored-TS path, so a YAML/JSON run parks again with the identical message. Start a new run instead. ([flows#504](https://github.com/AgentWorkforce/flows/issues/504)) - **`flows schedule` has no `--local-agent` at all.** A scheduled flow with an agent step parks forever, silently, on a timer. Drive it from cron with `flows run --local-agent` until that changes. ([flows#503](https://github.com/AgentWorkforce/flows/issues/503)) +### Watching a run: `flows status`, and nothing else + +`flows status ` is the observability surface, and it is good — a live step table with per-step attempts, durations, gate verdicts, spend and transcript paths, plus per-attempt transcripts under `.relayflowd/runs//steps//`. Poll it with `watch`. + +Three things to know: + +- **There is no observer link for a local run** and no channel to watch it in. v2 has no relaycast channel concept at all, and the `--local-agent` observer URL is empty or fails to mint ([flows#341](https://github.com/AgentWorkforce/flows/issues/341), [flows#264](https://github.com/AgentWorkforce/flows/issues/264)). +- **`flows logs ` is documented but refused** `invalid_invocation` in 2.0.22. Don't build a workflow around it. +- **`LEASE OVERDUE by Nm` is usually a display artifact**, not a stall — the lease is not renewed while an agent is mid-turn. Check the worker process before concluding anything. + +And the step count will not match your spec: the kernel adds a lowered `.gate` step per gated step, so a 58-step spec reports 67. + ### `flows check` passing does not mean the flow will run `check` is a pure compile-and-preflight — no daemon, no socket, no data dir. It's the fast, safe way to validate a flow before ever running it. `--json` works on `check`/`run`/`resume`; it does **not** exist on `tick start`, `hn-monitor start`, or `observer`. The printed usage above only lists `` for `check`, but it accepts `.flow.ts` too — verified in this skill's own **Verified against** section (`flows check hello.flow.ts` passes); the tool's own `--help` text is just incomplete on this point. From 8637e4a7275eecb15549f887aa2aee3997f133b5 Mon Sep 17 00:00:00 2001 From: Proactive Runtime Bot Date: Sun, 20 Sep 2026 09:40:57 -0700 Subject: [PATCH 03/12] refactor(skills): lead with durable craft, quarantine bug workarounds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A skill that teaches "artifact_exists silently fails for dot-directories" is carrying a bug's water, and it rots the day the bug is fixed. The previous two commits mixed that material into the body. writing-relayflows: - New "Open bugs you must author around" section, stamped CLI 2.0.22 / 2026-09-20, one row per bug with its workaround and a delete-when issue link. Ten rows, all deletable independently. - The inline sections now carry only the lesson that survives the fix. The two broken gates collapse to "don't gate an agent step — check it in the step after", which is craft, not a workaround. - Watching a run keeps two durable facts: the reported step count exceeds the authored one (lowered .gate steps), and a gate script's own side-effect log is NOT a progress signal, because agents run those same commands on themselves. flows status is the only caller-accurate view. relay-80-100-workflow: - Same split on the agent-gate advice. - New: cross-check your gates against each other. Nothing validates that the assertions you write are mutually satisfiable. A seam-rules gate required a file that an edit-gate rejected as out-of-scope; no implementation could pass both, and the agent began reverting correct work to appease it. Also filed AgentWorkforce/flows#514: flows check already computes the facts that would have prevented five dead runs (it prints REQUIRES naming the agent steps) and spends none of them. Most of the quarantine section exists because of that one gap. Co-Authored-By: Claude Opus 5 --- prpm.json | 6 +-- skills/relay-80-100-workflow/SKILL.md | 12 ++++-- skills/writing-relayflows/SKILL.md | 60 ++++++++++++--------------- 3 files changed, 37 insertions(+), 41 deletions(-) diff --git a/prpm.json b/prpm.json index bd4c1e45..d911e88a 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.2.1", + "version": "1.3.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -95,7 +95,7 @@ }, { "name": "writing-relayflows", - "version": "1.1.1", + "version": "1.2.0", "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) \u2014 see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", "format": "claude", "subtype": "skill", @@ -154,7 +154,7 @@ }, { "name": "relay-80-100-workflow", - "version": "1.1.1", + "version": "1.2.0", "description": "Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, the v1-to-v2 translation (no failOnError/captureOutput/{{steps.X.output}}) and its evidence-recorder replacement, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", "format": "claude", "subtype": "skill", diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index 952376ec..c3632720 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -69,17 +69,21 @@ It is more machinery than `failOnError: false`, and it buys something v1 did not ### Don't gate agent steps in v2 -v1's `verification: { type: 'exit_code' }` on an agent step has no v2 equivalent, and the closest thing — `subprocess_gate` — is a trap: the lowering runs the command under `stdio: 'inherit'` and the daemon's stdio is captured nowhere, so a failure arrives as `exit=1` with empty `stdout_tail` **and** empty `stderr_tail`, nothing in `relayflowd.log`, and nothing in the CLI output ([flows#511](https://github.com/AgentWorkforce/flows/issues/511)). A 58-step campaign flow lost a full run to exactly this: the agent succeeded, its gate failed three times, and the verdict its gate command printed on every path reached nowhere. +v1's `verification: { type: 'exit_code' }` on an agent step has no v2 equivalent, and as of CLI 2.0.22 neither named gate is usable in its place: `subprocess_gate`'s output is not captured ([flows#511](https://github.com/AgentWorkforce/flows/issues/511)) and `artifact_exists` cannot see a dot-directory ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)). `writing-relayflows` carries the detail and the delete-when conditions. -`artifact_exists` is not the escape hatch either: it reads the worker's journaled `artifacts` list, which **omits dot-directories** — so it can never pass for a file under `.workflow-artifacts/` or similar ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)). Measured in one run: 6,837 artifacts journaled, zero under the artifact directory the agent had demonstrably written to. - -With both named gates unusable on an agent step, the answer is **no gate on the agent step at all** — put enforcement in the deterministic recorded step that follows it, checking the disk. That is the same advice as *Keep Repairable Gates On The Critical Path* below, and v2 makes it mandatory rather than merely wise. +Those two will be fixed. The rule that outlives them will not: **put enforcement in the deterministic recorded step that follows the agent, not in a gate on it.** It is the same rule as *Keep Repairable Gates On The Critical Path* below — a dropped agent transport should read as "nothing was written", which is repairable, rather than as a crashed run. v2 currently makes it mandatory; it was always the better shape. ### Two more v2 facts that will cost you a run each - **`flows run` parks at the first agent step without `--local-agent`.** The message reads as a missing worker, not a missing flag. - **`permissions` on an agent step is journaled and not enforced.** Declare it for reviewability; do not treat a `flows run` as sandboxed. Your implementation agents share one checkout, so partition their lanes by path *and* sequence them, and make an `edit-gate` that rejects out-of-scope changes. +### Cross-check your gates against each other + +Nothing validates that the assertions you write are mutually satisfiable, and agents pay for it when they are not. In one campaign a `seam-rules` gate **required** a file to be edited while an `edit-gate` **rejected** that same file as out-of-scope. No implementation could pass both; the agent began reverting correct work to appease the contradiction, and it read as the agent failing. + +If one gate names a path, assert at authoring time that every other gate permits it. That check is a few lines and it found a real contradiction immediately. + ## Core Principle: Test In The Workflow The key insight: **run tests as deterministic steps inside the workflow itself**. Don't just write test files — execute them, verify they pass, fix failures, and re-run. The workflow doesn't commit until tests are green. diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 113367f9..737fa5f8 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -216,32 +216,11 @@ Verified for real: `flows check` on exactly this step passed (`CHECK PASSED`, ex An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. Always give an `agent`/`llm` step a real `verification`, not just the default. -### A failing `subprocess_gate` tells you nothing +### Don't gate an agent step — check it in the step after -`subprocess_gate` looks like the general-purpose escape hatch. It has a trap worth knowing before you build a flow around it. +Neither named gate is usable on an agent step today: `subprocess_gate`'s output is not captured, and `artifact_exists` cannot see a dot-directory. Both are open bugs — see **Open bugs you must author around** below for the detail and the issue links. -The lowering (`packages/sdk/src/named-gate-lowering.ts`) spawns your command with `stdio: 'inherit'`, and the daemon's stdio is captured nowhere. When the gate fails you get, in the journal: - -```json -{"completionReason":"retries_exhausted","output":{"exit_code":1,"stderr_tail":"","stdout_tail":""}} -``` - -`relayflowd.log` is empty too, and nothing reaches the `flows run` output. A gate command that prints a detailed diagnosis on every path prints it into the void. Tracked at [flows#511](https://github.com/AgentWorkforce/flows/issues/511). - -Two consequences for how you author: - -- **Prefer `artifact_exists` or `regex_match` on agent steps.** Both decide in-process from the journal, so their verdicts are legible and replay-stable. -- **If you must use `subprocess_gate`, make the command write its own verdict to a file** under your artifact directory. That file is the only record you will get. - -More generally: put enforcement in a deterministic step *after* the agent rather than in a gate *on* the agent. A dropped agent transport then reads as "nothing was written" — a repairable fact — instead of as a crashed run. - -### `artifact_exists` cannot see a dot-directory - -The other named gate you reach for on an agent step has its own blind spot. It reads the worker's journaled `artifacts` list, and that list **omits paths under dot-directories** — which is where artifact conventions usually put things. - -Measured in one run: an agent journaled 6,837 artifacts, `{target: 6832, crates: 5}`, and **zero** under `.workflow-artifacts/`, while provably having written a file there. A later step then died gating on a 25 KB review sitting on disk. It is not a `.gitignore` effect — `target/` is gitignored too and is included in full. ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)) - -So with both named gates unusable on an agent step, the shape that works is the same one: **a deterministic step after the agent that checks the disk and records its verdict.** Which is where enforcement belongs anyway. +The durable lesson outlives both fixes: **put enforcement in a deterministic step after the agent, not in a gate on it.** A dropped agent transport then reads as "nothing was written" — a repairable fact — instead of as a crashed run, and the verdict is legible because a deterministic step journals its output. This is the same advice `relay-80-100-workflow` gives as *keep repairable gates on the critical path*; v2 makes it mandatory rather than merely wise. ### There is no `failOnError: false` @@ -340,17 +319,13 @@ Three details worth having in advance: - **`flows resume --local-agent ` is a trap on a spec run.** The flag is accepted and then ignored: `resumeFlow` attaches a worker only on the authored-TS path, so a YAML/JSON run parks again with the identical message. Start a new run instead. ([flows#504](https://github.com/AgentWorkforce/flows/issues/504)) - **`flows schedule` has no `--local-agent` at all.** A scheduled flow with an agent step parks forever, silently, on a timer. Drive it from cron with `flows run --local-agent` until that changes. ([flows#503](https://github.com/AgentWorkforce/flows/issues/503)) -### Watching a run: `flows status`, and nothing else - -`flows status ` is the observability surface, and it is good — a live step table with per-step attempts, durations, gate verdicts, spend and transcript paths, plus per-attempt transcripts under `.relayflowd/runs//steps//`. Poll it with `watch`. +### Watching a run -Three things to know: +`flows status ` is the observability surface, and it is good: a live step table with per-step attempts, durations, gate verdicts, spend and transcript paths, plus per-attempt transcripts under `.relayflowd/runs//steps//`. Poll it with `watch`. -- **There is no observer link for a local run** and no channel to watch it in. v2 has no relaycast channel concept at all, and the `--local-agent` observer URL is empty or fails to mint ([flows#341](https://github.com/AgentWorkforce/flows/issues/341), [flows#264](https://github.com/AgentWorkforce/flows/issues/264)). -- **`flows logs ` is documented but refused** `invalid_invocation` in 2.0.22. Don't build a workflow around it. -- **`LEASE OVERDUE by Nm` is usually a display artifact**, not a stall — the lease is not renewed while an agent is mid-turn. Check the worker process before concluding anything. +Two durable facts. **The step count will not match your spec** — the kernel adds a lowered `.gate` step per gated step, so a 58-step spec reports 67. And **a side-effect log written by your own gate scripts is not a progress signal**: agents run those same commands on themselves while working, so the log mixes flow steps, agent self-checks and your own manual runs. `flows status` is the only caller-accurate view. -And the step count will not match your spec: the kernel adds a lowered `.gate` step per gated step, so a 58-step spec reports 67. +For what is missing (observer link, `flows logs`) see the open-bugs section. ### `flows check` passing does not mean the flow will run @@ -387,8 +362,7 @@ Both are real, both are exit 2 — just from different code paths, so don't be s - **Named-agent maps in TypeScript** (`agents: { reviewer: { cli, model } }` + reuse across steps by name) — YAML/JSON only today. Tracked for TS composition via `use:` at [flows#300](https://github.com/AgentWorkforce/flows/issues/300). - **`recoveryMode`, `surfaces`, `memory`** on agent steps — real YAML/JSON fields with no TypeScript equivalent. Author that step in YAML and reach it from TypeScript with `f.dispatch` if you need them. (`budget` *does* have a TypeScript home — the `FlowHeader` — and `AgentOptions` has grown `permissions`, `cwd` and `transport` since this skill was first written.) -- **Whether `permissions` does anything.** It is validated and journaled and **not enforced** — the kernel says so itself: *"Carried as data in gate 1; enforcement lands with agent dispatch."* Declare it so the intent is reviewable, but do not treat a `flows run` as sandboxed; an agent step can read and write the whole checkout. ([flows#487](https://github.com/AgentWorkforce/flows/issues/487), [flows#442](https://github.com/AgentWorkforce/flows/issues/442)) -- **`options.cwd` on an agent step.** Declared by `@relayflows/surface` and `@relayflows/sdk`, rejected by the runtime binary at the same version, with `invalid_spec: unknown field "cwd"`. Do not plan a per-step worktree around it yet. ([flows#489](https://github.com/AgentWorkforce/flows/issues/489)) +- **`permissions` and `options.cwd`** — both are accepted and neither works as written. See **Open bugs you must author around**. - **Cloud execution** (`flows run --cloud`), **triggers/webhooks**, **memory retrieval**, and the **`f.mcp`**/**`f.slack`** helper namespaces — each is its own surface with its own gotchas; see the [Relayflows product docs](https://agentrelay.com/docs/relayflows) for what's shipped versus designed-but-not-yet-implemented. - The **older `@relayflows/core` `WorkflowBuilder`** engine — see `writing-agent-relay-workflows` and `migrating-persona-to-relayflow` in this repo. @@ -412,6 +386,24 @@ Both are real, both are exit 2 — just from different code paths, so don't be s | `--local-agent` | CLI | **required** for any local run with an `agent` step | | `f.run(cmd, { timeout })` | TS | **default lease 30s, max 15m** — set it for anything touching the network | +## Open bugs you must author around + +**Verified against CLI 2.0.22 on 2026-09-20.** Everything in this section describes a bug, not a design. Each row says when to delete it — check the issue before trusting the workaround, and remove the row once it closes. Nothing else in this skill depends on these. + +| Symptom | Workaround | Delete when | +| --- | --- | --- | +| A failing `subprocess_gate` journals `exit=1` with **empty** stdout and stderr; `relayflowd.log` is empty too. The lowering runs your command under `stdio: 'inherit'` and the daemon's stdio is captured nowhere. | Don't gate agent steps. If you must, have the command write its verdict to a file — that file is the only record you get. | [flows#511](https://github.com/AgentWorkforce/flows/issues/511) | +| `artifact_exists` can never pass for a path under a dot-directory. The worker's journaled `artifacts` list omits them — one run journaled 6,837 paths, `{target: 6832, crates: 5}`, and zero under `.workflow-artifacts/` it had demonstrably written to. Not a `.gitignore` effect; `target/` is ignored too and included in full. | Check the disk from a deterministic step instead. | [flows#513](https://github.com/AgentWorkforce/flows/issues/513) | +| `flows resume --local-agent ` accepts the flag and ignores it on a spec run, parking again identically. | Start a new run. Use `flows run --reuse-from ` to reuse completed steps — it keys on `step_spec_hash` plus resolved input, so an edited step re-executes and the rest do not. | [flows#504](https://github.com/AgentWorkforce/flows/issues/504) | +| `flows schedule` has no `--local-agent`, so a scheduled agent flow parks forever, silently, on a timer. | Drive it from cron with `flows run --local-agent`. | [flows#503](https://github.com/AgentWorkforce/flows/issues/503) | +| `flows logs ` is documented but refused `invalid_invocation`. | `flows status`, and the transcripts under `.relayflowd/runs/`. | — | +| No observer link and no channel for a local run. | `flows status`; there is no shareable live view. | [flows#341](https://github.com/AgentWorkforce/flows/issues/341), [flows#264](https://github.com/AgentWorkforce/flows/issues/264) | +| `LEASE OVERDUE by Nm` in `flows status` usually means an agent is mid-turn, not that anything is stuck. | Check the worker process before concluding a stall. | — | +| `permissions` on an agent step is validated and journaled and **never enforced**. A `flows run` is not sandboxed; agent steps can read and write the whole checkout. | Declare it for reviewability. Partition agents by path *and* sequence them, and gate on out-of-scope changes yourself. | [flows#487](https://github.com/AgentWorkforce/flows/issues/487) | +| `options.cwd` on an agent step is declared by surface+sdk and rejected by the runtime at the same version. | Don't plan a per-step worktree around it. | [flows#489](https://github.com/AgentWorkforce/flows/issues/489) | +| `retries_exhausted` reports only the last attempt, hiding the first — which is usually the one that explains the failure. | Read the journal directly. | [flows#506](https://github.com/AgentWorkforce/flows/issues/506) | +| `flows check` never contacts the daemon, so a green check says nothing about whether the daemon will accept the spec, and it stays silent on run-time properties it can already compute. | Budget a shakedown run against a throwaway target. Treat green unit tests plus `CHECK PASSED` as necessary and nowhere near sufficient. | [flows#502](https://github.com/AgentWorkforce/flows/issues/502), [flows#514](https://github.com/AgentWorkforce/flows/issues/514) | + ## Verified against **Updated 2026-09-20** against `AgentWorkforce/flows@e73b315e` (origin/main, CLI 2.0.22) while authoring a 58-step campaign flow for `AgentWorkforce/relay` (`flows/migrate/native-delivery.spec.ts`). The `--local-agent` park, the silent `subprocess_gate` failure, the eight gate types, the `.gate()` postfix form, the three-argument `flow()` and the `f.run` 30-second lease were each read from the source cited beside them or observed in a real run — the park and the gate silence cost one run each. The `f.gitlab`/`f.github` writeback asymmetry, the `cwd` skew and the `retries_exhausted` behaviour come from the *Relayflows v2 Field Report* (2026-09-18, Julian Fann), corroborated against the same tree. From c73762e4f5c18f68aa02620fb2577174c212da3b Mon Sep 17 00:00:00 2001 From: Proactive Runtime Bot Date: Sun, 20 Sep 2026 12:47:25 -0700 Subject: [PATCH 04/12] feat(skills): the lessons from running a 67-step 80-to-100 workflow for real MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Everything here was paid for once. **Green gates are necessary and nowhere near sufficient.** A tree passed every deterministic gate — scope, manifest routing, targeted selection, invariant tests with a mutation transcript, fmt/clippy/release build, typecheck, 3,395 unit tests, all five E2E parity suites — and a fresh adversarial reviewer then returned 15 findings, 2 disqualifying, verdict "do not seal". Gates check properties you thought to name; the review catches what you did not. Moved this to the top, because it reframes everything under it. **A requirement that induces a hazard must gate that hazard.** The workflow demanded proof that tests bite (mutate, capture the failure, restore). An implementer skipped the restore, shipping a mutation probe on a live delivery path whose `&body[..1]` panics on a multi-byte first char. Every gate passed it. If you ask an agent to temporarily break something, gate that it was put back — and prove that gate bites. Plus four smaller rules: give every repair owner an explicit "if green, do nothing" clause (one burned 21 minutes inventing work against a 2-7 minute norm; 4 of 7 prompts lacked it); seal the product tree, not just the evidence directory, or your reviewer is bound to a hash of the wrong thing; verify at the granularity the risk lives at (256 test functions before and after, ~90 assertions deleted from inside them); and retry a flaky gate once while recording which attempt passed, so a suite that only ever passes on retry stays visible. Co-Authored-By: Claude Opus 5 --- skills/relay-80-100-workflow/SKILL.md | 50 +++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index c3632720..1995259a 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -84,6 +84,16 @@ Nothing validates that the assertions you write are mutually satisfiable, and ag If one gate names a path, assert at authoring time that every other gate permits it. That check is a few lines and it found a real contradiction immediately. +## Green gates are necessary and nowhere near sufficient + +Before the mechanics, the finding that reframes all of them. + +A 67-step workflow of this exact shape produced a tree on which **every deterministic gate was green** — scope, feature-manifest routing, targeted-verification selection, invariant tests with a mutation transcript, `cargo fmt`/`clippy -D warnings`/release build, typecheck, 3,395 unit tests, and all five end-to-end parity suites. A fresh adversarial reviewer then read that tree and returned **15 findings, 2 of them disqualifying**, with the verdict *"do not seal"*: a mutation probe shipping on a live delivery path that could panic the process, a seam whose coordinator was rebuilt per call so three of its four contract invariants had no production effect, and an adapter that had stopped observing whether its write landed — so a dead writer read as a successful delivery. + +None of that is reachable by a gate. Gates check *properties you thought to name*. The review catches *the thing you did not think to name* — and on this run, that was most of the risk. + +So: build the gates, and do not treat them as the finish line. The review/fix rounds below are not a formality bolted onto a working pipeline; on the evidence, they are where the defects were actually found. + ## Core Principle: Test In The Workflow The key insight: **run tests as deterministic steps inside the workflow itself**. Don't just write test files — execute them, verify they pass, fix failures, and re-run. The workflow doesn't commit until tests are green. @@ -594,6 +604,46 @@ Output: .run({ cwd: process.cwd() }); ``` +## A requirement that induces a hazard must gate that hazard + +This one cost a near-miss, and it generalises further than it looks. + +The workflow required proof that tests bite: *mutate the guarded code, capture the failing transcript, restore it.* An implementer did the first two steps and skipped the third, leaving this in shipping product code on a live delivery path: + +```rust +// MUTATION: drop the addressee and truncate the body. +if std::env::var("RELAY_MUTATION_LOSSY_FORMAT").is_ok() { + return format!("Relay message from {}:\n\n{}", delivery.from, &delivery.body[..1]); +} +``` + +`&body[..1]` panics on a multi-byte first character. + +**Every deterministic gate passed that tree.** The edit gate saw a changed file in scope; the invariant gate saw four correctly-named tests and a mutation transcript; typecheck, clippy, the release build and all five end-to-end parity suites were green. Only the adversarial reviewer caught it. + +The instruction "prove the test fails," read literally by an agent, is an instruction to damage production code. If you ask for that proof, you must also gate the residue: + +```js +// refuse mutation scaffolding anywhere in product source +for (const file of walk('src').filter(f => f.endsWith('.rs'))) { + if (/MUTATION|_MUTATION_/.test(readFileSync(file, 'utf8'))) problems.push(`residue in ${file}`); +} +``` + +Then prove *that* gate bites, by reinstating a one-line probe and watching it go red. + +Generalise it: **any instruction that tells an agent to temporarily break something needs a matching check that it was put back.** Temporarily lowering a timeout, stubbing a provider, disabling a guard — same shape, same hazard. + +## Four smaller rules, each learned the expensive way + +**Give every repair owner an explicit "if the gate is green, do nothing" clause.** A repair agent handed a green gate will invent work rather than conclude there is none — one burned 21 minutes on a gate reporting `not-required` against a 2–7 minute norm, because its prompt said "make the scenario real" with no branch for the passing case. Audit them: 4 of 7 in a mature workflow were missing it. + +**Seal the product tree, not just the evidence directory.** If your signoff step binds a reviewer to an artifact digest, digest the changed source files too. Otherwise the reviewer is bound to a hash of *evidence files* while the code those files describe can change underneath them. Prove it bites: a one-line source edit must change the digest. + +**Verify at the granularity the risk lives at.** Checking "256 test functions before, 256 after" and concluding nothing was deleted is wrong — roughly 90 assertions had been removed from *inside* those functions. Count what can actually be weakened, not the container it sits in. + +**Retry a flaky gate once, and record which attempt passed.** Contention between suites is not a regression; a run 35 steps deep died to a `Verified: 2/3, Failed: 0` that passed 3/3 standalone three times. A retry does not weaken the gate — the command must still pass — but the recorded tail has to say `passed on attempt 2`, so a suite that only ever passes on retry stays visible instead of being quietly laundered green. + ## Checklist: Is Your Workflow 80-to-100? | Check | How | From d32a85d03961062f2a1115646841a6e8f4545404 Mon Sep 17 00:00:00 2001 From: Proactive Runtime Bot Date: Sun, 20 Sep 2026 12:47:38 -0700 Subject: [PATCH 05/12] chore(skills): bump relay-80-100-workflow to 1.3.0 The version bump was missed on c73762e; the SKILL.md changes shipped without it. Co-Authored-By: Claude Opus 5 --- prpm.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/prpm.json b/prpm.json index d911e88a..611ef1a7 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.3.0", + "version": "1.4.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -154,7 +154,7 @@ }, { "name": "relay-80-100-workflow", - "version": "1.2.0", + "version": "1.3.0", "description": "Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, the v1-to-v2 translation (no failOnError/captureOutput/{{steps.X.output}}) and its evidence-recorder replacement, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", "format": "claude", "subtype": "skill", From bbc4aa1cea97b7758c8ac450a8826386c14b64eb Mon Sep 17 00:00:00 2001 From: Khaliq Date: Sun, 20 Sep 2026 17:36:34 -0700 Subject: [PATCH 06/12] feat(skills): rewrite relay-80-100-workflow as a Relayflows v2 skill v2 is now the assumed and only engine. Every example is v2-native (`f.run`/`f.agent`, `flow()` header budget, `dependsOn`, `maxIterations`, `--local-agent`); the v1 builder survives as one migration note, not as the skill's spine. The evidence recorder is now the spine: a script that runs a command, journals {name, command, exitCode, verdict, tail, runId} and always exits 0, plus a require-green step that recomputes the verdict from the recordings. That is what replaces failOnError, captureOutput and {{steps.X.output}} at once, and it carries the run stamp that stops --reuse-from resurrecting a previous run's verdict. The campaign lessons are kept and sharpened, with their evidence: green gates are necessary and nowhere near sufficient (15 findings, 2 disqualifying, on an all-green tree) leads the skill; a requirement that induces a hazard must gate that hazard (the shipped &body[..1] mutation probe); a test that guards a helper does not guard the call site; repair owners need a "do nothing" clause; seal the product tree; verify at the granularity the risk lives at; retry a flaky gate once and record which attempt passed; cross-check gates against each other. Gate-set specifics now cross-reference writing-relayflows instead of being restated here, since the shipped set moves between releases. 675 -> 309 lines. Co-Authored-By: Claude Opus 5 --- README.md | 4 +- prpm.json | 7 +- skills/relay-80-100-workflow/SKILL.md | 790 +++++++------------------- 3 files changed, 218 insertions(+), 583 deletions(-) diff --git a/README.md b/README.md index 82abf45d..dd18aadd 100644 --- a/README.md +++ b/README.md @@ -11,10 +11,10 @@ Package metadata lives in [prpm.json](prpm.json). The repo currently publishes ` | [choosing-swarm-patterns](skills/choosing-swarm-patterns/SKILL.md) | 1.1.5 | Pick the right Agent Relay orchestration pattern across the 10 core swarm patterns plus specialized patterns. | | [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 1.6.24 | Build multi-agent workflows with WorkflowBuilder, DAG dependencies, Relayfile-backed Slack human assistance, integration subscriptions, waitFor gates, review-depth review/fix loops, channels, and chat-native coordination recipes. | | [setting-up-relayfile](skills/setting-up-relayfile/SKILL.md) | 1.1.1 | Set up Relayfile mounts and writeback for provider files through local filesystem access. | -| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.0.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | +| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.2.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | | [using-agent-relay](skills/using-agent-relay/SKILL.md) | 1.4.0 | Participant-side MCP reference for a **registered** relay agent (spawned worker / registered lead): messaging, channels, threads, reactions, search, inbox, actions, and worker spawn/release. Counterpart to `orchestrating-agent-relay`. | | [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.3.1 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention. | -| [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 1.0.9 | Author workflows that close the 80-to-100 validation gap with repair-aware test, verify, review-depth review/fix with test hardening, and commit gates. | +| [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 2.0.0 | Close the 80-to-100 validation gap in a Relayflows v2 flow: the evidence recorder that turns a red check into work for a repair agent, repairable gates on the critical path, edit and hazard gates, and the fresh-eyes review rounds that catch what green gates miss. | | [review-fix-signoff-loop](skills/review-fix-signoff-loop/SKILL.md) | 1.0.2 | Loop review, repair, validation, and fresh-context dual-agent signoff until independent reviewers both satisfy the verdict contract. | | [trigger-autocomplete-catalog](skills/trigger-autocomplete-catalog/SKILL.md) | 1.0.0 | Enforce webhook/event trigger autocomplete coverage through KNOWN_TRIGGER_CATALOG in @relayfile/adapter-core. | | [activity-summary](skills/activity-summary/SKILL.md) | 1.0.0 | Answer "what did I work on yesterday" questions by reading `digests/yesterday.md` first instead of crawling provider directories. | diff --git a/prpm.json b/prpm.json index 611ef1a7..90514b81 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.4.0", + "version": "1.5.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -154,11 +154,12 @@ }, { "name": "relay-80-100-workflow", - "version": "1.3.0", - "description": "Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging - covers the 80-to-100 pattern with review-depth fresh-eyes review/fix loops with test hardening, the v1-to-v2 translation (no failOnError/captureOutput/{{steps.X.output}}) and its evidence-recorder replacement, PGlite in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates, and full lifecycle from implementation through passing tests to commit", + "version": "2.0.0", + "description": "Use when writing a Relayflows v2 flow (@relayflows/surface / @relayflows/sdk, CLI `flows`) that must fully validate a feature end-to-end before it is sealed or merged - the 80-to-100 gap. Covers the evidence-recorder pattern that replaces v1's failOnError/captureOutput, repairable gates on the critical path, edit and hazard gates, gate cross-checking, fresh-eyes review rounds, and what a wall of green gates still misses.", "format": "claude", "subtype": "skill", "tags": [ + "relayflows", "workflow", "testing", "e2e-validation", diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index 1995259a..943b158d 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -1,675 +1,309 @@ --- name: relay-80-100-workflow -description: Use when writing agent-relay or Relayflows v2 workflows that must fully validate features end-to-end before merging. Covers the 80-to-100 pattern - going beyond "code compiles" to "feature works, tested E2E locally." Includes the v1-to-v2 translation (v2 has no failOnError, captureOutput or {{steps.X.output}}) and the evidence-recorder pattern that replaces them, repair-before-failure validation gates, review-depth fresh-eyes review/fix loops with test hardening, PGlite for in-memory Postgres testing, mock sandbox patterns, test-fix-rerun loops, verify gates after every edit, and the full lifecycle from implementation through passing tests to commit. +description: Use when writing a Relayflows v2 flow (@relayflows/surface / @relayflows/sdk, CLI `flows`) that must fully validate a feature end-to-end before it is sealed, committed, or merged - the 80-to-100 gap. Covers the evidence-recorder pattern that makes a red check work for an agent instead of the end of the run, repairable gates on the critical path, edit and hazard gates, fresh-eyes review rounds, and what a wall of green gates still misses. --- -# Writing 80-to-100 Validated Workflows +# Writing 80-to-100 Validated Flows ## Overview -Most agent workflows get features to ~80%: code written, types check, maybe a build passes. This skill covers the **80-to-100 gap** — making workflows that fully validate features end-to-end before committing. The goal: every feature merged via these workflows is **tested, verified, and known-working**, not just "it compiles." +Most agent workflows get a feature to ~80%: code written, types check, maybe a build passes. This skill covers the **80-to-100 gap** — making a flow that fully validates a feature before it commits anything. The goal is that everything these flows produce is **tested, verified and known-working**, not just "an agent said it was done." -## When to Use +Everything here is written for **Relayflows v2** — `@relayflows/surface` / `@relayflows/sdk`, CLI `flows`. For the authoring surface itself (the ladder, gates, `cli`/`model` resolution, `flows.json`, refusal shapes) see `writing-relayflows`. This skill is only about the validation shape you build on top of it. -- Writing workflows where the deliverable must be **production-ready**, not just code-complete -- Features that touch databases, APIs, or infrastructure that can be tested locally -- Any workflow where "it compiles" is not sufficient proof of correctness -- When you want confidence that the commit actually works before deploying +> Migrating from the old `@relayflows/core` `WorkflowBuilder`? v2 has no `failOnError`, no `captureOutput`, no `{{steps.X.output}}` templating and no `.onError()`. Every deterministic step is gated on exit code with no opt-out. **The evidence recorder** below is what replaces all four. See `writing-agent-relay-workflows` for the old engine. -## Which engine are you writing for? +## When to use -**Every code example below is the v1 builder** — `@agent-relay/sdk/workflows`'s `workflow(...).step(...)`, with `failOnError`, `captureOutput`, `{{steps.X.output}}` and `.onError()`. The patterns are engine-independent; those four knobs are not. If you are writing a **Relayflows v2** flow (`@relayflows/surface` / `@relayflows/sdk`, CLI `flows`), read the next section before copying anything — a v2 spec has none of them and following these examples literally produces something that will not compile. +- The deliverable must be **production-ready**, not just code-complete. +- The feature touches databases, APIs or infrastructure that can be exercised locally. +- "It compiles" is not sufficient proof of correctness. +- You want the commit at the end of the flow to represent code that is proven working. -See `writing-relayflows` for the v2 authoring surface itself. - -## The v2 translation - -v2 has three step verbs (`deterministic` / `llm` / `agent`) and gates every deterministic step on exit code **with no opt-out**. That single fact is what every translation below works around. - -| v1 | v2 | -| --- | --- | -| `.timeout(ms)` | `budget.maxWallclockMs`, or `{ budget: { wallclock: '110m' } }` in a TS `FlowHeader` | -| `retries: n` | `maxIterations: n + 1` | -| `failOnError: false` | **nothing** — see the recorder below | -| `captureOutput: true` | **nothing** — output is journaled, but not addressable from a later step's command | -| `{{steps.X.output}}` | **nothing** — no template interpolation between steps | -| `verification: { type: 'file_exists' }` | the `artifact_exists` named gate | -| `verification: { type: 'exit_code' }` on an agent step | no equivalent — see "don't gate agent steps" | -| `.pattern('dag')`, `.maxConcurrency(n)` | nothing; `dependsOn` is the only ordering | -| `.onError('retry')` | nothing; a failed step fails the run (it is resumable with `flows resume`) | -| `.channel(...)`, `.idleNudge(...)`, `preset`/`role` | nothing | - -### The evidence recorder replaces `failOnError` + `captureOutput` + `{{steps.X.output}}` - -All three v1 knobs existed to serve one pattern: run a check, let it be red, hand its output to the agent built to answer it. In v2 you rebuild that with a small script of your own: - -```js -// gates.mjs record --name --command-base64 -// Runs the command, writes {name, command, exitCode, verdict, tail} to -// evidence/.json, and ALWAYS exits 0. -// -// gates.mjs require-green --names a,b,c -// Reads those files back. This is the only thing that says "green". -``` - -The flow then reads: - -```ts -det('unit-tests', record('unit-tests', 'npx vitest run')); // always exits 0 -agentStep({ id: 'repair-ts', agent: 'fixer', dependsOn: ['unit-tests'], - task: ['Read evidence/unit-tests.json. Green means do nothing.', - 'Red means fix it and rerun until the recorder writes green.'] }); -det('ts-final', record('unit-tests', 'npx vitest run'), ['repair-ts']); -det('ts-assert', gate('require-green', '--names unit-tests'), ['ts-final']); // this one may fail -``` - -It is more machinery than `failOnError: false`, and it buys something v1 did not have: the repair agent reads a **file**, not an interpolated string, so it sees the whole tail; and final acceptance recomputes the verdict from recordings rather than trusting any agent's report. Base64-encode the command — it will contain `&&`, pipes and quotes, and the calling shell must not reinterpret them. +## Green gates are necessary and nowhere near sufficient -`|| true` is the other option. It works and it throws the exit code away, which means a later gate has nothing to read and a forgotten gate ships red work silently. Tracked upstream at [flows#509](https://github.com/AgentWorkforce/flows/issues/509). +Read this before the mechanics; it reframes all of them. -### Don't gate agent steps in v2 +A 67-step flow of exactly this shape produced a tree on which **every deterministic gate was green** — scope, feature-manifest routing, targeted-verification selection, invariant tests with a mutation transcript, `cargo fmt`, `clippy -D warnings`, a release build, typecheck, 3,395 unit tests and all five end-to-end parity suites. A fresh adversarial reviewer then read that tree and returned **15 findings, 2 of them disqualifying**, with the verdict *"do not seal"*: -v1's `verification: { type: 'exit_code' }` on an agent step has no v2 equivalent, and as of CLI 2.0.22 neither named gate is usable in its place: `subprocess_gate`'s output is not captured ([flows#511](https://github.com/AgentWorkforce/flows/issues/511)) and `artifact_exists` cannot see a dot-directory ([flows#513](https://github.com/AgentWorkforce/flows/issues/513)). `writing-relayflows` carries the detail and the delete-when conditions. +- a mutation probe left shipping on a live delivery path, able to panic the process; +- a seam whose coordinator was rebuilt per call, so three of its four contract invariants had no production effect; +- an adapter that had stopped observing whether its write landed, so a dead writer read as a successful delivery. -Those two will be fixed. The rule that outlives them will not: **put enforcement in the deterministic recorded step that follows the agent, not in a gate on it.** It is the same rule as *Keep Repairable Gates On The Critical Path* below — a dropped agent transport should read as "nothing was written", which is repairable, rather than as a crashed run. v2 currently makes it mandatory; it was always the better shape. +None of that is reachable by a gate. Gates check *properties you thought to name*. Review catches *the thing you did not think to name* — and on that run, that was most of the risk. -### Two more v2 facts that will cost you a run each +So: build the gates, and do not treat them as the finish line. The review rounds below are not a formality bolted onto a working pipeline. They are where the defects were actually found. -- **`flows run` parks at the first agent step without `--local-agent`.** The message reads as a missing worker, not a missing flag. -- **`permissions` on an agent step is journaled and not enforced.** Declare it for reviewability; do not treat a `flows run` as sandboxed. Your implementation agents share one checkout, so partition their lanes by path *and* sequence them, and make an `edit-gate` that rejects out-of-scope changes. +## The evidence recorder -### Cross-check your gates against each other +v2 fails a deterministic step the moment its command exits non-zero, and there is no opt-out. But an 80-to-100 flow does not want a red test run to end the run — a red test run is **work for the repair agent**. `{ … } || true` discards the exit code, which leaves a later gate nothing to read and lets a forgotten gate ship red work silently. -Nothing validates that the assertions you write are mutually satisfiable, and agents pay for it when they are not. In one campaign a `seam-rules` gate **required** a file to be edited while an `edit-gate` **rejected** that same file as out-of-scope. No implementation could pass both; the agent began reverting correct work to appease the contradiction, and it read as the agent failing. +The shape that works is a small script of your own that journals verdicts to files: -If one gate names a path, assert at authoring time that every other gate permits it. That check is a few lines and it found a real contradiction immediately. +```js +// gates.mjs — the entire contract, in two subcommands. +// +// node gates.mjs record --name unit --command-base64 [--retries 1] +// Decodes and runs the command, writes evidence/.json: +// { name, command, exitCode, verdict: 'green' | 'red', +// attempts, passedOnAttempt, tail, runId } +// and ALWAYS exits 0. +// +// node gates.mjs require-green --names unit,clippy,e2e +// Reads those files back and exits non-zero, printing the red tails, +// if any verdict is not 'green' or any runId is not this run's. +``` -## Green gates are necessary and nowhere near sufficient +Three details are load-bearing: -Before the mechanics, the finding that reframes all of them. +- **Base64-encode the command.** It will contain `&&`, pipes and quotes, and the calling shell must not reinterpret them. +- **Stamp each record with the run that produced it** (`runId`, from the flow's input or an env var). `flows run --reuse-from ` keys on `step_spec_hash`, and a recorder step *always exits 0*, so it is always eligible for reuse — an unstamped recording resurrects a previous run's verdict, and `require-green` happily reads it as today's evidence. +- **`require-green` is the only thing that says "green."** Final acceptance recomputes the verdict from recordings; it never trusts an agent's report of its own work. -A 67-step workflow of this exact shape produced a tree on which **every deterministic gate was green** — scope, feature-manifest routing, targeted-verification selection, invariant tests with a mutation transcript, `cargo fmt`/`clippy -D warnings`/release build, typecheck, 3,395 unit tests, and all five end-to-end parity suites. A fresh adversarial reviewer then read that tree and returned **15 findings, 2 of them disqualifying**, with the verdict *"do not seal"*: a mutation probe shipping on a live delivery path that could panic the process, a seam whose coordinator was rebuilt per call so three of its four contract invariants had no production effect, and an adapter that had stopped observing whether its write landed — so a dead writer read as a successful delivery. +Compared to the old `failOnError: false` this is more machinery, and it buys something the old engine did not have: the repair agent reads a **file**, not an interpolated string, so it sees the whole tail rather than a truncated template. -None of that is reachable by a gate. Gates check *properties you thought to name*. The review catches *the thing you did not think to name* — and on this run, that was most of the risk. +### The four-step ladder -So: build the gates, and do not treat them as the finish line. The review/fix rounds below are not a formality bolted onto a working pipeline; on the evidence, they are where the defects were actually found. +Every meaningful gate is four steps: record, repair, re-record, assert. -## Core Principle: Test In The Workflow +```ts +import { flow } from '@relayflows/surface'; + +const b64 = (cmd: string) => Buffer.from(cmd, 'utf8').toString('base64'); +const record = (name: string, cmd: string) => + `node gates.mjs record --name ${name} --command-base64 ${b64(cmd)}`; +const requireGreen = (...names: string[]) => + `node gates.mjs require-green --names ${names.join(',')}`; + +export default flow('ship-feature', { budget: { wallclock: '110m' } }, async (f) => { + await f.run(record('unit', 'npx vitest run'), { timeout: '15m' }); // exits 0 either way + + await f.agent('repair-unit', { + cli: 'claude', + task: [ + 'Read evidence/unit.json.', + 'If verdict is "green", do nothing and say so. Do not invent work.', + 'If verdict is "red", read the tail, fix the source or the test,', + 'and rerun `npx vitest run` until it passes.', + ].join('\n'), + }); -The key insight: **run tests as deterministic steps inside the workflow itself**. Don't just write test files — execute them, verify they pass, fix failures, and re-run. The workflow doesn't commit until tests are green. + await f.run(record('unit', 'npx vitest run'), { timeout: '15m' }); + await f.run(requireGreen('unit')); // the only step here allowed to be red + f.done('success'); +}); ``` -implement → write tests → run tests → fix failures → re-run → build check → regression check → commit -``` - -This means the commit at the end of the workflow represents code that is **proven working**, not just code that an agent wrote and claimed works. - -## Repair Before Failure - -An 80-to-100 workflow should not stop merely because a test, typecheck, lint, schema, or E2E gate turns red. That red output is work for the agent team. Capture it, hand it to a repair owner, fix it, and rerun. Workflow-owned validation gates should never terminate the run with `FAILED`. If the team exhausts its repair budget or hits an external blocker such as missing credentials, wrong repository, or unsafe dirty worktree, write a `BLOCKED_NO_COMMIT` artifact and end without committing or opening a PR instead of crashing the workflow. - -Use this shape for every meaningful gate: -1. `run-*`: deterministic command with `captureOutput: true` and `failOnError: false`. -2. `fix-*`: agent step that reads `{{steps.run-*.output}}`, fixes source/tests/config, and reruns the command locally until green. -3. `verify-*`: deterministic rerun, usually still `failOnError: false`, followed by a final repair step if red. -4. `commit-if-green`: deterministic step that reruns the full acceptance command and commits only when every exit code is zero. If anything is still red, it writes `BLOCKED_NO_COMMIT` with the failing evidence and exits successfully so the workflow reports a handled blocked state, not a runtime failure. +The same thing in YAML, where `dependsOn` is the only ordering primitive: + +```yaml +version: '0.1.0' +name: ship-feature +steps: + - id: unit + type: deterministic + command: 'node gates.mjs record --name unit --command-base64 ' + - id: repair-unit + type: agent + dependsOn: [unit] + cli: claude + instruction: | + Read evidence/unit.json. Green means do nothing. + Red means fix it and rerun until the recorder writes green. + - id: unit-final + type: deterministic + dependsOn: [repair-unit] + command: 'node gates.mjs record --name unit --command-base64 ' + - id: unit-assert + type: deterministic + dependsOn: [unit-final] + command: 'node gates.mjs require-green --names unit' +``` -AgentWorkforce/relay#827 added repair-aware reliability to the SDK (`.reliable()` / `.repairable()` and repair-aware retry-mode workflows). Prefer those presets when available, but still model explicit repair owners when gate output needs domain-specific fixing. +In TypeScript, ordering is the await sequence and `Promise.all` is first-class for fan-out. In YAML/JSON it is `dependsOn` and nothing else — there is no `pattern`, no `maxConcurrency`. An agent's retry budget is `maxIterations`, which counts the first attempt: the old `retries: 2` is `maxIterations: 3`. -## Keep Repairable Gates On The Critical Path +### Retry a flaky gate once, and record which attempt passed -Repair-before-failure only works after the workflow reaches a deterministic gate. If a long-running interactive agent step is a hard dependency for the first gate, then a dropped PTY, agent spawn error, or transport failure can stop the workflow before the repair loop ever sees evidence. +Contention between suites is not a regression. A run 35 steps deep died to a `Verified: 2/3, Failed: 0` from a suite that passed 3/3 standalone three times in a row. Give `record` a `--retries 1`: the command must still pass, so the gate is not weakened, but the recorded tail has to say `passed on attempt 2`. A suite that only ever passes on retry then stays visible instead of being quietly laundered green. -For large rollouts, treat implementation agents as advisory producers and put a deterministic reconciliation step on the critical path: +## Enforce after the agent, not on it -1. Start implementation/review agents in parallel if useful, but require them to write durable artifacts such as `.workflow-artifacts//runtime.md`, self-review notes, changed-file lists, and command evidence. -2. Add `implementation-reconcile`: a deterministic step that inspects `git status --short -- `, required files, artifact files, and diff stats. It should use `captureOutput: true` and `failOnError: false`. -3. Add `repair-implementation-reconcile`: a focused repair owner that reads the reconcile output and finishes missing artifacts or code before validation gates run. -4. Make discovery, typecheck, E2E, and final acceptance depend on the reconcile/repair path, not directly on every long-lived implementation agent. -5. Keep the final commit deterministic and green-only; red final evidence becomes a repair/blocking artifact, not a failed workflow. +Do not hang the enforcement of an agent's work on a gate attached to that agent step. Put it in the deterministic recorded step that follows. -This shape prevents "agent transport failed" from masquerading as "the product failed." The product still has to pass the same gates; the difference is that the workflow can reach the gates and repair them. +A gate on an agent step makes a dropped transport look like a crashed run. The same work, checked by the next deterministic step, reads as *"nothing was written"* — a repairable fact, with a journaled tail explaining it. (`writing-relayflows` has the current per-gate caveats; the shipped gate set moves between releases, and at least one gate has silently swallowed its own output.) Independently of any of that, this is the better shape, and it is the same rule as keeping repairable gates on the critical path. -## Squad Review Before Final Acceptance +### Keep repairable gates on the critical path -For high-stakes implementation workflows, validation should include human-like review structure, not only command gates. Use small implementation squads and make review state durable: +Repair-before-failure only helps once the flow *reaches* a deterministic gate. If a long-running agent step is a hard dependency of the first gate, a dropped worker stops the flow before the repair loop ever sees evidence. -1. Split independent scopes into 2-3 agent squads. Each squad has an implementer, a shadow reviewer, and optionally a validation/test owner. -2. The shadow reviewer follows the implementer while work is happening and flags spec drift early. -3. Before external review, the implementer writes a self-reflection artifact under `.workflow-artifacts//` covering spec coverage, changed files, tests/proofs, repo-rule alignment, and known risks. -4. A fresh self-review agent reads the actual files, AGENTS.md / CLAUDE.md, recent related work, and local conventions. It writes findings to disk. -5. The implementer repairs valid findings, then deterministic gates rerun from captured output. -6. After all squads converge, run the selected review-depth fresh-eyes review/fix path. Light requires `review-claude` -> `fix-loop` and gates final review pass on `post-fix-validation`. Standard adds `final-review-claude` -> `final-fix-claude` and gates final review pass on `final-fix-claude`. Deep requires the standard Claude path plus `review-codex` -> `fix-loop-codex` -> `final-review-codex` -> `final-fix-codex` and gates final review pass on `final-fix-codex`. -7. If the selected review path still finds issues, run another explicit fix pass or write `BLOCKED_NO_COMMIT` with exact evidence. -8. Commit or PR creation is allowed only after the selected review-depth path, final-review-pass gate, final deterministic acceptance, and scoped diff/regression gates are green. Otherwise write a `BLOCKED_NO_COMMIT` artifact with exact evidence. +1. Treat implementation agents as advisory producers. Require them to write durable artifacts: a changed-file list, command evidence, self-review notes. +2. Add a deterministic `reconcile` recorder that inspects `git status --short -- `, required files and diff stats. +3. Add a repair owner that reads the reconcile evidence and finishes whatever is missing. +4. Make typecheck, E2E and final acceptance depend on the reconcile/repair path — not directly on every long-lived agent. +5. Keep the final commit deterministic and green-only. Red final evidence becomes a `BLOCKED_NO_COMMIT` artifact, not a crashed run. -This keeps "100%" tied to both executable evidence and independent review over the final state. +This stops "the agent transport failed" from masquerading as "the product failed." The product still has to pass the same gates; the difference is that the flow can reach them. -## The Test-Fix-Rerun Pattern +### Partition the agents' lanes yourself -Every testable feature in a workflow should follow this four-step pattern: +`permissions` on an agent step is journaled and **not** enforced — a `flows run` is not a sandbox, and every agent shares one checkout. Declare `permissions` for reviewability, then partition lanes by path *and* sequence the agents that touch adjacent ones, and make an edit gate that rejects out-of-scope changes. And pass `--local-agent`: without it a flow containing any agent step parks at the first one, with a message that reads as a missing worker rather than a missing flag. -```typescript -// Step 1: Run tests (allow failure — we expect issues on first run) -.step('run-tests', { - type: 'deterministic', - dependsOn: ['create-tests'], - command: 'npx tsx --test tests/my-feature.test.ts 2>&1 | tail -60', - captureOutput: true, - failOnError: false, // <-- Don't fail the workflow, let the agent fix it -}) +## Verify every edit -// Step 2: Agent reads output, fixes issues, re-runs until green -.step('fix-tests', { - agent: 'tester', - dependsOn: ['run-tests'], - task: `Check the test output and fix any failures. +Never trust that an agent edited a file. After every agent edit, record a deterministic check and route its evidence to a repair owner. -Test output: -{{steps.run-tests.output}} +```ts +await f.agent('edit-schema', { cli: 'claude', task: 'Edit lib/db/schema.ts …' }); -If all tests passed, do nothing. -If there are failures: -1. Read the failing test file and source files -2. Fix the issues (could be in test or source) -3. Re-run: npx tsx --test tests/my-feature.test.ts -4. Keep fixing until ALL tests pass.`, - verification: { type: 'exit_code' }, -}) +await f.run(record('edit-schema', ` + if [ -z "$(git status --short -- lib/db/schema.ts)" ]; then echo NOT_MODIFIED; exit 1; fi + grep -q my_new_table lib/db/schema.ts || { echo MISSING_TABLE; exit 1; } + echo EDIT_OK +`)); -// Step 3: Deterministic rerun — capture result for a final repair pass -.step('run-tests-final', { - type: 'deterministic', - dependsOn: ['fix-tests'], - command: 'npx tsx --test tests/my-feature.test.ts 2>&1', - captureOutput: true, - failOnError: false, -}) +await f.agent('repair-edit-schema', { + cli: 'claude', + task: 'Read evidence/edit-schema.json. Green means do nothing. Red means make the edit land.', +}); -// Step 4: Repair again if the rerun is still red -.step('fix-tests-final', { - agent: 'tester', - dependsOn: ['run-tests-final'], - task: `If the final test rerun passed, record the green evidence. -If it failed, fix the remaining issue and rerun until green: -{{steps.run-tests-final.output}}`, - verification: { type: 'exit_code' }, -}) +await f.run(record('edit-schema', /* same command */)); +await f.run(requireGreen('edit-schema')); ``` -**Why four steps instead of one?** -- The first run captures output for the agent to diagnose -- The agent step can iterate (read errors, fix, re-run) multiple times -- The final deterministic run is still evidence-based, but a repair agent sees it before the workflow stops -- The last repair step keeps the workflow aligned with the agent-team model instead of ending on a fixable failure +**Use `git status --short -- `, not `git diff --quiet`.** `git diff` only sees tracked changes, so a valid new package, test directory or generated artifact is misclassified as "no changes." `git diff --quiet` is fine only for tracked-only edits to files that already exist. -## PGlite: In-Memory Postgres for Database Testing +**Verify** that the file was modified, and that key content exists (a table name, an exported symbol, an import). **Do not verify** exact content, formatting, line counts or byte sizes — agents format differently and those checks are pure brittleness. -When your feature touches the database, use **PGlite** — a WASM-based Postgres that runs in-process. No Docker, no external services, no flaky network dependencies. +### Gate the hazard, not a tool's mode -### Setup +A gate that asserts *"the selector reported `targeted`"* rather than *"no runtime path is unmapped"* can be unsatisfiable by construction. In one campaign, registering a feature required editing the manifest, and editing the manifest forced the selector into `full-smoke` unconditionally. The gate failed correct work — and it read as the agent's fault. Assert the condition you actually care about, read out of the tool's structured output, not its summary verdict. -Install as a dev dependency in the workflow: - -```typescript -.step('install-pglite', { - type: 'deterministic', - command: 'npm install --save-dev @electric-sql/pglite 2>&1 | tail -5', - captureOutput: true, -}) -``` +### Verify at the granularity the risk lives at -### Test Helper Pattern +"256 test functions before, 256 after, so nothing was deleted" is wrong: roughly 90 assertions had been removed from *inside* those functions. Count what can actually be weakened, not the container it sits in. -Create a reusable helper that boots an in-memory Postgres with your schema: +## A requirement that induces a hazard must gate that hazard -```typescript -// tests/helpers/pglite-db.ts -import { PGlite } from '@electric-sql/pglite'; -import { drizzle } from 'drizzle-orm/pglite'; -import * as schema from '../../packages/web/lib/db/schema.js'; +This one cost a near-miss, and it generalises much further than it looks. -// Raw DDL matching your Drizzle schema — PGlite doesn't run Drizzle migrations -const MY_TABLE_DDL = ` -CREATE TABLE IF NOT EXISTS my_table ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - name TEXT NOT NULL, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); -`; +The flow required proof that tests bite: *mutate the guarded code, capture the failing transcript, restore it.* An implementer did the first two and skipped the third, leaving this in shipping product code on a live delivery path: -export async function createTestDb() { - const pg = new PGlite(); - await pg.exec(MY_TABLE_DDL); - const db = drizzle(pg, { schema }); - return { db, pg, schema, cleanup: () => pg.close() }; +```rust +// MUTATION: drop the addressee and truncate the body. +if std::env::var("RELAY_MUTATION_LOSSY_FORMAT").is_ok() { + return format!("Relay message from {}:\n\n{}", delivery.from, &delivery.body[..1]); } ``` -### PGlite Gotchas - -| Issue | Fix | -|-------|-----| -| `pgcrypto` extension not available | Use `gen_random_uuid()` (built-in since PG 13) or generate UUIDs in app code | -| UUID columns | PGlite supports UUID natively — no special handling needed | -| `drizzle-orm/pglite` import | Exists since drizzle-orm 0.30+. If not found, check version. | -| Index creation | PGlite supports standard CREATE INDEX — no limitations | -| Concurrent writes | PGlite is single-connection. Test concurrent logic with sequential assertions. | - -### Test Structure - -```typescript -// tests/my-feature.test.ts -import { describe, it } from 'node:test'; -import assert from 'node:assert/strict'; -import { randomUUID } from 'node:crypto'; -import { createTestDb } from './helpers/pglite-db.js'; - -describe('my feature', () => { - it('does the thing correctly', async () => { - const { db, schema, cleanup } = await createTestDb(); - try { - // Arrange - const testId = randomUUID(); - // Act — use your module against the real (in-memory) Postgres - // Assert - assert.equal(result.name, 'expected'); - } finally { - await cleanup(); - } - }); -}); -``` +`&body[..1]` panics on a multi-byte first character. -## Verify Gates After Every Edit - -Never trust that an agent edited a file correctly. Add a deterministic verify gate after every agent edit step: - -```typescript -// Agent edits a file -.step('edit-schema', { - agent: 'impl', - dependsOn: ['read-schema'], - task: `Edit packages/web/lib/db/schema.ts...`, - verification: { type: 'exit_code' }, -}) - -// Deterministic verification — did the edit actually land? -.step('verify-schema', { - type: 'deterministic', - dependsOn: ['edit-schema'], - command: `if git diff --quiet packages/web/lib/db/schema.ts; then echo "NOT MODIFIED"; exit 1; fi -grep "my_new_table" packages/web/lib/db/schema.ts >/dev/null && echo "OK" || (echo "MISSING"; exit 1)`, - failOnError: false, - captureOutput: true, -}) -.step('fix-schema-verification', { - agent: 'impl', - dependsOn: ['verify-schema'], - task: `Fix the schema edit if verification failed. Output:\n{{steps.verify-schema.output}}`, - verification: { type: 'exit_code' }, -}) -``` +**Every deterministic gate passed that tree.** The edit gate saw a changed file in scope; the invariant gate saw four correctly-named tests and a mutation transcript; typecheck, clippy, the release build and all five parity suites were green. Only the adversarial reviewer caught it. -**What to verify:** -- File was actually modified (`git diff --quiet` returns non-zero) -- Key content exists (grep for table names, function names, imports) -- For new files: `file_exists` verification type -- For new directories, package trees, generated files, or mixed tracked/untracked - edits: use `git status --short -- `, because `git diff --quiet` - ignores untracked files - -**Gate the hazard, not a tool's mode.** A gate that asserts *"the selector reported `targeted`"* rather than *"no runtime path is unmapped"* can be unsatisfiable by construction — in our case registering a feature required editing the manifest, and editing the manifest forced the selector into `full-smoke` unconditionally. The gate failed correct work, and it would have read as the agent's fault. Assert the condition you actually care about, from the tool's structured output, not its summary verdict. - -**What NOT to verify:** -- Exact content (too brittle — agents format differently) -- Line counts or byte sizes (meaningless) - -### Edit Gates That Include New Files - -When an agent may create new files or package directories, do not use -`git diff --quiet -- ` as the only edit gate. It only sees tracked -changes, so a valid new package can be misclassified as "no changes." - -Use `git status --short -- ` and keep the first gate repairable: - -```typescript -.step('edit-gate-capture', { - type: 'deterministic', - dependsOn: ['implement'], - command: `if [ -z "$(git status --short -- packages/new-adapter tests docs)" ]; then - echo "NO_CHANGES" - exit 1 -fi -echo "EDIT_GATE_OK"`, - captureOutput: true, - failOnError: false, -}) -.step('fix-edit-gate', { - agent: 'impl', - dependsOn: ['edit-gate-capture'], - task: `If the edit gate reported NO_CHANGES, inspect the acceptance contract -and current git status, then add the missing source/test/artifacts. - -Gate output: -{{steps.edit-gate-capture.output}} - -If it already passed, do nothing.`, - verification: { type: 'exit_code' }, -}) -.step('edit-gate-final', { - type: 'deterministic', - dependsOn: ['fix-edit-gate'], - command: `if [ -z "$(git status --short -- packages/new-adapter tests docs)" ]; then - echo "NO_CHANGES" - exit 1 -fi -echo "EDIT_GATE_FINAL_OK"`, - captureOutput: true, - failOnError: true, -}) -``` +Read literally by an agent, "prove the test fails" is an instruction to damage production code. If you ask for that proof, you must gate the residue: -Rule of thumb: `git diff --quiet` is fine for tracked-only edits to known -files. Use `git status --short -- ` for materialization gates that may -include new tests, docs, generated artifacts, or package directories. - -## Mock Sandbox Pattern - -When testing code that interacts with Daytona sandboxes, use inline mock objects matching the existing test conventions: - -```typescript -const daytona = { - create: async () => ({ - id: 'sandbox-id', - process: { - executeCommand: async (cmd, cwd, env) => ({ - result: 'output', - exitCode: 0, - }), - }, - fs: { - uploadFile: async () => undefined, - }, - getUserHomeDir: async () => '/home/daytona', - }), - remove: async () => undefined, -}; +```js +// refuse mutation scaffolding anywhere in product source +for (const file of walk('src').filter((f) => f.endsWith('.rs'))) { + if (/MUTATION|_MUTATION_/.test(readFileSync(file, 'utf8'))) problems.push(`residue in ${file}`); +} ``` -For testing that your code calls the right methods, record calls in an array: +Then prove *that* gate bites, by reinstating a one-line probe and watching it go red. -```typescript -const emitted: EmitEventOptions[] = []; -const mockClient: SessionEventClient = { - emit: async (opts) => { emitted.push(opts); }, - getEvents: async () => [], - getLatestSequence: async () => 0, -}; +Generalise it: **any instruction that tells an agent to temporarily break something needs a matching check that it was put back.** Temporarily lowering a timeout, stubbing a provider, disabling a guard — same shape, same hazard. -// ... run the code ... +### A test that guards a helper does not guard the call site -assert.equal(emitted.length, 4); -assert.equal(emitted[0].eventType, 'sandbox_created'); -``` +A mutation proof must mutate **where the bug would be written**, not where the abstraction lives. Mutating the shared helper and watching the suite go red proves only that the helper is covered; the call site that forgot to use the helper is exactly the defect you were trying to exclude, and it survives untouched. Point the probe at the production call path. -## Regression Testing +## Cross-check your gates against each other -After your new tests pass, always run the **existing test suite** to catch regressions: +Nothing validates that the assertions you write are mutually satisfiable, and the agents pay for it when they are not. In one campaign a `seam-rules` gate **required** a file to be edited while an `edit-gate` **rejected** that same file as out-of-scope. No implementation could pass both. The agent began reverting correct work to appease the contradiction — and it read as the agent failing. -```typescript -.step('run-existing-tests', { - type: 'deterministic', - dependsOn: ['fix-build'], - command: 'npm run orchestrator:test 2>&1 | tail -40', - captureOutput: true, - failOnError: false, -}) +If one gate names a path, assert at authoring time that every other gate permits it. That check is a few lines, and it found a real contradiction immediately. -.step('fix-regressions', { - agent: 'impl', - dependsOn: ['run-existing-tests'], - task: `Check the full test suite for regressions caused by our changes. +## Give every repair owner a "do nothing" clause -Test output: -{{steps.run-existing-tests.output}} +A repair agent handed a green gate will invent work rather than conclude there is none. One burned 21 minutes on a gate reporting `not-required`, against a 2–7 minute norm, because its prompt said "make the scenario real" with no branch for the passing case. -If all tests passed, do nothing. -If EXISTING tests broke, read the failing test, find what we broke, fix it. -Most likely cause: constructor signatures changed, new required fields added -without defaults, or import paths shifted. +Every repair prompt needs an explicit *"if the evidence is green, do nothing and say so."* Audit them: 4 of 7 in an otherwise mature flow were missing it. -Run: npm run orchestrator:test -Fix until all tests pass.`, - verification: { type: 'exit_code' }, -}) -``` +## Seal the product tree, not just the evidence -## Full Workflow Template - -Here's the complete pattern for a feature that touches the database: - -```typescript -import { workflow } from '@relayflows/core'; - -const result = await workflow('my-feature') - .description('Add feature X with full E2E validation') - .pattern('dag') - .channel('wf-my-feature') - .maxConcurrency(3) - .timeout(3_600_000) - .repairable() - - .agent('impl', { cli: 'claude', preset: 'worker', retries: 2 }) - .agent('tester', { cli: 'claude', preset: 'worker', retries: 2 }) - - // ── Phase 1: Read ──────────────────────────────────────────────── - .step('read-target', { - type: 'deterministic', - command: 'cat path/to/file.ts', - captureOutput: true, - }) - - // ── Phase 2: Implement ─────────────────────────────────────────── - .step('edit-target', { - agent: 'impl', - dependsOn: ['read-target'], - task: `Edit path/to/file.ts. Current contents: -{{steps.read-target.output}} - -Only edit this one file.`, - verification: { type: 'exit_code' }, - }) - .step('verify-target', { - type: 'deterministic', - dependsOn: ['edit-target'], - command: 'git diff --quiet path/to/file.ts && (echo "NOT MODIFIED"; exit 1) || echo "OK"', - failOnError: false, - captureOutput: true, - }) - .step('fix-target-verification', { - agent: 'impl', - dependsOn: ['verify-target'], - task: `Fix the target edit if verification failed. Output:\n{{steps.verify-target.output}}`, - verification: { type: 'exit_code' }, - }) - - // ── Phase 3: Test infrastructure ───────────────────────────────── - .step('install-pglite', { - type: 'deterministic', - command: 'npm install --save-dev @electric-sql/pglite 2>&1 | tail -5', - captureOutput: true, - }) - .step('create-test-helpers', { - agent: 'tester', - dependsOn: ['install-pglite'], - task: 'Create tests/helpers/pglite-db.ts with ...', - verification: { type: 'file_exists', value: 'tests/helpers/pglite-db.ts' }, - }) - .step('create-tests', { - agent: 'tester', - dependsOn: ['create-test-helpers', 'fix-target-verification'], - task: 'Create tests/my-feature.test.ts with ...', - verification: { type: 'file_exists', value: 'tests/my-feature.test.ts' }, - }) - - // ── Phase 4: Test-fix-rerun loop ───────────────────────────────── - .step('run-tests', { - type: 'deterministic', - dependsOn: ['create-tests'], - command: 'npx tsx --test tests/my-feature.test.ts 2>&1 | tail -60', - captureOutput: true, - failOnError: false, - }) - .step('fix-tests', { - agent: 'tester', - dependsOn: ['run-tests'], - task: `Fix any test failures. Output:\n{{steps.run-tests.output}}`, - verification: { type: 'exit_code' }, - }) - .step('run-tests-final', { - type: 'deterministic', - dependsOn: ['fix-tests'], - command: 'npx tsx --test tests/my-feature.test.ts 2>&1', - captureOutput: true, - failOnError: false, - }) - .step('fix-tests-final', { - agent: 'tester', - dependsOn: ['run-tests-final'], - task: `If the final test rerun is red, fix and rerun until green. Output:\n{{steps.run-tests-final.output}}`, - verification: { type: 'exit_code' }, - }) - - // ── Phase 5: Build + regression ────────────────────────────────── - .step('build-check', { - type: 'deterministic', - dependsOn: ['fix-tests-final'], - command: 'npx tsc --noEmit 2>&1 | tail -20; echo "EXIT: $?"', - captureOutput: true, - failOnError: false, - }) - .step('fix-build', { - agent: 'impl', - dependsOn: ['build-check'], - task: `Fix type errors if any. Output:\n{{steps.build-check.output}}`, - verification: { type: 'exit_code' }, - }) - .step('run-existing-tests', { - type: 'deterministic', - dependsOn: ['fix-build'], - command: 'npm test 2>&1 | tail -40', - captureOutput: true, - failOnError: false, - }) - .step('fix-regressions', { - agent: 'impl', - dependsOn: ['run-existing-tests'], - task: `Fix regressions if any. Output:\n{{steps.run-existing-tests.output}}`, - verification: { type: 'exit_code' }, - }) - - // ── Phase 6: Commit ────────────────────────────────────────────── - .step('commit', { - type: 'deterministic', - dependsOn: ['fix-regressions'], - command: [ - 'npx tsx --test tests/my-feature.test.ts', - 'npm test', - 'git add ', - 'git commit -m "feat: ..."', - ].join(' && '), - captureOutput: true, - failOnError: false, - }) - .step('repair-commit', { - agent: 'impl', - dependsOn: ['commit'], - task: `If commit failed, fix the blocker, rerun the feature and regression tests, and create the commit. -If commit passed, confirm the commit subject. -Output: -{{steps.commit.output}}`, - verification: { type: 'exit_code' }, - }) - .step('verify-commit-created', { - type: 'deterministic', - dependsOn: ['repair-commit'], - command: 'git log -1 --pretty=%s | grep -q "^feat: " && echo "COMMIT_OK" || (echo "COMMIT_MISSING"; exit 1)', - captureOutput: true, - failOnError: true, - }) - - .onError('retry', { maxRetries: 2, retryDelayMs: 10_000 }) - .run({ cwd: process.cwd() }); -``` +If a signoff step binds a reviewer to an artifact digest, digest the **changed source files** too. Otherwise the reviewer is bound to a hash of *evidence files* while the code those files describe changes underneath them. Prove it bites: a one-line source edit must change the digest. -## A requirement that induces a hazard must gate that hazard +## Review before acceptance -This one cost a near-miss, and it generalises further than it looks. +Command gates are not review. For high-stakes work, make review state durable and put it on the critical path: -The workflow required proof that tests bite: *mutate the guarded code, capture the failing transcript, restore it.* An implementer did the first two steps and skipped the third, leaving this in shipping product code on a live delivery path: +1. Split independent scopes into small squads: an implementer, a shadow reviewer who flags spec drift while the work happens, and optionally a test owner. +2. Before external review, the implementer writes a self-reflection artifact — spec coverage, changed files, proofs, repo-rule alignment, known risks. +3. A **fresh-context** reviewer reads the actual files, `AGENTS.md` / `CLAUDE.md`, recent related work and local conventions, and writes findings to disk. Fresh context is the point: a reviewer that watched the work being built inherits its blind spots. +4. The implementer repairs valid findings; the deterministic gates re-record from scratch. +5. Scale the depth to the stakes — one Claude review/fix round, then a second round, then an independent Codex review/fix round for the deepest tier. Gate final acceptance on the *last* fix step of whichever tier you chose. +6. Commit or PR creation is allowed only after the review path, final deterministic acceptance and scoped diff/regression gates are all green. Otherwise write `BLOCKED_NO_COMMIT` with the exact evidence and end without committing. -```rust -// MUTATION: drop the addressee and truncate the body. -if std::env::var("RELAY_MUTATION_LOSSY_FORMAT").is_ok() { - return format!("Relay message from {}:\n\n{}", delivery.from, &delivery.body[..1]); -} -``` +## Local test substrates -`&body[..1]` panics on a multi-byte first character. +Tests that need Docker or a network do not belong in a gate you intend to rerun four times. -**Every deterministic gate passed that tree.** The edit gate saw a changed file in scope; the invariant gate saw four correctly-named tests and a mutation transcript; typecheck, clippy, the release build and all five end-to-end parity suites were green. Only the adversarial reviewer caught it. +**PGlite** (`@electric-sql/pglite`) is a WASM Postgres running in-process — no Docker, no external service, no flaky network. Boot it with raw DDL matching your schema (it does not run Drizzle migrations), and hand the client to the code under test: -The instruction "prove the test fails," read literally by an agent, is an instruction to damage production code. If you ask for that proof, you must also gate the residue: +```ts +import { PGlite } from '@electric-sql/pglite'; +import { drizzle } from 'drizzle-orm/pglite'; +import * as schema from '../../lib/db/schema.js'; -```js -// refuse mutation scaffolding anywhere in product source -for (const file of walk('src').filter(f => f.endsWith('.rs'))) { - if (/MUTATION|_MUTATION_/.test(readFileSync(file, 'utf8'))) problems.push(`residue in ${file}`); +export async function createTestDb() { + const pg = new PGlite(); + await pg.exec(MY_TABLE_DDL); // raw DDL, kept in step with schema.ts + return { db: drizzle(pg, { schema }), pg, schema, cleanup: () => pg.close() }; } ``` -Then prove *that* gate bites, by reinstating a one-line probe and watching it go red. +Gotchas: no `pgcrypto` (use the built-in `gen_random_uuid()`); single connection, so exercise concurrency with sequential assertions; `drizzle-orm/pglite` needs drizzle-orm 0.30+. The real hazard is DDL drift — if the test DDL and `schema.ts` disagree, the tests pass against a schema that does not exist. Derive one from the other, or gate them against each other. -Generalise it: **any instruction that tells an agent to temporarily break something needs a matching check that it was put back.** Temporarily lowering a timeout, stubbing a provider, disabling a guard — same shape, same hazard. - -## Four smaller rules, each learned the expensive way - -**Give every repair owner an explicit "if the gate is green, do nothing" clause.** A repair agent handed a green gate will invent work rather than conclude there is none — one burned 21 minutes on a gate reporting `not-required` against a 2–7 minute norm, because its prompt said "make the scenario real" with no branch for the passing case. Audit them: 4 of 7 in a mature workflow were missing it. - -**Seal the product tree, not just the evidence directory.** If your signoff step binds a reviewer to an artifact digest, digest the changed source files too. Otherwise the reviewer is bound to a hash of *evidence files* while the code those files describe can change underneath them. Prove it bites: a one-line source edit must change the digest. - -**Verify at the granularity the risk lives at.** Checking "256 test functions before, 256 after" and concluding nothing was deleted is wrong — roughly 90 assertions had been removed from *inside* those functions. Count what can actually be weakened, not the container it sits in. +For external services (sandboxes, event clients), prefer an inline mock that **records calls into an array**, then assert on the recording. Asserting "it did not throw" proves nothing about what it called: -**Retry a flaky gate once, and record which attempt passed.** Contention between suites is not a regression; a run 35 steps deep died to a `Verified: 2/3, Failed: 0` that passed 3/3 standalone three times. A retry does not weaken the gate — the command must still pass — but the recorded tail has to say `passed on attempt 2`, so a suite that only ever passes on retry stays visible instead of being quietly laundered green. +```ts +const emitted: EmitEventOptions[] = []; +const mockClient: SessionEventClient = { emit: async (o) => { emitted.push(o); }, /* … */ }; +// … +assert.equal(emitted.length, 4); +assert.equal(emitted[0].eventType, 'sandbox_created'); +``` -## Checklist: Is Your Workflow 80-to-100? +## Checklist: is your flow 80-to-100? | Check | How | -|-------|-----| -| Tests exist | `file_exists` verification on test file | -| Tests actually run | Deterministic step executes them | -| Test failures get fixed | Agent step reads output, fixes, re-runs | -| Final test run is repairable | Deterministic rerun captures output, then a repair owner gets one more pass | -| Build passes | `npx tsc --noEmit` deterministic step | -| No regressions | Existing test suite runs after changes | -| Every edit is verified and repairable | `git diff --quiet` + grep for tracked-only edits; `git status --short -- ` when new files/packages may appear; then a fix step | -| Commit only happens after green evidence | Final commit step reruns acceptance checks and commits only on zero exit codes | - -## Common Anti-Patterns +|---|---| +| Tests exist | Deterministic check for the file, recorded | +| Tests actually run | A recorder step runs them; nobody's summary is trusted | +| Failures become work, not a dead run | Recorder always exits 0; a repair owner reads `evidence/.json` | +| Green is recomputed, not reported | `require-green` reads the recordings at the end | +| Evidence belongs to this run | Every record stamped with `runId`; `--reuse-from` can't resurrect a stale one | +| Flaky gates stay visible | `--retries 1`, and the tail names the attempt that passed | +| Every edit is verified and repairable | `git status --short -- ` + a content grep, then a repair step | +| Temporary breakage is put back | A residue gate for every "prove it fails" instruction, proven to bite | +| Gates are mutually satisfiable | Authoring-time cross-check of every path a gate names | +| Fresh eyes read the final tree | Review rounds on the critical path, findings written to disk | +| Commit only after green evidence | Final step recomputes acceptance; red writes `BLOCKED_NO_COMMIT` | + +## Common anti-patterns | Anti-pattern | Why it fails | Fix | -|-------------|-------------|-----| -| Tests written but never executed | Agent claims they pass, they don't | Add deterministic `run-tests` step | -| Single `failOnError: true` test run | First failure kills workflow, no chance to fix | Use repairable run-fix-rerun-final-fix loops | -| No regression test | New feature works, old features break | Run `npm test` after build check | -| Agent asked to "write and run tests" in one step | Agent writes tests, runs them, they fail, it edits, output is garbled | Separate write/run/fix into distinct steps | -| PGlite DDL doesn't match Drizzle schema | Tests pass on wrong schema | Derive DDL from schema.ts or test with real migration | -| Final test output not handed to an agent | Broken tests can stop the run or get ignored | Add a final repair owner before commit | -| Testing only happy path | Edge cases break in prod | Specify edge case tests in the task prompt | -| No verify gate after agent edits | Agent exits 0 without writing anything | Add `git diff --quiet` check after every edit, then route failures to a repair step | -| `git diff --quiet` for new package/test directories | Untracked files are invisible, so valid new artifacts can look like "no changes" | Use `git status --short -- ` and a repairable capture → fix → final gate pattern | -| Committing after `failOnError: false` without checking exits | Broken work can be committed because the shell step returned successfully | In `commit-if-green`, record each exit code and skip commit unless all are zero | -| Copying a v1 `.step({ failOnError, captureOutput })` example into a v2 flow | Those fields do not exist in v2; the spec will not compile | Use the evidence recorder; see **The v2 translation** | -| Gating a v2 agent step on `subprocess_gate` | A failure reports `exit=1` with empty stdout and stderr, and the verdict is unrecoverable | `artifact_exists`, or no gate plus a deterministic gate after the step | +|---|---|---| +| Tests written but never executed | The agent claims they pass; they don't | A recorder step runs them | +| A bare check as the gate | The first red kills the run, with no chance to repair | Record → repair → re-record → assert | +| `\|\| true` instead of a recorder | Throws the exit code away; a forgotten gate then ships red work silently | Journal `{exitCode, verdict, tail}` to a file | +| Unstamped evidence + `--reuse-from` | Recorder steps always exit 0, so they are always reusable — stale verdicts read as fresh | Stamp `runId`; `require-green` rejects foreign stamps | +| Gating enforcement on the agent step | A dropped transport reads as a crashed run instead of "nothing was written" | Enforce in the deterministic step after | +| Repair prompt with no green branch | The agent invents work against a passing gate | "If the evidence is green, do nothing and say so" | +| `git diff --quiet` for new files | Untracked files are invisible, so valid new packages look like "no changes" | `git status --short -- ` | +| Asserting a tool's summary verdict | Can be unsatisfiable by construction; fails correct work | Assert the hazard, from structured output | +| Counting containers, not contents | 256 tests before and after hid ~90 deleted assertions | Verify at the granularity the risk lives at | +| Mutating the helper to prove coverage | Proves the helper is covered, not the call site that forgot it | Mutate where the bug would be written | +| Sealing only the evidence directory | The reviewer is bound to a hash of the wrong thing | Digest the changed source too, and prove it bites | +| Treating all-green as done | 15 findings, 2 disqualifying, on an all-green tree | Fresh-eyes review rounds before acceptance | +| Copying a v1 `.step({ failOnError, captureOutput })` | Those fields do not exist in v2; the flow will not compile | The evidence recorder | From f0ecd77a76d8357f935526da2308dc82d13888cc Mon Sep 17 00:00:00 2001 From: khaliqgant Date: Sun, 20 Sep 2026 17:45:04 -0700 Subject: [PATCH 07/12] docs(orchestrating-agent-relay): 12.4 PR-owner GitHub subscribe Audit the orchestrating skill against Agent Relay 12.4 and make subscribing the live PR-owner agent to the PR's GitHub provider resource the durable default. Cover rebinding after respawn, unsubscribe/release hygiene, no duplicate writable helper subscriptions, and verifying actual event delivery. Salvage still-valid #56 workflow lessons (idle detection, per-turn interactive spawn, LLM-as-oracle, repair-step counterweight) into writing-agent-relay-workflows. --- README.md | 4 +- prpm.json | 6 +- skills/orchestrating-agent-relay/SKILL.md | 166 ++++++++++++++++-- skills/writing-agent-relay-workflows/SKILL.md | 22 +++ 4 files changed, 180 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 82abf45d..f5a7f9c8 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,11 @@ Package metadata lives in [prpm.json](prpm.json). The repo currently publishes ` | Skill | Version | Description | |-------|---------|-------------| | [choosing-swarm-patterns](skills/choosing-swarm-patterns/SKILL.md) | 1.1.5 | Pick the right Agent Relay orchestration pattern across the 10 core swarm patterns plus specialized patterns. | -| [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 1.6.24 | Build multi-agent workflows with WorkflowBuilder, DAG dependencies, Relayfile-backed Slack human assistance, integration subscriptions, waitFor gates, review-depth review/fix loops, channels, and chat-native coordination recipes. | +| [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 1.6.25 | Build multi-agent workflows with WorkflowBuilder, DAG dependencies, Relayfile-backed Slack human assistance, integration subscriptions, waitFor gates, review-depth review/fix loops, channels, and chat-native coordination recipes. | | [setting-up-relayfile](skills/setting-up-relayfile/SKILL.md) | 1.1.1 | Set up Relayfile mounts and writeback for provider files through local filesystem access. | | [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.0.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | | [using-agent-relay](skills/using-agent-relay/SKILL.md) | 1.4.0 | Participant-side MCP reference for a **registered** relay agent (spawned worker / registered lead): messaging, channels, threads, reactions, search, inbox, actions, and worker spawn/release. Counterpart to `orchestrating-agent-relay`. | -| [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.3.1 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention. | +| [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.4.0 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention, including GitHub PR-owner integration subscriptions. | | [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 1.0.9 | Author workflows that close the 80-to-100 validation gap with repair-aware test, verify, review-depth review/fix with test hardening, and commit gates. | | [review-fix-signoff-loop](skills/review-fix-signoff-loop/SKILL.md) | 1.0.2 | Loop review, repair, validation, and fresh-context dual-agent signoff until independent reviewers both satisfy the verdict contract. | | [trigger-autocomplete-catalog](skills/trigger-autocomplete-catalog/SKILL.md) | 1.0.0 | Enforce webhook/event trigger autocomplete coverage through KNOWN_TRIGGER_CATALOG in @relayfile/adapter-core. | diff --git a/prpm.json b/prpm.json index 55cf53e5..51bdfd4a 100644 --- a/prpm.json +++ b/prpm.json @@ -47,7 +47,7 @@ }, { "name": "writing-agent-relay-workflows", - "version": "1.6.24", + "version": "1.6.25", "description": "Use when building multi-agent workflows with relay broker-sdk. Covers conversation vs pipeline coordination, WorkflowBuilder/DAG steps, agents, {{steps.X.output}} chaining, Relayfile-backed human assistance, Slack answer injection, integration subscriptions, waitFor gates, repairable verification, review-depth fresh-eyes review/fix loops with test hardening, channels, chat-native recipes, error handling, event listeners, step sizing, lead+workers teams, and parallel waves.", "format": "claude", "subtype": "skill", @@ -136,8 +136,8 @@ }, { "name": "orchestrating-agent-relay", - "version": "2.3.1", - "description": "The canonical way to run agent-relay - self-bootstrap the local broker and autonomously spawn, monitor, and coordinate a team of worker agents without human intervention. Covers infrastructure startup, agent spawning, lifecycle monitoring, message-based reading via the relay MCP, and team coordination.", + "version": "2.4.0", + "description": "The canonical way to run agent-relay - self-bootstrap the local broker and autonomously spawn, monitor, and coordinate a team of worker agents without human intervention. Covers infrastructure startup, agent spawning, lifecycle monitoring, message-based reading via the relay MCP, GitHub PR-owner integration subscriptions, and team coordination.", "format": "claude", "subtype": "skill", "tags": [ diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index 23b8d937..491d40e6 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -1,6 +1,6 @@ --- name: orchestrating-agent-relay -description: The canonical way to run agent-relay - self-bootstrap the local broker and autonomously spawn, monitor, and coordinate a team of worker agents without human intervention. Covers infrastructure startup, agent spawning, lifecycle monitoring, message-based reading via the relay MCP, and team coordination. +description: The canonical way to run agent-relay - self-bootstrap the local broker and autonomously spawn, monitor, and coordinate a team of worker agents without human intervention. Covers infrastructure startup, agent spawning, lifecycle monitoring, message-based reading via the relay MCP, GitHub PR-owner integration subscriptions, and team coordination. --- # Orchestrating Agent Relay @@ -66,7 +66,12 @@ reference is the **`using-agent-relay`** skill. | Read worker replies (MCP) | `check_inbox(limit: 20)` / `list_messages(channel: "general")` | | Give a human a follow-along link | `agent-relay observer` | | Inspect a worker's TTY | `agent-relay node agent attach Worker1 --mode view` | -| Release worker | `agent-relay node agent release Worker1` | +| Fleet-wide agent inventory | `agent-relay fleet agent list` | +| Subscribe PR-owner to GitHub PR | `agent-relay integration subscribe github --resource '/github/repos///pulls//**' --to @Owner` | +| List integration bindings | `agent-relay integration subscribe --list` | +| Unsubscribe a provider resource | `agent-relay integration unsubscribe github --resource '...'` | +| Failed deliveries | `agent-relay node deadletters` | +| Release worker | `agent-relay node agent release Worker1` / `agent-relay fleet release Worker1` | | Stop broker | `agent-relay node down` | ## Bootstrap Flow @@ -146,9 +151,12 @@ add_agent( ``` `node agent spawn` takes the provider as a positional argument -(`claude`, `codex`, `gemini`, `droid`, …) and `--name` / `--task` / `--channels` -/ `--model` / `--cwd` flags. By default the agent joins the `general` channel and -runs in `interactive` spawn mode; pass `--exit-after-task` for a one-shot worker. +(`claude`, `codex`, `gemini`, `grok`, `opencode`, `muse`, `devin`, `droid`, …) +and `--name` / `--task` / `--channels` / `--model` / `--cwd` flags. By default +the agent joins the `general` channel and runs in `interactive` spawn mode; pass +`--exit-after-task` for a one-shot worker. Fleet placement uses +`agent-relay fleet spawn ` (or MCP `spawn` / `add_agent`); `droid` remains +high-risk for delegation. > **Expect a 30–60s gap between spawn and the first ACK.** A worker shows in > `node agent list` within ~5s (the process is up), but the underlying CLI @@ -226,13 +234,19 @@ agent-relay node metrics ### Step 4: Release Workers +Unsubscribe any provider resources that worker owned (see +[Subscribe the live PR owner](#subscribe-the-live-pr-owner)) before release. + ```text remove_agent(name: "Worker1", reason: "Work accepted") ``` -CLI equivalent: +CLI equivalent — prefer the control-plane form when the worker may not be on +this broker's default `connection.json`: ```bash +agent-relay fleet release Worker1 +# local-only fallback: agent-relay node agent release Worker1 ``` @@ -425,11 +439,19 @@ Quick Reference. Then enforce this protocol: over the relay MCP — never `node tail` (that streams broker debug events, not worker messages). See the "Channel vs DM" section for the full reading model -- Poll `agent-relay node agent list` for worker liveness; set a wall-clock - fallback so a silently-dead worker can't hang the loop +- Poll `agent-relay node agent list` (local) or `agent-relay fleet agent list` + (cross-node) for worker liveness; set a wall-clock fallback so a + silently-dead worker can't hang the loop +- The moment a worker opens or owns a GitHub PR, subscribe **that live + PR-owner identity** with `agent-relay integration subscribe github + --resource '/github/repos///pulls//**' --to @`. + Do not subscribe helpers to the same glob. After respawn under a new + name, rebind. Verify a real GitHub event arrives in `check_inbox`, not + just that `--list` shows a binding. Unsubscribe owned resources before + release - If a human is watching, give them a follow-along link with - `agent-relay observer` and print the URL it returns. Never print the - workspace key or put it in a URL + `agent-relay observer` (or MCP `get_observer_url`) and print the URL it + returns. Never print the workspace key or put it in a URL ``` ## Multi-Round Review Loops (DONE → NO-GO → fix → re-review) @@ -476,7 +498,8 @@ Do NOT self-remove — stay alive for re-review." Always pass branch + commit SHA + the **complete** reviewer verdict. A fresh worker has none of the loop's history; a summarized verdict loses the -specifics it needs to fix. +specifics it needs to fix. Rebind any GitHub PR subscription to the new +live identity (see below) — a new name does not inherit the old route. ### Detecting a silently-dead worker @@ -492,6 +515,116 @@ worker still thinking. Defenses: - Always set a wall-clock fallback (e.g. a ScheduleWakeup ~30 min out) so a silently-dead worker can't hang the loop forever waiting on a message that will never arrive. +- Across nodes, `agent-relay fleet agent list` joins live broker agents + against the workspace roster. `idle` means the harness is at its prompt, + not that the assigned work is complete. Roster-only presence is not a live + worker. + +## Subscribe the live PR owner + +Whenever a worker **opens or owns a PR**, subscribe **that live PR-owner +agent** to the PR's GitHub provider resource immediately — the moment the PR +number exists, not after the first review comment. Review comments, pushes, +check changes, and discussion then wake the owner instead of waiting on the +next poll. + +```bash +agent-relay integration subscribe github \ + --resource '/github/repos///pulls//**' \ + --to @ +``` + +`--to` is `@agent` or `#channel`. Point it at the owner handle. `--spawn ` +can launch and confirm a recipient before binding; do not use it to replace an +already-live owner. `--events` defaults to Relay `message.created,thread.reply`. + +`subscribe` is not a chat listener. It provisions three artifacts: + +1. A Relay inbound webhook (`webhookId`) delivering into the recipient's + identity-bound channel +2. A Relayfile webhook subscription (`webhookSubscriptionId`) filtered by + `path_glob` +3. A Relayfile binding keyed by `(provider, resolved path glob)` + +`--resource` is a Relayfile VFS glob. `owner/repo` resolves to repository +scope; provider URLs are not accepted. A PR glob does **not** include GitHub +issue-comment paths (`/issues//comments/**`) even when those comments sit +on the PR. Reviews, root review-comments, and checks are inventoried at +repository-level `reviews/`, `comments/`, and `checks/` paths — a PR directory +does not cover them. Subscribe the PR glob first; add those extra globs only +when the owner must wake on those exact records. + +List bindings with `agent-relay integration subscribe --list`. This is +distinct from `agent-relay integration subscription list`, which lists Relay +workspace event subscriptions (`message.created`, …), and from +`agent-relay integration webhook create `, which mints an inbound +webhook URL that posts into a channel (it does **not** take a URL or `--event`; +that older shape is stale). + +### Ownership transfer / rebinding after respawn + +Bindings follow the **recipient identity**, not the OS process. + +- **Same name, identity kept** (`fleet release` / `node agent release` without + `--delete-agent`): the binding still points at `@Owner`. Respawn that name, + confirm it is live, and re-subscribe only if `--list` no longer shows + `@Owner` on that resource. +- **New name or deleted identity** (`--delete-agent`, or a fresh + `Implementer2`): the new agent does **not** inherit the old subscription. + Subscribe the new live identity to the same `--resource` immediately. The + binding key is `(provider, resolved path)` — a re-subscribe **replaces** the + previous route (create-first). Record the prior `webhookId` / + `webhookSubscriptionId` so you can confirm they retired. +- Do not repair a failed spawn by inviting a helper onto the owner's channel + or by pointing `--to` at a disposable worker. + +### Helper agents: no second writable subscribe + +Only the PR owner gets `--to @Owner` on that PR glob. A second `subscribe` to +the same `(github, path glob)` **replaces** the owner's binding. Reviewers, +shadows, and one-shot helpers read the PR through git/`gh` or channel traffic; +they do not get their own writable subscription to the owner's resource. + +### Unsubscribe / release hygiene + +Unsubscribe **owned** resources before releasing the owner, then verify the +IDs are gone: + +```bash +agent-relay integration subscribe --list +agent-relay integration webhook list +agent-relay integration webhook list-inbound +agent-relay integration unsubscribe github \ + --resource '/github/repos///pulls//**' +# confirm the binding, webhookId, and webhookSubscriptionId disappeared +agent-relay fleet release +``` + +Unsubscribe does not delete agents or channels. Do not unsubscribe by guessed +name, and do not delete a webhook another binding still references. A failed +re-subscribe must leave the prior working binding in place — do not tear it +down first. + +### Verify delivery, not just binding creation + +A successful `subscribe` (or a 2xx webhook create) is **not** proof that +GitHub events will wake the agent. Verify in this order: + +1. **Binding** — `agent-relay integration subscribe --list` shows + `(github, resolved PR glob) → @Owner`. +2. **Webhook inventory** — `integration webhook list` / `list-inbound` shows + the new `webhookId` (and the prior id retired after a replacement). +3. **Live recipient** — `fleet agent list` or `node agent list` shows `@Owner` + live on a node. Roster-only is not enough. +4. **Actual event** — after a real GitHub comment, review, check, or push, the + owner receives a Relay message from `github` whose body names the event and + a Relayfile path under that glob. Read it with `check_inbox` / + `list_messages`. Optional broker corroboration: a delivery-injected event + for that agent; `node deadletters` must not hold it. + +If steps 1–3 pass and step 4 does not, the subscription is not working. Do not +treat Cloud/Nango ingress success, an HTTP 200, or a channel post you wrote +yourself as delivery proof. ## Lifecycle Events @@ -730,9 +863,16 @@ this whole workaround once `relay#1446` lands rather than letting it outlive the | Targeted `fleet spawn` fails with `Targeted Fleet spawn requires an agent token` | Pass `--token` or set `RELAY_AGENT_TOKEN`; mint one with `agent-relay agent register --type system` (capture it without echoing). `--task` is also mandatory and the error only surfaces one problem at a time | | Node shows `online` but never receives a spawn | `online` ≠ available. Check `capabilities` contains `spawn:*` — a record can be live with no spawn capacity. Confirm with a throwaway targeted spawn, verified by `pgrep` **on the target host**, then release | | Harness blocks `sleep 25; check_inbox ...` | Bare foreground `sleep` wait loops are disallowed in harnessed environments. Run the poll loop with `run_in_background` (or Monitor + until-loop); the inline `sleep` snippets show logic only | -| Worker self-removed; can't send review fixes | Instruct workers not to self-remove until told. If already gone, spawn a fresh worker and re-inject branch + commit SHA + full verdict (see Multi-Round Review Loops) | +| Worker self-removed; can't send review fixes | Instruct workers not to self-remove until told. If already gone, spawn a fresh worker and re-inject branch + commit SHA + full verdict; rebind any GitHub PR subscription to the new live identity (see [Subscribe the live PR owner](#subscribe-the-live-pr-owner)) | | Told the user to open an observer URL built from the workspace key | That is an admin credential in a query string, and the realtime endpoint rejects it. Run `agent-relay observer` (or `get_observer_url`) and share the `ot_live_` URL it returns | -| Worker died silently; loop hangs | Inbox polling fires on messages only. Poll `agent-relay node agent list` for liveness and set a wall-clock fallback (~30 min ScheduleWakeup) | +| Worker died silently; loop hangs | Inbox polling fires on messages only. Poll `agent-relay node agent list` / `fleet agent list` for liveness and set a wall-clock fallback (~30 min ScheduleWakeup) | +| Opened a PR but did not subscribe the live owner | Review comments, pushes, and checks will not wake anyone. Immediately `integration subscribe github --resource '/github/repos///pulls//**' --to @` | +| Subscribed a helper/reviewer to the same PR glob | Binding key is `(provider, resolved path)` — the second subscribe **replaces** the owner's writable route. Only the PR-owner identity gets `--to @Owner` | +| Treated `--list` / webhook create as delivery proof | Binding creation is not a wake. Confirm a real GitHub event arrives as a Relay message from `github` with a Relayfile path under that glob (`check_inbox` / `list_messages`); check `node deadletters` if it does not | +| Respawned the owner under a new name and left the old binding | New identities do not inherit subscriptions. Re-subscribe `--to @NewOwner` on the same `--resource`; confirm the prior webhook/subscription IDs retired | +| Released the owner without unsubscribing | Unsubscribe owned resources, verify IDs disappeared, then `fleet release`. Unsubscribe does not delete agents | +| Used `integration webhook create --event …` | Current CLI takes `` (inbound webhook into a channel). Provider wakes use `integration subscribe`, not that older URL/event shape | +| Used `integration subscription list` to inspect PR bindings | That lists Relay workspace event subscriptions. Provider bindings are `integration subscribe --list` | ## Prerequisites diff --git a/skills/writing-agent-relay-workflows/SKILL.md b/skills/writing-agent-relay-workflows/SKILL.md index 35445e8b..7e27f40c 100644 --- a/skills/writing-agent-relay-workflows/SKILL.md +++ b/skills/writing-agent-relay-workflows/SKILL.md @@ -64,6 +64,8 @@ The point of an agent team workflow is not to discover a red gate and stop. The Avoid hard-stop gates (`failOnError: true` with no repair step) in workflows that are supposed to be self-healing. Even cheap preconditions such as missing credentials, wrong repository, or an unsafe dirty worktree should normally write a clear `BLOCKED_*` artifact and exit cleanly. For implementation, build, test, lint, schema, artifact, and review failures, model the fix path in the workflow. +**Counterweight — don't reflexively add a repair step.** This rule applies when the gate has a real, agent-fixable failure surface. A deterministic gate that sits downstream of deterministic work (e.g. a transcript line-count check after a series of deterministic grader steps) is correct by construction — there is no failure mode for a repair agent to address. Wiring a repair step there just spawns a fresh agent for a guaranteed no-op, paying the spawn cost (and often a follow-on idle-timeout wait) for nothing. Keep the hard gate, skip the repair. Reserve repair branches for gates where a real agent action could plausibly turn red into green. + ## Human Assistance and Relayfile Events Relayflows can block on humans and provider events without shelling out to Slack, GitHub, Linear, or the Relayfile CLI. Declare the integration intent in the workflow and let the SDK/runtime use the existing local Relayfile/Pear connection. @@ -266,6 +268,7 @@ Before writing the workflow, decide *how the agents will coordinate*. The relay | Shape | What it is | Use when | |---|---|---| | **Conversation** (chat-native) | Interactive agents share a channel; messages, `@-mentions`, and ambient awareness drive coordination. Lead and workers spawn in parallel and self-organize. The relay is the coordination layer, not just transport. | Multi-file work, peer review loops, cross-agent feedback, dynamic re-planning, multi-PR coordination, anything with a human-in-the-loop escape, swarms where workers pick up each other's output. | +| **Per-turn interactive spawn** (bounded interactive) | Interactive PTY agents (no `preset`), but each step is bounded to *one* turn that exits as soon as its file/message is produced. Agents still join the channel — they just don't live across turns. | Turn-taking flows where each turn is short, well-defined, and reads its inputs from a deterministic source (file on disk, prior grade step). Exercises the PTY + channel-injection path without the idle-detection-vs-wait-for-event race that bites long-running interactive agents. | | **Pipeline** (one-shot DAG) | Each step runs as a one-shot subprocess (`claude -p`, `codex exec`); steps hand off via `{{steps.X.output}}` text injection. No agents are alive at the same time; no chat happens. | Linear, well-specified transformations; deterministic data passing; no live agent-to-agent coordination during implementation. The selected review-depth path and deterministic final gates still apply. | **Default to Conversation for any non-trivial work.** Pipeline DAGs are simpler to reason about but they do not exercise the relay primitive — they are a Unix pipe with extra steps. If you would happily write the same task as a single shell pipeline, pipeline-shape is fine. Otherwise, you almost certainly want a Conversation shape. @@ -1723,6 +1726,22 @@ Edit files as assigned. Report completion. Fix issues from feedback.`, Once you're in the Interactive Team shape, the channel is your coordination medium. These are recipes for using it well — they are *prompt-authoring patterns*, not new SDK surface. All of them assume interactive agents (no `preset`) sharing a `.channel('wf-...')`. +> **Idle detection beats "wait for X" prompts.** The runtime treats a silent interactive PTY as complete after a short idle window (~30 seconds). A task prompt that tells an agent to *wait for message Y* / *exit only on `GAME_OVER`* / *stay until @reviewer posts a verdict* will lose that race if the awaited event is more than ~30s away — the runtime will mark the agent complete and tear it down before the message arrives. This affects every recipe below: **Q/A, Broadcast/Ack, Peer Review, and Hand-Off all encode a wait.** +> +> **Two ways to handle it:** +> +> 1. **Per-agent escape hatch — set `idleThresholdSecs` on the agent.** The builder option `idleThresholdSecs` overrides the per-agent idle window (default 30s; `0` disables idle detection entirely). Use this when a specific agent reliably has long quiet stretches by design: +> ```typescript +> .agent('long-running-reviewer', { cli: 'claude', idleThresholdSecs: 0 }) +> ``` +> Pick the lowest value that comfortably exceeds your worst-case wait — `0` is safest but means a genuinely-hung agent will sit forever. +> +> 2. **Restructure to Per-turn interactive spawn.** For multi-turn flows where some turns are slow (LLM reasoning, manual review, paused work), use the **Per-turn interactive spawn** shape from the [coordination table](#choose-your-coordination-style--conversation-vs-pipeline): one bounded step per turn, with a deterministic step in between carrying state on disk. You still get interactive PTY + channel; the session is just short enough to never hit the idle window. +> +> The chat-native recipes below work as written when the next expected event reliably arrives well inside the idle window, or when you've extended the window per (1). +> +> Symptom to watch for in logs: a player exits with `Completion inferred from clean process exit (code 0) — no coordination signal was required` *before* the event it was supposed to wait for arrived; or a lead step that drives waits via shell `sleep` calls. + #### 1. Question / Answer (blocking ask) When agent A needs information only agent B has, instruct A to **post a direct question and wait for a reply** rather than guessing or proceeding. @@ -2230,6 +2249,9 @@ When you set `.pattern('supervisor')` (or `hub-spoke`, `fan-out`), the runner au |---------|-----| | Treating relay as transport, not as a coordination layer (every step is `preset: 'worker'`, every handoff is `{{steps.X.output}}`) | Default to **Conversation shape** for non-trivial work — interactive agents on a shared channel. Pipeline-shape is only correct when the work could be expressed as a `bash \| bash \| bash` pipe. | | Interactive agents on a channel whose task strings don't tell them to talk to each other | Pick a [Chat-Native Coordination Recipe](#chat-native-coordination-recipes) (Q/A, Broadcast/Ack, Peer Review, Standup, Hand-Off) and bake it into the task prompt — otherwise you're paying for a chat substrate you're not using | +| Task prompt that tells an interactive agent to wait for a later channel event | Idle detection (~30s) will complete the PTY before the event arrives. Set `idleThresholdSecs` (0 disables) or restructure to [Per-turn interactive spawn](#choose-your-coordination-style--conversation-vs-pipeline) | +| Using an LLM as a wait/grade oracle (`sleep 30` loops, "guess whether the secret matches") | If the comparison is computable (string match, test pass/fail, regex, schema validity), grade it in a **deterministic step**. LLMs should *produce*, not adjudicate fixed rules | +| Adding a repair step downstream of a gate that has no failure mode an agent can fix | "Workflows repair before they fail" applies when the gate has a real, agent-fixable failure surface. A deterministic gate downstream of deterministic work is correct by construction — a repair step there just burns a spawn cycle on a guaranteed no-op. Skip the repair, keep the hard gate | | All workflows run sequentially | Group independent workflows into parallel waves (4-7x speedup) | | Every step depends on the previous one | Only add `dependsOn` when there's a real data dependency | | Self-review step with no timeout | Set `timeout: 300_000` (5 min) — Codex hangs in non-interactive review | From 8f0f45fc9509f7a4c0edc156b8ee1e2ceddb8974 Mon Sep 17 00:00:00 2001 From: khaliqgant Date: Sun, 20 Sep 2026 17:49:56 -0700 Subject: [PATCH 08/12] fix(skills): match PR-owner subscribe and idle-window claims to live 12.4 delivery The PR glob wakes events whose resource_ref is that PR even when the inventory file lives under /reviews or /comments. Per-turn interactive spawn only beats idle detection when each turn itself finishes inside the window. --- skills/orchestrating-agent-relay/SKILL.md | 21 ++++++++++++------- skills/writing-agent-relay-workflows/SKILL.md | 4 ++-- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index 491d40e6..bd440116 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -524,8 +524,9 @@ worker still thinking. Defenses: Whenever a worker **opens or owns a PR**, subscribe **that live PR-owner agent** to the PR's GitHub provider resource immediately — the moment the PR -number exists, not after the first review comment. Review comments, pushes, -check changes, and discussion then wake the owner instead of waiting on the +number exists, not after the first review comment. Events whose +`resource_ref` is that PR (review submitted/edited, review comments, PR +pushes, PR-tied discussion) then wake the owner instead of waiting on the next poll. ```bash @@ -547,12 +548,16 @@ already-live owner. `--events` defaults to Relay `message.created,thread.reply`. 3. A Relayfile binding keyed by `(provider, resolved path glob)` `--resource` is a Relayfile VFS glob. `owner/repo` resolves to repository -scope; provider URLs are not accepted. A PR glob does **not** include GitHub -issue-comment paths (`/issues//comments/**`) even when those comments sit -on the PR. Reviews, root review-comments, and checks are inventoried at -repository-level `reviews/`, `comments/`, and `checks/` paths — a PR directory -does not cover them. Subscribe the PR glob first; add those extra globs only -when the owner must wake on those exact records. +scope; provider URLs are not accepted. The canonical PR glob is the owner +binding. Matching uses the event's `resource_ref` (for example +`/github/repos/__/pulls/by-id/.json`), not only the inventory +file path — so a review stored at `/github/repos///reviews/.json` +still wakes a `/pulls//**` owner when it is that PR's review. Extra globs +are for records that **never** carry this PR's `resource_ref` (a repo-wide +check on another SHA, an issue-comment path that only names `/issues//`, +or an independently inventoried pending review you must stage before its +terminal event). Do not give those extra globs to a helper; they are still +writable owner routes. One `(github, glob)` binding per resource. List bindings with `agent-relay integration subscribe --list`. This is distinct from `agent-relay integration subscription list`, which lists Relay diff --git a/skills/writing-agent-relay-workflows/SKILL.md b/skills/writing-agent-relay-workflows/SKILL.md index 7e27f40c..77dbfece 100644 --- a/skills/writing-agent-relay-workflows/SKILL.md +++ b/skills/writing-agent-relay-workflows/SKILL.md @@ -268,7 +268,7 @@ Before writing the workflow, decide *how the agents will coordinate*. The relay | Shape | What it is | Use when | |---|---|---| | **Conversation** (chat-native) | Interactive agents share a channel; messages, `@-mentions`, and ambient awareness drive coordination. Lead and workers spawn in parallel and self-organize. The relay is the coordination layer, not just transport. | Multi-file work, peer review loops, cross-agent feedback, dynamic re-planning, multi-PR coordination, anything with a human-in-the-loop escape, swarms where workers pick up each other's output. | -| **Per-turn interactive spawn** (bounded interactive) | Interactive PTY agents (no `preset`), but each step is bounded to *one* turn that exits as soon as its file/message is produced. Agents still join the channel — they just don't live across turns. | Turn-taking flows where each turn is short, well-defined, and reads its inputs from a deterministic source (file on disk, prior grade step). Exercises the PTY + channel-injection path without the idle-detection-vs-wait-for-event race that bites long-running interactive agents. | +| **Per-turn interactive spawn** (bounded interactive) | Interactive PTY agents (no `preset`), but each step is bounded to *one* turn that exits as soon as its file/message is produced. Agents still join the channel — they just don't live across turns. | Turn-taking flows where each turn is short, well-defined, finishes **inside the idle window**, and reads its inputs from a deterministic source (file on disk, prior grade step). Cuts the long-running wait-for-event race; a silent turn that itself exceeds ~30s still needs `idleThresholdSecs`. | | **Pipeline** (one-shot DAG) | Each step runs as a one-shot subprocess (`claude -p`, `codex exec`); steps hand off via `{{steps.X.output}}` text injection. No agents are alive at the same time; no chat happens. | Linear, well-specified transformations; deterministic data passing; no live agent-to-agent coordination during implementation. The selected review-depth path and deterministic final gates still apply. | **Default to Conversation for any non-trivial work.** Pipeline DAGs are simpler to reason about but they do not exercise the relay primitive — they are a Unix pipe with extra steps. If you would happily write the same task as a single shell pipeline, pipeline-shape is fine. Otherwise, you almost certainly want a Conversation shape. @@ -1736,7 +1736,7 @@ Once you're in the Interactive Team shape, the channel is your coordination medi > ``` > Pick the lowest value that comfortably exceeds your worst-case wait — `0` is safest but means a genuinely-hung agent will sit forever. > -> 2. **Restructure to Per-turn interactive spawn.** For multi-turn flows where some turns are slow (LLM reasoning, manual review, paused work), use the **Per-turn interactive spawn** shape from the [coordination table](#choose-your-coordination-style--conversation-vs-pipeline): one bounded step per turn, with a deterministic step in between carrying state on disk. You still get interactive PTY + channel; the session is just short enough to never hit the idle window. +> 2. **Restructure to Per-turn interactive spawn.** For multi-turn flows, use the **Per-turn interactive spawn** shape from the [coordination table](#choose-your-coordination-style--conversation-vs-pipeline): one bounded step per turn, with a deterministic step in between carrying state on disk. You still get interactive PTY + channel. This only beats idle detection when **each turn finishes well inside the window** — a slow reasoning turn still needs `idleThresholdSecs` (option 1). Do not treat per-turn spawn as a substitute for raising the threshold on a turn that is itself quiet for >30s. > > The chat-native recipes below work as written when the next expected event reliably arrives well inside the idle window, or when you've extended the window per (1). > From 19676901776cd775d50389c770db085c4d4e8660 Mon Sep 17 00:00:00 2001 From: khaliqgant Date: Sun, 20 Sep 2026 17:56:48 -0700 Subject: [PATCH 09/12] fix(orchestrating-agent-relay): default PR-owner recipe covers reviews, checks, comments Codex P1: the single /pulls//** command did not match the promised wake set. Subscribe the owner to the adapter nested reviews/status globs plus GitHub issue-comment paths as well. --- skills/orchestrating-agent-relay/SKILL.md | 84 ++++++++++++++--------- 1 file changed, 52 insertions(+), 32 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index bd440116..b53a2e5f 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -67,7 +67,7 @@ reference is the **`using-agent-relay`** skill. | Give a human a follow-along link | `agent-relay observer` | | Inspect a worker's TTY | `agent-relay node agent attach Worker1 --mode view` | | Fleet-wide agent inventory | `agent-relay fleet agent list` | -| Subscribe PR-owner to GitHub PR | `agent-relay integration subscribe github --resource '/github/repos///pulls//**' --to @Owner` | +| Subscribe PR-owner to GitHub PR | `agent-relay integration subscribe github --resource '/github/repos///pulls//**' --to @Owner` (also `pulls//reviews/**`, `pulls//status/**`, `issues//comments/**`) | | List integration bindings | `agent-relay integration subscribe --list` | | Unsubscribe a provider resource | `agent-relay integration unsubscribe github --resource '...'` | | Failed deliveries | `agent-relay node deadletters` | @@ -443,9 +443,9 @@ Quick Reference. Then enforce this protocol: (cross-node) for worker liveness; set a wall-clock fallback so a silently-dead worker can't hang the loop - The moment a worker opens or owns a GitHub PR, subscribe **that live - PR-owner identity** with `agent-relay integration subscribe github - --resource '/github/repos///pulls//**' --to @`. - Do not subscribe helpers to the same glob. After respawn under a new + PR-owner identity** to the PR glob **and** `pulls//reviews/**`, + `pulls//status/**`, and `issues//comments/**` (same `--to @`). + Do not subscribe helpers to those globs. After respawn under a new name, rebind. Verify a real GitHub event arrives in `check_inbox`, not just that `--list` shows a binding. Unsubscribe owned resources before release @@ -523,16 +523,29 @@ worker still thinking. Defenses: ## Subscribe the live PR owner Whenever a worker **opens or owns a PR**, subscribe **that live PR-owner -agent** to the PR's GitHub provider resource immediately — the moment the PR -number exists, not after the first review comment. Events whose -`resource_ref` is that PR (review submitted/edited, review comments, PR -pushes, PR-tied discussion) then wake the owner instead of waiting on the -next poll. +agent** to the PR's GitHub provider resources immediately — the moment the +PR number exists, not after the first review comment. Conversation comments, +reviews, review comments, pushes, and check changes then wake the owner +instead of waiting on the next poll. + +The default recipe is four owner bindings. Each `--resource` is a distinct +binding key, so these do not replace each other. Paths match the GitHub +adapter layout used by `writing-agent-relay-workflows` (`pulls//reviews/**`, +`pulls//status/**`) plus GitHub's issue-comment path for PR conversation +comments: ```bash -agent-relay integration subscribe github \ - --resource '/github/repos///pulls//**' \ - --to @ +OWNER=@ +REPO='/github/repos//' +N= +for resource in \ + "$REPO/pulls/$N/**" \ + "$REPO/pulls/$N/reviews/**" \ + "$REPO/pulls/$N/status/**" \ + "$REPO/issues/$N/comments/**" +do + agent-relay integration subscribe github --resource "$resource" --to "$OWNER" +done ``` `--to` is `@agent` or `#channel`. Point it at the owner handle. `--spawn ` @@ -548,16 +561,16 @@ already-live owner. `--events` defaults to Relay `message.created,thread.reply`. 3. A Relayfile binding keyed by `(provider, resolved path glob)` `--resource` is a Relayfile VFS glob. `owner/repo` resolves to repository -scope; provider URLs are not accepted. The canonical PR glob is the owner -binding. Matching uses the event's `resource_ref` (for example -`/github/repos/__/pulls/by-id/.json`), not only the inventory -file path — so a review stored at `/github/repos///reviews/.json` -still wakes a `/pulls//**` owner when it is that PR's review. Extra globs -are for records that **never** carry this PR's `resource_ref` (a repo-wide -check on another SHA, an issue-comment path that only names `/issues//`, -or an independently inventoried pending review you must stage before its -terminal event). Do not give those extra globs to a helper; they are still -writable owner routes. One `(github, glob)` binding per resource. +scope; provider URLs are not accepted. Matching also uses the event's +`resource_ref` (for example `/github/repos/__/pulls/by-id/.json`), +not only the inventory file path — so a review whose file is +`/github/repos///reviews/.json` can still wake a `/pulls//**` +binding when `resource_ref` names that PR. Do **not** rely on that alone for +conversation comments or checks: GitHub conversation comments on a PR are +inventoried under `/issues//comments/**`, and status/check records under +`/pulls//status/**`. Subscribe those globs in the default recipe above. +Do not give any of them to a helper; they are still writable owner routes. +One `(github, glob)` binding per resource. List bindings with `agent-relay integration subscribe --list`. This is distinct from `agent-relay integration subscription list`, which lists Relay @@ -585,10 +598,11 @@ Bindings follow the **recipient identity**, not the OS process. ### Helper agents: no second writable subscribe -Only the PR owner gets `--to @Owner` on that PR glob. A second `subscribe` to -the same `(github, path glob)` **replaces** the owner's binding. Reviewers, -shadows, and one-shot helpers read the PR through git/`gh` or channel traffic; -they do not get their own writable subscription to the owner's resource. +Only the PR owner gets `--to @Owner` on those owner globs. A second +`subscribe` to the same `(github, path glob)` **replaces** that route. +Reviewers, shadows, and one-shot helpers read the PR through git/`gh` or +channel traffic; they do not get their own writable subscription to the +owner's resources. ### Unsubscribe / release hygiene @@ -599,9 +613,15 @@ IDs are gone: agent-relay integration subscribe --list agent-relay integration webhook list agent-relay integration webhook list-inbound -agent-relay integration unsubscribe github \ - --resource '/github/repos///pulls//**' -# confirm the binding, webhookId, and webhookSubscriptionId disappeared +for resource in \ + "/github/repos///pulls//**" \ + "/github/repos///pulls//reviews/**" \ + "/github/repos///pulls//status/**" \ + "/github/repos///issues//comments/**" +do + agent-relay integration unsubscribe github --resource "$resource" +done +# confirm each binding, webhookId, and webhookSubscriptionId disappeared agent-relay fleet release ``` @@ -623,8 +643,8 @@ GitHub events will wake the agent. Verify in this order: live on a node. Roster-only is not enough. 4. **Actual event** — after a real GitHub comment, review, check, or push, the owner receives a Relay message from `github` whose body names the event and - a Relayfile path under that glob. Read it with `check_inbox` / - `list_messages`. Optional broker corroboration: a delivery-injected event + a Relayfile path (or `resource_ref`) for this PR. Read it with `check_inbox` + / `list_messages`. Optional broker corroboration: a delivery-injected event for that agent; `node deadletters` must not hold it. If steps 1–3 pass and step 4 does not, the subscription is not working. Do not @@ -871,7 +891,7 @@ this whole workaround once `relay#1446` lands rather than letting it outlive the | Worker self-removed; can't send review fixes | Instruct workers not to self-remove until told. If already gone, spawn a fresh worker and re-inject branch + commit SHA + full verdict; rebind any GitHub PR subscription to the new live identity (see [Subscribe the live PR owner](#subscribe-the-live-pr-owner)) | | Told the user to open an observer URL built from the workspace key | That is an admin credential in a query string, and the realtime endpoint rejects it. Run `agent-relay observer` (or `get_observer_url`) and share the `ot_live_` URL it returns | | Worker died silently; loop hangs | Inbox polling fires on messages only. Poll `agent-relay node agent list` / `fleet agent list` for liveness and set a wall-clock fallback (~30 min ScheduleWakeup) | -| Opened a PR but did not subscribe the live owner | Review comments, pushes, and checks will not wake anyone. Immediately `integration subscribe github --resource '/github/repos///pulls//**' --to @` | +| Opened a PR but did not subscribe the live owner | Review comments, pushes, and checks will not wake anyone. Immediately subscribe `@Owner` to `/pulls//**`, `/pulls//reviews/**`, `/pulls//status/**`, and `/issues//comments/**` | | Subscribed a helper/reviewer to the same PR glob | Binding key is `(provider, resolved path)` — the second subscribe **replaces** the owner's writable route. Only the PR-owner identity gets `--to @Owner` | | Treated `--list` / webhook create as delivery proof | Binding creation is not a wake. Confirm a real GitHub event arrives as a Relay message from `github` with a Relayfile path under that glob (`check_inbox` / `list_messages`); check `node deadletters` if it does not | | Respawned the owner under a new name and left the old binding | New identities do not inherit subscriptions. Re-subscribe `--to @NewOwner` on the same `--resource`; confirm the prior webhook/subscription IDs retired | From 9a247cca42c5a708c4314ac10a503b88ee257c00 Mon Sep 17 00:00:00 2001 From: Khaliq Date: Sun, 20 Sep 2026 19:58:31 -0700 Subject: [PATCH 10/12] skills: deprecate the v1 WorkflowBuilder skill, retarget its v2-relevant lessons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retarget pass over the folded-in #108 content plus the open review findings on this PR. Deprecation (v1 -> v2): - writing-agent-relay-workflows 1.6.25 -> 2.0.0. DEPRECATED banner at the top of SKILL.md and a rewritten frontmatter description, so a skill listing shows the deprecation and the pointer to writing-relayflows without opening the file. All v1 reference material kept intact below. - migrating-persona-to-relayflow 1.0.0 -> 1.1.0. Same short pointer, and corrects the stale "v2 is still under construction" bullet — v2 ships today as @relayflows/surface / @relayflows/sdk, CLI `flows`. - agent-relay-starter collection 1.1.0 -> 1.2.0: writing-relayflows is now the required workflow-authoring skill; the v1 skill drops to optional. Retargeted out of the v1 skill into relay-80-100-workflow 2.0.0 -> 2.1.0 (the v2 validation skill), because both lessons are about the ladder, not about either engine's syntax: - do not add a repair step to a gate with no agent-fixable failure surface - grade computable things in a deterministic step, never with an LLM Both also get an anti-pattern row. The idle-detection / per-turn spawn material stayed in the v1 skill: idleThresholdSecs, preset and interactive channel agents have no v2 equivalent. Review findings resolved in relay-80-100-workflow: - "None of that is reachable by a gate" was too categorical; the residue gate later in the skill shows the opposite - the re-record example used `/* same command */`, which does not compile; the command is now a named const reused by both record calls - require-green now writes BLOCKED_NO_COMMIT before its non-zero exit, so the artifact the skill requires actually exists when acceptance is red - `npx vitest run` -> `npx --no vitest run`, with a note to run tools from the project's own lockfile (CWE-494) writing-relayflows 1.2.0 -> 1.3.0: reconciles "always gate an agent step" with relay-80-100-workflow's "enforce after the agent, not on it" — the two skills contradicted each other. Where the check goes now depends on whether a red result should stop the run or become repairable work. Root agent-workforce-skills 1.5.0 -> 1.6.0. Co-Authored-By: Claude Opus 5 --- README.md | 11 ++--- prpm.json | 26 ++++++----- .../migrating-persona-to-relayflow/SKILL.md | 8 +++- skills/relay-80-100-workflow/SKILL.md | 45 ++++++++++++++----- skills/writing-agent-relay-workflows/SKILL.md | 20 ++++++++- skills/writing-relayflows/SKILL.md | 4 +- 6 files changed, 84 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index ba1edab3..2b33d655 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,19 @@ Skills, slash commands, and a Claude Code plugin for building multi-agent systems with Agent Relay. -Package metadata lives in [prpm.json](prpm.json). The repo currently publishes `agent-workforce-skills` version `1.1.6`. +Package metadata lives in [prpm.json](prpm.json). The repo currently publishes `agent-workforce-skills` version `1.6.0`. ## Published Skills | Skill | Version | Description | |-------|---------|-------------| | [choosing-swarm-patterns](skills/choosing-swarm-patterns/SKILL.md) | 1.1.5 | Pick the right Agent Relay orchestration pattern across the 10 core swarm patterns plus specialized patterns. | -| [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 1.6.25 | Build multi-agent workflows with WorkflowBuilder, DAG dependencies, Relayfile-backed Slack human assistance, integration subscriptions, waitFor gates, review-depth review/fix loops, channels, and chat-native coordination recipes. | +| [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 2.0.0 | **DEPRECATED** - the superseded v1 `@relayflows/core` WorkflowBuilder engine. Use [`writing-relayflows`](skills/writing-relayflows/SKILL.md) for v2; keep this only to maintain an existing v1 workflow. | | [setting-up-relayfile](skills/setting-up-relayfile/SKILL.md) | 1.1.1 | Set up Relayfile mounts and writeback for provider files through local filesystem access. | -| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.2.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | +| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.3.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | | [using-agent-relay](skills/using-agent-relay/SKILL.md) | 1.4.0 | Participant-side MCP reference for a **registered** relay agent (spawned worker / registered lead): messaging, channels, threads, reactions, search, inbox, actions, and worker spawn/release. Counterpart to `orchestrating-agent-relay`. | | [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.4.0 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention, including GitHub PR-owner integration subscriptions. | -| [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 2.0.0 | Close the 80-to-100 validation gap in a Relayflows v2 flow: the evidence recorder that turns a red check into work for a repair agent, repairable gates on the critical path, edit and hazard gates, and the fresh-eyes review rounds that catch what green gates miss. | +| [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 2.1.0 | Close the 80-to-100 validation gap in a Relayflows v2 flow: the evidence recorder that turns a red check into work for a repair agent, repairable gates on the critical path, edit and hazard gates, when *not* to add a repair step, and the fresh-eyes review rounds that catch what green gates miss. | | [review-fix-signoff-loop](skills/review-fix-signoff-loop/SKILL.md) | 1.0.2 | Loop review, repair, validation, and fresh-context dual-agent signoff until independent reviewers both satisfy the verdict contract. | | [trigger-autocomplete-catalog](skills/trigger-autocomplete-catalog/SKILL.md) | 1.0.0 | Enforce webhook/event trigger autocomplete coverage through KNOWN_TRIGGER_CATALOG in @relayfile/adapter-core. | | [activity-summary](skills/activity-summary/SKILL.md) | 1.0.0 | Answer "what did I work on yesterday" questions by reading `digests/yesterday.md` first instead of crawling provider directories. | @@ -67,10 +67,11 @@ npx prpm install collections/agent-relay-starter --as codex,claude This collection includes: - `@agent-relay/choosing-swarm-patterns` -- `@agent-relay/writing-agent-relay-workflows` +- `@agent-relay/writing-relayflows` - `@agent-workforce/trail-snippet` - optional `@agent-relay/relay-80-100-workflow` - optional `@agent-relay/review-fix-signoff-loop` +- optional `@agent-relay/writing-agent-relay-workflows` (deprecated v1 engine) Install the `relayfile-workspace` collection when you want the full Relayfile workspace primitive stack: diff --git a/prpm.json b/prpm.json index dd7f5592..81aa74c8 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.5.0", + "version": "1.6.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -10,7 +10,7 @@ "packages": [ { "name": "migrating-persona-to-relayflow", - "version": "1.0.0", + "version": "1.1.0", "description": "Use when migrating an existing defineAgent persona (in AgentWorkforce/agents or AgentWorkforce/internal-agents) to run its LLM-heavy work through a Relayflow v1 workflow via ctx.workflow.run(). Covers scope decisions, the 4-step durable-workflow shape, colocation with the persona, ctx.workflow.run() invocation, red-first + resume tests, useSubscription semantics for realtime triggers, and the agentworkforce >= 4.1.44 CI-deploy bump. Reference migration: hn-monitor PR #126.", "format": "claude", "subtype": "skill", @@ -47,8 +47,8 @@ }, { "name": "writing-agent-relay-workflows", - "version": "1.6.25", - "description": "Use when building multi-agent workflows with relay broker-sdk. Covers conversation vs pipeline coordination, WorkflowBuilder/DAG steps, agents, {{steps.X.output}} chaining, Relayfile-backed human assistance, Slack answer injection, integration subscriptions, waitFor gates, repairable verification, review-depth fresh-eyes review/fix loops with test hardening, channels, chat-native recipes, error handling, event listeners, step sizing, lead+workers teams, and parallel waves.", + "version": "2.0.0", + "description": "DEPRECATED - the superseded v1 @relayflows/core WorkflowBuilder engine (failOnError, captureOutput, {{steps.X.output}}, .onError()). All new work uses writing-relayflows (v2: @relayflows/surface / @relayflows/sdk, CLI `flows`). Keep this only to read or maintain an existing v1 workflow: conversation vs pipeline coordination, DAG steps, idle detection, human assistance, waitFor gates, review-depth review/fix loops, channels, and parallel waves.", "format": "claude", "subtype": "skill", "languages": [ @@ -95,8 +95,8 @@ }, { "name": "writing-relayflows", - "version": "1.2.0", - "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) \u2014 see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", + "version": "1.3.0", + "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) — see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", "format": "claude", "subtype": "skill", "languages": [ @@ -154,7 +154,7 @@ }, { "name": "relay-80-100-workflow", - "version": "2.0.0", + "version": "2.1.0", "description": "Use when writing a Relayflows v2 flow (@relayflows/surface / @relayflows/sdk, CLI `flows`) that must fully validate a feature end-to-end before it is sealed or merged - the 80-to-100 gap. Covers the evidence-recorder pattern that replaces v1's failOnError/captureOutput, repairable gates on the critical path, edit and hazard gates, gate cross-checking, fresh-eyes review rounds, and what a wall of green gates still misses.", "format": "claude", "subtype": "skill", @@ -479,7 +479,7 @@ "id": "agent-relay-starter", "name": "Agent Relay Starter", "description": "Essential skills for building multi-agent systems with Agent Relay - swarm pattern selection, workflow authoring, and trail debugging", - "version": "1.1.0", + "version": "1.2.0", "category": "development", "tags": [ "multi-agent", @@ -496,10 +496,16 @@ "reason": "Decision framework for picking the right multi-agent orchestration pattern" }, { - "packageId": "@agent-relay/writing-agent-relay-workflows", + "packageId": "@agent-relay/writing-relayflows", "version": "latest", "required": true, - "reason": "WorkflowBuilder API guide for authoring DAG-based multi-agent workflows with review-depth review/fix loops and test hardening" + "reason": "Authoring guide for the current Relayflows v2 engine (@relayflows/surface / @relayflows/sdk, CLI `flows`)" + }, + { + "packageId": "@agent-relay/writing-agent-relay-workflows", + "version": "latest", + "required": false, + "reason": "DEPRECATED v1 WorkflowBuilder reference - install only to maintain an existing v1 workflow" }, { "packageId": "@agent-workforce/trail-snippet", diff --git a/skills/migrating-persona-to-relayflow/SKILL.md b/skills/migrating-persona-to-relayflow/SKILL.md index 7a78e740..74240f29 100644 --- a/skills/migrating-persona-to-relayflow/SKILL.md +++ b/skills/migrating-persona-to-relayflow/SKILL.md @@ -5,6 +5,12 @@ description: Use when migrating an existing `defineAgent` persona (in AgentWorkf # Migrating a persona to Relayflow v1 +> **v1 engine.** This skill targets the superseded `@relayflows/core` engine via +> `ctx.workflow.run()`. The current engine is **v2** — `@relayflows/surface` / +> `@relayflows/sdk`, CLI `flows` — documented in +> [`writing-relayflows`](../writing-relayflows/SKILL.md). Use this skill only to move an +> existing persona onto v1's durable execution today. + ## When to use this skill You are working on a persona in `AgentWorkforce/agents` or `AgentWorkforce/internal-agents` that today does its work inline in a `defineAgent({ handler })` — LLM calls via `ctx.llm(...)`, ranking, curation, review — and you want to move the LLM-heavy portion behind a durable Relayflow v1 workflow via `ctx.workflow.run()` so that the work is exactly-once resumable, journaled, and step-level retriable. @@ -16,7 +22,7 @@ Read the RFC first: [`docs/RFC-0001-everything-is-a-relayflow.md`](https://githu ## Do not use this skill for - **Authoring a v1 workflow from scratch** — use [`writing-agent-relay-workflows`](../writing-agent-relay-workflows/SKILL.md). -- **Cloud v2 execution** — v2 is still under construction (see the RFC gate 3–4 work). Migrate to v1 first; v2 becomes an opt-in `relayflowVersion: "v2"` flip on the same `ctx.workflow.run()` call once v2 is generally available. +- **Authoring for the v2 engine** — v2 (`@relayflows/surface`/`@relayflows/sdk`, CLI `flows`, packages `2.0.x`) is a different product that ships today; see [`writing-relayflows`](../writing-relayflows/SKILL.md). This skill and `writing-agent-relay-workflows` cover the superseded **v1** engine. Personas still migrate to v1 first: v2 becomes an opt-in `relayflowVersion: "v2"` flip on the same `ctx.workflow.run()` call once the persona runtime supports it. - **Repointing a workflow between packages** (`@agent-relay/sdk/workflows` → `@relayflows/core`) — that is covered in `writing-agent-relay-workflows`. ## The seven-step recipe diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index 943b158d..91bc4de4 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -30,7 +30,7 @@ A 67-step flow of exactly this shape produced a tree on which **every determinis - a seam whose coordinator was rebuilt per call, so three of its four contract invariants had no production effect; - an adapter that had stopped observing whether its write landed, so a dead writer read as a successful delivery. -None of that is reachable by a gate. Gates check *properties you thought to name*. Review catches *the thing you did not think to name* — and on that run, that was most of the risk. +None of those defects was caught by the gates on that run. Gates check *properties you thought to name*. Review catches *the thing you did not think to name* — and on that run, that was most of the risk. So: build the gates, and do not treat them as the finish line. The review rounds below are not a formality bolted onto a working pipeline. They are where the defects were actually found. @@ -50,15 +50,17 @@ The shape that works is a small script of your own that journals verdicts to fil // and ALWAYS exits 0. // // node gates.mjs require-green --names unit,clippy,e2e -// Reads those files back and exits non-zero, printing the red tails, -// if any verdict is not 'green' or any runId is not this run's. +// Reads those files back. If any verdict is not 'green' or any runId is +// not this run's, it FIRST writes evidence/BLOCKED_NO_COMMIT.md naming +// each red record and its tail, THEN exits non-zero. ``` Three details are load-bearing: - **Base64-encode the command.** It will contain `&&`, pipes and quotes, and the calling shell must not reinterpret them. - **Stamp each record with the run that produced it** (`runId`, from the flow's input or an env var). `flows run --reuse-from ` keys on `step_spec_hash`, and a recorder step *always exits 0*, so it is always eligible for reuse — an unstamped recording resurrects a previous run's verdict, and `require-green` happily reads it as today's evidence. -- **`require-green` is the only thing that says "green."** Final acceptance recomputes the verdict from recordings; it never trusts an agent's report of its own work. +- **Run tools from the project's own lockfile** — `npx --no vitest run`, or a package script. A bare `npx` silently fetches a package the repo never pinned, so a gate can pass against a version nobody reviewed. +- **`require-green` is the only thing that says "green."** Final acceptance recomputes the verdict from recordings; it never trusts an agent's report of its own work. It is also the step that writes `BLOCKED_NO_COMMIT` — write the artifact *before* the non-zero exit, because v2 stops the run on that exit and nothing downstream will get the chance. Compared to the old `failOnError: false` this is more machinery, and it buys something the old engine did not have: the repair agent reads a **file**, not an interpolated string, so it sees the whole tail rather than a truncated template. @@ -76,7 +78,7 @@ const requireGreen = (...names: string[]) => `node gates.mjs require-green --names ${names.join(',')}`; export default flow('ship-feature', { budget: { wallclock: '110m' } }, async (f) => { - await f.run(record('unit', 'npx vitest run'), { timeout: '15m' }); // exits 0 either way + await f.run(record('unit', 'npx --no vitest run'), { timeout: '15m' }); // exits 0 either way await f.agent('repair-unit', { cli: 'claude', @@ -84,11 +86,11 @@ export default flow('ship-feature', { budget: { wallclock: '110m' } }, async (f) 'Read evidence/unit.json.', 'If verdict is "green", do nothing and say so. Do not invent work.', 'If verdict is "red", read the tail, fix the source or the test,', - 'and rerun `npx vitest run` until it passes.', + 'and rerun `npx --no vitest run` until it passes.', ].join('\n'), }); - await f.run(record('unit', 'npx vitest run'), { timeout: '15m' }); + await f.run(record('unit', 'npx --no vitest run'), { timeout: '15m' }); await f.run(requireGreen('unit')); // the only step here allowed to be red f.done('success'); @@ -133,6 +135,22 @@ Do not hang the enforcement of an agent's work on a gate attached to that agent A gate on an agent step makes a dropped transport look like a crashed run. The same work, checked by the next deterministic step, reads as *"nothing was written"* — a repairable fact, with a journaled tail explaining it. (`writing-relayflows` has the current per-gate caveats; the shipped gate set moves between releases, and at least one gate has silently swallowed its own output.) Independently of any of that, this is the better shape, and it is the same rule as keeping repairable gates on the critical path. +### Do not add a repair step to a gate that has nothing to repair + +The ladder applies where the gate has a real, agent-fixable failure surface. A +deterministic gate sitting downstream of deterministic work — a transcript +line-count check after a series of deterministic recorder steps — is correct by +construction; there is no failure mode a repair agent could act on. Wiring one in +anyway spawns a fresh agent for a guaranteed no-op, and pays the spawn cost plus, +often, a follow-on idle wait. Keep the hard gate, skip the repair. + +### Grade computable things in a deterministic step, never with an LLM + +If the comparison is computable — a string match, a test's pass/fail, a regex, schema +validity — grade it in a `run` step and record the verdict. An `llm` or `agent` step +asked to adjudicate a fixed rule will disagree with itself across runs, and its verdict +is not evidence. Agents *produce*; deterministic steps *judge*. + ### Keep repairable gates on the critical path Repair-before-failure only helps once the flow *reaches* a deterministic gate. If a long-running agent step is a hard dependency of the first gate, a dropped worker stops the flow before the repair loop ever sees evidence. @@ -154,20 +172,21 @@ This stops "the agent transport failed" from masquerading as "the product failed Never trust that an agent edited a file. After every agent edit, record a deterministic check and route its evidence to a repair owner. ```ts -await f.agent('edit-schema', { cli: 'claude', task: 'Edit lib/db/schema.ts …' }); - -await f.run(record('edit-schema', ` +const editSchemaCheck = ` if [ -z "$(git status --short -- lib/db/schema.ts)" ]; then echo NOT_MODIFIED; exit 1; fi grep -q my_new_table lib/db/schema.ts || { echo MISSING_TABLE; exit 1; } echo EDIT_OK -`)); +`; + +await f.agent('edit-schema', { cli: 'claude', task: 'Edit lib/db/schema.ts …' }); +await f.run(record('edit-schema', editSchemaCheck)); await f.agent('repair-edit-schema', { cli: 'claude', task: 'Read evidence/edit-schema.json. Green means do nothing. Red means make the edit land.', }); -await f.run(record('edit-schema', /* same command */)); +await f.run(record('edit-schema', editSchemaCheck)); await f.run(requireGreen('edit-schema')); ``` @@ -306,4 +325,6 @@ assert.equal(emitted[0].eventType, 'sandbox_created'); | Mutating the helper to prove coverage | Proves the helper is covered, not the call site that forgot it | Mutate where the bug would be written | | Sealing only the evidence directory | The reviewer is bound to a hash of the wrong thing | Digest the changed source too, and prove it bites | | Treating all-green as done | 15 findings, 2 disqualifying, on an all-green tree | Fresh-eyes review rounds before acceptance | +| A repair step on a gate with no agent-fixable failure | Burns a spawn cycle on a guaranteed no-op, plus the idle wait after it | Ladder only where an agent action could turn red green | +| An `llm`/`agent` step used as a grading oracle | A computable rule adjudicated by a model disagrees with itself across runs | Grade it in a `run` step and record the verdict | | Copying a v1 `.step({ failOnError, captureOutput })` | Those fields do not exist in v2; the flow will not compile | The evidence recorder | diff --git a/skills/writing-agent-relay-workflows/SKILL.md b/skills/writing-agent-relay-workflows/SKILL.md index 77dbfece..f2181580 100644 --- a/skills/writing-agent-relay-workflows/SKILL.md +++ b/skills/writing-agent-relay-workflows/SKILL.md @@ -1,10 +1,28 @@ --- name: writing-agent-relay-workflows -description: Use when building multi-agent workflows with relay broker-sdk. Covers conversation vs pipeline coordination, WorkflowBuilder/DAG steps, agents, {{steps.X.output}} chaining, Relayfile-backed human assistance, Slack answer injection, integration subscriptions, waitFor gates, repairable verification, review-depth fresh-eyes review/fix loops with test hardening, channels, chat-native recipes, error handling, event listeners, step sizing, lead+workers teams, and parallel waves. +description: DEPRECATED - documents the superseded v1 `@relayflows/core` WorkflowBuilder engine (`workflow().pattern().agent().step().run()`, `failOnError`, `captureOutput`, `{{steps.X.output}}`, `.onError()`). Use `writing-relayflows` for the current v2 engine (`@relayflows/surface`/`@relayflows/sdk`, CLI `flows`) on all new work. Keep this open only to read or maintain an existing v1 workflow: conversation vs pipeline coordination, DAG steps, idle detection, Relayfile-backed human assistance, waitFor gates, review-depth review/fix loops, channels, chat-native recipes, and parallel waves. --- # Writing Agent Relay Workflows +> ## ⚠️ DEPRECATED — this is the v1 engine +> +> This skill documents **Relayflows v1**: `@relayflows/core`'s `WorkflowBuilder`, the +> chained builder (`workflow('name').pattern('dag').agent(...).step(...).run()`) with +> `failOnError`, `captureOutput`, `{{steps.X.output}}` templating and `.onError()`. +> **v1 is superseded.** +> +> **New work goes to the v2 engine** — `@relayflows/surface`'s `flow()` and the +> YAML/JSON dialect compiled by `@relayflows/sdk`, CLI binary `flows`, packages `2.0.x`. +> Read [`writing-relayflows`](../writing-relayflows/SKILL.md) for the authoring surface +> and [`relay-80-100-workflow`](../relay-80-100-workflow/SKILL.md) for the v2 validation +> shape. None of `failOnError`, `captureOutput`, `{{steps.X.output}}`, `.onError()` or +> `idleThresholdSecs` exists in v2, so following this skill for a v2 flow produces +> something that will not compile. +> +> Everything below is kept intact for people reading or maintaining an **existing v1 +> workflow**. + ## Overview The relay broker-sdk workflow system orchestrates multiple AI agents (Claude, Codex, Gemini, Aider, Goose) through typed DAG-based workflows. Workflows can be written in **TypeScript** (preferred), **Python**, or **YAML**. diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 3df7474d..0bcc902e 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -194,7 +194,9 @@ const review = await f.agent('review', { task: '…', cli: 'claude' }) [flows#449](https://github.com/AgentWorkforce/flows/pull/449) (open at time of writing) adds journaled `artifact_exists` gates and real predicate-gate support (the executor runs the closure once, on the journaled value, and journals the verdict) — check whether it's merged before assuming predicate gates still don't work. Until it lands, `subprocess_gate`/`regex_match`/`word_count_bounds`/`references_input` are the only gates that actually run in TypeScript. -An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. Always give an `agent`/`llm` step a real, config-object gate — never the default, never a predicate. +An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. So never leave an `agent`/`llm` step on the default gate, and never on a predicate. + +**Where to put that check depends on what you want a red result to *do*.** A config-object gate on the step *ends the run* when it fails, and cannot distinguish "the agent produced the wrong thing" from "the agent's transport dropped" — both read as a crashed step. If a red result should instead become repairable work, move the same assertion into the deterministic step that follows the agent, where it reads as a journaled fact with a tail. See `relay-80-100-workflow` for that shape. Gate on the step when a red result really should stop the run. ## Helpers: journaled effects, not just Slack From b41db79a8be9a9fcce153c8b0d6ae270f130ad58 Mon Sep 17 00:00:00 2001 From: khaliqgant Date: Sun, 20 Sep 2026 22:48:38 -0700 Subject: [PATCH 11/12] docs: address PR review and distinguish subscription delivery from wake Session-Id: 01a0c27e-dc09-7ca1-945a-a174599050ea --- skills/orchestrating-agent-relay/SKILL.md | 31 +++++++++++++++-------- skills/relay-80-100-workflow/SKILL.md | 19 ++++++++++---- skills/writing-relayflows/SKILL.md | 16 ++++++------ 3 files changed, 43 insertions(+), 23 deletions(-) diff --git a/skills/orchestrating-agent-relay/SKILL.md b/skills/orchestrating-agent-relay/SKILL.md index b53a2e5f..51f76eab 100644 --- a/skills/orchestrating-agent-relay/SKILL.md +++ b/skills/orchestrating-agent-relay/SKILL.md @@ -525,19 +525,21 @@ worker still thinking. Defenses: Whenever a worker **opens or owns a PR**, subscribe **that live PR-owner agent** to the PR's GitHub provider resources immediately — the moment the PR number exists, not after the first review comment. Conversation comments, -reviews, review comments, pushes, and check changes then wake the owner -instead of waiting on the next poll. +reviews, review comments, pushes, and check changes can then wake the owner. +Binding creation alone does not establish end-to-end delivery; keep a bounded +GitHub polling fallback until the event types and harness wake are verified. The default recipe is four owner bindings. Each `--resource` is a distinct binding key, so these do not replace each other. Paths match the GitHub adapter layout used by `writing-agent-relay-workflows` (`pulls//reviews/**`, `pulls//status/**`) plus GitHub's issue-comment path for PR conversation -comments: +comments. Replace every angle-bracket placeholder in these shell examples +with the actual owner handle, repository, and PR number before execution: ```bash -OWNER=@ +OWNER='@' REPO='/github/repos//' -N= +N='' for resource in \ "$REPO/pulls/$N/**" \ "$REPO/pulls/$N/reviews/**" \ @@ -622,7 +624,7 @@ do agent-relay integration unsubscribe github --resource "$resource" done # confirm each binding, webhookId, and webhookSubscriptionId disappeared -agent-relay fleet release +agent-relay fleet release '' ``` Unsubscribe does not delete agents or channels. Do not unsubscribe by guessed @@ -644,10 +646,19 @@ GitHub events will wake the agent. Verify in this order: 4. **Actual event** — after a real GitHub comment, review, check, or push, the owner receives a Relay message from `github` whose body names the event and a Relayfile path (or `resource_ref`) for this PR. Read it with `check_inbox` - / `list_messages`. Optional broker corroboration: a delivery-injected event - for that agent; `node deadletters` must not hold it. - -If steps 1–3 pass and step 4 does not, the subscription is not working. Do not + / `list_messages`. Capture the GitHub event ID, Relay message ID, and + timestamps; test discussion, review, push, and check events separately. +5. **Idle harness wake** — with the owner idle before the event, confirm a new + agent turn consumes that message and rereads the authoritative PR/check + state. A channel message, inbox read, or broker delivery-injected event + alone does not prove an idle harness resumed. Inspect `node deadletters` + for failed delivery. + +If steps 1–3 pass but step 4 does not, delivery remains unverified. If step 4 +passes but step 5 does not, the remaining gap is harness wake/consumption. +In particular, a delivered comment does not establish `check_run.completed` +delivery or idle wake across other harnesses. Report those as separate +limitations and keep the polling fallback for unverified paths. Do not treat Cloud/Nango ingress success, an HTTP 200, or a channel post you wrote yourself as delivery proof. diff --git a/skills/relay-80-100-workflow/SKILL.md b/skills/relay-80-100-workflow/SKILL.md index 91bc4de4..1315b219 100644 --- a/skills/relay-80-100-workflow/SKILL.md +++ b/skills/relay-80-100-workflow/SKILL.md @@ -52,10 +52,12 @@ The shape that works is a small script of your own that journals verdicts to fil // node gates.mjs require-green --names unit,clippy,e2e // Reads those files back. If any verdict is not 'green' or any runId is // not this run's, it FIRST writes evidence/BLOCKED_NO_COMMIT.md naming -// each red record and its tail, THEN exits non-zero. +// every invalid record (including foreign-run green records), with name, +// verdict, expected runId, actual runId, and tail, THEN exits non-zero. +// Missing or unreadable records are invalid too; report unavailable fields. ``` -Three details are load-bearing: +Four details are load-bearing: - **Base64-encode the command.** It will contain `&&`, pipes and quotes, and the calling shell must not reinterpret them. - **Stamp each record with the run that produced it** (`runId`, from the flow's input or an env var). `flows run --reuse-from ` keys on `step_spec_hash`, and a recorder step *always exits 0*, so it is always eligible for reuse — an unstamped recording resurrects a previous run's verdict, and `require-green` happily reads it as today's evidence. @@ -169,15 +171,22 @@ This stops "the agent transport failed" from masquerading as "the product failed ## Verify every edit -Never trust that an agent edited a file. After every agent edit, record a deterministic check and route its evidence to a repair owner. +Never trust that an agent edited a file. Establish a clean pre-edit baseline for the target paths, then record a deterministic check after every agent edit and route its evidence to a repair owner. If existing work must be preserved, snapshot it and compare against that snapshot instead; do not clean it away. ```ts const editSchemaCheck = ` - if [ -z "$(git status --short -- lib/db/schema.ts)" ]; then echo NOT_MODIFIED; exit 1; fi + state=$(git status --short -- lib/db/schema.ts) || exit 1 + if [ -z "$state" ]; then echo NOT_MODIFIED; exit 1; fi grep -q my_new_table lib/db/schema.ts || { echo MISSING_TABLE; exit 1; } echo EDIT_OK `; +// Stop before editing if this path already has staged, unstaged, or untracked work. +await f.run(` + state=$(git status --short -- lib/db/schema.ts) || exit 1 + if [ -n "$state" ]; then echo DIRTY_BASELINE; exit 1; fi +`); + await f.agent('edit-schema', { cli: 'claude', task: 'Edit lib/db/schema.ts …' }); await f.run(record('edit-schema', editSchemaCheck)); @@ -192,7 +201,7 @@ await f.run(requireGreen('edit-schema')); **Use `git status --short -- `, not `git diff --quiet`.** `git diff` only sees tracked changes, so a valid new package, test directory or generated artifact is misclassified as "no changes." `git diff --quiet` is fine only for tracked-only edits to files that already exist. -**Verify** that the file was modified, and that key content exists (a table name, an exported symbol, an import). **Do not verify** exact content, formatting, line counts or byte sizes — agents format differently and those checks are pure brittleness. +**Verify** that the file changed from the pre-edit baseline (reuse that same baseline for repair and final acceptance), and that key content exists (a table name, an exported symbol, an import). **Do not verify** exact content, formatting, line counts or byte sizes — agents format differently and those checks are pure brittleness. ### Gate the hazard, not a tool's mode diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 0bcc902e..72087a3b 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -126,7 +126,7 @@ export const FLOW_COMPLETION_REASONS = ['success', 'step_failed', 'canceled', 'b An authored body can meaningfully call `f.done('success')`, `f.done('step_failed')`, `f.done('needs_human')` (parks the run — it's not a kernel cancellation), or `f.done('declined')` (a deliberate decision not to act — also not a cancellation). `'canceled'` and `'budget_exceeded'` exist in the same closed set but are the **kernel's** to record; a body calling them itself is calling the wrong verdict for what actually happened. The verdict for a human declining a durable approval is `f.done('declined')`, not `f.done('canceled')` — a human declining is a decision, not the kernel calling off the run. (Correct verdict *if* the approval call reaches that point — `f.human` itself doesn't execute yet; see **Human approval and dispatch**.) -`Step` (`dist/step.d.ts`) is a `PromiseLike` with one extra method, `.gate(...)` — see **Verification gates**, which has a critical caveat for the predicate form. Every awaited step must actually be awaited — an unawaited or manually-`.then()`-chained step is refused (`unawaited_step` / a manual `.then()` is not an await), not silently dropped. +`Step` (`dist/step.d.ts`) is a `PromiseLike` with one extra method, `.gate(...)` — see **Verification gates**, which distinguishes current predicate support from older-runtime limitations. Every awaited step must actually be awaited — an unawaited or manually-`.then()`-chained step is refused (`unawaited_step` / a manual `.then()` is not an await), not silently dropped. ## Verification gates @@ -172,7 +172,7 @@ gate(predicate: (value: T) => boolean, because?: string): Step; // replay — so the executor refuses this branch with unsupported_gate." ``` -**Verified for real, this session, against `flows@main` (post-v2.0.16, current as of 2026-09-17):** any flow using the predicate form — `.gate((out) => out.trim().length > 0, "some reason")` — fails at `flows run` time with: +**Historical limitation before flows#449 (post-v2.0.16, tested 2026-09-17):** a flow using the predicate form — `.gate((out) => out.trim().length > 0, "some reason")` — fails at `flows run` time with: ``` FAILED [protocol_error] relayflowd could not complete the run request: unsupported_gate: @@ -180,7 +180,7 @@ postfix .gate(predicate) closures cannot be journaled; use .gate({type: "…", slice-P named gate instead. ``` -This reproduced identically on two different real example flows (`examples/pr-review-pipeline`, `examples/dependency-upgrade-bot` in `AgentWorkforce/flows`) that both used the predicate form — neither got past their first gate. **Always use the config-object form**: +This reproduced identically on two different real example flows (`examples/pr-review-pipeline`, `examples/dependency-upgrade-bot` in `AgentWorkforce/flows`) that both used the predicate form — neither got past their first gate. **On those older runtimes, use the config-object form**: ```ts const outdated = await f.run('npm outdated --json 2>/dev/null || true') @@ -192,11 +192,11 @@ const review = await f.agent('review', { task: '…', cli: 'claude' }) (Verified for real: `.gate({ type: 'subprocess_gate', command: 'test -n "$INPUT" && echo "$INPUT" | grep -q pkg' })` against a step outputting `{"pkg":"left-pad"}` passes — `$INPUT`, not `$FLOWS_INPUT`, is what the author's command sees. `FLOWS_INPUT` is an env var the SDK's own generated wrapper reads internally to extract the value before invoking your command; it is never visible to `command` itself.) -[flows#449](https://github.com/AgentWorkforce/flows/pull/449) (open at time of writing) adds journaled `artifact_exists` gates and real predicate-gate support (the executor runs the closure once, on the journaled value, and journals the verdict) — check whether it's merged before assuming predicate gates still don't work. Until it lands, `subprocess_gate`/`regex_match`/`word_count_bounds`/`references_input` are the only gates that actually run in TypeScript. +[flows#449](https://github.com/AgentWorkforce/flows/pull/449) merged on 2026-09-17. Relayflows v2.0.22 supports journaled `artifact_exists` and predicate gates: the executor runs the predicate closure on the journaled value and journals its verdict. Older installations need the version-specific limitations above. -An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. So never leave an `agent`/`llm` step on the default gate, and never on a predicate. +An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. Validate every `agent`/`llm` result, either directly with a supported gate or in a following deterministic verifier. -**Where to put that check depends on what you want a red result to *do*.** A config-object gate on the step *ends the run* when it fails, and cannot distinguish "the agent produced the wrong thing" from "the agent's transport dropped" — both read as a crashed step. If a red result should instead become repairable work, move the same assertion into the deterministic step that follows the agent, where it reads as a journaled fact with a tail. See `relay-80-100-workflow` for that shape. Gate on the step when a red result really should stop the run. +**Where to put that check depends on what you want a red result to do.** A config-object or predicate gate directly on the agent/LLM step ends the run when it fails; use it when that failure should be terminal. For repairable failures, the following deterministic verifier must explicitly read the preceding output or workspace, produce its own verification result, and record it for repair. Output-dependent gates inspect the step they guard: moving the same gate unchanged would inspect the verifier's output, not the agent's. Use the record/repair/re-record/assert pattern in `relay-80-100-workflow`, gating the final verification result after repair. ## Helpers: journaled effects, not just Slack @@ -406,7 +406,7 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe ## Common mistakes - **Forgetting `version` in a YAML/JSON `FlowSpec`.** Required, not optional. -- **Using a predicate `.gate((v) => …, reason)` in TypeScript.** Refused at runtime (`unsupported_gate`) until [flows#449](https://github.com/AgentWorkforce/flows/pull/449) lands. Use a config-object gate — see **Verification gates**. +- **Using a predicate gate on an older runtime.** Runtimes before [flows#449](https://github.com/AgentWorkforce/flows/pull/449) reject it with `unsupported_gate`; v2.0.22 supports journaled predicates. See **Verification gates**. - **Putting `tools`, `budget`, or any other `FlowHeader` field into `flows.json`.** `flows.json` only accepts `cli`, `executors`, `models`, `mcp`, `deploy` — anything else is `config_invalid`. - **Assuming `flows.json`'s `models` sets a default model.** It only validates models already declared elsewhere; it never selects one. - **Declaring `.on(...)` trigger handlers without registering the provider in `flows.json`'s `executors`.** Even a flow you only ever run directly (never via a real webhook) needs this, or it's refused `no_executor` before your input is even read. @@ -434,7 +434,7 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe | `f.human(question, {to})` | TS only | **not executable yet** — `unsupported_verb`; typechecks, fails at `flows run`. [flows#400](https://github.com/AgentWorkforce/flows/issues/400) | | `f.dispatch(flow, input)` | TS only | **not executable yet** — `unsupported_verb`; typechecks, fails at `flows run` | | `f.done(reason)` | TS / kernel | one of `success \| step_failed \| needs_human \| declined` from an authored body; `canceled \| budget_exceeded` are kernel-only | -| `.gate({type: '...', ...})` | TS | the only `.gate()` form that runs today — config object, never a predicate | +| `.gate({type: '...', ...})` | TS | config-object gates; v2.0.22 also supports `.gate(predicate, because)` | | `options.transport: 'relay'` | TS/YAML `agent` step | dispatch to Agent Relay's task infra; request+receipt, not live chat | | `Promise.all([...steps])` | TS | first-class supported concurrent fan-out | | `flows.json`: `cli, executors, models, mcp, deploy` | project config | exact accepted key set — nothing else | From b373c76ea82ae05644d2317b7e5e9c79de71999c Mon Sep 17 00:00:00 2001 From: Khaliq Date: Sun, 20 Sep 2026 22:55:53 -0700 Subject: [PATCH 12/12] fix(writing-relayflows): reconcile the 2.0.16/2.0.22 merge against published 2.0.22 The skill was refreshed against @relayflows/surface 2.0.16 and then merged with notes taken at 2.0.22, leaving both halves standing. Every claim below was re-read from the published 2.0.22 .d.ts and dist/*.js (npm latest; there is no 2.0.23), and the losing side deleted rather than hedged. - AgentOptions.permissions: it IS a TypeScript option, added in 2.0.17, and PermissionsSpec is exported from the package root. The SDK validates it, lowers it to file_globs/network_allowlist/access_preset and journals it, and nothing enforces it (preflight warns permissions_unenforced; flows#442). Removed from the two "YAML/JSON-only, no TS equivalent" lists; the remaining YAML-only agent fields are recoveryMode, surfaces, output and the agent: selector. - Predicate .gate() and the artifact_exists gate ship (flows#449 landed). The skill claimed both were refused with unsupported_gate "until #449 lands"; the quick-reference repeated it. Now stated once: both forms run, one per step, and only a config object is provable by flows check. - f.human executes: it parks on a kernel wait.human, answered with flows answer human- yes|no and continued by flows resume. Only f.dispatch is still unsupported_verb. - Ctx.human returns Step, not Promise. - Narrowed the unknown-field claim: a bad flow header throws a TypeError at runtime; a bad AgentOptions field is a tsc error, not a runtime one. - Verified against now records 2.0.22 and names which side of each contradiction won. Already correct at HEAD and left alone: run(command, options?), f.done's six FLOW_COMPLETION_REASONS, and the absence of budget.maxWallclockMs (FlowHeader.budget is string | {tokens?, dollars?, wallclock?}). writing-relayflows 1.3.0 -> 1.4.0, root 1.6.0 -> 1.7.0. Co-Authored-By: Claude Opus 5 --- README.md | 4 +- prpm.json | 4 +- skills/writing-relayflows/SKILL.md | 100 ++++++++++++++--------------- 3 files changed, 54 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index 2b33d655..b9a79cf4 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Skills, slash commands, and a Claude Code plugin for building multi-agent systems with Agent Relay. -Package metadata lives in [prpm.json](prpm.json). The repo currently publishes `agent-workforce-skills` version `1.6.0`. +Package metadata lives in [prpm.json](prpm.json). The repo currently publishes `agent-workforce-skills` version `1.7.0`. ## Published Skills @@ -11,7 +11,7 @@ Package metadata lives in [prpm.json](prpm.json). The repo currently publishes ` | [choosing-swarm-patterns](skills/choosing-swarm-patterns/SKILL.md) | 1.1.5 | Pick the right Agent Relay orchestration pattern across the 10 core swarm patterns plus specialized patterns. | | [writing-agent-relay-workflows](skills/writing-agent-relay-workflows/SKILL.md) | 2.0.0 | **DEPRECATED** - the superseded v1 `@relayflows/core` WorkflowBuilder engine. Use [`writing-relayflows`](skills/writing-relayflows/SKILL.md) for v2; keep this only to maintain an existing v1 workflow. | | [setting-up-relayfile](skills/setting-up-relayfile/SKILL.md) | 1.1.1 | Set up Relayfile mounts and writeback for provider files through local filesystem access. | -| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.3.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | +| [writing-relayflows](skills/writing-relayflows/SKILL.md) | 1.4.0 | Author a Relayflows v2 flow (`@relayflows/surface`/`@relayflows/sdk`, the `flows` CLI) in TypeScript or YAML/JSON — the run/llm/agent ladder, verification gates, cli/model resolution, flows.json, and real `flows check`/`run` refusal shapes. Not the older `@relayflows/core` WorkflowBuilder engine. | | [using-agent-relay](skills/using-agent-relay/SKILL.md) | 1.4.0 | Participant-side MCP reference for a **registered** relay agent (spawned worker / registered lead): messaging, channels, threads, reactions, search, inbox, actions, and worker spawn/release. Counterpart to `orchestrating-agent-relay`. | | [orchestrating-agent-relay](skills/orchestrating-agent-relay/SKILL.md) | 2.4.0 | The canonical way to run agent-relay: self-bootstrap the broker (`agent-relay node up`) and autonomously spawn, monitor, and coordinate a worker team over the relay MCP without human intervention, including GitHub PR-owner integration subscriptions. | | [relay-80-100-workflow](skills/relay-80-100-workflow/SKILL.md) | 2.1.0 | Close the 80-to-100 validation gap in a Relayflows v2 flow: the evidence recorder that turns a red check into work for a repair agent, repairable gates on the critical path, edit and hazard gates, when *not* to add a repair step, and the fresh-eyes review rounds that catch what green gates miss. | diff --git a/prpm.json b/prpm.json index 81aa74c8..46566a3b 100644 --- a/prpm.json +++ b/prpm.json @@ -1,6 +1,6 @@ { "name": "agent-workforce-skills", - "version": "1.6.0", + "version": "1.7.0", "description": "Skills for multi-agent coordination - swarm patterns, workflow building, relay usage, and orchestrating agent-relay", "author": "khaliqgant", "organization": "agent-relay", @@ -95,7 +95,7 @@ }, { "name": "writing-relayflows", - "version": "1.3.0", + "version": "1.4.0", "description": "Use when authoring a Relayflows flow (@relayflows/surface / @relayflows/sdk, the v2 journal-based engine, CLI `flows`) in TypeScript or YAML/JSON. Covers the run/llm/agent ladder, human/dispatch/done, verification gates, cli/model resolution, flows.json, and flows check/run/resume refusal shapes. Not for the older @relayflows/core WorkflowBuilder (chained .pattern('dag')/.agent()/.step() calls) — see writing-agent-relay-workflows / migrating-persona-to-relayflow instead.", "format": "claude", "subtype": "skill", diff --git a/skills/writing-relayflows/SKILL.md b/skills/writing-relayflows/SKILL.md index 72087a3b..77671e6f 100644 --- a/skills/writing-relayflows/SKILL.md +++ b/skills/writing-relayflows/SKILL.md @@ -73,11 +73,11 @@ steps: model: claude-sonnet-4-6 ``` -**Version note.** This skill was refreshed against `@relayflows/surface@2.0.16` and `@relayflows/sdk@2.0.16` — the current published npm packages, not a source build. Both examples above pass `flows check` for real against those versions (see **Verified against**). `2.0.17` shipped since, fixing the `flows run --cloud` bug documented below (**Running it**) — that one claim is re-verified against `2.0.17`; everything else in this document was checked against `2.0.16` and hasn't been independently re-run on `2.0.17`, though nothing in its release notes suggests the other claims changed. +**Version note.** Every type in this skill is read from the published `@relayflows/surface@2.0.22` / `@relayflows/sdk@2.0.22` `.d.ts` files — the current npm `latest`, not a source build (see **Verified against**). Three things changed between `2.0.16` and `2.0.22`, and each is stated below at its shipped shape rather than at both: `AgentOptions.permissions` landed in `2.0.17`; predicate `.gate()` and the `artifact_exists` gate landed with [flows#449](https://github.com/AgentWorkforce/flows/pull/449); and `f.human` became executable. ## The real `Ctx` contract (TypeScript) -From the shipped `@relayflows/surface@2.0.16` `.d.ts` files (`dist/context.d.ts`, `dist/step.d.ts`, `dist/flow.d.ts`, `dist/completion.d.ts`): +From the shipped `@relayflows/surface@2.0.22` `.d.ts` files (`dist/context.d.ts`, `dist/step.d.ts`, `dist/flow.d.ts`, `dist/completion.d.ts`): ```ts export interface AgentResult { @@ -85,9 +85,16 @@ export interface AgentResult { artifacts: string[]; } +export interface PermissionsSpec { // exported from the package root + fileGlobs?: string[]; + networkAllowlist?: string[]; + accessPreset?: 'readonly' | 'readwrite'; +} + export interface AgentOptions { task: string; workspace?: string; + permissions?: PermissionsSpec; // since 2.0.17 — declared and journaled, NOT enforced cli?: string; model?: string; cwd?: string; // working dir for the CLI subprocess; defaults to the flow-runner's cwd @@ -106,7 +113,7 @@ export interface Ctx extends Helpers { // Helpers = f.slack, f.github, f.linea llm(strings: TemplateStringsArray, ...values: unknown[]): Step; llm(prompt: string, options: LlmOptions): Step; agent(name: string, options: AgentOptions): Step; - human(question: string, options: { to: string }): Promise; + human(question: string, options: { to: string }): Step; dispatch(flow: string, input: unknown): Promise; done(reason: FlowCompletionReason): void; cloud: CloudHelper; @@ -114,7 +121,9 @@ export interface Ctx extends Helpers { // Helpers = f.slack, f.github, f.linea } ``` -Do not add fields to `AgentOptions`/`Ctx` that aren't in this list — they don't exist in the shipped SDK. In particular: **no flow-level named-agent selector on `f.agent`, no `recoveryMode`/`permissions`/`surfaces`** on the TypeScript call site. Those are YAML/JSON step fields today. `FlowHeader` (`dist/flow.d.ts`) allows `use`, `identity`, `memory`, `budget`, `tools`, `workspace` — no bare `agents` map key (TypeScript composes reusable flows via `use: string[]` instead — see **Named agents and flow composition**). Passing an unknown header field or an unknown `AgentOptions` field throws a `TypeError` at authoring time, before anything is journaled. +Do not add fields to `AgentOptions`/`Ctx` that aren't in this list — they don't exist in the shipped SDK. In particular: **no flow-level named-agent selector on `f.agent`, no `recoveryMode`/`surfaces`/`output`** on the TypeScript call site. Those stay YAML/JSON step fields. `FlowHeader` (`dist/flow.d.ts`) allows `use`, `identity`, `memory`, `budget`, `tools`, `workspace` — no bare `agents` map key (TypeScript composes reusable flows via `use: string[]` instead — see **Named agents and flow composition**). An unknown *header* field throws a `TypeError` at authoring time, before anything is journaled; an unknown `AgentOptions` field is caught by `tsc`, not at runtime. + +**`permissions` is a real TypeScript option and it does not sandbox anything.** Since `2.0.17` it is on `AgentOptions`, validated by the SDK, lowered to `file_globs`/`network_allowlist`/`access_preset` and journaled with the step — and nothing reads it to gate a file or a network call. `flows check` warns `permissions_unenforced` on any declaration, including `{}`. Enforcement is [flows#442](https://github.com/AgentWorkforce/flows/issues/442). Declare it to record intent; do not rely on it to contain an agent. `run()`'s second argument is a real, current option: `{ timeout?: string | number }` — a duration string (`"15m"`) or milliseconds. Every real cookbook example that runs something slower than a few seconds sets this explicitly. @@ -124,9 +133,9 @@ Do not add fields to `AgentOptions`/`Ctx` that aren't in this list — they don' export const FLOW_COMPLETION_REASONS = ['success', 'step_failed', 'canceled', 'budget_exceeded', 'needs_human', 'declined'] as const; ``` -An authored body can meaningfully call `f.done('success')`, `f.done('step_failed')`, `f.done('needs_human')` (parks the run — it's not a kernel cancellation), or `f.done('declined')` (a deliberate decision not to act — also not a cancellation). `'canceled'` and `'budget_exceeded'` exist in the same closed set but are the **kernel's** to record; a body calling them itself is calling the wrong verdict for what actually happened. The verdict for a human declining a durable approval is `f.done('declined')`, not `f.done('canceled')` — a human declining is a decision, not the kernel calling off the run. (Correct verdict *if* the approval call reaches that point — `f.human` itself doesn't execute yet; see **Human approval and dispatch**.) +An authored body can meaningfully call `f.done('success')`, `f.done('step_failed')`, `f.done('needs_human')` (parks the run — it's not a kernel cancellation), or `f.done('declined')` (a deliberate decision not to act — also not a cancellation). `'canceled'` and `'budget_exceeded'` exist in the same closed set but are the **kernel's** to record; a body calling them itself is calling the wrong verdict for what actually happened. The verdict for a human declining a durable approval is `f.done('declined')`, not `f.done('canceled')` — a human declining is a decision, not the kernel calling off the run. -`Step` (`dist/step.d.ts`) is a `PromiseLike` with one extra method, `.gate(...)` — see **Verification gates**, which distinguishes current predicate support from older-runtime limitations. Every awaited step must actually be awaited — an unawaited or manually-`.then()`-chained step is refused (`unawaited_step` / a manual `.then()` is not an await), not silently dropped. +`Step` (`dist/step.d.ts`) is a `PromiseLike` with one extra method, `.gate(...)` — see **Verification gates**. Every awaited step must actually be awaited — an unawaited or manually-`.then()`-chained step is refused (`unawaited_step` / a manual `.then()` is not an await), not silently dropped. ## Verification gates @@ -137,6 +146,7 @@ Verification is control flow, not decoration — a gate decides whether a step a ```ts export type VerificationSpec = ExitCodeGate | OutputContainsGate | JsonSchemaGate | NamedDataGate; // NamedDataGate = ReferencesInputGate | SubprocessGate | WordCountBoundsGate | RegexMatchGate +// | ArtifactExistsGate ``` - `exit_code` — implicit default for `deterministic` steps. Not configurable; writing it explicitly is allowed and compiles to the same thing as omitting it. @@ -144,6 +154,7 @@ export type VerificationSpec = ExitCodeGate | OutputContainsGate | JsonSchemaGat - `json_schema` — step output validates against a JSON Schema (`boolean | Record`). Used for structured LLM/agent output; `output?: JsonOutputSchema` on an `llm`/`agent` step is sugar that compiles to this. - `subprocess_gate` — runs `command` under `/bin/sh`, judged on its exit code. The gate that actually works for "check a file the agent wrote": `{ type: 'subprocess_gate', command: 'test -s review.md' }`. If you also need to inspect the step's own output (via `from_output`), the SDK's generated wrapper reads its internal `FLOWS_INPUT` env var, extracts the value, and passes it to *your* command as `$INPUT` — `FLOWS_INPUT` itself is never visible to the author's command, only `$INPUT` is: `{ type: 'subprocess_gate', command: 'echo "$INPUT" | grep -q PASSED', from_output: ['summary'] }`. - `regex_match` — the step's output (or a value at `in_output_at`) matches `pattern`, evaluated by a non-backtracking RE2 engine (only `i`, `m`, `s` flags). +- `artifact_exists` — passes when the step's journaled `artifacts` list contains `path` (working-directory-relative POSIX). It reads the journal, not the disk, so it sees exactly what the worker measured at `step.completed`; a path the worker never journaled reads as missing even when the file is there. - `word_count_bounds`, `references_input` — narrower named gates; see `packages/sdk/src/named-gates.ts` in the flows repo for the exact contract. ```yaml @@ -159,32 +170,26 @@ export type VerificationSpec = ExitCodeGate | OutputContainsGate | JsonSchemaGat ### TypeScript (`Step.gate(...)`) — read this before you write one -`Step` has **two** overloads, and only one of them works today: +`Step` has **two** overloads, and **one `.gate()` per step** — a second throws `unsupported_gate`: ```ts gate(config: NamedGate): Step; -// NamedGate = ReferencesInputNamedGate | SubprocessNamedGate | WordCountBoundsNamedGate | RegexMatchNamedGate -// — the surface-visible subset of NamedDataGate above. This one is journal-honest and runs. +// NamedGate = ReferencesInputNamedGate | SubprocessNamedGate | WordCountBoundsNamedGate +// | RegexMatchNamedGate | ArtifactExistsNamedGate +// — the surface-visible subset of NamedDataGate above. Data, not code: `flows check` can prove it. gate(predicate: (value: T) => boolean, because?: string): Step; -// The type itself documents why this is dangerous — straight from the shipped .d.ts: -// "Predicate gates cannot be journaled — the JavaScript closure would not survive -// replay — so the executor refuses this branch with unsupported_gate." +// Runs since flows#449. The executor calls the closure once, on the journaled value, and +// journals the VERDICT as a lowered `.gate` step — so resume and replay read the +// recorded verdict and never re-run the function. A false verdict fails the run as +// `gate_failed`, carrying `because`. ``` -**Historical limitation before flows#449 (post-v2.0.16, tested 2026-09-17):** a flow using the predicate form — `.gate((out) => out.trim().length > 0, "some reason")` — fails at `flows run` time with: - -``` -FAILED [protocol_error] relayflowd could not complete the run request: unsupported_gate: -postfix .gate(predicate) closures cannot be journaled; use .gate({type: "…", …}) with a -slice-P named gate instead. -``` - -This reproduced identically on two different real example flows (`examples/pr-review-pipeline`, `examples/dependency-upgrade-bot` in `AgentWorkforce/flows`) that both used the predicate form — neither got past their first gate. **On those older runtimes, use the config-object form**: +The predicate form was refused with `unsupported_gate` through `2.0.16`; on `2.0.22` it runs. **Prefer the config-object form anyway** — `flows check` cannot prove a predicate (docs/SURFACE.md §6), so a named gate is the only kind a preflight can tell you about before you spend a run: ```ts const outdated = await f.run('npm outdated --json 2>/dev/null || true') - .gate({ type: 'subprocess_gate', command: 'test -n "$INPUT"' }); // not .gate((out) => out.length > 0, '…') + .gate({ type: 'subprocess_gate', command: 'test -n "$INPUT"' }); // provable by `flows check` const review = await f.agent('review', { task: '…', cli: 'claude' }) .gate({ type: 'subprocess_gate', command: 'test -s review.md' }); // checks a file the agent wrote, ignores $INPUT @@ -192,8 +197,6 @@ const review = await f.agent('review', { task: '…', cli: 'claude' }) (Verified for real: `.gate({ type: 'subprocess_gate', command: 'test -n "$INPUT" && echo "$INPUT" | grep -q pkg' })` against a step outputting `{"pkg":"left-pad"}` passes — `$INPUT`, not `$FLOWS_INPUT`, is what the author's command sees. `FLOWS_INPUT` is an env var the SDK's own generated wrapper reads internally to extract the value before invoking your command; it is never visible to `command` itself.) -[flows#449](https://github.com/AgentWorkforce/flows/pull/449) merged on 2026-09-17. Relayflows v2.0.22 supports journaled `artifact_exists` and predicate gates: the executor runs the predicate closure on the journaled value and journals its verdict. Older installations need the version-specific limitations above. - An agent step rarely fails by crashing; it fails by returning something plausible and wrong, which a plain retry-on-error never catches. Validate every `agent`/`llm` result, either directly with a supported gate or in a following deterministic verifier. **Where to put that check depends on what you want a red result to do.** A config-object or predicate gate directly on the agent/LLM step ends the run when it fails; use it when that failure should be terminal. For repairable failures, the following deterministic verifier must explicitly read the preceding output or workspace, produce its own verification result, and record it for repair. Output-dependent gates inspect the step they guard: moving the same gate unchanged would inspect the verifier's output, not the agent's. Use the record/repair/re-record/assert pattern in `relay-80-100-workflow`, gating the final verification result after repair. @@ -289,7 +292,7 @@ const planner = { cli: 'claude', model: 'claude-opus-5' }; await f.agent('plan', { ...planner, task: '…' }); ``` -What TypeScript *does* have, which the map doesn't give you, is `FlowHeader.use?: string[]` — "relative paths to reusable authored flows composed by this body" ([flows#300](https://github.com/AgentWorkforce/flows/issues/300), closed). This is a real field in the shipped `2.0.16` type; this skill hasn't independently run a flow that exercises it, so treat its exact runtime semantics as a pointer to verify against `docs/SURFACE.md` and the flows repo, not as tested guidance the way the rest of this document is. +What TypeScript *does* have, which the map doesn't give you, is `FlowHeader.use?: string[]` — "relative paths to reusable authored flows composed by this body" ([flows#300](https://github.com/AgentWorkforce/flows/issues/300), closed). This is a real field in the shipped `2.0.22` type; this skill hasn't independently run a flow that exercises it, so treat its exact runtime semantics as a pointer to verify against `docs/SURFACE.md` and the flows repo, not as tested guidance the way the rest of this document is. ## Parallel agents @@ -324,30 +327,25 @@ What this actually is, per `docs/AGENT-RELAY-TRANSPORT.md`: the step calls `POST Don't describe this as "agents talking to each other" — there's no documented pattern in this skill's scope for two flow steps to hold a live back-and-forth over Relay channels while both are running. What's real: the step's agent becomes a genuine Agent Relay workspace participant while it runs (so a human, or another agent with the right access, can DM it and potentially steer it — per the type's own wording), and the flow still only sees a request-and-durable-receipt shape, not an open channel. Multi-agent *coordination* inside a flow today means sequential handoff (one step's return value feeds the next) or parallel fan-out (**Parallel agents**, above) — not live messaging. -## Human approval and dispatch (declared, not yet runnable) +## Human approval and dispatch -`Ctx` declares `human(question, {to})` and `dispatch(flow, input)`, and both typecheck and pass `flows check`. **Neither actually runs today.** Verified for real against `@relayflows/surface@2.0.16`: +`Ctx` declares `human(question, {to})` and `dispatch(flow, input)`. **`f.human` runs on `2.0.22`; `f.dispatch` still does not** — the executor throws `unsupported_verb: the initial authored executor does not lower f.dispatch`. Both typecheck and pass `flows check`, so `flows check` green tells you nothing about which is which. -``` -$ flows run human.flow.ts --local-agent --input '{}' -FAILED [protocol_error] relayflowd could not complete the run request: -unsupported_verb: the initial authored executor does not lower f.human - -$ flows run dispatch.flow.ts --local-agent --input '{}' -FAILED [protocol_error] relayflowd could not complete the run request: -unsupported_verb: the initial authored executor does not lower f.dispatch -``` - -Type availability does not establish executable support for either verb — the shipped executor throws `unsupported_verb` for both, identically, regardless of what the rest of the flow does. [flows#400](https://github.com/AgentWorkforce/flows/issues/400) tracks wiring up durable human approval; there's no dedicated tracking issue found yet for `f.dispatch`. Until one of these lands, treat both as authoring-surface-only: +`f.human` parks the run durably on a kernel `wait.human`, keyed `human-` in the order the body asked. A person answers, and `flows resume` continues the body from that line: ```ts -// This typechecks and passes `flows check` — it will FAIL at `flows run` time. -// Do not ship a flow whose only path to done() runs through f.human or f.dispatch. const ok = await f.human(`Ship this?\n${plan.summary}`, { to: 'khaliq' }); -if (!ok) return f.done('declined'); // correct verdict, if this call ever executes +if (!ok) return f.done('declined'); // a person's "no" is declined, never canceled +``` + +``` +$ flows answer human-1 yes --by khaliq --note 'reviewed the diff' +$ flows resume ``` -If you need durable approval today, do it in YAML with `permissions`/`recoveryMode: manual` (parks the run as `needs_human` with a diff for a person to resolve), or split the decision out to a separate, human-triggered flow run rather than a mid-flow `f.human` call. +`to` names who is asked and is recorded with the question — **it is not a delivery address**; nothing notifies that person for you. The kernel closes a wait once, so the first answer wins, and it records `attribution: "client_asserted"` because the daemon socket, not the kernel, authenticated whoever `--by` names. The answer step takes a `.gate(config)` like any other step. One caveat: `f.human` needs a durable root run to park in — under a runner that has none it refuses with `unsupported_verb`, so run it with `flows run`. + +For `f.dispatch`, split the work into a separate, independently triggered flow run. Do not ship a flow whose only path to `done()` goes through it. **The same "types accept it, the executor doesn't" trap applies to `workspace` on an agent step under `--local-agent`.** Neither an annotated (`'acme/api: readonly'`) nor a bare (`'acme/api'`) value works — both fail identically: @@ -356,7 +354,7 @@ REFUSED [invalid_spec] unsupported_workspace_permission: The local agent worker stream-only steps. Remove workspace or attach a worker that holds its revision pins. ``` -It isn't the `: readonly`/`: readwrite` annotation syntax that's rejected — it's that the local-agent worker doesn't hold workspace revision pins at all. Omit `workspace` entirely for a step you intend to run with `--local-agent`; it's a real, working field only against a worker that supports it (Cloud's). +It isn't the `: readonly`/`: readwrite` annotation syntax that's rejected — it's that the local-agent worker doesn't hold workspace revision pins at all. Omit `workspace` entirely for a step you intend to run with `--local-agent`; it's a real, working field only against a worker that supports it (Cloud's). Against a worker that does hold pins, the suffix is separately refused, and the remedy the SDK's own message names is `f.agent`'s `permissions` option — which records the scope and does not enforce it. ## Running it: `flows check` / `run` / `deploy` / `schedule` @@ -406,7 +404,7 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe ## Common mistakes - **Forgetting `version` in a YAML/JSON `FlowSpec`.** Required, not optional. -- **Using a predicate gate on an older runtime.** Runtimes before [flows#449](https://github.com/AgentWorkforce/flows/pull/449) reject it with `unsupported_gate`; v2.0.22 supports journaled predicates. See **Verification gates**. +- **Attaching two `.gate()` calls to one step.** `unsupported_gate`: a step takes one. Also note a predicate gate is invisible to `flows check` — prefer a config object when preflight should be able to prove it. - **Putting `tools`, `budget`, or any other `FlowHeader` field into `flows.json`.** `flows.json` only accepts `cli`, `executors`, `models`, `mcp`, `deploy` — anything else is `config_invalid`. - **Assuming `flows.json`'s `models` sets a default model.** It only validates models already declared elsewhere; it never selects one. - **Declaring `.on(...)` trigger handlers without registering the provider in `flows.json`'s `executors`.** Even a flow you only ever run directly (never via a real webhook) needs this, or it's refused `no_executor` before your input is even read. @@ -421,7 +419,7 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe - **Triggers/webhooks** (`.on(github.pull_request(...), ...)`), **memory retrieval**, and the full **helper** catalog's per-provider quirks (`f.mcp`, provider-specific settings) — each is its own surface; see the [Relayflows product docs](https://agentrelay.com/docs/relayflows) and the [flows-cookbook](https://github.com/AgentWorkforce/flows-cookbook) for real, run-verified examples of each. - **Named-agent map equivalent in TypeScript beyond `use:`** — `FlowHeader.use` exists in the shipped type but this skill hasn't independently run a flow that exercises it; verify current semantics against `docs/SURFACE.md` before relying on this section alone. -- **YAML-only agent step fields with no TypeScript equivalent**: `recoveryMode`, `permissions`, `surfaces`, and enforced `workspace` (TypeScript's `workspace` string is accepted by the type but rejected by the local-agent worker — see **Human approval and dispatch**). Author that step in YAML if you need them; `f.dispatch` is not a working escape hatch for this today (see below). +- **YAML-only agent step fields with no TypeScript equivalent**: `recoveryMode`, `surfaces`, `output`, the `agent:` named-agent selector, and enforced `workspace` (TypeScript's `workspace` string is accepted by the type but rejected by the local-agent worker — see **Human approval and dispatch**). Author that step in YAML if you need them; `f.dispatch` is not a working escape hatch for this today (see below). `permissions` is **not** on this list — it is a TypeScript option too (see **The real `Ctx` contract**). - The **older `@relayflows/core` `WorkflowBuilder`** engine — see `writing-agent-relay-workflows` and `migrating-persona-to-relayflow` in this repo. ## Quick reference @@ -431,10 +429,10 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe | `f.run(command, {timeout?})` / `type: deterministic` | both | shell command, implicit `exit_code` gate | | `f.llm(...)` / `type: llm` | both | bare model call, no workspace | | `f.agent(name, opts)` / `type: agent` | both | harnessed coding agent, returns `{summary, artifacts}` | -| `f.human(question, {to})` | TS only | **not executable yet** — `unsupported_verb`; typechecks, fails at `flows run`. [flows#400](https://github.com/AgentWorkforce/flows/issues/400) | -| `f.dispatch(flow, input)` | TS only | **not executable yet** — `unsupported_verb`; typechecks, fails at `flows run` | +| `f.human(question, {to})` | TS only | parks the run on a durable `wait.human`; `flows answer human- yes\|no` then `flows resume` | +| `f.dispatch(flow, input)` | TS only | **not executable** — `unsupported_verb`; typechecks and passes `flows check`, fails at `flows run` | | `f.done(reason)` | TS / kernel | one of `success \| step_failed \| needs_human \| declined` from an authored body; `canceled \| budget_exceeded` are kernel-only | -| `.gate({type: '...', ...})` | TS | config-object gates; v2.0.22 also supports `.gate(predicate, because)` | +| `.gate(config)` / `.gate(predicate, because)` | TS | one per step; both run since flows#449, but only a config object is provable by `flows check` | | `options.transport: 'relay'` | TS/YAML `agent` step | dispatch to Agent Relay's task infra; request+receipt, not live chat | | `Promise.all([...steps])` | TS | first-class supported concurrent fan-out | | `flows.json`: `cli, executors, models, mcp, deploy` | project config | exact accepted key set — nothing else | @@ -446,6 +444,8 @@ Both are real, both are exit 2 — the same underlying problem can print a diffe ## Verified against -`@relayflows/surface@2.0.16` and `@relayflows/sdk@2.0.16` (the published npm packages) — types read directly from the shipped `.d.ts` files, not from documentation or memory. Beyond static type-checking, the claims in this refresh that carry a **real run**, not just `flows check`, are backed by the recipes in [`AgentWorkforce/flows-cookbook`](https://github.com/AgentWorkforce/flows-cookbook) — each recipe's own README states exactly what was run, when, and what the result was (a real local run, a real Cloud deploy, or both), rather than duplicating that evidence here where it will go stale. If a claim in this skill and a cookbook recipe's README disagree, trust whichever was verified more recently — check the README's date. +**`@relayflows/surface@2.0.22` and `@relayflows/sdk@2.0.22`** — the current npm `latest` (there is no `2.0.23`). Every type, union and option in this skill was read directly from those packages' shipped `.d.ts` and `dist/*.js`, not from documentation or memory. + +This revision resolved a set of contradictions left by an earlier refresh that was verified at `2.0.16` and then merged with notes taken at `2.0.22`. Each was settled against the published package, and the losing side was deleted rather than hedged: `AgentOptions.permissions` exists on the TypeScript call site (added in `2.0.17`, validated and journaled, not enforced — preflight warns `permissions_unenforced`); `run(command, options?)` takes a second `{timeout}` argument; `f.done` takes the six `FLOW_COMPLETION_REASONS`; there is no `budget.maxWallclockMs` (`FlowHeader.budget` is `string | {tokens?, dollars?, wallclock?}`); predicate `.gate()` and the `artifact_exists` gate both ship; and `f.human` executes while `f.dispatch` still does not. Beyond static type-checking, the claims in this refresh that carry a **real run**, not just `flows check`, are backed by the recipes in [`AgentWorkforce/flows-cookbook`](https://github.com/AgentWorkforce/flows-cookbook) — each recipe's own README states exactly what was run, when, and what the result was (a real local run, a real Cloud deploy, or both), rather than duplicating that evidence here where it will go stale. If a claim in this skill and a cookbook recipe's README disagree, trust whichever was verified more recently — check the README's date. -The predicate-gate failure, the `flows.json` schema refusal, the `--data-dir` git-hygiene interaction, and the `flows run --cloud` bug were all reproduced directly while building that cookbook, on `AgentWorkforce/flows@main` past the `v2.0.16` tag (2026-09-17). A subsequent review pass (Devin, on this PR) flagged three more claims worth checking rather than trusting on sight — `f.human`/`f.dispatch` executability, the `workspace: readonly` annotation, and the `subprocess_gate` env var name — all three were independently re-run against the real CLI, confirmed as real breaks (one of the review's own suggested fixes, for `workspace`, turned out not to work either — the bare form fails identically to the annotated one), and corrected in the sections above rather than taken on the reviewer's word alone. +The `flows.json` schema refusal, the `--data-dir` git-hygiene interaction, and the `flows run --cloud` bug were all reproduced directly while building that cookbook, on `AgentWorkforce/flows@main` past the `v2.0.16` tag (2026-09-17); they are unchanged in `2.0.22`'s shipped code. The predicate-gate refusal reproduced there too — and it is the one claim `2.0.22` overturned, which is why the section above now documents the working form instead. An earlier review pass (Devin, on this PR) flagged three more claims worth checking rather than trusting on sight — `f.human`/`f.dispatch` executability, the `workspace: readonly` annotation, and the `subprocess_gate` env var name. The last two were re-run against the real CLI and confirmed as real breaks, still present in `2.0.22` (one of the review's own suggested fixes, for `workspace`, turned out not to work either — the bare form fails identically to the annotated one). The first has since split: `f.human` executes on `2.0.22`, `f.dispatch` does not.