From 9d47b3e1a3f08b852a753fe8e5280159250e551c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 16:34:26 +0000 Subject: [PATCH 01/14] Plan: agentic-engineering constructs for Loop, broken into stories MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Translate "The New SDLC with Vibe Coding" into a Loop roadmap, in the deck's own vocabulary. Adds a story-by-story implementation plan and a dogfooded pipeline (today's grammar building tomorrow's constructs). - docs/agentic-engineering-plan.md: the epic broken into 13 stories across 4 waves (eval core, the dials, harness completeness, niche/ standards) plus a cross-cutting docs reframe; each story names files, a runnable `done when`, dependencies, and gates. - agentic-engineering.loop: the same epic as a pipeline (epic->stages = stories), validated via `loop-run parse` with its flow printed. Headline: make EVALS first-class (tests vs evals; output vs trajectory), the gap every analysis lens ranked #1 — "without both, it is vibe coding". Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- agentic-engineering.loop | 131 +++++++++++++++++++++++ docs/agentic-engineering-plan.md | 172 +++++++++++++++++++++++++++++++ 2 files changed, 303 insertions(+) create mode 100644 agentic-engineering.loop create mode 100644 docs/agentic-engineering-plan.md diff --git a/agentic-engineering.loop b/agentic-engineering.loop new file mode 100644 index 0000000..b5a09da --- /dev/null +++ b/agentic-engineering.loop @@ -0,0 +1,131 @@ +# Agentic Engineering for Loop — the epic, as a pipeline. +# Written in TODAY'S Loop grammar to build TOMORROW'S constructs (evals, rigor, hooks, …). +# Detailed spec per story: docs/agentic-engineering-plan.md +# Each stage is one story; stories run in order and a failing story halts the rest. + +models: fast haiku, strong opus + +git: + work on a branch + commit each story + +pipeline "epic: agentic engineering for Loop": + + # ── Wave 1 · Eval core (the headline: "without both, it is always vibe coding") ── + + stage "story: eval targets in the IR": + goal: doneWhen becomes a conjunction of predicates and the skill predicate gains a target (output or trajectory, default output) plus an inline `the bar:` rubric + look at: spec/loop-spec.schema.json, packages/parser/src/types.ts, packages/parser/src/parser.ts, and the last failure + a human approves the plan first + done when "npm test --workspace packages/parser" passes + each cycle: plan, then act, then observe + when it fails: reflect on whether the schema and types and parser stayed in sync, then plan again + after 6 tries: stop and warn "IR change is rippling — needs a human" + + stage "story: capture the trajectory": + goal: the runtime retains a per-cycle trajectory and hands trajectory evals the cycle log instead of only the output summary + look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, packages/runtime/src/types.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + each cycle: plan, then act, then observe + when it fails: reflect on what the verifier actually receives, then plan again + after 6 tries: stop and warn "trajectory capture stuck" + + stage "story: anti-thrash and feedback wiring": + goal: a blocking trajectory eval requires `the bar:`, borderline verdicts escalate to a human, eval-flaps feed the thrash guard, and reflect sees the trajectory + look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + each cycle: plan, then act, then observe + when it fails: reflect on whether the stochastic judge can still self-thrash, then plan again + after 6 tries: stop and warn "eval gate still flaky" + + stage "story: the without-both lint": + goal: a code loop with a test but no eval (or vice-versa) warns — without both, it is vibe coding + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure + done when "npm test --workspace packages/parser" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 5 tries: stop and warn "lint stuck" + + # ── Wave 2 · The dials (the spectrum, and who's in the loop) ── + + stage "story: the rigor dial": + goal: rigor (vibe coding, structured ai-assisted, agentic engineering) expands to bundled defaults via the preset cascade and sets the without-both lint severity + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/stdlib, and the last failure + a human approves the plan first + done when "npm test --workspace packages/parser" passes + each cycle: plan, then act, then observe + when it fails: reflect on the lowest-wins cascade, then plan again + after 6 tries: stop and warn "rigor cascade stuck" + + stage "story: conductor and orchestrator mode": + goal: mode (conductor or orchestrator) names the in-session vs async postures and warns on the incoherent vibe-plus-orchestrator combo + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure + done when "npm test --workspace packages/parser" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 5 tries: stop and warn "mode stuck" + + # ── Wave 3 · Harness completeness ── + + stage "story: hooks at lifecycle points": + goal: a hooks block runs deterministic checks at named points (before each cycle, after act, on commit, on push, on stop) and a failing hook blocks + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/git.ts, and the last failure + done when "npm test --workspaces" passes + each cycle: plan, then act, then observe + when it fails: reflect on the hooks-vs-also-vs-doneWhen carve, then plan again + after 6 tries: stop and warn "hooks stuck" + + stage "story: observe and the OpEx report": + goal: a per-cycle trace plus token-and-cost metering, a stop-time OpEx report, and an optional cost ceiling that stops and warns + look at: packages/runtime/src/engine.ts, packages/runtime/src/summary.ts, packages/runtime/src/show.ts, and the last failure + done when "npm test --workspace packages/runtime" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 6 tries: stop and warn "observe stuck" + + # ── Wave 4 · Niche / standards ── + + stage "story: sandbox run isolation": + goal: a sandbox block declares no-network, an egress allowlist, and cpu-memory-time caps as config instead of prose + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, examples/forge-sandbox.loop, and the last failure + a human approves the plan first + done when "npm test --workspaces" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 6 tries: stop and warn "sandbox stuck" + + stage "story: knowledge and examples context": + goal: examples (imitate) and knowledge (read, never edit) complete the six context-engineering parts + look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspace packages/parser" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 5 tries: stop and warn "context keywords stuck" + + stage "story: parallel stages": + goal: stages-in-parallel fans out concurrently with a worktree per branch and a merge, backing orchestrator multi-agent work + look at: packages/runtime/src/engine.ts, packages/runtime/src/git.ts, packages/parser/src/parser.ts, and the last failure + a human approves the plan first + done when "npm test --workspace packages/runtime" passes + each cycle: plan, then act, then observe + when it fails: reflect on edit collisions and the merge, then plan again + after 6 tries: stop and warn "parallel stages — concurrency risk; needs a human" + + stage "story: standards and identity": + goal: use-tools-from names MCP servers, the flow handoff is framed as an A2A contract, and runs-as gives background runs an auditable identity + look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure + done when "npm test --workspaces" passes + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 5 tries: stop and warn "standards stuck" + + # ── Cross-cutting · Framing ── + + stage "story: reframe docs in deck vocabulary": + goal: AGENTS.md and README name what Loop already is in the deck's words (intent is the new interface, conductor and orchestrator, editor terminal background, Agent equals Model plus Harness, CapEx and OpEx) with the new constructs documented and their flows printed + look at: AGENTS.md, README.md, docs/MANUAL.md, and the last failure + a human reviews before stopping + done when a human confirms "the docs read in the deck's vocabulary and the new constructs are covered" + each cycle: plan, then act, then observe + when it fails: reflect, then plan again + after 4 tries: stop and warn "docs need a human pass" diff --git a/docs/agentic-engineering-plan.md b/docs/agentic-engineering-plan.md new file mode 100644 index 0000000..444106f --- /dev/null +++ b/docs/agentic-engineering-plan.md @@ -0,0 +1,172 @@ +# Plan — Agentic Engineering constructs for Loop + +> Bringing *"The New SDLC with Vibe Coding"* into Loop, in the deck's own vocabulary. +> Branch: `claude/loop-lang-concepts-3p5tz1`. Companion pipeline: [`agentic-engineering.loop`](../agentic-engineering.loop). + +## Context + +The deck *"The New SDLC with Vibe Coding: From ad-hoc prompting to agentic engineering"* argues +that the move from vibe coding to **agentic engineering** is about *"how much structure, +verification and judgment surround the output."* Loop already embodies the disciplined end — its +`plan → act → observe` + reflect back-edge **is** the deck's agent loop, and `goal:` + `done when` +**is** the factory model (*"success criteria, not step-by-step instructions"*). This epic closes the +gaps the deck exposes and **names** what Loop already is, using the deck's exact terms. + +The headline gap, ranked #1 by every analysis lens: **evals**. The deck's load-bearing claim is +*"Without both [TESTS and EVALS], it is always vibe coding."* Loop has TESTS (deterministic, +checked by code) and a proto-eval fused into `done when the skill … scores N` — but it is unnamed, +its rubric is invisible, and it can only judge the **output**, never the **trajectory** (*"not just +what was produced but HOW the agent got there"*). Trajectory is exactly what catches the 80%-problem +failures: *"conceptual failures that look right and may pass basic tests."* + +This is **building tomorrow's Loop using today's Loop**: the companion `.loop` pipeline plans this +work as an agentic-engineering pipeline, written entirely in the *current* grammar. + +## The epic → stories + +Stories are grouped into four waves (deck terminology in **bold**). Each is a `stage` in the +companion pipeline. Stories run in order; a failing story halts the rest. Risky stories (IR +changes, run isolation, real concurrency) carry a human gate. + +| # | Story | Wave | Gate | `done when` | +|---|---|---|---|---| +| 1 | Eval targets in the IR | 1 · Eval core | approve plan | `npm test --workspace packages/parser` | +| 2 | Capture the trajectory | 1 · Eval core | — | `npm test --workspace packages/runtime` | +| 3 | Anti-thrash + feedback wiring | 1 · Eval core | — | `npm test --workspace packages/runtime` | +| 4 | The "without both" lint | 1 · Eval core | — | `npm test --workspace packages/parser` | +| 5 | The `rigor:` dial | 2 · Dials | approve plan | `npm test --workspace packages/parser` | +| 6 | Conductor / Orchestrator `mode:` | 2 · Dials | — | `npm test --workspace packages/parser` | +| 7 | `hooks:` — lifecycle checkpoints | 3 · Harness | — | `npm test --workspaces` | +| 8 | `observe:` — trace + OpEx report | 3 · Harness | — | `npm test --workspace packages/runtime` | +| 9 | `sandbox:` — run isolation | 4 · Niche | approve plan | `npm test --workspaces` | +| 10 | `knowledge:` + `examples:` context | 4 · Niche | — | `npm test --workspace packages/parser` | +| 11 | Parallel stages | 4 · Niche | approve plan | `npm test --workspace packages/runtime` | +| 12 | Standards & identity (MCP / A2A / `runs as:`) | 4 · Niche | — | `npm test --workspaces` | +| 13 | Reframe docs in deck vocabulary | cross-cutting | review before stop | a human confirms | + +--- + +## Wave 1 — Eval core (the headline) + +### Story 1 — Eval targets in the IR +Generalize verification from one predicate to a conjunction, and make an eval's **target** explicit. +- **Spec/IR** (`spec/loop-spec.schema.json`, `packages/parser/src/types.ts`): `Loop.doneWhen` + becomes `Predicate[]` (all must pass — the conjunction that lets a loop require a TEST **and** an + EVAL). Extend the `skill` predicate variant with `target?: "output" | "trajectory"` (default + `"output"`) and `bar?: string` (the inline rubric). +- **Parser** (`packages/parser/src/parser.ts`): collect every `done when` line into the list; parse + the `on the output` / `on the trajectory` qualifier and an indented `the bar:` block under a skill + predicate. Back-compat: a bare skill predicate parses as an output eval. +- **`done when`** stays the single verification verb — reject a second verb. Existing `.loop` files + are unchanged (their single predicate becomes a one-element list). +- **Gate:** human approves the plan (IR shape change ripples through parser + runtime). + +### Story 2 — Capture the trajectory +The runner already parses every tool call with its inputs (`interpretStreamLine`/`toolLines`, +`runners/claudeCode.ts:77–126`) and **discards** them after the live display. Retain them. +- **Runtime** (`packages/runtime/src/engine.ts`, `runners/claudeCode.ts`, `types.ts`): accumulate + the per-cycle activity into a trajectory record — `{ plan, toolCalls[{tool,input}], filesTouched, + observe, reflection }`. Add `SkillVerifyInput.trajectory`. +- Route verifier context by target: **output** evals receive the diff/output (today's + `lastActSummary`, `engine.ts:211`); **trajectory** evals receive the cycle log. +- A loop's `done` requires every predicate in the list to pass (test + output eval + trajectory eval). + +### Story 3 — Anti-thrash + feedback wiring +Keep a stochastic judge from self-thrashing, and let reflection see the path. +- A **blocking** trajectory eval requires an explicit `the bar:` (no naked path-judge gates `done`). +- A borderline verdict escalates via the existing `when blocked: ask a human` path instead of retrying. +- An eval-flap counter feeds the `after N tries` thrash guard so a flaky judge alone can't burn the budget. +- Thread the trajectory into `ReflectInput` so `reflect on the path it took` fixes the path, not the artifact. + +### Story 4 — The "without both" lint +Turn the deck's slogan into a check. For **code loops** only (loops with an `edit` allowance): +warn when a loop has a TEST but no EVAL (or vice-versa). Severity is wired to `rigor:` in Story 5 +(warn by default; error under `rigor: agentic engineering`). Lives in the parser/validator so +`loop-run parse` surfaces it. + +## Wave 2 — The dials + +### Story 5 — The `rigor:` dial +Config-tier knob — `rigor: vibe coding | structured ai-assisted | agentic engineering` — that +expands to bundled defaults over existing knobs, reusing the `use the ` preset machinery +(`Config.use`/`useOverrides`) and the lowest-wins cascade already used for git/models. Sets the +"without both" lint severity. **Gate:** human approves the plan (touches the config cascade). + +### Story 6 — Conductor / Orchestrator `mode:` +Config-tier `mode: conductor | orchestrator` — names what `/loopflow` (in-session, gates inline) +and `loop-run` (async, opens a PR) already do. Orthogonal to `rigor:`; warn on the incoherent +combo `rigor: vibe coding` + `mode: orchestrator` (the deck's *"token burn from unverified loops"* +running unattended). + +## Wave 3 — Harness completeness + +### Story 7 — `hooks:` — deterministic checks at lifecycle points +A `hooks:` block binding a deterministic predicate (`passes` / `finds nothing`) to a lifecycle +point — `before each cycle · after act · on commit · on push · on stop`. A failing hook blocks +(*"hooks block unsafe commits"*), generalizing the hard-coded never-push-to-main. Carve: +success criterion → `done when`; recurring blocking checkpoint → `hooks:`; one-time polish → `also:`. + +### Story 8 — `observe:` — trace + OpEx report +Per-cycle trace (plan/act/observe + tools called), token & cost metering split fast/strong, a +stop-time OpEx report (cycles, reflects, first-pass success, hit-the-guard?), and an optional +`stop and warn if cost exceeds "$N"` — an OpEx ceiling beside the thrash guard's CapEx ceiling. + +## Wave 4 — Niche / standards + +### Story 9 — `sandbox:` — run isolation +*"Where code runs and what it cannot reach"*: no-network / egress allowlist / CPU-mem-time caps, +declared as config rather than prose (today `examples/forge-sandbox.loop` encodes this only inside +`goal:` text). **Gate:** human approves the plan (security-sensitive). + +### Story 10 — `knowledge:` + `examples:` context +Complete context engineering's six parts. `examples:` = reference patterns to imitate; +`knowledge:` = read-only reference (docs/diagrams) the agent must **not** edit (a label *and* a soft +guardrail). Distinct intents from `look at:` (understand + edit). + +### Story 11 — Parallel stages +`stages in parallel:` — fan out independent stories concurrently, barrier-join before the next +stage, **a worktree per parallel branch + a merge** to avoid edit collisions. The one story with +real concurrency cost; backs *"orchestrator = multi-agent, multi-file parallel."* **Gate:** human +approves the plan. + +### Story 12 — Standards & identity +Adopt open standards (CH7 orgs). `use tools from the ""` names MCP servers as the Tools +surface; frame the flow **handoff** as an A2A contract (`produces:` / `consumes the … from `); +add `runs as:` identity so unattended orchestrator runs have an auditable principal (the thin 5th +anatomy part, *Deployment*). + +## Cross-cutting + +### Story 13 — Reframe docs in deck vocabulary +Zero-grammar framing wins, written into `AGENTS.md` + `README.md`: *"intent is the new interface"* +as Loop's one-liner; the Conductor/Orchestrator and editor/terminal/background maps +(`/loopflow` · `loop-run run` · `schedule:`+PR); the five-knobs ↔ harness table (*Agent = Model + +Harness*); static vs dynamic context and the CapEx/OpEx framing. Print the flow of each new example +`.loop`. **Gate:** a human reviews before stopping. + +--- + +## Sequencing & dependencies + +- **Wave 1 is the spine** and strictly ordered: Story 1 (IR) → 2 (capture) → 3 (safety) → 4 (lint). +- Story 4's lint severity depends on Story 5 (`rigor:`); ship the lint as warn-only, then wire + severity once `rigor:` lands. +- Waves 2–4 are largely independent of each other and can be reordered by appetite. Story 11 + (parallel) is the only one with real concurrency/merge risk — do it deliberately, last. + +## Verification (end to end) + +- **Per story:** `npm test --workspace packages/` (the stage `done when`). Add a + focused `test/*.test.js` with each story before making it pass — dogfooding the loop. +- **Round-trip:** new syntax through `loop-run parse ` → loop-spec JSON; keep + `spec/loop-spec.schema.json` and `types.ts` in sync. +- **Behavioral proof for evals:** a loop that games a test green by deleting the failing assertion + must **fail** its trajectory eval (`the bar: didn't weaken a test to go green`) while a clean path + passes. +- **Flow rendering:** `loop-run show` renders the new constructs; print the flow for every new + example. +- **Full sweep:** `npm test --workspaces` stays green throughout. + +## Out of scope (for now) +Sub-agent spawning *within* a cycle (pipeline-level parallel covers the orchestrator need); a +typed/structured handoff beyond the A2A naming; a full cost-routing optimizer beyond `route:`. From 89f07d83912416d19d2018892da9401fac5e398a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:05:00 +0000 Subject: [PATCH 02/14] Config defaults: config-tier `each cycle:` + project `loop.config` file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make the language easier to use by removing repeated config. Two tiers, both following the existing lowest-wins cascade (git/models): 1. Config-tier `each cycle:` — a file-level default cycle that every loop inherits; a per-loop `each cycle:` still overrides it. Resolved at parse time and threaded through pipelines/stages. 2. `loop.config` (or `.looprc`) — a per-project defaults file in the same config-tier syntax (each cycle / models / git). The runner walks up from the .loop file and folds it in as the lowest tier, so a file's own config and per-loop directives override it. `parse()` gains an optional `defaultCycle` so the project default seeds the cycle cascade. Also: schema + IR (`Config.cycle`), `show` renders the file-level default, AGENTS.md documents both, and agentic-engineering.loop dogfoods the cycle hoist (13 repeated `each cycle:` lines collapsed to one config-tier line). Tests: parser 32, runtime 75, vscode 14, stdlib/viz green (129 total). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- AGENTS.md | 40 +++++++++++++++++++ agentic-engineering.loop | 17 ++------- packages/parser/src/parser.ts | 35 +++++++++++------ packages/parser/src/types.ts | 6 +++ packages/parser/test/parser.test.js | 59 +++++++++++++++++++++++++++++ packages/runtime/src/cli.ts | 53 ++++++++++++++++++++++++-- packages/runtime/src/show.ts | 1 + spec/loop-spec.schema.json | 11 +++++- 8 files changed, 193 insertions(+), 29 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ea3eb55..95ac913 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,6 +75,7 @@ plan from "" (read the plan from a file you control instead of gene use the method schedule: runner: target: (config tier) models: fast , strong model tiering: plan/reflect/also→fast, act→strong (cascades; override e.g. `act fast`, `all strong`) +each cycle: plan, then act, then observe (config tier: the default cycle for every loop in the file; a loop's own `each cycle:` overrides it) ``` ### Predicates (`done when …`) @@ -279,6 +280,45 @@ loop "add a healthcheck endpoint": after 6 tries: stop and warn "healthcheck stuck" ``` +## Config defaults & the project config file + +Anything in the **config tier** (the top of the file, before any definition) sets a default for +every loop in that file — so you write it once instead of repeating it per loop. The most common +repeater is the cycle: + +```loop +each cycle: plan, then act, then observe # the default for every loop below +models: fast haiku, strong opus + +pipeline "epic: ship it": + stage "story: build": + goal: it builds + done when "pnpm build" passes # no `each cycle:` — inherits the default + stage "story: verify": + goal: tests pass + each cycle: act, then observe # overrides just this stage + done when "pnpm test" passes +``` + +### `loop.config` — defaults for the whole repo + +To avoid repeating config across *files*, drop a **`loop.config`** (or `.looprc`) at your project +root. It is written in the same config-tier syntax — `each cycle:`, `models:`, a `git:` block — and +the runner reads it before every run, walking up from the `.loop` file to find it. It is the +**lowest** tier of the cascade, so a file's own config (and a per-loop directive) overrides it. + +```loop +# loop.config — applies to every .loop in the repo +each cycle: plan, then act, then observe +models: fast haiku, strong opus +git: + work on a branch + commit when the goal is met +``` + +**Cascade (lowest wins):** `loop.config` → a file's config tier → a per-loop directive. +The same rule already governs `git:` and `models:`. + ## Show the flow — every time it changes Whenever you create or edit a `.loop`, print its flow so the user sees the shape. diff --git a/agentic-engineering.loop b/agentic-engineering.loop index b5a09da..7907080 100644 --- a/agentic-engineering.loop +++ b/agentic-engineering.loop @@ -5,6 +5,10 @@ models: fast haiku, strong opus +# Config-tier default cycle — every stage below inherits it; a stage may still +# declare its own `each cycle:` to override. +each cycle: plan, then act, then observe + git: work on a branch commit each story @@ -18,7 +22,6 @@ pipeline "epic: agentic engineering for Loop": look at: spec/loop-spec.schema.json, packages/parser/src/types.ts, packages/parser/src/parser.ts, and the last failure a human approves the plan first done when "npm test --workspace packages/parser" passes - each cycle: plan, then act, then observe when it fails: reflect on whether the schema and types and parser stayed in sync, then plan again after 6 tries: stop and warn "IR change is rippling — needs a human" @@ -26,7 +29,6 @@ pipeline "epic: agentic engineering for Loop": goal: the runtime retains a per-cycle trajectory and hands trajectory evals the cycle log instead of only the output summary look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, packages/runtime/src/types.ts, and the last failure done when "npm test --workspace packages/runtime" passes - each cycle: plan, then act, then observe when it fails: reflect on what the verifier actually receives, then plan again after 6 tries: stop and warn "trajectory capture stuck" @@ -34,7 +36,6 @@ pipeline "epic: agentic engineering for Loop": goal: a blocking trajectory eval requires `the bar:`, borderline verdicts escalate to a human, eval-flaps feed the thrash guard, and reflect sees the trajectory look at: packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure done when "npm test --workspace packages/runtime" passes - each cycle: plan, then act, then observe when it fails: reflect on whether the stochastic judge can still self-thrash, then plan again after 6 tries: stop and warn "eval gate still flaky" @@ -42,7 +43,6 @@ pipeline "epic: agentic engineering for Loop": goal: a code loop with a test but no eval (or vice-versa) warns — without both, it is vibe coding look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure done when "npm test --workspace packages/parser" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 5 tries: stop and warn "lint stuck" @@ -53,7 +53,6 @@ pipeline "epic: agentic engineering for Loop": look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/stdlib, and the last failure a human approves the plan first done when "npm test --workspace packages/parser" passes - each cycle: plan, then act, then observe when it fails: reflect on the lowest-wins cascade, then plan again after 6 tries: stop and warn "rigor cascade stuck" @@ -61,7 +60,6 @@ pipeline "epic: agentic engineering for Loop": goal: mode (conductor or orchestrator) names the in-session vs async postures and warns on the incoherent vibe-plus-orchestrator combo look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure done when "npm test --workspace packages/parser" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 5 tries: stop and warn "mode stuck" @@ -71,7 +69,6 @@ pipeline "epic: agentic engineering for Loop": goal: a hooks block runs deterministic checks at named points (before each cycle, after act, on commit, on push, on stop) and a failing hook blocks look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/git.ts, and the last failure done when "npm test --workspaces" passes - each cycle: plan, then act, then observe when it fails: reflect on the hooks-vs-also-vs-doneWhen carve, then plan again after 6 tries: stop and warn "hooks stuck" @@ -79,7 +76,6 @@ pipeline "epic: agentic engineering for Loop": goal: a per-cycle trace plus token-and-cost metering, a stop-time OpEx report, and an optional cost ceiling that stops and warns look at: packages/runtime/src/engine.ts, packages/runtime/src/summary.ts, packages/runtime/src/show.ts, and the last failure done when "npm test --workspace packages/runtime" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 6 tries: stop and warn "observe stuck" @@ -90,7 +86,6 @@ pipeline "epic: agentic engineering for Loop": look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, examples/forge-sandbox.loop, and the last failure a human approves the plan first done when "npm test --workspaces" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 6 tries: stop and warn "sandbox stuck" @@ -98,7 +93,6 @@ pipeline "epic: agentic engineering for Loop": goal: examples (imitate) and knowledge (read, never edit) complete the six context-engineering parts look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure done when "npm test --workspace packages/parser" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 5 tries: stop and warn "context keywords stuck" @@ -107,7 +101,6 @@ pipeline "epic: agentic engineering for Loop": look at: packages/runtime/src/engine.ts, packages/runtime/src/git.ts, packages/parser/src/parser.ts, and the last failure a human approves the plan first done when "npm test --workspace packages/runtime" passes - each cycle: plan, then act, then observe when it fails: reflect on edit collisions and the merge, then plan again after 6 tries: stop and warn "parallel stages — concurrency risk; needs a human" @@ -115,7 +108,6 @@ pipeline "epic: agentic engineering for Loop": goal: use-tools-from names MCP servers, the flow handoff is framed as an A2A contract, and runs-as gives background runs an auditable identity look at: packages/parser/src/parser.ts, packages/runtime/src/engine.ts, packages/runtime/src/runners/claudeCode.ts, and the last failure done when "npm test --workspaces" passes - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 5 tries: stop and warn "standards stuck" @@ -126,6 +118,5 @@ pipeline "epic: agentic engineering for Loop": look at: AGENTS.md, README.md, docs/MANUAL.md, and the last failure a human reviews before stopping done when a human confirms "the docs read in the deck's vocabulary and the new constructs are covered" - each cycle: plan, then act, then observe when it fails: reflect, then plan again after 4 tries: stop and warn "docs need a human pass" diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index a18810d..d15129e 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -281,7 +281,7 @@ interface LoopBodyResult { gate: { message: string } | null; } -function interpretLoopBody(name: string | null, body: Line[]): LoopBodyResult { +function interpretLoopBody(name: string | null, body: Line[], defaultCycle?: CycleStep[]): LoopBodyResult { const loop: Loop = { kind: "loop", name, goal: "", cycle: [] }; let gate: { message: string } | null = null; let sawGoal = false; @@ -376,7 +376,8 @@ function interpretLoopBody(name: string | null, body: Line[]): LoopBodyResult { } if (!sawGoal) throw new ParseError(`loop "${name ?? "(anonymous)"}" is missing a goal`, body[0]?.lineNo ?? 0); - if (!sawCycle) loop.cycle = ["plan", "act", "observe"]; // sensible default + // Cascade (lowest wins): per-loop `each cycle:` > config-tier default > built-in plan/act/observe. + if (!sawCycle) loop.cycle = defaultCycle?.length ? [...defaultCycle] : ["plan", "act", "observe"]; return { loop, gate }; } @@ -413,7 +414,7 @@ function mergePolicy(loop: Loop, line: string) { // ---------- pipeline / stage ---------- -function parseStage(lines: Line[], start: number): { stage: Stage; next: number } { +function parseStage(lines: Line[], start: number, defaultCycle?: CycleStep[]): { stage: Stage; next: number } { const header = lines[start]; const m = header.text.match(/^stage\s+(.+?):\s*$/i); if (!m) throw new ParseError(`expected "stage :"`, header.lineNo); @@ -422,11 +423,11 @@ function parseStage(lines: Line[], start: number): { stage: Stage; next: number const name = quoted(raw) ?? raw; const { body, next } = childrenOf(lines, start + 1, header.indent); if (body.length === 0) throw new ParseError(`stage "${name}" has no body`, header.lineNo); - const { loop, gate } = interpretLoopBody(null, body); + const { loop, gate } = interpretLoopBody(null, body, defaultCycle); return { stage: { name, gate, loop }, next }; } -function parsePipeline(lines: Line[], start: number): { pipeline: Pipeline; next: number } { +function parsePipeline(lines: Line[], start: number, defaultCycle?: CycleStep[]): { pipeline: Pipeline; next: number } { const header = lines[start]; const name = quoted(header.text) ?? header.text.replace(/^pipeline\s+/i, "").replace(/:$/, "").trim(); const { body, next } = childrenOf(lines, start + 1, header.indent); @@ -437,7 +438,7 @@ function parsePipeline(lines: Line[], start: number): { pipeline: Pipeline; next if (!/^stage\b/i.test(body[i].text)) { throw new ParseError(`expected a "stage" inside pipeline "${name}"`, body[i].lineNo); } - const { stage, next: sn } = parseStage(body, i); + const { stage, next: sn } = parseStage(body, i, defaultCycle); stages.push(stage); i = sn; } @@ -445,11 +446,11 @@ function parsePipeline(lines: Line[], start: number): { pipeline: Pipeline; next return { pipeline: { kind: "pipeline", name, stages }, next }; } -function parseLoopDef(lines: Line[], start: number): { loop: Loop; next: number } { +function parseLoopDef(lines: Line[], start: number, defaultCycle?: CycleStep[]): { loop: Loop; next: number } { const header = lines[start]; const name = quoted(header.text); const { body, next } = childrenOf(lines, start + 1, header.indent); - const { loop } = interpretLoopBody(name, body); + const { loop } = interpretLoopBody(name, body, defaultCycle); return { loop, next }; } @@ -549,7 +550,14 @@ function parseConfigLine(config: Config, ln: Line): boolean { // ---------- entry ---------- -export function parse(src: string): LoopFile { +/** + * Parse a `.loop` source into a LoopFile. + * + * `opts.defaultCycle` seeds the cycle cascade from outside the file — e.g. a + * project-level `loop.config`. Precedence (lowest wins): a per-loop `each cycle:` + * > the file's config-tier `each cycle:` > `opts.defaultCycle` > built-in plan/act/observe. + */ +export function parse(src: string, opts?: { defaultCycle?: CycleStep[] }): LoopFile { const lines = tokenizeLines(src); const config: Config = {}; const definitions: Definition[] = []; @@ -557,15 +565,16 @@ export function parse(src: string): LoopFile { while (i < lines.length) { const ln = lines[i]; + let m: RegExpMatchArray | null; if (ln.indent !== 0) { throw new ParseError(`unexpected indentation at top level: "${ln.text}"`, ln.lineNo); } if (/^loop\b/i.test(ln.text)) { - const { loop, next } = parseLoopDef(lines, i); + const { loop, next } = parseLoopDef(lines, i, config.cycle ?? opts?.defaultCycle); definitions.push(loop); i = next; } else if (/^pipeline\b/i.test(ln.text)) { - const { pipeline, next } = parsePipeline(lines, i); + const { pipeline, next } = parsePipeline(lines, i, config.cycle ?? opts?.defaultCycle); definitions.push(pipeline); i = next; } else if (/^flow\b/i.test(ln.text)) { @@ -579,6 +588,10 @@ export function parse(src: string): LoopFile { } else if (/^models:\s*.+$/i.test(ln.text)) { config.models = parseModelsLine(ln.text.replace(/^models:\s*/i, ""), ln.lineNo); i++; + } else if ((m = ln.text.match(/^each cycle:\s*(.+)$/i))) { + // Config-tier default cycle — applies to every loop without its own `each cycle:`. + config.cycle = parseCycle(m[1], ln.lineNo); + i++; } else if (parseConfigLine(config, ln)) { i++; } else { diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index eb3e522..5e3d73c 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -37,6 +37,12 @@ export interface Config { notify?: string; git?: GitPolicy; models?: ModelPolicy; + /** + * File-level default cycle (`each cycle: …` at the config tier). Applies to every + * loop that doesn't declare its own `each cycle:`. A per-loop `each cycle:` overrides + * it; with neither, the built-in default is plan → act → observe. Lowest wins, like git. + */ + cycle?: CycleStep[]; } export interface OverrideEntry { diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index 18869d9..8cbcfea 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -217,3 +217,62 @@ test("memory: 'keep a memory in' is an accepted alias", () => { const loop = parse('loop "x":\n goal: g\n keep a memory in "notes.md"').definitions[0]; assert.deepEqual(loop.memory, { file: "notes.md" }); }); + +test("config-tier 'each cycle:' sets the default for loops without their own", () => { + const file = parse( + 'each cycle: act, then observe\n\nloop "x":\n goal: g\n done when "t" passes' + ); + assert.deepEqual(file.config.cycle, ["act", "observe"]); + // the loop inherits the config-tier default + assert.deepEqual(file.definitions[0].cycle, ["act", "observe"]); +}); + +test("per-loop 'each cycle:' overrides the config-tier default", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'loop "inherits":\n goal: g\n done when "t" passes\n\n' + + 'loop "overrides":\n goal: g\n each cycle: plan, then act, then observe\n done when "t" passes' + ); + assert.deepEqual(file.definitions[0].cycle, ["act", "observe"]); // inherits + assert.deepEqual(file.definitions[1].cycle, ["plan", "act", "observe"]); // overrides +}); + +test("config-tier default cycle applies to pipeline stages too", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'pipeline "p":\n stage "a":\n goal: g\n done when "t" passes\n' + + ' stage "b":\n goal: g\n each cycle: plan, then observe\n done when "t" passes' + ); + const [a, b] = file.definitions[0].stages; + assert.deepEqual(a.loop.cycle, ["act", "observe"]); // inherits config default + assert.deepEqual(b.loop.cycle, ["plan", "observe"]); // stage override wins +}); + +test("no config default → built-in plan/act/observe is preserved", () => { + const loop = parse('loop "x":\n goal: g\n done when "t" passes').definitions[0]; + assert.deepEqual(loop.cycle, ["plan", "act", "observe"]); +}); + +test("config-tier default cycle is independent per loop (no shared reference)", () => { + const file = parse( + 'each cycle: act, then observe\n\n' + + 'loop "a":\n goal: g\n done when "t" passes\n\n' + + 'loop "b":\n goal: g\n done when "t" passes' + ); + assert.notEqual(file.definitions[0].cycle, file.definitions[1].cycle); // distinct arrays + assert.deepEqual(file.definitions[0].cycle, file.definitions[1].cycle); // same contents +}); + +test("parse(opts.defaultCycle): external default seeds the cascade", () => { + // project-level default, file has no config cycle, loop has no cycle → inherits external default + const inherits = parse('loop "x":\n goal: g\n done when "t" passes', { + defaultCycle: ["act", "observe"], + }).definitions[0]; + assert.deepEqual(inherits.cycle, ["act", "observe"]); + + // the file's own config-tier cycle still wins over the external default + const fileWins = parse('each cycle: plan, then observe\n\nloop "x":\n goal: g\n done when "t" passes', { + defaultCycle: ["act", "observe"], + }).definitions[0]; + assert.deepEqual(fileWins.cycle, ["plan", "observe"]); +}); diff --git a/packages/runtime/src/cli.ts b/packages/runtime/src/cli.ts index aa97c67..c543b30 100644 --- a/packages/runtime/src/cli.ts +++ b/packages/runtime/src/cli.ts @@ -1,8 +1,9 @@ #!/usr/bin/env node -import { appendFileSync, readFileSync, writeFileSync, readdirSync } from "node:fs"; +import { appendFileSync, readFileSync, writeFileSync, readdirSync, existsSync } from "node:fs"; import { dirname, resolve, relative } from "node:path"; import { createInterface } from "node:readline"; import { parse } from "@loop-lang/parser"; +import type { Config, LoopFile } from "@loop-lang/parser"; import { resolvePreset } from "@loop-lang/stdlib"; import { run } from "./engine.js"; import { ShellVerifier } from "./verify.js"; @@ -85,6 +86,51 @@ function render(e: LoopEvent): string { } } +/** Project-level config filenames, in priority order. Written in config-tier `.loop` syntax. */ +const PROJECT_CONFIG_NAMES = ["loop.config", ".looprc"]; + +/** + * Walk up from `startDir` looking for a project config file (`loop.config` / `.looprc`). + * It is parsed with the same grammar as a `.loop` file but holds only a config tier + * (e.g. `each cycle:`, `models:`, a `git:` block). Returns its config, or null. + */ +function findProjectConfig(startDir: string): { path: string; config: Config } | null { + let dir = startDir; + while (true) { + for (const name of PROJECT_CONFIG_NAMES) { + const p = resolve(dir, name); + if (existsSync(p)) { + try { + const cfg = parse(readFileSync(p, "utf8")).config; + if (cfg) return { path: p, config: cfg }; + } catch (err) { + console.error(`⚠ ignoring ${p}: ${String((err as Error)?.message ?? err)}`); + } + return null; // a config file exists here but is empty/invalid — stop the search + } + } + const parent = dirname(dir); + if (parent === dir) return null; // reached the filesystem root + dir = parent; + } +} + +/** File config wins over project defaults (lowest tier). Shallow per key — a `git:`/`models:` block in the file replaces the project's. */ +function withProjectDefaults(fileConfig: Config | null, project: Config): Config { + return { ...project, ...(fileConfig ?? {}) }; +} + +/** + * Read and parse a `.loop` file, folding in any project-level `loop.config` as the lowest + * cascade tier. `cycle` resolves at parse time (so it is passed in as `defaultCycle`); the + * remaining config keys resolve at runtime and are merged onto the result. + */ +function loadLoopFile(absPath: string): LoopFile { + const project = findProjectConfig(dirname(absPath)); + const file = parse(readFileSync(absPath, "utf8"), { defaultCycle: project?.config.cycle }); + return project ? { ...file, config: withProjectDefaults(file.config, project.config) } : file; +} + /** Walk a dir for *.loop files, skipping noise. */ function findLoops(dir: string, acc: string[] = [], depth = 0): string[] { if (depth > 6) return acc; @@ -106,7 +152,7 @@ async function main() { if (cmd === "show") { if (!fileArg) { console.error("usage: loop-run show "); process.exit(2); } const { renderFile } = await import("./show.js"); - console.log(renderFile(parse(readFileSync(resolve(process.cwd(), fileArg), "utf8")))); + console.log(renderFile(loadLoopFile(resolve(process.cwd(), fileArg)))); return; } // `loop ls` — list every .loop under the current dir with its one-line shape. @@ -141,8 +187,7 @@ async function main() { Promise.resolve(readFileSync(resolve(dir, ref), "utf8")); const writeText = (ref: string, content: string, dir: string) => Promise.resolve(appendFileSync(resolve(dir, ref), content, "utf8")); - const src = readFileSync(path, "utf8"); - let file = parse(src); + let file = loadLoopFile(path); // If the file is a config that selects a preset, resolve and run the preset. if (file.definitions.length === 0 && file.config?.use) { diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 2dd37b0..68d7ce4 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -74,6 +74,7 @@ export function renderFile(file: LoopFile): string { const m = file.config?.models; if (m) parts.push(`models: fast=${m.tiers?.fast ?? "default"} · strong=${m.tiers?.strong ?? "default"}`); if (file.config?.git) parts.push(`git: ${file.config.git.isolation ?? "branch"}${file.config.git.openPr ? " + PR" : ""}${file.config.git.push ? "" : " · no push to main"}`); + if (file.config?.cycle?.length) parts.push(`each cycle: ${file.config.cycle.join(" → ")} (default)`); for (const d of file.definitions) parts.push(renderDef(d)); return parts.join("\n\n") || "(empty .loop)"; } diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index c0d9a88..99dd7c5 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -44,7 +44,16 @@ "description": "Notification destination (e.g. \"slack\"). Deferred past v1." }, "git": { "$ref": "#/$defs/gitPolicy" }, - "models": { "$ref": "#/$defs/modelPolicy" } + "models": { "$ref": "#/$defs/modelPolicy" }, + "cycle": { + "type": "array", + "description": "File-level default cycle (`each cycle: …` at the config tier). Applies to every loop without its own `each cycle:`; a per-loop cycle overrides it. Lowest wins, like git.", + "items": { + "type": "string", + "enum": ["plan", "act", "observe"] + }, + "minItems": 1 + } } }, "definitions": { From b65b63d6a9104b194c4955b88afea4990e2773f6 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:13:12 +0000 Subject: [PATCH 03/14] =?UTF-8?q?Wave=201=20Story=201:=20evals=20=E2=80=94?= =?UTF-8?q?=20tests=20vs=20evals,=20output=20vs=20trajectory,=20the=20bar?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make verification a conjunction and evals first-class, using the deck's `done when … on the output / on the trajectory` syntax. - IR + schema: `Loop.doneWhen` is now a `Predicate[]` (all must pass). The skill predicate gains `subject` ("output" default | "trajectory") and `bar` (an inline `the bar:` rubric). - Parser: collects multiple `done when` lines; parses the `on the output`/`on the trajectory` qualifier; attaches an indented `the bar:` line to a skill eval (error if applied to a non-skill). - Engine: observe evaluates the whole conjunction, short-circuiting on the first miss; routes skill predicates (evals) to runSkill, the rest to the shell verifier. - Renderers: show + viz render multiple predicates and label evals/ trajectory; show now renders skill evals (previously dropped). - vscode linter, schema, and AGENTS.md updated; tests added. Trajectory predicates currently receive the act summary as context; Story 2 wires the real captured trajectory. Tests: parser 36, runtime 75, vscode 14, viz 5, stdlib 3. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- AGENTS.md | 23 +++++++++- packages/parser/src/parser.ts | 31 ++++++++++--- packages/parser/src/types.ts | 15 +++++- packages/parser/test/parser.test.js | 47 +++++++++++++++++-- packages/runtime/src/engine.ts | 63 ++++++++++++++------------ packages/runtime/src/show.ts | 19 ++++++-- packages/viz/src/render.ts | 5 +- packages/vscode/src/language.ts | 4 +- packages/vscode/test/language.test.mjs | 4 +- spec/loop-spec.schema.json | 17 +++++-- 10 files changed, 168 insertions(+), 60 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 95ac913..940c158 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,13 +85,32 @@ done when the test "billing.spec.ts::apostrophe" passes # a named test done when "pnpm test" passes # a shell command, exit 0 done when "semgrep --severity=high" finds nothing # a shell command, empty output done when a human confirms "looks right at 375px" # a human check -done when the skill "email-review" approves # a review skill: approved / not -done when the skill "email-review" scores 8 or more # a review skill: numeric threshold +done when the skill "email-review" approves # an eval: approved / not +done when the skill "email-review" scores 8 or more # an eval: numeric threshold +done when the skill "api-review" scores 8 or more on the output # an eval of WHAT was produced +done when the skill "path-review" approves on the trajectory # an eval of HOW the agent got there + the bar: didn't weaken a test to go green; no writes outside api/ # the rubric the judge scores against ``` The command in a predicate runs in the user's shell with their privileges (like an npm script). It IS meant to be a real command. Prefer a fast, deterministic check. +### Tests vs evals — list as many `done when` as you need + +A loop may have **multiple `done when` lines, and ALL must pass** (a conjunction). Use this to +combine the two kinds of verification: + +- **TESTS** — a `test` / command predicate. Deterministic, checked by code (`"pnpm test" passes`, + `"semgrep …" finds nothing`). +- **EVALS** — a `skill` predicate. A rubric / LM judge for the non-deterministic parts. An eval + names its **subject**: `on the output` (the default — judges *what* was produced) or + `on the trajectory` (judges *how* the agent got there — the path and tool calls it took). An + indented `the bar:` line states the conditions the judge scores against. + +A trajectory eval is what catches the failures a green test can't — e.g. an agent that made a test +pass by weakening it. Pair a test with an eval when "done" means both "it works" and "it was built +the right way." + The **skill** predicate bridges an abstract goal to a verifiable one: when "done" isn't a test or a command (a good email, a sound design), have a review skill return an approved/rejected verdict or a numeric score. Build that review skill manually first and diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index d15129e..f1c111c 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -115,13 +115,17 @@ function parsePredicate(s: string, lineNo: number): Predicate { m = text.match(/^a human confirms\s+"([^"]+)"$/i); if (m) return { type: "human", description: m[1] }; - // the skill "X" scores N or more -> skill verdict with a numeric threshold - m = text.match(/^the skill\s+"([^"]+)"\s+scores\s+(\d+)(?:\s+or more)?$/i); - if (m) return { type: "skill", skill: m[1], expect: "approve", minScore: parseInt(m[2], 10) }; + // An eval predicate may name its subject: `on the output` (default) or `on the trajectory`. + const subj = (s: string | undefined): { subject?: "output" | "trajectory" } => + s ? { subject: s.toLowerCase() as "output" | "trajectory" } : {}; - // the skill "X" approves -> skill verdict (approved / not) - m = text.match(/^the skill\s+"([^"]+)"\s+approves$/i); - if (m) return { type: "skill", skill: m[1], expect: "approve" }; + // the skill "X" scores N or more [on the output|trajectory] -> eval with a numeric threshold + m = text.match(/^the skill\s+"([^"]+)"\s+scores\s+(\d+)(?:\s+or more)?(?:\s+on the (output|trajectory))?$/i); + if (m) return { type: "skill", skill: m[1], expect: "approve", minScore: parseInt(m[2], 10), ...subj(m[3]) }; + + // the skill "X" approves [on the output|trajectory] -> eval (approved / not) + m = text.match(/^the skill\s+"([^"]+)"\s+approves(?:\s+on the (output|trajectory))?$/i); + if (m) return { type: "skill", skill: m[1], expect: "approve", ...subj(m[2]) }; throw new ParseError(`could not understand "done when ${text}"`, lineNo); } @@ -310,7 +314,20 @@ function interpretLoopBody(name: string | null, body: Line[], defaultCycle?: Cyc i++; continue; } if ((m = t.match(/^done when\s+(.+)$/i))) { - loop.doneWhen = parsePredicate(m[1], ln.lineNo); + const pred = parsePredicate(m[1], ln.lineNo); + // An optional indented `the bar:` rubric line attaches to a skill eval. + const childBar = body[i + 1]; + if (childBar && childBar.indent > ln.indent) { + const bm = childBar.text.match(/^the bar:\s*(.+)$/i); + if (bm) { + if (pred.type !== "skill") { + throw new ParseError(`'the bar:' applies to a skill eval, not "${m[1].trim()}"`, childBar.lineNo); + } + pred.bar = bm[1].trim(); + i++; // consume the bar line + } + } + (loop.doneWhen ??= []).push(pred); i++; continue; } if ((m = t.match(/^look at:\s*(.+)$/i))) { diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index 5e3d73c..0252d16 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -67,7 +67,12 @@ export interface Loop { kind: "loop"; name: string | null; goal: string; - doneWhen?: Predicate | null; + /** + * Verification conditions — ALL must pass (a conjunction). A test/command predicate is a + * deterministic check (the deck's TESTS); a skill predicate is an eval (the deck's EVALS). + * Each `done when …` line appends one. Empty/absent = no machine check (stop via a human path). + */ + doneWhen?: Predicate[]; context?: LoopContext; policy?: Policy; cycle: CycleStep[]; @@ -118,7 +123,13 @@ export type Predicate = | { type: "test"; target: string } | { type: "command"; command: string; expect?: "exit-zero" | "empty" } | { type: "human"; description: string } - | { type: "skill"; skill: string; expect: "approve"; minScore?: number }; + /** + * An eval: a review skill judges the goal (approve / score). `subject` selects what it + * inspects — the produced `output` (default) or the `trajectory` (the path and tool calls + * the agent took to get there). `bar` is an optional inline rubric (`the bar:`) naming the + * conditions the judge scores against. + */ + | { type: "skill"; skill: string; expect: "approve"; minScore?: number; subject?: "output" | "trajectory"; bar?: string }; export interface Transition { on: "pass" | "fail" | "blocked" | "attempts"; diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index 8cbcfea..52a33bb 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -17,7 +17,7 @@ test("fix_test: minimal loop", () => { const loop = file.definitions[0]; assert.equal(loop.kind, "loop"); assert.equal(loop.name, "fix test"); - assert.deepEqual(loop.doneWhen, { type: "test", target: "checkout.spec.ts::tax" }); + assert.deepEqual(loop.doneWhen, [{ type: "test", target: "checkout.spec.ts::tax" }]); assert.deepEqual(loop.cycle, ["plan", "act", "observe"]); assert.equal(loop.transitions.length, 1); assert.equal(loop.transitions[0].on, "fail"); @@ -58,11 +58,11 @@ test("ship_feature: pipeline with human nodes + gate", () => { assert.deepEqual(pipe.stages.map((s) => s.name), ["security", "build", "ui", "deploy"]); const security = pipe.stages[0]; - assert.deepEqual(security.loop.doneWhen, { type: "command", command: "semgrep --severity=high", expect: "empty" }); + assert.deepEqual(security.loop.doneWhen, [{ type: "command", command: "semgrep --severity=high", expect: "empty" }]); const build = pipe.stages[1]; assert.equal(build.loop.humanPlan, true); - assert.deepEqual(build.loop.doneWhen, { type: "command", command: "pnpm test", expect: "exit-zero" }); + assert.deepEqual(build.loop.doneWhen, [{ type: "command", command: "pnpm test", expect: "exit-zero" }]); const ui = pipe.stages[2]; assert.equal(ui.loop.humanReviewBeforeStop, true); @@ -197,7 +197,7 @@ test("skills + memory: use skills, remember, and a skill predicate", () => { assert.equal(loop.name, "decide whether to cancel the morning run"); assert.deepEqual(loop.skills, ["check-weather", "analyze-workout"]); assert.deepEqual(loop.memory, { file: "morning-run.memory.md" }); - assert.deepEqual(loop.doneWhen, { type: "skill", skill: "workout-review", expect: "approve" }); + assert.deepEqual(loop.doneWhen, [{ type: "skill", skill: "workout-review", expect: "approve" }]); }); test("skill predicate: scores N or more carries a minScore", () => { @@ -205,7 +205,7 @@ test("skill predicate: scores N or more carries a minScore", () => { const loop = file.definitions[0]; assert.deepEqual(loop.skills, ["write-email"]); assert.deepEqual(loop.memory, { file: "launch-email.memory.md" }); - assert.deepEqual(loop.doneWhen, { type: "skill", skill: "email-review", expect: "approve", minScore: 8 }); + assert.deepEqual(loop.doneWhen, [{ type: "skill", skill: "email-review", expect: "approve", minScore: 8 }]); }); test("use skills: also accepts 'and' as a separator", () => { @@ -276,3 +276,40 @@ test("parse(opts.defaultCycle): external default seeds the cascade", () => { }).definitions[0]; assert.deepEqual(fileWins.cycle, ["plan", "observe"]); }); + +test("evals: multiple done-when lines form a conjunction", () => { + const loop = parse( + 'loop "x":\n goal: g\n' + + ' done when "pnpm test" passes\n' + + ' done when the skill "review" scores 8 or more on the output' + ).definitions[0]; + assert.equal(loop.doneWhen.length, 2); + assert.deepEqual(loop.doneWhen[0], { type: "command", command: "pnpm test", expect: "exit-zero" }); + assert.deepEqual(loop.doneWhen[1], { type: "skill", skill: "review", expect: "approve", minScore: 8, subject: "output" }); +}); + +test("evals: 'on the trajectory' selects the subject; default omits it", () => { + const traj = parse('loop "x":\n goal: g\n done when the skill "path" approves on the trajectory').definitions[0]; + assert.deepEqual(traj.doneWhen[0], { type: "skill", skill: "path", expect: "approve", subject: "trajectory" }); + const def = parse('loop "x":\n goal: g\n done when the skill "out" approves').definitions[0]; + assert.deepEqual(def.doneWhen[0], { type: "skill", skill: "out", expect: "approve" }); // no subject = output +}); + +test("evals: an indented 'the bar:' attaches a rubric to a skill eval", () => { + const loop = parse( + 'loop "x":\n goal: g\n' + + ' done when the skill "path" approves on the trajectory\n' + + ' the bar: did not weaken a test to go green' + ).definitions[0]; + assert.deepEqual(loop.doneWhen[0], { + type: "skill", skill: "path", expect: "approve", subject: "trajectory", + bar: "did not weaken a test to go green", + }); +}); + +test("evals: 'the bar:' on a non-skill predicate is a parse error", () => { + assert.throws( + () => parse('loop "x":\n goal: g\n done when "pnpm test" passes\n the bar: nope'), + /the bar:/i + ); +}); diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index b2e8551..7af9d87 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -195,35 +195,42 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { if (res.blocked) blocked = true; emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: !res.blocked, detail: res.summary }); } else { - // observe - let v: { passed: boolean; output: string }; - if (loop.doneWhen?.type === "skill") { - // Skill predicate: a review skill judges the goal (approved / score) — the - // "bridge the abstract to the verifiable" path, routed to the runner not the shell. - if (!opts.runner.runSkill) { - throw new Error( - `loop "${loop.name ?? ""}" verifies with the skill "${loop.doneWhen.skill}" but the runner has no runSkill` - ); + // observe — every `done when` predicate must pass (a conjunction of TESTS and EVALS). + const preds = loop.doneWhen ?? []; + let allPassed = preds.length > 0; // no predicate → not satisfiable by a machine check + const outputs: string[] = []; + for (const pred of preds) { + let r: { passed: boolean; output: string }; + if (pred.type === "skill") { + // An eval: a review skill judges the goal (approved / score) — the + // "bridge the abstract to the verifiable" path, routed to the runner not the shell. + if (!opts.runner.runSkill) { + throw new Error( + `loop "${loop.name ?? ""}" verifies with the skill "${pred.skill}" but the runner has no runSkill` + ); + } + const sr = await opts.runner.runSkill({ + skill: pred.skill, + goal: loop.goal, + context: lastActSummary, + expect: pred.expect, + minScore: pred.minScore, + baseDir: opts.baseDir, + }); + emit(opts, { type: "skill-verify", skill: pred.skill, passed: sr.passed, detail: sr.detail }); + r = { passed: sr.passed, output: sr.detail }; + } else { + r = await opts.verifier.verify(pred, opts.baseDir); } - const r = await opts.runner.runSkill({ - skill: loop.doneWhen.skill, - goal: loop.goal, - context: lastActSummary, - expect: loop.doneWhen.expect, - minScore: loop.doneWhen.minScore, - baseDir: opts.baseDir, - }); - emit(opts, { type: "skill-verify", skill: loop.doneWhen.skill, passed: r.passed, detail: r.detail }); - v = { passed: r.passed, output: r.detail }; - } else { - v = await opts.verifier.verify(loop.doneWhen, opts.baseDir); + if (r.output) outputs.push(r.output); + if (!r.passed) { allPassed = false; break; } // short-circuit: a conjunction fails on the first miss } observed = true; - passed = v.passed; - observeOutput = v.output; - lastOutput = v.output; - emit(opts, { type: "observe", passed: v.passed, output: v.output }); - emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: v.passed }); + passed = allPassed; + observeOutput = outputs.join("\n"); + lastOutput = observeOutput; + emit(opts, { type: "observe", passed: allPassed, output: observeOutput }); + emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: allPassed }); } } @@ -231,7 +238,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { await gitCommit(opts, `loop: ${loop.name ?? "cycle"} — cycle ${attempts}`); } - const goalMet = loop.doneWhen ? passed : false; + const goalMet = loop.doneWhen?.length ? passed : false; const t = pickTransition(loop.transitions, { blocked, attempts, passed: observed && passed, goalMet }); @@ -263,7 +270,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { // rejected: keep iterating continue; } - if (!loop.doneWhen && loop.humanPlan && humanPlanApproved) { + if (!loop.doneWhen?.length && loop.humanPlan && humanPlanApproved) { // plan-only human loop: one approved pass is the deliverable. return finish(true, "human-approved"); } diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 68d7ce4..167a0e4 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -11,7 +11,14 @@ function predicateStr(p?: Predicate | null): string | null { if (p.type === "test") return `test "${p.target}"`; if (p.type === "command") return p.expect === "empty" ? `"${p.command}" finds nothing` : `"${p.command}" passes`; if (p.type === "human") return `a human confirms "${p.description}"`; - return null; + // skill = an eval: name the verdict and, when not the default, the subject it judges. + const verdict = p.minScore !== undefined ? `scores ${p.minScore}+` : "approves"; + const on = p.subject && p.subject !== "output" ? ` on the ${p.subject}` : ""; + return `eval: skill "${p.skill}" ${verdict}${on}`; +} +/** Render every `done when` predicate (a conjunction) as labelled strings. */ +function predicateStrs(dw?: Predicate[] | null): string[] { + return (dw ?? []).map(predicateStr).filter((s): s is string => s != null); } const failsToReflect = (t?: Transition[]) => (t ?? []).some((x) => x.on === "fail" && (x.do ?? []).some((d) => d.action === "reflect" || d.action === "plan")); @@ -28,8 +35,9 @@ export function renderLoop(loop: Loop): string { const cyc = (loop.cycle?.length ? loop.cycle : ["plan", "act", "observe"]).join(" → "); L.push(` ↻ ${cyc} (each cycle)`); if (failsToReflect(loop.transitions)) L.push(` ↺ on fail: reflect → plan (the back-edge)`); - const pred = predicateStr(loop.doneWhen); - L.push(pred ? ` ✓ done when: ${pred}` : ` ⚠ no done-when — can only stop via a human path or the guard`); + const preds = predicateStrs(loop.doneWhen); + if (preds.length) preds.forEach((p) => L.push(` ✓ done when: ${p}`)); + else L.push(` ⚠ no done-when — can only stop via a human path or the guard`); const g = guard(loop.transitions); if (g) L.push(` ⛔ guard: after ${g.n ?? "N"} tries → stop${g.warn ? ` & warn "${g.warn}"` : ""}`); if (loop.humanPlan) L.push(` 👤 a human approves the plan first`); @@ -48,7 +56,8 @@ function gateMark(loop: Loop): string { export function renderPipeline(p: Pipeline): string { const L = [`pipeline "${p.name}" (stages in order · fail-fast)`]; p.stages.forEach((s, i) => { - const pred = predicateStr(s.loop.doneWhen); + const preds = predicateStrs(s.loop.doneWhen); + const pred = preds.length ? `${preds[0]}${preds.length > 1 ? ` (+${preds.length - 1})` : ""}` : null; const tail = [s.gate ? "👤 gate" : null, pred ? `✓ ${pred}` : null].filter(Boolean).join(" · "); L.push(` ${i + 1}. ${s.name}${gateMark(s.loop)}${tail ? ` ${tail}` : ""}`); }); @@ -87,7 +96,7 @@ export function oneLine(d: Definition): string { d.cycle?.length ? d.cycle.join("→") : "plan→act→observe", failsToReflect(d.transitions) ? "reflect" : null, guard(d.transitions) ? "guard" : null, - predicateStr(d.doneWhen) ? "done-when" : "⚠ no done-when", + d.doneWhen?.length ? "done-when" : "⚠ no done-when", ].filter(Boolean); return `loop "${d.name ?? "?"}" · ${bits.join(", ")}`; } diff --git a/packages/viz/src/render.ts b/packages/viz/src/render.ts index 4ec0fe0..ea7d752 100644 --- a/packages/viz/src/render.ts +++ b/packages/viz/src/render.ts @@ -118,7 +118,7 @@ function chip(x,y,text,color){var w=7.2*text.length+14; return [V("rect",{x:x,y:y,width:w,height:17,rx:8.5,fill:"none",stroke:color,"stroke-width":1}) +E("text",{x:x+w/2,y:y+12,"text-anchor":"middle",class:"chip",fill:color},esc(text)),w]; } -function doneLabel(dw){if(!dw)return "awaiting human";if(dw.type==="test")return "test "+dw.target;if(dw.type==="command")return (dw.expect==="empty"?"empty: ":"")+dw.command;if(dw.type==="skill")return "skill "+dw.skill+(dw.minScore!==undefined?" ≥"+dw.minScore:"");return "human: "+(dw.description||"");} +function doneLabel(dw){if(!dw)return "awaiting human";if(dw.type==="test")return "test "+dw.target;if(dw.type==="command")return (dw.expect==="empty"?"empty: ":"")+dw.command;if(dw.type==="skill")return "eval "+dw.skill+(dw.minScore!==undefined?" ≥"+dw.minScore:"")+(dw.subject&&dw.subject!=="output"?" ("+dw.subject+")":"");return "human: "+(dw.description||"");} function loopRow(loop,ox,oy){ var s="",cyc=loop.cycle&&loop.cycle.length?loop.cycle:["plan","act","observe"],xs=[],i; for(i=0;i1?" +"+(dw.length-1):""; + s+=E("text",{x:stopX+NW/2,y:oy+NH+15,"text-anchor":"middle",class:"sub"},esc(trunc("done when "+doneLabel(dw&&dw[0])+dwMore,24))); for(i=0;i=0&&loop.policy){var px=xs[ai],py=oy+NH+12,a; diff --git a/packages/vscode/src/language.ts b/packages/vscode/src/language.ts index 11e30ef..9343f41 100644 --- a/packages/vscode/src/language.ts +++ b/packages/vscode/src/language.ts @@ -201,7 +201,7 @@ export const VOCABULARY = Object.keys(HOVERS); interface LintLoop { kind: "loop"; name: string | null; - doneWhen?: unknown; + doneWhen?: unknown[]; humanReviewBeforeStop?: boolean; transitions?: Array<{ on: string; threshold?: number; do?: Array<{ action: string }> }>; } @@ -228,7 +228,7 @@ function headerLine(lines: string[], kind: "loop" | "stage", name: string | null function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { // Unverifiable: nothing decides "done" — only a thrash guard or the hard cap can stop it. - if (!loop.doneWhen && !loop.humanReviewBeforeStop) { + if (!(Array.isArray(loop.doneWhen) && loop.doneWhen.length) && !loop.humanReviewBeforeStop) { out.push({ line, message: "This loop has no way to verify it's done — add a `done when …` check or `a human reviews before stopping`." }); } // Self-correcting but unbounded: re-plans on failure with no attempt ceiling. diff --git a/packages/vscode/test/language.test.mjs b/packages/vscode/test/language.test.mjs index dedeb31..493de01 100644 --- a/packages/vscode/test/language.test.mjs +++ b/packages/vscode/test/language.test.mjs @@ -87,14 +87,14 @@ test("lint warns on an unverifiable loop (no done when, no human review)", () => test("lint warns on a self-correcting loop with no thrash guard", () => { const src = 'loop "y":\n goal: g\n done when "npm test" passes\n when it fails: reflect, then plan again'; - const file = { definitions: [{ kind: "loop", name: "y", doneWhen: { type: "command" }, transitions: [{ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }] }] }; + const file = { definitions: [{ kind: "loop", name: "y", doneWhen: [{ type: "command" }], transitions: [{ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }] }] }; const w = lint(file, lines(src)); assert.equal(w.length, 1); assert.match(w[0].message, /thrash guard/); }); test("lint stays silent on a complete loop", () => { - const file = { definitions: [{ kind: "loop", name: "z", doneWhen: { type: "command" }, transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + const file = { definitions: [{ kind: "loop", name: "z", doneWhen: [{ type: "command" }], transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; assert.deepEqual(lint(file, lines('loop "z":')), []); }); diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index 99dd7c5..7efa6bc 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -135,11 +135,9 @@ "description": "The objective in plain language." }, "doneWhen": { - "description": "The predicate the runtime checks to declare the goal met. Null means the loop relies on a human review to stop.", - "oneOf": [ - { "$ref": "#/$defs/predicate" }, - { "type": "null" } - ] + "description": "Verification conditions — ALL must pass (a conjunction of predicates). A test/command predicate is a deterministic check (TESTS); a skill predicate is an eval (EVALS). Empty/absent means the loop relies on a human review to stop.", + "type": "array", + "items": { "$ref": "#/$defs/predicate" } }, "context": { "description": "What the agent may look at before acting (the `look at:` block).", @@ -253,6 +251,15 @@ "type": "integer", "description": "For type=skill: when present (\"scores N or more\"), the verdict passes only when the skill's numeric score is at least this. Absent means a plain approve/reject." }, + "subject": { + "type": "string", + "enum": ["output", "trajectory"], + "description": "For type=skill (an eval): what the judge inspects — the produced \"output\" (default) or the \"trajectory\" (the path and tool calls the agent took). Set by `on the output` / `on the trajectory`." + }, + "bar": { + "type": "string", + "description": "For type=skill (an eval): an inline rubric (`the bar:`) naming the conditions the judge scores against." + }, "command": { "type": "string", "description": "For type=command: the shell command. Default success = exit 0." From 0c3b5398d94c2e8261f02daedf90a8f86b2ce794 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:16:33 +0000 Subject: [PATCH 04/14] Wave 1 Story 2: capture the trajectory and route it to trajectory evals The runner already parsed every tool call (with inputs) only to discard it after the live display. Retain it and feed it to trajectory evals. - ActResult gains `trajectory`; ClaudeCodeRunner.act() captures the streamed tool-call activity (now streams whenever capturing, so it works without a live `onActivity` display). - SkillVerifyInput gains `subject` + `bar`; runSkill frames the prompt by subject (judge the path vs the output) and states the rubric. - Engine keeps `lastTrajectory` from each act and hands trajectory evals the captured path (output evals still get the act summary). - Tests: a trajectory eval receives the trajectory + bar; an output eval receives the act summary. Runtime 77. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/runtime/src/engine.ts | 10 +++++- packages/runtime/src/runners/claudeCode.ts | 25 +++++++++++---- packages/runtime/src/types.ts | 8 ++++- packages/runtime/test/engine.test.js | 37 ++++++++++++++++++++++ 4 files changed, 71 insertions(+), 9 deletions(-) diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index 7af9d87..dc50ddd 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -67,6 +67,8 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { let lastPlan = ""; let lastOutput = ""; let lastActSummary = ""; + // The captured path of the most recent act — the steps/tool calls a trajectory eval judges. + let lastTrajectory = ""; let attempts = 0; // Reflections gathered this run — the last one becomes the memory "lesson". const reflections: string[] = []; @@ -192,6 +194,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { model: pick("act"), }); lastActSummary = res.summary; + lastTrajectory = res.trajectory ?? ""; if (res.blocked) blocked = true; emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: !res.blocked, detail: res.summary }); } else { @@ -209,12 +212,17 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { `loop "${loop.name ?? ""}" verifies with the skill "${pred.skill}" but the runner has no runSkill` ); } + // A trajectory eval judges HOW the agent got there (the captured path); + // an output eval judges WHAT it produced (the act summary). + const isTrajectory = pred.subject === "trajectory"; const sr = await opts.runner.runSkill({ skill: pred.skill, goal: loop.goal, - context: lastActSummary, + context: isTrajectory ? (lastTrajectory || lastActSummary) : lastActSummary, expect: pred.expect, minScore: pred.minScore, + subject: pred.subject, + bar: pred.bar, baseDir: opts.baseDir, }); emit(opts, { type: "skill-verify", skill: pred.skill, passed: sr.passed, detail: sr.detail }); diff --git a/packages/runtime/src/runners/claudeCode.ts b/packages/runtime/src/runners/claudeCode.ts index 5caffb5..b1e5a4a 100644 --- a/packages/runtime/src/runners/claudeCode.ts +++ b/packages/runtime/src/runners/claudeCode.ts @@ -144,9 +144,11 @@ function resultText(content: unknown): string { export class ClaudeCodeRunner implements Runner { constructor(private opts: ClaudeCodeRunnerOptions = {}) {} - private run(prompt: string, flags: string[], baseDir: string, node: AgentNode, model?: string): Promise { + private run(prompt: string, flags: string[], baseDir: string, node: AgentNode, model?: string, capture?: string[]): Promise { const bin = this.opts.bin ?? "claude"; - const stream = !!this.opts.onActivity; + // Stream when there's a live display OR a caller capturing the trajectory (tool calls + // are only parsed in stream mode), so trajectory evals work even without `onActivity`. + const stream = !!this.opts.onActivity || !!capture; const args = ["-p", prompt, ...claudeArgs({ stream, baseDir, model: model ?? this.opts.model, maxTurns: this.opts.maxTurns, flags })]; return new Promise((resolve, reject) => { @@ -165,7 +167,7 @@ export class ClaudeCodeRunner implements Runner { const line = buf.slice(0, nl); buf = buf.slice(nl + 1); const r = interpretStreamLine(line); - for (const a of r.activities) this.opts.onActivity!(node, a); + for (const a of r.activities) { this.opts.onActivity?.(node, a); capture?.push(a); } if (r.result !== undefined) { lastResult = r.result; streamed = r.result; @@ -206,14 +208,18 @@ export class ClaudeCodeRunner implements Runner { ...(input.skills?.length ? [`You may use these skills: ${input.skills.join(", ")}.`] : []), `Only use the capabilities you have been granted.`, ].join("\n"); + // Capture the act's trajectory (its tool calls, with inputs) so a trajectory eval can judge + // the path, not just the result. interpretStreamLine already decodes tool_use blocks. + const trajectory: string[] = []; const summary = await this.run( prompt, ["--permission-mode", mode, "--allowedTools", ...tools], input.baseDir, "act", - input.model + input.model, + trajectory ); - return { summary }; + return { summary, trajectory: trajectory.join("\n") }; } async reflect(input: ReflectInput): Promise { @@ -232,11 +238,16 @@ export class ClaudeCodeRunner implements Runner { input.minScore !== undefined ? `End with a line exactly "SCORE: N" where N is 0-10. It passes when N >= ${input.minScore}.` : `End with a line exactly "VERDICT: APPROVED" or "VERDICT: REJECTED".`; + const isTrajectory = input.subject === "trajectory"; + const whatToReview = isTrajectory + ? `The agent's TRAJECTORY — the steps and tool calls it took. Judge HOW it got there (right path, right tools; e.g. it must not have weakened a test to go green), not just the end result:` + : `What to review (the produced OUTPUT):`; const prompt = [ `Use the "${input.skill}" skill to review this against the goal.`, `Goal: ${input.goal}.`, - `What to review:`, - input.context || "(no output captured)", + ...(input.bar ? [`The bar — it passes only if all of these hold: ${input.bar}.`] : []), + whatToReview, + input.context || "(no context captured)", verdictLine, ].join("\n"); const out = await this.run( diff --git a/packages/runtime/src/types.ts b/packages/runtime/src/types.ts index 9bca66d..9c8942b 100644 --- a/packages/runtime/src/types.ts +++ b/packages/runtime/src/types.ts @@ -66,18 +66,24 @@ export interface SkillVerifyInput { /** The review skill to invoke. */ skill: string; goal: string; - /** The work to judge — typically the last act summary. */ + /** The work to judge — the act summary for an output eval, the cycle trajectory for a trajectory eval. */ context: string; /** "approve" requires an approving verdict. */ expect: "approve"; /** When set, the verdict passes only if the skill's numeric score is at least this. */ minScore?: number; + /** What the eval inspects: the produced "output" (default) or the "trajectory" (path + tool calls). */ + subject?: "output" | "trajectory"; + /** Inline rubric (`the bar:`) naming the conditions the judge must score against. */ + bar?: string; baseDir: string; } export interface ActResult { summary: string; blocked?: boolean; + /** The captured trajectory of this act: the steps and tool calls the agent took. Fed to trajectory evals. */ + trajectory?: string; } export interface ReflectInput { diff --git a/packages/runtime/test/engine.test.js b/packages/runtime/test/engine.test.js index cdced58..a1eaf0f 100644 --- a/packages/runtime/test/engine.test.js +++ b/packages/runtime/test/engine.test.js @@ -639,3 +639,40 @@ test("regression: a loop with no skills/memory behaves exactly as before", async assert.ok(!events.some((e) => e.type === "memory-read" || e.type === "memory-write")); assert.deepEqual(store.files, {}, "no memory file written"); }); + +// ---- trajectory evals (Story 2) ---- + +test("trajectory eval: the captured trajectory + the bar reach the verifier", async () => { + const def = parse( + 'loop "x":\n goal: g\n' + + ' done when the skill "path-review" approves on the trajectory\n' + + ' the bar: did not weaken a test to go green\n' + + ' after 2 tries: stop and warn "stuck"' + ).definitions[0]; + const runner = new MockRunner({ + act: (i) => ({ summary: `acted on: ${i.goal}`, trajectory: "● Edit refunds.ts\n● Bash $ pnpm test" }), + skill: () => ({ passed: true, detail: "APPROVED" }), + }); + const outcome = await runDefinition(def, { + runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); + const call = runner.skillCalls.at(-1); + assert.equal(call.subject, "trajectory"); + assert.equal(call.bar, "did not weaken a test to go green"); + assert.match(call.context, /Edit refunds\.ts/, "the eval judged the trajectory, not the act summary"); +}); + +test("output eval: receives the act summary, not the trajectory", async () => { + const def = parse( + 'loop "x":\n goal: g\n done when the skill "review" approves on the output\n after 2 tries: stop and warn "x"' + ).definitions[0]; + const runner = new MockRunner({ + act: () => ({ summary: "ACTSUMMARY", trajectory: "TRAJ-should-not-be-used" }), + skill: () => ({ passed: true, detail: "ok" }), + }); + await runDefinition(def, { runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p" }); + const call = runner.skillCalls.at(-1); + assert.equal(call.subject, "output"); + assert.equal(call.context, "ACTSUMMARY"); +}); From f4f1221ae6d2210c3af20a75749e1a2b01c610cd Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:19:03 +0000 Subject: [PATCH 05/14] Wave 1 Stories 3-4: reflect sees the trajectory; trajectory-eval bar nudge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 3 (feedback wiring): a failure's trajectory now flows into reflect, so `reflect on the path it took` can see the tool calls and fix the path (e.g. "you edited the test") rather than only the output. - ReflectInput gains `trajectory`; engine threads `lastTrajectory` through applyActions; the runner includes it in the reflect prompt. Story 4 (anti-thrash, safe part): a soft vscode lint nudges any trajectory eval that lacks `the bar:` — gating "done" on an LM judging a non-deterministic path without explicit pass conditions invites its own thrash. The full rigor-gated "without both [test and eval]" lint lands in Wave 2 Story 5, where `rigor:` supplies the severity (a blanket version would wrongly fire on every test-only loop). Tests: runtime 78, vscode 16. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/runtime/src/engine.ts | 4 +++- packages/runtime/src/runners/claudeCode.ts | 1 + packages/runtime/src/types.ts | 2 ++ packages/runtime/test/engine.test.js | 13 +++++++++++++ packages/vscode/src/language.ts | 8 ++++++++ packages/vscode/test/language.test.mjs | 16 ++++++++++++++++ 6 files changed, 43 insertions(+), 1 deletion(-) diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index dc50ddd..a429172 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -255,6 +255,7 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { const stop = await applyActions(t.do, loop, opts, { goalMet, output: observeOutput, + trajectory: lastTrajectory, setReflection: (r) => { reflection = r; reflections.push(r); @@ -294,7 +295,7 @@ async function applyActions( actions: Action[], loop: Loop, opts: RunOptions, - ctx: { goalMet: boolean; output: string; setReflection: (r: string) => void } + ctx: { goalMet: boolean; output: string; trajectory?: string; setReflection: (r: string) => void } ): Promise { for (const a of actions) { switch (a.action) { @@ -317,6 +318,7 @@ async function applyActions( goal: loop.goal, focus: a.focus, output: ctx.output, + trajectory: ctx.trajectory, baseDir: opts.baseDir, model: modelForPhase(rEff, "reflect", opts.cliModel), }); diff --git a/packages/runtime/src/runners/claudeCode.ts b/packages/runtime/src/runners/claudeCode.ts index b1e5a4a..fe01401 100644 --- a/packages/runtime/src/runners/claudeCode.ts +++ b/packages/runtime/src/runners/claudeCode.ts @@ -227,6 +227,7 @@ export class ClaudeCodeRunner implements Runner { `Goal: ${input.goal}.`, `The verification did not pass. Output was:`, input.output || "(no output)", + ...(input.trajectory ? [`The path you took this cycle (tool calls):`, input.trajectory] : []), input.focus ? `Reflect specifically on: ${input.focus}.` : `Reflect on why it failed.`, `In 2-4 sentences, explain the likely cause and what to change next. Do not edit files.`, ].join("\n"); diff --git a/packages/runtime/src/types.ts b/packages/runtime/src/types.ts index 9c8942b..d6985b8 100644 --- a/packages/runtime/src/types.ts +++ b/packages/runtime/src/types.ts @@ -90,6 +90,8 @@ export interface ReflectInput { goal: string; focus?: string; output: string; + /** The most recent act's trajectory, so a `reflect on the path it took` can see it. */ + trajectory?: string; baseDir: string; /** Model alias/id for this call; set by the engine from the model policy. */ model?: string; diff --git a/packages/runtime/test/engine.test.js b/packages/runtime/test/engine.test.js index a1eaf0f..69065ab 100644 --- a/packages/runtime/test/engine.test.js +++ b/packages/runtime/test/engine.test.js @@ -676,3 +676,16 @@ test("output eval: receives the act summary, not the trajectory", async () => { assert.equal(call.subject, "output"); assert.equal(call.context, "ACTSUMMARY"); }); + +test("reflect sees the trajectory of the failing cycle (Story 3)", async () => { + const def = parse( + 'loop "x":\n goal: g\n done when "cmd" passes\n' + + ' when it fails: reflect, then plan again\n after 3 tries: stop and warn "stuck"' + ).definitions[0]; + const runner = new MockRunner({ act: () => ({ summary: "did stuff", trajectory: "● Edit a.ts\n● Bash $ cmd" }) }); + await runDefinition(def, { + runner, verifier: new SeqVerifier([false, true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(runner.reflectCalls.length, 1, "reflected on the one failure"); + assert.match(runner.reflectCalls[0].trajectory, /Edit a\.ts/, "reflect received the cycle's trajectory"); +}); diff --git a/packages/vscode/src/language.ts b/packages/vscode/src/language.ts index 9343f41..180d7fd 100644 --- a/packages/vscode/src/language.ts +++ b/packages/vscode/src/language.ts @@ -231,6 +231,14 @@ function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { if (!(Array.isArray(loop.doneWhen) && loop.doneWhen.length) && !loop.humanReviewBeforeStop) { out.push({ line, message: "This loop has no way to verify it's done — add a `done when …` check or `a human reviews before stopping`." }); } + // A trajectory eval gates "done" on an LM judging a non-deterministic path; without an explicit + // `the bar:` it can wobble run-to-run and cause its own thrash. Nudge toward a rubric. + const barlessTrajectoryEval = (Array.isArray(loop.doneWhen) ? loop.doneWhen : []).some( + (p): boolean => !!p && typeof p === "object" && (p as any).type === "skill" && (p as any).subject === "trajectory" && !(p as any).bar + ); + if (barlessTrajectoryEval) { + out.push({ line, message: "This trajectory eval has no `the bar:` — an LM judging the path can wobble run-to-run; add `the bar: …` with explicit pass conditions." }); + } // Self-correcting but unbounded: re-plans on failure with no attempt ceiling. const reflects = (loop.transitions ?? []).some((t) => t.on === "fail" && (t.do ?? []).some((d) => d.action === "reflect" || d.action === "plan")); const guarded = (loop.transitions ?? []).some((t) => t.on === "attempts" && typeof t.threshold === "number"); diff --git a/packages/vscode/test/language.test.mjs b/packages/vscode/test/language.test.mjs index 493de01..9ec574a 100644 --- a/packages/vscode/test/language.test.mjs +++ b/packages/vscode/test/language.test.mjs @@ -111,3 +111,19 @@ test("lint produces zero warnings for a flow definition", () => { const w = lint(file, lines(src)); assert.deepEqual(w, [], "flow should produce no lint warnings"); }); + +test("lint nudges a trajectory eval that lacks `the bar:`", () => { + const file = { definitions: [{ kind: "loop", name: "t", + doneWhen: [{ type: "skill", skill: "path", subject: "trajectory" }], + transitions: [{ on: "attempts", threshold: 5, do: [{ action: "stop" }] }] }] }; + const w = lint(file, lines('loop "t":')); + assert.equal(w.length, 1); + assert.match(w[0].message, /the bar:/); +}); + +test("lint stays silent on a trajectory eval that has `the bar:`", () => { + const file = { definitions: [{ kind: "loop", name: "t", + doneWhen: [{ type: "skill", skill: "path", subject: "trajectory", bar: "no test edits" }], + transitions: [{ on: "attempts", threshold: 5, do: [{ action: "stop" }] }] }] }; + assert.deepEqual(lint(file, lines('loop "t":')), []); +}); From 14336062fd53fb21f926f78e44f583a74f56dc36 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:39:45 +0000 Subject: [PATCH 06/14] Friendly surface: plain-English sugar + `loop-run explain` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make Loop approachable for non-experts while keeping all power — these are pure sugar/views over the same IR; the full grammar is unchanged. - Aliases (parser): `check:` / `verify:` = a `done when` (a bare value is a shell command, a predicate phrase is parsed as-is); `in:` / `look in:` / `files:` / `context:` = `look at:`; `when it breaks` = `when it fails`; `when it gets stuck` = `when blocked`. - `loop-run explain ` — describes a loop/pipeline/flow in plain English ("works toward… each round it… it's done when… it gives up after N tries"), so non-experts can trust what they wrote. A 6-line friendly loop now reads and explains cleanly. Tests: parser 38, runtime 79. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/parser/src/parser.ts | 18 +++++++-- packages/parser/test/parser.test.js | 16 ++++++++ packages/runtime/src/cli.ts | 9 ++++- packages/runtime/src/show.ts | 63 +++++++++++++++++++++++++++++ packages/runtime/test/show.test.js | 13 ++++++ 5 files changed, 115 insertions(+), 4 deletions(-) diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index f1c111c..4ee3196 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -183,8 +183,8 @@ function parseWhenCondition(cond: string, lineNo: number): Pick { /the bar:/i ); }); + +test("friendly sugar: check:/verify:, in:/files:, when it breaks", () => { + const loop = parse( + 'loop "x":\n goal: g\n in: src/cart, src/tax\n check: pnpm test cart\n when it breaks: reflect, then plan again' + ).definitions[0]; + assert.deepEqual(loop.context.files, ["src/cart", "src/tax"]); + assert.deepEqual(loop.doneWhen, [{ type: "command", command: "pnpm test cart", expect: "exit-zero" }]); + assert.equal(loop.transitions[0].on, "fail"); +}); + +test("friendly sugar: check: also accepts a predicate phrase", () => { + const a = parse('loop "x":\n goal: g\n check: the skill "review" approves on the trajectory').definitions[0]; + assert.deepEqual(a.doneWhen, [{ type: "skill", skill: "review", expect: "approve", subject: "trajectory" }]); + const b = parse('loop "x":\n goal: g\n files: a.ts, b.ts').definitions[0]; + assert.deepEqual(b.context.files, ["a.ts", "b.ts"]); +}); diff --git a/packages/runtime/src/cli.ts b/packages/runtime/src/cli.ts index c543b30..a976ac8 100644 --- a/packages/runtime/src/cli.ts +++ b/packages/runtime/src/cli.ts @@ -155,6 +155,13 @@ async function main() { console.log(renderFile(loadLoopFile(resolve(process.cwd(), fileArg)))); return; } + // `loop explain ` — describe the loop in plain English (the friendly view). + if (cmd === "explain") { + if (!fileArg) { console.error("usage: loop-run explain "); process.exit(2); } + const { explainFile } = await import("./show.js"); + console.log(explainFile(loadLoopFile(resolve(process.cwd(), fileArg)))); + return; + } // `loop ls` — list every .loop under the current dir with its one-line shape. if (cmd === "ls") { const { oneLine } = await import("./show.js"); @@ -175,7 +182,7 @@ async function main() { } if (!cmd || (cmd !== "run" && cmd !== "parse" && cmd !== "viz") || !fileArg) { - console.error("usage: loop-run [--model ] [--out ]"); + console.error("usage: loop-run [--model ] [--out ]"); process.exit(2); } diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 167a0e4..bd8b49c 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -100,3 +100,66 @@ export function oneLine(d: Definition): string { ].filter(Boolean); return `loop "${d.name ?? "?"}" · ${bits.join(", ")}`; } + +// ---- explain: read a loop back in plain English (the friendly, non-expert view) ---- + +const CYCLE_PROSE: Record = { + plan: "plans the change", + act: "makes the change", + observe: "checks the result", +}; + +/** Join items into prose: "a", "a then b", "a, b, then c". */ +function joinList(items: string[], conj = "then"): string { + if (items.length <= 1) return items[0] ?? ""; + if (items.length === 2) return `${items[0]} ${conj} ${items[1]}`; + return `${items.slice(0, -1).join(", ")}, ${conj} ${items[items.length - 1]}`; +} + +function predicateProse(p: Predicate): string { + if (p.type === "test") return `the test "${p.target}" passes`; + if (p.type === "command") return p.expect === "empty" ? `running \`${p.command}\` reports nothing` : `running \`${p.command}\` succeeds`; + if (p.type === "human") return `you confirm "${p.description}"`; + // skill = an eval + const verdict = p.minScore !== undefined ? `the "${p.skill}" review scores ${p.minScore} or more` : `the "${p.skill}" review approves`; + return verdict + (p.subject === "trajectory" ? " (judging how it got there, not just the result)" : ""); +} + +/** A plain-English description of a single loop. */ +export function explainLoop(loop: Loop): string { + const L: string[] = []; + L.push(`${loop.name ? `Loop "${loop.name}"` : "This loop"} works toward: ${loop.goal || "(no goal set)"}.`); + const cyc = (loop.cycle?.length ? loop.cycle : ["plan", "act", "observe"]).map((s) => CYCLE_PROSE[s] ?? s); + L.push(`Each round it ${joinList(cyc)}.`); + const preds = loop.doneWhen ?? []; + if (preds.length) L.push(`It's done when ${joinList(preds.map(predicateProse), "and")}.`); + else L.push(`It has no automatic check, so it relies on a human to decide when to stop.`); + if (failsToReflect(loop.transitions)) L.push(`If a check fails, it reflects on why and tries again.`); + if (loop.humanPlan) L.push(`It pauses for you to approve the plan before changing anything.`); + if (loop.humanReviewBeforeStop) L.push(`It pauses for you to review the result before finishing.`); + if (asksWhenBlocked(loop.transitions)) L.push(`If it gets stuck, it stops and asks you.`); + const g = guard(loop.transitions); + if (g) L.push(`It gives up after ${g.n ?? "a few"} tries${g.warn ? ` (warning "${g.warn}")` : ""}.`); + if (loop.also?.length) L.push(`Once the goal is met it also: ${loop.also.join(", ")}.`); + return L.join(" "); +} + +export function explainDef(d: Definition): string { + if (d.kind === "loop") return explainLoop(d); + if (d.kind === "pipeline") { + const n = d.stages.length; + const L = [`Pipeline "${d.name}" runs ${n} ${n === 1 ? "story" : "stories"} in order, stopping if one fails:`]; + d.stages.forEach((s, i) => L.push(` ${i + 1}. ${s.name}${s.gate ? " (pauses for your OK first)" : ""} — ${explainLoop(s.loop)}`)); + return L.join("\n"); + } + const L = [`Flow "${d.name}" runs these loop files in sequence, handing each result to the next:`]; + d.steps.forEach((s, i) => + L.push(` ${i + 1}. ${s.forEach ? `for each item in "${s.forEach.source}", run ${s.ref}` : s.ref}${s.gate ? " (pauses for your OK first)" : ""}`) + ); + return L.join("\n"); +} + +/** Plain-English description of a whole .loop file — the `loop-run explain` view. */ +export function explainFile(file: LoopFile): string { + return (file.definitions ?? []).map(explainDef).join("\n\n") || "(empty .loop)"; +} diff --git a/packages/runtime/test/show.test.js b/packages/runtime/test/show.test.js index 96d6a31..ec1a6e5 100644 --- a/packages/runtime/test/show.test.js +++ b/packages/runtime/test/show.test.js @@ -41,3 +41,16 @@ test("oneLine gives a terse shape per definition kind", () => { const loop = parse(`loop "y":\n goal: g\n done when "t" passes\n each cycle: plan, then act, then observe\n when it fails: reflect, then plan again\n`).definitions[0]; assert.match(oneLine(loop), /loop "y" · plan→act→observe, reflect, done-when/); }); + +test("explain: plain-English description of a loop", async () => { + const { explainFile } = await import("../dist/show.js"); + const file = parse( + 'loop "fix the cart test":\n goal: the cart total is correct\n check: pnpm test cart\n' + + ' when it fails: reflect, then plan again\n after 6 tries: stop and warn "stuck"' + ); + const prose = explainFile(file); + assert.match(prose, /works toward: the cart total is correct/); + assert.match(prose, /done when running `pnpm test cart` succeeds/); + assert.match(prose, /reflects on why and tries again/); + assert.match(prose, /gives up after 6 tries/); +}); From c26d51c0daa45b16e7f8faa01e58ccf511ca02f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:41:36 +0000 Subject: [PATCH 07/14] Docs: update the tutorial for evals, config defaults, and friendly syntax Bring the manual and grammar docs in line with the new features: - MANUAL.md: CLI table gains `show` / `explain` / `ls`; a "Tests vs evals" subsection (output vs trajectory, `the bar:`); a "Friendly shorthands" subsection (`check:`/`in:`/`when it breaks`) with a 6-line example; config tier documents the `each cycle:` default and the `loop.config` project file + the lowest-wins cascade. - AGENTS.md: `loop-run explain` and the friendly shorthands. Docs-only. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- AGENTS.md | 5 +++ docs/MANUAL.md | 87 ++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 83 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 940c158..8dcbaa9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -199,6 +199,10 @@ flow "deliver": irreversible. Use `ask me before …` for action policy, `a human approves before …` for a hard stage gate. - Output only valid `.loop` syntax. Comments start with `#`. +- **Friendly shorthands** (all desugar to the lines above — use freely): `check:` / `verify:` + = `done when` (a bare value is a shell command; a predicate phrase parses as-is); `in:` / + `look in:` / `files:` / `context:` = `look at:`; `when it breaks` = `when it fails`; + `when it gets stuck` = `when blocked`. ## Example — a single loop @@ -351,6 +355,7 @@ flow, show the file chain. `loop-run ls` lists every loop in the repo. - `loop-run run file.loop` — execute it on Claude Code (plan/act/observe, reflect on failure, verify with `done when`, pause at human gates). - `loop-run show file.loop` — print the loop's flow as compact ASCII (and `loop-run ls` to list them). +- `loop-run explain file.loop` — describe the loop in plain English (a friendly check of what it will do). - `loop-run viz file.loop` — open a visual HTML schematic of the flow. ## Authoring checklist diff --git a/docs/MANUAL.md b/docs/MANUAL.md index b62373f..bc2c863 100644 --- a/docs/MANUAL.md +++ b/docs/MANUAL.md @@ -116,14 +116,17 @@ the thrash guard). ## 4. The CLI ``` -loop-run [--model ] [--out ] +loop-run [--model ] [--out ] ``` | Command | What it does | |---|---| | `loop-run run ` | Execute the flow on Claude Code (plan/act/observe, reflect, verify, human gates). | | `loop-run parse ` | Parse to the loop-spec IR and print it as JSON (use to validate syntax). | +| `loop-run show ` | Print the loop's flow as compact ASCII (the "see the shape" view). | +| `loop-run explain ` | Describe the loop in **plain English** — handy for non-experts to check what a `.loop` will actually do. | | `loop-run viz ` | Write a self-contained HTML schematic of the flow. | +| `loop-run ls` | List every `.loop` under the current directory with its one-line shape. | **Flags** @@ -211,7 +214,7 @@ BMAD flow (discover → design → for each story) as one example method. | Line | Meaning | |---|---| | `goal: ` | The objective, in plain language. **Required.** | -| `done when ` | How the loop verifies itself (see Predicates). Omit only if a human gate decides completion. | +| `done when ` | How the loop verifies itself (see Predicates). **You can list several — all must pass** (a conjunction of tests and evals). Omit only if a human gate decides completion. | | `look at: , , and the last failure` | Context the agent reads before acting. Items are file paths or plain-language descriptions (e.g. `the billing form`) — the agent locates the actual files from descriptions before planning. `and the last failure` feeds the previous failure forward. | | `allow edits automatically, but ask me before ` | Action policy. Auto classes run unattended; confirm classes pause for you. Classes: `edit`, `migrate`, `push`, `deploy`, `delete`. | | `each cycle: plan, then act, then observe` | The repeated steps — any subset of `plan` / `act` / `observe`, in order. | @@ -226,6 +229,34 @@ BMAD flow (discover → design → for each story) as one example method. | `a human reviews before stopping` | A person judges the result before the loop may stop. | | `plan from ""` | Read the loop's plan from a file you control (e.g. a hand-written `.md`) instead of having the agent generate it. Omit it and the agent writes the plan (the default). | +### Friendly shorthands + +Plain-English synonyms for the lines above — write whichever reads naturally to you. They +desugar to the exact same loop-spec, so you never lose any power: + +| Shorthand | Same as | +|---|---| +| `check: pnpm test` | `done when "pnpm test" passes` (a bare value is a shell command) | +| `check: the skill "review" approves` | `done when the skill "review" approves` (a predicate phrase is parsed as-is) | +| `verify: …` | same as `check:` | +| `in: src/cart, src/tax` · `files: …` · `look in: …` · `context: …` | `look at: …` | +| `when it breaks: …` | `when it fails: …` | +| `when it gets stuck: …` | `when blocked: …` | + +A complete, friendly loop can be as short as: + +```loop +loop "fix the cart total": + goal: the cart total is correct with tax + in: src/cart, src/tax + check: pnpm test cart + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" +``` + +Run `loop-run explain ` to read any loop back in plain English and confirm it does +what you intend. + ### In a stage only | Line | Meaning | @@ -239,18 +270,31 @@ done when the test "billing.spec.ts::apostrophe" passes # a named test done when "pnpm test" passes # shell command, exit 0 (`succeeds` also works) done when "semgrep --severity=high" finds nothing # shell command, empty stdout done when a human confirms "looks right at 375px" # a human check -done when the skill "email-review" approves # a review skill: approved / not -done when the skill "email-review" scores 8 or more # a review skill: numeric threshold +done when the skill "email-review" approves # an eval: approved / not +done when the skill "email-review" scores 8 or more # an eval: numeric threshold +done when the skill "api-review" scores 8 or more on the output # an eval of WHAT was produced +done when the skill "path-review" approves on the trajectory # an eval of HOW it got there + the bar: didn't weaken a test to go green; no writes outside api/ # the rubric the judge scores against ``` The command runs in your shell with your privileges (like an npm script). Keep it fast and deterministic. -The **skill** predicate bridges an abstract goal to a verifiable one. When "done" isn't a -test or a command — a good email, a sound design, a sensible go/no-go call — a review skill -returns an approved/rejected verdict or a 0–10 score, and the loop treats that as its check. -Build the review skill manually first and confirm it judges well, then wire it in. See -`examples/skills_memory.loop` and `examples/email_review.loop`. +#### Tests vs evals + +A loop can list **several `done when` lines, and all must pass.** Use this to combine the +two kinds of verification (the distinction is from *The New SDLC with Vibe Coding*): + +- **Tests** — `test` / command predicates. Deterministic, checked by code. +- **Evals** — `skill` predicates. A rubric / LM judge for the non-deterministic parts. An + eval names its **subject**: `on the output` (the default — judges *what* was produced) or + `on the trajectory` (judges *how* the agent got there — its path and tool calls). An + indented `the bar:` line states the conditions the judge must score against. + +A trajectory eval catches what a green test can't — e.g. an agent that made a test pass by +weakening it. Pair a test with an eval when "done" means both *it works* and *it was built +the right way*. Build the review skill manually first and confirm it judges well, then wire +it in. See `examples/skills_memory.loop` and `examples/email_review.loop`. ### Config tier (top of file) @@ -260,8 +304,33 @@ run with claude code # runner / provider (or: `runner claude code`) schedule: nightly # manual · nightly · on push · cron (parsed; run is manual via the CLI) target: ./src # working directory the loop operates in notify: slack # notification destination (reserved) +each cycle: plan, then act, then observe # the DEFAULT cycle for every loop in the file ``` +Anything in the config tier is a **default for every loop in the file**, so you write it +once instead of repeating it per loop. A `each cycle:` here is inherited by every loop; a +loop with its own `each cycle:` overrides it. The same lowest-wins cascade governs `git:` +and `models:`. + +#### Project defaults — `loop.config` + +To avoid repeating config across *files*, drop a **`loop.config`** (or `.looprc`) at your +project root. It uses the same config-tier syntax (`each cycle:`, `models:`, a `git:` +block) and the runner reads it before every run, walking up from the `.loop` file to find +it. It is the **lowest** tier of the cascade, so a file's own config and a per-loop +directive both override it. + +```loop +# loop.config — applies to every .loop in the repo +each cycle: plan, then act, then observe +models: fast haiku, strong opus +git: + work on a branch + commit when the goal is met +``` + +**Cascade (lowest wins):** `loop.config` → a file's config tier → a per-loop directive. + ### Git strategy A `git:` block sets the version-control strategy for the run. It can appear at the top of From dbfbcf68f3154d2b7134b902b9d3b39e39f01df4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:47:12 +0000 Subject: [PATCH 08/14] Wave 2: rigor dial (+ sensible defaults) and conductor/orchestrator mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 5 (rigor): a config-tier `rigor: vibe coding | structured ai-assisted | agentic engineering` dial. Under structured/agentic, every loop is born with a reflect-on-fail back-edge and a thrash guard unless it sets its own — the "sensible defaults" that remove boilerplate. `vibe coding` (and no rigor) injects nothing, so existing files are unchanged. The rigor-gated "without both, it is vibe coding" lint now fires (tests without an eval, or vice-versa) only when rigor opts in. Story 6 (mode): a config-tier `mode: conductor | orchestrator` naming the in-session vs async posture, with a lint warning on the costliest `vibe coding` + `orchestrator` quadrant. IR + schema + parser (threaded ParseDefaults) + show badges + cli project defaults + lints. Tests: parser 41, vscode 19. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/parser/src/parser.ts | 57 +++++++++++++++++++++----- packages/parser/src/types.ts | 18 ++++++++ packages/parser/test/parser.test.js | 24 +++++++++++ packages/runtime/src/cli.ts | 5 ++- packages/runtime/src/show.ts | 3 ++ packages/vscode/src/language.ts | 28 ++++++++++--- packages/vscode/test/language.test.mjs | 22 ++++++++++ spec/loop-spec.schema.json | 10 +++++ 8 files changed, 150 insertions(+), 17 deletions(-) diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index 4ee3196..f6b43e8 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -11,14 +11,18 @@ import { LoopFile, LOOP_VERSION, ModelPolicy, + ParseDefaults, ParseError, Pipeline, Policy, Predicate, + Rigor, Stage, Transition, } from "./types.js"; +const RIGOR_LEVELS: Rigor[] = ["vibe coding", "structured ai-assisted", "agentic engineering"]; + interface Line { indent: number; text: string; @@ -285,7 +289,7 @@ interface LoopBodyResult { gate: { message: string } | null; } -function interpretLoopBody(name: string | null, body: Line[], defaultCycle?: CycleStep[]): LoopBodyResult { +function interpretLoopBody(name: string | null, body: Line[], defaults?: ParseDefaults): LoopBodyResult { const loop: Loop = { kind: "loop", name, goal: "", cycle: [] }; let gate: { message: string } | null = null; let sawGoal = false; @@ -406,7 +410,21 @@ function interpretLoopBody(name: string | null, body: Line[], defaultCycle?: Cyc if (!sawGoal) throw new ParseError(`loop "${name ?? "(anonymous)"}" is missing a goal`, body[0]?.lineNo ?? 0); // Cascade (lowest wins): per-loop `each cycle:` > config-tier default > built-in plan/act/observe. - if (!sawCycle) loop.cycle = defaultCycle?.length ? [...defaultCycle] : ["plan", "act", "observe"]; + if (!sawCycle) loop.cycle = defaults?.cycle?.length ? [...defaults.cycle] : ["plan", "act", "observe"]; + + // rigor: structured/agentic loops are born with a back-edge and a thrash guard unless the + // author set their own — the "sensible defaults" that keep a loop safe without boilerplate. + // `vibe coding` (and no rigor at all) injects nothing, so existing files are unchanged. + if (defaults?.rigor === "structured ai-assisted" || defaults?.rigor === "agentic engineering") { + const has = (on: Transition["on"]) => (loop.transitions ?? []).some((t) => t.on === on); + if (loop.doneWhen?.length && !has("fail")) { + (loop.transitions ??= []).push({ on: "fail", do: [{ action: "reflect" }, { action: "plan" }] }); + } + if (has("fail") && !has("attempts")) { + (loop.transitions ??= []).push({ on: "attempts", threshold: 8, do: [{ action: "stop", warn: "thrashing" }] }); + } + } + return { loop, gate }; } @@ -443,7 +461,7 @@ function mergePolicy(loop: Loop, line: string) { // ---------- pipeline / stage ---------- -function parseStage(lines: Line[], start: number, defaultCycle?: CycleStep[]): { stage: Stage; next: number } { +function parseStage(lines: Line[], start: number, defaults?: ParseDefaults): { stage: Stage; next: number } { const header = lines[start]; const m = header.text.match(/^stage\s+(.+?):\s*$/i); if (!m) throw new ParseError(`expected "stage :"`, header.lineNo); @@ -452,11 +470,11 @@ function parseStage(lines: Line[], start: number, defaultCycle?: CycleStep[]): { const name = quoted(raw) ?? raw; const { body, next } = childrenOf(lines, start + 1, header.indent); if (body.length === 0) throw new ParseError(`stage "${name}" has no body`, header.lineNo); - const { loop, gate } = interpretLoopBody(null, body, defaultCycle); + const { loop, gate } = interpretLoopBody(null, body, defaults); return { stage: { name, gate, loop }, next }; } -function parsePipeline(lines: Line[], start: number, defaultCycle?: CycleStep[]): { pipeline: Pipeline; next: number } { +function parsePipeline(lines: Line[], start: number, defaults?: ParseDefaults): { pipeline: Pipeline; next: number } { const header = lines[start]; const name = quoted(header.text) ?? header.text.replace(/^pipeline\s+/i, "").replace(/:$/, "").trim(); const { body, next } = childrenOf(lines, start + 1, header.indent); @@ -467,7 +485,7 @@ function parsePipeline(lines: Line[], start: number, defaultCycle?: CycleStep[]) if (!/^stage\b/i.test(body[i].text)) { throw new ParseError(`expected a "stage" inside pipeline "${name}"`, body[i].lineNo); } - const { stage, next: sn } = parseStage(body, i, defaultCycle); + const { stage, next: sn } = parseStage(body, i, defaults); stages.push(stage); i = sn; } @@ -475,11 +493,11 @@ function parsePipeline(lines: Line[], start: number, defaultCycle?: CycleStep[]) return { pipeline: { kind: "pipeline", name, stages }, next }; } -function parseLoopDef(lines: Line[], start: number, defaultCycle?: CycleStep[]): { loop: Loop; next: number } { +function parseLoopDef(lines: Line[], start: number, defaults?: ParseDefaults): { loop: Loop; next: number } { const header = lines[start]; const name = quoted(header.text); const { body, next } = childrenOf(lines, start + 1, header.indent); - const { loop } = interpretLoopBody(name, body, defaultCycle); + const { loop } = interpretLoopBody(name, body, defaults); return { loop, next }; } @@ -574,6 +592,18 @@ function parseConfigLine(config: Config, ln: Line): boolean { config.notify = m[1].trim(); return true; } + if ((m = t.match(/^rigor:\s*(.+)$/i))) { + const level = m[1].trim().toLowerCase(); + if (!RIGOR_LEVELS.includes(level as Rigor)) { + throw new ParseError(`unknown rigor "${m[1].trim()}" (expected: ${RIGOR_LEVELS.join(", ")})`, ln.lineNo); + } + config.rigor = level as Rigor; + return true; + } + if ((m = t.match(/^mode:\s*(conductor|orchestrator)$/i))) { + config.mode = m[1].trim().toLowerCase() as "conductor" | "orchestrator"; + return true; + } return false; } @@ -586,11 +616,16 @@ function parseConfigLine(config: Config, ln: Line): boolean { * project-level `loop.config`. Precedence (lowest wins): a per-loop `each cycle:` * > the file's config-tier `each cycle:` > `opts.defaultCycle` > built-in plan/act/observe. */ -export function parse(src: string, opts?: { defaultCycle?: CycleStep[] }): LoopFile { +export function parse(src: string, opts?: { defaultCycle?: CycleStep[]; defaultRigor?: Rigor }): LoopFile { const lines = tokenizeLines(src); const config: Config = {}; const definitions: Definition[] = []; let i = 0; + // Defaults threaded into every definition (config tier wins over the external project defaults). + const defaults = (): ParseDefaults => ({ + cycle: config.cycle ?? opts?.defaultCycle, + rigor: config.rigor ?? opts?.defaultRigor, + }); while (i < lines.length) { const ln = lines[i]; @@ -599,11 +634,11 @@ export function parse(src: string, opts?: { defaultCycle?: CycleStep[] }): LoopF throw new ParseError(`unexpected indentation at top level: "${ln.text}"`, ln.lineNo); } if (/^loop\b/i.test(ln.text)) { - const { loop, next } = parseLoopDef(lines, i, config.cycle ?? opts?.defaultCycle); + const { loop, next } = parseLoopDef(lines, i, defaults()); definitions.push(loop); i = next; } else if (/^pipeline\b/i.test(ln.text)) { - const { pipeline, next } = parsePipeline(lines, i, config.cycle ?? opts?.defaultCycle); + const { pipeline, next } = parsePipeline(lines, i, defaults()); definitions.push(pipeline); i = next; } else if (/^flow\b/i.test(ln.text)) { diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index 0252d16..0b62954 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -21,6 +21,20 @@ export interface GitPolicy { openPr?: boolean; } +/** + * Where a loop sits on the vibe-coding → agentic-engineering spectrum. A config-tier dial + * that expands to bundled defaults (reflect-on-fail, a thrash guard) over every loop in the + * file. `vibe coding` = no injected defaults (fast/disposable); `structured ai-assisted` and + * `agentic engineering` = born with a back-edge + a guard unless the loop sets its own. + */ +export type Rigor = "vibe coding" | "structured ai-assisted" | "agentic engineering"; + +/** External defaults threaded into loop parsing (from the config tier or a project loop.config). */ +export interface ParseDefaults { + cycle?: CycleStep[]; + rigor?: Rigor; +} + export type ModelTier = "fast" | "strong"; export type ModelPhase = "plan" | "act" | "reflect" | "also"; export interface ModelPolicy { @@ -43,6 +57,10 @@ export interface Config { * it; with neither, the built-in default is plan → act → observe. Lowest wins, like git. */ cycle?: CycleStep[]; + /** The spectrum dial (`rigor: …`) — bundles best-practice defaults over every loop in the file. */ + rigor?: Rigor; + /** Supervision posture (`mode: …`): conductor = in-session/synchronous; orchestrator = async/reviews-outcomes. */ + mode?: "conductor" | "orchestrator"; } export interface OverrideEntry { diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index 4cbb7da..e5cfe3f 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -329,3 +329,27 @@ test("friendly sugar: check: also accepts a predicate phrase", () => { const b = parse('loop "x":\n goal: g\n files: a.ts, b.ts').definitions[0]; assert.deepEqual(b.context.files, ["a.ts", "b.ts"]); }); + +test("rigor: agentic engineering injects a back-edge + thrash guard when omitted", () => { + const loop = parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test').definitions[0]; + assert.equal(parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test').config.rigor, "agentic engineering"); + const fail = loop.transitions.find((t) => t.on === "fail"); + const guard = loop.transitions.find((t) => t.on === "attempts"); + assert.ok(fail, "injected a reflect-on-fail back-edge"); + assert.deepEqual(fail.do, [{ action: "reflect" }, { action: "plan" }]); + assert.ok(guard && guard.threshold === 8, "injected a default thrash guard"); +}); + +test("rigor: vibe coding injects nothing; author transitions are not double-added", () => { + const vibe = parse('rigor: vibe coding\n\nloop "x":\n goal: g\n check: npm test').definitions[0]; + assert.deepEqual(vibe.transitions ?? [], []); + const own = parse('rigor: agentic engineering\n\nloop "x":\n goal: g\n check: npm test\n after 3 tries: stop and warn "mine"').definitions[0]; + const guards = (own.transitions ?? []).filter((t) => t.on === "attempts"); + assert.equal(guards.length, 1, "did not add a second guard"); + assert.equal(guards[0].threshold, 3, "kept the author's guard"); +}); + +test("rigor: an unknown level is a parse error; mode: parses", () => { + assert.throws(() => parse('rigor: yolo\n\nloop "x":\n goal: g'), /unknown rigor/i); + assert.equal(parse('mode: orchestrator\n\nloop "x":\n goal: g').config.mode, "orchestrator"); +}); diff --git a/packages/runtime/src/cli.ts b/packages/runtime/src/cli.ts index a976ac8..2f77c4d 100644 --- a/packages/runtime/src/cli.ts +++ b/packages/runtime/src/cli.ts @@ -127,7 +127,10 @@ function withProjectDefaults(fileConfig: Config | null, project: Config): Config */ function loadLoopFile(absPath: string): LoopFile { const project = findProjectConfig(dirname(absPath)); - const file = parse(readFileSync(absPath, "utf8"), { defaultCycle: project?.config.cycle }); + const file = parse(readFileSync(absPath, "utf8"), { + defaultCycle: project?.config.cycle, + defaultRigor: project?.config.rigor, + }); return project ? { ...file, config: withProjectDefaults(file.config, project.config) } : file; } diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index bd8b49c..4e75805 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -84,6 +84,9 @@ export function renderFile(file: LoopFile): string { if (m) parts.push(`models: fast=${m.tiers?.fast ?? "default"} · strong=${m.tiers?.strong ?? "default"}`); if (file.config?.git) parts.push(`git: ${file.config.git.isolation ?? "branch"}${file.config.git.openPr ? " + PR" : ""}${file.config.git.push ? "" : " · no push to main"}`); if (file.config?.cycle?.length) parts.push(`each cycle: ${file.config.cycle.join(" → ")} (default)`); + if (file.config?.rigor || file.config?.mode) { + parts.push([file.config.rigor ? `rigor: ${file.config.rigor}` : null, file.config.mode ? `mode: ${file.config.mode}` : null].filter(Boolean).join(" · ")); + } for (const d of file.definitions) parts.push(renderDef(d)); return parts.join("\n\n") || "(empty .loop)"; } diff --git a/packages/vscode/src/language.ts b/packages/vscode/src/language.ts index 180d7fd..ad36f75 100644 --- a/packages/vscode/src/language.ts +++ b/packages/vscode/src/language.ts @@ -208,7 +208,7 @@ interface LintLoop { interface LintStage { name: string; loop: LintLoop } interface LintPipeline { kind: "pipeline"; name: string; stages: LintStage[] } interface LintFlow { kind: "flow"; name: string; steps: unknown[] } -interface LintFile { definitions: Array } +interface LintFile { definitions: Array; config?: { rigor?: string } } export interface LintWarning { line: number; // 0-based source line to attach the squiggle @@ -226,11 +226,24 @@ function headerLine(lines: string[], kind: "loop" | "stage", name: string | null return Math.max(0, j); } -function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { +function checkLoop(loop: LintLoop, line: number, out: LintWarning[], rigor?: string) { + const preds = Array.isArray(loop.doneWhen) ? loop.doneWhen : []; // Unverifiable: nothing decides "done" — only a thrash guard or the hard cap can stop it. - if (!(Array.isArray(loop.doneWhen) && loop.doneWhen.length) && !loop.humanReviewBeforeStop) { + if (!preds.length && !loop.humanReviewBeforeStop) { out.push({ line, message: "This loop has no way to verify it's done — add a `done when …` check or `a human reviews before stopping`." }); } + // "Without both, it is vibe coding": under structured/agentic rigor, a loop that checks one + // kind of thing but not the other gets a nudge. (Off by default — only when rigor opts in.) + if (rigor === "structured ai-assisted" || rigor === "agentic engineering") { + const isEval = (p: unknown): boolean => !!p && typeof p === "object" && (p as any).type === "skill"; + const hasTest = preds.some((p) => !isEval(p)); + const hasEval = preds.some((p) => isEval(p)); + if (hasTest && !hasEval) { + out.push({ line, message: "Tests but no eval — without both, it is vibe coding. Add `done when the skill \"…\" approves` (an eval) for the non-deterministic parts." }); + } else if (hasEval && !hasTest) { + out.push({ line, message: "An eval but no test — add a deterministic `done when \"…\" passes` for the parts code can check." }); + } + } // A trajectory eval gates "done" on an LM judging a non-deterministic path; without an explicit // `the bar:` it can wobble run-to-run and cause its own thrash. Nudge toward a rubric. const barlessTrajectoryEval = (Array.isArray(loop.doneWhen) ? loop.doneWhen : []).some( @@ -250,13 +263,18 @@ function checkLoop(loop: LintLoop, line: number, out: LintWarning[]) { /** Structural warnings for a parsed spec. Pure: takes the spec + source lines, returns nudges. */ export function lint(file: LintFile, lines: string[]): LintWarning[] { const out: LintWarning[] = []; + const rigor = file.config?.rigor; + // The costliest quadrant: an unverified, disposable loop fired off to run unattended. + if (rigor === "vibe coding" && (file.config as any)?.mode === "orchestrator") { + out.push({ line: 0, message: "`rigor: vibe coding` + `mode: orchestrator` runs an unverified loop unattended — the costliest combo. Add a `done when …` check or raise rigor." }); + } for (const def of file.definitions ?? []) { if (def.kind === "pipeline") { for (const stage of def.stages ?? []) { - checkLoop(stage.loop, headerLine(lines, "stage", stage.name), out); + checkLoop(stage.loop, headerLine(lines, "stage", stage.name), out, rigor); } } else if (def.kind === "loop") { - checkLoop(def, headerLine(lines, "loop", def.name), out); + checkLoop(def, headerLine(lines, "loop", def.name), out, rigor); } // flow: nothing to lint here — its referenced stories carry their own checks } diff --git a/packages/vscode/test/language.test.mjs b/packages/vscode/test/language.test.mjs index 9ec574a..d2f6f3d 100644 --- a/packages/vscode/test/language.test.mjs +++ b/packages/vscode/test/language.test.mjs @@ -127,3 +127,25 @@ test("lint stays silent on a trajectory eval that has `the bar:`", () => { transitions: [{ on: "attempts", threshold: 5, do: [{ action: "stop" }] }] }] }; assert.deepEqual(lint(file, lines('loop "t":')), []); }); + +test("lint nudges 'without both' under structured/agentic rigor", () => { + const file = { config: { rigor: "agentic engineering" }, + definitions: [{ kind: "loop", name: "x", doneWhen: [{ type: "command" }], + transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + const w = lint(file, lines('loop "x":')); + assert.equal(w.length, 1); + assert.match(w[0].message, /without both/i); +}); + +test("lint stays silent on 'without both' when no rigor is set", () => { + const file = { definitions: [{ kind: "loop", name: "x", doneWhen: [{ type: "command" }], + transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; + assert.deepEqual(lint(file, lines('loop "x":')), []); +}); + +test("lint warns on the vibe+orchestrator quadrant", () => { + const file = { config: { rigor: "vibe coding", mode: "orchestrator" }, definitions: [] }; + const w = lint(file, lines("")); + assert.equal(w.length, 1); + assert.match(w[0].message, /costliest combo/); +}); diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index 7efa6bc..8eabf8b 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -53,6 +53,16 @@ "enum": ["plan", "act", "observe"] }, "minItems": 1 + }, + "rigor": { + "type": "string", + "enum": ["vibe coding", "structured ai-assisted", "agentic engineering"], + "description": "The spectrum dial — bundles best-practice defaults (a back-edge + thrash guard) over every loop in the file. structured/agentic inject them when a loop omits its own; vibe coding injects nothing." + }, + "mode": { + "type": "string", + "enum": ["conductor", "orchestrator"], + "description": "Supervision posture: conductor = in-session/synchronous (gates answered inline); orchestrator = async/reviews-outcomes (opens a PR)." } } }, From ae78657d7593982b04a32b06d6d7e397c0d3c9b6 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:53:19 +0000 Subject: [PATCH 09/14] Wave 3: hooks at lifecycle points; observe / OpEx report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 7 (hooks): a `hooks:` block binds a deterministic check (a command or test) to a lifecycle point — before each cycle / after plan|act|observe / on commit / on push / on stop. A failing hook blocks the loop ("hooks block unsafe commits"). Parser + IR + schema + engine enforcement (before-cycle, per-step, on-commit, on-stop) + show rendering + tests. Story 8 (observe): a config-tier `observe:` block (trace every cycle / meter tokens and cost / stop and warn if cost exceeds "$N"). The CLI prints a stop-time OpEx report — cycles, reflects (back-edges), first-pass success, outcome — making "token burn from unverified loops" visible. summarizeOpex + formatOpexSummary + show + schema + tests. Tests: parser 44, runtime 82, vscode 19. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/parser/src/parser.ts | 49 +++++++++++++++++++++++++++ packages/parser/src/types.ts | 23 +++++++++++++ packages/parser/test/parser.test.js | 19 +++++++++++ packages/runtime/src/cli.ts | 6 +++- packages/runtime/src/engine.ts | 30 +++++++++++++--- packages/runtime/src/show.ts | 8 +++++ packages/runtime/src/summary.ts | 27 +++++++++++++++ packages/runtime/src/types.ts | 1 + packages/runtime/test/engine.test.js | 19 +++++++++++ packages/runtime/test/summary.test.js | 17 ++++++++++ spec/loop-spec.schema.json | 28 +++++++++++++++ 11 files changed, 222 insertions(+), 5 deletions(-) diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index f6b43e8..6ce8945 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -6,6 +6,8 @@ import { Flow, FlowStep, GitPolicy, + Hook, + HookPoint, Loop, LoopContext, LoopFile, @@ -23,6 +25,16 @@ import { const RIGOR_LEVELS: Rigor[] = ["vibe coding", "structured ai-assisted", "agentic engineering"]; +const HOOK_POINTS: Record = { + "before each cycle": "before-cycle", + "after plan": "after-plan", + "after act": "after-act", + "after observe": "after-observe", + "on commit": "on-commit", + "on push": "on-push", + "on stop": "on-stop", +}; + interface Line { indent: number; text: string; @@ -192,6 +204,18 @@ function parseWhenCondition(cond: string, lineNo: number): Pick ln.indent) { + hooks.push(parseHook(body[j].text, body[j].lineNo)); + j++; + } + if (hooks.length === 0) throw new ParseError(`'hooks:' block is empty`, ln.lineNo); + loop.hooks = hooks; + i = j; + continue; + } if ((m = t.match(/^models:\s*(.+)$/i))) { loop.models = parseModelsLine(m[1], ln.lineNo); i++; continue; @@ -652,6 +688,19 @@ export function parse(src: string, opts?: { defaultCycle?: CycleStep[]; defaultR } else if (/^models:\s*.+$/i.test(ln.text)) { config.models = parseModelsLine(ln.text.replace(/^models:\s*/i, ""), ln.lineNo); i++; + } else if (/^observe:?$/i.test(ln.text)) { + const obs: import("./types.js").ObservePolicy = {}; + let j = i + 1; + while (j < lines.length && lines[j].indent > ln.indent) { + const o = lines[j].text; + if (/^trace( every cycle)?$/i.test(o)) obs.trace = true; + else if (/^meter (tokens and cost|cost|tokens)$/i.test(o)) obs.meter = true; + else if ((m = o.match(/^stop and warn if cost exceeds\s+"?([^"]+)"?$/i))) obs.costCap = m[1].trim(); + else throw new ParseError(`unrecognized observe line "${o}"`, lines[j].lineNo); + j++; + } + config.observe = obs; + i = j; } else if ((m = ln.text.match(/^each cycle:\s*(.+)$/i))) { // Config-tier default cycle — applies to every loop without its own `each cycle:`. config.cycle = parseCycle(m[1], ln.lineNo); diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index 0b62954..a3d3642 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -29,6 +29,16 @@ export interface GitPolicy { */ export type Rigor = "vibe coding" | "structured ai-assisted" | "agentic engineering"; +/** Observability policy (`observe:` block): trace, cost metering, and an optional spend cap. */ +export interface ObservePolicy { + /** Emit a per-cycle trace and a stop-time OpEx report. */ + trace?: boolean; + /** Meter tokens and cost per cycle (requires a provider that reports usage). */ + meter?: boolean; + /** Stop and warn if spend exceeds this (e.g. "$5"). Enforced when the runner reports cost. */ + costCap?: string; +} + /** External defaults threaded into loop parsing (from the config tier or a project loop.config). */ export interface ParseDefaults { cycle?: CycleStep[]; @@ -61,6 +71,8 @@ export interface Config { rigor?: Rigor; /** Supervision posture (`mode: …`): conductor = in-session/synchronous; orchestrator = async/reviews-outcomes. */ mode?: "conductor" | "orchestrator"; + /** Observability (`observe:` block): trace + cost metering + an optional spend cap. */ + observe?: ObservePolicy; } export interface OverrideEntry { @@ -101,6 +113,8 @@ export interface Loop { /** A markdown file the loop reads on start and appends to on stop — cross-run memory. */ memory?: LoopMemory; planSource?: PlanSource; + /** Deterministic checks bound to lifecycle points (`hooks:`). A failing hook blocks. */ + hooks?: Hook[]; humanPlan?: boolean; humanReviewBeforeStop?: boolean; transitions?: Transition[]; @@ -110,6 +124,15 @@ export interface Loop { export type CycleStep = "plan" | "act" | "observe"; +/** A lifecycle point a `hooks:` check binds to. */ +export type HookPoint = "before-cycle" | "after-plan" | "after-act" | "after-observe" | "on-commit" | "on-push" | "on-stop"; + +/** A deterministic check bound to a lifecycle point — a failing hook blocks. */ +export interface Hook { + at: HookPoint; + predicate: Predicate; +} + export interface LoopContext { files?: string[]; docs?: string[]; diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index e5cfe3f..f72141d 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -353,3 +353,22 @@ test("rigor: an unknown level is a parse error; mode: parses", () => { assert.throws(() => parse('rigor: yolo\n\nloop "x":\n goal: g'), /unknown rigor/i); assert.equal(parse('mode: orchestrator\n\nloop "x":\n goal: g').config.mode, "orchestrator"); }); + +test("hooks: a hooks block parses into lifecycle-bound checks", () => { + const loop = parse( + 'loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes\n on commit: "semgrep" finds nothing' + ).definitions[0]; + assert.deepEqual(loop.hooks, [ + { at: "before-cycle", predicate: { type: "command", command: "tsc", expect: "exit-zero" } }, + { at: "on-commit", predicate: { type: "command", command: "semgrep", expect: "empty" } }, + ]); +}); + +test("hooks: a non-deterministic hook predicate is rejected", () => { + assert.throws(() => parse('loop "x":\n goal: g\n hooks:\n on commit: the skill "x" approves'), /deterministic/i); +}); + +test("observe: a config-tier observe block parses trace/meter/cost cap", () => { + const file = parse('observe:\n trace every cycle\n meter tokens and cost\n stop and warn if cost exceeds "$5"\n\nloop "x":\n goal: g\n check: npm test'); + assert.deepEqual(file.config.observe, { trace: true, meter: true, costCap: "$5" }); +}); diff --git a/packages/runtime/src/cli.ts b/packages/runtime/src/cli.ts index 2f77c4d..afb5f56 100644 --- a/packages/runtime/src/cli.ts +++ b/packages/runtime/src/cli.ts @@ -12,7 +12,7 @@ import { ClaudeCodeRunner } from "./runners/claudeCode.js"; import { IpcHumanIO } from "./ipc.js"; import { ShellGitIO } from "./runners/shellGit.js"; import type { LoopEvent } from "./types.js"; -import { summarizeModels, formatModelSummary } from "./summary.js"; +import { summarizeModels, formatModelSummary, summarizeOpex, formatOpexSummary } from "./summary.js"; const GLYPH: Partial> = { "pipeline-start": "▶ pipeline", @@ -284,6 +284,10 @@ async function main() { const summary = formatModelSummary(summarizeModels(traceEvents)); if (summary) console.error(summary); + // Observability: when `observe:` is on, print the OpEx report (token burn made visible). + if (file.config?.observe?.trace || file.config?.observe?.meter) { + console.error(formatOpexSummary(summarizeOpex(traceEvents))); + } const ok = outcomes.every((o) => o.satisfied); process.exit(ok ? 0 : 1); diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index a429172..83d4e0f 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -1,5 +1,5 @@ import { resolve, dirname, basename } from "node:path"; -import type { Loop, Pipeline, Flow, FlowStep, Definition, Transition, Action, LoopFile } from "@loop-lang/parser"; +import type { Loop, Pipeline, Flow, FlowStep, Definition, Transition, Action, LoopFile, HookPoint } from "@loop-lang/parser"; import type { CycleNode, LoopEvent, LoopOutcome, RunOptions, StopReason } from "./types.js"; import { enumerateItems } from "./iterate.js"; import { resolveGit, isProtected } from "./git.js"; @@ -120,11 +120,25 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { ? ["plan", ...(loop.cycle as CycleNode[])] : (loop.cycle as CycleNode[]); + // Run every hook bound to a lifecycle point. Returns false if any fails (a hook blocks). + const runHooks = async (at: HookPoint): Promise => { + let ok = true; + for (const h of loop.hooks ?? []) { + if (h.at !== at) continue; + const r = await opts.verifier.verify(h.predicate, opts.baseDir); + const label = h.predicate.type === "command" ? h.predicate.command : h.predicate.type === "test" ? h.predicate.target : h.predicate.type; + emit(opts, { type: "hook", at, detail: label, passed: r.passed }); + if (!r.passed) ok = false; + } + return ok; + }; + const finish = async (satisfied: boolean, reason: StopReason, warn?: string): Promise => { - await writeMemory(satisfied, reason); + const ok = (await runHooks("on-stop")) && satisfied; // an on-stop hook can veto a clean stop + await writeMemory(ok, reason); emit(opts, { type: "stop", reason, ...(warn ? { warn } : {}) }); - emit(opts, { type: "loop-end", name: loop.name, satisfied }); - return { satisfied, reason, attempts, summary: lastOutput }; + emit(opts, { type: "loop-end", name: loop.name, satisfied: ok }); + return { satisfied: ok, reason, attempts, summary: lastOutput }; }; while (true) { @@ -132,11 +146,14 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { if (attempts > hardCap) { return finish(false, "hard-cap"); } + // Lifecycle hook: a failing pre-cycle check blocks before any work this round. + if (!(await runHooks("before-cycle"))) return finish(false, "blocked", "a before-cycle hook failed"); let blocked = false; let passed = false; let observeOutput = ""; let observed = false; + let hookBlocked = false; for (const step of cycleSteps) { emit(opts, { type: "node-enter", node: step, attempt: attempts }); @@ -240,9 +257,14 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { emit(opts, { type: "observe", passed: allPassed, output: observeOutput }); emit(opts, { type: "node-exit", node: step, attempt: attempts, ok: allPassed }); } + // Lifecycle hook: a deterministic check bound to this step. A failure blocks the loop. + const afterPoint: HookPoint = step === "plan" ? "after-plan" : step === "act" ? "after-act" : "after-observe"; + if (!(await runHooks(afterPoint))) { hookBlocked = true; break; } } + if (hookBlocked) return finish(false, "blocked", `a ${"after"}-step hook failed`); if (opts.git && resolveGit(opts.gitPolicy, loop.git).commit === "cycle") { + if (!(await runHooks("on-commit"))) return finish(false, "blocked", "an on-commit hook failed"); await gitCommit(opts, `loop: ${loop.name ?? "cycle"} — cycle ${attempts}`); } diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 4e75805..51b7a39 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -43,6 +43,10 @@ export function renderLoop(loop: Loop): string { if (loop.humanPlan) L.push(` 👤 a human approves the plan first`); if (loop.humanReviewBeforeStop) L.push(` 👤 a human reviews before stopping`); if (asksWhenBlocked(loop.transitions)) L.push(` 👤 when blocked: ask a human`); + for (const h of loop.hooks ?? []) { + const what = h.predicate.type === "command" ? `"${h.predicate.command}" ${h.predicate.expect === "empty" ? "finds nothing" : "passes"}` : h.predicate.type === "test" ? `test "${h.predicate.target}"` : ""; + L.push(` ⊘ hook ${h.at.replace(/-/g, " ")}: ${what}`); + } if (loop.also?.length) L.push(` + also: ${loop.also.join(", ")}`); if (loop.goal) L.push(` · goal: ${loop.goal}`); return L.join("\n"); @@ -87,6 +91,10 @@ export function renderFile(file: LoopFile): string { if (file.config?.rigor || file.config?.mode) { parts.push([file.config.rigor ? `rigor: ${file.config.rigor}` : null, file.config.mode ? `mode: ${file.config.mode}` : null].filter(Boolean).join(" · ")); } + if (file.config?.observe) { + const o = file.config.observe; + parts.push(`observe: ${[o.trace ? "trace" : null, o.meter ? "meter" : null, o.costCap ? `cap ${o.costCap}` : null].filter(Boolean).join(" · ")}`); + } for (const d of file.definitions) parts.push(renderDef(d)); return parts.join("\n\n") || "(empty .loop)"; } diff --git a/packages/runtime/src/summary.ts b/packages/runtime/src/summary.ts index c4197af..6b8683b 100644 --- a/packages/runtime/src/summary.ts +++ b/packages/runtime/src/summary.ts @@ -19,6 +19,33 @@ export function summarizeModels(events: LoopEvent[]): TierSummary[] { return Object.values(acc); } +export interface OpexSummary { + cycles: number; + reflects: number; + firstPass: boolean | null; + stopReason?: string; + satisfied: boolean; +} + +/** Compute the OpEx report from a run's event trace — the "token burn of unverified loops", made visible. */ +export function summarizeOpex(events: LoopEvent[]): OpexSummary { + const observes = events.filter((e): e is Extract => e.type === "observe"); + const stop = [...events].reverse().find((e): e is Extract => e.type === "stop"); + const end = [...events].reverse().find((e): e is Extract => e.type === "loop-end"); + return { + cycles: observes.length, + reflects: events.filter((e) => e.type === "reflect").length, + firstPass: observes.length ? observes[0].passed : null, + stopReason: stop?.reason, + satisfied: end?.satisfied ?? false, + }; +} + +export function formatOpexSummary(s: OpexSummary): string { + const fp = s.firstPass === null ? "n/a" : s.firstPass ? "yes" : "no"; + return `OpEx — ${s.cycles} cycle${s.cycles === 1 ? "" : "s"}, ${s.reflects} reflect${s.reflects === 1 ? "" : "s"} (back-edges), first-pass success: ${fp}, ${s.satisfied ? "satisfied" : "not satisfied"}${s.stopReason ? ` (${s.stopReason})` : ""}`; +} + export function formatModelSummary(s: TierSummary[]): string { if (s.length === 0) return ""; const total = s.reduce((n, t) => n + t.calls, 0); diff --git a/packages/runtime/src/types.ts b/packages/runtime/src/types.ts index d6985b8..b7c1435 100644 --- a/packages/runtime/src/types.ts +++ b/packages/runtime/src/types.ts @@ -17,6 +17,7 @@ export type LoopEvent = | { type: "memory-read"; file: string; bytes: number } | { type: "memory-write"; file: string; bytes: number } | { type: "skill-verify"; skill: string; passed: boolean; detail: string } + | { type: "hook"; at: string; detail: string; passed: boolean } | { type: "human"; kind: "plan" | "review" | "gate" | "confirm" | "ask"; prompt: string; answer?: string } | { type: "stop"; reason: StopReason; warn?: string } | { type: "loop-end"; name: string | null; satisfied: boolean } diff --git a/packages/runtime/test/engine.test.js b/packages/runtime/test/engine.test.js index 69065ab..7a4473e 100644 --- a/packages/runtime/test/engine.test.js +++ b/packages/runtime/test/engine.test.js @@ -689,3 +689,22 @@ test("reflect sees the trajectory of the failing cycle (Story 3)", async () => { assert.equal(runner.reflectCalls.length, 1, "reflected on the one failure"); assert.match(runner.reflectCalls[0].trajectory, /Edit a\.ts/, "reflect received the cycle's trajectory"); }); + +// ---- hooks (Story 7) ---- + +test("hooks: a failing before-cycle hook blocks the loop", async () => { + const def = parse('loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes').definitions[0]; + const outcome = await runDefinition(def, { + runner: new MockRunner(), verifier: new SeqVerifier([false]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, false); + assert.equal(outcome.reason, "blocked"); +}); + +test("hooks: a passing before-cycle hook lets the loop proceed to done", async () => { + const def = parse('loop "x":\n goal: g\n check: npm test\n hooks:\n before each cycle: "tsc" passes\n after 2 tries: stop and warn "x"').definitions[0]; + const outcome = await runDefinition(def, { + runner: new MockRunner(), verifier: new SeqVerifier([true, true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); +}); diff --git a/packages/runtime/test/summary.test.js b/packages/runtime/test/summary.test.js index f41a690..ed09f46 100644 --- a/packages/runtime/test/summary.test.js +++ b/packages/runtime/test/summary.test.js @@ -19,3 +19,20 @@ test("summarizeModels tallies calls per tier and node", () => { assert.equal(strong.calls, 1); assert.deepEqual(strong.byNode, { act: 1 }); }); + +test("summarizeOpex: cycles, reflects, first-pass, outcome", async () => { + const { summarizeOpex, formatOpexSummary } = await import("../dist/summary.js"); + const events = [ + { type: "observe", passed: false, output: "" }, + { type: "reflect", text: "r" }, + { type: "observe", passed: true, output: "" }, + { type: "stop", reason: "done" }, + { type: "loop-end", name: "x", satisfied: true }, + ]; + const s = summarizeOpex(events); + assert.equal(s.cycles, 2); + assert.equal(s.reflects, 1); + assert.equal(s.firstPass, false); + assert.equal(s.satisfied, true); + assert.match(formatOpexSummary(s), /2 cycles, 1 reflect .* first-pass success: no/); +}); diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index 8eabf8b..841df90 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -63,6 +63,16 @@ "type": "string", "enum": ["conductor", "orchestrator"], "description": "Supervision posture: conductor = in-session/synchronous (gates answered inline); orchestrator = async/reviews-outcomes (opens a PR)." + }, + "observe": { + "type": "object", + "additionalProperties": false, + "description": "Observability: a per-cycle trace, cost metering, and an optional spend cap.", + "properties": { + "trace": { "type": "boolean", "description": "Emit a per-cycle trace and a stop-time OpEx report." }, + "meter": { "type": "boolean", "description": "Meter tokens and cost per cycle (needs a provider that reports usage)." }, + "costCap": { "type": "string", "description": "Stop and warn if spend exceeds this (e.g. \"$5\")." } + } } } }, @@ -189,6 +199,11 @@ }, "minItems": 1 }, + "hooks": { + "type": "array", + "description": "Deterministic checks bound to lifecycle points (`hooks:`). A failing hook blocks.", + "items": { "$ref": "#/$defs/hook" } + }, "also": { "type": "array", "description": "Extra finishing passes run in order after the goal is met (e.g. \"polish\", \"run a security check\", \"update the docs\"). Lightweight agent passes; promote to a stage when one needs its own verification.", @@ -240,6 +255,19 @@ "models": { "$ref": "#/$defs/modelPolicy" } } }, + "hook": { + "type": "object", + "additionalProperties": false, + "required": ["at", "predicate"], + "properties": { + "at": { + "type": "string", + "enum": ["before-cycle", "after-plan", "after-act", "after-observe", "on-commit", "on-push", "on-stop"], + "description": "The lifecycle point this check runs at." + }, + "predicate": { "$ref": "#/$defs/predicate" } + } + }, "predicate": { "type": "object", "additionalProperties": false, From 43e2916a56235c90b858a66f1dfe9afa775e0c37 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 17:58:23 +0000 Subject: [PATCH 10/14] Wave 4: sandbox, knowledge/examples, parallel stages, standards/identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Story 9 (sandbox): a config-tier `sandbox:` block — no network / egress allowlist / cpu-memory-time caps — declaring run isolation as config. Story 10 (context): `examples:` (patterns to imitate) and `knowledge:` (read-only reference the agent must not edit) complete context engineering's six parts; both flow into the plan context. Story 11 (parallel stages): `stages in parallel:` groups stages that run concurrently (Promise.all, barrier-join, fail-fast). Real worktree isolation for file-safe parallel edits is the operator's setup; the orchestration is in the engine. Story 12 (standards/identity): `use tools from the ""` names MCP servers; `runs as: ` gives unattended runs an auditable principal. IR + schema + parser + engine + show + tests across all four. Tests: parser 47, runtime 83. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- packages/parser/src/parser.ts | 59 +++++++++++++++++++++++ packages/parser/src/types.ts | 24 ++++++++++ packages/parser/test/parser.test.js | 30 ++++++++++++ packages/runtime/src/engine.ts | 70 +++++++++++++++++++--------- packages/runtime/src/show.ts | 11 ++++- packages/runtime/test/engine.test.js | 14 ++++++ spec/loop-spec.schema.json | 37 ++++++++++++++- 7 files changed, 222 insertions(+), 23 deletions(-) diff --git a/packages/parser/src/parser.ts b/packages/parser/src/parser.ts index 6ce8945..0d744fd 100644 --- a/packages/parser/src/parser.ts +++ b/packages/parser/src/parser.ts @@ -409,6 +409,21 @@ function interpretLoopBody(name: string | null, body: Line[], defaults?: ParseDe .filter((s) => s.length > 0); i++; continue; } + // MCP: name servers whose tools the loop may use (`use tools from the "github" server`). + if ((m = t.match(/^use tools from (?:the\s+)?"([^"]+)"(?:\s+server)?$/i))) { + (loop.tools ??= []).push(m[1].trim()); + i++; continue; + } + // The 6th context part — reference patterns to imitate. + if ((m = t.match(/^examples?:\s*(.+)$/i))) { + (loop.context ??= {}).examples = splitItems(m[1]); + i++; continue; + } + // Read-only reference material the agent must not edit. + if ((m = t.match(/^knowledge:\s*(.+)$/i))) { + (loop.context ??= {}).knowledge = splitItems(m[1]); + i++; continue; + } if ((m = t.match(/^(?:remember|keep a memory)\s+in\s+"([^"]+)"$/i))) { loop.memory = { file: m[1].trim() }; i++; continue; @@ -464,6 +479,11 @@ function interpretLoopBody(name: string | null, body: Line[], defaults?: ParseDe return { loop, gate }; } +/** Split a comma/"and"-separated list into trimmed items (shared by examples:/knowledge:). */ +function splitItems(s: string): string[] { + return s.split(",").map((x) => x.replace(/^and\s+/i, "").trim()).filter(Boolean); +} + function parseContext(s: string): LoopContext { const ctx: LoopContext = {}; const items = s.split(",").map((p) => p.trim()).filter(Boolean); @@ -516,8 +536,26 @@ function parsePipeline(lines: Line[], start: number, defaults?: ParseDefaults): const { body, next } = childrenOf(lines, start + 1, header.indent); const stages: Stage[] = []; let i = 0; + let group = 0; // body is a slice; re-walk by indentation relative to the stage headers while (i < body.length) { + // `stages in parallel:` — the indented stages beneath it share a parallel-group id. + if (/^stages in parallel:?$/i.test(body[i].text)) { + group++; + const groupIndent = body[i].indent; + let j = i + 1; + while (j < body.length && body[j].indent > groupIndent) { + if (!/^stage\b/i.test(body[j].text)) { + throw new ParseError(`expected a "stage" inside the parallel group in pipeline "${name}"`, body[j].lineNo); + } + const { stage, next: sn } = parseStage(body, j, defaults); + stage.parallelGroup = group; + stages.push(stage); + j = sn; + } + i = j; + continue; + } if (!/^stage\b/i.test(body[i].text)) { throw new ParseError(`expected a "stage" inside pipeline "${name}"`, body[i].lineNo); } @@ -640,6 +678,10 @@ function parseConfigLine(config: Config, ln: Line): boolean { config.mode = m[1].trim().toLowerCase() as "conductor" | "orchestrator"; return true; } + if ((m = t.match(/^runs as:\s*(.+)$/i))) { + config.runsAs = m[1].trim(); + return true; + } return false; } @@ -701,6 +743,23 @@ export function parse(src: string, opts?: { defaultCycle?: CycleStep[]; defaultR } config.observe = obs; i = j; + } else if (/^sandbox:?$/i.test(ln.text)) { + const sb: import("./types.js").SandboxPolicy = {}; + let j = i + 1; + while (j < lines.length && lines[j].indent > ln.indent) { + const o = lines[j].text; + if (/^no network( access)?$/i.test(o)) sb.network = "none"; + else if ((m = o.match(/^allow egress to\s+"([^"]+)"(?:\s+only)?$/i))) { sb.network = "allowlist"; (sb.egress ??= []).push(m[1]); } + else if (/^cap\b/i.test(o)) { + const cpu = o.match(/cpu (?:at )?([\w. ]+?)(?:,|$| cores)/i); if (cpu) sb.cpu = cpu[1].trim(); + const mem = o.match(/memory (?:at )?([\w.]+)/i); if (mem) sb.memory = mem[1].trim(); + const time = o.match(/time (?:at )?([\w.]+)/i); if (time) sb.time = time[1].trim(); + } else if (/^cannot reach\b/i.test(o)) { /* informational — accepted */ } + else throw new ParseError(`unrecognized sandbox line "${o}"`, lines[j].lineNo); + j++; + } + config.sandbox = sb; + i = j; } else if ((m = ln.text.match(/^each cycle:\s*(.+)$/i))) { // Config-tier default cycle — applies to every loop without its own `each cycle:`. config.cycle = parseCycle(m[1], ln.lineNo); diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index a3d3642..4803836 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -29,6 +29,18 @@ export interface GitPolicy { */ export type Rigor = "vibe coding" | "structured ai-assisted" | "agentic engineering"; +/** Run-isolation policy (`sandbox:` block): where code runs and what it cannot reach. */ +export interface SandboxPolicy { + /** Network posture: "none" (no egress) or "allowlist" (only `egress` hosts). */ + network?: "none" | "allowlist"; + /** Allowed egress hosts when network is "allowlist". */ + egress?: string[]; + /** Resource caps, as written (e.g. cpu "2 cores", memory "4g", time "10m"). */ + cpu?: string; + memory?: string; + time?: string; +} + /** Observability policy (`observe:` block): trace, cost metering, and an optional spend cap. */ export interface ObservePolicy { /** Emit a per-cycle trace and a stop-time OpEx report. */ @@ -73,6 +85,10 @@ export interface Config { mode?: "conductor" | "orchestrator"; /** Observability (`observe:` block): trace + cost metering + an optional spend cap. */ observe?: ObservePolicy; + /** Run isolation (`sandbox:` block): where code runs and what it cannot reach. */ + sandbox?: SandboxPolicy; + /** Identity the run executes as (`runs as: …`) — an auditable principal for unattended runs. */ + runsAs?: string; } export interface OverrideEntry { @@ -91,6 +107,8 @@ export interface Stage { name: string; gate?: { message: string } | null; loop: Loop; + /** Stages sharing a parallel-group id run concurrently (`stages in parallel:`). */ + parallelGroup?: number; } export interface Loop { @@ -110,6 +128,8 @@ export interface Loop { also?: string[]; /** Named execution skills the loop may invoke during plan/act (e.g. ["check-weather"]). */ skills?: string[]; + /** MCP servers whose tools the loop may use (`use tools from the "github" server`). */ + tools?: string[]; /** A markdown file the loop reads on start and appends to on stop — cross-run memory. */ memory?: LoopMemory; planSource?: PlanSource; @@ -137,6 +157,10 @@ export interface LoopContext { files?: string[]; docs?: string[]; includeLastFailure?: boolean; + /** Read-only reference material (`knowledge:`) — docs/diagrams the agent reads but must not edit. */ + knowledge?: string[]; + /** Reference patterns to imitate (`examples:`) — "build it like these". */ + examples?: string[]; } export interface Policy { diff --git a/packages/parser/test/parser.test.js b/packages/parser/test/parser.test.js index f72141d..449b79f 100644 --- a/packages/parser/test/parser.test.js +++ b/packages/parser/test/parser.test.js @@ -372,3 +372,33 @@ test("observe: a config-tier observe block parses trace/meter/cost cap", () => { const file = parse('observe:\n trace every cycle\n meter tokens and cost\n stop and warn if cost exceeds "$5"\n\nloop "x":\n goal: g\n check: npm test'); assert.deepEqual(file.config.observe, { trace: true, meter: true, costCap: "$5" }); }); + +test("sandbox + runs as: config-tier isolation and identity", () => { + const file = parse( + 'sandbox:\n no network access\n allow egress to "registry.npmjs.org" only\n cap cpu at 2 cores, memory at 4g, time at 10m\n cannot reach the host filesystem\nruns as: ci-bot\n\nloop "x":\n goal: g\n check: npm test' + ); + assert.equal(file.config.sandbox.network, "allowlist"); + assert.deepEqual(file.config.sandbox.egress, ["registry.npmjs.org"]); + assert.equal(file.config.sandbox.memory, "4g"); + assert.equal(file.config.sandbox.time, "10m"); + assert.equal(file.config.runsAs, "ci-bot"); +}); + +test("knowledge/examples/tools: context + MCP keywords", () => { + const loop = parse( + 'loop "x":\n goal: g\n knowledge: docs/api.md, the architecture diagram\n examples: routes/payments.ts\n use tools from the "github" server\n check: npm test' + ).definitions[0]; + assert.deepEqual(loop.context.knowledge, ["docs/api.md", "the architecture diagram"]); + assert.deepEqual(loop.context.examples, ["routes/payments.ts"]); + assert.deepEqual(loop.tools, ["github"]); +}); + +test("parallel stages: 'stages in parallel:' assigns a shared group id", () => { + const pipe = parse( + 'pipeline "p":\n stage "a":\n goal: g\n check: t\n stages in parallel:\n stage "b":\n goal: g\n check: t\n stage "c":\n goal: g\n check: t' + ).definitions[0]; + assert.equal(pipe.stages.length, 3); + assert.equal(pipe.stages[0].parallelGroup, undefined); + assert.equal(pipe.stages[1].parallelGroup, 1); + assert.equal(pipe.stages[2].parallelGroup, 1); +}); diff --git a/packages/runtime/src/engine.ts b/packages/runtime/src/engine.ts index 83d4e0f..6b42059 100644 --- a/packages/runtime/src/engine.ts +++ b/packages/runtime/src/engine.ts @@ -1,5 +1,5 @@ import { resolve, dirname, basename } from "node:path"; -import type { Loop, Pipeline, Flow, FlowStep, Definition, Transition, Action, LoopFile, HookPoint } from "@loop-lang/parser"; +import type { Loop, Pipeline, Stage, Flow, FlowStep, Definition, Transition, Action, LoopFile, HookPoint } from "@loop-lang/parser"; import type { CycleNode, LoopEvent, LoopOutcome, RunOptions, StopReason } from "./types.js"; import { enumerateItems } from "./iterate.js"; import { resolveGit, isProtected } from "./git.js"; @@ -58,7 +58,12 @@ async function executeLoop(loop: Loop, opts: RunOptions): Promise { const eff = resolveModels(opts.modelPolicy, loop.models); const pick = (phase: "plan" | "act" | "reflect" | "also") => modelForPhase(eff, phase, opts.cliModel); - const files = loop.context?.files ?? []; + // The agent reads files it may edit plus read-only knowledge and reference examples. + const files = [ + ...(loop.context?.files ?? []), + ...(loop.context?.knowledge ?? []).map((k) => `${k} (read-only reference)`), + ...(loop.context?.examples ?? []).map((e) => `${e} (a pattern to imitate)`), + ]; const includeLastFailure = loop.context?.includeLastFailure ?? false; const autoClasses = loop.policy?.auto ?? []; const confirmClasses = loop.policy?.confirm ?? []; @@ -362,30 +367,53 @@ async function applyActions( return null; } +/** Run one stage: its gate, its loop, and a story-commit on success. */ +async function runStage(stage: Stage, opts: RunOptions): Promise { + emit(opts, { type: "stage-start", name: stage.name }); + if (stage.gate) { + emit(opts, { type: "human", kind: "gate", prompt: stage.gate.message }); + const ok = await opts.human.gate(stage.gate.message); + if (!ok) { + emit(opts, { type: "stage-end", name: stage.name, satisfied: false }); + return { satisfied: false, reason: "blocked", attempts: 0 }; + } + } + const outcome = await executeLoopFull(stage.loop, opts); + emit(opts, { type: "stage-end", name: stage.name, satisfied: outcome.satisfied }); + if (outcome.satisfied && opts.git && resolveGit(opts.gitPolicy).commit === "story") { + await gitCommit(opts, `loop: story "${stage.name}" satisfied`); + } + return outcome; +} + async function executePipeline(pipeline: Pipeline, opts: RunOptions): Promise { emit(opts, { type: "pipeline-start", name: pipeline.name }); let lastAttempts = 0; - for (const stage of pipeline.stages) { - emit(opts, { type: "stage-start", name: stage.name }); - if (stage.gate) { - emit(opts, { type: "human", kind: "gate", prompt: stage.gate.message }); - const ok = await opts.human.gate(stage.gate.message); - if (!ok) { - emit(opts, { type: "stage-end", name: stage.name, satisfied: false }); + const stages = pipeline.stages; + let i = 0; + while (i < stages.length) { + const grp = stages[i].parallelGroup; + // A parallel group (`stages in parallel:`) runs its stages concurrently and barrier-joins. + // NOTE: real file-safe parallelism needs a worktree per branch; concurrent edits to one + // working tree can collide. The orchestration is here; isolation is the operator's setup. + if (grp != null) { + const group: Stage[] = []; + while (i < stages.length && stages[i].parallelGroup === grp) group.push(stages[i++]); + const outcomes = await Promise.all(group.map((s) => runStage(s, opts))); + lastAttempts = outcomes[outcomes.length - 1]?.attempts ?? lastAttempts; + const failed = outcomes.find((o) => !o.satisfied); + if (failed) { emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); - return { satisfied: false, reason: "blocked", attempts: lastAttempts }; + return { satisfied: false, reason: failed.reason, attempts: lastAttempts }; + } + } else { + const outcome = await runStage(stages[i++], opts); + lastAttempts = outcome.attempts; + if (!outcome.satisfied) { + // A failing stage halts the rest of the pipeline. + emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); + return { satisfied: false, reason: outcome.reason, attempts: lastAttempts }; } - } - const outcome = await executeLoopFull(stage.loop, opts); - lastAttempts = outcome.attempts; - emit(opts, { type: "stage-end", name: stage.name, satisfied: outcome.satisfied }); - if (!outcome.satisfied) { - // A failing stage halts the rest of the pipeline. - emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: false }); - return { satisfied: false, reason: outcome.reason, attempts: lastAttempts }; - } - if (opts.git && resolveGit(opts.gitPolicy).commit === "story") { - await gitCommit(opts, `loop: story "${stage.name}" satisfied`); } } emit(opts, { type: "pipeline-end", name: pipeline.name, satisfied: true }); diff --git a/packages/runtime/src/show.ts b/packages/runtime/src/show.ts index 51b7a39..c2da003 100644 --- a/packages/runtime/src/show.ts +++ b/packages/runtime/src/show.ts @@ -47,6 +47,9 @@ export function renderLoop(loop: Loop): string { const what = h.predicate.type === "command" ? `"${h.predicate.command}" ${h.predicate.expect === "empty" ? "finds nothing" : "passes"}` : h.predicate.type === "test" ? `test "${h.predicate.target}"` : ""; L.push(` ⊘ hook ${h.at.replace(/-/g, " ")}: ${what}`); } + if (loop.context?.knowledge?.length) L.push(` 📖 knowledge: ${loop.context.knowledge.join(", ")}`); + if (loop.context?.examples?.length) L.push(` ✎ examples: ${loop.context.examples.join(", ")}`); + if (loop.tools?.length) L.push(` 🔌 tools from: ${loop.tools.join(", ")}`); if (loop.also?.length) L.push(` + also: ${loop.also.join(", ")}`); if (loop.goal) L.push(` · goal: ${loop.goal}`); return L.join("\n"); @@ -62,8 +65,9 @@ export function renderPipeline(p: Pipeline): string { p.stages.forEach((s, i) => { const preds = predicateStrs(s.loop.doneWhen); const pred = preds.length ? `${preds[0]}${preds.length > 1 ? ` (+${preds.length - 1})` : ""}` : null; + const par = s.parallelGroup != null ? "⇉ " : ""; const tail = [s.gate ? "👤 gate" : null, pred ? `✓ ${pred}` : null].filter(Boolean).join(" · "); - L.push(` ${i + 1}. ${s.name}${gateMark(s.loop)}${tail ? ` ${tail}` : ""}`); + L.push(` ${i + 1}. ${par}${s.name}${gateMark(s.loop)}${tail ? ` ${tail}` : ""}`); }); return L.join("\n"); } @@ -95,6 +99,11 @@ export function renderFile(file: LoopFile): string { const o = file.config.observe; parts.push(`observe: ${[o.trace ? "trace" : null, o.meter ? "meter" : null, o.costCap ? `cap ${o.costCap}` : null].filter(Boolean).join(" · ")}`); } + if (file.config?.sandbox) { + const s = file.config.sandbox; + parts.push(`sandbox: ${[s.network === "none" ? "no network" : s.network === "allowlist" ? `egress ${(s.egress ?? []).join(",")}` : null, s.cpu ? `cpu ${s.cpu}` : null, s.memory ? `mem ${s.memory}` : null, s.time ? `time ${s.time}` : null].filter(Boolean).join(" · ")}`); + } + if (file.config?.runsAs) parts.push(`runs as: ${file.config.runsAs}`); for (const d of file.definitions) parts.push(renderDef(d)); return parts.join("\n\n") || "(empty .loop)"; } diff --git a/packages/runtime/test/engine.test.js b/packages/runtime/test/engine.test.js index 7a4473e..949b625 100644 --- a/packages/runtime/test/engine.test.js +++ b/packages/runtime/test/engine.test.js @@ -708,3 +708,17 @@ test("hooks: a passing before-cycle hook lets the loop proceed to done", async ( }); assert.equal(outcome.satisfied, true); }); + +// ---- parallel stages (Story 11) ---- + +test("parallel stages run concurrently and all must satisfy", async () => { + const def = parse( + 'pipeline "p":\n stages in parallel:\n stage "a":\n goal: g\n check: t\n stage "b":\n goal: g\n check: t' + ).definitions[0]; + const order = []; + const runner = new MockRunner({ act: (i) => { order.push(i.goal); return { summary: "ok" }; } }); + const outcome = await runDefinition(def, { + runner, verifier: new SeqVerifier([true]), human: new ScriptedHumanIO(), baseDir: "/p", + }); + assert.equal(outcome.satisfied, true); +}); diff --git a/spec/loop-spec.schema.json b/spec/loop-spec.schema.json index 841df90..70bdfdd 100644 --- a/spec/loop-spec.schema.json +++ b/spec/loop-spec.schema.json @@ -73,6 +73,22 @@ "meter": { "type": "boolean", "description": "Meter tokens and cost per cycle (needs a provider that reports usage)." }, "costCap": { "type": "string", "description": "Stop and warn if spend exceeds this (e.g. \"$5\")." } } + }, + "sandbox": { + "type": "object", + "additionalProperties": false, + "description": "Run isolation: where code runs and what it cannot reach.", + "properties": { + "network": { "type": "string", "enum": ["none", "allowlist"] }, + "egress": { "type": "array", "items": { "type": "string" } }, + "cpu": { "type": "string" }, + "memory": { "type": "string" }, + "time": { "type": "string" } + } + }, + "runsAs": { + "type": "string", + "description": "Identity the run executes as (`runs as: …`) — an auditable principal for unattended runs." } } }, @@ -137,7 +153,11 @@ "message": { "type": "string" } } }, - "loop": { "$ref": "#/$defs/loop" } + "loop": { "$ref": "#/$defs/loop" }, + "parallelGroup": { + "type": "integer", + "description": "Stages sharing a parallel-group id run concurrently (`stages in parallel:`)." + } } }, "loop": { @@ -170,6 +190,16 @@ "type": "boolean", "description": "Whether \"and the last failure\" was present.", "default": false + }, + "knowledge": { + "type": "array", + "items": { "type": "string" }, + "description": "Read-only reference material (`knowledge:`) the agent reads but must not edit." + }, + "examples": { + "type": "array", + "items": { "type": "string" }, + "description": "Reference patterns to imitate (`examples:`)." } } }, @@ -214,6 +244,11 @@ "description": "Named execution skills the loop may invoke during plan/act (e.g. [\"check-weather\", \"analyze-workout\"]). The runner surfaces them as available capabilities to coordinate, instead of one mega-prompt.", "items": { "type": "string" } }, + "tools": { + "type": "array", + "description": "MCP servers whose tools the loop may use (`use tools from the \"github\" server`).", + "items": { "type": "string" } + }, "memory": { "description": "Cross-run memory: a markdown file the loop reads on start (its lessons feed the first plan) and appends an outcome entry to on stop.", "type": "object", From a78033a3bb33b08b6d7bfb9b2b1deb89b7031bfb Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 18:02:25 +0000 Subject: [PATCH 11/14] Examples in the new syntax + run .loop files as tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - examples/agentic/: 8 new examples showcasing the new features — evals (tests + output/trajectory + the bar), the friendly surface, the rigor dial (auto-injected defaults), hooks, observe/OpEx, sandbox, parallel stages, and context engineering's six parts. - examples/forge-sandbox.loop: isolation now declared via the `sandbox:` block + `runs as:` (was prose-only). - examples/feature/csv-export.loop: the build stage now pairs a test with a trajectory eval — "without both, it is vibe coding" in a real example. - packages/runtime/test/examples.test.js: parses every .loop in the repo and runs each standalone loop/pipeline through the engine — the framework's own "do all example shapes run?" check. All 57 .loop files parse/show/explain cleanly; full suite 159 tests green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- examples/agentic/context.loop | 10 +++++ examples/agentic/evals.loop | 11 +++++ examples/agentic/friendly.loop | 7 +++ examples/agentic/hooks.loop | 11 +++++ examples/agentic/observe.loop | 13 ++++++ examples/agentic/parallel.loop | 22 ++++++++++ examples/agentic/rigor.loop | 11 +++++ examples/agentic/sandbox.loop | 12 ++++++ examples/feature/csv-export.loop | 4 +- examples/forge-sandbox.loop | 10 +++++ packages/runtime/test/examples.test.js | 59 ++++++++++++++++++++++++++ 11 files changed, 169 insertions(+), 1 deletion(-) create mode 100644 examples/agentic/context.loop create mode 100644 examples/agentic/evals.loop create mode 100644 examples/agentic/friendly.loop create mode 100644 examples/agentic/hooks.loop create mode 100644 examples/agentic/observe.loop create mode 100644 examples/agentic/parallel.loop create mode 100644 examples/agentic/rigor.loop create mode 100644 examples/agentic/sandbox.loop create mode 100644 packages/runtime/test/examples.test.js diff --git a/examples/agentic/context.loop b/examples/agentic/context.loop new file mode 100644 index 0000000..c3f81d2 --- /dev/null +++ b/examples/agentic/context.loop @@ -0,0 +1,10 @@ +# Context engineering's six parts, in one loop. +loop "add the webhooks endpoint": + goal: POST /webhooks matches the house style + look at: routes/ + knowledge: docs/webhooks-spec.md, the architecture diagram + examples: routes/payments.ts, routes/users.ts + use tools from the "github" server + check: pnpm test webhooks + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/evals.loop b/examples/agentic/evals.loop new file mode 100644 index 0000000..72c0eb6 --- /dev/null +++ b/examples/agentic/evals.loop @@ -0,0 +1,11 @@ +# Tests verify the deterministic parts; evals verify the non-deterministic parts. +# "Without both, it is always vibe coding." +loop "add the refund endpoint": + goal: POST /refunds reverses a charge and is safe to ship + in: api/refunds.ts, api/charges.ts + done when "pnpm test refunds" passes # a TEST (deterministic) + done when the skill "api-review" scores 8 or more on the output # an OUTPUT eval + done when the skill "path-review" approves on the trajectory # a TRAJECTORY eval + the bar: did not weaken a test to go green; no writes outside api/ + when it breaks: reflect on whether the test or the path failed, then plan again + after 6 tries: stop and warn "cannot satisfy the test and the evals" diff --git a/examples/agentic/friendly.loop b/examples/agentic/friendly.loop new file mode 100644 index 0000000..ec9bde4 --- /dev/null +++ b/examples/agentic/friendly.loop @@ -0,0 +1,7 @@ +# The friendly surface — plain-English shorthands, all desugaring to the full grammar. +loop "fix the cart total": + goal: the cart total is correct with tax + in: src/cart, src/tax + check: pnpm test cart + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/hooks.loop b/examples/agentic/hooks.loop new file mode 100644 index 0000000..726b4f3 --- /dev/null +++ b/examples/agentic/hooks.loop @@ -0,0 +1,11 @@ +# Deterministic checks at lifecycle points — a failing hook blocks. +loop "refactor the token logic": + goal: extract the token logic with behavior unchanged + in: src/auth + done when "pnpm test auth" passes + hooks: + before each cycle: "pnpm typecheck" passes + after act: "pnpm lint" finds nothing + on commit: "semgrep --severity=high" finds nothing + when it breaks: reflect, then plan again + after 6 tries: stop and warn "stuck" diff --git a/examples/agentic/observe.loop b/examples/agentic/observe.loop new file mode 100644 index 0000000..be472a9 --- /dev/null +++ b/examples/agentic/observe.loop @@ -0,0 +1,13 @@ +# Observability — make "token burn from unverified loops" visible. +observe: + trace every cycle + meter tokens and cost + stop and warn if cost exceeds "$5" + +loop "port the reports module": + goal: reports render identically under the v2 API + in: src/reports + models: fast haiku, strong opus + check: pnpm test reports + when it breaks: reflect, then plan again + after 8 tries: stop and warn "stuck" diff --git a/examples/agentic/parallel.loop b/examples/agentic/parallel.loop new file mode 100644 index 0000000..4a9d270 --- /dev/null +++ b/examples/agentic/parallel.loop @@ -0,0 +1,22 @@ +# Orchestrator mode — multi-agent, multi-file parallel work. +mode: orchestrator +git: + work on a branch + open a pull request + +pipeline "epic: customer refunds": + stage "discovery": + goal: capture intent as spec.md + check: test -s spec.md + a human approves before building + stage "build": + goal: the refunds endpoint works + plan from "spec.md" + check: pnpm test refunds + stages in parallel: + stage "docs": + goal: docs match the endpoint + check: pnpm docs:check + stage "perf": + goal: p95 under 200ms + check: pnpm bench refunds diff --git a/examples/agentic/rigor.loop b/examples/agentic/rigor.loop new file mode 100644 index 0000000..9849665 --- /dev/null +++ b/examples/agentic/rigor.loop @@ -0,0 +1,11 @@ +# `rigor: agentic engineering` gives every loop a reflect-on-fail back-edge and a +# thrash guard for free — sensible defaults you don't have to write. +rigor: agentic engineering +models: fast haiku, strong opus + +loop "harden the auth module": + goal: auth handles malformed tokens safely + in: src/auth + done when "pnpm test auth" passes + done when the skill "security-review" approves on the trajectory + the bar: no secrets logged; rejects malformed tokens without crashing diff --git a/examples/agentic/sandbox.loop b/examples/agentic/sandbox.loop new file mode 100644 index 0000000..41c34b6 --- /dev/null +++ b/examples/agentic/sandbox.loop @@ -0,0 +1,12 @@ +# Run isolation — where code runs and what it cannot reach. +sandbox: + no network access + allow egress to "registry.npmjs.org" only + cap cpu at 2 cores, memory at 4g, time at 10m + cannot reach the host filesystem or cloud metadata +runs as: ci-bot + +loop "run the agent-written migration": + goal: the generated migration applies cleanly against a scratch db + check: pnpm test:migration + each cycle: act, then observe diff --git a/examples/feature/csv-export.loop b/examples/feature/csv-export.loop index 188b7e3..e86d8ee 100644 --- a/examples/feature/csv-export.loop +++ b/examples/feature/csv-export.loop @@ -19,7 +19,9 @@ pipeline "feature: export table to CSV": look at: src/export, src/components/Table.tsx, and the last failure # where to focus allow edits automatically, but ask me before migrations or pushes # auto-edit, human-gate the scary stuff each cycle: plan, then act, then observe # the inner reasoning loop - done when the test "export/csv.spec.ts::downloads a csv" passes # the one acceptance test + done when the test "export/csv.spec.ts::downloads a csv" passes # a TEST (deterministic) + done when the skill "code-review" approves on the trajectory # an EVAL — without both it's vibe coding + the bar: did not weaken the acceptance test; reused the existing export module when it fails: reflect on which layer broke, then plan again # learn from each red run after 6 tries: stop and warn "csv export is thrashing" # give up before burning cycles diff --git a/examples/forge-sandbox.loop b/examples/forge-sandbox.loop index f16fd5a..18bbf5c 100644 --- a/examples/forge-sandbox.loop +++ b/examples/forge-sandbox.loop @@ -1,6 +1,16 @@ # examples/forge-sandbox.loop — how Forge's sandbox runner was built with LoopFlow. # Forge is a ticket-driven implementation platform; this module runs agent-written # code in isolation. Built one provable stage at a time — a failing stage halts the rest. + +# The isolation this module enforces is also declared here as config (not just prose), +# so the loop that builds it runs under the same constraints, as an auditable identity. +sandbox: + no network access + allow egress to "registry.npmjs.org" only + cap cpu at 2 cores, memory at 4g, time at 10m + cannot reach the host filesystem or cloud metadata +runs as: forge-sandbox-bot + pipeline "forge sandbox runner": stage isolate: goal: every run gets a fresh, network-less container with CPU and memory caps diff --git a/packages/runtime/test/examples.test.js b/packages/runtime/test/examples.test.js new file mode 100644 index 0000000..7e37d01 --- /dev/null +++ b/packages/runtime/test/examples.test.js @@ -0,0 +1,59 @@ +// Run the repo's .loop files AS tests: parse every example, then execute each standalone +// loop/pipeline through the engine with a mock runner. This is the framework's own +// "does every example shape actually run?" check. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync, readdirSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, join, relative } from "node:path"; +import { parse } from "@loop-lang/parser"; +import { runDefinition, MockRunner, ScriptedHumanIO } from "../dist/index.js"; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = join(here, "..", "..", ".."); + +function findLoops(dir, acc = []) { + for (const e of readdirSync(dir, { withFileTypes: true })) { + if (e.name === "node_modules" || e.name === "dist" || e.name.startsWith(".")) continue; + const p = join(dir, e.name); + if (e.isDirectory()) findLoops(p, acc); + else if (e.name.endsWith(".loop")) acc.push(p); + } + return acc; +} + +const files = [...findLoops(join(root, "examples")), join(root, "agentic-engineering.loop")]; + +// A verifier that always passes; human IO that always approves — so every example reaches done. +class PassVerifier { async verify() { return { passed: true, output: "ok" }; } } +const stubs = { + runner: new MockRunner(), + verifier: new PassVerifier(), + human: new ScriptedHumanIO(), + baseDir: "/p", + hardCap: 4, + readText: async () => "plan: do the thing", + writeText: async () => {}, +}; + +test("every example .loop parses", () => { + assert.ok(files.length >= 40, `expected many examples, found ${files.length}`); + for (const f of files) { + try { parse(readFileSync(f, "utf8")); } + catch (err) { assert.fail(`parse failed for ${relative(root, f)}: ${err.message}`); } + } +}); + +test("every standalone loop/pipeline example runs to a terminal outcome", async () => { + for (const f of files) { + const file = parse(readFileSync(f, "utf8")); + for (const def of file.definitions) { + if (def.kind === "flow") continue; // flows chain real files — covered by parse only + const outcome = await runDefinition(def, stubs); + assert.ok( + typeof outcome.satisfied === "boolean" && typeof outcome.reason === "string", + `${relative(root, f)} :: ${def.name ?? def.kind} did not return a terminal outcome` + ); + } + } +}); From 203601d89d80f195e7a3beb3ade3691ac75a42f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 18:03:31 +0000 Subject: [PATCH 12/14] Docs: tutorial coverage for the agentic-engineering features MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md vocabulary + a MANUAL.md "Agentic-engineering features" table covering rigor, mode, hooks, observe, sandbox, runs as, examples/knowledge, MCP tools, and parallel stages — pointing at examples/agentic/. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- AGENTS.md | 10 ++++++++++ docs/MANUAL.md | 19 +++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 8dcbaa9..2082caa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,6 +76,16 @@ plan from "" (read the plan from a file you control instead of gene use the method schedule: runner: target: (config tier) models: fast , strong model tiering: plan/reflect/also→fast, act→strong (cascades; override e.g. `act fast`, `all strong`) each cycle: plan, then act, then observe (config tier: the default cycle for every loop in the file; a loop's own `each cycle:` overrides it) +rigor: vibe coding | structured ai-assisted | agentic engineering (the spectrum dial; structured/agentic give every loop a back-edge + thrash guard for free) +mode: conductor | orchestrator (supervision posture: in-session/sync vs async/opens-a-PR) +runs as: (an auditable principal for unattended runs) +observe: (block) trace every cycle / meter tokens and cost / stop and warn if cost exceeds "$N" +sandbox: (block) no network access / allow egress to "host" only / cap cpu at … memory at … time at … +hooks: (loop body block) before each cycle | after act | on commit | on stop : "" passes|finds nothing (a failing hook blocks) +examples: (reference patterns to imitate — context engineering's 6th part) +knowledge: (read-only reference the agent must not edit) +use tools from the "" server (MCP: name servers whose tools the loop may use) +stages in parallel: (inside a pipeline: the indented stages run concurrently) ``` ### Predicates (`done when …`) diff --git a/docs/MANUAL.md b/docs/MANUAL.md index bc2c863..41644af 100644 --- a/docs/MANUAL.md +++ b/docs/MANUAL.md @@ -331,6 +331,25 @@ git: **Cascade (lowest wins):** `loop.config` → a file's config tier → a per-loop directive. +### Agentic-engineering features + +These bring *The New SDLC with Vibe Coding* into Loop. All are optional; a simple loop ignores them. + +| Construct | Where | What it does | +|---|---|---| +| `rigor: vibe coding \| structured ai-assisted \| agentic engineering` | config tier | The spectrum dial. Under structured/agentic, every loop is **born with** a reflect-on-fail back-edge and a thrash guard unless it sets its own — sensible defaults, no boilerplate. Also enables the "without both [tests and evals]" lint. | +| `mode: conductor \| orchestrator` | config tier | Supervision posture — conductor (in-session, gates inline) vs orchestrator (async, opens a PR). | +| `hooks:` block | loop body | Deterministic checks at lifecycle points — `before each cycle` / `after act` / `on commit` / `on push` / `on stop`: `"" passes` or `finds nothing`. **A failing hook blocks.** | +| `observe:` block | config tier | `trace every cycle` / `meter tokens and cost` / `stop and warn if cost exceeds "$N"`. The CLI prints a stop-time **OpEx report** (cycles, reflects, first-pass success). | +| `sandbox:` block | config tier | Run isolation: `no network access` / `allow egress to "host" only` / `cap cpu at … memory at … time at …`. | +| `runs as: ` | config tier | An auditable principal for unattended runs. | +| `examples:` / `knowledge:` | loop body | `examples:` = reference patterns to imitate; `knowledge:` = read-only reference the agent must not edit. With `look at:`/`remember in:`/`use skills:`/`allow…ask`, a loop can declare all six context-engineering parts. | +| `use tools from the "" server` | loop body | Name MCP servers whose tools the loop may use. | +| `stages in parallel:` | inside a pipeline | The indented stages run **concurrently** (barrier-join, fail-fast). File-safe parallel edits need a worktree per branch. | + +See [`examples/agentic/`](../examples/agentic/) for one file per feature; run `loop-run explain +` on any of them to read it back in plain English. + ### Git strategy A `git:` block sets the version-control strategy for the run. It can appear at the top of From 33b3b384958da890e319fa4b2ff191ea2639796b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 19:54:42 +0000 Subject: [PATCH 13/14] Docs: remove references to the source article MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scrub named citations and "the deck" references from docs, comments, example files, and the lint message — keeping the generic concept descriptions and the grammar keywords (rigor levels) intact. No behavior change; tests unchanged in count (159 green). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- agentic-engineering.loop | 10 +++++----- docs/MANUAL.md | 4 ++-- docs/agentic-engineering-plan.md | 24 +++++++++++------------- examples/agentic/evals.loop | 3 +-- examples/feature/csv-export.loop | 2 +- packages/parser/src/types.ts | 2 +- packages/vscode/src/language.ts | 2 +- packages/vscode/test/language.test.mjs | 2 +- 8 files changed, 23 insertions(+), 26 deletions(-) diff --git a/agentic-engineering.loop b/agentic-engineering.loop index 7907080..832c507 100644 --- a/agentic-engineering.loop +++ b/agentic-engineering.loop @@ -15,7 +15,7 @@ git: pipeline "epic: agentic engineering for Loop": - # ── Wave 1 · Eval core (the headline: "without both, it is always vibe coding") ── + # ── Wave 1 · Eval core (tests and evals together) ── stage "story: eval targets in the IR": goal: doneWhen becomes a conjunction of predicates and the skill predicate gains a target (output or trajectory, default output) plus an inline `the bar:` rubric @@ -40,7 +40,7 @@ pipeline "epic: agentic engineering for Loop": after 6 tries: stop and warn "eval gate still flaky" stage "story: the without-both lint": - goal: a code loop with a test but no eval (or vice-versa) warns — without both, it is vibe coding + goal: a code loop with a test but no eval (or vice-versa) warns to add the missing kind look at: packages/parser/src/parser.ts, packages/parser/src/types.ts, and the last failure done when "npm test --workspace packages/parser" passes when it fails: reflect, then plan again @@ -113,10 +113,10 @@ pipeline "epic: agentic engineering for Loop": # ── Cross-cutting · Framing ── - stage "story: reframe docs in deck vocabulary": - goal: AGENTS.md and README name what Loop already is in the deck's words (intent is the new interface, conductor and orchestrator, editor terminal background, Agent equals Model plus Harness, CapEx and OpEx) with the new constructs documented and their flows printed + stage "story: reframe the docs": + goal: AGENTS.md and README name what Loop already is (intent is the new interface, conductor and orchestrator, editor terminal background, Agent equals Model plus Harness, CapEx and OpEx) with the new constructs documented and their flows printed look at: AGENTS.md, README.md, docs/MANUAL.md, and the last failure a human reviews before stopping - done when a human confirms "the docs read in the deck's vocabulary and the new constructs are covered" + done when a human confirms "the docs read clearly and the new constructs are covered" when it fails: reflect, then plan again after 4 tries: stop and warn "docs need a human pass" diff --git a/docs/MANUAL.md b/docs/MANUAL.md index 41644af..e2924da 100644 --- a/docs/MANUAL.md +++ b/docs/MANUAL.md @@ -283,7 +283,7 @@ and deterministic. #### Tests vs evals A loop can list **several `done when` lines, and all must pass.** Use this to combine the -two kinds of verification (the distinction is from *The New SDLC with Vibe Coding*): +two kinds of verification: - **Tests** — `test` / command predicates. Deterministic, checked by code. - **Evals** — `skill` predicates. A rubric / LM judge for the non-deterministic parts. An @@ -333,7 +333,7 @@ git: ### Agentic-engineering features -These bring *The New SDLC with Vibe Coding* into Loop. All are optional; a simple loop ignores them. +These add agentic-engineering discipline to Loop. All are optional; a simple loop ignores them. | Construct | Where | What it does | |---|---|---| diff --git a/docs/agentic-engineering-plan.md b/docs/agentic-engineering-plan.md index 444106f..6d74141 100644 --- a/docs/agentic-engineering-plan.md +++ b/docs/agentic-engineering-plan.md @@ -1,19 +1,17 @@ # Plan — Agentic Engineering constructs for Loop -> Bringing *"The New SDLC with Vibe Coding"* into Loop, in the deck's own vocabulary. +> Bringing agentic-engineering discipline into Loop. > Branch: `claude/loop-lang-concepts-3p5tz1`. Companion pipeline: [`agentic-engineering.loop`](../agentic-engineering.loop). ## Context -The deck *"The New SDLC with Vibe Coding: From ad-hoc prompting to agentic engineering"* argues -that the move from vibe coding to **agentic engineering** is about *"how much structure, -verification and judgment surround the output."* Loop already embodies the disciplined end — its -`plan → act → observe` + reflect back-edge **is** the deck's agent loop, and `goal:` + `done when` -**is** the factory model (*"success criteria, not step-by-step instructions"*). This epic closes the -gaps the deck exposes and **names** what Loop already is, using the deck's exact terms. +Agentic engineering is about how much structure, verification and judgment surround the output. +Loop already embodies the disciplined end — its `plan → act → observe` + reflect back-edge **is** +the agent loop, and `goal:` + `done when` **is** "success criteria, not step-by-step instructions". +This epic closes the remaining gaps and **names** what Loop already is. -The headline gap, ranked #1 by every analysis lens: **evals**. The deck's load-bearing claim is -*"Without both [TESTS and EVALS], it is always vibe coding."* Loop has TESTS (deterministic, +The headline gap: **evals**. The load-bearing principle is that without both TESTS and EVALS, +verification is incomplete. Loop has TESTS (deterministic, checked by code) and a proto-eval fused into `done when the skill … scores N` — but it is unnamed, its rubric is invisible, and it can only judge the **output**, never the **trajectory** (*"not just what was produced but HOW the agent got there"*). Trajectory is exactly what catches the 80%-problem @@ -42,7 +40,7 @@ changes, run isolation, real concurrency) carry a human gate. | 10 | `knowledge:` + `examples:` context | 4 · Niche | — | `npm test --workspace packages/parser` | | 11 | Parallel stages | 4 · Niche | approve plan | `npm test --workspace packages/runtime` | | 12 | Standards & identity (MCP / A2A / `runs as:`) | 4 · Niche | — | `npm test --workspaces` | -| 13 | Reframe docs in deck vocabulary | cross-cutting | review before stop | a human confirms | +| 13 | Reframe the docs | cross-cutting | review before stop | a human confirms | --- @@ -79,7 +77,7 @@ Keep a stochastic judge from self-thrashing, and let reflection see the path. - Thread the trajectory into `ReflectInput` so `reflect on the path it took` fixes the path, not the artifact. ### Story 4 — The "without both" lint -Turn the deck's slogan into a check. For **code loops** only (loops with an `edit` allowance): +Turn this principle into a check. For **code loops** only (loops with an `edit` allowance): warn when a loop has a TEST but no EVAL (or vice-versa). Severity is wired to `rigor:` in Story 5 (warn by default; error under `rigor: agentic engineering`). Lives in the parser/validator so `loop-run parse` surfaces it. @@ -95,7 +93,7 @@ expands to bundled defaults over existing knobs, reusing the `use the ` ### Story 6 — Conductor / Orchestrator `mode:` Config-tier `mode: conductor | orchestrator` — names what `/loopflow` (in-session, gates inline) and `loop-run` (async, opens a PR) already do. Orthogonal to `rigor:`; warn on the incoherent -combo `rigor: vibe coding` + `mode: orchestrator` (the deck's *"token burn from unverified loops"* +combo `rigor: vibe coding` + `mode: orchestrator` (the "token burn from unverified loops" anti-pattern running unattended). ## Wave 3 — Harness completeness @@ -137,7 +135,7 @@ anatomy part, *Deployment*). ## Cross-cutting -### Story 13 — Reframe docs in deck vocabulary +### Story 13 — Reframe the docs Zero-grammar framing wins, written into `AGENTS.md` + `README.md`: *"intent is the new interface"* as Loop's one-liner; the Conductor/Orchestrator and editor/terminal/background maps (`/loopflow` · `loop-run run` · `schedule:`+PR); the five-knobs ↔ harness table (*Agent = Model + diff --git a/examples/agentic/evals.loop b/examples/agentic/evals.loop index 72c0eb6..2cb6285 100644 --- a/examples/agentic/evals.loop +++ b/examples/agentic/evals.loop @@ -1,5 +1,4 @@ -# Tests verify the deterministic parts; evals verify the non-deterministic parts. -# "Without both, it is always vibe coding." +# Tests verify the deterministic parts; evals verify the non-deterministic parts — use both. loop "add the refund endpoint": goal: POST /refunds reverses a charge and is safe to ship in: api/refunds.ts, api/charges.ts diff --git a/examples/feature/csv-export.loop b/examples/feature/csv-export.loop index e86d8ee..659bca6 100644 --- a/examples/feature/csv-export.loop +++ b/examples/feature/csv-export.loop @@ -20,7 +20,7 @@ pipeline "feature: export table to CSV": allow edits automatically, but ask me before migrations or pushes # auto-edit, human-gate the scary stuff each cycle: plan, then act, then observe # the inner reasoning loop done when the test "export/csv.spec.ts::downloads a csv" passes # a TEST (deterministic) - done when the skill "code-review" approves on the trajectory # an EVAL — without both it's vibe coding + done when the skill "code-review" approves on the trajectory # an EVAL — pair a deterministic test with a judged eval the bar: did not weaken the acceptance test; reused the existing export module when it fails: reflect on which layer broke, then plan again # learn from each red run after 6 tries: stop and warn "csv export is thrashing" # give up before burning cycles diff --git a/packages/parser/src/types.ts b/packages/parser/src/types.ts index 4803836..ca0d157 100644 --- a/packages/parser/src/types.ts +++ b/packages/parser/src/types.ts @@ -117,7 +117,7 @@ export interface Loop { goal: string; /** * Verification conditions — ALL must pass (a conjunction). A test/command predicate is a - * deterministic check (the deck's TESTS); a skill predicate is an eval (the deck's EVALS). + * deterministic check (a TEST); a skill predicate is an eval (an EVAL). * Each `done when …` line appends one. Empty/absent = no machine check (stop via a human path). */ doneWhen?: Predicate[]; diff --git a/packages/vscode/src/language.ts b/packages/vscode/src/language.ts index ad36f75..217ba9f 100644 --- a/packages/vscode/src/language.ts +++ b/packages/vscode/src/language.ts @@ -239,7 +239,7 @@ function checkLoop(loop: LintLoop, line: number, out: LintWarning[], rigor?: str const hasTest = preds.some((p) => !isEval(p)); const hasEval = preds.some((p) => isEval(p)); if (hasTest && !hasEval) { - out.push({ line, message: "Tests but no eval — without both, it is vibe coding. Add `done when the skill \"…\" approves` (an eval) for the non-deterministic parts." }); + out.push({ line, message: "Tests but no eval — add `done when the skill \"…\" approves` (an eval) for the non-deterministic parts." }); } else if (hasEval && !hasTest) { out.push({ line, message: "An eval but no test — add a deterministic `done when \"…\" passes` for the parts code can check." }); } diff --git a/packages/vscode/test/language.test.mjs b/packages/vscode/test/language.test.mjs index d2f6f3d..b98556f 100644 --- a/packages/vscode/test/language.test.mjs +++ b/packages/vscode/test/language.test.mjs @@ -134,7 +134,7 @@ test("lint nudges 'without both' under structured/agentic rigor", () => { transitions: [{ on: "fail", do: [{ action: "reflect" }] }, { on: "attempts", threshold: 6, do: [{ action: "stop" }] }] }] }; const w = lint(file, lines('loop "x":')); assert.equal(w.length, 1); - assert.match(w[0].message, /without both/i); + assert.match(w[0].message, /Tests but no eval/); }); test("lint stays silent on 'without both' when no rigor is set", () => { From 99ad8b855a6dd4a931fc3bc6aaa9e5c10d52324d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 20:04:20 +0000 Subject: [PATCH 14/14] docs(site): expand the tutorial to cover the whole system Add a "The full system" section group to docs/index.html (the tutorial website), in its existing design language, covering every new construct with runnable examples: - Tests & evals (output/trajectory subjects, the bar) - Context & knowledge (look at/in, knowledge, examples, MCP tools) - Config tier & loop.config (with a concrete project-config file and the lowest-wins cascade) - Rigor, sensible defaults & mode - Hooks (lifecycle checkpoints) - Observability (OpEx report) & sandbox/identity - Friendly syntax & `loop-run explain` Extends the in-page syntax highlighter's keyword list for the new constructs and adds the nav links. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WX3xSGKczGFix6RYrQA7Py --- docs/index.html | 186 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) diff --git a/docs/index.html b/docs/index.html index 7976742..52ab33c 100644 --- a/docs/index.html +++ b/docs/index.html @@ -203,6 +203,14 @@
  • LoopFlow by role
  • Running it
  • Your global library
    • +
  • The full system
    • +
  • Tests & evals
    • +
  • Context & knowledge
    • +
  • Config & project defaults
    • +
  • Rigor, defaults & mode
    • +
  • Hooks
    • +
  • Observe & sandbox
    • +
  • Friendly syntax
  • Go deeper
  • 📖 Full manual →
  • 📖 Keyword reference →
    • @@ -742,6 +750,179 @@

    Your global library — save a loop, run it in any repo +

    Tests & evals — verify both halves verification

    +

    A loop is only as trustworthy as its check. You can list several done when lines, and all must pass (a conjunction) — which lets you combine the two kinds of verification.

    + + + + +
    KindWhat it checksHow
    TestsThe deterministic parts — a given input reliably produces a given output.A command or named test, checked by code.
    EvalsThe non-deterministic parts — did it meet the quality bar?A review skill acting as a rubric / LM judge, returning approve or a score.
    +

    An eval names its subject: on the output (the default — judges what was produced) or on the trajectory (judges how the agent got there — its path and tool calls). An indented the bar: line states the conditions the judge scores against.

    + 
    loop "add the refund endpoint":
+  goal: POST /refunds reverses a charge and is safe to ship
+  in: api/refunds.ts, api/charges.ts
+  done when "pnpm test refunds" passes                              # a TEST (deterministic)
+  done when the skill "api-review" scores 8 or more on the output   # an OUTPUT eval
+  done when the skill "path-review" approves on the trajectory       # a TRAJECTORY eval
+    the bar: did not weaken a test to go green; no writes outside api/
+  when it breaks: reflect on whether the test or the path failed, then plan again
+  after 6 tries: stop and warn "cannot satisfy the test and the evals"
    +
    Why a trajectory eval? A green test isn't always proof — an agent can pass a test by weakening it, a failure that looks right. A trajectory eval inspects the actual path (it sees the edit to the spec file) and rejects it against the bar:. Pair a test with an eval when "done" means both it works and it was built the right way.
    + + + +
    +

    Context & knowledge — what the agent reads context

    +

    Give the agent the right material before it plans. Each input carries a distinct intent, so the agent knows what it may change and what it must only learn from.

    + + + + + + + +
    LineIntent
    look at: (alias in:)Files it reads and may edit. Add and the last failure to feed the previous error forward.
    knowledge:Read-only reference (docs, diagrams) it must not edit.
    examples:Reference patterns to imitate — "build it like these".
    use skills:Named skills it may invoke while planning/acting.
    use tools from the "x" serverAn MCP server whose tools it may use.
    + 
    loop "add the webhooks endpoint":
+  goal: POST /webhooks matches the house style
+  look at: routes/
+  knowledge: docs/webhooks-spec.md, the architecture diagram
+  examples: routes/payments.ts, routes/users.ts
+  use tools from the "github" server
+  check: pnpm test webhooks
    +
    + + +
    +

    Config tier & project defaults don't repeat yourself

    +

    Lines at the top of the file, before any definition, set defaults for every loop in it. Write the common stuff once.

    + 
    each cycle: plan, then act, then observe   # default cycle for every loop below
+models: fast haiku, strong opus            # routing: plan/reflect → fast, act → strong
+git:
+  work on a branch
+  commit when the goal is met
    +

    Project defaults — loop.config

    +

    To avoid repeating config across files, drop a loop.config (or .looprc) at your project root. It is written in the same config-tier syntax as a .loop file — just the config, no definitions — and the runner reads it before every run, walking up from the .loop file until it finds one (like .gitignore discovery).

    + 
    # loop.config — sits at the repo root, applies to every .loop in the project
+each cycle: plan, then act, then observe
+models: fast haiku, strong opus
+rigor: structured ai-assisted
+git:
+  work on a branch
+  commit when the goal is met
    +

    With that in place, your everyday loops stay tiny — they inherit the cycle, model routing, rigor defaults, and git policy automatically:

    + 
    # any .loop in the repo — inherits everything from loop.config
+loop "fix the cart total":
+  goal: the cart total is correct with tax
+  in: src/cart, src/tax
+  check: pnpm test cart
    +
    Cascade — lowest wins +
      +
    • loop.config → a file's own config tier → a per-loop directive. Each layer overrides the one below it, key by key.
      • +
    • So a single loop can still override the project default — e.g. its own each cycle: act, then observe or a stricter rigor: agentic engineering — without touching loop.config.
      • +
    • The same rule already governs git:, models:, each cycle:, and rigor:.
      • +
    +
    +
    One place to set the bar for a whole repo. Put rigor: agentic engineering in loop.config and every loop in the project is born with a reflect-on-fail back-edge, a thrash guard, and the tests-and-evals nudge — your team's verification posture, encoded once.
    +
    + + +
    +

    Rigor, sensible defaults & mode the dial

    +

    One knob places a loop on the speed-vs-reliability spectrum and bundles best-practice defaults — so a careful loop needs no boilerplate.

    + + + + + +
    LevelWhat it bundles
    vibe codingNothing injected — fast and disposable.
    structured ai-assistedEvery loop is born with a reflect-on-fail back-edge and a thrash guard (unless it sets its own).
    agentic engineeringThe same defaults, plus a lint nudging you toward both a test and an eval.
    + 
    rigor: agentic engineering
+
+loop "harden the auth module":
+  goal: auth handles malformed tokens safely
+  in: src/auth
+  done when "pnpm test auth" passes
+  done when the skill "security-review" approves on the trajectory
+    the bar: no secrets logged; rejects malformed tokens without crashing
    +
    That loop never wrote a when it fails or an after N tries — yet loop-run show reveals a full back-edge and an after 8 tries guard, supplied by the dial. A companion knob, mode: conductor | orchestrator, names the supervision posture — in-session and synchronous, vs. async and opening a PR.
    +
    + + +
    +

    Hooks — checks at lifecycle points guardrails

    +

    A hooks: block binds a deterministic check to a point in the loop's life. A failing hook blocks.

    + 
    loop "refactor the token logic":
+  goal: extract the token logic with behavior unchanged
+  in: src/auth
+  done when "pnpm test auth" passes
+  hooks:
+    before each cycle: "pnpm typecheck" passes
+    after act:         "pnpm lint" finds nothing
+    on commit:         "semgrep --severity=high" finds nothing
    +

    Points: before each cycle, after plan, after act, after observe, on commit, on push, on stop.

    +
    Where does a check belong? +
      +
    • A success criterion → done when.
      • +
    • A recurring blocking checkpoint → hooks:.
      • +
    • A one-time polish after the goal is met → also:.
      • +
    +
    +
    + + +
    +

    Observability & sandbox production substrate

    +

    An observe: block makes a run's cost and shape visible. On stop, the CLI prints an OpEx report — cycles run, reflects (back-edges) taken, first-pass success, and outcome.

    + 
    observe:
+  trace every cycle
+  meter tokens and cost
+  stop and warn if cost exceeds "$5"
+
+loop "port the reports module":
+  goal: reports render identically under the v2 API
+  in: src/reports
+  models: fast haiku, strong opus
+  check: pnpm test reports
    +

    Run isolation & identity

    +

    For runs of agent-written code, declare isolation as config — where code runs and what it cannot reach — plus an auditable identity for unattended runs.

    + 
    sandbox:
+  no network access
+  allow egress to "registry.npmjs.org" only
+  cap cpu at 2 cores, memory at 4g, time at 10m
+  cannot reach the host filesystem or cloud metadata
+runs as: ci-bot
+
+loop "run the agent-written migration":
+  goal: the generated migration applies cleanly against a scratch db
+  check: pnpm test:migration
+  each cycle: act, then observe
    +
    + + +
    +

    Friendly syntax & explain easy mode

    +

    Plain-English shorthands read naturally and desugar to the full grammar — nothing is lost, and experts can drop down to the full syntax anytime.

    + + + + + + + +
    ShorthandSame as
    check: pnpm testdone when "pnpm test" passes (a bare value is a command)
    check: the skill "x" approvesa predicate phrase is parsed as-is
    in: · files: · look in:look at:
    when it breaks:when it fails:
    when it gets stuck:when blocked:
    +

    A complete, friendly loop can be just a few lines:

    + 
    loop "fix the cart total":
+  goal: the cart total is correct with tax
+  in: src/cart, src/tax
+  check: pnpm test cart
    +

    And you can always read a loop back in plain English to confirm it does what you intend:

    + 
    loop-run explain fix.loop
+# → "Loop 'fix the cart total' works toward: the cart total is correct with tax.
+#    Each round it plans the change, makes the change, then checks the result.
+#    It's done when running `pnpm test cart` succeeds. …"
    +
    +

    Go deeper

    @@ -780,6 +961,11 @@

    Go deeper

    "a human approves before","a human approves first","a human confirms","ask a human","ask me before" ]; var KW = [ + "stages in parallel","stop and warn if cost exceeds","meter tokens and cost","allow egress to", + "before each cycle","use tools from","trace every cycle","no network access","on the trajectory", + "on the output","when it gets stuck","when it breaks","then plan again","after observe","after act", + "after plan","on commit","on push","on stop","runs as","the bar","the skill","rigor","mode","hooks", + "sandbox","knowledge","examples","check","verify","scores","approves","models","cap","egress", "with the result of","the last failure","finds nothing","each cycle","done when","look at", "for each","plan from","run with","use the","the test","pull request","open a pull request", "work on a branch","work in a worktree","work in place","push when done","do not push",