From 48ad3800856eba11603dfbd1f1bb42dac26a3c33 Mon Sep 17 00:00:00 2001 From: Johannes Schickling Date: Tue, 10 Nov 2026 18:58:22 +0100 Subject: [PATCH 1/3] docs(st3): design for compiling data-DSL missions into programs (program engine M3) Orchestrated missions: the native steps as the data DSL renders them plus one generated program step that releases them; construct mapping, release protocol, published-mission cutover and state classes, parity tiers, latency, slices, risks and open questions. Section 9 of the program engine design points to it. --- docs/st3/README.md | 2 + docs/st3/program-engine-data-dsl.md | 347 ++++++++++++++++++ docs/st3/program-engine.md | 23 +- .../program-engine-data-dsl-design.json | 11 + 4 files changed, 367 insertions(+), 16 deletions(-) create mode 100644 docs/st3/program-engine-data-dsl.md create mode 100644 release-notes/program-engine-data-dsl-design.json diff --git a/docs/st3/README.md b/docs/st3/README.md index c80ea7a625e..bd2261ac667 100644 --- a/docs/st3/README.md +++ b/docs/st3/README.md @@ -12,6 +12,8 @@ Use these documents for implementation details: - [Mission graph runtime](mission-graph-runtime.md) defines the KDL language and execution model. - [Program engine](program-engine.md) is the accepted design for missions whose body is a program: where it runs, its journal, versioning by bundle hash, authority, child runs and visibility. +- [Data-DSL missions as programs](program-engine-data-dsl.md) is the proposed design for compiling + data-DSL missions into programs: orchestrated missions, released steps, parity tests and slices. - [Configuration and KDL](../site/reference/configuration.md) defines day-to-day publication and revision workflows. - [Native driver contracts](native-driver-contracts.md) retains launch, scope, helper, harness-state and context obligations after catalog-runner retirement. - [Owned sets](owned-sets.md) defines complete membership publication, source ordering and retirement. diff --git a/docs/st3/program-engine-data-dsl.md b/docs/st3/program-engine-data-dsl.md new file mode 100644 index 00000000000..d9e0701e4f1 --- /dev/null +++ b/docs/st3/program-engine-data-dsl.md @@ -0,0 +1,347 @@ +# Data-DSL missions as programs + +Status: proposed design for milestone M3 of the [program engine](program-engine.md#build-plan), +awaiting the owner's answers to the [open questions](#open-questions). Behaviour this document +describes is released only as far as the slices that shipped it. + +The program engine runs `Fractal.mission(intent, program)`. Decision D6 makes it the one execution +model: a mission authored in the `fractal` package's data DSL compiles into a program and runs on +the same engine, while the native step graph stays the substrate for worker claims, gates and +products (R12). This document specifies that compilation, what it changes for published missions, +how parity with the native graph is proved, and the slices that ship it. + +## The shape + +A data-DSL mission compiles into an **orchestrated mission**: its native steps exactly as the data +DSL renders them, one generated `program` step, and a mission-level `orchestrated-by` naming that +step. The program decides when each native step starts; the reconciler runs everything else as it +does for any mission. + +```kdl +mission "docs/review" report-to="agent/docs/editor" state="ready" timeout="2h" { + orchestrated-by "program" + input "change" kind="text" + goal "Review one documentation change." + completion { + when "all-steps-exhausted" + } + step "draft" { + assigned-to "agent/docs/writer" + goal "Draft the correction for ${input.change}." + produces { + resource "mission-run/${ST_MISSION_RUN}/draft" { + kind "custom.docs.draft" + } + } + } + step "review" { + assigned-to "agent/docs/editor" + depends-on { + step "draft" "completed" + } + goal "Review the draft." + gate "approved" { + field "state" "resource/mission-run/${ST_MISSION_RUN}/draft" "is" "approved" + } + } + step "program" timeout="2h" { + program "/bundles/.mjs" sha256="" + } +} +``` + +Everything except `orchestrated-by` and `step "program"` is byte-identical to the native rendering +of the same mission, so the KDL a person reads, the step definitions and their hashes do not change. + +| Owned by the program (orchestration) | Owned by the native graph (substrate) | +|---|---| +| when each normal-phase step or loop starts (its release) | the step lifecycle: baselines, selector, readiness, declarations, worker claim and report, nested and used missions, products, gates, retry attempts, timeouts | +| parallelism between released steps | the normal-phase transition: the first final failure ends normal work and cancels the rest ("another step failed") | +| constructs that run inside the body: local execs (§5), decisions over held values (§6) | mission products and gates, the completion frontier, the final phase, the run outcome | +| | reopening a failed run, revisions, cancellation | + +## Options considered + +| | A. Every step inside the program; worker steps as worker turns (child runs, §6 of the engine) | **B. Program releases native steps (chosen)** | C. One shared interpreter bundle, graph read from the declaration | +|---|---|---|---| +| Interpolation (`${ST_MISSION_RUN}`) in products, gates and goals | names the child run, so a later step's gate on `resource/mission-run/${ST_MISSION_RUN}/…` reads the wrong subject | unchanged | unchanged | +| Run-scoped agents (`agent/${ST_MISSION_RUN}/…`) | one agent per child run: a worker loses its session between steps | one per run, as in the native graph | as B | +| Gates that wait on graph state (field, exists, merged, ci-passed, exec, human feedback) | need a graph-read and wait API in the host contract | native, unchanged | as B | +| `missions show`, step-run subjects, work claims | child runs instead of steps | unchanged plus one program step | as B | +| In-flight run cut over by revision | impossible (no shared step definitions) | completed steps carry forward (identical definitions) | as B | +| Bundle per mission revision | yes | yes | no, but the interpreter then changes with the daemon, against R04 | +| Engine work | a graph-read and wait API for gates | release and its admission rule (§2) | B plus a declaration read in the host contract | + +B keeps native semantics wherever the program has nothing to decide, so parity holds by +construction for those parts and the parity tests concentrate on admission. + +## Spec + +### 1. Declaration + +- `orchestrated-by "STEP"` on a mission names a normal-phase step that declares `program` and has + no `depends-on`. The mission must have no other `program` step. +- Every other normal-phase node (step or loop) of an orchestrated mission is **released**: the + reconciler admits it only after the program releases it. Final-phase steps (`finally`) are never + released; the final phase runs natively after the normal phase, also when the run is cancelled, + which a program cannot outlive. +- A released node's `depends-on` stays declared and keeps its meaning as a guard: the daemon refuses + a release whose step dependencies do not hold, and the program fails with that defect. Graph + predicate dependencies and baselines are evaluated natively after release, as in any mission. +- `orchestrated-by` is outside every step definition, so the step definition hashes of an + orchestrated mission equal those of its native rendering. +- A member that has not advertised the orchestrated-mission capability refuses the declaration at + apply, as `mission_program=1` fences `program` steps. This is also the protocol fence for §2: a + bundle that releases steps never runs on an engine without `release`. + +### 2. Release (host protocol and HostContext) + +| Direction | Message | Fields | Meaning | +|---|---|---|---| +| host → daemon | `release` | `step`, `attempt`, `seq`, `key`, `path`, `env?` | admit the native node `path` of this run's generation; `env` adds string variables to its exec and work context | +| daemon → host | `resolved` | `step`, `attempt`, `seq`, `key`, `exit` | the released node ended; its result is journaled under `key` | + +`HostContextV1` gains `release(request: { key, path, env? }): Promise`. Success is +`{ step_run, status: 'completed', attempt, products?: Record }`; failure is +`{ _tag: 'StepFailed', step_run, status: 'failed' | 'cancelled', attempt, reason }`, written when +the node is terminal and no native retry attempt remains. + +- The journal key of a released node is `step:PATH`, its `program.step` entry names the step run in + `child`. A release is idempotent per generation: releasing a node that already ran (a resumed + attempt, a successor generation that carried it forward) resolves at once from its native state. +- The engine learns a node's end from its store transition, as for child runs, and requests an + evaluation of the run on each release, so admission does not wait for a reconciler pass. +- In an orchestrated mission the program step never carries forward into a successor generation: a + reopened or revised run starts the program again, which re-releases every node; carried-forward + nodes resolve at once and only the rest run. +- When a released node fails finally, the reconciler ends the normal phase and cancels the program + step with the other unfinished normal work ("another step failed"), exactly as for any normal + step, so the reopen rules (`work retry`, revision of a failed run) see the same causes as for the + native rendering. +- Nodes the program never releases are settled exactly as the reconciler settles nodes it never + admits; the parity oracle (§9) pins that per dependency state. +- `missions show`, the run view and `missions waiting-on` fold `step:` keys into the native step + rows they name, so a released step is shown once. + +### 3. The generated program + +The loader compiles each data-DSL mission into a module whose default export is +`Fractal.orchestrate(graph)`, bundled and pinned as any program mission +([§11 Publication](program-engine.md#11-program-api)). `graph` is the mission's normal-phase nodes +with their step dependencies and any in-body constructs (§5, §6); `Fractal.orchestrate` is the +interpreter in the `fractal` package. + +```ts +// what Fractal.orchestrate runs for each normal-phase node, all under one Effect.all +const node = (n: Node) => + Effect.gen(function* () { + const deps = yield* Effect.all(n.dependsOn.map((d) => Deferred.await(done[d.step]))) + const exit = n.dependsOn.every((d, i) => holds(d.state, deps[i])) + ? yield* Effect.exit(Fractal.release({ id: `step:${n.path}`, path: n.path })) + : notReleased // satisfies no dependency, as a never-admitted step stays pending + yield* Deferred.succeed(done[n.path], exit) + }) +yield* Effect.all(graph.nodes.map(node), { concurrency: 'unbounded', discard: true }) +``` + +- Its control flow reads only journaled resolutions and the run's inputs (no clock, randomness or + I/O), so replay reproduces it (§5 of the engine). +- Keys derive from step paths, never from source order or generated code positions, and every path + is unique, so concurrent releases never take `id#n` keys and a revision that adds or removes a + step does not move another step's key. +- The program completes when every node is resolved or not released; it fails only on defects + (bundle, host, refused journal entry, refused release). The run's outcome comes from the native + rules, never from the program's value. + +### 4. Construct mapping + +| Data DSL | KDL | Compiled | +|---|---|---| +| `Fractal.step(...)`, worker (`assignedTo`, `availableTo`, run-scoped agents, `freshContext`, `keepOpen`, `capacity`) | `step` with `assigned-to`… | released native step | +| `after: [a, b]` / `dependsOn: [completed(a)]`, `Fractal.sequence(a, b, c)` | `depends-on { step "a" "completed" }` | the node awaits `a`'s resolution, then releases; the native guard holds by construction | +| siblings without mutual dependencies | roots or disjoint `depends-on` | released concurrently (`Effect.all`, unbounded) | +| `failed(a)` / `terminal(a)` dependencies | `step "a" "failed"` / `"terminal"` | released only when the state holds, as the native guard admits it | +| graph predicate dependency, `baseline` | `depends-on { field … }`, `baseline` | native on the released step | +| agentless `exec` step + `exit_code is 0` gate | `exec` + `gate { field "exit_code" … }` | released native step; local execs inline (§5) | +| gates on a step (field, exists, has/lacks, empty, document, merged, ci-passed, exec, LLM) | `gate "NAME" { … }` | native on the released step: they wait on graph state or a worker's report, which the program does not hold | +| human gate on a worker step (approve or feedback) | `gate type="human"` | native: it is part of the worker's report loop | +| human gate on an agentless step (an approval checkpoint) | agentless step + `gate type="human"` | released native step; an in-body `Fractal.ask` with deadline and default after Q3 (§6) | +| field gate over an earlier step's typed product | `gate { field … "resource/…" … }` | native; a TS predicate over the product (`gate.field(h.output, v => …)`) is an in-body decision (§6) | +| products (`produces`, `Fractal.product`) | `produces { … }` | native; the resolution carries them, decoded with the product's Effect Schema for in-body use | +| inputs (one Effect Schema Struct; `Fractal.input`) | `input "NAME" kind="text"\|"resource"` | native inputs unchanged; the program receives the payload as a Struct of strings (a resource input is its pinned subject) | +| `retry { attempts; backoff }` | `retry { … }` | native attempts on the released step; resolution waits for the last | +| step `timeout`, mission `timeout` | `timeout=` | native; the program step's timeout is the mission's | +| `onFailure: h` | `finally` step plus a guard | unchanged native lowering: after a final failure the normal phase is over, so compensation is final-phase work that also runs on cancel; for an inline step (§5) it is `Effect.onError` with the failed key in scope | +| `finally` steps | `finally { … }` | native final phase | +| `completion` frontier | `completion { … }` | native | +| `usesMission` / `waitFor` (revision or produced mission), `producesMission`, nested `mission` | `uses-mission`, `produces-mission`, nested `mission` | native on the released step (already a linked child run) | +| `loop` | `loop { … }` | released native node; rounds stay native child missions | +| `after-run`, `terminal`/`pty`, `document`, declarations in steps (resources, docs, observers, subscriptions) | as rendered | native on the released step | +| mission header: `concurrentRuns`, `lanes`, `authority`, `schedule`, report settings, mission gates and products | as rendered | native | +| run-data side channels | files in the run directory | unchanged; typed step values exist only in-body (§5, §6) and in hand-written programs | + +### 5. Inline local execs + +An agentless step whose only work is an `exec` on the run's own member with `restart "never"`, whose +only gate is `exit_code is 0` on that exec, and that declares no products, declarations, documents, +baselines or graph predicate dependencies compiles to +`Fractal.step({ id: 'exec:ID' }, Fractal.command(argv, { env, cwd }))` failing typed on a non-zero +exit. `retry { attempts n; backoff }` becomes `Effect.retry` with `Schedule.recurs(n - 1)` and the +backoff (keys `exec:ID#2`…), and `timeout` a typed `StepTimedOut` failure that ends the command's +process group. + +The command acts as `mission-run/RUN` under the mission's `authority` block (R07), which may differ +from the actor of a native exec. A mission without an `authority` block keeps its execs native, so the +compile never changes who writes. Inline steps run again in a successor generation (T03); whether a +same-revision reopen carries their journal entries forward is Q4. + +### 6. Decisions over held values + +The capabilities the native graph cannot express, available only because the mission is a program: + +- **Predicates over typed products:** `Fractal.gate.field(review.output, (v) => v.verdict === 'GO')` + on a step is evaluated in-body on the resolved product after the step completes; false fails the + program with `GateFailed { step, gate }`, journaled under `gate:STEP/NAME`, which fails the run. +- **Approval checkpoints as asks:** an agentless step whose only gate is a human approve gate + compiles to `Fractal.ask({ id: 'ask:STEP/NAME', for: reviewer, title, request: { review } })`, + which adds deadlines and defaults (Q3). +- **Values into steps:** `release` carries `env`, so a compensation or follow-up step receives the + failed step's path and reason, or an inline step's value, without a run-directory side channel. + +### 7. Latency + +| Hop | Native graph | Released native step | In-body (inline exec, decision, ask) | +|---|---|---|---| +| step end → next step admitted | ~3.3 s measured (2 s pass spacing × 2 passes) | engine hop 0.01–0.02 s (spike) plus one run evaluation; target under 0.1 s, measured in slice 4 | 0.01–0.02 s (spike); R13 under 50 ms | +| run start → first step | one reconciler pass | one pass for the program step, then a release | as released | +| worker step | the agent's wake and work dominate | same | — | + +A routine landing measured 11.4 s on the native graph and 4.0 s as a program: the gain is the +in-body path. A graph of released worker steps gains the admission hop only, and a native +reconciler woken by events instead of pass spacing would gain most of that too. + +### 8. Published missions and state + +| State | Class | What happens | +|---|---|---| +| Mission revisions published from KDL by hand | authoritative | unchanged; native orchestration keeps running them (DQ7) | +| Revisions published from the data DSL before the cutover | authoritative | keep running natively until republished; a schedule pinned to `mission@revision` keeps starting that revision until re-pinned | +| Revisions published from the data DSL after the cutover | authoritative, additive | a new revision of the same mission ID (`orchestrated-by` plus the program step); new runs use the engine | +| Runs in flight at the cutover | authoritative | finish on their revision (R05); `missions revise RUN` cuts one over, carrying completed steps forward because the step definitions are identical | +| `program.step` journal entries (`step:`, `exec:`, `gate:`, `ask:` keys) | authoritative, additive | new keys under the existing claim kind; no reader arm | +| Run view, `waiting-on` and `missions show` projections | rebuildable from claims | rebuilt; no migration | +| Bundles | generated, content-addressed | a new hash is a new blob | +| Rendered KDL fixtures and goldens | generated | regenerated | + +No old record changes meaning, so no converter is needed: native orchestration is the reader for +revisions without `orchestrated-by`. Its deletion trigger is a query, not a date: no ready mission +and no active run whose revision lacks `orchestrated-by` on any member (DQ7). + +Rollback is the previous release of the `fractal` package: its rendering republishes the native +form as the next revision, and runs started on the orchestrated revision finish on it or are +revised back, again carrying completed steps forward. + +Upgrading the `fractal` package changes the interpreter inlined in every bundle, so it revises +every compiled mission once (§2 of the engine: the bundle pins the runtime library). In-flight runs +are unaffected; the cost is one revision per mission per package upgrade (Q5). + +### 9. Parity + +Parity means the same mission, rendered natively and compiled, produces the same **observable +run**: + +- per node: terminal status, attempt count, claimant, products (subject and fields), gate results, + person answers, child runs (mission, inputs, outcome); +- the happens-before order between node ends and starts; +- the run's phases, outcome and reasons, final-phase steps and faults; +- the effects of reopen (`work retry`), revision and cancellation. + +The trace excludes the program step and its `program.step` claims, timestamps (only order counts), +and run, generation and step-run identifiers (canonically renamed). Each in-body construct adds one +named normalization row (an inline exec's key for its native step and `exec/` subject, an ask's +answer for a human gate result); parity is checked after it. + +**Tier 1: oracle and properties (every PR, in `pnpm --filter fractal test`).** A TypeScript model of +the native admission and normal-phase rules, written from the reconciler +(`step_dependencies_hold`, `mission_completion_selected` and the normal-phase transition in +`crates/fractal-reconcile/src/reconcile.rs`), serves as the native substrate for the compiled +program running on the in-memory host (`src/program-host.fixture.ts`). Property tests generate +graphs from Effect Schema arbitraries: up to 12 nodes, random acyclic dependencies with +`completed`/`failed`/`terminal` states, worker, agentless and exec kinds, retry attempts, final +steps, completion frontiers and scripted outcomes per attempt. They assert that every release +satisfies its guard, that the released set and resolutions equal the oracle's, that the trace +equals the oracle's native trace, and that killing the host at a random journal entry and replaying +reproduces the same trace. + +**Tier 2: golden runs on a scratch daemon (fixtures per PR, seeded sample scheduled).** Each fixture +(`daily-review`, `pr-landing`, `upstream-repin`) and a fixed-seed sample of tier-1 graphs is +published twice to isolated scratch instances (`fractal/testing`'s `scratchDaemonLayer`): the +native rendering and the compiled one. A scripted worker claims and reports with scripted products +and a scripted person answers gates and asks. The normalized traces must be equal and match the +stored golden trace (`FRACTAL_UPDATE_GOLDENS=1` rewrites it). Scenario cases: a final failure and +`work retry` reopening the run; cancellation during a step with a final phase; a mission gate +failure; step and mission timeouts; a loop; daemon SIGKILL and host kill mid-run; an in-flight +native run revised to the compiled revision. The same lane validates the oracle: a tier-2 trace that +differs from the oracle's is an oracle bug before it is a compiler bug. + +**Rendering parity (every PR).** For every fixture and for a corpus of published data-DSL trees, the +compiled KDL equals the native KDL plus exactly the `orchestrated-by` node and the program step. + +## Slices + +Stacked pull requests of about 300 changed lines; each updates this document and the affected docs +to describe what it shipped. + +| # | Slice | Size | Base | +|---|---|---|---| +| 1 | **Declaration:** `orchestrated-by` grammar (parse, render, validation rules of §1), capability advertisement and refusal, step definition hashes independent of it, `schema.md` | ~250 | M1.1a on `main` | +| 2a | **Admission:** released nodes and the guard, `step:` journal entries with `child`, resolution from store transitions and at once for terminal or carried-forward nodes, run evaluation on release, no carry-forward of the program step, settling unreleased nodes, view folding | ~350 | 1 and M1.2 | +| 2b | **Protocol:** `release`/`resolved` in the host, `HostContextV1.release`, `Fractal.release` in `fractal/effect` | ~200 | 2a | +| 3a | **Interpreter and oracle:** `Fractal.orchestrate`, the tier-1 oracle and property tests on the in-memory host | ~350 | M1.5; parallel with 1–2 | +| 3b | **Compiler:** the loader compiles data-DSL missions (opt-in per tree) to orchestrated KDL plus bundle; rendering-parity test | ~250 | 1, 3a and M2 | +| 4 | **Golden harness:** tier 2 on scratch daemons, scenario cases, admission-hop and worker-memory measurements | ~400 | 2b, 3b | +| 5 | **Cutover:** the loader always compiles data-DSL missions (the opt-in and the native-only rendering path removed); docs; consumers republish a canary mission, then every tree | ~150 | 4 green and the owner's go | +| 6 | **Inline local execs** (§5) with their normalization row | ~250 | 5 | +| 7 | **Decisions over held values** (§6): typed product predicates, approval asks (after Q3), `env` into released steps | ~300 | 5 | + +Slices 1 and 3a start at once; 2a/2b and 3b follow; 4 joins both lines; 6 and 7 are independent +after 5. + +## Risks + +- **The host becomes the orchestration path for every data-DSL run.** A host outage stops admission + for all of them (T02); running native steps go on, and DQ8 replaces a silent host. Native-only + missions never depended on the host. +- **A Bun worker per run for the run's whole life.** Data-DSL runs wait hours on agents, and a + waiting program stays loaded (§11 of the engine). Slice 4 measures worker memory per run; if + concurrent runs exceed the member's budget, evicting waiting programs and replaying them on + `resolved` becomes a slice before 5 (Q2). +- **Native rules the oracle misreads.** Unreleased nodes, `failed`/`terminal` dependencies under the + first-failure-ends-normal-work rule, loops and reopen are edges the tier-2 lane must cover before + the cutover; a mismatch blocks slice 5. +- **Two orchestrators until DQ7.** The reconciler keeps admitting hand-written KDL missions, so + admission rules change in two places until native orchestration retires. +- **Revision churn on package upgrades** (§8, Q5). +- **Authority enforcement and inline execs.** Under enforced run authority an inline exec that writes + outside the mission's `authority` block is refused where its native form was allowed; §5's rule and + the golden harness catch it before it ships. + +## Open questions + +- **Q1 Released native steps.** This document refines §9 of the engine: worker steps stay native + steps the program releases, instead of worker turns as child runs. Accept? +- **Q2 Waiting programs.** Evict a program whose fibers all wait on releases, asks or child runs, and + replay it on `resolved`; build it only if slice 4's measurement requires it, or before the + cutover regardless? +- **Q3 Approval checkpoints.** Compile an agentless human approve gate to `Fractal.ask` (deadlines, + defaults, one person-facing mechanism; the person sees an ask instead of a gate review), or keep + the native gate and offer `Fractal.ask` only as a new construct? +- **Q4 Inline journals across a reopen.** For a same-revision successor generation, carry the + completed entries of inline keys forward (the native graph carries completed execs forward), for + compiled programs only or for every program (which changes §5 of the engine)? +- **Q5 Interpreter versioning.** Accept one revision per compiled mission per `fractal` package + upgrade, or ship the interpreter with the daemon and give up R04 for orchestration code? +- **Q6 DQ7.** Retire native admission once no ready mission or active run lacks `orchestrated-by`, + converting hand-written KDL missions by publishing them through the data DSL? +- **Q7 Releases in hand-written programs.** Expose `Fractal.release` to authors, so a program can use + native worker steps (shared run-scoped agents, native gates) without a child run per worker turn? diff --git a/docs/st3/program-engine.md b/docs/st3/program-engine.md index a7e2526b377..0149efc7a49 100644 --- a/docs/st3/program-engine.md +++ b/docs/st3/program-engine.md @@ -287,21 +287,12 @@ Choice: a child run. It needs no standing owner and reuses §6. ### 9. One execution model -The data DSL lowers into a program instead of a step graph. A data-DSL mission becomes a generated -program body whose bundle the publisher pins; the KDL a person reads stays a projection. - -| Data DSL construct | Lowers to | -|---|---| -| `depends-on { step "x" completed }` | `yield*` order | -| sibling steps without mutual dependencies | `Effect.all([...], { concurrency: 'unbounded' })` | -| agentless `exec` step + `exit_code is 0` gate | `Fractal.step({ id }, Fractal.command(argv))` failing on a non-zero exit | -| `assigned-to` worker step, products | worker turn (§6); products are its success schema | -| field / exists / exec gate | a step that reads through the host contract and fails typed | -| human gate | `Fractal.ask` | -| `retry { attempts n }` | `Effect.retry(Schedule.recurs(n - 1))` | -| `finally { revert }` | `Effect.onError` / `catchTag` with the failed key in scope | -| `uses-mission` | `Fractal.run` (§6) | -| run-data side channels | step values | +The data DSL compiles into programs ([data-DSL missions as programs](program-engine-data-dsl.md)). +A data-DSL mission becomes an orchestrated mission: its native steps as the data DSL renders them, +plus one generated `program` step that decides when each of them starts. The native step graph +stays the substrate for worker claims, gates, products, retries and the final phase; ordering, +parallelism (`Effect.all`), local execs and decisions over step values run in the program body. +The KDL a person reads is the native rendering plus `orchestrated-by` and the program step. Native graph execution remains for published KDL missions until each is republished from the DSL. The `Fractal.steps.*` catalogue (classify, check, merge, publish, verify, revert, review) is one set @@ -369,7 +360,7 @@ on a scratch instance at each. |---|---| | M1 Engine | `program` steps and the `program.step` journal (§5); the supervised host with a worker per program, replay, eager cancellation, recorded effect process groups (DQ2) and liveness pings (DQ8); the step definition names its bundle by `bundle_hash` and publishing the declaration stores the blob, so neither the field nor definition hashes change in later milestones (§4); linked child runs with cancellation cascade and serialized sections (§6, §7); the `effect/workflow` binding (§11); mission-run authority (§3); inner steps in the run view and `waiting-on`, and the `program/stalled` and `program/host` faults (§8). | | M2 Publication | `fractal apply` carries a program mission's bundle to the daemon it publishes on and validates it (§4); the `fractal` package's loader renders `Fractal.mission(intent, program)` to a bundle and its KDL declaration. Factory's landing mission becomes the first production program. | -| M3 One model | The data DSL compiles into programs (§9), with byte and behaviour parity for existing data-DSL missions; published KDL missions keep running on the native graph until they are republished. | +| M3 One model | The data DSL compiles into programs (§9, [data-DSL missions as programs](program-engine-data-dsl.md)), with byte and behaviour parity for existing data-DSL missions; published KDL missions keep running on the native graph until they are republished. | ## Spike results (scratch, macOS, load 80–280) diff --git a/release-notes/program-engine-data-dsl-design.json b/release-notes/program-engine-data-dsl-design.json new file mode 100644 index 00000000000..4c59cd5586b --- /dev/null +++ b/release-notes/program-engine-data-dsl-design.json @@ -0,0 +1,11 @@ +{ + "version": 1, + "summary": "Documentation only: the proposed design for compiling data-DSL missions into programs (docs/st3/program-engine-data-dsl.md), with its parity tests and slices", + "replay": {"status": "none", "detail": "Documentation only; no claim, projection or replay change."}, + "schema": {"status": "none", "detail": "Documentation only; no table, index or migration change."}, + "rules": {"status": "none", "detail": "Documentation only; no rule or authority change."}, + "compatibility": {"status": "none", "detail": "Documentation only; no client, wire, CLI or behaviour change."}, + "downtime": {"status": "none", "detail": "No restart or migration."}, + "manual": {"status": "none", "detail": "No operator action is required."}, + "recovery": {"status": "none", "detail": "Revert the commit."} +} From ea58e9382892370ca0bf22ce6c10abeb4fdb6d7b Mon Sep 17 00:00:00 2001 From: Johannes Schickling Date: Tue, 10 Nov 2026 22:52:51 +0100 Subject: [PATCH 2/3] docs(st3): resolve review of the data-DSL compilation design - the orchestrator is not normal work for frontiers, exhaustion and the outcome - in-body constructs keep their nodes (settled-by, program gates), so dependents, frontiers and onFailure guards never dangle and parity names each difference - explicit released set: top-level declared nodes only; after-run, nested and round steps, the orchestrator and finally steps are admitted natively - capability advertised only with admission and release; compiler based on 2b - native agentless handoff figures; reserved orchestrator ID; tier-2 scenarios for frontier+finally, --after, nested, settled steps and program gates --- docs/st3/program-engine-data-dsl.md | 423 +++++++++++++++++----------- docs/st3/program-engine.md | 9 +- 2 files changed, 267 insertions(+), 165 deletions(-) diff --git a/docs/st3/program-engine-data-dsl.md b/docs/st3/program-engine-data-dsl.md index d9e0701e4f1..6e3fb5e43b2 100644 --- a/docs/st3/program-engine-data-dsl.md +++ b/docs/st3/program-engine-data-dsl.md @@ -13,13 +13,13 @@ how parity with the native graph is proved, and the slices that ship it. ## The shape A data-DSL mission compiles into an **orchestrated mission**: its native steps exactly as the data -DSL renders them, one generated `program` step, and a mission-level `orchestrated-by` naming that -step. The program decides when each native step starts; the reconciler runs everything else as it -does for any mission. +DSL renders them, one generated `program` step named `orchestrator`, and a mission-level +`orchestrated-by` naming that step. The program decides when each native step starts; the +reconciler runs everything else as it does for any mission. ```kdl mission "docs/review" report-to="agent/docs/editor" state="ready" timeout="2h" { - orchestrated-by "program" + orchestrated-by "orchestrator" input "change" kind="text" goal "Review one documentation change." completion { @@ -44,20 +44,22 @@ mission "docs/review" report-to="agent/docs/editor" state="ready" timeout="2h" { field "state" "resource/mission-run/${ST_MISSION_RUN}/draft" "is" "approved" } } - step "program" timeout="2h" { + step "orchestrator" timeout="2h" { program "/bundles/.mjs" sha256="" } } ``` -Everything except `orchestrated-by` and `step "program"` is byte-identical to the native rendering -of the same mission, so the KDL a person reads, the step definitions and their hashes do not change. +Apart from `orchestrated-by` and `step "orchestrator"`, the compiled KDL is byte-identical to the +native rendering of the same mission, so the KDL a person reads, the step definitions and their +hashes do not change. The only exceptions are the in-body constructs of §5 and §6. Each keeps its +node and step ID and changes only its own body, in the one way that section names. | Owned by the program (orchestration) | Owned by the native graph (substrate) | |---|---| -| when each normal-phase step or loop starts (its release) | the step lifecycle: baselines, selector, readiness, declarations, worker claim and report, nested and used missions, products, gates, retry attempts, timeouts | -| parallelism between released steps | the normal-phase transition: the first final failure ends normal work and cancels the rest ("another step failed") | -| constructs that run inside the body: local execs (§5), decisions over held values (§6) | mission products and gates, the completion frontier, the final phase, the run outcome | +| when each released node starts (§1) | the step lifecycle: baselines, selector, readiness, declarations, worker claim and report, nested and used missions, products, gates, retry attempts, timeouts | +| parallelism between released nodes | the normal-phase transition: the first final failure ends normal work and cancels the rest ("another step failed") | +| the outcome of settled steps (§5, §6) and the result of program gates (§6) | mission products and gates, the completion frontier, the final phase, the run outcome | | | reopening a failed run, revisions, cancellation | ## Options considered @@ -70,35 +72,48 @@ of the same mission, so the KDL a person reads, the step definitions and their h | `missions show`, step-run subjects, work claims | child runs instead of steps | unchanged plus one program step | as B | | In-flight run cut over by revision | impossible (no shared step definitions) | completed steps carry forward (identical definitions) | as B | | Bundle per mission revision | yes | yes | no, but the interpreter then changes with the daemon, against R04 | -| Engine work | a graph-read and wait API for gates | release and its admission rule (§2) | B plus a declaration read in the host contract | +| Engine work | a graph-read and wait API for gates | release and its admission rules (§1, §2) | B plus a declaration read in the host contract | B keeps native semantics wherever the program has nothing to decide, so parity holds by construction for those parts and the parity tests concentrate on admission. ## Spec -### 1. Declaration - -- `orchestrated-by "STEP"` on a mission names a normal-phase step that declares `program` and has - no `depends-on`. The mission must have no other `program` step. -- Every other normal-phase node (step or loop) of an orchestrated mission is **released**: the - reconciler admits it only after the program releases it. Final-phase steps (`finally`) are never - released; the final phase runs natively after the normal phase, also when the run is cancelled, - which a program cannot outlive. +### 1. Declaration and the released set + +- `orchestrated-by "STEP"` on a mission names a top-level normal-phase step that declares `program` + and no authored `depends-on`. The mission must have no other `program` step. A run started with + `--after RUN` adds its `after-run` dependency to the orchestrator as to every root step, so the + program starts once that run completes. +- The loader always names the step `orchestrator`. It refuses a data-DSL mission that declares a + step with that ID (`orchestrator-step-exists`), as a run started with `--after` refuses a mission + that declares its own `after-run` step. +- **Released nodes** are exactly the top-level normal-phase nodes the mission declares in `steps` + (steps and loops), except the orchestrator. The reconciler admits a released node only after the + program releases it. +- **Admitted natively** in an orchestrated mission, as in any mission: + - the orchestrator; + - final-phase (`finally`) steps; + - the synthetic `after-run` step of a run started with `--after RUN`; + - steps of a nested `mission`, admitted under their released parent step; + - the steps of loop rounds, which run in each round's child mission under the released loop. - A released node's `depends-on` stays declared and keeps its meaning as a guard: the daemon refuses a release whose step dependencies do not hold, and the program fails with that defect. Graph predicate dependencies and baselines are evaluated natively after release, as in any mission. -- `orchestrated-by` is outside every step definition, so the step definition hashes of an - orchestrated mission equal those of its native rendering. -- A member that has not advertised the orchestrated-mission capability refuses the declaration at - apply, as `mission_program=1` fences `program` steps. This is also the protocol fence for §2: a - bundle that releases steps never runs on an engine without `release`. +- `orchestrated-by` is outside every step definition. Every released node's definition hash is + therefore its native one, except for the settled steps and program gates of §5 and §6. +- A member advertises the orchestrated-mission capability only once it runs both the admission rules + (§2) and the release protocol (§2). Until then every member refuses `orchestrated-by` at apply, + as `mission_program=1` fences `program` steps, so a bundle that releases steps never runs on an + engine without `release`. The in-body constructs raise the capability level: `settled-by` (§5) + needs level 2 and a program gate (§6) needs level 3. A member below the level refuses the + declaration. -### 2. Release (host protocol and HostContext) +### 2. Release, resolution and the run outcome | Direction | Message | Fields | Meaning | |---|---|---|---| -| host → daemon | `release` | `step`, `attempt`, `seq`, `key`, `path`, `env?` | admit the native node `path` of this run's generation; `env` adds string variables to its exec and work context | +| host → daemon | `release` | `step`, `attempt`, `seq`, `key`, `path`, `env?` | admit the released node `path` of this run's generation; `env` adds string variables to its exec and work context | | daemon → host | `resolved` | `step`, `attempt`, `seq`, `key`, `exit` | the released node ended; its result is journaled under `key` | `HostContextV1` gains `release(request: { key, path, env? }): Promise`. Success is @@ -106,33 +121,48 @@ construction for those parts and the parity tests concentrate on admission. `{ _tag: 'StepFailed', step_run, status: 'failed' | 'cancelled', attempt, reason }`, written when the node is terminal and no native retry attempt remains. -- The journal key of a released node is `step:PATH`, its `program.step` entry names the step run in - `child`. A release is idempotent per generation: releasing a node that already ran (a resumed - attempt, a successor generation that carried it forward) resolves at once from its native state. -- The engine learns a node's end from its store transition, as for child runs, and requests an +**Release.** +- A released node's journal key is `step:PATH`, and its `program.step` entry names the step run in + `child`. +- A release is idempotent per generation. Releasing a node that already ran (in a resumed attempt, + or carried forward into a successor generation) resolves at once from its native state. +- The engine learns that a node ended from its store transition, as for child runs. It requests an evaluation of the run on each release, so admission does not wait for a reconciler pass. -- In an orchestrated mission the program step never carries forward into a successor generation: a - reopened or revised run starts the program again, which re-releases every node; carried-forward - nodes resolve at once and only the rest run. -- When a released node fails finally, the reconciler ends the normal phase and cancels the program - step with the other unfinished normal work ("another step failed"), exactly as for any normal - step, so the reopen rules (`work retry`, revision of a failed run) see the same causes as for the - native rendering. -- Nodes the program never releases are settled exactly as the reconciler settles nodes it never - admits; the parity oracle (§9) pins that per dependency state. -- `missions show`, the run view and `missions waiting-on` fold `step:` keys into the native step - rows they name, so a released step is shown once. +- The orchestrator never carries forward into a successor generation. A reopened or revised run + starts the program again, which re-releases every node; carried-forward nodes resolve at once and + only the rest run. +- A node the program never releases is settled exactly as the reconciler settles a node it never + admits, by the same rules for each dependency state. The parity oracle (§9) pins this down. + +**The orchestrator is not normal work for the run's outcome.** +- `all-steps-exhausted` and a `completion { depends-on … }` frontier are evaluated without the + orchestrator. +- When the normal phase ends, the reconciler settles the orchestrator and the engine cancels its + attempt on the host: + - `completed` with "the normal phase completed" when normal work completed; + - `cancelled` with "another step failed" when it did not. +- A cancelled orchestrator never makes a run failed. A mission whose frontier holds while the + orchestrator is still running therefore completes, also when it has `finally` steps, exactly as + its native rendering does. +- Only the orchestrator's own failure fails the run, as a failed normal step does. That failure is + always a defect: a bundle, host or protocol fault, a refused journal entry or a refused release. +- When a released node fails finally, the reconciler ends the normal phase as it does for any normal + step. The reopen rules (`work retry`, revision of a failed run) therefore see the same causes as + for the native rendering. + +**Visibility.** `missions show`, the run view and `missions waiting-on` fold `step:`, `exec:`, +`ask:` and `gate:` keys into the native step rows they name, so a released step is shown once. ### 3. The generated program The loader compiles each data-DSL mission into a module whose default export is -`Fractal.orchestrate(graph)`, bundled and pinned as any program mission -([§11 Publication](program-engine.md#11-program-api)). `graph` is the mission's normal-phase nodes -with their step dependencies and any in-body constructs (§5, §6); `Fractal.orchestrate` is the -interpreter in the `fractal` package. +`Fractal.orchestrate(graph)`, bundled and pinned like any program mission +([§11 Publication](program-engine.md#11-program-api)). `graph` holds the released nodes with their +step dependencies, the work of each settled step (§5, §6) and each program gate's predicate (§6). +`Fractal.orchestrate` is the interpreter in the `fractal` package. ```ts -// what Fractal.orchestrate runs for each normal-phase node, all under one Effect.all +// what Fractal.orchestrate runs for each released node, all under one Effect.all const node = (n: Node) => Effect.gen(function* () { const deps = yield* Effect.all(n.dependsOn.map((d) => Deferred.await(done[d.step]))) @@ -144,14 +174,19 @@ const node = (n: Node) => yield* Effect.all(graph.nodes.map(node), { concurrency: 'unbounded', discard: true }) ``` -- Its control flow reads only journaled resolutions and the run's inputs (no clock, randomness or - I/O), so replay reproduces it (§5 of the engine). -- Keys derive from step paths, never from source order or generated code positions, and every path - is unique, so concurrent releases never take `id#n` keys and a revision that adds or removes a - step does not move another step's key. -- The program completes when every node is resolved or not released; it fails only on defects - (bundle, host, refused journal entry, refused release). The run's outcome comes from the native - rules, never from the program's value. +A settled step's fiber releases the node, runs its work, settles it (§5) and then awaits its +resolution. Program gates are answered by a handler that the interpreter installs for the whole +attempt (§6). + +- Its control flow reads only journaled resolutions, journaled in-body results and the run's + inputs (no clock, randomness or I/O), so replay reproduces it (§5 of the engine). +- Keys derive from step paths and gate names, never from source order or generated code + positions. Every path is unique, so concurrent fibers never take `id#n` keys, and a revision + that adds or removes a step does not move another step's key. Only the retry of an inline exec + repeats a key (`exec:PATH#2`…), inside one fiber. +- The program ends when every node is resolved or not released, or when the reconciler settles it + at the end of the normal phase (§2). The run's outcome comes from the native rules, never from + the program's value. ### 4. Construct mapping @@ -162,63 +197,103 @@ yield* Effect.all(graph.nodes.map(node), { concurrency: 'unbounded', discard: tr | siblings without mutual dependencies | roots or disjoint `depends-on` | released concurrently (`Effect.all`, unbounded) | | `failed(a)` / `terminal(a)` dependencies | `step "a" "failed"` / `"terminal"` | released only when the state holds, as the native guard admits it | | graph predicate dependency, `baseline` | `depends-on { field … }`, `baseline` | native on the released step | -| agentless `exec` step + `exit_code is 0` gate | `exec` + `gate { field "exit_code" … }` | released native step; local execs inline (§5) | +| agentless `exec` step + `exit_code is 0` gate | `exec` + `gate { field "exit_code" … }` | released native step; a local exec settled by the program (§5) | | gates on a step (field, exists, has/lacks, empty, document, merged, ci-passed, exec, LLM) | `gate "NAME" { … }` | native on the released step: they wait on graph state or a worker's report, which the program does not hold | | human gate on a worker step (approve or feedback) | `gate type="human"` | native: it is part of the worker's report loop | -| human gate on an agentless step (an approval checkpoint) | agentless step + `gate type="human"` | released native step; an in-body `Fractal.ask` with deadline and default after Q3 (§6) | -| field gate over an earlier step's typed product | `gate { field … "resource/…" … }` | native; a TS predicate over the product (`gate.field(h.output, v => …)`) is an in-body decision (§6) | +| human gate on an agentless step (an approval checkpoint) | agentless step + `gate type="human"` | released native step; with Q3, a step settled by `Fractal.ask` (§6) | +| field gate over an earlier step's product | `gate { field … "resource/…" … }` | native | +| TS predicate over a step's typed product (`Fractal.gate.field(h.output, v => …)`) | `gate "NAME" { program }` | a program gate on the step (§6); it has no native form | | products (`produces`, `Fractal.product`) | `produces { … }` | native; the resolution carries them, decoded with the product's Effect Schema for in-body use | | inputs (one Effect Schema Struct; `Fractal.input`) | `input "NAME" kind="text"\|"resource"` | native inputs unchanged; the program receives the payload as a Struct of strings (a resource input is its pinned subject) | -| `retry { attempts; backoff }` | `retry { … }` | native attempts on the released step; resolution waits for the last | -| step `timeout`, mission `timeout` | `timeout=` | native; the program step's timeout is the mission's | -| `onFailure: h` | `finally` step plus a guard | unchanged native lowering: after a final failure the normal phase is over, so compensation is final-phase work that also runs on cancel; for an inline step (§5) it is `Effect.onError` with the failed key in scope | +| `retry { attempts; backoff }` | `retry { … }` | native attempts on the released step, and resolution waits for the last; in-body for a settled exec (§5) | +| step `timeout`, mission `timeout` | `timeout=` | native, settled steps included; the orchestrator's timeout is the mission's | +| `onFailure: h` | `finally` step plus a guard | unchanged native lowering, also on a settled step, which keeps its node: compensation is final-phase work that also runs on cancel | | `finally` steps | `finally { … }` | native final phase | -| `completion` frontier | `completion { … }` | native | -| `usesMission` / `waitFor` (revision or produced mission), `producesMission`, nested `mission` | `uses-mission`, `produces-mission`, nested `mission` | native on the released step (already a linked child run) | +| `completion` (all-steps-exhausted or a frontier) | `completion { … }` | native, evaluated without the orchestrator (§2) | +| `usesMission` / `waitFor` (revision or produced mission), `producesMission`, nested `mission` | `uses-mission`, `produces-mission`, nested `mission` | native on the released step (already a linked child run); nested steps admitted natively (§1) | | `loop` | `loop { … }` | released native node; rounds stay native child missions | | `after-run`, `terminal`/`pty`, `document`, declarations in steps (resources, docs, observers, subscriptions) | as rendered | native on the released step | +| a run started with `--after RUN` | synthetic `after-run` step | admitted natively; the orchestrator waits for it (§1) | | mission header: `concurrentRuns`, `lanes`, `authority`, `schedule`, report settings, mission gates and products | as rendered | native | -| run-data side channels | files in the run directory | unchanged; typed step values exist only in-body (§5, §6) and in hand-written programs | +| run-data side channels | files in the run directory | unchanged; `env` on release (§6) replaces them where a step needs a value | + +### 5. Settled local execs + +An agentless step is settled by the program when all of these hold: -### 5. Inline local execs +- its only work is an `exec` on the run's own member with `restart "never"`; +- its only gate is `exit_code is 0` on that exec; +- it declares no products, declarations, documents, baselines or graph-predicate dependencies; +- its mission declares an `authority` block. -An agentless step whose only work is an `exec` on the run's own member with `restart "never"`, whose -only gate is `exit_code is 0` on that exec, and that declares no products, declarations, documents, -baselines or graph predicate dependencies compiles to -`Fractal.step({ id: 'exec:ID' }, Fractal.command(argv, { env, cwd }))` failing typed on a non-zero -exit. `retry { attempts n; backoff }` becomes `Effect.retry` with `Schedule.recurs(n - 1)` and the -backoff (keys `exec:ID#2`…), and `timeout` a typed `StepTimedOut` failure that ends the command's -process group. +The step keeps its node, its ID, its `depends-on`, its `timeout` and its place in the KDL. Its +`exec`, its gate and its `retry` are replaced by one `settled-by "program"`: + +```kdl +step "check" timeout="10m" { + depends-on { + step "classify" "completed" + } + settled-by "program" +} +``` -The command acts as `mission-run/RUN` under the mission's `authority` block (R07), which may differ -from the actor of a native exec. A mission without an `authority` block keeps its execs native, so the -compile never changes who writes. Inline steps run again in a successor generation (T03); whether a -same-revision reopen carries their journal entries forward is Q4. +| Direction | Message | Fields | Meaning | +|---|---|---|---| +| host → daemon | `settle` | `step`, `attempt`, `seq`, `key`, `path`, `exit` | set the released settled step `path` to completed (Success) or failed (Failure, with its reason) | + +- A released settled step is working until it is settled. The program runs + `Fractal.step({ id: 'exec:PATH' }, Fractal.command(argv, { env, cwd }))` and fails it typed on a + non-zero exit. `retry { attempts n; backoff }` becomes `Effect.retry` with + `Schedule.recurs(n - 1)` and the backoff (keys `exec:PATH#2`…). The program then sends `settle`, + and the daemon writes the step's state and acknowledges. +- The native step's `timeout` stays authoritative. When it fires, the step fails, the normal phase + ends, and the cancelled orchestrator ends the command's process group. +- Dependents, completion frontiers and the `finally` guard an `onFailure` lowers to all keep naming + the step, so no reference dangles. Their definitions are unchanged. +- The settled step's own definition differs from its native form. A native run revised onto the + compiled revision therefore runs it again, which is idempotent by T01. +- `work retry` on a failed settled step reopens the run as for any step. The new generation's + program releases it again and runs the command. A completed settled step carries forward like any + completed step, and its release resolves at once. +- The command acts as `mission-run/RUN` under the mission's `authority` block (R07), which may + differ from the actor of a native exec. A mission without an `authority` block keeps its execs + native, so the compile never changes who writes. ### 6. Decisions over held values The capabilities the native graph cannot express, available only because the mission is a program: -- **Predicates over typed products:** `Fractal.gate.field(review.output, (v) => v.verdict === 'GO')` - on a step is evaluated in-body on the resolved product after the step completes; false fails the - program with `GateFailed { step, gate }`, journaled under `gate:STEP/NAME`, which fails the run. -- **Approval checkpoints as asks:** an agentless step whose only gate is a human approve gate - compiles to `Fractal.ask({ id: 'ask:STEP/NAME', for: reviewer, title, request: { review } })`, - which adds deadlines and defaults (Q3). -- **Values into steps:** `release` carries `env`, so a compensation or follow-up step receives the - failed step's path and reason, or an inline step's value, without a run-directory side channel. +- **Program gates.** `Fractal.gate.field(review.output, (v) => v.verdict === 'GO')` renders as + `gate "NAME" { program }` on the step. The reconciler evaluates it like any gate, after the + step's products hold: it asks the engine, which sends the host + `evaluate { step, attempt, seq, key, path, gate, step_attempt, products }`. The interpreter + decodes the products with the step's Effect Schemas, runs the predicate and journals the result + under `gate:PATH/NAME@STEP_ATTEMPT`. The reconciler reads that entry as the gate's result for + that attempt. A false predicate therefore fails the attempt like any failed gate: native `retry` + gives the worker another attempt, and `work retry` redoes the step. A replayed evaluation answers + from the journal. +- **Approval checkpoints as asks** (with Q3). An agentless step whose only gate is a human approve + gate keeps its node, which becomes `settled-by "program"`. The program asks + `Fractal.ask({ id: 'ask:PATH', for: reviewer, title, request: { review }, deadline, default })` + and settles the step completed on approval and failed on rejection. +- **Values into steps.** `release` carries `env`, so a compensation or follow-up step receives the + failed step's path and reason, or a settled step's value, without a run-directory side channel. ### 7. Latency -| Hop | Native graph | Released native step | In-body (inline exec, decision, ask) | +| Hop | Native graph | Released native step | In-body (settled exec, program gate, ask) | |---|---|---|---| -| step end → next step admitted | ~3.3 s measured (2 s pass spacing × 2 passes) | engine hop 0.01–0.02 s (spike) plus one run evaluation; target under 0.1 s, measured in slice 4 | 0.01–0.02 s (spike); R13 under 50 ms | -| run start → first step | one reconciler pass | one pass for the program step, then a release | as released | -| worker step | the agent's wake and work dominate | same | — | +| agentless step end → next agentless step started | ~0.2 s (p50 0.21 s, p95 0.32 s), because a run is evaluated again at once from an exec's exit | engine hop 0.01–0.02 s (spike) plus the run evaluation that admits the node; about the native figure, measured in slice 4 | 0.01–0.02 s (spike); R13 under 50 ms | +| step end → worker step claimed | admission as above, then the seat's wake, which dominates; not measured separately | same | — | +| run start → first step | one reconciler pass | one pass for the orchestrator, then a release | as released | -A routine landing measured 11.4 s on the native graph and 4.0 s as a program: the gain is the -in-body path. A graph of released worker steps gains the admission hop only, and a native -reconciler woken by events instead of pass spacing would gain most of that too. +Releasing native steps is about as fast as native admission, not faster. It is not what makes the +model fast. The gain is the in-body path: a settled exec skips exec declaration, convergence and +observation, and a program gate skips a reconciler round. Each still pays one settle or evaluate +hop. The landing comparison in the [engine design](program-engine.md#why) (11.4 s native, 4.0 s as +a program) was measured before a run was evaluated again from an exec's exit. With that change, the +native side of that comparison shrinks to about 0.2 s per agentless hop. ### 8. Published missions and state @@ -226,8 +301,8 @@ reconciler woken by events instead of pass spacing would gain most of that too. |---|---|---| | Mission revisions published from KDL by hand | authoritative | unchanged; native orchestration keeps running them (DQ7) | | Revisions published from the data DSL before the cutover | authoritative | keep running natively until republished; a schedule pinned to `mission@revision` keeps starting that revision until re-pinned | -| Revisions published from the data DSL after the cutover | authoritative, additive | a new revision of the same mission ID (`orchestrated-by` plus the program step); new runs use the engine | -| Runs in flight at the cutover | authoritative | finish on their revision (R05); `missions revise RUN` cuts one over, carrying completed steps forward because the step definitions are identical | +| Revisions published from the data DSL after the cutover | authoritative, additive | a new revision of the same mission ID (`orchestrated-by` plus the orchestrator); new runs use the engine | +| Runs in flight at the cutover | authoritative | finish on their revision (R05); `missions revise RUN` cuts one over, carrying completed steps forward because their definitions are identical (settled steps excepted, §5) | | `program.step` journal entries (`step:`, `exec:`, `gate:`, `ask:` keys) | authoritative, additive | new keys under the existing claim kind; no reader arm | | Run view, `waiting-on` and `missions show` projections | rebuildable from claims | rebuilt; no migration | | Bundles | generated, content-addressed | a new hash is a new blob | @@ -237,13 +312,13 @@ No old record changes meaning, so no converter is needed: native orchestration i revisions without `orchestrated-by`. Its deletion trigger is a query, not a date: no ready mission and no active run whose revision lacks `orchestrated-by` on any member (DQ7). -Rollback is the previous release of the `fractal` package: its rendering republishes the native -form as the next revision, and runs started on the orchestrated revision finish on it or are +Rollback is the previous release of the `fractal` package. Its rendering republishes the native +form as the next revision. Runs started on the orchestrated revision either finish on it or are revised back, again carrying completed steps forward. Upgrading the `fractal` package changes the interpreter inlined in every bundle, so it revises every compiled mission once (§2 of the engine: the bundle pins the runtime library). In-flight runs -are unaffected; the cost is one revision per mission per package upgrade (Q5). +are unaffected; the cost is one revision per mission per package upgrade (Q4). ### 9. Parity @@ -256,36 +331,65 @@ run**: - the run's phases, outcome and reasons, final-phase steps and faults; - the effects of reopen (`work retry`), revision and cancellation. -The trace excludes the program step and its `program.step` claims, timestamps (only order counts), -and run, generation and step-run identifiers (canonically renamed). Each in-body construct adds one -named normalization row (an inline exec's key for its native step and `exec/` subject, an ask's -answer for a human gate result); parity is checked after it. - -**Tier 1: oracle and properties (every PR, in `pnpm --filter fractal test`).** A TypeScript model of -the native admission and normal-phase rules, written from the reconciler -(`step_dependencies_hold`, `mission_completion_selected` and the normal-phase transition in -`crates/fractal-reconcile/src/reconcile.rs`), serves as the native substrate for the compiled -program running on the in-memory host (`src/program-host.fixture.ts`). Property tests generate -graphs from Effect Schema arbitraries: up to 12 nodes, random acyclic dependencies with -`completed`/`failed`/`terminal` states, worker, agentless and exec kinds, retry attempts, final -steps, completion frontiers and scripted outcomes per attempt. They assert that every release -satisfies its guard, that the released set and resolutions equal the oracle's, that the trace -equals the oracle's native trace, and that killing the host at a random journal entry and replaying -reproduces the same trace. - -**Tier 2: golden runs on a scratch daemon (fixtures per PR, seeded sample scheduled).** Each fixture -(`daily-review`, `pr-landing`, `upstream-repin`) and a fixed-seed sample of tier-1 graphs is -published twice to isolated scratch instances (`fractal/testing`'s `scratchDaemonLayer`): the -native rendering and the compiled one. A scripted worker claims and reports with scripted products -and a scripted person answers gates and asks. The normalized traces must be equal and match the -stored golden trace (`FRACTAL_UPDATE_GOLDENS=1` rewrites it). Scenario cases: a final failure and -`work retry` reopening the run; cancellation during a step with a final phase; a mission gate -failure; step and mission timeouts; a loop; daemon SIGKILL and host kill mid-run; an in-flight -native run revised to the compiled revision. The same lane validates the oracle: a tier-2 trace that -differs from the oracle's is an oracle bug before it is a compiler bug. - -**Rendering parity (every PR).** For every fixture and for a corpus of published data-DSL trees, the -compiled KDL equals the native KDL plus exactly the `orchestrated-by` node and the program step. +The trace excludes the orchestrator and its `program.step` claims, and it excludes timestamps (only +order counts). Run, generation and step-run identifiers are renamed canonically. Each in-body +construct has one named normalization row, and parity is checked after normalization: + +| Construct | Native trace | Compiled trace | Normalized as | +|---|---|---|---| +| settled exec (§5) | an `exec/` subject, the `exit_code` gate result, native attempts | `exec:PATH`, `exec:PATH#n` entries, one settle | the step's status and attempt count (n) | +| approval ask (§6) | a human gate review on the step | an `ask:PATH` answer and one settle | the step's status and the person's decision | +| program gate (§6) | none (no native form) | `gate:PATH/NAME@n` entries | golden trace only | + +**Tier 1: oracle and properties (every PR, in `pnpm --filter fractal test`).** +- **Oracle.** A TypeScript model of the native admission and normal-phase rules, written from the + reconciler, serves as the native substrate. It covers `step_dependencies_hold`, + `mission_completion_selected` and the normal-phase transition in + `crates/fractal-reconcile/src/reconcile.rs`, plus the orchestrator rules of §2. The compiled + program runs against it on the in-memory host (`src/program-host.fixture.ts`). +- **Generated graphs.** Property tests generate graphs from Effect Schema arbitraries: + - up to 12 nodes, with random acyclic dependencies in `completed`, `failed` and `terminal` states; + - worker, agentless, exec and settled kinds; + - retry attempts, final steps and completion frontiers; + - scripted outcomes per attempt. +- **Assertions.** + - Every release satisfies its guard. + - The released set and the resolutions equal the oracle's. + - The trace equals the oracle's native trace. + - Killing the host at a random journal entry and replaying reproduces the same trace. +- **Limit.** The oracle is also the substrate, so tier 1 proves only that the interpreter agrees + with the model. Bugs in how the program interacts with the reconciler show up only in tier 2. + +**Tier 2: golden runs on a scratch daemon (fixtures per PR, seeded sample scheduled).** +- **Setup.** Each fixture (`daily-review`, `pr-landing`, `upstream-repin`) and a fixed-seed sample + of tier-1 graphs is published twice to isolated scratch instances (`fractal/testing`'s + `scratchDaemonLayer`): once as the native rendering, once compiled. +- **Drivers.** A scripted worker claims and reports with scripted products, and a scripted person + answers gates and asks. +- **Check.** The normalized traces must be equal and match the stored golden trace + (`FRACTAL_UPDATE_GOLDENS=1` rewrites it). A tier-2 trace that differs from the oracle's is treated + as an oracle bug before it is treated as a compiler bug. +- **Scenarios.** Each scenario runs both forms: + - a final failure, then `work retry` reopening the run; + - a completion frontier that holds while later released nodes are unreleased, with a `finally` + step (the run completes, §2); + - a run started with `--after RUN`; + - a released step with a nested mission; + - a loop of several rounds; + - cancellation during a step with a final phase; + - a mission gate failure; + - step and mission timeouts, including the timeout of a settled step; + - a failed settled exec with a dependent step and an `onFailure` compensation; + - `work retry` of a settled step; + - a false program gate under native `retry`; + - daemon SIGKILL and host kill mid-run; + - an in-flight native run revised onto the compiled revision. + +**Rendering parity (every PR).** The test runs over every fixture and a corpus of published +data-DSL trees. The compiled KDL must equal the native KDL plus exactly two nodes, the +`orchestrated-by` node and the orchestrator. The only other allowed difference is each settled step +whose body is replaced by `settled-by "program"`. A mission with a program gate has no native +rendering, so its parity is golden-only. ## Slices @@ -294,54 +398,51 @@ to describe what it shipped. | # | Slice | Size | Base | |---|---|---|---| -| 1 | **Declaration:** `orchestrated-by` grammar (parse, render, validation rules of §1), capability advertisement and refusal, step definition hashes independent of it, `schema.md` | ~250 | M1.1a on `main` | -| 2a | **Admission:** released nodes and the guard, `step:` journal entries with `child`, resolution from store transitions and at once for terminal or carried-forward nodes, run evaluation on release, no carry-forward of the program step, settling unreleased nodes, view folding | ~350 | 1 and M1.2 | -| 2b | **Protocol:** `release`/`resolved` in the host, `HostContextV1.release`, `Fractal.release` in `fractal/effect` | ~200 | 2a | +| 1 | **Declaration:** `orchestrated-by` grammar (parse, render, the rules of §1), the reserved orchestrator ID, refusal at apply on every member, step definition hashes independent of it, `schema.md` | ~250 | M1.1a on `main` | +| 2a | **Admission:** the released set and the guard, the orchestrator outcome rules, `step:` entries with `child`, resolution from store transitions (at once for terminal or carried-forward nodes), run evaluation on release, settling unreleased nodes, view folding | ~350 | 1 and M1.2 | +| 2b | **Protocol and capability:** `release`/`resolved` in the host, `HostContextV1.release`, `Fractal.release` in `fractal/effect`; the member advertises the capability, so `orchestrated-by` is accepted from here on | ~200 | 2a | | 3a | **Interpreter and oracle:** `Fractal.orchestrate`, the tier-1 oracle and property tests on the in-memory host | ~350 | M1.5; parallel with 1–2 | -| 3b | **Compiler:** the loader compiles data-DSL missions (opt-in per tree) to orchestrated KDL plus bundle; rendering-parity test | ~250 | 1, 3a and M2 | -| 4 | **Golden harness:** tier 2 on scratch daemons, scenario cases, admission-hop and worker-memory measurements | ~400 | 2b, 3b | -| 5 | **Cutover:** the loader always compiles data-DSL missions (the opt-in and the native-only rendering path removed); docs; consumers republish a canary mission, then every tree | ~150 | 4 green and the owner's go | -| 6 | **Inline local execs** (§5) with their normalization row | ~250 | 5 | -| 7 | **Decisions over held values** (§6): typed product predicates, approval asks (after Q3), `env` into released steps | ~300 | 5 | +| 3b | **Compiler:** the loader compiles data-DSL missions (opt-in per tree) to orchestrated KDL plus bundle, refusing the reserved ID; rendering-parity test | ~250 | 2b, 3a and M2 | +| 4a | **Golden harness:** tier 2 on scratch daemons, the three fixtures and the seeded sample | ~250 | 3b | +| 4b | **Scenarios and measurements:** the scenario cases of §9 for released steps, admission-hop and worker-memory measurements | ~200 | 4a | +| 5 | **Cutover:** the loader always compiles data-DSL missions (the opt-in and the native-only rendering path removed); docs; consumers republish a canary mission, then every tree | ~150 | 4b green and the owner's go | +| 6 | **Settled local execs** (§5): `settled-by` grammar and admission, `settle`, capability level 2, compile, normalization row and their scenarios | ~300 | 5 | +| 7 | **Decisions over held values** (§6): program gates and `evaluate`, capability level 3, approval asks (with Q3), `env` into released steps, and their scenarios | ~300 | 5 | -Slices 1 and 3a start at once; 2a/2b and 3b follow; 4 joins both lines; 6 and 7 are independent -after 5. +Slices 1 and 3a start at once. 2a and 2b follow 1, and 3b joins them. 4a and 4b follow, then the +cutover. 6 and 7 are independent after 5. ## Risks - **The host becomes the orchestration path for every data-DSL run.** A host outage stops admission - for all of them (T02); running native steps go on, and DQ8 replaces a silent host. Native-only + for all of them (T02). Running native steps go on, and DQ8 replaces a silent host. Native-only missions never depended on the host. - **A Bun worker per run for the run's whole life.** Data-DSL runs wait hours on agents, and a - waiting program stays loaded (§11 of the engine). Slice 4 measures worker memory per run; if + waiting program stays loaded (§11 of the engine). Slice 4b measures worker memory per run. If concurrent runs exceed the member's budget, evicting waiting programs and replaying them on `resolved` becomes a slice before 5 (Q2). -- **Native rules the oracle misreads.** Unreleased nodes, `failed`/`terminal` dependencies under the - first-failure-ends-normal-work rule, loops and reopen are edges the tier-2 lane must cover before - the cutover; a mismatch blocks slice 5. +- **Native rules the oracle misreads.** These edges are listed in tier 2's scenarios, and a mismatch + blocks slice 5: + - unreleased nodes; + - `failed`/`terminal` dependencies under the rule that the first failure ends normal work; + - frontiers; + - `--after` runs; + - nested missions and loops; + - reopen. - **Two orchestrators until DQ7.** The reconciler keeps admitting hand-written KDL missions, so admission rules change in two places until native orchestration retires. -- **Revision churn on package upgrades** (§8, Q5). -- **Authority enforcement and inline execs.** Under enforced run authority an inline exec that writes - outside the mission's `authority` block is refused where its native form was allowed; §5's rule and - the golden harness catch it before it ships. +- **Revision churn on package upgrades** (§8, Q4). +- **Authority enforcement and settled execs.** Under enforced run authority, a settled exec that + writes outside the mission's `authority` block is refused where its native form was allowed. + §5's rule and the golden harness catch this before it ships. ## Open questions -- **Q1 Released native steps.** This document refines §9 of the engine: worker steps stay native - steps the program releases, instead of worker turns as child runs. Accept? -- **Q2 Waiting programs.** Evict a program whose fibers all wait on releases, asks or child runs, and - replay it on `resolved`; build it only if slice 4's measurement requires it, or before the - cutover regardless? -- **Q3 Approval checkpoints.** Compile an agentless human approve gate to `Fractal.ask` (deadlines, - defaults, one person-facing mechanism; the person sees an ask instead of a gate review), or keep - the native gate and offer `Fractal.ask` only as a new construct? -- **Q4 Inline journals across a reopen.** For a same-revision successor generation, carry the - completed entries of inline keys forward (the native graph carries completed execs forward), for - compiled programs only or for every program (which changes §5 of the engine)? -- **Q5 Interpreter versioning.** Accept one revision per compiled mission per `fractal` package - upgrade, or ship the interpreter with the daemon and give up R04 for orchestration code? -- **Q6 DQ7.** Retire native admission once no ready mission or active run lacks `orchestrated-by`, - converting hand-written KDL missions by publishing them through the data DSL? -- **Q7 Releases in hand-written programs.** Expose `Fractal.release` to authors, so a program can use - native worker steps (shared run-scoped agents, native gates) without a child run per worker turn? +| # | Question | Needed before | +|---|---|---| +| Q1 | **Released native steps.** This document refines §9 of the engine: worker steps stay native steps the program releases, instead of worker turns as child runs. Accept? Once answered, it is recorded as decided here and in the engine's decision table. | merging this design | +| Q2 | **Waiting programs.** Evict a program whose fibers all wait on releases, asks or child runs, and replay it on `resolved`. Build it only if slice 4b's measurement requires it, or before the cutover regardless? | slice 5 | +| Q3 | **Approval checkpoints.** Compile an agentless human approve gate to `Fractal.ask` (deadlines, defaults, one person-facing mechanism, but the person sees an ask instead of a gate review), or keep the native gate and offer `Fractal.ask` only as a new construct? | slice 7 | +| Q4 | **Interpreter versioning.** Accept one revision per compiled mission per `fractal` package upgrade, or ship the interpreter with the daemon and give up R04 for orchestration code? | slice 3b | +| Q5 | **DQ7.** Retire native admission once no ready mission or active run lacks `orchestrated-by`, converting hand-written KDL missions by publishing them through the data DSL? | after slice 5 | +| Q6 | **Releases in hand-written programs.** Expose `Fractal.release` to authors, so a program can use native worker steps (shared run-scoped agents, native gates) without a child run per worker turn? | after slice 5 | diff --git a/docs/st3/program-engine.md b/docs/st3/program-engine.md index 0149efc7a49..266f398e086 100644 --- a/docs/st3/program-engine.md +++ b/docs/st3/program-engine.md @@ -289,10 +289,11 @@ Choice: a child run. It needs no standing owner and reuses §6. The data DSL compiles into programs ([data-DSL missions as programs](program-engine-data-dsl.md)). A data-DSL mission becomes an orchestrated mission: its native steps as the data DSL renders them, -plus one generated `program` step that decides when each of them starts. The native step graph -stays the substrate for worker claims, gates, products, retries and the final phase; ordering, -parallelism (`Effect.all`), local execs and decisions over step values run in the program body. -The KDL a person reads is the native rendering plus `orchestrated-by` and the program step. +plus one generated `program` step, `orchestrator`, that decides when each of them starts and is not +normal work for the run's outcome. The native step graph stays the substrate for worker claims, +gates, products, retries, completion and the final phase; ordering, parallelism (`Effect.all`), +settled local execs and decisions over step values run in the program body. The KDL a person reads +is the native rendering plus `orchestrated-by` and the orchestrator. Native graph execution remains for published KDL missions until each is republished from the DSL. The `Fractal.steps.*` catalogue (classify, check, merge, publish, verify, revert, review) is one set From a448bb7c556067fba1f471331922f03fcec60dc9 Mon Sep 17 00:00:00 2001 From: Johannes Schickling Date: Tue, 10 Nov 2026 22:55:32 +0100 Subject: [PATCH 3/3] docs(st3): restrict in-body constructs to released nodes; revision reruns settled dependents - settled-by and gate { program } are refused (loader and apply) on final, nested and round steps, loop until and mission gates - revising a native run onto the compiled revision restarts each settled step and its transitive dependents; tier-2 scenario states the expected trace --- docs/st3/program-engine-data-dsl.md | 32 ++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/docs/st3/program-engine-data-dsl.md b/docs/st3/program-engine-data-dsl.md index 6e3fb5e43b2..711ac223870 100644 --- a/docs/st3/program-engine-data-dsl.md +++ b/docs/st3/program-engine-data-dsl.md @@ -108,6 +108,13 @@ construction for those parts and the parity tests concentrate on admission. engine without `release`. The in-body constructs raise the capability level: `settled-by` (§5) needs level 2 and a program gate (§6) needs level 3. A member below the level refuses the declaration. +- `settled-by "program"` and `gate { program }` are allowed only on released nodes, because only + there does an orchestrator attempt exist to answer them. The loader and `apply` both refuse + them on the following, where they would wait until the step or mission timed out: + - final-phase steps; + - steps of nested missions and loop rounds; + - a loop's `until` gates; + - mission-level gates. ### 2. Release, resolution and the run outcome @@ -251,8 +258,13 @@ step "check" timeout="10m" { ends, and the cancelled orchestrator ends the command's process group. - Dependents, completion frontiers and the `finally` guard an `onFailure` lowers to all keep naming the step, so no reference dangles. Their definitions are unchanged. -- The settled step's own definition differs from its native form. A native run revised onto the - compiled revision therefore runs it again, which is idempotent by T01. +- The settled step's own definition differs from its native form. Revising a native run onto the + compiled revision therefore restarts the settled step and every transitive dependent without + prior completion ([mission revisions](mission-graph-runtime.md#immutable-generations): a changed + step and every transitive dependent start again). In a landing-style mission whose settled + `check` precedes worker steps and `merge`, all of them run again; only steps that do not depend + on a settled step carry forward. Revise only runs whose settled steps have not completed, or + accept the rerun (idempotent by T01). - `work retry` on a failed settled step reopens the run as for any step. The new generation's program releases it again and runs the command. A completed settled step carries forward like any completed step, and its release resolves at once. @@ -272,7 +284,7 @@ The capabilities the native graph cannot express, available only because the mis under `gate:PATH/NAME@STEP_ATTEMPT`. The reconciler reads that entry as the gate's result for that attempt. A false predicate therefore fails the attempt like any failed gate: native `retry` gives the worker another attempt, and `work retry` redoes the step. A replayed evaluation answers - from the journal. + from the journal. A program gate is allowed only on a released node (§1). - **Approval checkpoints as asks** (with Q3). An agentless step whose only gate is a human approve gate keeps its node, which becomes `settled-by "program"`. The program asks `Fractal.ask({ id: 'ask:PATH', for: reviewer, title, request: { review }, deadline, default })` @@ -302,7 +314,7 @@ native side of that comparison shrinks to about 0.2 s per agentless hop. | Mission revisions published from KDL by hand | authoritative | unchanged; native orchestration keeps running them (DQ7) | | Revisions published from the data DSL before the cutover | authoritative | keep running natively until republished; a schedule pinned to `mission@revision` keeps starting that revision until re-pinned | | Revisions published from the data DSL after the cutover | authoritative, additive | a new revision of the same mission ID (`orchestrated-by` plus the orchestrator); new runs use the engine | -| Runs in flight at the cutover | authoritative | finish on their revision (R05); `missions revise RUN` cuts one over, carrying completed steps forward because their definitions are identical (settled steps excepted, §5) | +| Runs in flight at the cutover | authoritative | finish on their revision (R05); `missions revise RUN` cuts one over, carrying completed steps forward because their definitions are identical, except settled steps and their transitive dependents, which start again (§5) | | `program.step` journal entries (`step:`, `exec:`, `gate:`, `ask:` keys) | authoritative, additive | new keys under the existing claim kind; no reader arm | | Run view, `waiting-on` and `missions show` projections | rebuildable from claims | rebuilt; no migration | | Bundles | generated, content-addressed | a new hash is a new blob | @@ -383,7 +395,13 @@ construct has one named normalization row, and parity is checked after normaliza - `work retry` of a settled step; - a false program gate under native `retry`; - daemon SIGKILL and host kill mid-run; - - an in-flight native run revised onto the compiled revision. + - an in-flight native run revised onto the compiled revision, in two variants. Without settled + steps, every completed step carries forward. With a completed settled `check` before a + completed worker step and a pending `merge`, the expected trace is: `check`, the worker step + and `merge` start again in the successor generation; steps independent of `check` carry + forward; the outcome is the native rendering's for that order. +- **Refusals.** `settled-by` and `gate { program }` on a final, nested or round step, in a loop's + `until`, or as a mission gate are refused by the loader and by `apply` (§1). **Rendering parity (every PR).** The test runs over every fixture and a corpus of published data-DSL trees. The compiled KDL must equal the native KDL plus exactly two nodes, the @@ -406,8 +424,8 @@ to describe what it shipped. | 4a | **Golden harness:** tier 2 on scratch daemons, the three fixtures and the seeded sample | ~250 | 3b | | 4b | **Scenarios and measurements:** the scenario cases of §9 for released steps, admission-hop and worker-memory measurements | ~200 | 4a | | 5 | **Cutover:** the loader always compiles data-DSL missions (the opt-in and the native-only rendering path removed); docs; consumers republish a canary mission, then every tree | ~150 | 4b green and the owner's go | -| 6 | **Settled local execs** (§5): `settled-by` grammar and admission, `settle`, capability level 2, compile, normalization row and their scenarios | ~300 | 5 | -| 7 | **Decisions over held values** (§6): program gates and `evaluate`, capability level 3, approval asks (with Q3), `env` into released steps, and their scenarios | ~300 | 5 | +| 6 | **Settled local execs** (§5): `settled-by` grammar and admission, `settle`, capability level 2, compile, refusal outside released nodes, normalization row and their scenarios | ~300 | 5 | +| 7 | **Decisions over held values** (§6): program gates and `evaluate`, capability level 3, refusal outside released nodes (final, nested and round steps, loop `until`, mission gates) in the loader and at `apply`, approval asks (with Q3), `env` into released steps, and their scenarios | ~300 | 5 | Slices 1 and 3a start at once. 2a and 2b follow 1, and 3b joins them. 4a and 4b follow, then the cutover. 6 and 7 are independent after 5.