S2.6 — Skill execution harness: fixture repository, host runner, mutation recorder

Size: L. Set by the repository owner at the S2.2 design gate (#41), ruling
G4, 2026-09-06. iAI did not author that decision —
docs/design/03-workflow.md:271.

Why this is a Story and not a task inside S2.2

docs/milestones/M2.md:107-109 sets the verification for every skill Story as:

"Each skill runs twice against the same fixture repository and the second run
makes zero mutations."

Nothing in this repository can execute a skill. Measured at 9c15f04:

Searched	Result
every `Bun.spawn` / `spawnSync` in the tree	`git ls-files`, `tsc`, `git config`, and tests re-invoking the linters. No LLM, no host, no skill
`packages/adapter-claude`, `packages/adapter-opencode`	25 and 20 lines; pure `Decision` → exit-code/stdout mapping. No process launch
`packages/installer`	10 lines; `describeInstall` returns a string
`agents/`	empty
`.github/workflows/ci.yml`	six lint/test jobs plus `smoke-install`, which at `:161-165` prints that no real install smoke test is possible

packages/core/src/gh/* is pure argv construction, enforced by
no-io-in-pure-modules; it never executes anything. There is no fixture
repository, no host runner, no recording proxy and no dry-run mode.

So the milestone's verification sentence describes a capability that must be
built from nothing. docs/milestones/M2.md:105 lists it as half of one
indicative task row inside S2.2 — a Story about writing four markdown files.
That is a three-build-target project wearing a task's clothing, and none of the
three is a planning skill.

Build targets

#	Target	Work
1	A fixture repository	a disposable git tree plus a GitHub state fixture a skill can be run against, and torn down. Must be reproducible in CI, where no network and no private repo are available
2	A host runner	something that actually executes a `SKILL.md` through a host. This is where the real cost is, and where the design question is: a live model call is non-deterministic and costs money per CI run; a recorded transcript is deterministic but proves only that the recording is stable
3	A mutation recorder	capture every mutating command a run issues, so "the second run makes zero mutations" is a diff over a recorded command list rather than a claim

The design question this Story must answer first

A live-model harness and a replay harness verify different things, and only
one of them verifies idempotency.

A replayed transcript proves the recording is stable. It cannot detect that a
skill would issue a second gh issue create on a real second run, because the
recording never made the first one. A live harness can, and is non-deterministic,
rate-limited and costed per run.

The honest answer may be both, at different rungs. That is the first thing to
decide, and it should be decided before any code, because it determines all
three build targets above.

What depends on this

S2.2, S2.3, S2.4 and S2.5 all defer their idempotency verification here.
S2.2's CLAIM-41.8 is restated to a stated re-entry contract, verified
model-judged by review — honest about what it is, per
references/verification.md:45-46, rather than a stronger-sounding class than
the check supports. The other three Stories will need the same restatement until
this lands.

Sequencing note. This Story is most valuable after at least one skill
exists to run, so it is not a blocker on S2.2 — it is the reason S2.2's claim is
model-judged instead of false.

Done when

The live-versus-replay question is decided and recorded
The three build targets exist, with a denominator: the harness reports how many
skills it ran, and something asserts that number is non-zero
At least one real skill is run twice and the zero-mutation property is
measured, not asserted
The restated CLAIM-41.8 in S2.2, and its equivalents in S2.3-S2.5, are
either discharged or re-pointed here explicitly

Tasks

S2.6.1 — The adapter port, and the ARCHITECTURE.md build target #318 The adapter port, and the ARCHITECTURE.md build target
S2.6.2 — The fixture repository, and the synthetic goals fixture #319 The fixture repository, and the synthetic goals fixture
S2.6.3 — The stateful fake forge: able to lie, and non-enforcing #320 The stateful fake forge: able to lie, and non-enforcing
S2.6.4 — The two-surface mutation recorder #321 The two-surface mutation recorder
S2.6.5 — The anti-vacuity assertion set, asserted by the runner #322 The anti-vacuity assertion set, asserted by the runner
S2.6.6 — The seam rung and its CI job #323 The seam rung and its CI job
S2.6.7 — The live rung, never required #324 The live rung, never required
S2.6.8 — The M2.md and M4.md corrections, line-neutral #325 The M2.md and M4.md corrections, line-neutral

Eight tasks. docs/milestones/M2.md:105 allotted this Story half of one indicative task row. The Design measured ten build targets; these eight are that table, with the two smallest folded into their parents.

#318 is sequenced first by gate ruling G3 — the adapter port is owed to the architecture regardless, both rungs need it, and it is where the first-contact bill of six untested gh families lands.

The port in #318 is SHARED with #47's task-do (#49), per G3 here and G4 there, ruled in the same sitting. It must not be built twice.

#319 is blocked on #292 if a public goals fixture is wanted; otherwise it ships a declared-synthetic one. goal-create's real input is PRIVATE by fail-safe default and no public checkout can supply it.

#41: apply the gate ruling

#41: story-level verification evidence for S2.2

@dev-pmallapp

iai-design

Design: docs/design/stories/293.md — pinned at bc25b778bec99ab10278f1f87be6ca7df347f3ba (gate ruling applied), on story/293-s2-6-skill-execution-harness.

Eleven claims registered: CLAIM-293.1–.3, NEVER-293.4, NEVER-293.5, CLAIM-293.6, NEVER-293.7, CLAIM-293.8, CLAIM-293.9, NEVER-293.10, CLAIM-293.11. Ten build targets, six decisions, six open gate questions.

The issue's capability table, re-measured at `5884a40`

This issue's table was measured at 9c15f04. Three of five rows are stale.

Row	Status now
spawn sites	stale in count — 16 call sites in 11 files, not 12. S2.2 added four. The conclusion "No LLM, no host, no skill" is confirmed: the complete set of executables this repository invokes is `git`, `tsc`, `bun`, `gh`, `python`/`awk`/`grep`
adapters 25 / 20 lines	still true, verbatim
installer 10 lines	still true — plus the package declares a `bin` resolving to a module that only exports a function
`agents/` empty	still true
`ci.yml:161-165`	still true, line numbers unchanged
—	new: `skills/` is no longer empty. 4 directories, 4 `SKILL.md`, 508 lines. `skill-lint` reports 4 scanned, not zero

That last row retires an argument three source comments still make — two in test/lint.test.ts, one in packages/core/test/binding-isolation.test.ts. All three are now false and must not be cited.

What the Design concluded

1. The adapter port does not exist. Four contract documents assign execution to an adapter. No gh argv this repository has ever built has ever been executed; six operation families are tested against golden argv only. There is no single choke point to wrap — "gh" is argv[0] at 22 sites across 6 modules — but the type-level one already exists, so the recorder can be designed into the executor rather than retrofitted. Sequenced as build target 1.

2. "Live versus replay" is one axis of three. Idempotency is f(f(s)) = f(s) over repository state — a property of world and sequencing, not of the oracle. Decision 6 of docs/design/stories/41.md is precisely right about token replay against no world from a clean state, and says nothing about any oracle against a stateful world from run 1's state. The candidate set is eight, not two.

3. Removing the model buys a different claim about a different population. A model-free harness verifies a human's transcription of the SKILL.md, not the model reading it. This is the category error at its fifth occurrence — the fourth landed inside a gate-approved claim and is #307, still open. Mutation testing cannot catch it: perturbing the harness kills mutations, perturbing a SKILL.md kills nothing. NEVER-293.5 turns that asymmetry into a published number.

4. Two structural constraints that cannot both be met by one architecture. A required live check is illegal under NEVER-9.8 (.github/workflows/ci.yml:8-25: a required job may never carry an if:, because a skipped job reports success), and M2 cannot exit without a live run (docs/milestones/M2.md:240-241 requires a measured token cost, which only a live call produces).

5. A fake forge that always answers correctly cannot fail. 4 of 4 skill bodies name rate-limited-read-as-absence as their damaging failure mode. The mirror defect is worse because it passes: if the fake rejects a duplicate title, run 2 is clean for the fake's reason — a skill shadowed by its own test double.

6. There are two mutation surfaces. An argv-only recorder scores an identical-bytes Design rewrite as zero mutations, silently covering two of four skills.

Two claims are deliberately not `tool-checked`

references/verification.md:45-46 — "Choosing a stronger-sounding class than the check supports is the failure this taxonomy exists to prevent."

CLAIM-293.11 is model-judged. Harness holds a procedure, SKILL.md holds prose; agreement is meaning. The denominator is tool-checked, only the verdict is not.
CLAIM-293.9 is human-attested. n=1 over a non-deterministic function is an existence proof, not a property.

Raised here and recorded nowhere else

goal-create has no public Phase-0 input. Its goals source is a gitignored symlink into a private repository and is PRIVATE by fail-safe default (references/data-classification.md:66-76). Its scenarios are blocked on #292 unless a declared-synthetic fixture ships — and a live run against the real source on a cloud model would be a designed egress violation, pinned by NEVER-293.10.

Chain

903 tests / 889 pass / 0 fail / 14 skip / 3170 expect() · lint 0 across 95 files · typecheck clean · skill-lint 4 SKILL.md + 12 contract files, 0 errors · claim-lint 113 files, 0 errors, 1 warning (the expected docs/design/stories/9.md one; count unchanged) · design-spine now scans 11 Designs / 114 sections.

expect() moved 3160 → 3170. The delta is +10, not the +2 of an evidence artifact — packages/core/test/design-spine.test.ts:28 reads the directory at run time and :93 loops the ten spine sections per Design. A data point for #299: the per-artifact delta is a function of which corpus the artifact joins.

The gate is open — see the ## iai-gate comment. No code should be written before G1a, G2 and G6 are ruled.

Gate RULED, 2026-09-07 — ruling applied

Decided by the repository owner (@dev-pmallapp). iAI did not author the decision — docs/design/03-workflow.md:271. All seven parts accepted as recommended (G1a, G1b, G2, G3, G4, G5, G6).

This sentinel is repinned to bc25b778bec99ab10278f1f87be6ca7df347f3ba, the commit that carries the applied ruling. The text the approval was given against remains addressable at d59a95810544811c0e30106df0c447dd6bebf43b.

See the ## iai-gate comment, updated in place, for the full ruling.

@dev-pmallapp

iai-gate

RULED — 2026-09-07. All seven parts accepted as recommended.

Decided by the repository owner (@dev-pmallapp). iAI did not author this
decision, did not choose between the shapes, and did not originate any part of
it — docs/design/03-workflow.md:271. This comment relays the ruling with
attribution and is the same comment that carried the questions; it was updated
in place, not stacked.

The ruling was given as correct at this point in time. Two parts are forward
commitments and are the things to revisit if the facts move: G6 binds M4 to
a fixture repository this Story owns, and G2 binds every later Story that
reads a green seam rung.

#	Question	Ruling
G1a	Two rungs — an offline seam rung required in CI, and a live rung never required — rather than one architecture?	YES
G1b	Is token replay excluded by name, with the reason recorded?	YES, exclude
G2	May the seam rung's green let a later Story upgrade `CLAIM-41.8` / `CLAIM-47.4` to `tool-checked`?	NO — may never upgrade
G3	Does #293 build the adapter port as its first build target?	YES
G4	Must the fixture forge be able to lie, and permissive exactly where GitHub is permissive?	YES, both halves
G5	A new build target, and does `ARCHITECTURE.md` gain a Build Targets row?	YES to both
G6	Is the fixture repository the one M4/S4.5 consumes?	ONE, owned by #293, consumed by M4

Applied at bc25b778bec99ab10278f1f87be6ca7df347f3ba. The Gate section is retitled RULED, each Decision carries its ruling banner, and the G6 row of Not resolved here is struck — what remains there is the docs/milestones/M4.md:215 restatement, which lands with Build Target 10.

What the ruling settles beyond this Story

G2 is the load-bearing one, and it is a prohibition rather than a permission. The seam rung will be green, required, and visibly about idempotency — and it does not test the model reading the markdown. Three claims in S2.2's evidence are already marked stated, not executed — needs #293. Without G2, the predictable next step is a later author reading a green harness and upgrading CLAIM-41.8 to tool-checked; nothing in the repository would catch that, because mutation testing cannot and claim-lint checks that a class is one of three names, not that the check supports it.

G2 is therefore what makes the seam rung safe to build at all, and NEVER-293.5 is its mechanical half — a test whose PASS condition is that the harness does not notice a mutated SKILL.md, publishing the size of the transcription gap as a number.

G1a is a consequence, not a preference. A required live check is structurally illegal under NEVER-9.8 (.github/workflows/ci.yml:8-25), and M2 cannot exit without a live run (docs/milestones/M2.md:240-241). Two immovable constraints; no single architecture satisfies both.

iai-test-plan

Test plan: docs/test-plans/293-plan.md — pinned at 017137713af9dbbdd8f5f4ef369885afa4e4bf67, written against the ruled Design at bc25b77.

22 cases — 21 P0 / 1 P1 / 0 P2, in seven tables. Coverage verified in both directions: 11 claims defined, 11 anchored, 0 uncovered, 0 dangling. The forward direction is not checked by claim-lint — only anchor-dangling is — so it was asserted by hand. Case 4 of docs/test-plans/41-plan.md records that gap.

Corpus: 8 `real` / 13 `synthetic` / 1 `none`

The synthetic majority is structural and declared per case, not sloppy: a disposable git tree and a fake forge are invented on purpose. The real cases are the ones that read the repository as it is — the gh barrel, the skill roster, the CI job list, the required-context list.

The five cases that carry the weight

Case	Why it matters
3 + 7	Case 3 pairs `run1.mutations > 0` with `run2.mutations == 0` and asserts them in that order. Case 7 is its negative: a scenario whose run 1 mutates nothing fails. Without case 7, the identity function scores a perfect idempotency result
6	Mutation J2 made executable — delete a verb directory and the run must go red. A loop over a corpus stays green when the corpus shrinks
9	The case that proves the harness can fail. An injected truncated-list lie must make a real second run mutate. Without it the seam rung is `smoke-install` with a bigger budget, and `.github/workflows/ci.yml:137-146` names that disqualifier itself
8	Drives the fake forge directly with a duplicate-creating argv and asserts it accepts. A fake that refuses makes run 2 clean for the fake's reason — the S1.5 shadowing trap one layer out
16 + 17	These PASS when the harness fails to notice a mutated `SKILL.md`. That is gate ruling G2 made mechanical: the transcription gap becomes a number in an artifact instead of a footnote. Case 17 asserts `CLAIM-41.8` is still `model-judged` after this Story ships

Cases 16 and 17 will look like bugs to the next reader. They are not. Do not "fix" them — deleting them deletes the only published measurement of what the seam rung does not verify.

Finding 37 is discharged for this harness

Cases 21 and 22: the verdict is the exit code plus a JSON artifact, never a grep of stdout — and case 22 seeds a fixture repository with the runner's own success phrases to make case 21 provable rather than stated. docs/evidence/ is full of those exact phrases, so this is not a hypothetical corpus. Finding 37's recorded owner is "any future harness"; this is it.

Deviations from the milestone

docs/milestones/M2.md:107-109 seeds "the second run makes zero mutations." It is not wrong; it is incomplete in the one direction that matters. "Zero mutations" with no > 0 beside it is satisfied perfectly by a harness that never ran anything. Three deviations, all ruled at the gate — see the plan's Deviations table.

Chain

903 tests / 889 pass / 0 fail / 14 skip / 3170 expect() · lint 0 across 95 files · typecheck clean · skill-lint 0 errors · claim-lint 114 files, 0 errors, 1 warning (the expected docs/design/stories/9.md one).

Denominators: testplan-corpus 9 → 10 plans, 49 → 56 case tables, 220 → 242 cases.

expect() is UNCHANGED at 3170. A plan is +0, unlike a Design's +10 — no test loops over the plan roster, while design-spine.test.ts:92-96 loops the ten spine sections per Design. Second data point for #299: the per-artifact delta is a function of which corpus the artifact joins, and for plans it is zero.

Next: /iai:task-create 293 — 10 build targets, with the adapter port sequenced first per gate ruling G3.

#47: the test plan for S2.3 — 20 cases, 12 of 12 claims covered

#315: the ## Tasks section carries prose, and the rewrite must not ea…

#315: record the verification evidence

S2.6 is complete

The integration PR #312 merged into main at dab09ccd5e5f5f73592ca6d97110a51a775a4327 (2026-09-23T14:01:13Z).

All eight tasks are CLOSED/COMPLETED, each closed explicitly citing its own merge commit, with the issue state re-read before its box was ticked — a task PR targets the story branch, so closing keywords never fire.

Task	Merge
#318 the adapter port + `ARCHITECTURE.md` row	PR #326
#319 fixture repository + synthetic goals fixture	PR #327
#320 stateful fake forge, able to lie, non-enforcing	PR #329 @ `e1c2421`
#321 two-surface mutation recorder	PR #332 @ `27ac716`
#322 anti-vacuity assertion set, asserted by the runner	PR #333 @ `6e05328`
#323 the seam rung and its CI job	PR #335 @ `49d3ec5`
#324 the live rung, never required	PR #338 @ `60c187b`
#325 `M2.md`/`M4.md` corrections, line-neutral	PR #339 @ `adb9969`

Transitioned to status:resolved and closed per the Story state table at docs/design/03-workflow.md:195: Closed — closed, status:resolved, enters when the integration PR merges into main.

What this Story leaves behind, deliberately

No gh argv this repository builds has ever been executed, and no build target owns paying that bill #330 — no gh argv this repository builds has ever been executed. Six tasks deferred it; S2.6.7 — The live rung, never required #324's live rung was the first thing that could pay it, and the bill is still recorded rather than paid.
shouldRetryResponse answers retry: false for the transient connection error the reference tells you to retry #331 — shouldRetryResponse answers retry: false for the transient connection error the reference tells you to retry.
classify() never consults classifyPath, so USER/-derived content under a recognised key clears the cloud egress gate #336, M2's token-cost exit criterion can only be met with a session total today, not a per-verb measurement #337 — raised by S2.6.7 — The live rung, never required #324, both needing a human ruling.
Promote the seam rung to a seventh required context #334 — promoting seam to a seventh required context. Its first precondition is now met: the seam job has reported success on main at dab09cc. It was deliberately kept off this checklist and must not be read as owed here.
The fake forge's two unmodelled shapes — no milestone PATCH, and --milestone ignored on gh issue edit. Two ## Re-entry branches are consequently dead, for a correct reason, and only a real forge can exercise them.

Navigation Menu

Repository navigation

S2.6 — Skill execution harness: fixture repository, host runner, mutation recorder #293

Why this is a Story and not a task inside S2.2

Build targets

The design question this Story must answer first

What depends on this

Done when

Related

Tasks

iai-design

The issue's capability table, re-measured at `5884a40`

What the Design concluded

Two claims are deliberately not `tool-checked`

Raised here and recorded nowhere else

Chain

Gate RULED, 2026-09-07 — ruling applied

iai-gate

RULED — 2026-09-07. All seven parts accepted as recommended.

What the ruling settles beyond this Story

Next

iai-test-plan

Corpus: 8 `real` / 13 `synthetic` / 1 `none`

The five cases that carry the weight

Finding 37 is discharged for this harness

Deviations from the milestone

Chain

S2.6 is complete

What this Story leaves behind, deliberately

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

S2.6 — Skill execution harness: fixture repository, host runner, mutation recorder #293

Description

Why this is a Story and not a task inside S2.2

Build targets

The design question this Story must answer first

What depends on this

Done when

Related

Tasks

Activity

dev-pmallapp commented on Sep 7, 2026

iai-design

The issue's capability table, re-measured at 5884a40

What the Design concluded

Two claims are deliberately not tool-checked

Raised here and recorded nowhere else

Chain

Gate RULED, 2026-09-07 — ruling applied

dev-pmallapp commented on Sep 7, 2026

iai-gate

RULED — 2026-09-07. All seven parts accepted as recommended.

What the ruling settles beyond this Story

Next

dev-pmallapp commented on Sep 7, 2026

iai-test-plan

Corpus: 8 real / 13 synthetic / 1 none

The five cases that carry the weight

Finding 37 is discharged for this harness

Deviations from the milestone

Chain

dev-pmallapp commented on Sep 23, 2026

S2.6 is complete

What this Story leaves behind, deliberately

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

The issue's capability table, re-measured at `5884a40`

Two claims are deliberately not `tool-checked`

Corpus: 8 `real` / 13 `synthetic` / 1 `none`