S2.6.5 — The anti-vacuity assertion set, asserted by the runner #322

Activity

1. dev-pmallapp

   added this to the M2 — Universal lifecycle milestone on Sep 7, 2026
2. dev-pmallapp

   added

   type:taskOne unit of work within a StoryOne unit of work within a Story

   domain:devSoftware development packSoftware development pack

   iaiCreated or managed by iAICreated or managed by iAI

   on Sep 7, 2026
3. dev-pmallapp

   added a parent issue on Sep 7, 2026

   • S2.6 — Skill execution harness: fixture repository, host runner, mutation recorder #293
4. dev-pmallapp commented on Sep 10, 2026

   

   dev-pmallapp

   on Sep 10, 2026

   OwnerAuthor

   More actions

   iai-evidence

   Verdict: PASS. Artifact: docs/evidence/322-20260910T042657Z.md, pinned at 95c0af46749e5b9d912c7419beaa64c695309e8d. PR #333.

   Build target 6 of docs/design/stories/293.md. Cases 3, 5, 6, 7, 11, 21, 22 PASS. CLAIM-293.3 and NEVER-293.4 are met for the harness's transcription of each skill, which is a different population from the skill.

   bun run skill-harness exists for the first time. The command named in seven test-plan cases was in no package.json in this tree until now.

   Chain

   1038 tests / 1024 pass / 0 fail / 14 skip / 3777 expect() · lint 0 across 121 files · typecheck clean · skill-lint 4 + 12, 0 errors · claim-lint 120 files, 0 errors, 1 warning (the expected stories/9.md one) · bun run skill-harness exit 0 · both verifier scripts PASS, required contexts still six. CI on PR #333 is 7/7.

   The measurement

   Scenario	run 1	run 2	Re-entry rows
   goal-create/absent-milestone	mutations 1, reads 3	0, reads 3	3
   story-create/two-unstoried-rows	mutations 2, reads 4	0, reads 4	4
   story-design/design-and-sentinel	mutations 4, reads 4	0, reads 4	4
   story-test-plan/plan-and-sentinel	mutations 4, reads 5	0, reads 5	5
   story-design/sentinel-only	forge only — 1, reads 4	0, reads 4	4
   story-test-plan/file-only	worktree only — 3, reads 5	0, reads 5	5

   Every threshold in the right-hand column is read off skills/*/SKILL.md at run time. run2.reads >= rows is what separates "decided to do nothing" from "did nothing" — a run 2 with zero mutations and zero reads did not run.

   The last two scenarios are the surface-separation pair, which is why the mutation counter is the union of both surfaces: a forge-only count scores story-test-plan/file-only as a vacuous run 1, and mutation M8 measures exactly that.

   Mutation testing — 16 injected, 13 killed, 3 survivors

   #	Mutation	Result
   M1	the roster emptied	KILLED — the CLI, on the empty-roster message specifically
   M2	a verb directory renamed away	KILLED — coverage mismatch naming the verb, plus the pinned-count failure
   M3	goal-create's create step disabled	KILLED — run-1-made-no-mutation; artifact shows run2: null
   M4	run 2 executed before run 1's > 0 check	KILLED by the unit test only
   M5	countReads becomes the subtraction	KILLED by the unit test only
   M6	the reads threshold hardcoded to 4	KILLED — by goal-create, not the skill predicted
   M7	per-skill coverage replaced by an aggregate	KILLED by the unit test only
   M8	mutations counts the forge only	KILLED — by the worktree-only scenario
   M9	the run-2 tree-hash check deleted	SURVIVED — entailed by the evidence being empty
   M10	a fresh fake forge before run 2	KILLED, after a fix
   M11	the verdict re-derived from rendered text	KILLED, after a fix — statically
   M12	renderArtifact forces verdict: "pass"	KILLED, after a fix
   M13	the wrapper hardcodes process.exit(0)	KILLED
   M14	test-only weakening	SURVIVED — the control half
   M14b	M14 plus a real defect	KILLED, by a different test
   S1	two ## Re-entry rows reordered	SURVIVED, by design

   Three survived their first run, and each was a real gap now closed and re-killed:

   1. M10 crashed instead of reporting. A fresh forge before run 2 made a transcription throw and the runner died with an unhandled rejection — no verdict, no artifact, no scenario named, exactly when the artifact is most wanted. Now a scenario-threw failure, one failure per cause.
   2. M11 was behaviourally equivalent, so no dynamic test could ever kill it. renderReport(...).includes("verdict: pass") returns the identical answer for every fixture in the corpus. The kill had to be static: decideVerdict's own source may not contain renderReport, renderArtifact, SUCCESS_PHRASES, .includes(" or stdout, with the extracted body asserted non-empty first. This is finding 37, whose recorded owner was "any future harness".
   3. M12 exposed an unasserted field. renderArtifact could emit verdict: "pass" on a failing run with every test green, because the only agreement assertion compared exit codes and the real roster never fails. S2.6.6 — The seam rung and its CI job #323's CI job may read that artifact — the disagreement would have surfaced as a green job over a red run.

   M4, M5 and M7 leave bun run skill-harness exit 0 and unchanged, and die only to synthetic scenarios built to trip them:

   A green run of the real roster is not evidence that the runner's checks work. It is evidence that the roster passes them. A task that shipped only the real roster would have shipped all three undetected.

   M9 survived everywhere and is kept anyway — the tree-hash check is entailed by run 2's mutation evidence being empty over this roster, retained because the issue mandates it and M10 is its one independent killer. Recorded as redundancy rather than hidden. S1 survives by design and must not be "fixed": reordering a ## Re-entry row changes not one assertion, which is NEVER-293.5 working.

   Findings raised

   1. S2.6.4 — The two-surface mutation recorder #321's write-seam lint was RED AGAINST CORRECT CODE, and the class was already written down. It matched the bare word writeFile anywhere in a module; runner.ts exposes a context method that only delegates to recorder.writeFile and imports no fs at all. packages/exec/test/port.test.ts had already rewritten its launcher detector from a call shape to an import for exactly this reason, and the standing rule is that for an assertion — unlike a ban — a false positive must be fixed. Knowing the class did not prevent reproducing it one task later. The detector now reads the capability off the import list, with a regression test asserting a delegating module is a non-writer.
   2. A read may not be computed by subtraction. isMutatingGhArgv returns false for every git argv by construction, so calls.length - attemptedMutations.length scores git add and git commit as reads and a run 2 that committed four times would satisfy reads >= 4. M5 is the measurement: the subtraction passes the real roster and dies only to a deliberately mixed one.
   3. The fake forge models neither a milestone PATCH nor --milestone on gh issue edit. Two ## Re-entry branches are dead in this roster for a correct reason. They were left implemented and unexercised rather than forced: a scenario built to reach them would be built around the fake's gaps. Owner: S2.6.6 — The seam rung and its CI job #323's audit and S2.6.7 — The live rung, never required #324's live rung.

   What this does NOT establish

   • The transcriptions are UNAUDITED. CLAIM-293.11 and case 18 are S2.6.6 — The seam rung and its CI job #323's. This task authored them and deliberately did not rule on their faithfulness — that would be self-attestation.
   • Nothing here tests the model reading a SKILL.md. The runner prints three NOT VERIFIED lines saying so on every run. Gate ruling G2 forbids reading this green as licence to upgrade CLAIM-41.8 or CLAIM-47.4.
   • CLAIM-293.6 half (ii) is still NOT MET — --inject is deliberately absent; case 9 closes at S2.6.6 — The seam rung and its CI job #323.
   • No CI job, and the required contexts are still six. Build target 7 is a six-site edit and is S2.6.6 — The seam rung and its CI job #323's.
   • The first-contact bill (No gh argv this repository builds has ever been executed, and no build target owns paying that bill #330) is still unpaid — all six scenarios answered every gh call from S2.6.3 — The stateful fake forge: able to lie, and non-enforcing #320's fake. shouldRetryResponse answers retry: false for the transient connection error the reference tells you to retry #331 untouched.
5. dev-pmallapp

   added

   status:resolvedWork complete, awaiting closureWork complete, awaiting closure

   on Sep 10, 2026
6. dev-pmallapp commented on Sep 10, 2026

   

   dev-pmallapp

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Closed by the merge of PR #333 into story/293-s2-6-skill-execution-harness at 6e05328.

   A task PR targets the story branch, so GitHub's closing keywords never fire (CONTRIBUTING.md, Pull requests); this closure is explicit and cites the merge commit.

   Verified PASS before merge — evidence docs/evidence/322-20260910T042657Z.md, ## iai-evidence sentinel posted, CI 7/7 on the PR head 95c0af4.

   Carried forward to #323, deliberately not closed here:

   • CLAIM-293.11 / case 18 — the four transcriptions are unaudited; ruling on them here would have been self-attestation.
   • CLAIM-293.6 half (ii) — --inject is deliberately absent from the CLI; case 9 needs the truncated-list lie armed per scenario.
   • Build target 7 — the seam rung, its job-list edit and the seventh required context.
7. dev-pmallapp

   closed this as completedon Sep 10, 2026