Skip to content

S2.6.5 — The anti-vacuity assertion set, asserted by the runner - #333

Merged
dev-pmallapp merged 7 commits into
story/293-s2-6-skill-execution-harnessfrom
task/322-anti-vacuity-runner
Sep 10, 2026
Merged

dev-pmallapp merged 7 commits into
story/293-s2-6-skill-execution-harnessfrom
task/322-anti-vacuity-runner

Conversation

@dev-pmallapp

Copy link
Copy Markdown
Owner

Build target 6 of docs/design/stories/293.md. Anchors CLAIM-293.3 and NEVER-293.4.

Verdict: PASS for cases 3, 5, 6, 7, 11, 21 and 22. CLAIM-293.3 and NEVER-293.4 are met for the harness's transcription of each skill — a different population from the skill, and the runner says so in its own output on every run.

Evidence: docs/evidence/322-20260910T042657Z.md

bun run skill-harness exists for the first time. The command named in seven test-plan cases was in no package.json in this tree until now.

Chain

1038 tests / 1024 pass / 0 fail / 14 skip / 3777 expect() · lint 0 across 121 files · typecheck clean · skill-lint 4 + 12, 0 errors · claim-lint 120 files, 0 errors, 1 warning (the expected stories/9.md one) · bun run skill-harness exit 0 over six scenarios · both verifier scripts PASS with the required contexts still six — the seam rung's CI job is #323's six-site edit and is deliberately not here.

What it ships

  • packages/harness/src/runner.ts — runHarness, and decideVerdict, the one pure function that turns typed observations into an exit code. Thirteen denominators, each reported on its own line whether it passed or failed.
  • Four transcriptions, transcription-<verb>.ts, one step per ## Re-entry row, each read attributed to the row it decides.
  • scenario-roster.ts — six frozen scenarios covering 4 of 4 skills, including a forge-only and a worktree-only scenario so neither surface can be silently unwired.
  • re-entry.ts and argv-kind.ts — the Re-entry parser promoted out of a test file so there is exactly one, and a positive argv classifier so a read is never computed by subtraction.
  • scripts/skill-harness.ts — the only writer; it passes verdict.exitCode to process.exit with no branch.

Measured, per skill

goal-create 3 Re-entry rows / run 2 read 3 · story-create 4 / 4 · story-design 4 / 4 · story-test-plan 5 / 5 — every threshold read off skills/*/SKILL.md at run time. That is what separates "decided to do nothing" from "did nothing".

Mutation testing — 16 injected, 13 killed, 3 survivors

Three mutations survived their first run. Each was a real gap, each is now closed and re-killed — that is the most useful part of this PR:

  • M10 crashed instead of reporting: a fresh forge before run 2 made a transcription throw, and the runner died with no verdict, no artifact, no scenario named. Now a scenario-threw failure, one per cause.
  • M11 was behaviourally equivalent — deriving the exit code from renderReport(...).includes("verdict: pass") gives the identical answer for every fixture, so no dynamic test could ever kill it. Killed by a static check on decideVerdict's own source. Finding 37, whose recorded owner was any future harness.
  • M12 exposed an unasserted field: renderArtifact could emit verdict: "pass" on a failing run and everything passed. S2.6.6 — The seam rung and its CI job #323's CI job may read that file, so the disagreement would have surfaced as a green job over a red run.

M4, M5 and M7 leave bun run skill-harness exit 0 and unchanged. They die only to synthetic scenarios built to trip them:

A green run of the real roster is not evidence that the runner's checks work. It is evidence that the roster passes them.

M9 survived everywhere and is kept — run2.treeHash == run1.treeHash is entailed by run 2's evidence being empty over this roster; it is retained because the issue mandates it and M10 is its only independent killer. S1 survives by design: reordering a ## Re-entry row changes nothing, which is NEVER-293.5 working.

Findings

  1. S2.6.4 — The two-surface mutation recorder #321's write-seam lint was red against correct code. It matched the bare word writeFile, and a context method that only delegates to the recorder tripped it. The repository had already recorded this class — port.test.ts rewrote its launcher detector from a call shape to an import for the same reason — and it was reproduced one task later anyway. The detector now reads the capability off the import list, with a regression test.
  2. A read may not be computed by subtraction. isMutatingGhArgv is false for every git argv, so calls.length - attemptedMutations.length scores git commit as a read.
  3. The fake models neither a milestone PATCH nor --milestone on gh issue edit. Two Re-entry branches are consequently dead here — for a correct reason — and were left unexercised rather than forced around the fake's gaps.

Not established, deliberately: the transcriptions are unaudited (CLAIM-293.11, case 18 — #323); nothing tests the model reading a SKILL.md; CLAIM-293.6 half (ii) is still NOT MET; no CI job; #330 is still unpaid — every gh call was answered from #320's fake; #331 untouched.

Refs #322

@dev-pmallapp
dev-pmallapp marked this pull request as ready for review September 10, 2026 04:37
@dev-pmallapp
dev-pmallapp merged commit 6e05328 into story/293-s2-6-skill-execution-harness Sep 10, 2026
7 checks passed
@dev-pmallapp
dev-pmallapp deleted the task/322-anti-vacuity-runner branch September 10, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants