Repository navigation
S2.6.5 — The anti-vacuity assertion set, asserted by the runner - #333
Merged
dev-pmallapp merged 7 commits intoSep 10, 2026
Merged
dev-pmallapp merged 7 commits into
dev-pmallapp merged 7 commits into
Conversation
3 tasks
dev-pmallapp
marked this pull request as ready for review
September 10, 2026 04:37
dev-pmallapp
merged commit Sep 10, 2026
6e05328
into
story/293-s2-6-skill-execution-harness
7 checks passed
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build target 6 of
docs/design/stories/293.md. AnchorsCLAIM-293.3andNEVER-293.4.Verdict: PASS for cases 3, 5, 6, 7, 11, 21 and 22.
CLAIM-293.3andNEVER-293.4are met for the harness's transcription of each skill — a different population from the skill, and the runner says so in its own output on every run.Evidence:
docs/evidence/322-20260910T042657Z.mdbun run skill-harnessexists for the first time. The command named in seven test-plan cases was in nopackage.jsonin this tree until now.Chain
1038 tests / 1024 pass / 0 fail / 14 skip / 3777
expect()· lint 0 across 121 files · typecheck clean · skill-lint 4 + 12, 0 errors · claim-lint 120 files, 0 errors, 1 warning (the expectedstories/9.mdone) ·bun run skill-harnessexit 0 over six scenarios · both verifier scripts PASS with the required contexts still six — the seam rung's CI job is #323's six-site edit and is deliberately not here.What it ships
packages/harness/src/runner.ts—runHarness, anddecideVerdict, the one pure function that turns typed observations into an exit code. Thirteen denominators, each reported on its own line whether it passed or failed.transcription-<verb>.ts, one step per## Re-entryrow, each read attributed to the row it decides.scenario-roster.ts— six frozen scenarios covering 4 of 4 skills, including a forge-only and a worktree-only scenario so neither surface can be silently unwired.re-entry.tsandargv-kind.ts— the Re-entry parser promoted out of a test file so there is exactly one, and a positive argv classifier so a read is never computed by subtraction.scripts/skill-harness.ts— the only writer; it passesverdict.exitCodetoprocess.exitwith no branch.Measured, per skill
goal-create3 Re-entry rows / run 2 read 3 ·story-create4 / 4 ·story-design4 / 4 ·story-test-plan5 / 5 — every threshold read offskills/*/SKILL.mdat run time. That is what separates "decided to do nothing" from "did nothing".Mutation testing — 16 injected, 13 killed, 3 survivors
Three mutations survived their first run. Each was a real gap, each is now closed and re-killed — that is the most useful part of this PR:
scenario-threwfailure, one per cause.renderReport(...).includes("verdict: pass")gives the identical answer for every fixture, so no dynamic test could ever kill it. Killed by a static check ondecideVerdict's own source. Finding 37, whose recorded owner was any future harness.renderArtifactcould emitverdict: "pass"on a failing run and everything passed. S2.6.6 — The seam rung and its CI job #323's CI job may read that file, so the disagreement would have surfaced as a green job over a red run.M4, M5 and M7 leave
bun run skill-harnessexit 0 and unchanged. They die only to synthetic scenarios built to trip them:M9 survived everywhere and is kept —
run2.treeHash == run1.treeHashis entailed by run 2's evidence being empty over this roster; it is retained because the issue mandates it and M10 is its only independent killer. S1 survives by design: reordering a## Re-entryrow changes nothing, which isNEVER-293.5working.Findings
writeFile, and a context method that only delegates to the recorder tripped it. The repository had already recorded this class —port.test.tsrewrote its launcher detector from a call shape to an import for the same reason — and it was reproduced one task later anyway. The detector now reads the capability off the import list, with a regression test.isMutatingGhArgvis false for everygitargv, socalls.length - attemptedMutations.lengthscoresgit commitas a read.PATCHnor--milestoneongh issue edit. Two Re-entry branches are consequently dead here — for a correct reason — and were left unexercised rather than forced around the fake's gaps.Not established, deliberately: the transcriptions are unaudited (
CLAIM-293.11, case 18 — #323); nothing tests the model reading aSKILL.md;CLAIM-293.6half (ii) is still NOT MET; no CI job; #330 is still unpaid — everyghcall was answered from #320's fake; #331 untouched.Refs #322