Skip to content

S2.6.4 — The two-surface mutation recorder - #332

Merged
dev-pmallapp merged 2 commits into
story/293-s2-6-skill-execution-harnessfrom
task/321-two-surface-mutation-recorder
Sep 10, 2026
Merged

dev-pmallapp merged 2 commits into
story/293-s2-6-skill-execution-harnessfrom
task/321-two-surface-mutation-recorder

Conversation

@dev-pmallapp

Copy link
Copy Markdown
Owner

Build target 5 of docs/design/stories/293.md. Anchors CLAIM-293.8.

Verdict: PASS for case 4 and case 10. CLAIM-293.8 is met for the recorder; its per-skill threshold — at least one scenario per file-writing skill with run1.tree_hash != pre.tree_hash — needs a transcription and a runner and closes at #323, which must not read this as having closed it.

Evidence: docs/evidence/321-20260909T125257Z.md

Chain at 761bb1d

987 tests / 973 pass / 0 fail / 14 skip / 3513 expect() · lint 0 across 109 files · typecheck clean · skill-lint 4 + 12, 0 errors · claim-lint 117 files, 0 errors, 1 warning (the expected stories/9.md one). The evidence artifact adds +2 expect() on top, per #299.

What it ships

  • packages/harness/src/mutation-recorder.ts — two surfaces, reported as two separately named lists, never one total. The forge surface reuses isMutatingGhArgv; there is no second argv classifier.
  • A write seam, because the third clause is an ATTEMPT clause. git status, git hash-object, a --stat diff and an mtime comparison all sample effect, and the defining property of a byte-identical rewrite is that it leaves no effect. The seam is the worktree's createRecordingPort.
  • Three worktree observables, not one. git commit --allow-empty after a byte-identical rewrite moves HEAD while leaving HEAD^{tree} byte-identical, so a tree-hash-only recorder cannot tell a committed identical-bytes rewrite from an uncommitted one.
  • No verdict anywhere. mutationEvidence returns a list whose emptiness is the zero-mutation result, mirroring ghOk([]). A boolean is the assertion shape under which every surface-unwiring mutation survives.
  • attemptedMutations is typed GhPlan — an argv list with no exit code — so && c.exitCode === 0 is a type error downstream rather than a review miss.

Mutation testing — 14 injected, 12 killed, 2 survivors by design

M3 and M4 kill disjoint halves: if (identical) return; kills only the two identical-bytes scenarios, and identical := existedBefore kills only the differing-bytes ones. A suite without case 10 ships M3 green.

M5/M6 is #320's M9/M9b one layer up: weakening the matrix assertion to a length check survives alone, and the kill for the paired defect comes from a different test. A weakened assertion is not detectable by the test it weakens.

M12's asymmetry: an exit-code filter is killed only by a lie that refuses. The silent-no-op lie exits 0, so the property "records the attempt, not the success" is observable only through a double that says no.

S2 survives — the fake made to enforce a duplicate leaves this suite green — while M12 dies. Same property, opposite sides.

Findings

  1. Every effect-sampling alternative to the write seam is blind to an identical-bytes rewrite by construction.
  2. The seam has a run-time bypass detector, and its blind spot is the same one the seam exists to cover — stated in the module, the test and the artifact so nobody reads it as complete.
  3. Tracked-ness is recorded, never filtered. A Design is untracked at the moment it is created; filtering on it would score story-design as zero mutations.
  4. M8's prediction was wrong and is recorded as wrong — it kills C as well as E. The conclusion held; the stated mechanism did not.

The first-contact bill (#330) is still unpaid — this task answered every gh call from #320's fake. #331 is untouched: the transient mode was injected to prove a failed attempt is recorded, and no retry logic was built on it.

Refs #321

@dev-pmallapp
dev-pmallapp marked this pull request as ready for review September 9, 2026 12:58
@dev-pmallapp
dev-pmallapp merged commit 27ac716 into story/293-s2-6-skill-execution-harness Sep 10, 2026
7 checks passed
@dev-pmallapp
dev-pmallapp deleted the task/321-two-surface-mutation-recorder branch September 10, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants