Skip to content

Cut the model-read ecosystem specifics from mutation-testing: the audit skill #17733

Description

@kyle-sexton

Part 2 of 3 of #16752. Refs #15703.

Problem

The mutation-testing audit skill carries per-runner filter forms, one-tool pointers and language-flavored examples that the model reads.

Rows (7: R11 6, R8 1)

Row Rule Kind Cited at (as of 5b66a7f) Fact
F0281 R8 K4 plugins/mutation-testing/skills/audit/context/tool-test-restriction.md:1-13 restriction table records probe #9687 of Stryker.NET 5.0.0 (filter applies to the coverage run, keeps NoCoverage)
F0282 R11 K3 plugins/mutation-testing/skills/audit/context/tool-test-restriction.md:15-29 per-runner test-command filter forms (vitest, jest, mocha, Maven, Gradle, PHPUnit, pytest, unittest)
F0287 R11 K7 plugins/mutation-testing/skills/audit/context/config-fallback.md:32-38 bare dotnet test sentence in the never-guess-the-command rule
F0288 R11 K5 plugins/mutation-testing/skills/audit/SKILL.md:212-221 StrykerJS inPlace pointer
F0298 R11 K7 plugins/mutation-testing/skills/audit/context/suppression.md:145-156 src/pricing.ts in the sample suppression entry
F0299 R11 K5 plugins/mutation-testing/skills/audit/SKILL.md:430-436 Hypothesis derandomize and fast-check seed links
F0300 R11 K7 plugins/mutation-testing/skills/audit/context/persist-findings.md:120-130 .py, .cs, Makefile as examples of "a root that looks like source" (not the guard's allowlist)

Files

  • Cut: plugins/mutation-testing/skills/audit/SKILL.md, .../audit/context/tool-test-restriction.md, .../audit/context/config-fallback.md, .../audit/context/suppression.md, .../audit/context/persist-findings.md.
  • Outside references at d46efa9: scripts/docs-anchor-baseline.txt:84 (audit SKILL.md), :85 (tool-test-restriction.md, count 5); .performance/ratchets.json:182 (suppression.md skill-stack-fences ceiling 1); docs/skill-cheat-sheet.md:176 (mention only). The parent's scripts/spoke-plugin-root-baseline.txt entry for persist-findings.md is no longer on main.

Rules this part follows (from #16752, rules revision 5)

  • R11 (shipped knowledge): model-read K2, K3, K4, K5 or K7 text. Generic text replaces it only at a routing site where the skill still takes a language-specific step; anywhere else it is deleted with no replacement. An item kept only because a removed item consumes it goes in the same pull request.
  • R4 (consumer default): the value picks a tool or library for the consumer. It goes: no bundled default survives (ADR 0101 decision 2, "Ship no bundled default"); the choice is resolved from the consumer through docs/contracts/ecosystem-resolution/.
  • R8 (measured fact): cut it like the other rows and mark it in the pull request body as pending its targeted case; the host session's eval (below) decides whether it comes back, annotated portability-ok: measured <case-id>.
  • AC1's named exceptions are the kept classes of ADR 0101 (docs/adr/0101-resolve-ecosystem-specifics-from-the-consumer-repository.md, Decision and the 2026-11-17 amendment). Bundled defaults that pick a tool, command or library are retired wherever they appear (Interview Q32, D4 of the Brief).
  • Line numbers below were read at 5b66a7fa and may have moved; find each row by its cited text.

After the pull request opens (host session, not the lane)

The lane opens the pull request with Closes this issue and stops; it adds no label. The pull request stays held for the eval:

  • Hold line: a sweep or host session adds do-not-merge and posts, from a listed human's account, a comment with <!-- pr-hold reason=eval --> on a line of its own.
  • Targeted case per changed skill: one before-and-after case on sonnet at 3 runs a side, --case, --runs 3, a --max-cost-usd ceiling; opus only to confirm a sonnet loss; a new case gets a 1 to 3 run smoke first; start no batch after a usage-limit error (owner decision of 2026-11-16).
  • Controls: the owner's Q53 decision of 2026-11-17 (owner-decisions.md L202) reads: "each cut pull request is gated on a no-loss compare alone, with no wrong-text side". It supersedes Cut the model-read ecosystem specifics from mutation-testing, starting with its per-tool tables #16752's Controls line and its AC7 sentence ("each partial cut carries a detection-control result"). A whole-page deletion still states its predicted outcome in the pull request body before the first batch, and the record says the case cannot detect wrong text.
  • On a LOSS reading: a trace read of each dropped case and an opus confirm; only a loss traced to the changed text counts.
  • R8 rows: the row's targeted case with and without it; on a case drop or a mean fall of 0.15 or more without it, restore the row annotated portability-ok: measured <case-id>; a deterministic grader states before the run in how many of n runs it must fail.
  • Record: on a VALID no-loss run, post from a listed human's account <!-- eval-pair verdict=no-loss validity=VALID head=<40-hex head SHA> --> naming the current head, with no "do not merge" text (docs/contracts/loop-lane/README.md, "Cross-lane PR hold"). pr-merge's "Release eval holds whose record matches the head" step (.github/scripts/pr-merge/release-eval-hold.mjs) then removes do-not-merge; a later push needs a new record.

Out of scope

Rows outside this part's table, the parent's kept rows, every other plugin, toolchain's bundled defaults (#15733), and widening the portability gate (#15732).

Done when

  • Every row in this part's table is cut, replaced at a routing site, or (R8) cut and marked pending its targeted case; a row whose cited text is already gone on main is listed as GONE.
  • The pull request body states this part's row count (7) and lists every row id with its rule and outcome; F0281 is marked pending its targeted case.
  • Each R11 row's script-reader search is quoted in the pull request body; a row whose file a script reads is not cut, and is listed in the pull request body under a Script readers heading.
  • The pull request body names, for the audit skill, the existing eval case that reads the cut text, or says none does.
  • The pull request body says, for each changed file, whether the cut deletes the whole page or part of it.
  • The skill-stack-fences ceiling for plugins/mutation-testing/skills/audit/context/suppression.md in .performance/ratchets.json is lowered if the cut removes its fence, and scripts/docs-anchor-baseline.txt:84-85 match the remaining anchors.
  • AC10: no new enforcement-placing skill is added.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 21, 2026
  2. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none
    Blast radius: five mutation-testing audit skill files (SKILL.md and four context pages), plus the docs-anchor baseline and the suppression.md ratchet ceiling; read by the model running that skill and by the docs and performance checks.
    Reversible: yes; revert the pull request, or restore F0281 later with a portability-ok: measured <case-id> mark if its targeted case shows a loss.
    Open questions: none
    I understood this as part 2 of 3 of #16752: cut, replace at a routing site, or (F0281, R8) cut and mark pending its targeted case the 7 listed ecosystem-specific rows in the audit skill, and keep the anchor baseline and ratchet ceiling in step. The design is already chosen and the done-when is exact, so nothing is missing. Item 3 (quoting each R11 row's script-reader search in the pull request body) and the later eval are host-side steps and do not hold the rest. Note that SKILL.md invokes a dotnet-test-selection.sh script, so check that search before cutting any row whose file a script reads. #17731 is the sibling part covering different rows and does not block this issue. I found no duplicate among open issues and no merged pull request that already fixes this.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 21, 2026
  4. melodic-automation-lanes commented on Nov 21, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (tracker-output: "Each R11 row's script-reader search is quoted in the pull request body; a row..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. melodic-automation-lanes commented on Nov 21, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Cut, replace at a routing site, or (F0281, R8) cut and mark pending its targeted case the 7 listed ecosystem-specific rows in the mutation-testing audit skill, with the docs-anchor baseline and ratchets ceiling kept in step.

    Split. Done-when item 3 of the issue body needs a person: it is a check a person runs, listed below, and the lane does not do it. This issue keeps the rest of its done-when.

    Not done by the lane: checks a person runs

    External state: none

    Dependencies

    • Sibling parts of #​16752 (#​17731 setup and tooling page) cover other rows and are not blockers; a later host-session eval gates merge, not the lane's work.

    Related work

    Sources read

    • plugins/mutation-testing/skills/audit/SKILL.md:217: StrykerJS inPlace pointer still present (F0288)
    • plugins/mutation-testing/skills/audit/SKILL.md:432: Hypothesis derandomize pointer still present (F0299)
    • plugins/mutation-testing/skills/audit/SKILL.md:125: a dotnet-test-selection.sh script is invoked from the skill, so a script reads related text; check before cutting
    • plugins/mutation-testing/skills/audit/context/config-fallback.md:37: bare dotnet test sentence present (F0287)
    • plugins/mutation-testing/skills/audit/context/suppression.md:152: src/pricing.ts sample entry present (F0298)
    • plugins/mutation-testing/skills/audit/context/tool-test-restriction.md:12: probe table and per-runner filters present (F0281, F0282)
    • scripts/docs-anchor-baseline.txt:85: tool-test-restriction.md count 5; line 84 is audit SKILL.md count 1
    • .performance/ratchets.json:182: suppression.md skill-stack-fences ceiling is 1

    Automated refinement by the lanes App. Intake triages the issue again with it.

  6. melodic-automation-lanes commented on Nov 21, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"fe2d150469ea66cf75b2e0bb6dfa5e3a6d10714b52040c322dc2cf961f08d3f5","body_sha256":"723003d0221c823c293dea14347e4930a24c25980108b0b42668cbcb14091a43"}

  7. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor
    No description provided.
  8. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor
    No description provided.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-triage

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions