You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(mutation-testing): #16229's Sonnet compare cannot decide non-inferiority on 3 cases in #16063 #16357
The #16063 two-version compare of #16229 reads NOT MET for mutation-testing on Sonnet: over 3 cases the paired with-arm change is +0.04 with a 95% interval of -0.28 to +0.35 (noise-report.py, not shown non-inferior). The interval is wide because there are only 3 cases at 3 runs per side; au-light-survivors-unclassified fell .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2 failing runs before/after) and au-no-test-command-asks rose .67 to 1.00. Opus is non-inferior (+0.00).
The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
A before and after eval of mutation-testing on Sonnet, sized so the interval can decide the 0.05 margin (more runs per case, or more cases that load the changed text), both sides run together, is recorded and reads non-inferior at margin 0.05; or the record states why that cannot be shown with the cases available.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side au-light-survivors-unclassified runs against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.
Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.
Possibly related, but it targets different cases (the 12 se- cases): #16358. I found no merged pull request that already covers this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Record: the drop is not the #16229 text, and non-inferiority at margin 0.05 cannot be shown with the cases available (this session's reading, not an owner decision).
Read from the kept #16063 compare: the six Sonnet runs of au-light-survivors-unclassified and their traces, side taken from the file stem.
Side, run
Score
Failing graders
What the trace shows
before 1
1.00
none
before 2
1.00
none
before 3
0.33
survivor-unclassified, writes-skip-token
A shell glob error (no matches found: .claude/*.yaml) hid the listing; the reply says "No mutation-testing config exists at any layer", so the light level was never read.
after 1
0.67
writes-skip-token
Read the config and skipped triage ("so I skipped the fresh-context triage. The survivor is unclassified"), but left out the literal verification: skipped (light) token.
after 2
1.00
none
after 3
0.00
all three
Same glob error (no matches found: .claude/mutation*); "No mutation-testing config exists in the repo"; ran at the default level and dispatched a "Triage surviving mutant" agent.
Not the grader or the case: after run 1 really omits the token the unchanged skill line asks for.
The worst run on each side fails the same way, a config-discovery miss after a shell glob error, which then fails different graders downstream. That is why the per-grader counts differ between the sides.
Opus read this case .89 before and .89 after.
Second done-when item, by its second branch ("or the record states why that cannot be shown with the cases available"):
From the observed per-case variance, a script mirroring noise-report.py's rule (interval over per-case deltas, lower end at least -0.05, a case drop of one third counts as a loss) gives the chance that a version with no real change passes: 0.20 at 3 runs a side, 0.24 at 6, 0.39 at 20, 0.79 at 150, 0.85 at 200. Reaching about 80% needs 150 to 200 runs a side per case, 900 to 1,200 runs. The script reproduces the record's +0.04 and -0.28 lower end.
Checked by this session directly: the quoted trace text of before run 3, after run 1 and after run 3, and that #16229's audit-skill diff has no triage line. Relayed from a worker's read and not re-run by me: the simulation figures and the case-loading survey.
Found while reading, not fixed here: the audit skill's Phase 0 config bullet does not name the team file path, and a glob error in the same shell call can hide the folder listing. Naming docs/conventions/mutation-testing.yaml in that bullet would address the miss seen on both sides; it would be a text change with its own before and after case.
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side
au-light-survivors-unclassifiedruns against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.
Possibly related, but it targets different cases (the 12
se-cases): #16358. I found no merged pull request that already covers this.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 17, 2026
Record: the drop is not the #16229 text, and non-inferiority at margin 0.05 cannot be shown with the cases available (this session's reading, not an owner decision).
Read from the kept #16063 compare: the six Sonnet runs of
au-light-survivors-unclassifiedand their traces, side taken from the file stem.survivor-unclassified,writes-skip-tokenno matches found: .claude/*.yaml) hid the listing; the reply says "No mutation-testing config exists at any layer", so thelightlevel was never read.writes-skip-tokenverification: skipped (light)token.no matches found: .claude/mutation*); "No mutation-testing config exists in the repo"; ran at the default level and dispatched a "Triage surviving mutant" agent.First done-when item, the cause:
config-fallback.md, the schema and the setup files.Second done-when item, by its second branch ("or the record states why that cannot be shown with the cases available"):
au-*cases load the audit text feat(mutation-testing): accept any mutation tool id and name one with no shipped adapter #16229 changed. The sixpr-*cases fire the principles skill; the twosf-*cases were added later and exist on neither side; the twelvese-*cases load the setup text, which is test(mutation-testing): run a before and after eval of #16229's setup-skill text over the 12 se- cases #16358's compare.noise-report.py's rule (interval over per-case deltas, lower end at least -0.05, a case drop of one third counts as a loss) gives the chance that a version with no real change passes: 0.20 at 3 runs a side, 0.24 at 6, 0.39 at 20, 0.79 at 150, 0.85 at 200. Reaching about 80% needs 150 to 200 runs a side per case, 900 to 1,200 runs. The script reproduces the record's +0.04 and -0.28 lower end.Checked by this session directly: the quoted trace text of before run 3, after run 1 and after run 3, and that #16229's audit-skill diff has no triage line. Relayed from a worker's read and not re-run by me: the simulation figures and the case-loading survey.
Found while reading, not fixed here: the audit skill's Phase 0 config bullet does not name the team file path, and a glob error in the same shell call can hide the folder listing. Naming
docs/conventions/mutation-testing.yamlin that bullet would address the miss seen on both sides; it would be a text change with its own before and after case.