You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(mutation-testing): run a before and after eval of #16229's setup-skill text over the 12 se- cases #16358
#16229 changed the mutation-testing setup skill's text (plugins/mutation-testing/skills/setup/SKILL.md lines 73-76 and 96-98, skills/setup/templates/config-template.md, and the config schema description). By the owner's choice in the #16063 compare (D303), that text was not compared and the 12 se-* setup cases were not run, so its effect is not measured.
The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
Done when
A before and after eval of the 12 se-* setup cases, on Sonnet and Opus, before 59d6f9d110 against after 08fcb9062e (or the current main text against its parent where it has changed), both sides run together, 3 runs per side, is recorded here with the per-grader pass counts per side.
The record reads non-inferior at margin 0.05 with no case drop on each model; or each loss is named with its cause and carried in its own issue.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: #16229 changed the mutation-testing setup skill text, and its effect was never measured because the 12 se-* setup cases were skipped in the #16063 compare. This issue asks for a paired before/after eval (Sonnet and Opus, 3 runs per side) and a non-inferiority record at margin 0.05. The numeric bars are stated as the filing session's judgment, to be set by whoever registers the eval from measured rates. That does not leave an outcome open.
The eval runs themselves may need a host or an attended run, so the issue keeps its host-run label.
Related, not a duplicate: #16357 covers the Sonnet compare for 3 cases in #16063. No merged pull request in the recent list appears to already do this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (eval: "A before and after eval of the 12 se-* setup cases, on Sonnet and Opus, befor..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Record: the setup text of #16229 reads non-inferior on Sonnet over the 12 se-* cases (this session's reading, not an owner decision).
Compare: before 59d6f9d110 against after 08fcb9062e; the setup text, schema and the 12 cases are unchanged on main since, so the pair is the issue's own. Sonnet, 3 runs a side, both sides in one launch, judge claude-opus-5-5, Claude Code 2.1.326.
Registered pass rule, fixed before the compare: both files valid, no case left out, no case drop, and noise-report.py at margin 0.05. Result: paired change over 12 cases -0.01, 95% interval -0.04 to +0.03, non-inferior at margin 0.05; no case drop; no run excluded.
Opus was not run. The registration follows the owner's eval gate (Sonnet 3 a side, Opus only to confirm a loss), which differs from this issue's text asking for both models; there was no Sonnet loss to confirm. The issue's first done-when item is therefore met for Sonnet only.
Cost: 36.14 USD list-price estimate (smoke 0.63, before 17.39, after 18.12). The registration's expected figure was 12.00; the cost basis was borrowed from other cases and ran low by about three times.
One amendment after the first smoke run, before any counted run: the registration's own check for "skill loaded from this side's copy" looked for a trace line 2.1.326 no longer writes; it now reads the trace's init record (exactly one mutation-testing plugin, at this side's copy). The first smoke run was re-graded offline and not re-run. Pass rule, bars, exclusions and flags were unchanged.
Read with it: graders that pass 0 of 3 on both sides (legacy-setup-2, legacy-setup-3, legacy-setup-4, offers-alternative, timeout-not-derived, retired-info) measure nothing about this change either way; they are a case-quality finding, not a loss.
with-arm mean: 0.82, 95% interval 0.71 to 0.93 (normal, 12 cases)
cost: 18.12 USD for the whole suite (list-price estimate)
no without-arm runs in this result, so there is no delta to read
with-arm pass count at threshold 0.80: 9 of 12, 95% interval 0.51 to 0.99 (normal)
judge agreement: case se-apply-proposes-tool-writes-config, grader legacy-setup-2: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-apply-user-layer-no-repo-write, grader legacy-setup-11: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-arid-entry-missing-keys, grader names-missing-keys: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-arid-entry-missing-keys, grader legacy-setup-5: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-check-reports-every-probe, grader legacy-setup-1: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-check-unconfigured-info, grader legacy-setup-10: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-gradle-name-filter-notice, grader exercised-refuses-notice: unanimous in 2 of 3 runs (8 of 9 votes match their run's majority)
judge agreement: case se-gradle-name-filter-notice, grader legacy-setup-9: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-no-score-threshold, grader offers-alternative: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-no-score-threshold, grader legacy-setup-4: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-personal-arid-not-applied, grader promotion-and-floor: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-personal-arid-not-applied, grader legacy-setup-6: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-red-baseline-still-configures, grader timeout-not-derived: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-red-baseline-still-configures, grader legacy-setup-3: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-retired-team-file-migrates, grader keys-carried: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-retired-team-file-migrates, grader legacy-setup-12: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-test-command-needs-placeholder, grader legacy-setup-8: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
judge agreement: case se-untracked-config-offers-add, grader legacy-setup-7: unanimous in 3 of 3 runs (9 of 9 votes match their run's majority)
with-arm change (this minus baseline), paired over 12 cases: -0.01, 95% interval -0.04 to +0.03
compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.04 is at or above -0.05)
Activity
github-actions commentedon Nov 15, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: #16229 changed the mutation-testing setup skill text, and its effect was never measured because the 12
se-*setup cases were skipped in the #16063 compare. This issue asks for a paired before/after eval (Sonnet and Opus, 3 runs per side) and a non-inferiority record at margin 0.05. The numeric bars are stated as the filing session's judgment, to be set by whoever registers the eval from measured rates. That does not leave an outcome open.The eval runs themselves may need a host or an attended run, so the issue keeps its
host-runlabel.Related, not a duplicate: #16357 covers the Sonnet compare for 3 cases in #16063. No merged pull request in the recent list appears to already do this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (eval: "A before and after eval of the 12 se-* setup cases, on Sonnet and Opus, befor..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 17, 2026
Record: the setup text of #16229 reads non-inferior on Sonnet over the 12
se-*cases (this session's reading, not an owner decision).59d6f9d110against after08fcb9062e; the setup text, schema and the 12 cases are unchanged on main since, so the pair is the issue's own. Sonnet, 3 runs a side, both sides in one launch, judgeclaude-opus-5-5, Claude Code 2.1.326.noise-report.pyat margin 0.05. Result: paired change over 12 cases -0.01, 95% interval -0.04 to +0.03, non-inferior at margin 0.05; no case drop; no run excluded.legacy-setup-2,legacy-setup-3,legacy-setup-4,offers-alternative,timeout-not-derived,retired-info) measure nothing about this change either way; they are a case-quality finding, not a loss.Per-grader pass counts per side
Sonnet gate (batches 2 and 3)
Sonnet BEFORE (59d6f9d): batch 2, file: OK
| Case | Grader | Passed (launched) | Passed (counted) |
|---|---|---|---|
| se-apply-proposes-tool-writes-config | arid-record-written | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | config-written | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | diff-target-from-origin-head | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | legacy-setup-2 | 0/3 | 0/3 |
| se-apply-proposes-tool-writes-config | no-tool-install | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | legacy-setup-11 | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | names-headless-configure | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | names-plugin-configure | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-edit | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-git-add-or-commit | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-test-run | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-write | 3/3 | 3/3 |
| se-arid-entry-missing-keys | legacy-setup-5 | 2/3 | 2/3 |
| se-arid-entry-missing-keys | names-missing-keys | 3/3 | 3/3 |
| se-arid-entry-missing-keys | reports-malformed | 3/3 | 3/3 |
| se-arid-entry-missing-keys | runs-suppression-lint | 3/3 | 3/3 |
| se-check-reports-every-probe | legacy-setup-1 | 3/3 | 3/3 |
| se-check-reports-every-probe | no-edit | 3/3 | 3/3 |
| se-check-reports-every-probe | no-write | 3/3 | 3/3 |
| se-check-reports-every-probe | pass-fail-info-table | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-baseline | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-config-layers | 2/3 | 2/3 |
| se-check-reports-every-probe | probe-diff-target | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-ecosystem | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-flakiness | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-runner | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-suppression-record | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-tool | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-tracked | 3/3 | 3/3 |
| se-check-unconfigured-info | config-info | 3/3 | 3/3 |
| se-check-unconfigured-info | legacy-setup-10 | 3/3 | 3/3 |
| se-check-unconfigured-info | names-apply | 3/3 | 3/3 |
| se-check-unconfigured-info | names-fallback-manual | 3/3 | 3/3 |
| se-check-unconfigured-info | no-edit | 3/3 | 3/3 |
| se-check-unconfigured-info | no-write | 3/3 | 3/3 |
| se-gradle-name-filter-notice | exercised-refuses-notice | 2/3 | 2/3 |
| se-gradle-name-filter-notice | legacy-setup-9 | 1/3 | 1/3 |
| se-gradle-name-filter-notice | names-name-filter | 3/3 | 3/3 |
| se-gradle-name-filter-notice | proposes-pit | 3/3 | 3/3 |
| se-no-score-threshold | legacy-setup-4 | 0/3 | 0/3 |
| se-no-score-threshold | no-threshold-edit | 3/3 | 3/3 |
| se-no-score-threshold | no-threshold-written | 3/3 | 3/3 |
| se-no-score-threshold | offers-alternative | 0/3 | 0/3 |
| se-personal-arid-not-applied | info-not-fail | 3/3 | 3/3 |
| se-personal-arid-not-applied | legacy-setup-6 | 0/3 | 0/3 |
| se-personal-arid-not-applied | personal-only-not-applied | 3/3 | 3/3 |
| se-personal-arid-not-applied | promotion-and-floor | 3/3 | 3/3 |
| se-red-baseline-still-configures | baseline-fail | 2/3 | 2/3 |
| se-red-baseline-still-configures | config-written | 3/3 | 3/3 |
| se-red-baseline-still-configures | legacy-setup-3 | 0/3 | 0/3 |
| se-red-baseline-still-configures | timeout-not-derived | 0/3 | 0/3 |
| se-retired-team-file-migrates | config-written | 3/3 | 3/3 |
| se-retired-team-file-migrates | keys-carried | 3/3 | 3/3 |
| se-retired-team-file-migrates | legacy-setup-12 | 3/3 | 3/3 |
| se-retired-team-file-migrates | no-git-add-or-commit | 3/3 | 3/3 |
| se-retired-team-file-migrates | old-file-kept | 3/3 | 3/3 |
| se-retired-team-file-migrates | retired-info | 0/3 | 0/3 |
| se-test-command-needs-placeholder | legacy-setup-8 | 3/3 | 3/3 |
| se-test-command-needs-placeholder | no-edit | 3/3 | 3/3 |
| se-test-command-needs-placeholder | no-write | 3/3 | 3/3 |
| se-test-command-needs-placeholder | proposes-placeholder | 3/3 | 3/3 |
| se-test-command-needs-placeholder | test-command-fail | 3/3 | 3/3 |
| se-untracked-config-offers-add | legacy-setup-7 | 3/3 | 3/3 |
| se-untracked-config-offers-add | no-git-add-or-commit | 3/3 | 3/3 |
| se-untracked-config-offers-add | offers-git-add | 3/3 | 3/3 |
| se-untracked-config-offers-add | probe-check-ignore | 3/3 | 3/3 |
| se-untracked-config-offers-add | probe-ls-files | 3/3 | 3/3 |
| se-untracked-config-offers-add | untracked-fail | 3/3 | 3/3 |
Sonnet AFTER (08fcb90): batch 3, file: OK
| Case | Grader | Passed (launched) | Passed (counted) |
|---|---|---|---|
| se-apply-proposes-tool-writes-config | arid-record-written | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | config-written | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | diff-target-from-origin-head | 3/3 | 3/3 |
| se-apply-proposes-tool-writes-config | legacy-setup-2 | 0/3 | 0/3 |
| se-apply-proposes-tool-writes-config | no-tool-install | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | legacy-setup-11 | 2/3 | 2/3 |
| se-apply-user-layer-no-repo-write | names-headless-configure | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | names-plugin-configure | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-edit | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-git-add-or-commit | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-test-run | 3/3 | 3/3 |
| se-apply-user-layer-no-repo-write | no-write | 3/3 | 3/3 |
| se-arid-entry-missing-keys | legacy-setup-5 | 2/3 | 2/3 |
| se-arid-entry-missing-keys | names-missing-keys | 3/3 | 3/3 |
| se-arid-entry-missing-keys | reports-malformed | 3/3 | 3/3 |
| se-arid-entry-missing-keys | runs-suppression-lint | 3/3 | 3/3 |
| se-check-reports-every-probe | legacy-setup-1 | 3/3 | 3/3 |
| se-check-reports-every-probe | no-edit | 3/3 | 3/3 |
| se-check-reports-every-probe | no-write | 3/3 | 3/3 |
| se-check-reports-every-probe | pass-fail-info-table | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-baseline | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-config-layers | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-diff-target | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-ecosystem | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-flakiness | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-runner | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-suppression-record | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-tool | 3/3 | 3/3 |
| se-check-reports-every-probe | probe-tracked | 3/3 | 3/3 |
| se-check-unconfigured-info | config-info | 3/3 | 3/3 |
| se-check-unconfigured-info | legacy-setup-10 | 3/3 | 3/3 |
| se-check-unconfigured-info | names-apply | 3/3 | 3/3 |
| se-check-unconfigured-info | names-fallback-manual | 3/3 | 3/3 |
| se-check-unconfigured-info | no-edit | 3/3 | 3/3 |
| se-check-unconfigured-info | no-write | 3/3 | 3/3 |
| se-gradle-name-filter-notice | exercised-refuses-notice | 1/3 | 1/3 |
| se-gradle-name-filter-notice | legacy-setup-9 | 0/3 | 0/3 |
| se-gradle-name-filter-notice | names-name-filter | 3/3 | 3/3 |
| se-gradle-name-filter-notice | proposes-pit | 3/3 | 3/3 |
| se-no-score-threshold | legacy-setup-4 | 0/3 | 0/3 |
| se-no-score-threshold | no-threshold-edit | 3/3 | 3/3 |
| se-no-score-threshold | no-threshold-written | 3/3 | 3/3 |
| se-no-score-threshold | offers-alternative | 0/3 | 0/3 |
| se-personal-arid-not-applied | info-not-fail | 3/3 | 3/3 |
| se-personal-arid-not-applied | legacy-setup-6 | 1/3 | 1/3 |
| se-personal-arid-not-applied | personal-only-not-applied | 3/3 | 3/3 |
| se-personal-arid-not-applied | promotion-and-floor | 3/3 | 3/3 |
| se-red-baseline-still-configures | baseline-fail | 3/3 | 3/3 |
| se-red-baseline-still-configures | config-written | 3/3 | 3/3 |
| se-red-baseline-still-configures | legacy-setup-3 | 0/3 | 0/3 |
| se-red-baseline-still-configures | timeout-not-derived | 0/3 | 0/3 |
| se-retired-team-file-migrates | config-written | 3/3 | 3/3 |
| se-retired-team-file-migrates | keys-carried | 3/3 | 3/3 |
| se-retired-team-file-migrates | legacy-setup-12 | 3/3 | 3/3 |
| se-retired-team-file-migrates | no-git-add-or-commit | 3/3 | 3/3 |
| se-retired-team-file-migrates | old-file-kept | 3/3 | 3/3 |
| se-retired-team-file-migrates | retired-info | 0/3 | 0/3 |
| se-test-command-needs-placeholder | legacy-setup-8 | 3/3 | 3/3 |
| se-test-command-needs-placeholder | no-edit | 3/3 | 3/3 |
| se-test-command-needs-placeholder | no-write | 3/3 | 3/3 |
| se-test-command-needs-placeholder | proposes-placeholder | 2/3 | 2/3 |
| se-test-command-needs-placeholder | test-command-fail | 3/3 | 3/3 |
| se-untracked-config-offers-add | legacy-setup-7 | 3/3 | 3/3 |
| se-untracked-config-offers-add | no-git-add-or-commit | 3/3 | 3/3 |
| se-untracked-config-offers-add | offers-git-add | 3/3 | 3/3 |
| se-untracked-config-offers-add | probe-check-ignore | 3/3 | 3/3 |
| se-untracked-config-offers-add | probe-ls-files | 3/3 | 3/3 |
| se-untracked-config-offers-add | untracked-fail | 3/3 | 3/3 |
verdict: sonnet PASS; opus not run