Skip to content

test(mutation-testing): #16229's Sonnet compare cannot decide non-inferiority on 3 cases in #16063 #16357

Closed

Description

@kyle-sexton

Refs: #16063

The #16063 two-version compare of #16229 reads NOT MET for mutation-testing on Sonnet: over 3 cases the paired with-arm change is +0.04 with a 95% interval of -0.28 to +0.35 (noise-report.py, not shown non-inferior). The interval is wide because there are only 3 cases at 3 runs per side; au-light-survivors-unclassified fell .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2 failing runs before/after) and au-no-test-command-asks rose .67 to 1.00. Opus is non-inferior (+0.00).

The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

  • The after-side runs of au-light-survivors-unclassified on Sonnet are read against the two graders, and the cause is stated with quotes: the feat(mutation-testing): accept any mutation tool id and name one with no shipped adapter #16229 text, the case, or the grader.
  • A before and after eval of mutation-testing on Sonnet, sized so the interval can decide the 0.05 margin (more runs per case, or more cases that load the changed text), both sides run together, is recorded and reads non-inferior at margin 0.05; or the record states why that cannot be shown with the cases available.

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: the #16063 two-version compare of #16229 could not show mutation-testing non-inferior on Sonnet. Over 3 cases the paired change is +0.04, with a 95% interval of -0.28 to +0.35. The issue asks for two things. First, read the after-side au-light-survivors-unclassified runs against both graders and state the cause with quotes. Second, record a larger before and after Sonnet eval that reads non-inferior at margin 0.05, or record why the available cases cannot show that.

    Nothing is missing that blocks triage. The eval runs may need a host run, which the lane routes separately.

    Possibly related, but it targets different cases (the 12 se- cases): #16358. I found no merged pull request that already covers this.

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "A before and after eval of mutation-testing on Sonnet, sized so the interval ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. kyle-sexton commented on Nov 17, 2026

    @kyle-sexton
    MemberAuthor

    Record: the drop is not the #16229 text, and non-inferiority at margin 0.05 cannot be shown with the cases available (this session's reading, not an owner decision).

    Read from the kept #16063 compare: the six Sonnet runs of au-light-survivors-unclassified and their traces, side taken from the file stem.

    Side, run Score Failing graders What the trace shows
    before 1 1.00 none
    before 2 1.00 none
    before 3 0.33 survivor-unclassified, writes-skip-token A shell glob error (no matches found: .claude/*.yaml) hid the listing; the reply says "No mutation-testing config exists at any layer", so the light level was never read.
    after 1 0.67 writes-skip-token Read the config and skipped triage ("so I skipped the fresh-context triage. The survivor is unclassified"), but left out the literal verification: skipped (light) token.
    after 2 1.00 none
    after 3 0.00 all three Same glob error (no matches found: .claude/mutation*); "No mutation-testing config exists in the repo"; ran at the default level and dispatched a "Triage surviving mutant" agent.

    First done-when item, the cause:

    Second done-when item, by its second branch ("or the record states why that cannot be shown with the cases available"):

    Checked by this session directly: the quoted trace text of before run 3, after run 1 and after run 3, and that #16229's audit-skill diff has no triage line. Relayed from a worker's read and not re-run by me: the simulation figures and the case-loading survey.

    Found while reading, not fixed here: the audit skill's Phase 0 config bullet does not name the team file path, and a glob error in the same shell call can hide the folder listing. Naming docs/conventions/mutation-testing.yaml in that bullet would address the miss seen on both sides; it would be a text change with its own before and after case.

  7. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions