Skip to content

Accept any recorded or inferred ecosystem id and any mutation tool value - part 4 of 4 #16063

Closed

Description

@melodic-automation-lanes

Part 4 of 4 of #15710, split at intake (owner Q93). The done-when below is copied from #15710 as written.

Done when

  • A full /testing:audit scan of this repository shows zero changed findings before and after (dogfood).
  • Each touched script's test file passes; the two-version compare is non-inferior and its record is in the pull request body.
  • AC7: nothing is cut here, so no control result is owed.
  • AC10: no new enforcement-placing skill is added here.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 14, 2026
  2. github-actions commented on Nov 14, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: #16355, #16356, #16357, #16358

    Decision recorded.
    Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.

    • Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
    • Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
    • Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.

    Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.

    Acceptance criteria

    #16063 (comment)

    This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 14, 2026
  4. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    ContributorAuthor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    ContributorAuthor

    {"v":1,"actor_id":337595936,"title_sha256":"c403775406470bfeca5bde4219ef8256e01fafcb209620635796475088871d96","body_sha256":"b9ff2193f8cd973eaf3a3c019a49d8cb678efe1db69545ed6ee720f2582c25e0"}

  6. 5 remaining items

  7. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    Member

    #16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)

    Computed by the registered result.py and noise-report.py (--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).

    Versions. toolchain and verification: BEFORE 7645e1c6ac, AFTER d87f47101e (#16228). mutation-testing: BEFORE 59d6f9d110, AFTER 08fcb9062e (#16229). Six plugin copies built with git archive; diff -rq showed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.

    Not compared. The mutation-testing setup-skill text (setup/SKILL.md:73-76,96-98, setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12 se-* cases were not run. A follow-up issue carries them.

    Verdicts

    Compare Under amendments B, C, D By the registration's letter
    toolchain sonnet NOT MET: -0.01, 95% interval -0.06 to +0.04, not shown non-inferior NOT MET (C's letter also drops ck-consumer-config-wins: -0.01, -0.06 to +0.05)
    toolchain opus MET: +0.00, -0.03 to +0.03, non-inferior (see the batch-225 note) NOT MET: batch 225 void, ck-clean-tree-branch-diff dropped (18 cases: +0.01, -0.02 to +0.04)
    verification sonnet NOT MET: +0.03, -0.01 to +0.07 with case drop confirm-proof-level-user-strict-wins 0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0) NOT MET: batches 39 and 40 void
    verification opus MET: +0.04, +0.00 to +0.07 same
    mutation-testing sonnet NOT MET: +0.04, -0.28 to +0.35, not shown non-inferior (3 cases) same
    mutation-testing opus MET: +0.00, +0.00 to +0.00 same

    Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.

    Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep changed-read); BEFORE leaks 0.

    New cases (by design, AFTER only): ck-recorded-filter-runs and ln-inferred-filter meet the bar on both models; every grader passed 3 of 3.

    Amendments

    • B (owner, Q167 (a)): run 2 of batches 39 and 40 (confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.
    • C (hub, written after the event; confirmed by the owner, Q177 (a)): the batch-95 launch (sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, both error: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.
    • D (owner, Q173 (a)): run 3 of batch 225 (ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.
    • How B and D were applied: a result-step helper copied each affected file with only the errored run removed (it refuses a run with no error); unchanged copies of exclude.py (--runs 2 --min 2) and result.py then ran; all three files read VALID. Each exclusion was applied on the side where it occurred.

    Run-level exclusions for sandbox denials (D299; each denied call read from its trace)

    Batch Compare, side Case, run Denied call
    58 verification sonnet AFTER confirm-prep-light-does-not-block r1 Glob **/setup-apply.mjs at the AFTER copy
    97 toolchain sonnet BEFORE ck-detects-changed-ecosystems r3 Read context/dotnet.md, context/python.md
    106 toolchain sonnet AFTER ck-skips-lint-only-surfaces r2 Read context/bash.md
    162 verification opus BEFORE confirm-intent-from-plan-brief r4 Bash sed -n ... reference/config.md
    168 verification opus BEFORE confirm-light-skips-verifier r4 Glob of the copy root (a Read of a mistyped path not counted at the copy)
    189 verification opus AFTER confirm-prep-light-does-not-block r2, r5 Glob setup-apply.mjs; Bash sed ... reference/config.md
    210 verification opus BEFORE confirm-retired-assumption-confirmed r3 Grep of the copy root

    No case fell below its minimum (confirm-prep-light-does-not-block opus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.

    Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)

    toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.

    toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).

    verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.

    verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.

    mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.

    Trace checks (tracegrep.py plus a bounded grep over all 936 kept traces): no is not a valid shim or Shell command permission check failed hits. The only command not found hits are cargo in ln-inferred-filter (sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.

    Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.

    Cost and time (file-level costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)

    Model Spent Judge share Expected Worst
    sonnet 63.70 USD 12.71 86.25 288.34
    opus 153.50 USD 18.41 201.49 465.07
    total 217.20 USD, plus 0.22 for the killed batch-95 attempt 287.74 753.41

    Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.

    Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.

    Follow-ups. #16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).

    session: GitHub Support

  8. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    Member

    Decision recorded.
    Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.

    • Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
    • Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
    • Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.

    Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.

    Acceptance criteria

    session: GitHub Support

  9. kyle-sexton commented on Nov 17, 2026

    @kyle-sexton
    Member

    Decision recorded.

    Owner, 2026-11-17, typed in the GitHub Work session in answer to "does this issue close on its four follow-up records?", verbatim: "go w/ recommendations". The recommendation shown to him was to close it with every gap stated; the alternatives shown were to run the dogfood scan first, to run the paid compares first, or to amend the done-when. He was also told that #16355 and #16357 were closed by a session without his decision.

    Re-read of this issue's done-when against the follow-up records, as the Q178 (a) decision set out. The original compare record is here: Opus non-inferior for all three plugins; Sonnet not shown non-inferior for toolchain, verification and mutation-testing.

    Per plugin that lost on Sonnet

    Not run, or not met as written

    • The Sonnet compare was not re-run in its registered shape for any of the three plugins. The 43-case verification compare, the 19-case toolchain compare at 6 runs a side, and a sized mutation-testing compare were not run.
    • Opus was not run on the 12 setup cases.
    • The compare record is in this issue's comments, not in a pull request body: this part shipped no pull request, and #16228 and #16229 say the compare was not run there.
    • A full /testing:audit scan across the whole change was not run. The only dogfood scan is #16231's, before and after its own adapter change: 120 findings both times over 2,404 test files. The new test file from feat(mutation-testing): accept any mutation tool id and name one with no shipped adapter #16229 was not covered by any scan.
    • The touched test files were left to CI; CI Status and the Bash test shards were green on #16229 and #16231. No record names the files run.
    • AC7 and AC10 hold: nothing was cut here, and no skill was added.

    Closing. Posted by an agent session.

  10. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions