You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Accept any recorded or inferred ecosystem id and any mutation tool value - part 4 of 4 #16063
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #16355, #16356, #16357, #16358
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
#16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)
Computed by the registered result.py and noise-report.py (--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).
Versions. toolchain and verification: BEFORE 7645e1c6ac, AFTER d87f47101e (#16228). mutation-testing: BEFORE 59d6f9d110, AFTER 08fcb9062e (#16229). Six plugin copies built with git archive; diff -rq showed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.
Not compared. The mutation-testing setup-skill text (setup/SKILL.md:73-76,96-98, setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12 se-* cases were not run. A follow-up issue carries them.
Verdicts
Compare
Under amendments B, C, D
By the registration's letter
toolchain sonnet
NOT MET: -0.01, 95% interval -0.06 to +0.04, not shown non-inferior
NOT MET (C's letter also drops ck-consumer-config-wins: -0.01, -0.06 to +0.05)
toolchain opus
MET: +0.00, -0.03 to +0.03, non-inferior (see the batch-225 note)
NOT MET: batch 225 void, ck-clean-tree-branch-diff dropped (18 cases: +0.01, -0.02 to +0.04)
verification sonnet
NOT MET: +0.03, -0.01 to +0.07 with case drop confirm-proof-level-user-strict-wins 0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0)
NOT MET: batches 39 and 40 void
verification opus
MET: +0.04, +0.00 to +0.07
same
mutation-testing sonnet
NOT MET: +0.04, -0.28 to +0.35, not shown non-inferior (3 cases)
same
mutation-testing opus
MET: +0.00, +0.00 to +0.00
same
Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.
Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep changed-read); BEFORE leaks 0.
New cases (by design, AFTER only): ck-recorded-filter-runs and ln-inferred-filter meet the bar on both models; every grader passed 3 of 3.
Amendments
B (owner, Q167 (a)): run 2 of batches 39 and 40 (confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.
C (hub, written after the event; confirmed by the owner, Q177 (a)): the batch-95 launch (sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, both error: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.
D (owner, Q173 (a)): run 3 of batch 225 (ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.
How B and D were applied: a result-step helper copied each affected file with only the errored run removed (it refuses a run with no error); unchanged copies of exclude.py (--runs 2 --min 2) and result.py then ran; all three files read VALID. Each exclusion was applied on the side where it occurred.
Run-level exclusions for sandbox denials (D299; each denied call read from its trace)
Batch
Compare, side
Case, run
Denied call
58
verification sonnet AFTER
confirm-prep-light-does-not-block r1
Glob **/setup-apply.mjs at the AFTER copy
97
toolchain sonnet BEFORE
ck-detects-changed-ecosystems r3
Read context/dotnet.md, context/python.md
106
toolchain sonnet AFTER
ck-skips-lint-only-surfaces r2
Read context/bash.md
162
verification opus BEFORE
confirm-intent-from-plan-brief r4
Bash sed -n ... reference/config.md
168
verification opus BEFORE
confirm-light-skips-verifier r4
Glob of the copy root (a Read of a mistyped path not counted at the copy)
189
verification opus AFTER
confirm-prep-light-does-not-block r2, r5
Glob setup-apply.mjs; Bash sed ... reference/config.md
210
verification opus BEFORE
confirm-retired-assumption-confirmed r3
Grep of the copy root
No case fell below its minimum (confirm-prep-light-does-not-block opus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.
Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)
toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.
toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).
verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.
verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.
mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.
Trace checks (tracegrep.py plus a bounded grep over all 936 kept traces): no is not a valid shim or Shell command permission check failed hits. The only command not found hits are cargo in ln-inferred-filter (sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.
Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.
Cost and time (file-level costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)
Model
Spent
Judge share
Expected
Worst
sonnet
63.70 USD
12.71
86.25
288.34
opus
153.50 USD
18.41
201.49
465.07
total
217.20 USD, plus 0.22 for the killed batch-95 attempt
287.74
753.41
Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.
Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.
Follow-ups.#16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
Owner, 2026-11-17, typed in the GitHub Work session in answer to "does this issue close on its four follow-up records?", verbatim: "go w/ recommendations". The recommendation shown to him was to close it with every gap stated; the alternatives shown were to run the dogfood scan first, to run the paid compares first, or to amend the done-when. He was also told that #16355 and #16357 were closed by a session without his decision.
Re-read of this issue's done-when against the follow-up records, as the Q178 (a) decision set out. The original compare record is here: Opus non-inferior for all three plugins; Sonnet not shown non-inferior for toolchain, verification and mutation-testing.
Per plugin that lost on Sonnet
toolchain, #16355: the four lost runs were read from their traces (record). None traces to the feat(toolchain): accept any recorded or inferred ecosystem id in check, lint and confirm #16228 text: two ended on a note after the result-recording script failed in the eval sandbox, one cut its own file listing short, one was judged against criteria that predate a status the skill already had. The same note-only ending occurs on the before side. Result: closed as noise, by the issue's second branch. No eval was run.
setup text not compared, #16358: the 12 setup cases on Sonnet read non-inferior (-0.01, 95% interval -0.04 to +0.03), with no case drop (36.14 USD) (record).
Not run, or not met as written
The Sonnet compare was not re-run in its registered shape for any of the three plugins. The 43-case verification compare, the 19-case toolchain compare at 6 runs a side, and a sized mutation-testing compare were not run.
Opus was not run on the 12 setup cases.
The compare record is in this issue's comments, not in a pull request body: this part shipped no pull request, and #16228 and #16229 say the compare was not run there.
Activity
github-actions commentedon Nov 14, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #16355, #16356, #16357, #16358
#16063 (comment)
This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 14, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 14, 2026
{"v":1,"actor_id":337595936,"title_sha256":"c403775406470bfeca5bde4219ef8256e01fafcb209620635796475088871d96","body_sha256":"b9ff2193f8cd973eaf3a3c019a49d8cb678efe1db69545ed6ee720f2582c25e0"}
5 remaining items
kyle-sexton commentedon Nov 15, 2026
#16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)
Computed by the registered
result.pyandnoise-report.py(--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).Versions. toolchain and verification: BEFORE
7645e1c6ac, AFTERd87f47101e(#16228). mutation-testing: BEFORE59d6f9d110, AFTER08fcb9062e(#16229). Six plugin copies built withgit archive;diff -rqshowed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.Not compared. The mutation-testing setup-skill text (
setup/SKILL.md:73-76,96-98,setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12se-*cases were not run. A follow-up issue carries them.Verdicts
-0.01, 95% interval -0.06 to +0.04, not shown non-inferiorck-consumer-config-wins:-0.01, -0.06 to +0.05)+0.00, -0.03 to +0.03, non-inferior (see the batch-225 note)ck-clean-tree-branch-diffdropped (18 cases:+0.01, -0.02 to +0.04)+0.03, -0.01 to +0.07with case dropconfirm-proof-level-user-strict-wins0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0)+0.04, +0.00 to +0.07+0.04, -0.28 to +0.35, not shown non-inferior (3 cases)+0.00, +0.00 to +0.00Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.
Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep
changed-read); BEFORE leaks 0.New cases (by design, AFTER only):
ck-recorded-filter-runsandln-inferred-filtermeet the bar on both models; every grader passed 3 of 3.Amendments
confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, botherror: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.exclude.py(--runs 2 --min 2) andresult.pythen ran; all three files read VALID. Each exclusion was applied on the side where it occurred.Run-level exclusions for sandbox denials (D299; each denied call read from its trace)
**/setup-apply.mjsat the AFTER copycontext/dotnet.md,context/python.mdcontext/bash.mdsed -n ... reference/config.mdsetup-apply.mjs; Bashsed ... reference/config.mdNo case fell below its minimum (
confirm-prep-light-does-not-blockopus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)
toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.
toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).
verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.
verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.
mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.
Trace checks (
tracegrep.pyplus a bounded grep over all 936 kept traces): nois not a valid shimorShell command permission check failedhits. The onlycommand not foundhits arecargoinln-inferred-filter(sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.
Cost and time (file-level
costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.
Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.
Follow-ups. #16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).
session: GitHub Support
kyle-sexton commentedon Nov 15, 2026
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
Acceptance criteria
session: GitHub Support
kyle-sexton commentedon Nov 17, 2026
Decision recorded.
Owner, 2026-11-17, typed in the GitHub Work session in answer to "does this issue close on its four follow-up records?", verbatim: "go w/ recommendations". The recommendation shown to him was to close it with every gap stated; the alternatives shown were to run the dogfood scan first, to run the paid compares first, or to amend the done-when. He was also told that #16355 and #16357 were closed by a session without his decision.
Re-read of this issue's done-when against the follow-up records, as the Q178 (a) decision set out. The original compare record is here: Opus non-inferior for all three plugins; Sonnet not shown non-inferior for toolchain, verification and mutation-testing.
Per plugin that lost on Sonnet
Not run, or not met as written
/testing:auditscan across the whole change was not run. The only dogfood scan is #16231's, before and after its own adapter change: 120 findings both times over 2,404 test files. The new test file from feat(mutation-testing): accept any mutation tool id and name one with no shipped adapter #16229 was not covered by any scan.Closing. Posted by an agent session.