You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(verification): #16228 reads a Sonnet case drop and a plugin-copy denial asymmetry in the #16063 compare #16356
The #16063 two-version compare of #16228 reads NOT MET for verification on Sonnet on two pass-rule items, though the paired change over 43 cases is non-inferior (+0.03, 95% interval -0.01 to +0.07):
Case drop: confirm-proof-level-user-strict-wins fell from 0.89 to 0.56 on the with arm (report-line-strict failed 2 after-side runs, user-strict-boxes 1 more than before; one judge split vote, 8 of 9).
Plugin-copy denial asymmetry: one after-side run (batch 58, confirm-prep-light-does-not-block run 1) had a sandbox-denied Glob for **/setup-apply.mjs at the after plugin copy, and no before-side run had one.
Opus is MET on the same cases.
The numeric bars below are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
Any text change goes in its own pull request, and a before and after eval of verification on Sonnet over the 43 cases, 6 runs per side for the dropped case, both sides run together, is recorded and shows no case drop and no denial asymmetry; or the record states why the drop is noise, with evidence.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a follow-up to the #16063 compare of #16228 on Sonnet. It has two parts:
Case drop: confirm-proof-level-user-strict-wins fell from 0.89 to 0.56.
Denied Glob: one after-side run had a sandbox-denied Glob at the plugin copy, and no before-side run did.
The work is to find the cause of each from the run records and to fix the text if the text is at fault. A before and after Sonnet eval over the 43 cases follows, or a recorded reason that the drop is noise.
Nothing is needed from the reporter. Reading the runs should settle the cause. The paired eval is a host-run step, which admission routes separately.
Related but not duplicates: #16355 is the same kind of Sonnet compare loss for #16228 in the toolchain plugin. #16063 is the parent change. No recently merged pull request already fixes this.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "Any text change goes in its own pull request, and a before and after eval of ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Owner, 2026-11-17, typed in the GitHub Work session in answer to "does this issue close without the 43-case compare?", verbatim: "go w/ recommendations". The recommendation shown to him was to close on the two-case record. The alternatives shown were to run the 43-case compare first, or to leave the issue open.
Eval on the feat(toolchain): accept any recorded or inferred ecosystem id in check, lint and confirm #16228 pair (Sonnet, registered before any run, both sides on the fixed case files): confirm-proof-level-user-strict-wins 1.00 before and after, both graders 6 of 6 a side; confirm-prep-light-does-not-block 0 counted-denial runs on both sides, 3 runs a side. Verdict: Sonnet PASS; Opus not run. Cost 2.89 USD.
Not run: done-when item 3's before and after compare over all 43 cases. No skill text changed, so the issue's "any text change goes in its own pull request" did not apply.
Left for a later change, not filed: giving setup-apply.mjs a path in the confirm skill.
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
I read this as a follow-up to the #16063 compare of #16228 on Sonnet. It has two parts:
confirm-proof-level-user-strict-winsfell from 0.89 to 0.56.The work is to find the cause of each from the run records and to fix the text if the text is at fault. A before and after Sonnet eval over the 43 cases follows, or a recorded reason that the drop is noise.
Nothing is needed from the reporter. Reading the runs should settle the cause. The paired eval is a host-run step, which admission routes separately.
Related but not duplicates: #16355 is the same kind of Sonnet compare loss for #16228 in the toolchain plugin. #16063 is the parent change. No recently merged pull request already fixes this.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "Any text change goes in its own pull request, and a before and after eval of ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).test(verification): describe the strict-wins case's graders as they n…
kyle-sexton commentedon Nov 17, 2026
Decision recorded.
Owner, 2026-11-17, typed in the GitHub Work session in answer to "does this issue close without the 43-case compare?", verbatim: "go w/ recommendations". The recommendation shown to him was to close on the two-case record. The alternatives shown were to run the 43-case compare first, or to leave the issue open.
Record (#16684, merged as 4139bb7):
report-line-strictcould not match a bold-colon line or a table row; three stored replies it wrongly failed are quoted in test(verification): let the proof-level strict graders read bold-colon lines and open boxes #16684. The judge graderuser-strict-boxesvoted the same content both ways ("Empty." cells failed in one run and passed in another). Both graders are fixed there, with new pass and fail samples; the real miss (the layer on the next sub-bullet) still fails.setup-apply.mjs, so it is not an effect of the feat(toolchain): accept any recorded or inferred ecosystem id in check, lint and confirm #16228 text. The confirm skill names that file without a path, which is what sends runs searching.confirm-proof-level-user-strict-wins1.00 before and after, both graders 6 of 6 a side;confirm-prep-light-does-not-block0 counted-denial runs on both sides, 3 runs a side. Verdict: Sonnet PASS; Opus not run. Cost 2.89 USD.Not run: done-when item 3's before and after compare over all 43 cases. No skill text changed, so the issue's "any text change goes in its own pull request" did not apply.
Left for a later change, not filed: giving
setup-apply.mjsa path in the confirm skill.Closing. Posted by an agent session.