You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
evals: changes merged without their eval while the gate is suspended (2026-11-21) #18018
On 2026-11-21 the owner suspended the before-and-after eval gate until open issues and PRs reach zero. Typed in the hub session at ~19:1xZ, verbatim: "just fucking stop the evals, get the shit pushed in and merged. I want to get issues and PRs down to zero, and then we're just going to have to focus on the stuff that's a priority, like evals". So these changes merge on the code and security review lanes without the no-loss eval that D367 asked for.
After open issues and PRs reach zero, the owner decides which eval suites are worth keeping, using the eval-value measurement in the hub's .work/eval-value/REPORT.md. For each kept suite, its skipped changes above are evaluated, or recorded as accepted without one.
Any PR that closes one of the issues above after today is added here.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Blast radius: this tracking issue only. It records which changes merged without an eval and who reads it. No code or workflow is touched until the owner picks the suites to keep.
Reversible: yes; edit or close the record. Any later evals or accepted-without-eval notes are separate comments.
Open questions: none
I read this as a tracking record. The eval gate is suspended, and the issues and PRs listed merge on code and security review alone. The follow-up is to decide, once open issues and PRs reach zero, which eval suites to keep, using the eval-value report. Each kept suite then gets the skipped changes evaluated or recorded as accepted without an eval. No duplicate is open. It relates to #16756, which is listed in it. The follow-up is deferred by design until the open count reaches zero.
Eval gate value, measured 2026-11-21 (read-only, over 2026-11-07 to 2026-11-21; gating actually ran about 4.5 days, 11-17 to 11-21). Data and scripts: the hub's .work/eval-value/.
Limit: a PASS cannot show a cut lost nothing. Each record says its cases have "not been shown to detect wrong text".
Share of wait: eval holds were 8.0 of 107.3 item-hours (7.5%) in the 17:17Z heat map window. Build (46.6) and admission (24.0) are larger.
Starting point when evals return: keep a suite only where it has caught a real change (the five above), fix graders before gating on them, and stop gating cuts whose cases cannot detect wrong text.
Adding from the handoff-redesign chain (#17877, which closes in favor of this issue):
PR feat(session-flow): compact within a task and hand off at a phase or task boundary #18022 (continuation router: /compact within a task, handoff then /clear at a phase or task boundary, ADR 0116) merged as 7953f48 without its eval. Its gate, per the owner on 2026-11-21, is a paired smoke: 17 resume-eval cases x after and compact x 2 runs = 68 runs. It passes when, on every case or all but one, compact is no worse than after on same-task cases and after is no worse than compact on boundary cases. Also the three wf-* routing cases (wf-same-task-compacts, wf-phase-boundary-hands-off, wf-unrelated-next-plain-clear) through /evals:plugin-eval.
Still open from feat(session-flow): finish the handoff shape-4 rollout after #17677 #17877: the parked live-reader resume-eval cases (need gh in the eval sandbox), and the real-session check AC9 (median context at the tenth resumed message and binding misses, by session-flow version, after the shape-4 and router releases).
Optional (owner, 2026-11-21): the statistical A/A before/after comparison, the before + A1 arm, and Codex reader runs.
The plugin copies are rebuilt from recorded commits: before e9a9d70, after 9fbd9b5, harness df315ae.
This issue now owns all remaining handoff-redesign work. The session handoff chain (handoff-redesign, 15 hops) is closed: shape 4 shipped in session-flow 9.19.0, and the router (#18022, ADR 0116) ships with release PR #18036.
Activity
github-actions commentedon Nov 21, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Blast radius: this tracking issue only. It records which changes merged without an eval and who reads it. No code or workflow is touched until the owner picks the suites to keep.
Reversible: yes; edit or close the record. Any later evals or accepted-without-eval notes are separate comments.
Open questions: none
I read this as a tracking record. The eval gate is suspended, and the issues and PRs listed merge on code and security review alone. The follow-up is to decide, once open issues and PRs reach zero, which eval suites to keep, using the eval-value report. Each kept suite then gets the skipped changes evaluated or recorded as accepted without an eval. No duplicate is open. It relates to #16756, which is listed in it. The follow-up is deferred by design until the open count reaches zero.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 21, 2026
The triage verdict left this issue at
needs-owner-decision:Reason:
intentkyle-sexton commentedon Nov 21, 2026
Eval gate value, measured 2026-11-21 (read-only, over 2026-11-07 to 2026-11-21; gating actually ran about 4.5 days, 11-17 to 11-21). Data and scripts: the hub's
.work/eval-value/.Starting point when evals return: keep a suite only where it has caught a real change (the five above), fix graders before gating on them, and stop gating cuts whose cases cannot detect wrong text.
kyle-sexton commentedon Nov 21, 2026
Adding from the handoff-redesign chain (#17877, which closes in favor of this issue):
/compactwithin a task, handoff then/clearat a phase or task boundary, ADR 0116) merged as 7953f48 without its eval. Its gate, per the owner on 2026-11-21, is a paired smoke: 17 resume-eval cases xafterandcompactx 2 runs = 68 runs. It passes when, on every case or all but one, compact is no worse than after on same-task cases and after is no worse than compact on boundary cases. Also the threewf-*routing cases (wf-same-task-compacts,wf-phase-boundary-hands-off,wf-unrelated-next-plain-clear) through/evals:plugin-eval.ghin the eval sandbox), and the real-session check AC9 (median context at the tenth resumed message and binding misses, by session-flow version, after the shape-4 and router releases).before + A1arm, and Codex reader runs.The plugin copies are rebuilt from recorded commits: before e9a9d70, after 9fbd9b5, harness df315ae.
kyle-sexton commentedon Nov 21, 2026
This issue now owns all remaining handoff-redesign work. The session handoff chain (handoff-redesign, 15 hops) is closed: shape 4 shipped in session-flow 9.19.0, and the router (#18022, ADR 0116) ships with release PR #18036.
Remaining, unchanged:
.work/handoff-audit/eval-scratch-removed.mdin the main checkout. A fail may mean reverting feat(session-flow): compact within a task and hand off at a phase or task boundary #18022.wf-*routing cases through/evals:plugin-eval.Close this issue when all three are done.