Skip to content

evals: changes merged without their eval while the gate is suspended (2026-11-21) #18018

Open

Description

@kyle-sexton

Refs: #16756

Problem

On 2026-11-21 the owner suspended the before-and-after eval gate until open issues and PRs reach zero. Typed in the hub session at ~19:1xZ, verbatim: "just fucking stop the evals, get the shit pushed in and merged. I want to get issues and PRs down to zero, and then we're just going to have to focus on the stuff that's a priority, like evals". So these changes merge on the code and security review lanes without the no-loss eval that D367 asked for.

Issues whose eval-gated label was removed: #17773, #17771, #17767, #17764, #17763, #17761, #17736, #16765, #16762, #16760, #16758, #16756, #16755, #16752.

PRs whose eval hold was released: #17926, #17933, #17938, #17990. #17932 already had its eval PASS posted.

Done when

  • After open issues and PRs reach zero, the owner decides which eval suites are worth keeping, using the eval-value measurement in the hub's .work/eval-value/REPORT.md. For each kept suite, its skipped changes above are evaluated, or recorded as accepted without one.
  • Any PR that closes one of the issues above after today is added here.

Activity

  1. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Nov 21, 2026
  2. github-actions commented on Nov 21, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none
    Blast radius: this tracking issue only. It records which changes merged without an eval and who reads it. No code or workflow is touched until the owner picks the suites to keep.
    Reversible: yes; edit or close the record. Any later evals or accepted-without-eval notes are separate comments.
    Open questions: none
    I read this as a tracking record. The eval gate is suspended, and the issues and PRs listed merge on code and security review alone. The follow-up is to decide, once open issues and PRs reach zero, which eval suites to keep, using the eval-value report. Each kept suite then gets the skipped changes evaluated or recorded as accepted without an eval. No duplicate is open. It relates to #16756, which is listed in it. The follow-up is deferred by design until the open count reaches zero.

    Automated triage. A maintainer reviews it.

  3. melodic-automation-lanes commented on Nov 21, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • held by needs-human
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  4. kyle-sexton commented on Nov 21, 2026

    @kyle-sexton
    MemberAuthor

    Eval gate value, measured 2026-11-21 (read-only, over 2026-11-07 to 2026-11-21; gating actually ran about 4.5 days, 11-17 to 11-21). Data and scripts: the hub's .work/eval-value/.

    Starting point when evals return: keep a suite only where it has caught a real change (the five above), fix graders before gating on them, and stop gating cuts whose cases cannot detect wrong text.

  5. kyle-sexton commented on Nov 21, 2026

    @kyle-sexton
    MemberAuthor

    Adding from the handoff-redesign chain (#17877, which closes in favor of this issue):

    • PR feat(session-flow): compact within a task and hand off at a phase or task boundary #18022 (continuation router: /compact within a task, handoff then /clear at a phase or task boundary, ADR 0116) merged as 7953f48 without its eval. Its gate, per the owner on 2026-11-21, is a paired smoke: 17 resume-eval cases x after and compact x 2 runs = 68 runs. It passes when, on every case or all but one, compact is no worse than after on same-task cases and after is no worse than compact on boundary cases. Also the three wf-* routing cases (wf-same-task-compacts, wf-phase-boundary-hands-off, wf-unrelated-next-plain-clear) through /evals:plugin-eval.
    • Still open from feat(session-flow): finish the handoff shape-4 rollout after #17677 #17877: the parked live-reader resume-eval cases (need gh in the eval sandbox), and the real-session check AC9 (median context at the tenth resumed message and binding misses, by session-flow version, after the shape-4 and router releases).
    • Optional (owner, 2026-11-21): the statistical A/A before/after comparison, the before + A1 arm, and Codex reader runs.

    The plugin copies are rebuilt from recorded commits: before e9a9d70, after 9fbd9b5, harness df315ae.

  6. kyle-sexton commented on Nov 21, 2026

    @kyle-sexton
    MemberAuthor

    This issue now owns all remaining handoff-redesign work. The session handoff chain (handoff-redesign, 15 hops) is closed: shape 4 shipped in session-flow 9.19.0, and the router (#18022, ADR 0116) ships with release PR #18036.

    Remaining, unchanged:

    • Router smoke: 68 runs (17 cases x after and compact x 2), at most 2 workers, once evals return (after open issues and PRs reach zero). Rebuild recipe: .work/handoff-audit/eval-scratch-removed.md in the main checkout. A fail may mean reverting feat(session-flow): compact within a task and hand off at a phase or task boundary #18022.
    • The three wf-* routing cases through /evals:plugin-eval.
    • AC9 on real sessions after the router release: median context at the tenth resumed message and binding misses, by session-flow version.

    Close this issue when all three are done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.

    Type

    Fields

    Work status

    needs-human

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions