fix: require one attempt to pass all fresh-evaluation gates - #2
Merged
Merged
Conversation
family_acceptance and family_clean were latched independently across attempts, so an acceptance-passing attempt followed by a clean-worker attempt combined into pass=true and exit 0 even though no single attempt met both gates (base a6cb005, evaluation/run_fresh_evaluation.sh:340-345, :362). Record each attempt as one record and compute the family pass only from one attempt: baseline failed as expected, harness_ok, independent acceptance pass, clean worker, failure_kind=none. The new attempts_acceptance_pass and attempts_worker_clean counters are evidence only and never feed the pass gate; a failed attempt still retries and does not stop other families. Add deterministic opposite_attempts fake-provider control (attempt 1 correct patch + step limit, attempt 2 weakened visible test + clean finish) and register it in run_controls.sh. Controls now 36 checks, 0 failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On base
a6cb005,evaluation/run_fresh_evaluation.shlatchedfamily_acceptanceandfamily_cleanindependently across attempts (:340-345,:362). An attempt that passed independent acceptance but failed the worker, followed by a retry that was worker-clean but failed acceptance, combined intopass=trueand exit 0 — no single attempt met both criteria.Reproduced deterministically against the base harness (see control below).
New behavior
<family>-attempts.tsv; the per-family record (<family>.result,summary.tsv) describes exactly one attempt: the passing one if any, otherwise the latest tried.harness_ok, independent acceptance pass, clean worker, andfailure_kind=none.attempts_acceptance_pass/attempts_worker_cleanare kept as evidence-only counters and never feed the pass gate.Regression control
New deterministic fake provider
evaluation/controls/fake_providers/opposite_attempts.sh(registered inrun_controls.sh): attempt 1 applies the correct1..n→1..=npatch and hits the step limit; attempt 2 cleanly finishes after weakening the visible test.patch_acceptance_pass=true,worker_clean_success=true,failure_kind=none, exit 0 (bug reproduced).family_pass=false, exit 1; attempt 1 =acceptance=true, clean=false, step_limit, attempt 2 =acceptance=false, clean=true, none, inevaluation/runs/controls-20260911T203623Z-opposite-attempts/.All pre-existing controls are kept: build failure, invalid baseline, acceptance failure, step limit, wall timeout, provider error path, same-RUN_ID evidence preservation, mixed families.
Test command
cargo fmt --check cargo check --locked --all-targets cargo clippy --locked --all-targets -- -D warnings cargo test --locked bash evaluation/run_controls.shObserved: fmt/check/clippy clean, 47 tests pass, controls 36 checks, 0 failures (was 28; the new opposite-attempts assertions add 8). Evidence:
evaluation/runs/controls-20260911T203623Z-*.Remaining limits
<family>-attempts.tsv.