Skip to content

fix: require one attempt to pass all fresh-evaluation gates - #2

Merged
rustfuture merged 1 commit into
mainfrom
fix/eval-attempt-consistency
Sep 11, 2026
Merged

rustfuture merged 1 commit into
mainfrom
fix/eval-attempt-consistency

Conversation

@rustfuture

Copy link
Copy Markdown
Owner

Problem

On base a6cb005, evaluation/run_fresh_evaluation.sh latched family_acceptance and family_clean independently across attempts (:340-345, :362). An attempt that passed independent acceptance but failed the worker, followed by a retry that was worker-clean but failed acceptance, combined into pass=true and exit 0 — no single attempt met both criteria.

Reproduced deterministically against the base harness (see control below).

New behavior

  • Each attempt is one record in <family>-attempts.tsv; the per-family record (<family>.result, summary.tsv) describes exactly one attempt: the passing one if any, otherwise the latest tried.
  • A family passes only when the same attempt has baseline failed as expected, harness_ok, independent acceptance pass, clean worker, and failure_kind=none.
  • attempts_acceptance_pass / attempts_worker_clean are kept as evidence-only counters and never feed the pass gate.
  • A failed attempt still triggers the controlled retry and does not stop other families.

Regression control

New deterministic fake provider evaluation/controls/fake_providers/opposite_attempts.sh (registered in run_controls.sh): attempt 1 applies the correct 1..n1..=n patch and hits the step limit; attempt 2 cleanly finishes after weakening the visible test.

  • Base harness: patch_acceptance_pass=true, worker_clean_success=true, failure_kind=none, exit 0 (bug reproduced).
  • Fixed harness: family_pass=false, exit 1; attempt 1 = acceptance=true, clean=false, step_limit, attempt 2 = acceptance=false, clean=true, none, in evaluation/runs/controls-20260911T203623Z-opposite-attempts/.

All pre-existing controls are kept: build failure, invalid baseline, acceptance failure, step limit, wall timeout, provider error path, same-RUN_ID evidence preservation, mixed families.

Test command

cargo fmt --check
cargo check --locked --all-targets
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked
bash evaluation/run_controls.sh

Observed: fmt/check/clippy clean, 47 tests pass, controls 36 checks, 0 failures (was 28; the new opposite-attempts assertions add 8). Evidence: evaluation/runs/controls-20260911T203623Z-*.

Remaining limits

  • The new fixture covers the two-attempt opposite case; other cross-attempt combinations (e.g. provider failure then acceptance failure) rely on the same per-attempt gate without a dedicated fixture.
  • Recorded per-family fields describe the passing/latest attempt only; full history stays in <family>-attempts.tsv.
  • No real-model evaluation was run for this change (per task); recorded real-run evidence is unchanged.

family_acceptance and family_clean were latched independently across attempts, so an acceptance-passing attempt followed by a clean-worker attempt combined into pass=true and exit 0 even though no single attempt met both gates (base a6cb005, evaluation/run_fresh_evaluation.sh:340-345, :362).

Record each attempt as one record and compute the family pass only from one attempt: baseline failed as expected, harness_ok, independent acceptance pass, clean worker, failure_kind=none. The new attempts_acceptance_pass and attempts_worker_clean counters are evidence only and never feed the pass gate; a failed attempt still retries and does not stop other families.

Add deterministic opposite_attempts fake-provider control (attempt 1 correct patch + step limit, attempt 2 weakened visible test + clean finish) and register it in run_controls.sh. Controls now 36 checks, 0 failures.
@rustfuture
rustfuture merged commit a9c6ad4 into main Sep 11, 2026
2 checks passed
@rustfuture
rustfuture deleted the fix/eval-attempt-consistency branch September 11, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant