Skip to content

audit: VerifierBoost recompute oscillation and legacy full-evaluation delegation regressions #304

Closed

Description

@Birfy

Follow-up to merged #273

Tracks the two distinct findings already posted in this source review. Exact main is 58ee78fb943eaf36c60e1f0bcdfe3af3301a6dc1; boost.py and wiring.py are byte-identical to reviewed head feca0bd. These findings are source-confirmed, not locally reproduced.

1. Recompute can deactivate/reactivate a validated correction without new data

Audit.recompute trains against its current fingerprint, then refreshes that fingerprint to include the active curve. The next recompute filters labels to the new fingerprint. With no new labels, the insufficient-label branch deactivates the fit. Its tag returns to boost:inactive, reproducing the old fingerprint; another recompute can fit the original pool again.

Acceptance: separate underlying scorer/training identity from correction/measurement identity, and retain a validated correction when the unchanged scorer merely lacks sufficient new training data. Preserve withdrawal when the actual scorer changes.

Proposed test (not run): seed sufficient inactive-version IMPROVEMENT labels, activate via recompute, repeat recompute without adding records, then add a sub-min_labels batch. Assert the active curve/version remains stable rather than oscillating; separately exercise a real scorer-version change.

2. A supported oracle_eval-only verifier fails when wrapped, even inactive

Both wrapper aliases call inner.full_eval directly. The engine's full_eval_of compatibility helper explicitly supports legacy oracle_eval-only verifiers. With oracle_shares_full_set=False, wrapping preserves the independence flag but masks the legacy method behind the wrapper's full_eval, which then raises AttributeError.

Acceptance: delegate through the supported full-evaluation compatibility path (including the wrapper's legacy alias), preserving raw independent evaluation and equivalence declarations.

Proposed test (not run): use an oracle_eval-only verifier with oracle_shares_full_set=False and an empty boost store. The unwrapped and wrapped independent gates should execute the same scorer and refuse a candidate whose independent score regresses, rather than failing with AttributeError. Cover active and inactive correction states.

Scope

Dependency: #273 is merged. Keep the two acceptance checks separate; bounded changes are boost/audit wiring, focused offline regressions and a changelog fragment. No implementation, local materialization/tests, paid calls, release or workflow changes are performed by this filing. Hosted CI is not an independent reproduction of these paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions