Repository navigation
audit.VerifierBoost: bootstrap gate activates a curve that is worse than a constant predictor (a pure bias shift passes 'validated') #306
Description
Symptom
VerifierBoost.retrain() (merged in #273) activates a calibration curve whenever the validation MSE beats the raw verifier's by min_gain. That gate measures calibration error only, and a pure bias shift satisfies it. A verifier that has no discriminating power at all (score uncorrelated with the oracle) but reads systematically high/low is therefore "validated", gets installed, and starts re-ranking candidates (cheap_eval → conflict resolution, fusion tournament, audit priority) with a curve that is itself fit mostly to noise.
Recent work calls out exactly this failure for self-evolved verifiers: "Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents" (arXiv 2610.11464, Oct 2026) finds that removing the anchor guards collapses the evolved verifier into a vacuous always-pass grader that still looks fine on downstream score, and concludes "downstream task score cannot certify a self-evolved verifier"; verifiers there are gated at birth on agreement with an anchored reference set, not on the loop's own score. UCoE (arXiv 2610.17609) likewise requires a candidate verifier to beat the incumbent on discrimination over declared contrast pairs before adoption. Our bootstrap gate has the anchor set (the IMPROVEMENT pool) but compares against the wrong baseline.
Reproduction (main @ c44633c; offline, ~1 s)
import random, types
import numpy as np
from agentdescent.audit.boost import VerifierBoost
rng = random.Random(1)
# verifier score = pure noise in [0.8, 1.0]; the oracle passes 30% of the time
recs = [types.SimpleNamespace(resolved=True,
verifier_score=0.8 + 0.2 * rng.random(),
oracle_score=1.0 if rng.random() < 0.3 else 0.0)
for _ in range(400)]
class Store:
def for_improvement(self, version=None): return recs
class Inner:
full_eval_matches_counts = False
def cheap_eval(self, a): return a
def learned_eval(self, a): return a, 0.1
b = VerifierBoost(Inner(), Store())
s = b.status()
print(s["active"], s["error_before"], s["error_after"]) # True 0.583 0.244 -> "validated"
f = np.array([r.verifier_score for r in recs]); y = np.array([r.oracle_score for r in recs])
print(np.corrcoef(f, y)[0, 1]) # -0.062: no signal
print(np.mean((y - y.mean())**2)) # 0.200: the trivial constant predictor already beats the "validated" 0.244Output observed: active: True | error_before 0.583 -> error_after 0.244, corr = -0.062, constant-mean MSE 0.200. The installed curve is worse than predicting the training mean yet passes because the raw baseline (0.583) is badly biased. version_tag() flips to boost:active:<hash>, so VerifierWatch also withdraws the existing calibration for a change that carries no information.
Expected behaviour
The bootstrap gate activates a curve only if it demonstrably adds ranking information, i.e. it must beat the best constant predictor (train-slice mean of oracle_score) on the validation slice by min_gain, in addition to beating the raw verifier. A pure bias correction should be recorded as a rejection ("no discrimination beyond a constant") and the boost stays inactive; the constant offset is the audit's Calibrator job (delta_hat), not the ranking layer's.
Suggested fix
In VerifierBoost.retrain (agentdescent/audit/boost.py):
- Compute
error_const = _mse([mean(train_oracle)] * len(val), val_oracle). - Require
error_after <= error_const * (1 - min_gain)as well as the existingerror_beforetest; adderror_consttoBoostState/status()(defaultmath.infkeeps old pickles/consumers working) and put it in the rejectionreason. - Optionally also reject a curve with
< 2distinct fitted levels (degenerate PAV block) with an explicit reason.
Do not touch the delegation of eval_counts/full_eval (raw) or the #304 items (recompute oscillation, oracle_eval-only delegation) — keep this PR to the gate.
Acceptance criteria
- New test in
tests/test_verifier_boost.py: the repro above →is_active is False, reason mentions the constant baseline,version_tag() == "boost:inactive". - A control test: a verifier that is monotone-informative but miscalibrated (e.g. oracle = 1 when
f > 0.9, scores in [0.8, 1.0]) still activates, so the existing positive tests stay green and the gate is not merely tightened. -
status()exposes the new baseline error; existing keys unchanged. - One
changelog.d/<issue>-boost-constant-baseline.mdfragment (Fixed). -
pytest -qpasses offline.
Activity
fix(audit): preserve measured error_const on constant-baseline reject…
audit: VerifierBoost bootstrap gate must beat a constant-mean baseline (