[HugoXK]

feat(audit): verifier bootstrap flywheel — the audit's labels improve the verifier, under a gate

The problem

The audit package already turns labelled records into a correction of the loop: Calibrator computes a rectification (delta_hat, se, resid_sd) that RectifiedAcceptance uses to discount the gate's evidence. What it never does is turn those labels into a better verifier — the correction is subtracted at the decision, and the scorer itself is left exactly as it was. AuditedReward always returns the verifier's score, and no code path rewrites the verifier function the aggregator consumes from the labels the audit collects. That is the missing half of "the audit learns" — the frontier direction (Self-Rewarding LM, V-STaR) where a judge improves itself from its own labeled data.

The dangerous part is the same as the valuable part: a verifier that learns from its own labels could learn to be confident-but-wrong. This PR closes the loop under a bootstrap gate and two hard invariants, so the flywheel cannot install a worse verifier and cannot substitute itself for the oracle.

What this PR adds

agentdescent/audit/boost.py — VerifierBoost, a drop-in VerifierProtocol wrapper (satisfies the engine's verifier seam, Policies(verifier=...)):

• The learner. From the audit's IMPROVEMENT pool it fits a monotone calibration curve g: verifier_score -> oracle-consistent score (isotonic regression — pool-adjacent-violators, numpy-only, so score order is preserved, which is exactly what a ranking correction must not break).
• The bootstrap gate. retrain() fits on a tail slice of the pool and validates on a held-out slice: the curve activates only if E[(g(f)-Y)²] ≤ E[(f-Y)²]·(1 − min_gain). A curve that does not provably help is recorded as a rejection in status() and never installed.
• What it may touch. When active, cheap_eval and learned_eval return g(score) — the scores the aggregator uses to rank candidates (conflict resolution, the fusion tournament) and the audit uses for its trust/priority signal. learned_eval's uncertainty is raised to the boost's own validation residual, so the audit spends less oracle budget on a verifier that demonstrably tracks the oracle.
• What it never touches. eval_counts and full_eval are delegated raw — the commit gate keeps the oracle's authority over what commits. The correction never substitutes for the oracle.

The governance invariants

• Two-pool separation. It trains on the IMPROVEMENT pool only; the CALIBRATION pool (which exists to measure the verifier, not change it) never enters the fit. The store's own for_calibration assertion already enforces the split; the boost reads only for_improvement().
• Version honesty. version_tag() changes when the boost activates and is stable while active. A caller wiring it as the audit's version_extra makes VerifierWatch withdraw the old calibration and re-measure the boosted verifier with fresh CALIBRATION labels — so the flywheel's output is measured by the same audit that produced its training labels. A boost that is not actually better is caught twice: by the bootstrap gate at install time, and by the new version's calibration afterwards.
• No labels, no change. min_labels shortfall leaves the boost inactive; the wrapped verifier is returned raw. Default behaviour is unchanged.

Wiring. attach(..., boost=...) and Audit.recompute() retrain the boost when a batch of labels resolves — the moment a flywheel should learn. version_tag() is meant for attach(version_extra=...) so the audit's version tracks the boost.

Backward compatibility

Component	Default	Old behaviour?
attach(boost=)	None	yes — no fitting, no correction, Audit.recompute() unchanged
VerifierBoost	inactive without labels	yes — raw delegation, version_tag == "boost:inactive"
VerifierBoost.cheap_eval/learned_eval	raw	yes, until a validated fit activates
VerifierBoost.eval_counts/full_eval	raw	yes — always, active or not
Audit dataclass	boost=None	yes — new optional field
audit/__init__ exports	unchanged	yes — three new symbols

Verification

Item	Result
Isotonic fit is monotone and approximates a constant-bias target	1 test
Correction preserves score order	1 test
No labels / too few labels -> inactive	2 tests
Genuine bias is learned and validated; corrected score closer to oracle	1 test
A boost that does not help is rejected at the gate	1 test
min_gain raises the bar (lenient accepts, strict refuses)	1 test
Commit gate (eval_counts/full_eval/rule_eval) stays raw when active	1 test
Learned uncertainty reflects the validation residual	1 test
Training uses the IMPROVEMENT pool only (a CALIBRATION outlier never enters)	1 test
version_tag changes on activation and is stable while active	1 test
Audit.recompute() retrains an attached boost; no-boost recompute unchanged	2 tests
The boost is a valid engine verifier (Policies(verifier=...), commits normally)	1 test
docs/api.md regenerated, test_api_reference green	exit 0
Regression: 135 tests (audit gate/calibrator/attach + api)	all green

Files

7 files changed, +618, -6

New:

• agentdescent/audit/boost.py (+~330) — VerifierBoost, BoostState, isotonic_fit
• tests/test_verifier_boost.py (+~300) — 14 tests

Modified:

• agentdescent/audit/wiring.py — Audit.boost, attach(boost=), recompute() retrains
• agentdescent/audit/__init__.py — exports
• agentdescent/__init__.py — VerifierBoost, BoostState, isotonic_fit
• docs/api.md — regenerated
• CHANGELOG.md — entry

[Birfy]

Thanks for this, @HugoXK. Using audit labels to improve the ranking layers, gated on a held-out check and with the commit gate left raw, is a direction worth having, and the PR is clean and well documented. The isotonic learner is small and dependency-free. eval_counts/full_eval stay raw. attach(boost=None) is a true no-op. The 14 new tests pass locally (tests/test_verifier_boost.py, py3.13), and the docs build is green. The test matrix was still pending when I looked.

I'm requesting changes because two of the safety properties the description relies on don't hold as wired, and one input makes the code crash. I reproduced each one with a short script against c5c448f:

Blocking

1. "Version honesty" isn't actually wired. The audit's version never changes when the boost activates.
AuditedReward.verifier_version is computed once in __init__ (verifier_fingerprint(verifier, extra=version_extra)), so attach(version_extra=boost.version_tag()) freezes the tag at attach time. Repro: build a store and a boost (inactive), then attach(..., version_extra=boost.version_tag(), boost=boost), append 200 biased IMPROVEMENT labels, and call audit.recompute(). The result is boost.is_active == True and version_tag() is boost:active:…, but audit.verifier_version is unchanged. VerifierWatch never fires, and the old calibration keeps applying. Passing the bound method instead doesn't help either, since repr() of a bound method doesn't change with state. The "caught twice" argument depends on this, so it needs real wiring. For example, recompute() could re-derive and propagate the version (reward, gate, watch) when boost.version_tag() changes, or verifier_version could become a live property. Either way, it needs an integration test that asserts audit.verifier_version changes and that the calibration for the old version is withdrawn after activation.

2. The training pool mixes verifier versions.
retrain() calls store.for_improvement() with no verifier_version, so labels scored by an older judge get fitted together with the current one. Repro: put 200 records from an "old" verifier biased by +0.4 next to 200 from an unbiased "new" one. The boost activates with g(0.5) = 0.361, which is a correction learned mostly from a verifier that is no longer running. This is the same staleness the store's version filter exists to prevent. Please filter to the version currently being scored. The cleanest way is probably to have the boost take the audit's version, or a callable for it, and call for_improvement(version).

3. A crash when min_labels <= min_val_labels.
n_val = max(min_val_labels, …) can consume every record, which leaves train empty. Then isotonic_fit([]) returns empty arrays, and _apply() raises IndexError: index 0 is out of bounds from self._vals[0]. That happens outside the try, and since auto=True it happens inside the constructor. Repro: 10 records with min_labels=10, min_val_labels=10. Please guard len(train) == 0 (or validate min_labels > min_val_labels in __init__) and add a regression test.

Should fix

4. The flywheel is learned and applied on different scales.
The curve is fit on AuditRecord.verifier_score, which is the per-unit (task, output) -> float reward that AuditedReward wraps. It's then applied to cheap_eval / learned_eval, which (in ThreeLayerVerifier) are artifact-level means over subsets plus noise, and 0.5*rule + 0.5*learned. Those aren't the same function, and for a non-linear g, g(mean f) != mean g(f). _FakeVerifier returns the same constant for every layer, so the tests can't see this. At minimum, the docstring should state the assumption that the wrapped verifier's layer scores are on the same scale as the audited per-unit reward. Ideally there'd be a test with a real ThreeLayerVerifier showing the corrected ranking moves closer to the oracle. A related point: because the audit keeps recording the raw per-unit reward, recalibrating after activation measures the raw reward again, not the boosted ranking layers. So even with (1) fixed, the second check doesn't actually measure the boost.

5. The uncertainty change does the opposite of what the docs say.
Audit priority is blast_radius * uncertainty / trust (aggregator.py ~L1189), so higher uncertainty means more oracle spend. max(self._resid_sd, uncertainty) can only raise it. In the repro, resid_sd = 0.037 and the inner uncertainty is 0.1, giving 0.1, unchanged. The module docstring and PR body say a boost that tracks the oracle makes the audit spend less. Either the docs or the code should change, and test_the_learned_uncertainty_reflects_the_validation_residual should pin the intended direction. Also note that resid_sd is an RMSE in score units, while the learned layer's uncertainty is a noise band, so taking max across the two needs a sentence of justification.

6. retrain() swaps the curve before the gate decides.
self._xs, self._vals are overwritten at fit time while _state may still say active from the previous fit. Evaluation is concurrent (eval_concurrency), so a cheap_eval in that window applies an unvalidated curve under the old version_tag. Fit into locals and assign _xs/_vals/_fit_hash/_resid_sd/_state together only after the gate passes.

7. The gate is weak by default.
min_gain=0.0 with <= accepts a curve that is merely equal on one random split. With n_val around 10 to 20, that's mostly noise, and isotonic fits overfit small train sets. I'd suggest a default min_gain > 0 (e.g. 0.05), or a paired bootstrap / sign test on the per-example squared-error differences. Also, len(val) >= self.min_val_labels is always true by construction, so that clause is dead code.

Nits

• Callable is imported but unused in boost.py.
• Typo: "uncertainy" in the module docstring.
• The PR description says the correction "preserves score order". It's non-decreasing, so flat PAV blocks can still turn distinct raw scores into ties, so "never inverts order" is more accurate.
• There's no linked issue. That's fine, but given how large the design surface is, an issue or design note first would help scope the follow-ups.
• CHANGELOG.md will probably conflict with the other open PRs touching ### Added. A rebase will be needed before merge.

Happy to re-review once 1–3 are fixed and 4–5 are either addressed or explicitly documented as limitations.

[Birfy]

P1: Preserve the wrapped verifier’s full-set equivalence declaration.

At c5c448f, boost.py lines 182–183 default full_eval_matches_counts to True when the inner verifier has no such attribute, and also prefer inherited True over an explicitly declared legacy oracle_shares_full_set=False. The existing shares_eval_counts helper deliberately treats both cases as independent measurements. After wrapping, it sees the wrapper’s newly declared True, so Aggregator._audit skips full_eval and substitutes count-derived scores. A candidate whose counts improve but whose independent full evaluation regresses loses that veto, even with an empty store and inactive boost.

Please derive this declaration with shares_eval_counts(verifier) and add regressions for missing declarations and explicit legacy False.

Source:

agentdescent/agentdescent/audit/boost.py

Lines 182 to 183 in c5c448f

	self.full_eval_matches_counts = bool(getattr(
	verifier, "full_eval_matches_counts", True))

Helper:

agentdescent/agentdescent/verifier.py

Lines 298 to 320 in c5c448f

	def shares_eval_counts(verifier: Any) -> bool:
	"""Does this verifier's full-set scorer return what ``eval_counts`` measured?
	True means the aggregator can reuse rates it has already paid for instead of
	buying the same sweep twice -- and, more importantly, instead of routing a
	commit decision through a call that degrades to a sub-sample once the budget
	is gone. Reads :attr:`~ThreeLayerVerifier.full_eval_matches_counts`, then the
	pre-0.5.1 ``oracle_shares_full_set``.
	"""
	# A *declaration* beats an inherited default. `ThreeLayerVerifier` sets
	# `full_eval_matches_counts = True` as a statement about how that class is
	# written, and a subclass inherits it -- so the escape hatch the attribute's
	# docstring offers ("simply does not define it") does not exist for a
	# subclass, and one that set the pre-0.5.1 `oracle_shares_full_set = False`
	# was ignored. Its independent expensive layer then went uncalled and the
	# gate degraded to the cheap measurement it existed to cross-check, with no
	# warning.
	if _declares(verifier, "full_eval_matches_counts"):
	return bool(getattr(verifier, "full_eval_matches_counts"))
	if _declares(verifier, "oracle_shares_full_set"):
	return bool(getattr(verifier, "oracle_shares_full_set"))
	return bool(getattr(verifier, "full_eval_matches_counts", False))

Consumer:

agentdescent/agentdescent/aggregator.py

Lines 1223 to 1232 in c5c448f

	if shares_eval_counts(self.verifier):
	oracle_base, oracle_cand = base_full, cand_full
	else:
	full_eval = full_eval_of(self.verifier)
	oracle_base = full_eval(artifact)
	oracle_cand = full_eval(best_state)
	agreed = (oracle_cand > oracle_base) == (cand_score > base_score)
	self.audit.update_trust(artifact_id, agreed)
	if oracle_cand <= oracle_base:
	prior.observe_delta(oracle_cand - oracle_base)

Source inspection only; local execution remains paused, so no runtime reproduction is claimed.

[Birfy]

Status check (four-hour routine, 2026-10-16 ~12:30 CST): no new pushes since the CHANGES_REQUESTED review; still CONFLICTING vs main. Prior requested changes still stand — skipping re-review until there is a new head.

[Birfy]

主分支已有较多更新，当前 PR 与 main 存在冲突。请 rebase/merge main 并解决冲突后推送，我们会尽快合入。谢谢！

[Birfy]

Status note (four-hour routine)

Still CONFLICTING with main and the prior CHANGES_REQUESTED review is outstanding. No new push since that review (c5c448f). Skipping a deep re-review until the branch is rebased/merged with main and the earlier feedback is addressed — please resolve the conflicts on your side when you pick this back up (we do not push into contributor branches unsolicited).

[Birfy]

Maintainer push on feat/verifier-boost (0e6a890):

• Merged current main; CHANGELOG → fragment changelog.d/273-verifier-boost.md
• Fixed CR blockers: version honesty on activation, version-filtered training, empty-train guard, atomic curve assign, shares_eval_counts preservation
• Docs: uncertainty floor + scale assumption; default min_gain=0.05
• Local boost/audit suites green

Auto-squash armed once CI is green.

[Birfy]

Re-approve on 0e6a890 after push race. Verifier-boost CR items + changelog fragment addressed; see prior review body / PR comment.

[Birfy]

Re-approve after main merge + docs/api.md regen (CI api-reference sync).

[Birfy]

Follow-up: merged main (post-#254) and regenerated docs/api.md so the api-reference sync test is green. Auto-squash re-armed.

[Birfy]

Two distinct source-review findings at feca0bd, after the previous shares_eval_counts fix:

1. P2: Repeated recompute can toggle a validated boost with unchanged data. Audit.recompute trains on the current fingerprint, then refreshes it to include the activated curve (wiring.py 103–111). The next recompute filters IMPROVEMENT records to that new fingerprint; with no new labels, retrain's insufficient-label branch deactivates the curve (boost.py 218–228). Its tag returns to boost:inactive (274–304), restoring the old fingerprint, so another recompute can reactivate the same old fit. Please distinguish the underlying scorer's training identity from the activated correction's measurement identity and retain a valid fit when unchanged-scorer training data is merely insufficient. Proposed regression: seed a sufficient inactive-version pool, activate, call recompute repeatedly without adding records, then add a sub-min_labels batch; assert stable active curve/version instead of oscillation.

2. P2: The wrapper breaks supported oracle_eval-only verifiers, even inactive. Both full_eval and oracle_eval call self._inner.full_eval directly (boost.py 347–351). The engine explicitly supports legacy oracle_eval through full_eval_of (verifier.py 275–296). With an oracle_eval-only inner declaring oracle_shares_full_set=False, the corrected equivalence flag now reaches the independent gate, but full_eval_of selects the wrapper method and that method raises AttributeError. Delegate via full_eval_of(self._inner); proposed regression: wrap a legacy-only independent verifier with an empty store and assert its full gate still executes and can veto a candidate, as it does unwrapped.

Sources:

agentdescent/agentdescent/audit/wiring.py

Lines 103 to 111 in feca0bd

	if self.boost is not None:
	# Train against labels scored under the version currently in force
	# (before activation changes the tag).
	self.boost.retrain(verifier_version=self.verifier_version)
	prev = self.verifier_version
	self.reward.refresh_verifier_version()
	if self.reward.verifier_version != prev:
	self.watch.check()
	return self.calibrator.recompute(self.verifier_version)

agentdescent/agentdescent/audit/boost.py

Lines 218 to 228 in feca0bd

	version = verifier_version
	if version is None and self._version_for_training is not None:
	version = self._version_for_training()
	recs = [r for r in self.store.for_improvement(version)
	if r.resolved and _finite(r.verifier_score)
	and _finite(r.oracle_score)]
	recs = recs[-self.max_records:]
	if len(recs) < self.min_labels:
	return self._deactivate(
	f"insufficient improvement labels ({len(recs)} < "
	f"{self.min_labels})")

agentdescent/agentdescent/audit/boost.py

Lines 347 to 351 in feca0bd

	def full_eval(self, artifact: Evolvable) -> float:
	return self._inner.full_eval(artifact)
	def oracle_eval(self, artifact: Evolvable) -> float: # legacy alias
	return self._inner.full_eval(artifact)

agentdescent/agentdescent/verifier.py

Lines 275 to 296 in feca0bd

	def full_eval_of(verifier: Any) -> Callable[[Evolvable], float]:
	"""The verifier's whole-held-out-set scorer, under whichever name it has.
	Prefers ``full_eval``; falls back to a pre-0.5.1 ``oracle_eval`` and says so
	once per verifier class, naming the class so the warning points at the file
	that has to change.
	"""
	fn = getattr(verifier, "full_eval", None)
	if fn is not None:
	return fn
	legacy = getattr(verifier, "oracle_eval", None)
	if legacy is None:
	raise AttributeError(
	f"{type(verifier).__name__} has neither full_eval nor oracle_eval; "
	"VerifierProtocol needs cheap_eval, learned_eval, eval_counts and "
	"full_eval")
	warnings.warn(
	f"{type(verifier).__name__} defines oracle_eval but not full_eval. The "
	"method was renamed in 0.5.1 -- oracle_eval is read for now and dropped "
	"in 0.7. Rename it; the behaviour is unchanged.",
	DeprecationWarning, stacklevel=2)
	return legacy

Source inspection only. These regression scenarios have not been executed here; local execution remains paused.