refactor(evaluation): one plugin class per candidate-evaluation policy, composed by mixins

[BobbyZhouZijian]

Why

Candidate evaluation was two objects glued together: a CandidateEvaluator that measured a candidate and a CandidateSelector that decided whether to publish it, wrapped at runtime by DefaultCandidateEvaluationPlugin. A method that wanted to own both halves had to write two classes, and a recipe had to know how to pair them — so "the selection policy" was never a thing you could point at, and a method that measures for itself (rather than through the training backend) had nowhere natural to put that code.

That came up while building a regression gate for OpenClaw-RL, which needs exactly that shape: its own measurement plus a best-checkpoint decision.

What

A plugin is one class that mixes in the methods it needs.

mixin	supplies
AlwaysSelectMixin	decide() — publish every evaluated candidate
RegressionGateMixin	decide() — best-checkpoint selection over one scalar metric (new)
ScoreComparisonMixin (cordis)	decide() — more task wins than losses
GEPASelectorMixin, MetaHarnessSelectorMixin	decide() — archive / population policies
BackendEvaluateMixin	evaluate() — delegate to the training backend

The mixins are abstract. CandidateSelector.decide and CandidateEvaluator.evaluate are @abstractmethod, and each mixin inherits CandidateEvaluationPlugin and implements one half of it — so the half it does not implement stays abstract and the mixin cannot be instantiated on its own:

AlwaysSelectMixin      abstract={evaluate}     BackendEvaluateMixin   abstract={decide}
RegressionGateMixin    abstract={evaluate}     ...Plugin              abstract={}

Declaring the abstract members on the protocols rather than on each mixin is what makes this compose: they sit last in every plugin's MRO, so a mixin's concrete half always wins over the inherited abstract one. The contracts stay runtime_checkable Protocols, so structural typing and the isinstance/issubclass checks are unchanged — only explicit subclasses gain the enforcement. Verified identical on Python 3.10, 3.11, 3.12 and 3.13.

A shipped plugin is the pairing: BackendAlwaysSelectPlugin, ScoreComparisonPlugin, GEPAPlugin, MetaHarnessPlugin. A method that measures for itself mixes a decide mixin into its own plugin and writes evaluate() directly.

RegressionGateMixin reads one scalar metric from the evaluation and selects only while the score stays within margin of the best selected so far; otherwise it rejects and serving holds the last selected weights. It's best-checkpoint selection made online: an objective past its peak cannot compound regressing steps into serving. The first candidate seeds the bar, so the initial climb is always admitted and the gate only bites once there is a peak to regress from. higher_is_better=False gates a metric that improves as it falls.

Contract change

Recipes now choose a plugin class, not a selector.

• CordisRecipe.candidate_selector → candidate_plugin, typed by a named CandidatePluginFactory Protocol (backend -> CandidateEvaluationPlugin).
• A dotted path in evolution.selection must now name a plugin class, not a selector.
• ScoreComparisonSelector → ScoreComparisonMixin + ScoreComparisonPlugin; GEPASelector → GEPASelectorMixin + GEPAPlugin; MetaHarnessSelector → MetaHarnessSelectorMixin + MetaHarnessPlugin.
• AlwaysSelect and DefaultCandidateEvaluationPlugin are gone, replaced by AlwaysSelectMixin and BackendAlwaysSelectPlugin.

CandidateEvaluator / CandidateSelector remain as the published protocols; only the built-ins changed shape.

Evidence

CI green on all 15 checks (lint, docs-gate, test and package on 3.10/3.11/3.12, the five real-harness jobs). Local: 2590 passed, 52 skipped (slime-dependency files excluded — no slime on this machine). mypy, ruff 0.14.7, isort 5.13.2, black 24.3.0, the Python design/statement policies, the README i18n check and the docs contract gate all pass.

The gate's effect on a real run is in the OpenClaw-RL MLX experiment on #325: ungated, the policy collapsed into an unbroken no-reply wall from session 51 to 71; gated, it kept answering through 17 of the 18 sessions in the same stretch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SuEa3FyrkW1xc3sjmAJSEB

[github-actions]

Merge Oncall: @BobbyZhouZijian

The merge oncall will route reviews and keep this pull request moving toward a clear decision.