[Feature] Check new model weights before replacing the serving model #5

Problem

A completed training step does not establish that the new model is safe to publish or better on real tasks. Recipes need configurable checks before replacing the currently served weights.

The shared train → evaluate → decide → activate/reject lifecycle already exists. This issue supplies the weight-specific measurements, policies and integration under #356.

Remaining work

1. Check training metrics

• Provide a configurable metric_threshold policy with minimum/maximum limits, required metrics and finite-value rules. Missing values, NaN and infinity reject by default unless explicitly allowed.
• Validate typed configuration before starting the runtime; retain an explicit always option.
• Record the policy/version, limits, observed values and reason. Emit selected/rejected counts with bounded reason labels.
• Define rejected-batch handling explicitly; default to consuming/discarding the attempted batch rather than retrying it forever.

A bundled weight recipe must be able to enable this through configuration without replacing its backend or build() method. Final configuration names remain to be agreed.

2. Compare behavior on held-out tasks

• Evaluate the candidate and current model on the same pinned task set before activation.
• Pass exact candidate/current identities, task-set identity and configuration to the evaluator. Pin model/tokenizer/template, scorer, sampling settings and seeds where applicable.
• Make candidate loading explicit, such as a temporary evaluation engine or shadow deployment, without silently changing the serving model.
• Bound concurrency, time, retries and cost. Missing or partial results, evaluator crashes/timeouts, incompatible checkpoints and ambiguous results cannot approve publication.
• Support configured absolute thresholds, allowed regressions and required task passes. Retain evaluation results and the decision in training and artifact/commit metadata.

Reuse #355's execution/results contract where available. #355 owns historical comparisons; this issue owns integration into weight publication. #698 proposes shared task-separation and failure rules; it does not complete weight loading or this integration.

Acceptance

• Metric tests cover passing and failing limits, missing/NaN/infinite values and policy errors.
• One real weight integration rejects a measured regression without activating the candidate, then publishes a passing candidate with its results attached.
• Failure and restart tests cover interrupted rejection and restoration of the current model.
• Documentation explains configuration, candidate-loading choices, cost, retained results and when always-select behavior is appropriate.

Existing work and verification

External evaluation.module/config plugins and the shared activation/rejection/recovery lifecycle are implemented. Merged PR #380 added composed plugins and RegressionGateMixin, which compares a scalar with the best previously selected score. That comparison is neither an absolute metric limit nor a fresh candidate/current-model comparison.

The review at main 91702b7f on 2026-09-15 reported 93 focused tests passing. A direct check found infinities selected, NaN rejected and a missing metric raising ValueError. These are dated findings, not a fresh validation of current main.

Open PR #325 includes an MLX/OpenClaw-RL candidate probe. The reviewed version compared candidate scores with historical best scores; it did not satisfy the paired behavioral requirement. Shared lifecycle tests likewise do not complete the checks above.

Originally migrated from reef-archive#111. The earlier standalone publication-hook proposal is superseded by the existing candidate lifecycle. This issue remains open and KIV.

AI assistance: Codex helped condense the issue while retaining its remaining requirements and dated verification limits.