lab: plan — version scoreboard + task matrix for the evolve harness

[Evanfeenstra]

Direction doc distilled from the post-mortem of prod run gaia-evolve/1788061734710 (the 0.68→0.80 result whose best generation never published, with both duplicate gradings landing inside a measured ±0.04 sampling floor).

• Thesis: every failure was the harness trusting narration where it owns ground truth — author-echoed versions vs the registry, summaries vs computable YAML diffs, single samples vs per-task vectors, run-end summaries vs the event log.
• Core restructure: measurements attach to versions (generations are just attempts — the stale-version bug becomes unrepresentable, and resamples are banked as incumbent data instead of discarded); a task×version matrix becomes the loop's native memory (bands, empirical flip-noise floor, bias-vs-variance via identical-answer detection, tool-error aggregation).
• Promotion: paired comparison on shared task vectors against the measured noise floor, replacing improveMargin/stopFitness-on-one-sample; cost enters the fitness per EVOLVE_SPEC §7.
• Also: foreach concurrency + incremental report persistence, diff-based briefings, an automated holdout beat, and a deletion table (what the simplification removes).
• Builds on evolve: skip no-op generations, guard re-scores, cap run budgets #1621 (no-op gate, re-score guard, budget caps) — that lands as-is; this is the layer above it.

Doc: mcp/src/lab/plans/evolve-scoreboard-and-task-matrix.md

🤖 Generated with Claude Code