lab: GAIA evolve harness on a generalized eval/evolve-loop - #1616
Merged
Conversation
…EC §9.5) Promote harvey/evolve-loop to the GENERIC eval/evolve-loop the spec called for: domain-neutral hill-climb over workflow versions, reading the gen digest's `fitness` (fallback meanPassRate) and naming it via `fitnessName` in briefings. harvey-evolve rewires onto it (pass-rate fitness, report field names unchanged). GAIA instance, mirroring harvey's produce/grade split: - gaia-run takes input.produceWorkflow (was hardcoded to gaia-produce) - gaia/evaluate gains a fromRun mode: unpacks a candidate run's answer in code (no-short-circuit template rule, §5.3.5) — a failed run scores as "" (honest zero), plus gold-stripped question/level metadata for digests - gaia/digest-results: accuracy as fitness, misses tagged wrong-answer / empty-answer / produce-error (§8's taxonomy, code-only), candidate answers + question excerpts, gold never enters or leaves - gaia-candidate-run: ai-stamped candidate via meta/run-workflow, graded fromRun - gaia-evolve-gen / gaia-evolve: meta/* author generations over the task set; improveMargin 0 (exact match has no judge noise — the residual produce-sampling noise is answered by held-out validation, per the report's TRAIN-score note) Offline checks: src/lab/gaia/evolve-smoke.ts (workflow parses, template guards, fromRun semantics, digest shapes, loop fitness/naming); harvey evolve-smoke updated for the generic loop. tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the harvey self-evolving setup to GAIA — and does it by generalizing the loop rather than copying it, per EVOLVE_SPEC §9.5 ("the GENERIC step remains open — this is the shape it should generalize").
The generic loop
harvey/evolve-loop→eval/evolve-loop(mcp/src/lab/eval/steps/evolve-loop.ts): same best-anchored hill-climb (briefings from all prior attempts, exploit→explore flip afterexploreAfterstalls, durable resume, run-control checkpoints), now domain-neutral. A domain plugs in its gen workflow plus a digest emittingfitness(fallbackmeanPassRate) and afitnessNamefor briefings.harvey-evolverewires onto it; its report fields (bestPassRateetc.) are unchanged.The GAIA instance
gaia-runnow takesinput.produceWorkflow(was hardcoded) — grades any seeded produce variant, matchingharvey-run.gaia/evaluatefromRunmode: unpacks a candidate run's answer in code (the template evaluator does not short-circuit — §5.3.5); a failed candidate run scores as"", an honest zero that never aborts a batch. Also returns gold-strippedquestion/levelfor digests.gaia/digest-results: accuracy asfitness, each miss taggedwrong-answer/empty-answer/produce-error(§8's formatting/persistence/tooling taxonomy, cheap code-only version), candidate answers + question excerpts for misses. Gold never enters or leaves this step.gaia-candidate-run: runs an ai-stamped candidate on one task viameta/run-workflow(own runId, fresh registry, refuses seeded workflows) and gradesfromRun.gaia-evolve-gen/gaia-evolve: meta/*-only author generations over the task set. GAIA-adapted author method: exact-match answer formatting is the top lever, then persistence, tooling, structure; candidate contract isinput { taskId }→{ taskId, answer, cost, steps }, and embeddinggaia/evaluatein a candidate (produce-time oracle) is called out as disqualifying at promotion review.Measurement discipline
improveMargin: 0for GAIA — exact-match scoring has no judge noise; any task flip counts. The residual produce-sampling noise is answered by held-out validation (the report's TRAIN-score note demands it), not a margin.Testing
npx tsx src/lab/gaia/evolve-smoke.ts— workflow parses/registry, template no-short-circuit guards,fromRunsemantics (honest zero, metadata degradation, mode exclusivity), digest shape normalization + miss taxonomy, loop fitness/naming/margin-0 ties.npx tsx src/lab/harvey/evolve-smoke.ts(updated for the generic loop) — passes.npx tsx src/lab/gaia/smoke.ts— passes.tsc --noEmitclean.Docs updated:
EVOLVE_SPEC.md§9.5,lab/AGENTS.md(eval + gaia sections).🤖 Generated with Claude Code