From 8c03ed335ab5a7c53df9e8081b4921b3ae85eafa Mon Sep 17 00:00:00 2001 From: Evanfeenstra Date: Fri, 28 Aug 2026 14:34:47 -0700 Subject: [PATCH] =?UTF-8?q?lab:=20GAIA=20evolve=20harness=20on=20a=20gener?= =?UTF-8?q?alized=20eval/evolve-loop=20(EVOLVE=5FSPEC=20=C2=A79.5)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Promote harvey/evolve-loop to the GENERIC eval/evolve-loop the spec called for: domain-neutral hill-climb over workflow versions, reading the gen digest's `fitness` (fallback meanPassRate) and naming it via `fitnessName` in briefings. harvey-evolve rewires onto it (pass-rate fitness, report field names unchanged). GAIA instance, mirroring harvey's produce/grade split: - gaia-run takes input.produceWorkflow (was hardcoded to gaia-produce) - gaia/evaluate gains a fromRun mode: unpacks a candidate run's answer in code (no-short-circuit template rule, §5.3.5) — a failed run scores as "" (honest zero), plus gold-stripped question/level metadata for digests - gaia/digest-results: accuracy as fitness, misses tagged wrong-answer / empty-answer / produce-error (§8's taxonomy, code-only), candidate answers + question excerpts, gold never enters or leaves - gaia-candidate-run: ai-stamped candidate via meta/run-workflow, graded fromRun - gaia-evolve-gen / gaia-evolve: meta/* author generations over the task set; improveMargin 0 (exact match has no judge noise — the residual produce-sampling noise is answered by held-out validation, per the report's TRAIN-score note) Offline checks: src/lab/gaia/evolve-smoke.ts (workflow parses, template guards, fromRun semantics, digest shapes, loop fitness/naming); harvey evolve-smoke updated for the generic loop. tsc clean. Co-Authored-By: Claude Fable 5 --- mcp/src/lab/AGENTS.md | 33 +- mcp/src/lab/createLabVein.ts | 2 +- mcp/src/lab/eval/seed.ts | 11 +- .../lab/{harvey => eval}/steps/evolve-loop.ts | 128 ++++---- mcp/src/lab/gaia/evolve-smoke.ts | 241 +++++++++++++++ mcp/src/lab/gaia/seed.ts | 23 ++ mcp/src/lab/gaia/steps/digest-results.ts | 157 ++++++++++ mcp/src/lab/gaia/steps/evaluate.ts | 50 +++- .../gaia/workflows/gaia-candidate-run.yaml | 91 ++++++ .../lab/gaia/workflows/gaia-evolve-gen.yaml | 281 ++++++++++++++++++ mcp/src/lab/gaia/workflows/gaia-evolve.yaml | 135 +++++++++ mcp/src/lab/gaia/workflows/gaia-run.yaml | 11 +- mcp/src/lab/harvey/evolve-smoke.ts | 29 +- mcp/src/lab/harvey/seed.ts | 5 +- .../harvey/workflows/harvey-evolve-gen.yaml | 4 +- .../lab/harvey/workflows/harvey-evolve.yaml | 15 +- vein/EVOLVE_SPEC.md | 28 +- 17 files changed, 1155 insertions(+), 89 deletions(-) rename mcp/src/lab/{harvey => eval}/steps/evolve-loop.ts (67%) create mode 100644 mcp/src/lab/gaia/evolve-smoke.ts create mode 100644 mcp/src/lab/gaia/steps/digest-results.ts create mode 100644 mcp/src/lab/gaia/workflows/gaia-candidate-run.yaml create mode 100644 mcp/src/lab/gaia/workflows/gaia-evolve-gen.yaml create mode 100644 mcp/src/lab/gaia/workflows/gaia-evolve.yaml diff --git a/mcp/src/lab/AGENTS.md b/mcp/src/lab/AGENTS.md index d12bc70cd..d6bd86691 100644 --- a/mcp/src/lab/AGENTS.md +++ b/mcp/src/lab/AGENTS.md @@ -429,7 +429,8 @@ are thin plumbing over `ctx.services.gaia.*`. `gaia-produce` (agent step; the produce system prompt, model, maxSteps: 50 and agentTools live in `params`; an `onError` fallback scores a blown-up agent as an empty wrong answer instead of killing the batch), `gaia-run` - (single task: produce → score) and `gaia-batch` ({ level, limit }: one + (single task: produce → score; `input.produceWorkflow` swaps in a seeded + produce variant) and `gaia-batch` ({ level, limit }: one score call for the whole batch). This is the harness that went 1/5 → 5/5 on the level-1 batch (EVOLVE_SPEC §1), promoted from the workspace where the assistant authored it. Seeding is content-hash reconciled — the @@ -438,6 +439,25 @@ are thin plumbing over `ctx.services.gaia.*`. authoring recipe is kept in `notes/GAIA.md` as an authoring eval; it is no longer the path to a working harness. +- **Evolve harness** (mirrors harvey's, on the generic `eval/evolve-loop`): + `gaia/digest-results` (verdict-channel digest — accuracy as `fitness`, + misses tagged wrong-answer / empty-answer / produce-error per EVOLVE_SPEC + §8's taxonomy, candidate answers + question excerpts, never gold), + `gaia-candidate-run` (runs an ai-stamped candidate on one task via + `meta/run-workflow`, scores its reported answer via `gaia/evaluate`'s + `fromRun` unpack — a failed run is an honest zero), `gaia-evolve-gen` + (one generation: meta/* author → pinned candidate over the task set → + digest) and `gaia-evolve` (baseline → hill-climb → report; + `improveMargin: 0` since exact-match has no judge noise — the residual + produce-sampling noise is answered by held-out validation, not a margin). + Candidate contract: input `{ taskId }`, last step outputs `taskId`, + `answer` (bare string), `cost`, `steps`; candidates may use + `gaia/get-task` / `gaia/pack-result` as steps but NEVER `gaia/evaluate` + (produce-time oracle) and never gaia/*, eval/*, meta/* as agentTools. + Scores are TRAIN scores — validate the best version on a held-out + `gaia-batch` slice before promoting. Offline checks: + `npx tsx src/lab/gaia/evolve-smoke.ts`. + - **Setup**: automatic (`gaia/bootstrap.ts`) — the one required env var is **`HF_TOKEN`**. First use materialises the dataset into `/vein/gaia`, installs the leaderboard Space's `scorer.py` (verified against the in-repo @@ -497,6 +517,17 @@ Domain-agnostic eval substrate, shared by every experiment. See §11.2) — the per-example results array is fed to reflect. Each entry carries its own gold (e.g. `{ owner, repo, expected }`), read by the eval workflow from `input`. (A single example is just a 1-entry `evalInputs`.) +- `eval/steps/evolve-loop.ts` (`eval/evolve-loop`) — the generic hill-climb + over WORKFLOW VERSIONS (EVOLVE_SPEC §5.3.3 generalized from the harvey + instance): runs a domain's one-generation workflow (author → run candidate + over tasks → digest) up to N generations, briefing each author with every + prior attempt anchored to the best-so-far, flipping exploit→explore after + `exploreAfter` non-improving attempts. Fitness is the generation digest's + `fitness` (fallback `meanPassRate`), named in briefings by `fitnessName`; + improvements must clear `improveMargin` (judge noise for LLM-judged + domains — harvey 0.02; produce-sampling noise for deterministic scorers — + gaia 0). Needs `services.optimizer`. Wired by `harvey-evolve` + (pass-rate) and `gaia-evolve` (accuracy). **Naming rule:** `eval/*` = generic. The eval *workflows* that wire these with a rubric/task/dataset belong to the experiment and are named `-…`. diff --git a/mcp/src/lab/createLabVein.ts b/mcp/src/lab/createLabVein.ts index e56d1b6a1..51381679e 100644 --- a/mcp/src/lab/createLabVein.ts +++ b/mcp/src/lab/createLabVein.ts @@ -178,7 +178,7 @@ export async function createLabVein( // spreading our `services` into a fresh object (standardServices + // artifacts + ours) — NOT the local `services`, which runs never see // again. Mutating the local bag here silently broke every consumer of - // `services.optimizer` (eval/optimize, harvey/evolve-loop): steps threw + // `services.optimizer` (eval/optimize, eval/evolve-loop): steps threw // "requires a services.optimizer capability" at run time. This is what // lets the optimize/evolve loops run sub-workflows. const optimizer: LabServices["optimizer"] = { diff --git a/mcp/src/lab/eval/seed.ts b/mcp/src/lab/eval/seed.ts index 88c143569..aeed346a9 100644 --- a/mcp/src/lab/eval/seed.ts +++ b/mcp/src/lab/eval/seed.ts @@ -12,15 +12,20 @@ import type { WorkspaceManager } from "vein"; * config. An experiment supplies its own eval WORKFLOWS that wire these steps * with its rubric / task / dataset (e.g. the concepts experiment ships * `concepts-eval*` in concepts/workflows, seeded by concepts/seed.ts): - * - `eval/score` — match produced vs expected by a `rubric`, recall-weighted. - * - `eval/reflect` — propose a better prompt from AGGREGATED results. - * - `eval/optimize` — eval → keep best → reflect loop (a detached job). + * - `eval/score` — match produced vs expected by a `rubric`, recall-weighted. + * - `eval/reflect` — propose a better prompt from AGGREGATED results. + * - `eval/optimize` — eval → keep best → reflect loop (a detached job). + * - `eval/evolve-loop` — hill-climb candidate WORKFLOW VERSIONS over + * generations (EVOLVE_SPEC §5.3.3 generalized): a domain supplies its + * one-generation workflow + a digest with a `fitness`; harvey-evolve and + * gaia-evolve are the two instances. */ const SEED_STEPS: Array<{ file: string; type: string }> = [ { file: "score.ts", type: "eval/score" }, { file: "reflect.ts", type: "eval/reflect" }, { file: "optimize.ts", type: "eval/optimize" }, + { file: "evolve-loop.ts", type: "eval/evolve-loop" }, ]; const HERE = dirname(fileURLToPath(import.meta.url)); diff --git a/mcp/src/lab/harvey/steps/evolve-loop.ts b/mcp/src/lab/eval/steps/evolve-loop.ts similarity index 67% rename from mcp/src/lab/harvey/steps/evolve-loop.ts rename to mcp/src/lab/eval/steps/evolve-loop.ts index 12c09afc3..e879374ca 100644 --- a/mcp/src/lab/harvey/steps/evolve-loop.ts +++ b/mcp/src/lab/eval/steps/evolve-loop.ts @@ -1,29 +1,47 @@ import { z, defineStep, type RunEvent } from "vein"; /** - * The HILL-CLIMB over workflow versions (EVOLVE_SPEC §5.3.3, the harvey - * instance): run `harvey-evolve-gen` up to maxGenerations times, feeding - * each generation a BRIEFING composed from the baseline digest plus every - * previous attempt's version, pass-rate, approach summary, and failure + * The GENERIC hill-climb over workflow versions (EVOLVE_SPEC §5.3.3 — the + * generalization §9.5 called for; promoted from the harvey instance): run a + * one-generation workflow up to maxGenerations times, feeding each + * generation a BRIEFING composed from the baseline digest plus every + * previous attempt's version, fitness, approach summary, and failure * digest — anchored to the best-so-far (never the latest, which may have * regressed; the same guarantee eval/optimize makes for prompts). * + * Domain-agnostic on purpose: the loop knows nothing about rubrics or + * scorers. A domain plugs in via + * - `genWorkflow`: its one-generation workflow (author → run candidate + * over tasks → digest), invoked with + * { tasks, mission, candidateName, generation, briefing } and returning + * { version?, summary?, changes?, missingSecrets?, authorCost?, digest } + * - the digest's FITNESS: the loop reads `digest.fitness` (falling back + * to `digest.meanPassRate`, the harvey digest's field) — a number in + * [0,1] that MUST have a gradient (harvey: criteria pass-rate, since + * binary all-pass has none; gaia: plain accuracy — binary per task is + * fine there because the set supplies the gradient). + * - `fitnessName`: how briefings name that number ("pass-rate", + * "accuracy", …) so authors read the right thing. + * * EXPLOIT vs EXPLORE: while attempts keep beating the best, the directive * says refine the best version. After `exploreAfter` consecutive * non-improving attempts, it flips: try a GENUINELY DIFFERENT approach, * with the already-tried approaches listed so "different" is checkable. * - * Judge noise: an improvement must clear `improveMargin` (default 0.02 — - * one criterion on a 50-criterion task) to count; smaller deltas are ties. + * Noise: an improvement must clear `improveMargin` to count; smaller + * deltas are ties. What the margin answers is per-domain — judge noise for + * LLM-judged benchmarks (harvey: 0.02 ≈ one criterion at n=50), produce- + * sampling noise for deterministic scorers (gaia: 0, any task flip counts, + * but the same caveat rides: validate on held-out tasks). * * Runs generations through `services.optimizer` (vein.run — same capability * eval/optimize uses), each as its own persisted run linked from this - * step's per-generation progress events. Stops on: stopPassRate reached, + * step's per-generation progress events. Stops on: stopFitness reached, * generations exhausted, or two consecutive generation-run failures (a * broken harness should not burn ten generations of budget). * * TRAIN-SET caveat rides on the output: every generation tunes against the - * same tasks; the final pass-rate is a train score (EVOLVE_SPEC §7). + * same tasks; the final fitness is a train score (EVOLVE_SPEC §7). */ interface AnyRec { @@ -67,7 +85,7 @@ interface GenEntry { gen: number; genRunId: string; version?: string; - passRate: number; + fitness: number; allPassCount?: number; summary?: string; changes?: unknown; @@ -82,19 +100,20 @@ interface GenEntry { export function composeBriefing(args: { baseWorkflow: string; candidateName: string; + fitnessName: string; baselineText: string; - baselinePassRate: number; + baselineFitness: number; generations: GenEntry[]; - best: { gen: number; version?: string; passRate: number; digestText: string }; + best: { gen: number; version?: string; fitness: number; digestText: string }; explore: boolean; sinceImprove: number; improveMargin: number; }): string { - const { generations, best } = args; + const { generations, best, fitnessName } = args; const lines: string[] = []; lines.push( - `BASELINE — the seeded produce workflow "${args.baseWorkflow}" was run on every task and graded (mean pass-rate ${args.baselinePassRate}):`, + `BASELINE — the seeded produce workflow "${args.baseWorkflow}" was run on every task and graded (mean ${fitnessName} ${args.baselineFitness}):`, ); lines.push(indent(args.baselineText, " ")); lines.push(""); @@ -108,9 +127,9 @@ export function composeBriefing(args: { lines.push(`- attempt ${g.gen}: FAILED to complete (${excerpt(g.error, 200)})`); continue; } - const delta = Math.round((g.passRate - args.baselinePassRate) * 1000) / 1000; + const delta = Math.round((g.fitness - args.baselineFitness) * 1000) / 1000; lines.push( - `- attempt ${g.gen} → published ${args.candidateName}@${g.version ?? "?"}: mean pass-rate ${g.passRate} (${delta >= 0 ? "+" : ""}${delta} vs baseline)`, + `- attempt ${g.gen} → published ${args.candidateName}@${g.version ?? "?"}: mean ${fitnessName} ${g.fitness} (${delta >= 0 ? "+" : ""}${delta} vs baseline)`, ); // The approach summary is the ONLY channel telling the EXPLORE // directive what has already been tried — keep it roomy enough that @@ -122,10 +141,12 @@ export function composeBriefing(args: { lines.push(""); if (best.gen < 0) { - lines.push(`BEST SO FAR: the baseline itself (mean pass-rate ${best.passRate}) — no attempt has beaten it yet.`); + lines.push( + `BEST SO FAR: the baseline itself (mean ${fitnessName} ${best.fitness}) — no attempt has beaten it yet.`, + ); } else { lines.push( - `BEST SO FAR: attempt ${best.gen} → ${args.candidateName}@${best.version} (mean pass-rate ${best.passRate}). Its result digest:`, + `BEST SO FAR: attempt ${best.gen} → ${args.candidateName}@${best.version} (mean ${fitnessName} ${best.fitness}). Its result digest:`, ); lines.push(indent(excerpt(best.digestText, 900), " ")); } @@ -150,36 +171,40 @@ export function composeBriefing(args: { lines.push( `DIRECTIVE — EXPLOIT. Improve FROM the best attempt: read ${args.candidateName}@${best.version} ` + `(meta/get-workflow with that EXACT version — the active version may be a worse later ` + - `attempt) and refine it: keep what worked, fix exactly what its failed criteria show.`, + `attempt) and refine it: keep what worked, fix exactly what its failures show.`, ); } lines.push( - `Deltas within ±${args.improveMargin} are judge noise — treat them as ties, not as signal to chase.`, + `Deltas within ±${args.improveMargin} are noise — treat them as ties, not as signal to chase.`, ); return lines.join("\n"); } export default defineStep({ - type: "harvey/evolve-loop", + type: "eval/evolve-loop", description: - "Hill-climb candidate produce workflows over generations: repeatedly run a generation workflow (default harvey-evolve-gen: author → run candidate over tasks → digest), composing each generation's briefing from the baseline digest + all previous attempts, anchored to the best-so-far, flipping to an explore directive after `exploreAfter` non-improving attempts. Requires services.optimizer. Config: tasks, mission, baseline (a harvey/digest-results object), candidateName, baseWorkflow?, genWorkflow?, maxGenerations? (default 5, max 20), stopPassRate? (default 1), improveMargin? (default 0.02), exploreAfter? (default 2), genParams? (paramOverrides for the generation workflow). Output: { candidate, baselinePassRate, bestGen, bestVersion, bestPassRate, improved, generations, totalKnownCost, stopReason }.", + "GENERIC hill-climb of candidate produce workflows over generations: repeatedly run a domain's one-generation workflow (author → run candidate over tasks → digest), composing each generation's briefing from the baseline digest + all previous attempts, anchored to the best-so-far, flipping to an explore directive after `exploreAfter` non-improving attempts. Fitness is the generation digest's `fitness` (fallback `meanPassRate`). Requires services.optimizer. Config: tasks, mission, baseline (a digest object with fitness/meanPassRate + text), candidateName, baseWorkflow, genWorkflow, fitnessName? (default 'pass-rate'), maxGenerations? (default 5, max 20), stopFitness? (default 1), improveMargin? (default 0.02), exploreAfter? (default 2), genParams? (paramOverrides for the generation workflow). Output: { candidate, baselineFitness, bestGen, bestVersion, bestFitness, improved, generations, totalKnownCost, stopReason }.", input: z.object({ - tasks: z.array(z.string()).min(1).describe("Harvey task ids the loop optimizes against (the TRAIN set)"), + tasks: z.array(z.string()).min(1).describe("task ids the loop optimizes against (the TRAIN set)"), mission: z.string().describe("the standing gap statement handed to every generation's author"), baseline: z .any() - .describe("the baseline harvey/digest-results object ({ meanPassRate, text, … }) — generation 0's anchor"), + .describe("the baseline digest object ({ fitness | meanPassRate, text, … }) — generation 0's anchor"), candidateName: z.string().describe("the candidate workflow name every generation publishes to (versioned lineage)"), - baseWorkflow: z.string().default("harvey-produce").describe("the seeded produce workflow the baseline ran"), - genWorkflow: z.string().default("harvey-evolve-gen").describe("the one-generation workflow the loop runs"), + baseWorkflow: z.string().describe("the seeded produce workflow the baseline ran"), + genWorkflow: z.string().describe("the domain's one-generation workflow the loop runs"), + fitnessName: z + .string() + .default("pass-rate") + .describe("how briefings name the fitness number ('pass-rate', 'accuracy', …)"), maxGenerations: z.number().int().min(1).max(20).default(5), - stopPassRate: z.number().min(0).max(1).default(1).describe("stop early once a generation's mean pass-rate reaches this"), + stopFitness: z.number().min(0).max(1).default(1).describe("stop early once a generation's fitness reaches this"), improveMargin: z .number() .min(0) .default(0.02) - .describe("an attempt must beat the best by MORE than this to count as an improvement (judge noise floor; 0.02 ≈ one criterion on a 50-criterion task)"), + .describe("an attempt must beat the best by MORE than this to count as an improvement (noise floor — judge noise for LLM-judged domains, produce-sampling noise for deterministic scorers)"), exploreAfter: z .number() .int() @@ -195,17 +220,17 @@ export default defineStep({ async run(cfg, ctx) { const opt = (ctx.services as { optimizer?: Optimizer } | undefined)?.optimizer; if (!opt) { - throw new Error("harvey/evolve-loop requires a `services.optimizer` capability (injected in createLabVein)."); + throw new Error("eval/evolve-loop requires a `services.optimizer` capability (injected in createLabVein)."); } const baseline = (cfg.baseline ?? {}) as AnyRec; - const baselinePassRate = num(baseline["meanPassRate"]) ?? 0; + const baselineFitness = num(baseline["fitness"]) ?? num(baseline["meanPassRate"]) ?? 0; const baselineText = typeof baseline["text"] === "string" && baseline["text"] ? (baseline["text"] as string) : excerpt(JSON.stringify(baseline), 800); - let best = { gen: -1, version: undefined as string | undefined, passRate: baselinePassRate, digestText: baselineText }; + let best = { gen: -1, version: undefined as string | undefined, fitness: baselineFitness, digestText: baselineText }; let sinceImprove = 0; let consecutiveFailures = 0; let totalKnownCost = 0; @@ -219,7 +244,7 @@ export default defineStep({ ts: new Date().toISOString(), runId: ctx.runId, path: `${ctx.path}#${gen}`, - stepType: "harvey/evolve-loop", + stepType: "eval/evolve-loop", iteration: gen, ...e, } as RunEvent); @@ -235,12 +260,12 @@ export default defineStep({ // from the journaled output so the loop continues where it left off. const journaled = ctx.journal?.[`${ctx.path}#${gen}`] as AnyRec | undefined; if (journaled) { - const passRate = num(journaled["passRate"]) ?? 0; + const fitness = num(journaled["fitness"]) ?? num(journaled["passRate"]) ?? 0; const entry: GenEntry = { gen, genRunId: String((journaled["runs"] as AnyRec[] | undefined)?.[0]?.["runId"] ?? ""), version: typeof journaled["version"] === "string" ? (journaled["version"] as string) : undefined, - passRate, + fitness, summary: typeof journaled["summary"] === "string" ? (journaled["summary"] as string) : undefined, digestText: typeof journaled["digestText"] === "string" ? (journaled["digestText"] as string) : undefined, explore: journaled["directive"] === "explore", @@ -248,15 +273,15 @@ export default defineStep({ generations.push(entry); consecutiveFailures = 0; totalKnownCost += num(journaled["knownCost"]) ?? 0; - if (passRate > best.passRate + cfg.improveMargin) { - best = { gen, version: entry.version, passRate, digestText: entry.digestText ?? "" }; + if (fitness > best.fitness + cfg.improveMargin) { + best = { gen, version: entry.version, fitness, digestText: entry.digestText ?? "" }; sinceImprove = 0; } else { sinceImprove++; } await emitGen(gen, { type: "step.replayed", output: journaled }); - if (passRate >= cfg.stopPassRate) { - stopReason = `stopPassRate ${cfg.stopPassRate} reached`; + if (fitness >= cfg.stopFitness) { + stopReason = `stopFitness ${cfg.stopFitness} reached`; break; } continue; @@ -266,8 +291,9 @@ export default defineStep({ const briefing = composeBriefing({ baseWorkflow: cfg.baseWorkflow, candidateName: cfg.candidateName, + fitnessName: cfg.fitnessName, baselineText, - baselinePassRate, + baselineFitness, generations, best, explore, @@ -277,7 +303,7 @@ export default defineStep({ const genStart = Date.now(); await emitGen(gen, { type: "step.start", - input: { gen, directive: explore ? "explore" : "exploit", bestPassRate: best.passRate, briefing: excerpt(briefing, 2000) }, + input: { gen, directive: explore ? "explore" : "exploit", bestFitness: best.fitness, briefing: excerpt(briefing, 2000) }, }); const run = await opt.run( @@ -295,7 +321,7 @@ export default defineStep({ if (run.status !== "success") { const message = run.error?.message ?? "unknown"; - generations.push({ gen, genRunId: run.runId, passRate: 0, explore, error: message }); + generations.push({ gen, genRunId: run.runId, fitness: 0, explore, error: message }); await emitGen(gen, { type: "step.error", durationMs: Date.now() - genStart, @@ -312,7 +338,7 @@ export default defineStep({ const out = (run.output ?? {}) as AnyRec; const digest = (out["digest"] ?? {}) as AnyRec; - const passRate = num(digest["meanPassRate"]) ?? 0; + const fitness = num(digest["fitness"]) ?? num(digest["meanPassRate"]) ?? 0; const digestResults = Array.isArray(digest["results"]) ? (digest["results"] as AnyRec[]) : []; const authorCost = num(out["authorCost"]) ?? 0; const produceCost = digestResults.reduce((s, r) => s + (num(r["cost"]) ?? 0), 0); @@ -322,7 +348,7 @@ export default defineStep({ gen, genRunId: run.runId, version: typeof out["version"] === "string" ? (out["version"] as string) : undefined, - passRate, + fitness, allPassCount: num(digest["allPassCount"]), summary: typeof out["summary"] === "string" ? (out["summary"] as string) : undefined, changes: out["changes"], @@ -334,8 +360,8 @@ export default defineStep({ }; generations.push(entry); - if (passRate > best.passRate + cfg.improveMargin) { - best = { gen, version: entry.version, passRate, digestText: entry.digestText ?? "" }; + if (fitness > best.fitness + cfg.improveMargin) { + best = { gen, version: entry.version, fitness, digestText: entry.digestText ?? "" }; sinceImprove = 0; } else { sinceImprove++; @@ -348,8 +374,8 @@ export default defineStep({ gen, directive: explore ? "explore" : "exploit", version: entry.version, - passRate, - bestPassRate: best.passRate, + fitness, + bestFitness: best.fitness, bestGen: best.gen, knownCost: Math.round((authorCost + produceCost) * 10000) / 10000, runs: [{ label: `generation ${gen}`, workflow: cfg.genWorkflow, runId: run.runId }], @@ -360,23 +386,23 @@ export default defineStep({ }, }); - if (passRate >= cfg.stopPassRate) { - stopReason = `stopPassRate ${cfg.stopPassRate} reached`; + if (fitness >= cfg.stopFitness) { + stopReason = `stopFitness ${cfg.stopFitness} reached`; break; } } return { candidate: cfg.candidateName, - baselinePassRate, + baselineFitness, bestGen: best.gen, bestVersion: best.version, - bestPassRate: best.passRate, + bestFitness: best.fitness, improved: best.gen >= 0, generations, totalKnownCost: Math.round(totalKnownCost * 10000) / 10000, stopReason, - note: "TRAIN scores — every generation tuned against the same tasks (EVOLVE_SPEC §7). Validate the best version on held-out tasks before promoting. totalKnownCost = author + produce costs; judge cost is not surfaced by the benchmark and is additional.", + note: "TRAIN scores — every generation tuned against the same tasks (EVOLVE_SPEC §7). Validate the best version on held-out tasks before promoting. totalKnownCost = author + produce costs; grading cost (where the benchmark bills it) is additional.", }; }, }); diff --git a/mcp/src/lab/gaia/evolve-smoke.ts b/mcp/src/lab/gaia/evolve-smoke.ts new file mode 100644 index 000000000..99978edfb --- /dev/null +++ b/mcp/src/lab/gaia/evolve-smoke.ts @@ -0,0 +1,241 @@ +/** + * Offline validation for the gaia evolve harness (the gaia instance of the + * generic eval/evolve-loop): + * 1. all six gaia workflows seed + parse into Flows (YAML → Flow schema) + * 2. the template expressions used in them resolve as intended + * 3. gaia/evaluate's fromRun mode unpacks candidate runs safely in code + * 4. gaia/digest-results normalizes every result shape + tags misses + * 5. eval/evolve-loop climbs the digest's `fitness` (accuracy) field + * No python, no dataset checkout, no LLM, no network. + * Run: npx tsx src/lab/gaia/evolve-smoke.ts + */ +import assert from "node:assert/strict"; +import { mkdtempSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { WorkspaceManager, buildRegistry, resolveConfig } from "vein"; +import { seedGaiaSteps, seedGaiaWorkflows } from "./seed.js"; +import { seedEvalSteps } from "../eval/seed.js"; +import { seedArtifactSteps } from "../artifacts/seed.js"; + +async function main() { + const base = mkdtempSync(join(process.cwd(), ".gaia-evolve-validate-")); + try { + const workspace = new WorkspaceManager(join(base, "ws")); + await seedGaiaSteps(workspace); + await seedEvalSteps(workspace); // eval/evolve-loop — the generic hill-climb gaia-evolve wires + await seedArtifactSteps(workspace); + await seedGaiaWorkflows(workspace); + + // 1. every seeded workflow parses into a Flow + for (const name of ["gaia-produce", "gaia-run", "gaia-batch", "gaia-candidate-run", "gaia-evolve-gen", "gaia-evolve"]) { + const flow = await workspace.getWorkflow(name); + assert.equal(flow.name, name); + assert.ok(Array.isArray(flow.steps) && flow.steps.length > 0, `${name} has steps`); + console.log(`✔ workflow parses: ${name} (${flow.steps.length} steps)`); + } + + // step types referenced by the workflows all exist in the registry + const { registry } = await buildRegistry(workspace.path); + const wanted = [ + "gaia/list-tasks", "gaia/get-task", "gaia/evaluate", "gaia/pack-result", + "gaia/summarize-batch", "gaia/digest-results", "eval/evolve-loop", + "artifacts/dir", "meta/run-workflow", "agent", "subflow", "foreach", + ]; + for (const t of wanted) assert.ok(registry[t], `registry has ${t}`); + console.log("✔ all referenced step types resolve"); + + // 2. template expressions used in the new workflows + const scope: Record = { + input: { taskId: "t-1", tasks: ["t-1", "t-2"] }, + run: { runId: "r1", status: "success", output: { taskId: "t-1", answer: "42", cost: 0.5, steps: 3 } }, + grade: { isCorrect: true, answer: "42", benchmarkRev: "abc" }, + vactive: { version: "v2" }, + vpin: {}, + params: {}, + }; + const r = (s: string) => (resolveConfig as any)(s, scope); + assert.equal(r("{{ input.produceWorkflow || 'gaia-produce' }}"), "gaia-produce"); + assert.equal(r("{{ vpin.version || vactive.version }}"), "v2"); + // The evaluator does NOT short-circuit — a failed candidate run has no + // `output`, so the workflows pass the WHOLE run into gaia/evaluate's + // fromRun and unpack in code, never `run.output.answer` in YAML. + const failedScope = { ...scope, run: { runId: "r2", status: "failed", error: { message: "boom" } } }; + assert.throws( + () => (resolveConfig as any)("{{ run.output ? run.output.answer : '' }}", failedScope), + /Cannot access/, + ); + assert.deepEqual((resolveConfig as any)("{{ run }}", failedScope), failedScope.run); + // gaia-candidate-run's grade fallback keeps isCorrect/answer shape-stable, + // so the result pack's one-level reads stay safe: + const fallbackScope = { ...scope, grade: { taskId: "t-1", isCorrect: false, answer: "", level: null, gradeError: "boom" } }; + assert.equal((resolveConfig as any)("{{ grade.isCorrect }}", fallbackScope), false); + assert.equal((resolveConfig as any)("{{ grade.benchmarkRev }}", fallbackScope), undefined); + console.log("✔ template expressions resolve (fallback, version pin, whole-run pass; no-short-circuit guarded)"); + + // 3. gaia/evaluate fromRun: unpack-in-code semantics against a faked + // scoring service (no python, no dataset). + const evaluate = registry["gaia/evaluate"]!; + const scored: any[] = []; + const fakeGaia = { + getTask: async (taskId: string) => { + if (taskId === "t-2") throw new Error("metadata unavailable"); // must not fail the grade + return { taskId, question: `question for ${taskId}`, level: 1, fileName: "" }; + }, + score: async (pairs: Array<{ taskId: string; answer: string }>) => { + scored.push(pairs); + const results = pairs.map((p) => ({ taskId: p.taskId, level: 1, correct: p.answer === "42" })); + const correct = results.filter((x) => x.correct).length; + return { + accuracy: results.length ? correct / results.length : 0, + correct, + total: results.length, + byLevel: { "1": { correct, total: results.length } }, + results, + benchmarkRev: "rev0", + scorerSha256: "sha0", + }; + }, + }; + const ctxStub = { runId: "r", path: "p", scope: {}, input: undefined, emit: async () => {}, services: {}, registry } as any; + const evalCtx = { ...ctxStub, services: { gaia: fakeGaia } }; + const ok: any = await evaluate.run( + evaluate.input.parse({ fromRun: { taskId: "t-1", run: { runId: "r1", status: "success", output: { answer: "42" } } } }), + evalCtx, + ); + assert.equal(ok.isCorrect, true); + assert.equal(ok.answer, "42"); + assert.equal(ok.level, 1); + assert.equal(ok.question, "question for t-1"); + assert.equal(ok.benchmarkRev, "rev0"); + // a failed run (no output) scores as "" — an honest zero, not a throw — + // and a failing metadata lookup degrades to question null, never a throw + const failed: any = await evaluate.run( + evaluate.input.parse({ fromRun: { taskId: "t-2", run: { runId: "r2", status: "failed", error: { message: "boom" } } } }), + evalCtx, + ); + assert.equal(failed.isCorrect, false); + assert.equal(failed.answer, ""); + assert.equal(failed.question, null); + assert.deepEqual(scored[1], [{ taskId: "t-2", answer: "" }]); + // exactly one of pairs/fromRun + await assert.rejects(() => evaluate.run(evaluate.input.parse({}), evalCtx), /exactly one/); + await assert.rejects( + () => + evaluate.run( + evaluate.input.parse({ pairs: [{ taskId: "t", answer: "a" }], fromRun: { taskId: "t", run: {} } }), + evalCtx, + ), + /exactly one/, + ); + console.log("✔ gaia/evaluate fromRun: unpacks in code, honest zero on failed runs, mode exclusivity"); + + // 4. digest step over every result shape that reaches it + const digest = registry["gaia/digest-results"]!; + const out: any = await digest.run( + digest.input.parse({ + results: [ + // gaia-candidate-run shape: boolean correct, cost/steps only + // inside runResult.output + { + taskId: "t-1", level: 1, correct: true, answer: "42", question: "What is…", + runResult: { runId: "x", status: "success", output: { cost: 0.5, steps: 12 } }, + }, + // gaia-run shape: correct as a COUNT + the score call's results array + { + taskId: "t-2", level: 1, correct: 0, total: 1, + results: [{ taskId: "t-2", level: 1, correct: false }], + question: "Which bird…", answer: "eagle (Aquila)", cost: 0.8, steps: 40, + }, + // produce blew up: onError fallback answer + error message + { taskId: "t-3", level: 2, correct: false, answer: "", question: "How many…", produceError: "AI_NoObjectGeneratedError" }, + // gave up: empty answer, no error + { taskId: "t-4", level: 2, correct: false, answer: "", question: "In what year…" }, + ], + }), + ctxStub, + ); + assert.equal(out.n, 4); + assert.equal(out.correctCount, 1); + assert.equal(out.accuracy, 0.25); + assert.equal(out.fitness, 0.25); // the field eval/evolve-loop reads + assert.deepEqual(out.byLevel, { "1": { correct: 1, total: 2 }, "2": { correct: 0, total: 2 } }); + assert.equal(out.results[0].correct, true); + assert.equal(out.results[0].cost, 0.5); // unpacked from runResult.output in code + assert.equal(out.results[0].steps, 12); + assert.equal(out.results[0].question, undefined); // question excerpts ride on MISSES only + assert.equal(out.results[1].correct, false); // count+results normalization + assert.equal(out.results[1].miss, "wrong-answer"); + assert.equal(out.results[2].miss, "produce-error"); + assert.equal(out.results[3].miss, "empty-answer"); + assert.ok(out.text.includes("accuracy 0.25 (1/4 correct)")); + assert.ok(out.text.includes("1 wrong-answer") && out.text.includes("1 empty-answer") && out.text.includes("1 produce-error")); + assert.ok(out.text.includes('answered: "eagle (Aquila)"')); + assert.ok(out.text.includes("question: Which bird…")); + assert.ok(out.text.includes("ERROR: AI_NoObjectGeneratedError")); + console.log("✔ gaia/digest-results: shape normalization, miss taxonomy, fitness, text"); + + // 5. the generic loop climbs the gaia digest's `fitness` field (accuracy) + // with improveMargin 0 — any task flip counts — and names the fitness + // "accuracy" in briefings. + const loop = registry["eval/evolve-loop"]!; + const genCalls: any[] = []; + const rates = [0.4, 0.6]; + const fakeOpt = { + run: async (_name: string, input: any) => { + genCalls.push(input); + const g = input.generation as number; + return { + runId: `genrun-${g}`, + status: "success", + output: { + candidate: input.candidateName, + version: `v${g + 1}`, + summary: `approach ${g}`, + authorCost: 1, + digest: { fitness: rates[g], text: `digest ${g}`, results: [{ cost: 2 }] }, + }, + }; + }, + }; + const loopOut: any = await loop.run( + loop.input.parse({ + tasks: ["t-1", "t-2"], + mission: "m", + baseline: { fitness: 0.4, text: "baseline digest" }, + candidateName: "gaia-produce-ai", + baseWorkflow: "gaia-produce", + genWorkflow: "gaia-evolve-gen", + fitnessName: "accuracy", + maxGenerations: 5, + stopFitness: 0.6, + improveMargin: 0, + exploreAfter: 2, + }), + { ...ctxStub, services: { optimizer: fakeOpt } }, + ); + assert.equal(genCalls.length, 2); // gen 1 hit stopFitness 0.6 + assert.equal(loopOut.stopReason, "stopFitness 0.6 reached"); + assert.equal(loopOut.bestGen, 1); + assert.equal(loopOut.bestVersion, "v2"); + assert.equal(loopOut.bestFitness, 0.6); + assert.equal(loopOut.baselineFitness, 0.4); + assert.equal(loopOut.improved, true); + assert.ok(genCalls[0].briefing.includes("mean accuracy 0.4")); + assert.ok(genCalls[1].briefing.includes('the seeded produce workflow "gaia-produce"')); + // margin 0 semantics: a TIE (0.4 vs baseline 0.4) does not become best + assert.ok(genCalls[1].briefing.includes("BEST SO FAR: the baseline itself")); + console.log("✔ eval/evolve-loop: climbs gaia `fitness`, accuracy naming, margin-0 tie handling"); + + console.log("\nALL GAIA EVOLVE VALIDATION CHECKS PASSED"); + } finally { + rmSync(base, { recursive: true, force: true }); + } +} + +main().then( + () => process.exit(0), + (err) => { + console.error(err); + process.exit(1); + }, +); diff --git a/mcp/src/lab/gaia/seed.ts b/mcp/src/lab/gaia/seed.ts index 388d26bcf..e1d21358f 100644 --- a/mcp/src/lab/gaia/seed.ts +++ b/mcp/src/lab/gaia/seed.ts @@ -18,6 +18,13 @@ import type { WorkspaceManager } from "vein"; * - `gaia/evaluate` — the real leaderboard scorer. HARNESS-ONLY: grant only * to harness workflows, never to a producing agent's `agentTools`. * - `gaia/pack-result`, `gaia/summarize-batch` — pure combiners. + * - `gaia/digest-results` — aggregate graded results into the evolve loop's + * propose digest (verdict channel only; accuracy as `fitness`). + * + * The evolve harness (gaia-candidate-run / gaia-evolve-gen / gaia-evolve) + * mirrors harvey's, driven by the generic `eval/evolve-loop`. All seeded + * UNSTAMPED, so the meta surface can read but never edit, run, or + * overwrite them. * * Seeding is content-hash reconciled: the committed copy is authoritative at * boot. A workspace-side evolution of these survives restarts only once it's @@ -29,6 +36,7 @@ const SEED_STEPS: Array<{ file: string; type: string }> = [ { file: "evaluate.ts", type: "gaia/evaluate" }, { file: "pack-result.ts", type: "gaia/pack-result" }, { file: "summarize-batch.ts", type: "gaia/summarize-batch" }, + { file: "digest-results.ts", type: "gaia/digest-results" }, ]; const HERE = dirname(fileURLToPath(import.meta.url)); @@ -49,6 +57,21 @@ const SEED_WORKFLOWS: Array<{ name: string; description: string }> = [ description: "GAIA batch harness: list-tasks (by level) -> first `limit` -> produce per task via gaia-produce -> one gaia/evaluate call -> merged report {accuracy, byLevel, perTask, totalCost, totalSteps}.", }, + { + name: "gaia-candidate-run", + description: + "Run an ai-stamped candidate produce workflow on ONE GAIA task via meta/run-workflow (own runId, fresh registry) and score its answer with the real scorer (fromRun unpack — a failed run scores as an honest zero). Input: { workflow, version?, taskId }. Output: { taskId, candidate, version, correct, answer, level, question, produceStatus, runResult, … }.", + }, + { + name: "gaia-evolve-gen", + description: + "ONE GENERATION of the gaia evolution loop: meta/* authoring agent publishes a candidate version -> gaia-candidate-run over the task set (pinned version) -> gaia/digest-results. Invoked by eval/evolve-loop with { tasks, mission, candidateName, generation, briefing }. Output: { candidate, generation, version, summary, changes, missingSecrets, authorCost, authorSteps, digest }.", + }, + { + name: "gaia-evolve", + description: + "GAIA authoring harness (hill-climb): baseline gaia-run over the task set -> digest -> eval/evolve-loop over gaia-evolve-gen generations (accuracy fitness, exact-match so improveMargin 0) -> report with best version vs baseline. TRAIN scores — validate the best version on held-out tasks before promoting. Input: { tasks: [taskId, …], mission, generations? }.", + }, ]; export async function seedGaiaWorkflows(workspace: WorkspaceManager): Promise { diff --git a/mcp/src/lab/gaia/steps/digest-results.ts b/mcp/src/lab/gaia/steps/digest-results.ts new file mode 100644 index 000000000..5a5674ff9 --- /dev/null +++ b/mcp/src/lab/gaia/steps/digest-results.ts @@ -0,0 +1,157 @@ +import { z, defineStep } from "vein"; + +/** + * Aggregate graded GAIA results into the compact digest the evolve loop's + * PROPOSE beat consumes (EVOLVE_SPEC §2: a proposer must see the aggregate + * across the dataset, never one example, or it overfits). + * + * GOLD DISCIPLINE: what leaves this step is the VERDICT channel only — + * correct/wrong per task, plus the candidate's OWN answer (its output, not + * the gold) and the question text (task-visible to every producer anyway). + * The gold never appears in any input here: gaia/evaluate results carry + * only { taskId, level, correct }, and gaia.getTask strips `Final answer`. + * + * FITNESS: plain accuracy, emitted as `fitness` (the field eval/evolve-loop + * reads). Binary per task is fine here — unlike harvey's all-pass score, + * the task SET supplies the gradient (each flip moves accuracy by 1/n). + * + * MISS ROUTING (EVOLVE_SPEC §8's taxonomy, the cheap code-only version): + * each miss is tagged from mechanical signals — produce ERROR (harness/ + * tooling blew up), EMPTY answer (the agent gave up or crashed into the + * fallback), or plain WRONG (formatting or substance — the answer excerpt + * is there so the author can tell which). The summary line counts them so + * an author sees at a glance which layer owns the misses. + * + * Input entries are gaia-run / gaia-candidate-run outputs. Field access is + * defensive: gaia-run reports `correct` as a COUNT with a per-task + * `results` array (its score call), gaia-candidate-run as a boolean. + */ + +interface AnyRec { + [k: string]: unknown; +} + +function truncate(s: string, max: number): string { + const t = s.replace(/\s+/g, " ").trim(); + return t.length > max ? t.slice(0, max) + " […]" : t; +} + +function str(v: unknown): string | undefined { + return typeof v === "string" && v.trim() ? v : undefined; +} + +/** An error field may be a string or an { message } object (a RunResult's + * error) — normalize both. */ +function errStr(v: unknown): string | undefined { + if (typeof v === "string") return str(v); + if (v && typeof v === "object") return str((v as AnyRec)["message"]); + return undefined; +} + +function num(v: unknown): number | undefined { + return typeof v === "number" && Number.isFinite(v) ? v : undefined; +} + +/** Normalize the three correctness shapes that reach this step: + * boolean (gaia-candidate-run), a single-entry score-results array + * (gaia-run carries its score call's `results`), or a 0/1 count with + * total 1 (gaia-run's `correct`). Unreadable → false, never true. */ +function correctOf(r: AnyRec): boolean { + if (typeof r["correct"] === "boolean") return r["correct"] as boolean; + const arr = Array.isArray(r["results"]) ? (r["results"] as AnyRec[]) : []; + if (arr.length === 1 && typeof arr[0]?.["correct"] === "boolean") return arr[0]["correct"] as boolean; + if (typeof r["correct"] === "number" && num(r["total"]) === 1) return r["correct"] === 1; + return false; +} + +export default defineStep({ + type: "gaia/digest-results", + description: + "Aggregate an array of graded GAIA results (gaia-run / gaia-candidate-run outputs) into a compact digest: accuracy as `fitness` (what eval/evolve-loop reads), per-task correct/wrong with the produced answer excerpt, miss tags (wrong-answer / empty-answer / produce-error), question excerpts for misses, and a preformatted `text` block for an LLM prompt. Gold never enters or leaves this step. Config: results (array), maxAnswerChars? (default 160), maxQuestionChars? (default 240). Output: { n, correctCount, accuracy, fitness, byLevel, results, text }.", + input: z.object({ + results: z.array(z.any()).describe("graded results, one per task (gaia-run / gaia-candidate-run outputs)"), + maxAnswerChars: z.number().int().positive().default(160).describe("max characters of the produced answer per task"), + maxQuestionChars: z + .number() + .int() + .positive() + .default(240) + .describe("max characters of the question excerpt shown for missed tasks"), + }), + output: z.any(), + async run(cfg) { + const entries = (cfg.results as AnyRec[]).map((raw, i) => { + const r = (raw ?? {}) as AnyRec; + const taskId = str(r["taskId"]) ?? `result ${i + 1}`; + const level = num(r["level"]) ?? null; + const correct = correctOf(r); + const answer = typeof r["answer"] === "string" ? (r["answer"] as string) : ""; + const error = errStr(r["error"]) ?? errStr(r["produceError"]) ?? errStr(r["gradeError"]); + // gaia-candidate-run carries cost/steps only inside the candidate's + // run result (YAML cannot deep-access a failed run's output — the + // template evaluator does not short-circuit), so unpack here in code. + const runOut = ((r["runResult"] as AnyRec | undefined)?.["output"] ?? {}) as AnyRec; + const cost = num(r["cost"]) ?? num(runOut["cost"]); + const steps = num(r["steps"]) ?? num(runOut["steps"]); + // Route each miss to the layer that owns it (§8): a produce error is + // harness/tooling, an empty answer is a give-up (the onError fallback + // or a bailed agent), a non-empty wrong answer is formatting or + // substance — the excerpt lets the author tell which. + const miss = correct ? null : error ? "produce-error" : answer.trim() === "" ? "empty-answer" : "wrong-answer"; + return { + taskId, + level, + correct, + answer: truncate(answer, cfg.maxAnswerChars), + ...(miss ? { miss } : {}), + ...(correct ? {} : { question: truncate(str(r["question"]) ?? "", cfg.maxQuestionChars) }), + ...(cost != null ? { cost } : {}), + ...(steps != null ? { steps } : {}), + ...(error ? { error: truncate(error, 240) } : {}), + }; + }); + + const n = entries.length; + const correctCount = entries.filter((e) => e.correct).length; + const accuracy = n ? Math.round((correctCount / n) * 1000) / 1000 : 0; + const byLevel: Record = {}; + for (const e of entries) { + const key = e.level == null ? "?" : String(e.level); + byLevel[key] ??= { correct: 0, total: 0 }; + byLevel[key].total += 1; + if (e.correct) byLevel[key].correct += 1; + } + const missCounts: Record = {}; + for (const e of entries) { + if (e.miss) missCounts[e.miss] = (missCounts[e.miss] ?? 0) + 1; + } + + // Preformatted for an LLM prompt ({{ digest.text }}) — objects + // interpolated into template strings would arrive as raw JSON. + const missSummary = Object.entries(missCounts) + .map(([k, v]) => `${v} ${k}`) + .join(", "); + const lines: string[] = [ + `${n} task(s) — accuracy ${accuracy} (${correctCount}/${n} correct)` + + (missSummary ? `; misses: ${missSummary}` : ""), + ]; + for (const e of entries) { + const lvl = e.level == null ? "" : ` (L${e.level})`; + const meta = [e.steps != null ? `steps ${e.steps}` : "", e.cost != null ? `$${e.cost}` : ""] + .filter(Boolean) + .join(", "); + if (e.correct) { + lines.push(`- ${e.taskId}${lvl} ✓ correct${meta ? ` (${meta})` : ""}`); + continue; + } + lines.push( + `- ${e.taskId}${lvl} ✗ ${e.miss?.toUpperCase()}${meta ? ` (${meta})` : ""}` + + (e.miss === "wrong-answer" ? ` — answered: "${e.answer}"` : ""), + ); + if (e.question) lines.push(` question: ${e.question}`); + if (e.error) lines.push(` ERROR: ${e.error}`); + } + + return { n, correctCount, accuracy, fitness: accuracy, byLevel, results: entries, text: lines.join("\n") }; + }, +}); diff --git a/mcp/src/lab/gaia/steps/evaluate.ts b/mcp/src/lab/gaia/steps/evaluate.ts index 85d1a948b..fbada638a 100644 --- a/mcp/src/lab/gaia/steps/evaluate.ts +++ b/mcp/src/lab/gaia/steps/evaluate.ts @@ -4,11 +4,24 @@ import { z, defineStep, type StepContext, type VeinCapabilities } from "vein"; * HARNESS-ONLY plumbing over ctx.services.gaia.score. Never expose this * step (or any gaia/*) as an agentTool to a producing agent — it would let * the agent see grading/scoring internals or self-grade. + * + * Two modes: + * - `pairs`: score explicit { taskId, answer } pairs (gaia-run/gaia-batch). + * - `fromRun`: score ONE candidate run's reported answer (gaia-candidate- + * run). The whole run result is passed in and unpacked HERE, in code — + * the template evaluator does not short-circuit, so YAML can never + * safely deep-access `run.output.answer` on a failed run (EVOLVE_SPEC + * §5.3.5). A failed run or missing/non-string answer scores as "" — a + * certainly-wrong answer, an honest zero, never an aborted batch. + * Output adds { taskId, answer, isCorrect, level, question } convenience + * fields (`isCorrect` — the report's `correct` is a COUNT; `question` + * comes from gaia.getTask, which strips the gold, and is null rather + * than fatal when the lookup fails — grading never dies on metadata). */ export default defineStep({ type: "gaia/evaluate", description: - "Score answer pairs against the GAIA gold set via ctx.services.gaia.score(pairs). HARNESS-ONLY — never grant to a producing agent's agentTools. Config: pairs [{ taskId, answer }]. Output: { accuracy, correct, total, byLevel, results, benchmarkRev, scorerSha256 }.", + "Score answers against the GAIA gold set via ctx.services.gaia.score(pairs). HARNESS-ONLY — never grant to a producing agent's agentTools. Config: pairs [{ taskId, answer }] OR fromRun { taskId, run } (a candidate RunResult — the answer is unpacked in code, '' when the run failed). Output: { accuracy, correct, total, byLevel, results, benchmarkRev, scorerSha256 }; fromRun adds { taskId, answer, isCorrect, level, question }.", input: z.object({ pairs: z .array( @@ -17,13 +30,44 @@ export default defineStep({ answer: z.string(), }), ) - .min(1), + .min(1) + .optional(), + fromRun: z + .object({ + taskId: z.string(), + run: z.any().describe("the candidate run's RunResult ({ runId, status, output?, error? })"), + }) + .optional(), }), output: z.any(), async run(cfg, ctx) { const c = ctx as StepContext; const gaia = c.services?.gaia; if (!gaia) throw new Error("gaia capability unavailable in this deployment"); - return await gaia.score(cfg.pairs); + if (!cfg.pairs === !cfg.fromRun) { + throw new Error("gaia/evaluate: provide exactly one of `pairs` or `fromRun`"); + } + + if (cfg.pairs) return await gaia.score(cfg.pairs); + + const { taskId, run } = cfg.fromRun!; + const output = (run && typeof run === "object" ? (run as any).output : undefined) ?? {}; + const answer = typeof output.answer === "string" ? output.answer : ""; + const report = await gaia.score([{ taskId, answer }]); + const first = report.results?.[0]; + let question: string | null = null; + try { + question = (await gaia.getTask(taskId))?.question ?? null; + } catch { + // metadata only (for the digest) — never fail a grade over it + } + return { + ...report, + taskId, + answer, + isCorrect: first?.correct === true, + level: first?.level ?? null, + question, + }; }, }); diff --git a/mcp/src/lab/gaia/workflows/gaia-candidate-run.yaml b/mcp/src/lab/gaia/workflows/gaia-candidate-run.yaml new file mode 100644 index 000000000..15f9d89d5 --- /dev/null +++ b/mcp/src/lab/gaia/workflows/gaia-candidate-run.yaml @@ -0,0 +1,91 @@ +name: gaia-candidate-run +# Run an AGENT-AUTHORED candidate produce workflow on ONE GAIA task, then +# score its answer with the real leaderboard scorer. The harness half of the +# EVOLVE_SPEC §5.2 loop (gaia instance, mirroring harvey-candidate-run): +# +# author (meta/*) → run (THIS) → grade (THIS) → reflect +# +# The candidate runs via meta/run-workflow (services.authoring), which +# (a) runs ONLY workflows stamped publisher "ai" — i.e. published via +# meta/publish-workflow. It refuses the seeded gaia-produce; baseline +# runs go through gaia-run instead. So this harness can never be +# pointed back at (or used to launder) the seeded harness surface. +# (b) rebuilds the step registry FRESH at call time — a candidate STEP +# authored earlier in the same enclosing run is visible (§5.3.1), +# which a plain subflow (start-of-run registry snapshot) can't do. +# (c) runs the candidate under its OWN runId — its artifacts dir (where +# gaia/get-task stages the attached file) is its own, so parallel +# candidate runs never collide. +# +# Grading passes the WHOLE run result to gaia/evaluate as `fromRun` and +# unpacks the answer in code (template expressions cannot guard deep access +# on a failed run's missing `output` — the evaluator does not +# short-circuit, §5.3.5). A failed candidate run scores as "" — an honest +# zero with the cause attributable via produceStatus/error — and a batch +# over many tasks always completes with one result per task. +# +# CANDIDATE CONTRACT (what the author must publish): input { taskId }, +# output (last step) includes taskId, answer (the bare final-answer +# string), cost, steps. +# +# Input: { workflow, version?, taskId } +# workflow: candidate name (publisher "ai"); version pins it (default: active) +# Output: { taskId, candidate, version, produceRunId, produceStatus, +# correct, level, answer, question, cost, steps, benchmarkRev, +# scorerSha256, runResult, error?, gradeError? } +steps: + - id: run + type: meta/run-workflow + config: + name: "{{ input.workflow }}" + version: "{{ input.version }}" + input: + taskId: "{{ input.taskId }}" + + # question/level/answer come from the grade step (fromRun mode unpacks the + # run's answer and looks up gold-stripped task metadata IN CODE), never + # from deep template access on the candidate's output (it may be missing). + - id: grade + type: gaia/evaluate + depends: run + options: + retry: { max: 1, delayMs: 15000 } + # Shape-stable fallback: every field the result step reads one level + # deep must exist here (§5.3.5). + onError: + id: grade_failed + type: gaia/pack-result + config: + taskId: "{{ input.taskId }}" + isCorrect: false + answer: "" + level: null + question: null + gradeError: "{{ $error.message }}" + config: + fromRun: + taskId: "{{ input.taskId }}" + run: "{{ run }}" + + - id: result + type: gaia/pack-result + depends: grade + config: + taskId: "{{ input.taskId }}" + candidate: "{{ input.workflow }}" + version: "{{ input.version }}" + produceRunId: "{{ run.runId }}" + produceStatus: "{{ run.status }}" + # object ({ message, … }), string (an authoring-gate refusal), or + # absent — gaia/digest-results normalizes all three. + error: "{{ run.error }}" + # The full candidate run result (runId, status, output, error) for + # humans debugging a candidate — inspect further via meta/get-run. + runResult: "{{ run }}" + correct: "{{ grade.isCorrect }}" + level: "{{ grade.level }}" + question: "{{ grade.question }}" + answer: "{{ grade.answer }}" + benchmarkRev: "{{ grade.benchmarkRev }}" + scorerSha256: "{{ grade.scorerSha256 }}" + gradeError: "{{ grade.gradeError }}" diff --git a/mcp/src/lab/gaia/workflows/gaia-evolve-gen.yaml b/mcp/src/lab/gaia/workflows/gaia-evolve-gen.yaml new file mode 100644 index 000000000..a05bfd7ef --- /dev/null +++ b/mcp/src/lab/gaia/workflows/gaia-evolve-gen.yaml @@ -0,0 +1,281 @@ +name: gaia-evolve-gen +# ONE GENERATION of the gaia evolution loop: author a candidate → run it +# over the task set → digest the scores. The outer loop (eval/evolve-loop, +# invoked by gaia-evolve) runs this workflow repeatedly, composing each +# generation's `briefing` from the baseline digest + every previous +# attempt's result, so successive authors hill-climb instead of starting +# blind (EVOLVE_SPEC §5.2's reflect beat, §5.3.3's loop-over-versions — +# the gaia instance, mirroring harvey-evolve-gen). +# +# The author is the standard meta/* authoring agent (§5.3.2 grants: meta/* +# and the editor tool, never bash, never graders). It sees correct/wrong +# verdicts and the candidate's own answers only — never the gold — and +# cannot grade its own candidate; grading happens here, after it finishes. +# +# Input: { tasks, mission, candidateName, generation, briefing } +# tasks: GAIA taskIds (the TRAIN set) +# briefing: composed by eval/evolve-loop — baseline digest, previous +# attempts (version, accuracy, approach, misses), best-so-far +# anchor, and the exploit-or-explore directive. +# Output: { candidate, generation, version, summary, changes, +# missingSecrets, authorCost, authorSteps, digest } +# digest: the gaia/digest-results object for this generation's candidate +# runs (fitness = accuracy is what the loop climbs). +steps: + - id: dir + type: artifacts/dir + config: + sub: author + + # ── propose: the authoring agent ─────────────────────────────────────── + - id: author + type: agent + depends: dir + options: + retry: { max: 1, delayMs: 15000 } + config: + cwd: "{{ dir.path }}" + system: "{{ params.authorSystem }}" + prompt: | + MISSION: + {{ input.mission }} + + TASK SET ({{ input.tasks.length }} GAIA benchmark task(s)): + {{ input.tasks }} + + This is GENERATION {{ input.generation }} of an evolution run. + + {{ input.briefing }} + + Author an improved CANDIDATE produce workflow named EXACTLY + "{{ input.candidateName }}", following the DIRECTIVE above. Address + miss patterns that RECUR across tasks (never one task's specifics). + Keep the hard contract, publish, smoke-test on ONE task from the + task set, and return the result. + {{ params.authorGuidance }} + model: "{{ params.authorModel }}" + maxSteps: "{{ params.authorMaxSteps }}" + # No bash, no web_search, no file browsing — the author's ONLY levers + # are the meta/* authoring steps (§5.3.2). The editor tool is kept so + # it can draft YAML in its (empty) scratch dir before publishing. + toolFilter: ["str_replace_based_edit_tool"] + agentTools: ["meta/*"] + schema: + type: object + properties: + candidate: + type: string + description: "the published candidate workflow name" + version: + type: string + description: "EXACT version string of the final healthy publish (from meta/publish-workflow)" + summary: + type: string + description: "the APPROACH this generation took and why — tied to the directive and the recurring miss patterns; the next generation reads this to know what has been tried" + changes: + type: array + items: { type: string } + description: "one line per concrete change" + missingSecrets: + type: array + description: "credentials the candidate needs that meta/list-secrets does not show — the capture artifact a human fulfills" + items: + type: object + properties: + name: { type: string } + why: { type: string } + required: [name, why] + additionalProperties: false + required: [candidate, version, summary] + additionalProperties: false + + # ── resolve: never trust the author's echo ───────────────────────────── + # The candidate NAME is harness-pinned (input.candidateName) — grading the + # string the author returned once zeroed two generations on a literal + # "placeholder" (in schema mode, one no-tool-call text turn ends the agent + # loop and IS the structured output). The VERSION falls back to the + # candidate's active version — the author's own last publish, since + # generations run sequentially — when the echoed pin is empty or bogus. + # meta/get-workflow returns { error } instead of throwing, so `vpin.version` + # is undefined on a bad pin and the `||` falls through to the active one. + - id: vactive + type: meta/get-workflow + depends: author + config: + name: "{{ input.candidateName }}" + + - id: vpin + type: meta/get-workflow + depends: author + config: + name: "{{ input.candidateName }}" + version: "{{ author.object.version }}" + + # ── evaluate: the pinned candidate over the task set ─────────────────── + - id: candeval + type: foreach + depends: [vpin, vactive] + config: + items: "{{ input.tasks }}" + body: + id: one + type: subflow + config: + workflow: gaia-candidate-run + input: + workflow: "{{ input.candidateName }}" + version: "{{ vpin.version || vactive.version }}" + taskId: "{{ $current }}" + + - id: canddigest + type: gaia/digest-results + depends: candeval + config: + results: "{{ candeval }}" + + - id: result + type: gaia/pack-result + depends: canddigest + config: + candidate: "{{ input.candidateName }}" + generation: "{{ input.generation }}" + # The version that was actually GRADED (resolved above) — the loop's + # briefing and EXPLOIT anchor read this, so an author's garbage echo + # must not poison the lineage. + version: "{{ vpin.version || vactive.version }}" + summary: "{{ author.object.summary }}" + changes: "{{ author.object.changes }}" + missingSecrets: "{{ author.object.missingSecrets }}" + authorCost: "{{ author.cost }}" + authorSteps: "{{ author.steps }}" + digest: "{{ canddigest }}" + +params: + authorModel: claude-sonnet-5 + # Generous — an author that reads several workflows, drafts steps, and + # smoke-tests burns calls fast; the publish-early rule in authorSystem is + # what protects a generation whose budget runs out anyway. + authorMaxSteps: 200 + # Extra per-run steering appended to the author's task prompt (experiment + # surface). + authorGuidance: "" + # The author's persona + method — layer-1 tunable like any other prompt. + authorSystem: | + You are a WORKFLOW AUTHOR inside a self-evolving eval harness. You + improve HOW a GAIA benchmark answer gets produced by authoring a new + version of a produce workflow. You never answer benchmark questions + yourself, and you never see the gold answers — only correct/wrong + verdicts plus the candidate's own (wrong) answers. + + You are ONE GENERATION in a hill-climbing loop: earlier generations' + attempts and results are in your task prompt, and your published version + and summary will be handed to the next generation. Follow the DIRECTIVE + in the briefing — build on the best attempt when told to exploit, and + genuinely change strategy when told to explore. The fitness is exact- + match ACCURACY over the task set: every task flip moves it by 1/n, and + the same workflow re-run can flip tasks by sampling luck alone — chase + recurring MISS PATTERNS, never one task's noise. + + WHAT MOVES GAIA SCORES, in rough order of leverage: + 1. ANSWER FORMATTING. Grading is quasi-exact string match. A correct + fact in the wrong FORM (units included when not asked, "two" vs 2, + thousands separators, list order, singular/plural, extra words) is + a miss. Look at each wrong answer in the digest: if it is + substantively right but shaped wrong, the fix is format discipline — + re-read-the-question rules, a final format-check pass, or a + separate fresh-eyes formatting step. + 2. PERSISTENCE. An EMPTY answer means the producer crashed or gave up. + Fixes live in retry/onError plumbing, step budgets (maxSteps), and + try-a-different-approach prompt rules. + 3. TOOLING. Wrong extractions (garbled PDF, misread spreadsheet, + unfetched page) need different tool guidance or a different + extraction path in the prompt. + 4. STRUCTURE. When one agent drifts, split the work: research vs + answer vs format-check as separate steps. + + Your only levers are the meta/* authoring tools (the workspace surface): + meta/get-workflow and meta/list-workflows to read existing YAML; + meta/list-steps, meta/search-steps, meta/get-step to discover step + types; meta/create-step / meta/edit-step to author new custom steps + (TypeScript, defineStep, imports from "vein" and node builtins); + meta/run-step to test ONE step cheaply; meta/publish-workflow to publish + the candidate; meta/run-workflow to run it; meta/list-runs, + meta/get-run, meta/search-runs to inspect those runs; meta/list-secrets + for available credential NAMES. + + METHOD + 1. Read the DIRECTIVE's anchor first: the best-so-far candidate version + (meta/get-workflow with the exact version) when exploiting, or the + base produce workflow when nothing has beaten the baseline. Your + candidate starts from that YAML, changed only where the evidence + points. + 2. PUBLISH EARLY. Before any deep step-authoring, publish a first + candidate version (meta/publish-workflow) that is a modest, safe + improvement on your anchor. The harness grades whatever your FINAL + answer names: a rough published candidate beats a perfect + unpublished one, and if you run out of steps with nothing published, + this generation's budget is wasted. Republishing the same name + creates a new version — that is how you iterate upward. + 3. BUDGET YOUR CALLS. You have a hard tool-call cap. Reserve the LAST + ~5 calls for: final meta/publish-workflow, one verification read, + and your final structured answer. Time-box debugging. + 4. SMOKE-TEST on ONE task from the task set (meta/run-workflow), then + inspect (meta/get-run, meta/search-runs). You cannot grade + correctness — the harness grades after you finish. Check STRUCTURE + only: the run succeeds and the output object carries taskId, a + non-empty answer string, cost, steps. Fix and republish until + healthy. Runs cost real money: one task, at most two smoke runs. + 5. Steps you (or a prior generation) already authored may exist — + check meta/list-steps before creating: edit and reuse them instead + of re-authoring from scratch. But when the DIRECTIVE says explore, + reuse must not drag you back into an approach that already failed. + + WORKFLOW DESIGN PATTERNS — vocabulary, not prescription. A candidate + may be ANY shape that honors the hard contract: compose these, adapt + them, or invent shapes not listed here. The one structural insight to + hold on to: an agent reviewing its own draft inside its own context is + anchored to it — only a SEPARATE agent step with a clean context gets + an unbiased re-read. A longer prompt cannot buy fresh eyes; structure + can. + - fresh-eyes format checker: after the answering agent, a separate + CHEAP agent step re-reads ONLY the question text + the draft answer + and emits the corrected bare answer (units, number form, list order, + extra words). Fits: substantively-right-but-shaped-wrong misses. + - researcher → answerer: a research agent gathers evidence into a + memo; the answering agent consumes the memo and the question. Fits: + wrong facts from shallow searching. + - verify-with-code: prompt rules (or a dedicated step) forcing a + python-via-bash computation for counting/logic/date questions. + Fits: arithmetic and bookkeeping slips. + - dual attempt + reconcile: two independent produce attempts and a + reconciler that picks (or re-derives) the answer when they disagree. + Expensive — buy it only if the digest shows sampling flips. + Every added step costs real money — buy structure only where the + digest shows a recurring miss it answers. + + HARD CONTRACT for the candidate (the harness runs it as-is): + - input { taskId }. + - use the gaia/get-task step (plumbing; the gold is stripped) to fetch + the question/level and stage the task's attached file into the run's + artifacts dir; give agent steps that dir as cwd via artifacts/dir, + exactly as the base workflow does. + - the LAST step (use gaia/pack-result) must output: taskId, question, + level, answer (the BARE final-answer string), cost, steps. + - NEVER use gaia/evaluate anywhere in the candidate — self-grading at + produce time is oracle access, and a candidate that embeds the + grader is disqualified at promotion review. NEVER grant gaia/*, + eval/*, or meta/* to any agent step's agentTools — graders and the + authoring surface are off-limits to producers. Steps you author + yourself and other registry namespaces are fair game. + - keep prompts in params (the tuning surface), keep the produce agent's + retry + onError empty-answer fallback so one bad task cannot kill a + batch. + - secrets: reference NAMES from meta/list-secrets only; never paste + credential values. If a credential you need does not exist, record + it in the missingSecrets output and make the candidate degrade + gracefully without it. + + When done, return: the candidate name, the EXACT version string of the + final healthy publish, a summary of the APPROACH this generation took + (the next generation reads it to know what has been tried), the list of + changes, and any missing secrets. diff --git a/mcp/src/lab/gaia/workflows/gaia-evolve.yaml b/mcp/src/lab/gaia/workflows/gaia-evolve.yaml new file mode 100644 index 000000000..c56a6bc19 --- /dev/null +++ b/mcp/src/lab/gaia/workflows/gaia-evolve.yaml @@ -0,0 +1,135 @@ +name: gaia-evolve +# The AUTHORING HARNESS (EVOLVE_SPEC §5.2 / §9.4) for GAIA — a HILL-CLIMB +# (§5.3.3, gaia instance of the generic eval/evolve-loop): baseline once, +# then up to maxGenerations author→run→grade cycles, each generation +# briefed with every previous attempt's version, accuracy, approach, and +# misses — anchored to the best-so-far, flipping to an explicit "try a +# genuinely different approach" directive after `exploreAfter` +# non-improving attempts. +# +# capture run the BASELINE produce workflow over the task set, score it +# with the real leaderboard scorer, digest the misses (tagged +# wrong-answer / empty-answer / produce-error — §8's taxonomy) +# loop eval/evolve-loop runs gaia-evolve-gen per generation: an +# authoring agent (meta/* only — §5.3.2) publishes a new +# version of the candidate, the harness runs it pinned over the +# same tasks and digests the scores; the loop composes the next +# generation's briefing and directive from the results +# promote the report: best version + its accuracy vs baseline, the +# full generation history, costs. Promotion stays a HUMAN act — +# point gaia-run's input.produceWorkflow at the best version, +# or fold its diff into the seeded gaia-produce, after review. +# Review must also check the candidate never embeds +# gaia/evaluate (produce-time oracle access — §6). +# +# MEASUREMENT DISCIPLINE (EVOLVE_SPEC §6/§7): +# - Scoring is exact-match, so there is no judge noise — improveMargin +# defaults to 0 (any task flip counts). The noise that remains is +# PRODUCE-SAMPLING noise: the same workflow re-run can flip tasks by +# luck, and on n tasks one flip is 1/n. Small task sets make single +# flips loud — n=5 is an anecdote (§7); prefer ~10+ train tasks. +# - Every score in this report is a TRAIN score on the very tasks the +# candidates were tuned against. Validate the best version on held-out +# tasks (gaia-batch over a different slice) before believing the delta. +# - Authors cannot grade: grading happens in the harness after each +# author finishes, and candidates run under their OWN runIds via +# meta/run-workflow (gaia-candidate-run). +# +# COST: roughly (1 + generations) × |tasks| × produce + each generation's +# author (incl. its ≤2 smoke runs). Scoring is a local python subprocess — +# free. Start small; raise input.generations (cap 20) to let it rip. +# +# Needs: ANTHROPIC_API_KEY, HF_TOKEN (or a populated GAIA_DIR checkout; +# git-lfs for attachments). +# Input: { tasks: [taskId, …], mission, generations? } +# mission: the standing gap statement handed to every generation's +# author (the digest evidence rides alongside it and, per +# the author method, outranks it). +# generations: override params.maxGenerations for this run (e.g. 10). +# Output: { candidate, bestGen, bestVersion, bestAccuracy, +# baselineAccuracy, improved, generations, totalKnownCost, +# stopReason, baseline, … } +steps: + # ── capture: baseline over the task set ──────────────────────────────── + - id: baseline + type: foreach + config: + items: "{{ input.tasks }}" + body: + id: one + type: subflow + config: + workflow: gaia-run + input: + taskId: "{{ $current }}" + + - id: basedigest + type: gaia/digest-results + depends: baseline + config: + results: "{{ baseline }}" + + # ── the hill-climb: author → run → grade, up to N generations ────────── + - id: evolve + type: eval/evolve-loop + depends: basedigest + config: + tasks: "{{ input.tasks }}" + mission: "{{ input.mission }}" + baseline: "{{ basedigest }}" + candidateName: "{{ params.candidateName }}" + baseWorkflow: "{{ params.baseWorkflow }}" + genWorkflow: "{{ params.genWorkflow }}" + fitnessName: accuracy + maxGenerations: "{{ input.generations || params.maxGenerations }}" + stopFitness: "{{ params.stopAccuracy }}" + improveMargin: "{{ params.improveMargin }}" + exploreAfter: "{{ params.exploreAfter }}" + genParams: "{{ params.genParams }}" + + # ── promote: the reviewable report ───────────────────────────────────── + - id: report + type: gaia/pack-result + depends: evolve + config: + mission: "{{ input.mission }}" + tasks: "{{ input.tasks }}" + candidate: "{{ evolve.candidate }}" + bestGen: "{{ evolve.bestGen }}" + bestVersion: "{{ evolve.bestVersion }}" + bestAccuracy: "{{ evolve.bestFitness }}" + baselineAccuracy: "{{ evolve.baselineFitness }}" + accuracyDelta: "{{ evolve.bestFitness - evolve.baselineFitness }}" + improved: "{{ evolve.improved }}" + generations: "{{ evolve.generations }}" + totalKnownCost: "{{ evolve.totalKnownCost }}" + stopReason: "{{ evolve.stopReason }}" + baseline: "{{ basedigest.results }}" + note: "{{ evolve.note }}" + +params: + # The produce workflow the baseline runs and generation-0 authors start + # from (read via meta/get-workflow). + baseWorkflow: gaia-produce + # The candidate's name. Every generation republishes it — a NEW VERSION + # each time, so one evolve run leaves a versioned lineage (and the report + # pins the best version, which may not be the latest). + candidateName: gaia-produce-ai + # The one-generation workflow the loop runs (author → candidate eval → + # digest). Its own params (authorModel, authorMaxSteps, authorSystem, …) + # are the author's experiment surface — override per run via genParams. + genWorkflow: gaia-evolve-gen + maxGenerations: 5 + # Stop early once a generation's accuracy reaches this (1 = every task — + # reachable on tuned-against train sets; generations are the real cap). + stopAccuracy: 1 + # Exact-match scoring has no judge noise — any single task flip counts as + # an improvement. The residual produce-sampling noise is answered by the + # held-out validation the report's note demands, not by a margin. + improveMargin: 0 + # Consecutive non-improving attempts before the directive flips from + # "refine the best" to "try a genuinely different approach". + exploreAfter: 2 + # Param overrides for gaia-evolve-gen (e.g. { authorModel: "...", + # authorMaxSteps: 120, authorGuidance: "..." }). Empty = its defaults. + genParams: {} diff --git a/mcp/src/lab/gaia/workflows/gaia-run.yaml b/mcp/src/lab/gaia/workflows/gaia-run.yaml index c637c7cdc..f1573d180 100644 --- a/mcp/src/lab/gaia/workflows/gaia-run.yaml +++ b/mcp/src/lab/gaia/workflows/gaia-run.yaml @@ -1,14 +1,18 @@ name: gaia-run # Single-task harness: get-task -> produce (agent) -> evaluate (REAL scorer). # The producing agent NEVER gets gaia/* tools (see gaia-produce's params). -# Input: { taskId } +# `input.produceWorkflow` swaps in a different produce workflow (a seeded +# variant; ai-stamped candidates go through gaia-candidate-run instead, +# which runs them via meta/run-workflow under their own runId). +# Input: { taskId, produceWorkflow? } # Output: { accuracy, correct, total, byLevel, results, benchmarkRev, -# scorerSha256, taskId, question, level, answer, cost, steps } +# scorerSha256, produceWorkflow, taskId, question, level, answer, +# cost, steps } steps: - id: produced type: subflow config: - workflow: gaia-produce + workflow: "{{ input.produceWorkflow || 'gaia-produce' }}" input: taskId: "{{ input.taskId }}" @@ -28,6 +32,7 @@ steps: byLevel: "{{ score.byLevel }}" results: "{{ score.results }}" benchmarkRev: "{{ score.benchmarkRev }}" + produceWorkflow: "{{ input.produceWorkflow || 'gaia-produce' }}" taskId: "{{ produced.taskId }}" question: "{{ produced.question }}" level: "{{ produced.level }}" diff --git a/mcp/src/lab/harvey/evolve-smoke.ts b/mcp/src/lab/harvey/evolve-smoke.ts index 6ebd9fb23..4bfda92c7 100644 --- a/mcp/src/lab/harvey/evolve-smoke.ts +++ b/mcp/src/lab/harvey/evolve-smoke.ts @@ -12,6 +12,7 @@ import { mkdtempSync, rmSync } from "node:fs"; import { join } from "node:path"; import { WorkspaceManager, buildRegistry, fileArtifactsCapability, resolveConfig } from "vein"; import { seedHarveySteps, seedHarveyWorkflows } from "./seed.js"; +import { seedEvalSteps } from "../eval/seed.js"; import { seedArtifactSteps } from "../artifacts/seed.js"; import { createLabVein } from "../createLabVein.js"; @@ -20,6 +21,7 @@ async function main() { try { const workspace = new WorkspaceManager(join(base, "ws")); await seedHarveySteps(workspace); + await seedEvalSteps(workspace); // eval/evolve-loop — the generic hill-climb harvey-evolve wires await seedArtifactSteps(workspace); await seedHarveyWorkflows(workspace); @@ -35,7 +37,7 @@ async function main() { const { registry } = await buildRegistry(workspace.path); const wanted = [ "harvey/get-task", "harvey/evaluate", "harvey/pack-result", "harvey/digest-results", - "harvey/evolve-loop", "artifacts/dir", "meta/run-workflow", "agent", "subflow", "foreach", + "eval/evolve-loop", "artifacts/dir", "meta/run-workflow", "agent", "subflow", "foreach", ]; for (const t of wanted) assert.ok(registry[t], `registry has ${t}`); console.log("✔ all referenced step types resolve"); @@ -135,8 +137,8 @@ async function main() { // 5. the hill-climb loop, driven by a fake optimizer: // rates 0.85, 0.86, 0.86, 0.95 vs baseline 0.90 (improveMargin 0.02, // exploreAfter 2) → gens 0-1 exploit and fail to improve, gens 2-3 - // get the EXPLORE directive, gen 3 beats best and hits stopPassRate. - const loop = registry["harvey/evolve-loop"]!; + // get the EXPLORE directive, gen 3 beats best and hits stopFitness. + const loop = registry["eval/evolve-loop"]!; const genCalls: any[] = []; const rates = [0.85, 0.86, 0.86, 0.95]; const fakeOpt = { @@ -163,19 +165,21 @@ async function main() { mission: "m", baseline: { meanPassRate: 0.9, text: "baseline digest" }, candidateName: "harvey-produce-ai", + baseWorkflow: "harvey-produce", + genWorkflow: "harvey-evolve-gen", maxGenerations: 6, - stopPassRate: 0.94, + stopFitness: 0.94, improveMargin: 0.02, exploreAfter: 2, }), loopCtx, ); assert.equal(genCalls.length, 4); // stopped at gen 3 (0.95 ≥ 0.94), not 6 - assert.equal(loopOut.stopReason, "stopPassRate 0.94 reached"); + assert.equal(loopOut.stopReason, "stopFitness 0.94 reached"); assert.equal(loopOut.bestGen, 3); assert.equal(loopOut.bestVersion, "v4"); - assert.equal(loopOut.bestPassRate, 0.95); - assert.equal(loopOut.baselinePassRate, 0.9); + assert.equal(loopOut.bestFitness, 0.95); + assert.equal(loopOut.baselineFitness, 0.9); assert.equal(loopOut.improved, true); assert.equal(loopOut.totalKnownCost, 12); // 4 gens × (author 1 + produce 2) // directive flip: gens 0-1 exploit, gens 2-3 explore @@ -189,7 +193,7 @@ async function main() { assert.ok(genCalls[3].briefing.includes("attempt 2")); assert.ok(genCalls[3].briefing.includes("harvey-produce-ai@v3")); assert.ok(genCalls[3].briefing.includes("BEST SO FAR: the baseline itself")); - console.log("✔ harvey/evolve-loop: best-anchoring, explore flip, history, early stop"); + console.log("✔ eval/evolve-loop: best-anchoring, explore flip, history, early stop"); // two consecutive generation failures abort the loop const failOpt = { run: async () => ({ runId: "x", status: "failed", error: { message: "boom" } }) }; @@ -199,6 +203,8 @@ async function main() { mission: "m", baseline: { meanPassRate: 0.9, text: "b" }, candidateName: "c", + baseWorkflow: "harvey-produce", + genWorkflow: "harvey-evolve-gen", maxGenerations: 6, }), { ...ctxStub, services: { optimizer: failOpt } }, @@ -206,15 +212,16 @@ async function main() { assert.equal(failOut.stopReason, "two consecutive generation failures"); assert.equal(failOut.generations.length, 2); assert.equal(failOut.improved, false); - console.log("✔ harvey/evolve-loop: aborts after consecutive failures"); + console.log("✔ eval/evolve-loop: aborts after consecutive failures"); // 6. REGRESSION — the optimizer capability must be visible to RUNS. // createVein SPREADS the caller's services into a fresh bag, so // createLabVein's post-construction injection must land on // vein.services (the effective bag), not the local one. This broke - // silently once: eval/optimize and harvey/evolve-loop threw + // silently once: eval/optimize and eval/evolve-loop threw // "requires a services.optimizer capability" at run time while the - // local bag looked fine. Prove it end to end: boot the real lab + // local bag looked fine (eval/optimize + eval/evolve-loop). Prove it + // end to end: boot the real lab // vein, publish a probe step + workflow, and assert a RUN sees // services.optimizer. // Construction-only requirement: concept services demand a provider key diff --git a/mcp/src/lab/harvey/seed.ts b/mcp/src/lab/harvey/seed.ts index 38fe48a8d..4115216fc 100644 --- a/mcp/src/lab/harvey/seed.ts +++ b/mcp/src/lab/harvey/seed.ts @@ -18,13 +18,16 @@ import type { WorkspaceManager } from "vein"; * fallbacks). * - `harvey/digest-results` — aggregate graded results into the propose * digest (verdict channel only; see the step header). + * + * The hill-climb loop itself is the GENERIC `eval/evolve-loop` (seeded by + * eval/seed.ts) — harvey-evolve wires it with harvey's gen workflow and + * pass-rate fitness. */ const SEED_STEPS: Array<{ file: string; type: string }> = [ { file: "get-task.ts", type: "harvey/get-task" }, { file: "evaluate.ts", type: "harvey/evaluate" }, { file: "pack-result.ts", type: "harvey/pack-result" }, { file: "digest-results.ts", type: "harvey/digest-results" }, - { file: "evolve-loop.ts", type: "harvey/evolve-loop" }, ]; const HERE = dirname(fileURLToPath(import.meta.url)); diff --git a/mcp/src/lab/harvey/workflows/harvey-evolve-gen.yaml b/mcp/src/lab/harvey/workflows/harvey-evolve-gen.yaml index 006404cb1..e86d2c3c3 100644 --- a/mcp/src/lab/harvey/workflows/harvey-evolve-gen.yaml +++ b/mcp/src/lab/harvey/workflows/harvey-evolve-gen.yaml @@ -1,6 +1,6 @@ name: harvey-evolve-gen # ONE GENERATION of the harvey evolution loop: author a candidate → run it -# over the task set → digest the grades. The outer loop (harvey/evolve-loop, +# over the task set → digest the grades. The outer loop (eval/evolve-loop, # invoked by harvey-evolve) runs this workflow repeatedly, composing each # generation's `briefing` from the baseline digest + every previous # attempt's result, so successive authors hill-climb instead of starting @@ -12,7 +12,7 @@ name: harvey-evolve-gen # grading happens here, after it finishes, exactly as in harvey-evolve. # # Input: { tasks, mission, candidateName, generation, briefing } -# briefing: composed by harvey/evolve-loop — baseline digest, previous +# briefing: composed by eval/evolve-loop — baseline digest, previous # attempts (version, pass-rate, approach, failures), best-so-far # anchor, and the exploit-or-explore directive. # Output: { candidate, generation, version, summary, changes, diff --git a/mcp/src/lab/harvey/workflows/harvey-evolve.yaml b/mcp/src/lab/harvey/workflows/harvey-evolve.yaml index 4c717c18f..140d0ff66 100644 --- a/mcp/src/lab/harvey/workflows/harvey-evolve.yaml +++ b/mcp/src/lab/harvey/workflows/harvey-evolve.yaml @@ -8,7 +8,7 @@ name: harvey-evolve # # capture run the BASELINE produce workflow over the task set, grade it # with the real benchmark eval, digest the aggregate misses -# loop harvey/evolve-loop runs harvey-evolve-gen per generation: +# loop eval/evolve-loop (generic) runs harvey-evolve-gen per generation: # an authoring agent (meta/* only — §5.3.2) publishes a new # version of the candidate, the harness runs it pinned over the # same tasks and digests the grades; the loop composes the next @@ -68,8 +68,10 @@ steps: maxCriteria: "{{ params.digestMaxCriteria }}" # ── the hill-climb: author → run → grade, up to N generations ────────── + # The loop is the GENERIC eval/evolve-loop; harvey's fitness is criteria + # pass-rate (the digest's meanPassRate — binary all-pass has no gradient). - id: evolve - type: harvey/evolve-loop + type: eval/evolve-loop depends: basedigest config: tasks: "{{ input.tasks }}" @@ -78,8 +80,9 @@ steps: candidateName: "{{ params.candidateName }}" baseWorkflow: "{{ params.baseWorkflow }}" genWorkflow: "{{ params.genWorkflow }}" + fitnessName: pass-rate maxGenerations: "{{ input.generations || params.maxGenerations }}" - stopPassRate: "{{ params.stopPassRate }}" + stopFitness: "{{ params.stopPassRate }}" improveMargin: "{{ params.improveMargin }}" exploreAfter: "{{ params.exploreAfter }}" genParams: "{{ params.genParams }}" @@ -94,9 +97,9 @@ steps: candidate: "{{ evolve.candidate }}" bestGen: "{{ evolve.bestGen }}" bestVersion: "{{ evolve.bestVersion }}" - bestPassRate: "{{ evolve.bestPassRate }}" - baselinePassRate: "{{ evolve.baselinePassRate }}" - passRateDelta: "{{ evolve.bestPassRate - evolve.baselinePassRate }}" + bestPassRate: "{{ evolve.bestFitness }}" + baselinePassRate: "{{ evolve.baselineFitness }}" + passRateDelta: "{{ evolve.bestFitness - evolve.baselineFitness }}" improved: "{{ evolve.improved }}" generations: "{{ evolve.generations }}" totalKnownCost: "{{ evolve.totalKnownCost }}" diff --git a/vein/EVOLVE_SPEC.md b/vein/EVOLVE_SPEC.md index 048c83b2e..bef401ddd 100644 --- a/vein/EVOLVE_SPEC.md +++ b/vein/EVOLVE_SPEC.md @@ -443,12 +443,26 @@ possible version of "capture." stays human. Offline checks: `mcp/src/lab/harvey/evolve-smoke.ts`. 5. **Generalize `eval/optimize`'s candidate** from prompt string to workflow ref + version (§5.3.3) — the change that lets one loop drive all three - layers. **Harvey instance built** — `harvey/evolve-loop` (lab): up to N - generations of `harvey-evolve-gen` (author → run pinned candidate → - digest), each briefed with every prior attempt's version/pass-rate/ + layers. **Done, twice over** — the loop is now the GENERIC + `eval/evolve-loop` (lab, `eval/steps/evolve-loop.ts`): up to N + generations of a domain's gen workflow (author → run pinned candidate → + digest), each briefed with every prior attempt's version/fitness/ approach/failures, anchored to the best-so-far (never the latest), with the directive flipping from exploit to "try a GENUINELY DIFFERENT - approach" after `exploreAfter` non-improving attempts. Fitness is - criteria pass-rate (binary all-pass has no gradient); improvements must - clear a judge-noise margin (default 0.02 ≈ one criterion at n=50). The - GENERIC step remains open — this is the shape it should generalize. + approach" after `exploreAfter` non-improving attempts. A domain plugs in + its gen workflow plus a digest emitting `fitness` (fallback + `meanPassRate`) and a `fitnessName`. Two instances wired: + - **harvey-evolve** — fitness is criteria pass-rate (binary all-pass has + no gradient); improvements must clear a judge-noise margin (0.02 ≈ one + criterion at n=50). + - **gaia-evolve** — fitness is plain accuracy (binary per task is fine: + the task SET is the gradient, each flip moves it 1/n); + `improveMargin: 0` since exact-match scoring has no judge noise — the + residual produce-sampling noise is answered by §7's held-out + validation, not a margin. `gaia/digest-results` tags each miss + wrong-answer / empty-answer / produce-error (§8's taxonomy as a + required batch output, the cheap code-only version), and + `gaia-candidate-run` grades via `gaia/evaluate`'s `fromRun` unpack + (§5.3.5: a failed candidate run scores as "" — an honest zero, never + an aborted batch). Offline checks: + `mcp/src/lab/gaia/evolve-smoke.ts`.