evolve: phase-1 measurement infra — foreach concurrency, partial run summaries, eval/matrix - #1624
Open
Evanfeenstra wants to merge 1 commit into
Open
evolve: phase-1 measurement infra — foreach concurrency, partial run summaries, eval/matrix#1624Evanfeenstra wants to merge 1 commit into
Evanfeenstra wants to merge 1 commit into
Conversation
…summaries, eval/matrix Phase 1 of plans/evolve-scoreboard-and-task-matrix.md (#1623), scoped to avoid #1621's files. Three independent pieces: vein foreach `concurrency` (default 1): bounded worker pool over iterations — results stay in input order, `#i` event paths and journal replay are unchanged, the cooperative checkpoint moves to just before each iteration starts (pause parks new starts, in-flight drain), a failure stops new starts and surfaces the lowest-index error, and cancellation always wins over an ordinary error. The prod evolve run spent 17h strictly serial on 225 produce runs; repeats are unaffordable without this. vein partial run summaries: `GET /workflows/:name/runs/:runId` now serves a summary reconstructed from the event log (`partial: true`, status from the live controller or "stale") instead of a 404 when `run.json` is missing — in-flight and crash-orphaned runs alike. The event log is already durable per-step; refusing to read it is what turned a dead 17-hour run into event-log archaeology. Reconstruction is a pure `summarizeFromEvents` (exported) over run.start / top-level step.end / step.error. No write-path changes: run.json stays terminal-only, so "summary exists = finalized" consumers are untouched. eval/matrix (seeded lab step): the task×version matrix across measurements — per-task bands (floor / movable / ceiling), an EMPIRICAL noise floor from same-version re-measurements (fitness deltas + task flips on identical YAML; UNKNOWN rather than 0 when no version has n≥2), and bias-vs-variance tags on never-correct tasks (byte-identical wrong answer ×≥3 = bias, distinct answers = variance). Verdict channel only — gold never enters (EVOLVE_SPEC §6). Offline smoke: `npx tsx src/lab/eval/matrix-smoke.ts`, fixtures modeled on prod run 1788061734710 ("36"×9 bias, the v11 0.76/0.80 resample pair). vein: 544/544 tests pass (6 new foreach-concurrency, 4 summarize, 1 endpoint). gaia evolve-smoke + matrix-smoke pass; both packages typecheck. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of the version-scoreboard restructure plan (#1623), deliberately scoped to files #1621 does not touch. Three independent pieces, all motivated by the post-mortem of prod run
gaia-evolve/1788061734710:foreachgetsconcurrency(default 1) — bounded worker pool; results in input order,#ipaths + journal replay unchanged, checkpoint before each start (pause parks new starts), lowest-index error wins, cancellation beats error. The prod run spent 17h strictly serial on 225 produce runs; repeated measurement is unaffordable without this.GET /runs/:runIdserves a partial summary instead of a 404 whenrun.jsonis missing — reconstructed from the durable event log via a new exportedsummarizeFromEvents(partial: truediscriminator, status from live controller orstale). No write-path change:run.jsonstays terminal-only. A dead 17-hour run no longer costs its whole report (and this works retroactively for every existing stale run).eval/matrixseeded lab step — the task×version matrix across measurements: floor/movable/ceiling bands, an empirical noise floor from same-version re-measurements (reported UNKNOWN, never 0, when no version has n≥2), and bias-vs-variance tags on never-correct tasks (byte-identical wrong answer ×≥3 = bias). Verdict channel only; gold never enters. Smoke fixtures model the prod pathologies ("36"×9, the v11 0.76/0.80 resample pair).Tests: vein 544/544 (11 new);
gaia/evolve-smoke+ neweval/matrix-smokepass; both packages typecheck.Follow-up (blocked on #1621 merging, then rebased on it): wire
baselineSamples+ the matrix intogaia-evolve, and the Phase-2 scoreboard loop.🤖 Generated with Claude Code