evolve: guard the author summary channel against filler echoes - #1620
Merged
Conversation
Observed live (gaia, prod): a schema-mode author ended on a bare text turn and echoed summary '' / 'placeholder' into two generations. The version fallback kept grading honest, but the junk summaries poisoned the briefing channel — the EXPLORE directive's 'pick an approach that is none of the above' is only checkable against real summaries — and left the human report unreadable. - eval/evolve-loop: usableSummary() rejects empty/short/filler echoes (placeholder, todo, n/a, …) and substitutes an honest marker pointing at the version YAML diff, in both the live and journal-replay paths. - gaia/harvey evolve-gen authorSystem: explicit final-message discipline — every schema field filled with real values, never filler, even when the step budget runs out. - gaia evolve-smoke: gen-0 fixture now echoes 'placeholder' and asserts the marker (not the filler) reaches the next briefing and the report. tsc clean; gaia + harvey evolve smokes pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On a live prod gaia-evolve run, the generation authors' structured output came back with
summary: ""(gen 0) andsummary: "placeholder"(gen 1) — the known schema-mode failure where one no-tool-call text turn ends the agent loop and becomes the structured output. The existing version fallback kept grading honest (real versions were graded, the climb was genuine), but the summary channel is load-bearing in two places it silently poisoned:generations[].summaryis the primary "what changed" recordFix
eval/evolve-loop(generic — harvey + gaia):usableSummary()rejects empty/short/filler echoes and substitutes an honest marker ((no usable approach summary reported … read this version's YAML diff)) in both the live and journal-replay paths. AI SDKOutput.objectdoesn't enforce schema constraints likeminLengthlocally, so the guard lives harness-side where it has teeth.authorSystem): explicit final-message discipline — the structured answer is harvested from the final turn, every field real, never filler, even when the budget runs out.placeholderand asserts the marker (never the filler) reaches the next briefing and the report.Testing
tsc --noEmitclean ·npx tsx src/lab/gaia/evolve-smoke.tsandsrc/lab/harvey/evolve-smoke.tspass.🤖 Generated with Claude Code