Skip to content

Match flat Diagnostics scores label to the per-example path - #68

Merged
KE7 merged 1 commit into
mainfrom
fix/flat-diagnostics-scores-header
Aug 26, 2026
Merged

Match flat Diagnostics scores label to the per-example path#68
KE7 merged 1 commit into
mainfrom
fix/flat-diagnostics-scores-header

Conversation

@KE7

@KE7 KE7 commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary

The mutation-prompt Diagnostics section has two renderers. The per-example
renderer relabels the reserved scores key to Scores (Higher is Better),
a header carried over from the upstream optimizer this repo derives its
Diagnostics format from. The flat/aggregate renderer — used when an
evaluator emits one side_info for the whole batch instead of per-example
records — rendered the same key as a raw scores: {...} line, so the same
data got two different presentations and the flat path silently dropped
the "higher is better" cue.

This PR makes the flat path relabel scores the same way, in its existing
label: value line shape (no markdown header, since that isn't this
path's style):

# before
## Diagnostics
  scores: {'acc': 0.5}

# after
## Diagnostics
  Scores (Higher is Better): {'acc': 0.5}

Per-example rendering is unchanged:

## Diagnostics

### Example ex_0
#### Scores (Higher is Better)
##### acc
0.5

This is a presentation-only change. side_info["scores"] is a data
contract read by the parser, executor, population, and the
objective/cartesian frontier modes — none of those, nor any evaluator, nor
the wire format, were touched. Only how the flat renderer displays the key
to the model changed.

Also fixed

_render_per_example_diagnostics's docstring cited the upstream GEPA
source it mirrors by file and line number
(src/gepa/strategies/instruction_proposal.py:54-95). Upstream line
numbers rot independently of this repo, so it's re-cited by file + symbol
instead: the format_samples closure inside
InstructionProposalSignature.prompt_renderer in
src/gepa/strategies/instruction_proposal.py. Verified against a fresh
clone of gepa-ai/gepaformat_samples (and the render_value closure
inside it) live exactly where the docstring says, nested in that method.
The attribution itself is kept, only the citation shape changed.

Test plan

  • Extended tests/unit/test_mutator.py::TestPerExampleDiagnostics
    with a test that renders both Diagnostics paths on the same
    scores key and asserts they carry the same label.
  • uv run python -m pytest — 923 passed
  • uv run ruff check src/ tests/ — all checks passed
  • uv run mypy --strict src/helix/ — no issues in 26 source files

The per-example Diagnostics renderer relabels the reserved `scores`
key to "Scores (Higher is Better)", carrying the higher-is-better cue
forward from the upstream GEPA optimizer. The flat/aggregate path (used
when an evaluator emits one side_info for the whole batch instead of
per-example records) rendered the same key as a raw `scores: {...}`
line, silently dropping that cue.

The flat path now relabels `scores` the same way, keeping its existing
`  label: value` line shape rather than growing a markdown header.
This is presentation only: side_info["scores"] is unchanged as the
data-contract key read by the parser, executor, population, and the
objective/cartesian frontier modes.

Also re-cite the ported GEPA closure by file + symbol instead of file +
line number, since upstream line numbers rot independently of this repo.
@KE7
KE7 force-pushed the fix/flat-diagnostics-scores-header branch from 87ea484 to 8a6da0b Compare August 26, 2026 10:57
@KE7
KE7 merged commit 313f4bb into main Aug 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant