feat(evolution): pluggable candidate-selection strategies - #64
Merged
Conversation
KE7
changed the title
KE7
force-pushed
the
feat/candidate-selection-strategies
branch
from
August 16, 2026 20:28
4fd6528 to
6460000
Compare
KE7
force-pushed
the
feat/candidate-selection-strategies
branch
from
August 23, 2026 21:57
5282c30 to
93f2d83
Compare
Add current_best, epsilon_greedy, and top_k_pareto candidate-selection strategies alongside the existing pareto default, matching upstream gepa.strategies.candidate_selector at parity. Config knob: evolution.candidate_selection_strategy (default "pareto", unchanged behavior) plus candidate_selection_epsilon/candidate_selection_top_k, declared and validated the same way proposal_top_k is (required for their own strategy, rejected otherwise). Also adds select_parents(), an additive batch-seam wrapper (loops the existing scalar draw) so a future joint-allocation strategy has a batch-shaped interface to land against without reopening this module. No changes to ParetoFrontier/population.py; the pareto strategy is untouched and the full existing suite passes unmodified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…parents Upstream GEPA's GEPAState.program_full_scores_val_set and per_program_tracked_scores are the identical expression — a comprehension over get_program_average_val_subset(i)[0], which returns sum(scores.values()) / num_samples. Both are means. select_top_k_pareto was mapping per_program_tracked_scores onto EvalResult.sum_score(), justified by a misreading of the val_evaluation_policy TODO as evidence the two arrays were expected to diverge. Sum and mean disagree whenever candidates have unequal numbers of scored instances, which HELIX permits, so this changed top-K membership, the filtered-front dominance sort key and the empty-filter fallback. All three now read aggregate_score(). current_best and epsilon_greedy already did. Also drops the unused select_parents batch seam. evolution.py builds one random.Random and shares it between candidate selection and the minibatch sampler, and PR #48's _plan_proposals draws each parent lazily inside its P iteration interleaved with per-slot minibatch draws — so batching all P parent draws consecutively would reorder the seeded stream. The helper could never have been substituted into the loop it was built for. Tests: the fallback test that encoded sum semantics is corrected and renamed, and a new unequal-cardinality test pins mean ranking. Both were verified to fail against the previous implementation. population.py is untouched; the default pareto path is unchanged. Two pre-existing issues found during validation are reported, not fixed: PYTHONHASHSEED-sensitive parent selection (reproduced across seeds) and ParetoFrontier.select_parent's own sum-vs-mean divergence from upstream. Verified against gepa-ai/gepa @ d26029fe, current main HEAD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… k>=pool
select_top_k_pareto's `if k >= len(frontier): return frontier.select_parent()`
shortcut fed sum_score() into dominance removal at k>=pool while every other
path in the function uses aggregate_score() (the mean) — a discontinuity in
a config knob in exactly the direction the prior commit's mean-vs-sum fix
was meant to eliminate. The top-k *membership* filter is indeed a no-op at
k>=pool, but that is not license to swap the score quantity fed to
_remove_dominated_programs, which is not a no-op whenever candidates carry
unequal numbers of scored instances.
Delete the shortcut; let the normal path run to completion so one mean
mapping feeds top-k ranking, dominance removal, and the empty-filter
fallback at every k, including k>=pool. This is deliberately NOT identical
to legacy `pareto` (population.py, untouched, 0-line diff), which keeps its
sum-based behaviour as previously scoped.
- Replace test_k_greater_equal_pool_matches_pareto (asserted the buggy
legacy-pareto-equivalence contract) with a test asserting the correct
mean-pareto contract, plus a dedicated regression test pinning the
reviewer's counterexample (a={i1:1.0}, b={i1:1.0,i2:0.9}, c={i2:1.0},
Random(1)) that the shipped shortcut selected "b" on and the mean path
provably cannot.
- Rename test_epsilon_one_never_consumes_extra_randomness_on_greedy_path
to test_epsilon_zero_..., matching what it actually exercises
(epsilon=0.0, not 1.0).
- Reword the module's Determinism docstring: a seeded run is NOT
reproducible end-to-end — string-id frontier sets are
PYTHONHASHSEED-sensitive in both `pareto` and `top_k_pareto`'s
dominance-removal path.
Both new/replaced tests were proven RED against the pre-fix code in a
detached scratch worktree before this fix was written.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_aggregate_score_or_floor returned EvalResult.aggregate_score() for a
result with empty instance_scores, and that method returns 0.0 for the
empty case (by design, for other callers like reporting/serialization).
Upstream's get_program_average_val_subset returns float("-inf") for a
program with no recorded subscores, so an unscored candidate was ranking
above any candidate with a negative finite mean instead of below every
scored candidate. Affected current_best, epsilon_greedy's greedy branch,
and top_k_pareto's ranking, dominance sort key, and fallback.
Floored at the selection site only (_aggregate_score_or_floor), leaving
EvalResult.aggregate_score itself and population.py untouched.
Second-order fix required by the floor: _first_argmax's strict-
improvement scan (`score > best_score`) left best_id unset once an
all-unscored pool ties every candidate at -inf (-inf > -inf is False).
Now takes the first candidate unconditionally
(`best_id is None or score > best_score`), matching upstream idxmax's
lst.index(max(lst)) first-index tie-break.
Also closes a coverage gap in TestCandidateSelectionConfig: adds one
parametrized strategy x epsilon-presence x top-k-presence matrix,
including both-irrelevant-knobs-set cases for pareto/current_best.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The candidate_selector module docstring had grown to 98 lines on a 270-line
module, most of it addressed to a reviewer rather than a reader: references to
unpublished repo conventions, a recital of which TODO comments upstream's
source carries, scope defences ("NOT addressed here", "deliberately does NOT"),
and derivations that a test already pins.
Keep the constraints, drop the notes:
- ranking is aggregate_score() (the mean), never sum_score()
- an unscored candidate floors to -inf and ranks below every scored one
- _first_argmax takes its first candidate unconditionally, because an
all-unscored pool ties at -inf and -inf > -inf is False
- no k >= len(frontier) shortcut to select_parent(), which ranks by sum
- pareto delegates to ParetoFrontier.select_parent, unchanged
GEPA attribution stays, restated as a plain pointer to gepa.strategies.
candidate_selector by symbol, matching eval_cache.py's existing phrasing —
no "naming credit only" hedge and no upstream file:line citations.
Comments and docstrings only. Verified prose-only by comparing the ast.unparse
of both revisions with docstrings stripped: candidate_selector.py is identical
in code, config.py differs only inside Field(description=...) literals, and the
single identifier change is one test renamed off "reviewer_counterexample".
Module docstring 99 -> 35 lines; the file 306 -> 212.
pytest 975 passed, ruff clean, mypy --strict clean — all unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A change on main converted ParetoFrontier.select_parent to rank by aggregate_score() (mean) instead of sum_score(), so the sum-vs-mean framing this branch's docstrings and tests used to describe the "pareto" strategy no longer matches the code. The "not equivalent to top_k_pareto" warning on select_top_k_pareto is still true, but its reason has changed: select_parent now also ranks by mean, so what distinguishes it is that it runs dominance removal over the whole active frontier with no top-k membership filter, where top_k_pareto restricts to the top-k candidates by score first. Verified by reading both functions and by directly comparing their internal eligible-candidate sets and frequency weights for representative pools. Two tests whose docstrings pinned a sum-vs-mean difference against select_parent no longer discriminate the two paths in the k >= len(frontier) regime they exercise (verified: identical eligible sets and frequency weights there now); their docstrings and one test name are rewritten to describe what they actually guard post-merge — a regression to sum-based ranking within select_top_k_pareto's own ranking step, not a difference from select_parent. No behavior change.
KE7
force-pushed
the
feat/candidate-selection-strategies
branch
from
August 26, 2026 11:12
30dd16a to
eba0ae0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
evolution.candidate_selection_strategywith the existingparetodefault pluscurrent_best,epsilon_greedy, andtop_k_pareto. The latter two require their strategy-specificcandidate_selection_epsilonorcandidate_selection_top_ksetting.Why
Different optimization runs need different, explicit parent-selection policies without changing the current default.
Review focus
Check
candidate_selector._aggregate_score_or_floor: new strategies rank by aggregate validation score and place unscored candidates below every scored candidate.Validation
uv run pytest tests/unit/test_candidate_selector.py tests/unit/test_config_new_fields.py -q— 92 passed in 0.20s. Diff: +920/−1 lines.