feat: auto-switch to rank-weighted selection based on archive state - #1412
Conversation
sample_parent() now automatically enables rank-weighted tournament selection when the archive reaches auto_rank_cell_threshold (default 8) occupied cells, indicating enough diversity that biasing toward stronger parents is beneficial. This mirrors the on_plateau() pattern of adapting strategy when the search state calls for it. On by default; opt out with auto_rank_weighted=False. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
d09c115 to
1671650
Compare
|
@ceo-review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Factory Review: KEEP
Verdict: KEEP
Reason: QA: CLEAN — 588 tests pass (1 pre-existing failure unrelated), lint clean, mypy clean. Auto-switch logic correct: activates at threshold, produces expected rank-proportional distribution, respects overrides. NOTE: PR description claims score_variance() and variance-based threshold that do not exist in code — description should be updated to match implementation. Stale test comment at test_population.py:226 references non-existent variance threshold.
QA Analysis
Adversarial QA Report — PR #1412
Feature: Auto-switch to rank-weighted selection based on archive size
Project type: Library (Python CLI with evolutionary search engine)
Date: 2026-08-30
Smoke Test
Command: uv run pytest tests/test_outer_loop/test_population.py -v
Result: PASS — 26/26 tests passed in 0.22s
Test Plan (derived from PR acceptance criteria)
- Existing test suite passes
- Full suite regression check
- Auto-switch activates with 10+ cells (rank bias observed)
- Auto-switch does NOT activate below threshold (uniform distribution)
- Explicit
rank_weighted=Trueworks independently of auto auto_rank_weighted=Falsedisables auto-switching- Engine.py call-site compatibility
- Lint and type-check pass
Feature Tests with Evidence
Test 1: Population test suite
Status: VERIFIED
Command: uv run pytest tests/test_outer_loop/test_population.py -v
Output: 26 passed in 0.22s — all tests pass including new TestAutoRankWeighted class (3 tests) and TestRankWeightedSelection class (2 tests).
Test 2: Full test suite regression check
Status: VERIFIED (with pre-existing failure)
Command: uv run pytest -x -v
Output: 588 passed, 1 failed in 9.50s
Note: The single failure is test_templatized_skill_validates[design-v2] — a pre-existing issue where the design-v2 skill body exceeds 600 lines (1123). This is NOT related to PR #1412 (no population/archive code involved). Confirmed by checking the failing test: it validates skill template line count, unrelated to tournament selection.
Test 3: Auto-switch activates with 10+ cells
Status: VERIFIED
Command: Custom Python script — MAPElitesArchive with 10 individuals, 2000 samples with tournament_size=1 (isolates weighting from tournament effect).
Output:
Archive size: 10
Best individual (ind-9) selected 357/2000 times = 17.8%
Uniform baseline would be 10.0%
All counts: [('ind-9', 357), ('ind-8', 331), ('ind-7', 281), ('ind-6', 276), ('ind-5', 234), ('ind-4', 163), ('ind-3', 151), ('ind-2', 111), ('ind-1', 76), ('ind-0', 20)]
Analysis: With rank weighting on 10 items, best (rank 10) should get ~18.2% (10/55). Observed 17.8% — matches expected distribution. Clear rank-proportional gradient from worst (1.0%) to best (17.8%).
Test 4: Auto-switch does NOT activate below threshold
Status: VERIFIED
Command: Custom Python script — MAPElitesArchive with 5 individuals (below default threshold of 8), 2000 samples with tournament_size=1.
Output:
Archive size: 5
Best individual (ind-4) selected 398/2000 times = 19.9%
Uniform baseline would be 20.0%
All counts: [('ind-2', 412), ('ind-3', 407), ('ind-0', 399), ('ind-4', 398), ('ind-1', 384)]
Analysis: All individuals between 19.2%-20.6% — textbook uniform distribution. No rank bias detected below threshold.
Test 5: Explicit rank_weighted=True works independently
Status: VERIFIED
Command: Custom Python script — 3 individuals (below threshold), rank_weighted=True explicitly, 3000 samples.
Output:
Archive size: 3 (below default threshold of 8)
Best (ind-2, score=2.0): 49.5%
Worst (ind-0, score=0.0): 16.7%
Analysis: With 3 items and weights [1,2,3], expected: best=50%, worst=16.7%. Observed: 49.5% and 16.7% — exact match. Explicit rank_weighted=True overrides the auto threshold check.
Test 6: auto_rank_weighted=False disables auto-switching
Status: VERIFIED
Command: Custom Python script — 10 individuals (above threshold), auto_rank_weighted=False, 2000 samples.
Output:
Archive size: 10 (above threshold, but auto disabled)
Best (ind-9): 9.8%
Uniform baseline: 10.0%
All counts: [('ind-1', 224), ('ind-4', 210), ..., ('ind-0', 175)]
Analysis: All individuals between 8.8%-11.2% — uniform distribution. Auto-switching correctly disabled despite being above threshold.
Test 7: Engine.py integration compatibility
Status: VERIFIED
Command: Custom Python script — inspected signature + called with engine.py's pattern: archive.sample_parent(3, rank_weighted=False).
Output:
sample_parent signature: (self, tournament_size: 'int' = 3, rank_weighted: 'bool' = False, auto_rank_weighted: 'bool' = True, auto_rank_cell_threshold: 'int' = 8)
Engine-style call (rank_weighted=False): got e-9
Engine-style call (rank_weighted=True): got e-8
auto_rank_weighted default: True
auto_rank_cell_threshold default: 8
Analysis: Engine.py (line 284) calls sample_parent(tournament_size, rank_weighted=config.rank_weighted_selection) — positional + keyword. The new params have defaults, so no breakage. The engine automatically benefits from auto-switching when rank_weighted=False and archive has 8+ cells.
Test 8: Lint and type-check
Status: VERIFIED
Command: uv run ruff check factory/outer_loop/population.py factory/outer_loop/engine.py
Output: All checks passed!
Command: uv run mypy factory/outer_loop/population.py factory/outer_loop/engine.py
Output: Success: no issues found in 2 source files
Edge Cases Probed
| Edge case | Result |
|---|---|
| Empty archive + auto_rank_weighted=True | Returns None (existing behavior preserved) |
| 1 individual + auto_rank_weighted=True | Falls through to random.sample (len < 2 guard on line 159) |
| Exactly 8 cells (threshold boundary) | Auto activates (>= check on line 156) |
| rank_weighted=True + auto_rank_weighted=True | rank_weighted takes priority (line 155: use_rank = rank_weighted) |
Acceptance Criteria Verification
| Criterion | Status |
|---|---|
sample_parent() auto-activates rank-weighted when archive >= 8 cells |
VERIFIED |
auto_rank_weighted defaults to True |
VERIFIED |
auto_rank_cell_threshold defaults to 8 |
VERIFIED |
Explicit rank_weighted=True still works |
VERIFIED |
auto_rank_weighted=False disables the feature |
VERIFIED |
| No regressions in engine.py integration | VERIFIED |
| Lint and type-check pass | VERIFIED |
| Existing tests pass | VERIFIED |
Adversarial Verdict: PASS
All 8 acceptance criteria verified with evidence. The auto-switch logic is correct: it activates at the right threshold, produces the expected rank-proportional distribution, respects both explicit overrides and the disable flag, and maintains backward compatibility with engine.py's existing call pattern. The single test failure in the full suite (design-v2 template size) is pre-existing and unrelated.
Posted by Factory CEO
Benchmark Resultsdevopsgym
Full JSON{
"benchmark": "devopsgym",
"instance_id": "build-maven-dependency-resolution",
"solver": "claude-code",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 6,
"status": "failed",
"timestamp": "20260830T211530Z",
"details": {
"solver": "claude-code",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": ""
}
}terminalbench
Full JSON{
"benchmark": "terminalbench",
"instance_id": "fix-git",
"solver": "claude-code",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 126,
"status": "success",
"timestamp": "20260830T211530Z",
"details": {
"solver": "claude-code",
"cost_usd": 0.62895675,
"input_tokens": 197447,
"output_tokens": 1346,
"cache_read_tokens": 111081,
"cache_creation_tokens": 0,
"trace_id": ""
}
}swebench
Full JSON{
"benchmark": "swebench",
"instance_id": "sympy__sympy-20590",
"solver": "claude-code",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 204,
"status": "success",
"timestamp": "20260830T211531Z",
"details": {
"solver": "claude-code",
"cost_usd": 0.603611,
"input_tokens": 267296,
"output_tokens": 3968,
"cache_read_tokens": 202807,
"cache_creation_tokens": 0,
"trace_id": ""
}
}harborindex
Full JSON{
"benchmark": "harborindex",
"instance_id": "bix-filter-chip-variants",
"solver": "claude-code",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 7,
"status": "failed",
"timestamp": "20260830T211532Z",
"details": {
"solver": "claude-code",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": ""
}
}featurebench
Full JSON{
"benchmark": "featurebench",
"instance_id": "pypa__packaging.013f3b03.test_metadata.e00b5801.lv1",
"solver": "claude-code",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 656,
"status": "success",
"timestamp": "20260830T211533Z",
"details": {
"pass_rate": 1,
"solver": "claude-code",
"cost_usd": 1.18995675,
"input_tokens": 1031127,
"output_tokens": 12113,
"cache_read_tokens": 966501,
"cache_creation_tokens": 0,
"trace_id": ""
}
}legacybench
Full JSON{
"benchmark": "legacybench",
"instance_id": "1907c2-c-debug-legacy-buddy-fix",
"solver": "claude-code",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 247,
"status": "success",
"timestamp": "20260830T211533Z",
"details": {
"solver": "claude-code",
"cost_usd": 0.8522289999999999,
"input_tokens": 482698,
"output_tokens": 8697,
"cache_read_tokens": 414228,
"cache_creation_tokens": 0,
"trace_id": ""
}
}featurebench
Full JSON{
"benchmark": "featurebench",
"instance_id": "pypa__packaging.013f3b03.test_metadata.e00b5801.lv1",
"solver": "factory",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 593,
"status": "success",
"timestamp": "20260830T211534Z",
"details": {
"pass_rate": 1,
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "d4482eea7e6cee0d978fca70609e2a1a"
}
}programbench
Full JSON{
"benchmark": "programbench",
"instance_id": "abishekvashok__cmatrix.5c082c6",
"solver": "claude-code",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 423,
"status": "success",
"timestamp": "20260830T211535Z",
"details": {
"solver": "claude-code",
"cost_usd": 1.1269920000000002,
"input_tokens": 470362,
"output_tokens": 11204,
"cache_read_tokens": 406189,
"cache_creation_tokens": 0,
"trace_id": ""
}
}terminalbench
Full JSON{
"benchmark": "terminalbench",
"instance_id": "fix-git",
"solver": "factory",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 100,
"status": "success",
"timestamp": "20260830T211535Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "d29cd92743d7ab5eaba4841b5acfbc5f"
}
}programbench
Full JSON{
"benchmark": "programbench",
"instance_id": "abishekvashok__cmatrix.5c082c6",
"solver": "factory",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 1184,
"status": "success",
"timestamp": "20260830T211536Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "a8158997ab14b91dd6e01ceec8d57c24"
}
}tomswe
Full JSON{
"benchmark": "tomswe",
"instance_id": "sympy__sympy-20590",
"solver": "claude-code",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 259,
"status": "success",
"timestamp": "20260830T211536Z",
"details": {
"solver": "claude-code",
"cost_usd": 0.432367,
"input_tokens": 255998,
"output_tokens": 3632,
"cache_read_tokens": 218834,
"cache_creation_tokens": 0,
"trace_id": ""
}
}devopsgym
Full JSON{
"benchmark": "devopsgym",
"instance_id": "build-maven-dependency-resolution",
"solver": "factory",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 6,
"status": "failed",
"timestamp": "20260830T211538Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": ""
}
}legacybench
Full JSON{
"benchmark": "legacybench",
"instance_id": "1907c2-c-debug-legacy-buddy-fix",
"solver": "factory",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 1850,
"status": "success",
"timestamp": "20260830T211538Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "0f312f16d8b6e4d68ed48e3c6e0d99a1"
}
}swebench
Full JSON{
"benchmark": "swebench",
"instance_id": "sympy__sympy-20590",
"solver": "factory",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 177,
"status": "success",
"timestamp": "20260830T211538Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "ff62f94f421bb87fc491c5b61c4c5033"
}
}tomswe
Full JSON{
"benchmark": "tomswe",
"instance_id": "sympy__sympy-20590",
"solver": "factory",
"passed": 1,
"total": 1,
"score": 1,
"resolved": true,
"duration_seconds": 171,
"status": "success",
"timestamp": "20260830T211540Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": "62213370f0253e80115ae9dc31ebc973"
}
}harborindex
Full JSON{
"benchmark": "harborindex",
"instance_id": "bix-filter-chip-variants",
"solver": "factory",
"passed": 0,
"total": 1,
"score": 0,
"resolved": false,
"duration_seconds": 8,
"status": "failed",
"timestamp": "20260830T211547Z",
"details": {
"solver": "factory",
"cost_usd": 0,
"input_tokens": 0,
"output_tokens": 0,
"cache_read_tokens": 0,
"cache_creation_tokens": 0,
"trace_id": ""
}
}devopsgym-claude-code: The benchmark never ran: Harbor couldn't resolve the dataset Detailed analysisFailure Analysis: devopsgym / build-maven-dependency-resolutionSolver: claude-code DiagnosisWhat went wrongThis isn't a bug in your code or in the agent's work on the task — the run never got as far as running the agent. It died during Harbor's job setup, while resolving the dataset. The actual errorThe bottom line is the whole story: The chain that produced itReading the traceback top-to-bottom, Harbor's CLI tried to start a job and resolve its dataset:
The Root causeThe
How to fix / move forward
Note also the earlier docstring text visible in the frames ( Want me to grep the benchmark harness config in this repo to find where the harborindex-claude-code: The task ID filter Detailed analysisFailure Analysis: harborindex / bix-filter-chip-variantsSolver: claude-code DiagnosisWhat went wrongThis isn't a failure of the agent or the code being benchmarked — the job never started. It aborted during Harbor's task-resolution phase, before any work was dispatched. The chainReading the traceback bottom-up:
The root causeThe task-ID filter didn't match because of a naming/namespace mismatch. The instance was requested as the bare name Two possibilities, both of which produce this exact error:
The secondary symptom (why it looks so ugly)The What to do
Bottom line: a task-selection/config error, not a code or agent failure. The benchmarked change was never even exercised. programbench-claude-code: The trace shows only harness setup and artifact collection—no test results or error output—so there's no evidence the agent ran or why it failed. Under 140 chars: "Trace only shows harness setup and trajectory collection; no test output or error, so the actual failure cause isn't captured." (137 chars) Detailed analysisFailure Analysis: programbench / abishekvashok__cmatrix.5c082c6Solver: claude-code DiagnosisWhat happenedThis run did not fail in the harness — it failed the task. The agent produced a reverse-engineered The task
The apparent contradiction
Both are correct, because they measure different things. Why Harbor said 1.0The Harbor verifier (
If all three succeed it writes Why the recorded result is 0.0For ProgramBench, So: the reverse-engineered implementation compiled and packaged (Harbor gate = pass) but did not match the original binary's behavior on the hidden tests (real eval = fail). Corroborating signals
About the trace you were givenThe Bottom lineNothing broke in the pipeline. The agent solved the packaging gate but not the behavioral task — its reconstructed One incidental note: there's a programbench-factory: Despite agents self-reporting all 16 discoveries "verified," the reverse-engineered cmatrix source still diverged from the binary on untested behavior, failing hidden tests. (139 chars) Detailed analysisFailure Analysis: programbench / abishekvashok__cmatrix.5c082c6Solver: factory DiagnosisWhat went wrongBottom line: The run "succeeded" as a process ( The taskReverse‑engineer an execute‑only The timeline (from the trace)
Then: The root causeThe validation loop is self‑referential. The adversarial reviewer only checks the builder's own recorded discoveries, re‑running the same class of probes (flags, error strings, exit codes). Nothing in the loop generates behavior neither agent thought to test. So "all 16 verified" means "the 16 things we looked at agree" — it says nothing about coverage. For Two secondary issues worth flagging
The lessonThe workflow's weakness on this benchmark is that its "adversarial" review validates claims already made rather than hunting for untested behavior. For a rich interactive program, differential testing is only as good as the diversity of inputs generated — and here the agents never expanded the probe space to the animation/rendering surface that the hidden tests grade. "All discoveries verified" was a coverage illusion, and the benchmark graded the gap. If you want, I can dig into whether the adversarial reviewer prompt ( devopsgym-factory: The dataset Detailed analysisFailure Analysis: devopsgym / build-maven-dependency-resolutionSolver: factory DiagnosisWhat went wrongThe run never started. It failed during Harbor's job-setup phase — before any agent, container, or the actual The failure, tracedThe bottom line is the whole story: Reading the stack top-down:
The Root causeThe Harbor package registry has no version published under the tag
This is a benchmark infrastructure / dataset-availability problem in Harbor's registry, not a code defect in this repo and not an agent failure. How to fix / move forward
One side note unrelated to the failure: a Want me to grep the benchmark configs in this repo to find where the harborindex-factory: Task filter Detailed analysisFailure Analysis: harborindex / bix-filter-chip-variantsSolver: factory DiagnosisWhat went wrongThis run never started a single task — it failed during job setup, before any agent or environment was provisioned. The root cause is a task-filter mismatch, not a bug in the factory or the agent. The failure chainReading the traceback bottom-up:
The scary-looking asyncio frames ( Why the filter didn't matchThe instance was requested as: But every task in this dataset is namespaced under So there are two things off:
Bottom lineThis is a benchmark harness configuration error, not a factory/agent failure. The run was invoked with a task instance name ( To fix it:
Also unrelated but worth noting: the run wrote Would you like me to grep the factory's benchmark-runner code to see where Overall: 62.5% accuracy (= +0.0% vs main) | $0.81 avg cost | 376s avg duration Comparison vs Main
Baseline: latest main branch run per benchmark+solver. ▲ = improvement, ▼ = regression. How these benchmarks runFactory solver: Runs Claude Code solver: Runs TerminalBench: Uses Harbor framework. Factory runs via custom ProgramBench: Both solvers run inside a Docker cleanroom container. See Config: |
1 participant
Summary
Builds on #1394 (rank-weighted tournament selection).
sample_parent()now auto-activates rank-weighted selection when the archive reachesauto_rank_cell_threshold(default 8) occupied cells, indicating enough diversity that biasing toward stronger parents is beneficial.This mirrors the
on_plateau()pattern — adapting selection pressure based on search state rather than requiring static configuration. Auto-switch is on by default; callers can opt out withauto_rank_weighted=False.Cell count is the sole trigger — score variance was considered but dropped because a fixed variance threshold is application-dependent (chess scores range -500 to +500, factory's default scores are 0-1 floats). Cell count is application-agnostic: enough diverse individuals in the archive → rank-weighted helps regardless of score scale.
Test plan
test_auto_activates_by_cell_count— triggers at 10 cells with threshold 8test_auto_stays_uniform_below_thresholds— 2 cells → stays uniformtest_auto_disabled—auto_rank_weighted=Falsekeeps uniformruff check+mypyclean🤖 Generated with Claude Code