feat(ledger): multiple live heads — fork, head_for_worker, branch-aware commit

[HugoXK]

feat(ledger): multiple live heads — fork, head_for_worker, branch-aware commit

The problem

The ledger held one dev branch. PopulationAggregator serialised candidates by rewriting it — select one, commit it to dev, all workers start from dev. No two workers could expand different candidates at once. selection.py's docstring said "Making the ledger hold concurrent branches, and redefining eta when head is plural, is a separate change."

What this PR adds

Ledger.fork(name, from_branch=DEV) — creates a head/<slot> branch with its own version vector. A commit against it does not touch dev. live_heads() lists dev plus all forks. discard_head cleans up. Fork names must start with head/ (git cannot have both dev as a ref file and dev/0 as a ref directory, so the prefix avoids the collision).

Candidate.branch + EvidenceCard.branch — carry the fork name through the selection and merge pipeline so the aggregator knows where to commit back.

Aggregator.head_for_worker(worker_id) — returns dev by default (every single-head caller). PopulationAggregator overrides it to cycle through head/<slot> forks, distributing workers across candidates.

PopulationAggregator.step(n_workers) — forks a head/<slot> per selected candidate. finalize discards all forks and commits the best candidate to dev.

_Candidate.branch + _branch_of(cards) — the merge pipeline carries the branch from evidence cards to the candidate, and _commit_with_retry commits to it.

Worker loops (sync evolution.py + async async_evolve.py) — _snapshot_for returns (artifact, base_v, branch). Each worker snapshots from head_for_worker(wid) instead of hardcoded Ledger.DEV. EvidenceCard carries branch=worker_branch.

Backward compatibility

Component	Default	Old behaviour?
head_for_worker	dev	yes
Candidate.branch	None	yes
EvidenceCard.branch	None	yes
PopulationAggregator._forks	[]	yes — no forks, head_for_worker returns dev
step(n_workers=1)	n_workers=1	yes — single head, old behaviour
_commit_with_retry(branch=DEV)	Ledger.DEV	yes
AggregatorProtocol.step	(n_workers=1)	yes — default param

Verification

Item	Result
Ledger fork/live_heads/discard	6 tests
Candidate.branch + EvidenceCard.branch	2 tests
head_for_worker default + cycle	2 tests
PopulationAggregator commit to fork	1 test
Default evolve unchanged	1 test
Core regression	178 passed, 0 failed

Files

10 files changed, +380, -23

Modified:

• agentdescent/ledger.py — fork, live_heads, discard_head, HEAD_PREFIX
• agentdescent/aggregator.py — head_for_worker, _branch_of, _Candidate.branch, _commit_with_retry(branch=), AggregatorProtocol.step(n_workers=1), _drain_audit_queue + _verifier_can_spend + audit_drain_per_step
• agentdescent/evolution.py — _snapshot_for returns 3-tuple, worker loop uses branch, EvolutionResult.audit_drained
• agentdescent/async_evolve.py — worker_branch, snapshot from it, EvidenceCard.branch
• agentdescent/population.py — step(n_workers) forks, head_for_worker cycles, finalize discards, _commit_state(branch=)
• agentdescent/selection.py — Candidate.branch
• agentdescent/evolvable.py — EvidenceCard.branch
• docs/api.md — regenerated
• CHANGELOG.md — entry
• tests/test_multi_head.py — 12 tests

[Birfy]

Request changes before merge. The new multi-head tests pass, but the existing suite fails across Python 3.9/3.11/3.12; I reproduced 5 failures in the focused async/pipelined tests.

1. Both drivers now unconditionally call aggregator.step(n_workers) and aggregator.head_for_worker(...). Existing custom aggregators implement the published zero-argument step() and need not implement head_for_worker; CI shows TypeError for PromptBreederPopulation and other implementations, and an async legacy aggregator's worker threads die with AttributeError. Keep these hooks optional for existing aggregators, with compatibility tests.
2. async_evolve._worker chooses worker_branch once at startup, before PopulationAggregator.step() creates forks. It never updates the branch after selection, so async workers continue to snapshot and submit against dev; the advertised parallel heads are never explored. Choose the branch when refreshing each worker snapshot and add an end-to-end async multi-head test.
3. The single-head EvidenceBuffer.ready() and _in_flight values changed from artifact ids to tuples, breaking existing async semantics and pipelined-gate tests. Preserve the existing single-head contract or update its consumers/tests deliberately. The currently failing CI must be green before merge.

[Birfy]

The older custom-aggregator compatibility and async refresh concerns appear addressed. A remaining blocker is that _Candidate.branch is recorded during preparation, but the commit call in _decide() still omits it. _commit_with_retry therefore uses its default dev branch for candidates prepared against head/<slot>, so fork work can overwrite dev and leave the fork unchanged. Pass the candidate branch through and add a test proving a fork candidate updates only its own branch.

[HugoXK]

Addressed the review: a fork candidate now commits to its own branch.

The problem: _Candidate.branch was recorded during preparation, but the commit call in _decide() defaulted to dev — so a candidate prepared against head/<slot> was committed to dev, overwriting the confirmed branch and leaving the fork unchanged.

The fix: _decide passes the candidate's branch through to _commit_with_retry (c.branch or Ledger.DEV), and the CAS-rebase path rebases against that same branch — the version vectors differ per fork, so rebasing onto dev would be a different world entirely.

Test: test_decide_commits_a_fork_candidate_to_its_own_branch proves a head/0 candidate lands on head/0 (version 2) and leaves dev at version 1. I also verified the failure mode: without the fix, the fork candidate writes to dev (reproduced directly).

Rebased onto current upstream main (PR #194 merged). 26 multi-head/ledger tests pass; 118 across the async/policy-contract/candidate suites pass.

[Birfy]

Scoped re-review of b766875: the outstanding _decide branch-routing finding is addressed. The candidate now forwards its own branch to _commit_with_retry, and CAS retry snapshots the same branch. Independently ran 161 focused multi-head/ledger/async semantics/parity/policy-contract/pipelined/candidate/aggregator tests: all passed.

Also exercised the retry beyond the new direct-commit test: start candidate on head/0 v1, concurrently advance head/0 with a fork-only update and dev with a different dev-only update, then submit the stale fork candidate. Retry commits head/0 v3 preserving its winning update plus our diff; dev's state/version stay unchanged. This verifies retry does not rebase against dev.

This is confirmation of that fix, not whole-PR approval yet. Exact-head tests run 38023411855 is still red on all Python 3.9/3.10/3.11/3.12 jobs. I inspected each job's failure summary: the failures are test_every_internal_anchor_exists and test_install_guide_names_the_current_number_of_dry_run_ports (broken install-page anchors and stale 'All 20 algorithm ports print' text). Docs build run 38023411845 is green. Please integrate the shared documentation corrections and obtain green CI on the resulting exact head before final approval.

[Birfy]

Requested CI rerun completed on unchanged current head b766875: attempt 2 is red. All four Python jobs repeat the two already identified install-page docs failures. There is one additional result worth separating from those: Python 3.9 job failed test_a_worker_does_not_read_the_ledger_once_per_rollout with 35 reads, 6 sweeps, 21 rollouts, versus budget 34. The other three versions passed that assertion.

Bounded diagnosis:

• The test is unchanged from base 31b2e0c. Its 3 * sweeps + 16 accounting assumes three per-sweep reads.
• Caller tracing on both base and PR head shows four routine reads per measured sweep: _merge_batch before and after, _prepare, and _stable_distance. The stable-distance snapshot already exists on the base, so this is not a new multi-head-only read. Workers made only 4–6 snapshots across 21 rollouts in these probes, rather than one per rollout.
• The exact test passes locally on Python 3.9.25. Fifteen controlled no-delay/1ms/5ms worker-snapshot-delay probes per revision on 3.9 also pass; ten earlier plain repetitions on 3.12.14 passed. I did not reproduce the exact 35/34 CI schedule, so I am not classifying that individual failure as harmless.
• The accounting weakness is independently observable on the base with supported smaller async_ratio values: at async_ratio=0, one run produced 91 reads over 19 sweeps (budget 73); at async_ratio=1, 48 reads over 9 sweeps (budget 43). These are diagnostic workload variations, not reproductions of the default CI invocation.

Suggested next step: make this regression assert worker-originated reads under a controlled snapshot/refresh schedule, with a negative control that injects a per-rollout read and fails. Account separately for known merger/promotion work. Merely increasing the slack constant or rerunning until green would not establish the intended property. No code/test bounds changed and no further hosted retries started.