feat(multi-artifact): evolve multiple artifacts via extra_artifacts worker loop, PP stages, atomic contract changes

[HugoXK]

What this does

evolve() and async_evolve() can now evolve any number of artifacts, not just the primary. This is the first half of the design's L-task cross-product (task-cluster × artifact): the second axis finally has somewhere to live.

What changed

extra_artifacts= — register more artifacts

Both engines take extra_artifacts: {aid: EvolvingArtifact} beside artifact_id. Each extra carries its own blast_radius, strategy and Contract. The worker loop rolls each out and proposes against it, so a proposal can name any registered artifact as its diff target.

Worker loop proposes against every artifact

• Sync (_artifact_targets / _propose_one in evolution.py): DP+extras rolls out primary and each extra; PP proposes against the stage's artifact only.
• Async (async_evolve.py): same cross-product on the barrier-free path (no parallel strategy, so it is primary + every extra).

PipelineParallel is no longer refused

_reject_pipeline_parallel lifts once extra_artifacts are registered, and each PP stage must name a registered artifact (unknown stages raise, not silently drop). evolve(parallel=PipelineParallel(...)) without extras still raises as before.

Contract dependency graph + atomic adaptation

• Contract gains depends_on (artifact ids whose contracts it relies on).
• A contract-breaking diff lands through Ledger.commit_atomic — the only path the ledger sanctions for a deliberate contract change, previously unreachable from any engine path (commit_atomic was "provided and tested but not used").
• Dependents' cached evaluations are evicted (MemoryCache/FileCache.invalidate(rendered)) so they re-measure under the superseded-then-replaced contract.

EvolveSpec supports extra_artifacts

compose() merges the field into extra_artifacts= for evolve().

Test & docs

• tests/test_multi_artifact.py — 10 tests: registration, cross-artifact diffs, per-artifact strategies, PP allowed/refused/unknown-stage, atomic contract commit, dependent cache eviction, commit_atomic all-or-nothing.
• Full suite passes (two pre-existing numba environment failures in test_era_algotune/test_porous_molecules fail on main too).
• docs/api.md regenerated (tools.gen_api_docs); concepts, aggregator, data-model, modules updated; LedgerProtocol gains commit_atomic; CHANGELOG updated.

Follow-up (Phase 2)

• TaskCluster artifact dimension — the triggering side of L-task (which artifact a task's failure implicates).
• adapters= in _commit_with_retry — worker-produced cross-artifact adapter states for the full atomic adaptation transaction.

[Birfy]

The multi-artifact path is otherwise covered by green CI, but the contract-breaking commit does not fulfill the atomic adaptation contract introduced here. _commit_with_retry() accepts adapters, yet its _decide() call site does not pass dependent artifact states; the transaction therefore commits only [candidate]. A breaking contract can land while Contract.depends_on artifacts remain unchanged. Please supply and validate the adapted dependents in the same commit_atomic() call, or narrow the API/docs and add an explicit rejection when required adapters are missing.

[HugoXK]

Addressed the review: the atomic adaptation transaction is now enforced, not assumed.

The problem: _commit_with_retry() accepted adapters, but _decide() never supplied dependent states, so a contract-breaking commit silently landed as just [candidate] — leaving artifacts whose Contract.depends_on names it registered against a superseded contract.

The fix (two parts):

1. Explicit rejection when required adapters are missing. _decide() now refuses a contract-breaking diff on an artifact with declared dependents when no adapted states are provided — a new missing-adapters MergeOutcome, before any commit. The breaking change cannot land while dependents remain unchanged; the worker that wants the break must also produce the adapters.

2. The atomic commit actually covers every state. _commit_with_retry() now builds the base version vector from the merge's full version vector, naming the candidate and each adapter — so commit_atomic CAS-checks all of them and the transaction is genuinely all-or-nothing (previously only the candidate's version was checked, and an adapter with a stale base could silently half-commit).

Tests:

• test_contract_breaking_with_dependents_but_no_adapters_is_refused — a breaking change to an artifact with dependents and no adapters is refused (missing-adapters), nothing commits.
• test_contract_breaking_with_adapters_commits_both_atomically — with adapters supplied, candidate + adapter land in one commit_atomic (both versions advance), and on_contract_change re-measures the dependents.
• test_contract_dependents_are_re_measured_after_a_breaking_change — direct Engine.invalidate_dependents eviction check.

Also rebased onto current upstream main (PR #194 merged). 12 multi-artifact tests pass; 177 across the multi-artifact/aggregator/ledger/contracts/evolution/API suites pass.

[Birfy]

Re-reviewed ec9bd9c. Missing-all adapters now correctly rejects, the complete happy path commits all supplied states, and the initial CAS vector covers them. 91 focused shipped tests passed. Two supplied-adapter safety gaps remain; independently reproduced using the same private _Candidate/ledger setup as the new atomic-adapter tests:

1. _decide only tests whether the adapter list is nonempty (aggregator.py:1364–1366), not whether it covers the required dependent IDs. With declared dependents [b,c] and only adapter b, the breaking change commits artifact v2 and b v2 while c stays v1. With required dependent b and an unrelated adapter instead, it also commits while b remains v1. Please validate complete required-ID coverage, plus adapter identity/duplicate handling, before committing. Add partial-set and wrong-target regression cases, not just empty versus complete lists.

2. CAS retry can overwrite a concurrent adapter update. Prepare the primary candidate and b adapter from snapshot b v1, then concurrently commit b v2 containing concurrent='keep me'. The first atomic CAS correctly fails. The retry (aggregator.py:1167–1171) refreshes the head and rebases only the primary candidate, retaining the old adapter state; it then expects b v2 and writes that stale adapter as b v3, deleting the concurrent field. Please reject the stale adaptation transaction or rebuild/rebase and revalidate every affected adapter; do not advance adapter expected versions without updating their states. Add a conflict test that verifies the competing adapter update survives, or that the whole transaction is safely refused.

The extra probes produced three failures (partial set, wrong target, stale-adapter overwrite) and two passing controls (missing-all rejection, complete atomic commit). The ordinary worker preparation path still does not populate adapters, so its safe rejection is a useful improvement; the supplied-adapter path being claimed/tested here still needs these guards before it is safe for custom integration.

Separate integration status: exact-head tests run 38023236790 fails across Python 3.9/3.10/3.11/3.12. Each inspected failure summary lists the shared install-page anchor and port-count tests; docs build 38023236789 succeeds. Those shared documentation failures should be corrected and the resulting exact-head matrix rerun, independently of the adapter findings.