A/B thesis batch v3: seeded blocks, resume, infrastructure rules, Jev stop and cap

[DevMortimer]

Prepares the pre-registered A/B batch (registration v3). It does not launch it.

Pricing and models

• Model B is claude-bridge/claude-sonnet-5-5 everywhere in eval/ and its docs. Model A stays deepseek/deepseek-flash. Batch sizes are unchanged: A 20 x 143 x 3 = 8,580 runs, B 20 x 40 x 3 = 2,400 runs.
• Prices come from the providers' price pages. Each DeepSeek call is priced by its own timestamp at the peak or off-peak rate; Sonnet 5.5 uses Anthropic's list prices.

Batch harness (eval/batch.mjs, scripts/eval-ab.mjs)

• Blocks: one per task x repeat, with the cell order shuffled by a registered seed; a block's runs go one after another.
• --resume DIR continues a stopped batch from runs.json, which is written atomically after every run. It refuses --typesafe-cap.
• Infrastructure failure: a run is one only when it ended on the error (its last assistant message, or the last of any turn, has stopReason error, or pi exited before any assistant message). An error pi retried and got past is a valid run, counted per run as providerErrorsRecovered.
• A failure is re-run after 1 and 5 minutes, then recorded with infraError, and its whole block leaves the metrics. More than 10% of a model's blocks excluded makes it inconclusive.
• 5 final failures in a row stop the batch (exit 4). All 5 run again on resume, not only the runs in flight; an isolated failure stays excluded.
• A failed Jev judgment in a warden run stops the batch (exit 3). A request in flight when pi exits is recorded as abandonedJevRequests. A run killed at the timeout with an unreadable Jev ledger has an unknown Jev cost: it leaves the dollar metric only, and the report counts it.
• --jev-usd-cap N stops the batch at N Jev dollars (exit 3). It is cumulative across resumes, counts re-runs, and may overshoot by the runs in flight.
• buildReport and buildWeakReport drop excluded blocks themselves, so a report rebuilt from runs.json matches the batch's report.

Docs

• eval/preregistration.md v3, with a registration-history line for these corrections (still before any batch run), eval/README.md, and CHANGELOG.md under Unreleased.

Commit under test: 5ff3c778193584ce2404b1b43a31f5a40ef6563d (main at 0.89.0). The launch check git diff 5ff3c778193584ce2404b1b43a31f5a40ef6563d HEAD -- src prints nothing. main is merged in and the version is 0.89.1.

npm run check: 1,412 tests pass.