A/B thesis batch v3: seeded blocks, resume, infrastructure rules, Jev stop and cap

CHANGELOG.md

@@ -4,7 +4,13 @@ Notable changes to pi-warden, newest first. Versions follow semver. The publishe
 
 How to keep this current: add the entry in the same pull request as the change, under `Unreleased`. The release commit renames `Unreleased` to the version it ships and adds its own notes. Entries before 0.10.0 are one-line summaries taken from the release commit headers; the detail for those is in `git log`.
 
-## Unreleased
+## 0.89.1
+
+### Added
+
+- A/B benchmark tooling, registration v3 (`eval/preregistration.md`): model B is `claude-bridge/claude-sonnet-5-5`; agent dollars come from the providers' price pages instead of Pi's catalog, each DeepSeek call priced by its own timestamp at the peak or off-peak rate (Chinese public holidays off-peak, from the State Council's 2026 notice) and Sonnet 5.5 at Anthropic's list prices; a failed Jev judgment in a `warden` run stops the batch and the run is not counted (`eval/jev-stop.mjs`, exit code 3); `scripts/eval-call-costs.mjs` lists every call of a report with its price window; `npm run eval:power` adds the wall-clock of each batch. Repo tooling; rides along with the next release.
+- A/B batch scheduling (`eval/batch.mjs`, `scripts/eval-ab.mjs`), pre-registration v3 revised before any batch run: the queue runs in blocks (one per task x repeat, repeat by repeat) with the cell order of each block shuffled by a seeded generator (`--seed`) and the block's runs dispatched one after another, so paired runs share a price window and the machine's load; `--resume DIR` continues a stopped batch from its `runs.json`, which is now rewritten after every run; a run that fails on an agent-model API error is re-run after 1 and 5 minutes, then recorded with `infraError` and its whole block left out of the metrics, and 5 such failures in a row stop the batch (exit code 4); a Jev request still in flight when a run's process exits no longer stops the batch (recorded as `abandonedJevRequests`); `--jev-usd-cap N` stops the batch at N Jev dollars (exit code 3); the launch commands run under `caffeinate -i`. Repo tooling; rides along with the next release.
+- A/B batch corrections, registration v3 corrected before any batch run: a run is an infrastructure failure only when it ended on the error (its last assistant message, or the last of any turn, stopped on an error, or pi exited before any assistant message); an error pi retried and got past is a valid run, counted as `providerErrorsRecovered`; after an exit-4 stop all 5 runs of the streak run again on resume (an isolated failure stays excluded); a run killed at the timeout with an unreadable Jev ledger has an unknown Jev cost, so it leaves the dollar metric only and the report counts it; `--resume` refuses `--typesafe-cap`; `buildReport` and `buildWeakReport` drop excluded blocks themselves. Repo tooling; rides along with the next release.
 
 ## 0.89.0
 

eval/README.md

@@ -56,10 +56,35 @@ calculation behind its batch size is `power.mjs` (`npm run eval:power`).
 - `cost.mjs` + `config.mjs`: dollars per run, agent plus Jev. Agent input, output,
   cache-read, and cache-write tokens come from the run's session log; Jev requests
   and input tokens from the run's own pi-typesafe usage ledger; both are priced from
-  the price table in `config.mjs` (per-model prices taken from Pi's model catalog,
-  the source named per entry; a model whose bill is a plan carries list-price
-  equivalents). Reported per run and per cell; a model the table
-  does not price reports tokens with no dollars.
+  the price table in `config.mjs` (the provider's own price page, named per entry,
+  never Pi's dollar figures; a model whose bill is a plan carries list-price
+  equivalents). A time-priced model (DeepSeek's peak and off-peak rates, with the
+  Chinese public holidays off-peak) is priced per call by the call's own timestamp.
+  Reported per run and per cell; a model the table does not price reports tokens
+  with no dollars. `scripts/eval-call-costs.mjs <report folder>` lists every call of
+  a report with its UTC time, price window, tokens, and dollars.
+- `jev-stop.mjs`: the Jev-error stop of the A/B batch. A `warden` run with a failed
+  Jev judgment (a failed request in the run's usage ledger, or a fallback in its warden
+  trace file: TypeSafe unavailable, judgments off, a spend-cap stop) stops the batch;
+  the runner leaves that run out of `runs.json` and exits with code 3. A request still
+  in flight when the run's process exited, with no fallback in the trace, is recorded as
+  `abandonedJevRequests` and stops nothing.
+- `batch.mjs`: how the batch is scheduled. One block per task x repeat, repeat by
+  repeat, with the cell order of each block shuffled by a seed (`--seed`, recorded in
+  `runs.json` and the report) and its runs dispatched one after another; `--resume DIR`
+  continues a stopped batch from `DIR/runs.json`; a run that fails on an agent-model API
+  error (it ended on one: the last assistant message of the run, or of any of its
+  turns, has `stopReason` `error`, or pi gone before any assistant message; an error pi
+  retried and got past is a valid run, counted as `providerErrorsRecovered`) is re-run
+  after 1 and 5 minutes (`--retry-delays 60,300`) and then
+  recorded with `infraError`, its whole block flagged `excludedBlock` and left out of
+  the report's metrics; 5 such failures in a row stop the batch with exit code 4, and all 5 run again on
+  resume;
+  `--jev-usd-cap N` stops the batch at N Jev dollars with exit code 3. Both stops
+  resume with `--resume`, which refuses `--typesafe-cap`. A run killed at the timeout
+  whose Jev ledger cannot be read has an unknown Jev cost (`cost.jevUnknown`): it stays
+  out of the dollar columns, and the report counts it. The report builders drop the
+  excluded blocks themselves, so a report rebuilt from `runs.json` matches.
 - `weak-tasks.mjs` + `weak.mjs`: the weak-model suite (`--suite weak`). Eight everyday
   requests, each with one trap and a scripted harm and success check read from the
   run's files, its bare origin, a sandbox (a `sudo` shim that logs and fails, global
@@ -70,7 +95,17 @@ calculation behind its batch size is `power.mjs` (`npm run eval:power`).
 - `reports/`: committed `report.md` + `runs.json` per run batch. Per-run evidence
   (session logs, tool output) stays local: it is heavy and never needed to reproduce.
 
-Run: `npm run eval:ab -- --repeats 3 --concurrency 6 --model <provider/model>`. Each run
+Run: `npm run eval:ab -- --repeats 3 --concurrency 6 --model <provider/model>`. A long
+batch runs under `caffeinate -i`, with a Jev dollar cap, and resumes after a stop:
+
+```
+caffeinate -i node scripts/eval-ab.mjs --model <provider/model> --repeats 143 --turns 12 --concurrency 10 --seed 20261001 --jev-usd-cap 8
+caffeinate -i node scripts/eval-ab.mjs --resume eval/reports/<batch folder> --concurrency 10 --jev-usd-cap 8
+```
+
+Exit codes: 0 finished, 2 bad arguments, 3 stopped by a Jev error or the Jev cap, 4
+stopped by 5 infrastructure failures in a row, 130 interrupted.
+ Each run
 gets its own temp project, its own local bare `origin`, and its own
 `PI_CODING_AGENT_DIR` seeded with your provider credentials and exactly one extension
 (pi-warden, in the two warden cells), so the cells differ by that extension and, for

eval/batch.d.mts

@@ -0,0 +1,83 @@
+/** Types for eval/batch.mjs. */
+
+export const DEFAULT_SEED: number;
+export const RETRY_DELAYS_MS: number[];
+export const INFRA_STREAK_LIMIT: number;
+export const EXCLUSION_LIMIT: number;
+
+export interface Block {
+  task: string;
+  repeat: number;
+  cells: string[];
+  family?: string;
+}
+
+export interface BatchRun {
+  task: string;
+  cell: string;
+  repeat: number;
+  family?: string;
+  infraError?: string;
+  excludedBlock?: boolean;
+  attempts?: number;
+  infraRetries?: string[];
+  [key: string]: unknown;
+}
+
+export interface BatchState {
+  runs: BatchRun[];
+  jevUsd: number;
+  stops?: unknown[];
+}
+
+export interface BatchStop {
+  kind: "jev-error" | "jev-cap" | "infra" | "interrupt";
+  reason?: string;
+  failures?: { task: string; cell: string; repeat: number; reason: string }[];
+  [key: string]: unknown;
+}
+
+export interface AttemptResult {
+  record?: BatchRun | null;
+  infra?: string | null;
+  jevError?: string | null;
+  jevUsd?: number;
+  aborted?: boolean;
+  summary?: string;
+}
+
+export interface InfraCounts {
+  perCell: Record<string, { runs: number; infraErrors: number; retriedOk: number; excludedRuns: number }>;
+  excludedBlocks: number;
+  plannedBlocks: number;
+  share: number;
+  inconclusive: boolean;
+  providerErrorsRecovered: number;
+  unknownJevCost: number;
+}
+
+export function seededRandom(seed: number): () => number;
+export function shuffledCells(cells: string[], seed: number, taskId: string, repeat: number): string[];
+export function blockOrder(input: { taskIds: string[]; cells: string[]; repeats: number; seed: number }): Block[];
+export function firstRuns(blocks: Block[], count: number): Block[];
+export function runKey(run: { task: string; cell: string; repeat: number }): string;
+export function infraReason(events: unknown[], run?: { timedOut?: boolean; code?: number | null; err?: string }): string | null;
+export function recoveredErrors(events: unknown[]): number;
+export function markExclusions(runs: BatchRun[]): BatchRun[];
+export function metricRuns(runs: BatchRun[]): BatchRun[];
+export function infraCounts(runs: BatchRun[], plannedBlocks: number): InfraCounts;
+export function infraSection(runs: BatchRun[], plannedBlocks: number): string[];
+export function runBatch(input: {
+  blocks: Block[];
+  state: BatchState;
+  execute: (run: { task: string; cell: string; repeat: number; family?: string }, attempt: number) => Promise<AttemptResult>;
+  persist?: (state: BatchState) => void;
+  concurrency?: number;
+  jevUsdCap?: number | null;
+  total?: number;
+  retryDelaysMs?: number[];
+  streakLimit?: number;
+  sleep?: (ms: number) => Promise<void>;
+  log?: (line: string) => void;
+  control?: { stop: { kind: string } | null; streak: unknown[] };
+}): Promise<{ stop: BatchStop | null }>;