Add Haiku 8-way runner with journal, resume and cost cap

[drewOrc]

Step 4, first of two PRs: the Claude Haiku runner for the 8-way routing comparison (docs/PLAN.md section 4 "LLM 對照", RQ4, AC6). Nothing here has called the API; the analysis (uncertainty, risk-coverage, fallback, oracle) is the next PR.

What it does

make llm sends every validation (3,100) and test (5,500) query to claude-haiku-4-5-20251001, temperature 0, max_tokens 20, with the system prompt of cost-aware-hybrid-router. The prompt is byte-identical to src/routers/llm_router.py there (SHA-256 560d22c5...5df574, pinned in tests/test_llm.py).

One record per query: split, index, query_sha256, gold_intent, gold_agent, raw_text, agent, parse_failed, input/output/cache tokens, cost_usd, latency_ms, attempts, request_id, stop_reason, identity_sha256, created_at. The query text is not stored: CLINC150 is public and pinned, and the hash ties each record to its row (a rerun refuses a journal whose hash disagrees with the dataset).

• Resume: calls are journaled as they finish (results/llm/haiku-8way.<id12>.journal.jsonl). A rerun calls only rows with no stored reply. The journal is keyed by an identity hash of model, prompt, temperature and max_tokens, so changing any of them reuses nothing. A half-written last line is trimmed and redone.
• Cost cap (MAX_USD, default 5): prints an upper-bound estimate first (prompt tokens from the token counting endpoint, one token per query byte, output at max_tokens). Each call reserves that bound before it starts; no call starts if spend so far for this identity (earlier runs included) plus in-flight reservations could pass the cap. Actual cost comes from usage, at US$1 / US$5 per MTok (Haiku 4.5).
• Retries: done in code with the SDK's max_retries=0. 429, 5xx, 408, 409 and connection errors back off 1, 2, 4, 8 s (or retry-after, capped at 60 s), up to 5 attempts. Other errors fail at once and stop new calls. A failed row is journaled and the run exits 1.
• Completion: results/llm/haiku-8way.jsonl is read back and must hold every (split, index) once with one identity, and its SHA-256 must match results/llm/haiku-8way.json and results/llm-manifest.json. Only then does it print completed 8600/8600 llm predictions. make verify-llm reruns the check.
• Smoke: make llm-smoke calls validation rows 0-19 into results/llm-smoke/, separate from the real results, and prints tokens, cost and the extrapolated cost for 8,600 rows.
• Key: without ANTHROPIC_API_KEY it exits 2 before loading data. The key is redacted from printed output, the journal and exception text, and exceptions are not chained to the SDK error.

Changes outside the runner

• classify used to retry every exception. Now it retries only the errors listed above.
• CI's test job runs uv sync --locked --group llm, so the tests raise the SDK's real exception classes at a fake client. No key is set and nothing calls the API. tests/test_llm_deps.py fails in CI if the group is missing, so the tests that skip without the SDK cannot all skip silently there.
• The parser differs from the old project's. The old one used a substring match and took the first hit in set order. This one takes an exact label, or exactly one in-scope agent name, else oos with parse_failed. Raw replies are stored, so the old rule can be applied without new calls.

Review fixes (second round)

• One runner per target: flock on results/llm/<name>.lock; a second process exits 2 instead of spending from the same cap.
• parse_agent counts oos as a word among the candidates; two or more candidates are ambiguous, so "oos (not travel_agent)" is oos with parse_failed.
• The journal tail is repaired before it is loaded: a whole last record missing its newline is kept (it was paid for); only a half line is cut.
• count_tokens uses the same retries and key redaction as the calls.
• Unexpected exceptions are journaled as failures and stop the run; calls in flight at Ctrl-C are waited for and journaled; a response above its token bound stops the run with exit 1, even when every row has a reply.
• The summary records the parser's hash and the cap's scope. The cap covers one identity of one target, so AC6's total is this summary plus the smoke summary.

Tests

421 passed locally (73 more than main), all with a fake client. make smoke passes.

Mutation check: 43 mutations of llm.py and llm_run.py, all 43 caught, including the five the review found surviving (character bound, journal row of another identity, changed gold label, manifest-only SHA change, row recorded twice).