PRD: Benchpress v3 - deterministic causal-reasoning frontier benchmark

[mark-allwyn]

Problem Statement

My existing LLM benchmarks are saturated. The general set (LLM-judge scored) no longer discriminates, and the causal multiple-choice set has top models clustered at 75-78% - a ceiling effect, not a ranking. A thinking-adaptive experiment confirmed the causal set rewards pattern-matching a correct letter, not reasoning: enabling adaptive thinking did not change scores. I need a new, genuinely hard benchmark where frontier models land around 50-60%, scored deterministically (no LLM judge), that compares heterogeneous models (reasoning vs non-reasoning) fairly, tracks models over time with honest statistics, and lets me add new metrics and benchmark domains without a rewrite.

Solution

Benchpress - a specialty benchmark for formal, verifiable-answer reasoning, causal-inference-first. Items are conjunctive multi-part: each item demands several machine-checkable parts (e.g. minimal sufficient adjustment set, numeric effect estimate, identifiability verdict) and counts as correct only if every part is correct, so the difficulty falls out of conjunction and guessing is dead. Items are procedurally generated from seeded templates with held-out parameters and randomized cover stories, so they cannot be memorized and difficulty is a tunable knob. Scoring is fully deterministic via typed per-part normalizers. Each model runs in its own best native configuration (thinking on if it has it), and results are reported as a Pareto frontier (accuracy vs thinking-tokens / latency / cost) with bootstrapped confidence intervals - never a single rank. Generated answer keys are independently verified before an item ships, response status is classified from stop_reason so refusals never masquerade as wrong answers, and content is refusal-proofed at generation time.

User Stories

1. As a benchmark maintainer, I want frontier models to score around 50-60%, so that the leaderboard discriminates among them instead of saturating.
2. As a benchmark maintainer, I want every item to require several machine-checkable parts that must all be correct, so that a model cannot succeed by pattern-matching a single answer.
3. As a benchmark maintainer, I want items generated procedurally from seeded templates, so that they cannot be memorized from training data.
4. As a benchmark maintainer, I want difficulty to be a tunable generation parameter, so that I can re-harden the benchmark by turning a knob rather than hand-authoring new traps.
5. As a benchmark maintainer, I want scoring to be fully deterministic with no LLM judge, so that scores are reproducible and free of judge bias and cost.
6. As a benchmark maintainer, I want answers emitted in a tagged-text protocol with the exact format and a worked example shown in every prompt, so that format-following is disclosed and not a gotcha, and parsing works across all providers.
7. As a benchmark maintainer, I want each answer part normalized by type (number with tolerance, node set, edge list, verdict, sequence), so that cosmetically different but equivalent answers are scored correctly.
8. As a benchmark maintainer, I want an unparseable or missing part to score as wrong for that part only, so that one malformed field does not void an otherwise-scorable item.
9. As a benchmark maintainer, I want each model run in its own best native configuration (thinking enabled where available), so that no model is handicapped by a one-size config.
10. As a benchmark maintainer, I want to capture thinking-tokens, latency, and cost per run, so that I can compare models on efficiency, not just accuracy.
11. As a benchmark maintainer, I want results presented as a Pareto frontier of accuracy against cost and thinking-tokens, so that a cheap fast model and an expensive reasoning model are each judged on their own frontier.
12. As a benchmark maintainer, I want models tiered (reasoning / non-reasoning, size class, open / closed), so that like is compared with like.
13. As a benchmark maintainer, I want bootstrapped 95% confidence intervals on every accuracy, so that I can tell whether one model genuinely beats another rather than reading noise.
14. As a benchmark maintainer, I want a headline conjunctive accuracy plus per-part marginal accuracies, so that I can see not just that a model failed but which sub-skill failed.
15. As a benchmark maintainer, I want per-skill and per-bundle breakdowns, so that I can see which causal concepts each model is weak on.
16. As a benchmark maintainer, I want results tagged with benchmark version and model launch date, so that I can track the capability frontier over time.
17. As a benchmark maintainer, I want a saturation detector that signals when the top models' confidence intervals overlap, so that I know when to re-harden to a new version.
18. As a benchmark maintainer, I want classical item statistics (p-value, item-total discrimination), so that I can flag dead items (everyone right) and broken items (everyone wrong).
19. As a benchmark maintainer, I want a fixed seeded item set per version that every model is tested on, so that comparisons are reproducible and direct.
20. As a benchmark maintainer, I want the generator code public but the seed and full item set private, so that the methodology is transparent while the canonical items resist contamination.
21. As a benchmark maintainer, I want a periodic fresh held-out audit sample, so that a large public-vs-fresh score gap reveals contamination or overfitting.
22. As a benchmark maintainer, I want every generated answer key independently verified (graphical solver cross-checked against a seeded linear-Gaussian simulation; fallacy arithmetic checked independently) before an item ships, so that I never repeat the v2 problem of miskeyed items.
23. As a benchmark maintainer, I want each response classified by status (ok / refusal / invalid_answer / truncated / api_error) from stop_reason, so that a refusal is never silently counted as a wrong answer.
24. As a benchmark maintainer, I want refusal and invalid counts shown as separate columns, so that the accuracy number is not corrupted by refusals.
25. As a benchmark maintainer, I want items generated only from refusal-neutral cover stories with a banned-vocabulary lint rejecting health/bio/security terms, so that models do not refuse on benign causal scenarios.
26. As a benchmark maintainer, I want to add a new formal domain (probability, algorithms, logic) as a registered module without touching the runner or stats, so that the benchmark grows without a rewrite.
27. As a benchmark maintainer, I want to add a new answer shape by registering one part-scorer, so that new question types are cheap to support.
28. As a benchmark maintainer, I want to add a new statistic by registering one metric function over stored results, so that new analyses do not require changing data collection.
29. As a benchmark maintainer, I want the runner to support multiple providers (Anthropic, OpenAI, Google, Ollama, Cohere, Bedrock), so that I can evaluate the full model roster.
30. As a benchmark maintainer, I want re-running scoring to never re-call the model API, so that scoring is cheap and offline.
31. As a benchmark maintainer, I want a per-model results file with multi-run history, so that reruns are preserved rather than overwritten.
32. As a benchmark maintainer, I want a thin end-to-end slice (one bundle, one model) working before volume work, so that every layer is proven to integrate early.
33. As a benchmark maintainer, I want a stats CLI and JSON export first and the dashboard deferred, so that I get useful results without building a large site up front.

Implementation Decisions

• New benchpress package at repo root; the v2 system stays archived under legacy/ as read-only reference. Drop the LLM judge, DeepEval, and the general (judged) set entirely.
• Three registries provide extensibility (generalizing the legacy CHECKERS dict): a module registry (benchmark generators), a part-scorer registry keyed by part type, and a metric registry (functions over stored results). Adding a domain, an answer shape, or a statistic each means registering one function.
• Module interface: a generator is a pure, deterministic function generator(seed, difficulty) -> (items, module_meta). Same seed yields identical items - this is the reproducibility contract.
• Item shape (from the design): an item has an id, module, bundle, variant, difficulty, the generation params, a fully rendered prompt (including the tagged-format spec and a worked example), and a list of parts. Each part has an id (also its tagged-text label), a part_type, the expected gold value, scorer params (e.g. numeric tolerance), and skill tags.
• Part-scorer interface: (gold_part, raw_field_or_none) -> part_result{correct, parsed, expected, note}. A missing or unparseable field yields correct=false, parsed=none - a wrong part, not a voided item. Five initial scorers: set_match, numeric_tolerance, sequence_match, edge_list, categorical.
• Answer protocol: tagged text, one labelled line per part, parsed by tolerant per-field extractors (case-insensitive, markdown-tolerant, last occurrence wins). No dependence on provider JSON or tool-calling modes. Per-type normalizers handle fractions/scientific-notation/percent for numbers, brace/bracket/delimiter/empty-set handling for sets, arrow and pair forms for edge lists, and synonym classes for verdicts.
• Scoring: item_correct = all parts correct (conjunctive). Persist per-part results for marginal diagnostics. Single-shot, no reparse retries, no LLM fallback.
• Dual-verification admission gate: no causal item enters the pool unless two independent methods agree on its key (graphical solver vs seeded linear-Gaussian simulation + partial correlation; fallacy arithmetic checked independently). Disagreement rejects the item.
• Status taxonomy: classify every response ok | refusal | invalid_answer | truncated | api_error from stop_reason + stop_details + extraction outcome. Refusal/invalid count as wrong but are reported in separate columns.
• Refusal-proofing: refusal-neutral cover stories only (logistics, manufacturing, agriculture, education, marketing); a banned-vocabulary lint rejects health/bio/security terms at generation time.
• Provider interface (ported from legacy): complete(prompt, params) -> (content, usage), one class per vendor, factory selects by config. The one change: capture thinking_tokens and the native config used into usage. Anthropic sends adaptive thinking for 4.6+/Fable and reads thinking tokens from the usage details; OpenAI reasoning models read reasoning tokens; Google reads thought tokens; others none unless exposed. The runner computes cost from a price table.
• Fairness model: run each model in its best native mode; never collapse to a single rank; report a Pareto frontier (accuracy vs thinking-tokens / latency / cost) and tier models. Budget-normalized comparison is an optional secondary view, not primary.
• Results schema: per-model JSON with runs[item_id] as a list (multi-run history). Each run stores native config used, content, latency, input/output/thinking tokens, cost, benchmark version, status, item_correct, and a parts array (part id, type, parsed, expected, correct, note). Gold values are not stored here; they regenerate from (seed, difficulty), keeping the public generator separate from private keys.
• Reproducibility & durability: a fixed seeded set per version ("Benchpress-Causal v1") is canonical; re-hardening bumps the version. Generator code public; seed and full item set private/held-out; only example items and aggregate results published. A periodic fresh held-out audit sample detects contamination.
• Statistics layer (decoupled, reads persisted results only): bootstrapped 95% CIs on accuracy; Pareto frontier; per-skill marginals; version + launch-date over-time tracking; saturation detector (top-N CI overlap); classical item statistics (p-value, item-total discrimination). IRT deferred until the tested-model population is large enough; it slots in later as a metric without changing data collection.
• Difficulty target is 50-60% for frontier models.
• CLI: generate, run, score, stats, export, audit. Scoring is offline and never re-calls the API.
• Reuse: port the legacy provider layer and the per-model JSON / multi-run-history persistence pattern; reuse the root config.yaml model registry (provider, model, company, type, launch_date, api_key_env, params).

Testing Decisions

Good tests assert externally observable behavior at the module boundaries, not internals. The seams, highest first:

• CLI seam (end-to-end): drive run against a mocked provider (canned API response) and assert an item flows generate -> raw -> scored -> leaderboard with the correct accuracy, status classification, and conjunctive correctness. Re-running score must not call the provider. Prior art: the legacy tests/test_run.py and tests/test_integration.py mocked-provider end-to-end tests.
• Generator seam: generator(seed, difficulty) twice yields byte-identical items (reproducibility); the dual-verification gate is exercised by asserting solver and simulation agree on many random instances and that injected disagreements reject the item; the banned-vocabulary lint is green over the full generated set.
• Part-scorer seam: each of the five scorers tested for exact-match, tolerant-format (e.g. {X, Z} vs X, Z, 7/2 vs 3.5, arrow forms, empty-set synonyms), and unparseable/missing -> wrong-part-not-exception. Prior art: the legacy tests/test_checks.py fixture-based checker tests.
• Provider seam: mock at the HTTP boundary and assert stop_reason -> status mapping (a refusal classifies as refusal, not invalid) and thinking-token extraction per vendor. Prior art: the legacy tests/test_providers.py.
• Metric seam: run metrics over synthetic persisted results (no provider) and assert the CI brackets the point estimate, each per-part marginal is >= the conjunctive accuracy, and refusal/invalid columns reconcile exactly with raw stop_reason counts.

Modules under test: tagged-text extractor, the five part-scorers, the causal generator + dual-verification gate + vocab lint, the ported providers (status + thinking-token mapping), the runner/scorer end-to-end, and the stats metrics. The thin vertical slice (one bundle, one model) is the first thing made green, including one live smoke call recording stop_reason and thinking_tokens.

Out of Scope

• The LLM judge, DeepEval, and the general (judged) eval set - dropped entirely.
• Additional formal domains (probability, algorithms, logic) - the architecture supports them as future modules, but only the causal module ships now.
• The dashboard / static site - deferred; a lean stats CLI and JSON export ship first.
• Calibration pilot, post-sweep consensus audit, and production runner hardening (resume, backoff, concurrency caps) - explicitly excluded from the causal-first build; may return later.
• IRT-based item/model placement - deferred until enough models are tested; classical item statistics first.
• Budget-normalized (equal-token) comparison as a primary methodology - optional secondary view only.
• k-times repeated sampling per item - the canonical run is once per item; repeated sampling is an optional later mode.

Further Notes

This PRD supersedes the prior 12-issue v3 plan (#1-12, now closed). Three pieces of that plan were folded in because they fix documented failures in our own causal work: the dual-verification gate (v2 shipped ~9 miskeyed causal transfer items), the status taxonomy (Fable 5's 21 causal refusals were recorded as wrong because the harness never captured stop_reason), and generation-time refusal-proofing (Fable refuses on health/bio causal scenarios). The remaining detail of those closed issues (calibration band rules, consensus-audit procedure, sandboxed code execution, math dual-computation) remains recoverable from the closed issues if a later phase needs it. Full design lives in plans/2026-06-23-benchpress-design.md.