Add benchmark harness to compare solver revisions for correctness and speed

AGENTS.md

@@ -0,0 +1,175 @@
+# AGENTS.md — running the benchmark to compare two solver revisions
+
+This file tells an agent everything needed to benchmark the simplex solver and
+decide whether one revision is a **correct** and **significantly faster**
+improvement over another. The infrastructure lives in [`bench/`](bench/); read
+[`bench/README.md`](bench/README.md) for the *why* (methodology). This file is the
+operational contract: the exact commands, and the invariants that are easy to
+violate without noticing.
+
+## TL;DR
+
+```bash
+cd bench
+uv run python evaluate.py --baseline <git-ref> --candidate <git-ref>
+```
+
+- Exit **0** = candidate ACCEPTED (correct AND significantly faster).
+- Exit **1** = candidate REJECTED (incorrect, regression, or no significant change).
+- Exit **2** = infrastructure error (bad ref, missing tool, unparseable output).
+
+Both `--baseline` and `--candidate` are **required**; there is no default. The
+baseline is the revision to beat; the candidate is the proposed change.
+
+Typical optimization loop:
+
+```bash
+# 1. make a change, commit it on a branch
+git checkout -b my-optim
+#    ...edit solver/...
+git commit -am "try X"
+
+# 2. compare it against the current mainline (or any ref)
+cd bench
+uv run python evaluate.py --baseline claude/recursing-austin-d29bcb --candidate my-optim
+#    exit 0 -> keep it; exit 1 -> discard/iterate
+```
+
+## What the command does, step by step
+
+1. Regenerates the `train/` and `heldout/` instance corpora from fixed seeds.
+2. For **each** ref (baseline, then candidate):
+   - `git worktree add --detach` the ref into a throwaway temp dir.
+   - `dune build` the solver *in that worktree* (its own isolated `_build`).
+   - Run every held-out instance through `simplex -json --repeat N` and parse the
+     JSON metrics (status, objective, pivots, in-process solve time).
+   - Correctness-check each result against the GLPK + Gurobi oracle consensus.
+   - `git worktree remove --force` the temp worktree.
+3. Compare the two reports: geometric-mean speedup + 95% bootstrap CI + pivot
+   ratio, and print the verdict.
+
+Only the **solver binary** varies between the two runs. Instances, oracle, and
+all statistics come from the *current* `bench/` checkout (see invariants below).
+
+## Prerequisites (must all be present)
+
+| Tool | Used for | Check |
+|---|---|---|
+| `uv` | Python env (numpy, scipy) | `uv --version` |
+| `dune` + opam switch with `zarith`, `menhirSdk` | building the solver | `dune --version`; `opam switch show` |
+| `glpsol` (GLPK) | correctness oracle #1 | `glpsol --version` |
+| `gurobi_cl` (Gurobi, licensed) | correctness oracle #2 | `gurobi_cl --version` |
+| `git` with worktree support | checking out each ref | `git --version` |
+
+Python deps are pinned in `bench/pyproject.toml` / `bench/uv.lock` and installed
+automatically by `uv run`. **Do not** `pip install` into system Python — this repo
+uses `uv`; always invoke via `uv run python …` from inside `bench/`.
+
+Both oracle solvers must agree on every instance. If GLPK and Gurobi ever
+disagree (status or objective), the harness raises rather than guessing — that
+means the benchmark itself is broken and the verdict cannot be trusted.
+
+## Options
+
+| Flag / env | Default | Meaning |
+|---|---|---|
+| `--baseline <ref>` | *(required)* | git ref for the revision to beat |
+| `--candidate <ref>` | *(required)* | git ref for the revision under test |
+| `--rule <name>` / `RULE=` | `bland` | pivot rule: `bland`, `max`, or `myrule` |
+| `--trials <N>` / `TRIALS=` | `15` | timed in-process re-solves per instance |
+| `--warmup <W>` / `WARMUP=` | `3` | untimed in-process warmup solves per instance |
+
+Raise `--trials` (e.g. 30) when judging a small (<~4 %) speedup: at the default
+the laptop noise floor can occasionally flip a borderline verdict. A self-vs-self
+comparison should stay REJECTED; if it doesn't, increase `--trials`.
+
+`--rule myrule` is **randomized** (mixes bland/max via `Random.bool`), so its
+pivot count and timing are non-deterministic run-to-run. Do not use it as the rule
+when judging a change — prefer `bland` (deterministic) or `max`. Keep the rule the
+**same** for baseline and candidate; the harness already does this.
+
+## Invariants — violate these and the result is wrong or the run fails
+
+1. **Both refs must contain the `-json` solver code.** The timing/metrics come
+   solely from the solver's own `simplex -json --repeat N` output; there is no
+   fallback. A ref older than the `-json` commit builds a solver that rejects the
+   flag, and the harness detects this **up front** and stops with **exit 2** and a
+   clear message naming the ref (e.g. "ref 'baseline:main' has no -json support …
+   pick a baseline that already has -json, or rebase the -json commit onto it").
+   It does *not* try to measure such a ref — mixing an old ref's timing with a new
+   ref's would produce a meaningless comparison. Fix: pick a baseline that already
+   has `-json`, or rebase the `-json` commit onto the old ref first.
+
+2. **Comparisons are between committed refs, never the working tree.** Each ref is
+   checked out fresh, so **uncommitted edits are not measured**. Commit your change
+   on a branch first. There is no "measure the working tree" mode.
+
+3. **Instances and oracle come from the *current* `bench/`, the solver from each
+   ref.** `evaluate.py` runs from your checkout; only the compiled solver differs
+   per ref. Therefore both refs must be compatible with the current `bench/` on two
+   contracts:
+   - the **input format** (`.in`: n / m / objective / bounds / matrix rows), and
+   - the **`-json` output schema** (`status`, `value`, `value_rat`, `pivots`,
+     `trials`, `time_min`, `time_median`).
+   If a change alters either contract, the harness must be updated in lockstep, and
+   you cannot meaningfully compare across that change with one `bench/` checkout.
+
+4. **The solver maximizes.** It solves `max cᵀx s.t. Ax ≤ b, x ≥ 0`. (The old
+   `Readme.md` says "min" — that is a documentation bug.) The LP converter and
+   oracles are built around **maximize**; do not "fix" this to minimize.
+
+5. **The verdict is decided on the held-out split only.** `instances/heldout/` uses
+   disjoint seeds from `instances/train/`. If you are hand-tuning a change, look at
+   `train/` — never at `heldout/` — or you overfit the very set that judges you.
+   Both are regenerated every run, so never commit generated instances or edit them
+   by hand.
+
+6. **"Significant" means the whole 95% CI is above 1.0.** A positive geomean alone
+   is **not** acceptance — a change is accepted only if the bootstrap CI lies
+   entirely above 1.0 (`compare.py`: `accepted = correctness AND lo > 1.0`). A
+   geomean of 1.02 with CI `[0.98, 1.06]` is REJECTED as noise. Do not
+   cherry-pick the "best instance" number from the output; it is printed only to
+   show the spread.
+
+7. **Timing is intrinsic solve cost, not wall time.** `--repeat N` times the solve
+   *in-process*, excluding OCaml startup and input parsing. So numbers here (µs–ms)
+   are much smaller than end-to-end `time simplex …` and are not comparable to it.
+
+8. **Pivot count is the noise-free anchor.** The output reports a pivot-ratio too.
+   If time changed but pivots did not, the change is per-pivot cost (e.g. rational
+   blow-up); if pivots changed but time did not, per-pivot cost moved the other
+   way. A change that reports `Pivot geomean ratio: 1.0000x` and a time win is the
+   cleanest kind (same search path, cheaper arithmetic).
+
+9. **`myrule` and any randomized code path break reproducibility.** The solver
+   calls `Random.self_init` and `myrule` uses `Random.bool`. For a trustworthy
+   verdict keep the rule deterministic (see Options).
+
+10. **Don't leave worktrees behind.** The harness cleans up its temp worktrees in a
+    `finally`. If a run is killed mid-way, check `git worktree list` and
+    `git worktree remove --force <path>` any stray `simplex-ref-*` entries before
+    the next run.
+
+## Reading the verdict
+
+```
+CORRECTNESS: passed (matches GLPK+Gurobi on all instances)
+Time geomean speedup : 1.4208x (+42.08%)     <- headline metric
+  95% bootstrap CI   : [1.2331, 1.6314]       <- must be entirely > 1.0 to accept
+Pivot geomean ratio  : 1.0000x (+0.00% ...)   <- deterministic sanity anchor
+Best  instance: match_10x8.in 2.341x          <- spread, not the headline
+Worst instance: rand_5x5.in  0.875x
+VERDICT: ACCEPTED (correct and significantly faster: CI entirely above 1.0)
+```
+
+A `[BAD]` flag on any instance during the run means the candidate disagreed with
+the oracle → automatic REJECT (incorrect), regardless of speed.
+
+## Extending the corpus
+
+Instance families and sizes live in `bench/gen_instances.py`. To add a family,
+add a `family_*` function and register it in `_split_spec` with **disjoint** seed
+ranges for train (base 0) vs heldout (base 1000). Larger/harder instances make
+timing signal clearer but slow every run. See `bench/README.md` for the honest
+notes on each family (e.g. Klee–Minty is *not* a worst case for this solver's
+`max` rule — it is kept only as large-coefficient rational stress).

bench/.gitignore

@@ -0,0 +1,17 @@
+# uv-managed virtualenv (recreated from uv.lock)
+.venv/
+
+# Generated instances (reproducible from gen_instances.py + fixed seeds)
+instances/train/
+instances/heldout/
+
+# Benchmark reports and recorded baselines (machine-specific, regenerable)
+*.json
+!uv.lock
+
+# Python caches
+__pycache__/
+*.pyc
+
+# Gurobi stray log (also suppressed via LogFile= in oracle.py)
+gurobi.log

bench/README.md

@@ -0,0 +1,165 @@
+# Simplex benchmarking infrastructure
+
+This directory is a **fitness function and correctness gate for an automated
+optimization loop**. The intended workflow is:
+
+1. Claude proposes a change to the solver (`solver/`).
+2. The harness checks the change is still **correct**.
+3. The harness checks the change is a **real, significant speedup** — not noise,
+   not an artifact of overfitting to the instances it was tuned on.
+
+Only changes that pass both gates should be accepted. The design deliberately
+follows the SIGPLAN *Empirical Evaluation Checklist* and Heiser's *Systems
+Benchmarking Crimes*, because the failure modes those documents warn about are
+exactly the ways an automated loop can fool itself.
+
+## Quick start
+
+```bash
+# ONE command: compare two git refs. Both are REQUIRED (no implicit default).
+uv run python evaluate.py --baseline HEAD --candidate my-optim-branch
+# exit 0 = ACCEPT, exit 1 = REJECT, exit 2 = infra error
+```
+
+So the loop is: commit the change on a branch, then compare it against the
+baseline ref. Each ref is checked out into its **own throwaway git worktree**,
+built there (`dune build`; deps come from the shared opam switch), and measured on
+the same held-out instances. The caller's checkout and `_build` are untouched, and
+both temp worktrees are removed afterwards.
+
+Everything runs through [`uv`](https://docs.astral.sh/uv/); dependencies (numpy,
+scipy) are pinned in `pyproject.toml` / `uv.lock`. The harness is pure Python —
+`evaluate.py` orchestrates the other modules by importing them directly.
+
+## The two gates
+
+### 1. Correctness — external oracle consensus
+
+Every instance's true optimum is computed by **two independent, external LP
+solvers** and our solver must agree with both:
+
+- **GLPK** (`glpsol` 5.0) — also the primary infeasible/unbounded status oracle.
+- **Gurobi** (`gurobi_cl` 12) — numeric cross-check, run with `DualReductions=0`
+  so it too distinguishes infeasible from unbounded.
+
+Why external? Grading the solver against its own frozen outputs (the cram tests
+in `test/`) only catches regressions against *current* behavior — and a proposer
+that edits `solver/` could also edit those expected outputs. GLPK and Gurobi are
+ground truth the proposer cannot touch. This is the checklist's *appropriate
+baseline* and Heiser's *don't evaluate against yourself*.
+
+The gate is verified to catch a silent wrong-answer bug: injecting a `2x`
+objective error is flagged `[BAD]` on every instance and rejected before any
+timing is even considered.
+
+### 2. Fitness — geometric mean of speedups, with a significance test
+
+`compare.py` computes, per instance, the ratio `baseline_time / candidate_time`
+(>1 means the candidate is faster) and summarizes with the **geometric mean** —
+the correct summary for normalized ratios (Fleming & Wallace; the checklist's
+*inappropriate summary statistics* and *ratios plotted incorrectly*).
+
+A speedup is only called **significant** if a **95% bootstrap confidence
+interval** over the per-instance log-ratios lies **entirely above 1.0**. This is
+the guard against *treating noise as signal*: on a laptop, timing has real
+variance, and without this test the loop would accept meaningless ±2% wobble.
+
+- Verified: comparing the solver **against an identical copy of itself** yields a
+  geomean of ~1.00 with a CI straddling 1.0 → correctly **REJECTED** (no signal).
+
+### Pivot count — a deterministic anchor
+
+`compare.py` also reports the geomean **pivot-count** ratio. Pivot count is
+deterministic (zero measurement noise), so:
+
+- If time improved but pivots didn't, the win is in **per-pivot cost** (e.g.
+  cheaper rational arithmetic) — which is real and worth keeping.
+- If pivots dropped but time didn't, the per-pivot cost went **up** (e.g.
+  rational blow-up) — a regression that a pivots-only metric would hide.
+
+Reporting both is Heiser's *measure all effects* / the checklist's *fails to
+measure all important effects*.
+
+## Instance sources
+
+A single instance distribution invites overfitting, so the corpus spans several
+**families**, each seeded for reproducibility (`gen_instances.py`):
+
+| Family | What it is | Why it's here |
+|---|---|---|
+| `rand_NxM` | dense uniform-random LPs, sizes 5×5 → 25×60 | scaling behavior; bulk of the geomean |
+| `assign_K` | balanced K×K assignment LPs | real combinatorial structure, integral optimum |
+| `match_LxR` | max-weight bipartite matching | sparse, decouples #vars from #constraints |
+| `kleeminty_D` | Klee–Minty cube (base-100 integer form) | large-coefficient stress on exact rationals |
+| `degen_N` | highly degenerate LPs | ratio-test ties, anti-cycling stress |
+
+Measured pivot counts differ substantially between the `bland` and `max` rules
+across the assignment, matching, random, and degeneracy families — i.e. the
+corpus genuinely exercises pivot-strategy differences, which is the signal an
+optimizer needs.
+
+> Honest note on Klee–Minty: the textbook cube forces `2^D − 1` pivots under
+> Dantzig's rule, but *this* solver's max-rule + lexicographic ratio test happens
+> to solve it in one pivot. It is therefore **not** a worst case here; it is kept
+> purely as large-coefficient arithmetic stress. This is documented in the
+> generator rather than quietly relabeled, to avoid overclaiming.
+
+## Overfitting guard: train / held-out split
+
+- `instances/train/` — the proposer **may** inspect and iterate against these.
+- `instances/heldout/` — used **only** for the final accept/reject verdict.
+
+The two splits use **disjoint seed ranges** (train: base 0, held-out: base 1000),
+so no held-out instance coincides with a train instance, and both are regenerated
+from those fixed seeds on every `evaluate.py` run. This is the direct analog of
+ML cross-validation — the checklist's *tested on training set* crime. A change
+that only wins by memorizing the train instances will not clear the held-out gate.
+
+## Files
+
+| File | Role |
+|---|---|
+| `gen_instances.py` | seeded generators for all instance families; `python gen_instances.py {train,heldout} --out DIR` |
+| `to_lp.py` | convert a `.in` instance to CPLEX LP format for the oracles |
+| `oracle.py` | run GLPK + Gurobi, return their consensus optimum (raises on disagreement) |
+| `run_bench.py` | `run_corpus()`: build solver, run corpus, correctness-check vs oracle, parse the solver's `-json` metrics → report dict |
+| `compare.py` | `compare()` / `format_verdict()`: baseline vs candidate → geomean + bootstrap CI + pivot ratio → verdict |
+| `evaluate.py` | the single loop entry point tying it all together (pure Python, imports the modules above) |
+
+## Measurement hygiene
+
+- **The solver times itself, with warmup + multiple samples.** In
+  `-json --repeat N --warmup W` mode the solver runs W untimed solves (default
+  W=3, via `--warmup`) to reach steady state, then N timed solves (default N=15,
+  via `--trials`), reporting the min and median over the N samples. The timing
+  loop is *in-process*, so it excludes process startup and input parsing — the
+  metric is intrinsic solve cost, not a ~4 ms startup floor. The **minimum** is
+  the point estimate (least perturbed by external interference for a
+  deterministic CPU-bound task); the median is recorded alongside.
+- **Near-threshold verdicts need more samples.** At the default `--trials 15` the
+  laptop noise floor is a few percent, so a *true* speedup under ~3–4 % can
+  occasionally flip a self-vs-self comparison to a false ACCEPT. For a change in
+  that range, raise `--trials` (e.g. 30) — a self-vs-self check then stays
+  correctly REJECTED, confirming the floor is below your signal.
+- Each ref is built with `dune build` in its own temporary git worktree, isolated
+  from every other checkout's `_build`.
+- Bootstrap uses a fixed seed (`compare.py`), so a verdict is reproducible given
+  the same two reports.
+
+## Known limitations (disclosed, not hidden)
+
+- **Comparisons are between committed refs.** The metric source is the solver's
+  own `-json` output, which must exist in *both* refs being compared. Uncommitted
+  working-tree edits are not measured directly — commit them on a branch first.
+- **Corpus size (19 instances).** Enough to detect a solid geomean shift, but
+  borderline changes (±a few %) can tip on laptop noise. For a high-stakes
+  accept, raise `TRIALS` and/or enlarge the corpus in `gen_instances.py`.
+- **Not a realistic workload.** These are synthetic/structured LPs, not
+  production models (e.g. Netlib). They span useful structure but a win here is
+  evidence, not proof, of a win on real-world LPs.
+- **Platform-specific timing.** Absolute times reflect this machine (Apple M4).
+  The geomean-ratio design is platform-robust, but re-record the baseline on the
+  machine where verdicts are made.
+- **Feasibility bias.** Random instances are generated feasible-at-origin to focus
+  on optimization work; phase-1 feasibility hunting is under-exercised outside the
+  degenerate family.