Add benchmark harness to compare solver revisions for correctness and speed#1
Merged
Merged
Conversation
Emit a single JSON object (status, objective as exact rational + float, pivots, and in-process solve timing) when -json is passed. --repeat N re-solves N times in-process so the reported time excludes process startup and parsing. Used by the bench/ harness as the metric source.
bench/ builds each of two git refs in its own throwaway worktree, runs a seeded corpus of LP instances through the solver's -json metrics mode, and decides whether the candidate is both correct and significantly faster than the baseline. - Correctness gate: every instance's optimum is cross-checked against the GLPK and Gurobi oracles, so a change is graded against independent ground truth rather than the solver's own frozen outputs. - Fitness gate: geometric mean of per-instance solve-time ratios with a 95% bootstrap confidence interval; a change is accepted only if the interval lies entirely above 1.0. Pivot count is reported alongside as a deterministic anchor. - Instances span random-dense, assignment, bipartite-matching, Klee-Minty, and degenerate families, split into disjoint train/heldout seed ranges to guard against overfitting; the verdict is decided on heldout only. evaluate.py is the entry point: uv run python evaluate.py --baseline <ref> --candidate <ref> AGENTS.md documents the commands, prerequisites, and invariants.
Run W untimed solves before the timed --repeat ones so the reported time_min/time_median reflect steady state rather than a cold first solve.
evaluate.py/run_bench.py gain --warmup (default 3), passed to the solver's -json --warmup. run_bench also fails fast with a clear message when a ref's solver lacks -json, instead of surfacing an unparseable-JSON error.
- Vectorize compare.py's bootstrap: the 10k-iteration Python loop becomes a single numpy draw (identical CI, faster and clearer). - Drop the unused standalone CLIs from compare.py and run_bench.py; evaluate.py is the sole entry point and imports the functions directly. - Remove dead imports (json in evaluate.py) and dead locals (n, m in two instance generators). - Inline the trivial index_by_instance/load helpers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
bench/, a benchmarking harness that decides whether one solver revision isboth correct and significantly faster than another. Intended as the
fitness function and correctness gate for an automated optimization loop.
Each ref is checked out into its own throwaway git worktree, built there, and
measured on the same held-out instances.
How it works
independent external solvers (GLPK + Gurobi). The solver is never graded against
its own frozen outputs, so the gate can't be fooled by editing test files.
bootstrap confidence interval; a change is accepted only if the interval lies
entirely above 1.0 (guards against reading noise as signal). Pivot count is
reported alongside as a deterministic anchor.
-json --repeat N --warmup Wmode that timesthe solve in-process, excluding process-startup/parse overhead. Metrics are
emitted as JSON and parsed by the harness.
matching, Klee–Minty, and degenerate families, split into disjoint train/heldout
seed ranges; the verdict is decided on heldout only.
Notes
solver/tableau.ml) is unchanged — this PR only addsthe benchmark tooling and a
-jsonmetrics-output mode to the CLI.AGENTS.mddocuments the commands, prerequisites, and invariants.benchmarking-crimes guidance (external baseline, geomean of ratios, significance
testing, held-out set).
uv,dune,glpsol, andgurobi_cl.