diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d793d4c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,175 @@ +# AGENTS.md — running the benchmark to compare two solver revisions + +This file tells an agent everything needed to benchmark the simplex solver and +decide whether one revision is a **correct** and **significantly faster** +improvement over another. The infrastructure lives in [`bench/`](bench/); read +[`bench/README.md`](bench/README.md) for the *why* (methodology). This file is the +operational contract: the exact commands, and the invariants that are easy to +violate without noticing. + +## TL;DR + +```bash +cd bench +uv run python evaluate.py --baseline --candidate +``` + +- Exit **0** = candidate ACCEPTED (correct AND significantly faster). +- Exit **1** = candidate REJECTED (incorrect, regression, or no significant change). +- Exit **2** = infrastructure error (bad ref, missing tool, unparseable output). + +Both `--baseline` and `--candidate` are **required**; there is no default. The +baseline is the revision to beat; the candidate is the proposed change. + +Typical optimization loop: + +```bash +# 1. make a change, commit it on a branch +git checkout -b my-optim +# ...edit solver/... +git commit -am "try X" + +# 2. compare it against the current mainline (or any ref) +cd bench +uv run python evaluate.py --baseline claude/recursing-austin-d29bcb --candidate my-optim +# exit 0 -> keep it; exit 1 -> discard/iterate +``` + +## What the command does, step by step + +1. Regenerates the `train/` and `heldout/` instance corpora from fixed seeds. +2. For **each** ref (baseline, then candidate): + - `git worktree add --detach` the ref into a throwaway temp dir. + - `dune build` the solver *in that worktree* (its own isolated `_build`). + - Run every held-out instance through `simplex -json --repeat N` and parse the + JSON metrics (status, objective, pivots, in-process solve time). + - Correctness-check each result against the GLPK + Gurobi oracle consensus. + - `git worktree remove --force` the temp worktree. +3. Compare the two reports: geometric-mean speedup + 95% bootstrap CI + pivot + ratio, and print the verdict. + +Only the **solver binary** varies between the two runs. Instances, oracle, and +all statistics come from the *current* `bench/` checkout (see invariants below). + +## Prerequisites (must all be present) + +| Tool | Used for | Check | +|---|---|---| +| `uv` | Python env (numpy, scipy) | `uv --version` | +| `dune` + opam switch with `zarith`, `menhirSdk` | building the solver | `dune --version`; `opam switch show` | +| `glpsol` (GLPK) | correctness oracle #1 | `glpsol --version` | +| `gurobi_cl` (Gurobi, licensed) | correctness oracle #2 | `gurobi_cl --version` | +| `git` with worktree support | checking out each ref | `git --version` | + +Python deps are pinned in `bench/pyproject.toml` / `bench/uv.lock` and installed +automatically by `uv run`. **Do not** `pip install` into system Python — this repo +uses `uv`; always invoke via `uv run python …` from inside `bench/`. + +Both oracle solvers must agree on every instance. If GLPK and Gurobi ever +disagree (status or objective), the harness raises rather than guessing — that +means the benchmark itself is broken and the verdict cannot be trusted. + +## Options + +| Flag / env | Default | Meaning | +|---|---|---| +| `--baseline ` | *(required)* | git ref for the revision to beat | +| `--candidate ` | *(required)* | git ref for the revision under test | +| `--rule ` / `RULE=` | `bland` | pivot rule: `bland`, `max`, or `myrule` | +| `--trials ` / `TRIALS=` | `15` | timed in-process re-solves per instance | +| `--warmup ` / `WARMUP=` | `3` | untimed in-process warmup solves per instance | + +Raise `--trials` (e.g. 30) when judging a small (<~4 %) speedup: at the default +the laptop noise floor can occasionally flip a borderline verdict. A self-vs-self +comparison should stay REJECTED; if it doesn't, increase `--trials`. + +`--rule myrule` is **randomized** (mixes bland/max via `Random.bool`), so its +pivot count and timing are non-deterministic run-to-run. Do not use it as the rule +when judging a change — prefer `bland` (deterministic) or `max`. Keep the rule the +**same** for baseline and candidate; the harness already does this. + +## Invariants — violate these and the result is wrong or the run fails + +1. **Both refs must contain the `-json` solver code.** The timing/metrics come + solely from the solver's own `simplex -json --repeat N` output; there is no + fallback. A ref older than the `-json` commit builds a solver that rejects the + flag, and the harness detects this **up front** and stops with **exit 2** and a + clear message naming the ref (e.g. "ref 'baseline:main' has no -json support … + pick a baseline that already has -json, or rebase the -json commit onto it"). + It does *not* try to measure such a ref — mixing an old ref's timing with a new + ref's would produce a meaningless comparison. Fix: pick a baseline that already + has `-json`, or rebase the `-json` commit onto the old ref first. + +2. **Comparisons are between committed refs, never the working tree.** Each ref is + checked out fresh, so **uncommitted edits are not measured**. Commit your change + on a branch first. There is no "measure the working tree" mode. + +3. **Instances and oracle come from the *current* `bench/`, the solver from each + ref.** `evaluate.py` runs from your checkout; only the compiled solver differs + per ref. Therefore both refs must be compatible with the current `bench/` on two + contracts: + - the **input format** (`.in`: n / m / objective / bounds / matrix rows), and + - the **`-json` output schema** (`status`, `value`, `value_rat`, `pivots`, + `trials`, `time_min`, `time_median`). + If a change alters either contract, the harness must be updated in lockstep, and + you cannot meaningfully compare across that change with one `bench/` checkout. + +4. **The solver maximizes.** It solves `max cᵀx s.t. Ax ≤ b, x ≥ 0`. (The old + `Readme.md` says "min" — that is a documentation bug.) The LP converter and + oracles are built around **maximize**; do not "fix" this to minimize. + +5. **The verdict is decided on the held-out split only.** `instances/heldout/` uses + disjoint seeds from `instances/train/`. If you are hand-tuning a change, look at + `train/` — never at `heldout/` — or you overfit the very set that judges you. + Both are regenerated every run, so never commit generated instances or edit them + by hand. + +6. **"Significant" means the whole 95% CI is above 1.0.** A positive geomean alone + is **not** acceptance — a change is accepted only if the bootstrap CI lies + entirely above 1.0 (`compare.py`: `accepted = correctness AND lo > 1.0`). A + geomean of 1.02 with CI `[0.98, 1.06]` is REJECTED as noise. Do not + cherry-pick the "best instance" number from the output; it is printed only to + show the spread. + +7. **Timing is intrinsic solve cost, not wall time.** `--repeat N` times the solve + *in-process*, excluding OCaml startup and input parsing. So numbers here (µs–ms) + are much smaller than end-to-end `time simplex …` and are not comparable to it. + +8. **Pivot count is the noise-free anchor.** The output reports a pivot-ratio too. + If time changed but pivots did not, the change is per-pivot cost (e.g. rational + blow-up); if pivots changed but time did not, per-pivot cost moved the other + way. A change that reports `Pivot geomean ratio: 1.0000x` and a time win is the + cleanest kind (same search path, cheaper arithmetic). + +9. **`myrule` and any randomized code path break reproducibility.** The solver + calls `Random.self_init` and `myrule` uses `Random.bool`. For a trustworthy + verdict keep the rule deterministic (see Options). + +10. **Don't leave worktrees behind.** The harness cleans up its temp worktrees in a + `finally`. If a run is killed mid-way, check `git worktree list` and + `git worktree remove --force ` any stray `simplex-ref-*` entries before + the next run. + +## Reading the verdict + +``` +CORRECTNESS: passed (matches GLPK+Gurobi on all instances) +Time geomean speedup : 1.4208x (+42.08%) <- headline metric + 95% bootstrap CI : [1.2331, 1.6314] <- must be entirely > 1.0 to accept +Pivot geomean ratio : 1.0000x (+0.00% ...) <- deterministic sanity anchor +Best instance: match_10x8.in 2.341x <- spread, not the headline +Worst instance: rand_5x5.in 0.875x +VERDICT: ACCEPTED (correct and significantly faster: CI entirely above 1.0) +``` + +A `[BAD]` flag on any instance during the run means the candidate disagreed with +the oracle → automatic REJECT (incorrect), regardless of speed. + +## Extending the corpus + +Instance families and sizes live in `bench/gen_instances.py`. To add a family, +add a `family_*` function and register it in `_split_spec` with **disjoint** seed +ranges for train (base 0) vs heldout (base 1000). Larger/harder instances make +timing signal clearer but slow every run. See `bench/README.md` for the honest +notes on each family (e.g. Klee–Minty is *not* a worst case for this solver's +`max` rule — it is kept only as large-coefficient rational stress). diff --git a/bench/.gitignore b/bench/.gitignore new file mode 100644 index 0000000..6b4a904 --- /dev/null +++ b/bench/.gitignore @@ -0,0 +1,17 @@ +# uv-managed virtualenv (recreated from uv.lock) +.venv/ + +# Generated instances (reproducible from gen_instances.py + fixed seeds) +instances/train/ +instances/heldout/ + +# Benchmark reports and recorded baselines (machine-specific, regenerable) +*.json +!uv.lock + +# Python caches +__pycache__/ +*.pyc + +# Gurobi stray log (also suppressed via LogFile= in oracle.py) +gurobi.log diff --git a/bench/README.md b/bench/README.md new file mode 100644 index 0000000..5afa2e9 --- /dev/null +++ b/bench/README.md @@ -0,0 +1,165 @@ +# Simplex benchmarking infrastructure + +This directory is a **fitness function and correctness gate for an automated +optimization loop**. The intended workflow is: + +1. Claude proposes a change to the solver (`solver/`). +2. The harness checks the change is still **correct**. +3. The harness checks the change is a **real, significant speedup** — not noise, + not an artifact of overfitting to the instances it was tuned on. + +Only changes that pass both gates should be accepted. The design deliberately +follows the SIGPLAN *Empirical Evaluation Checklist* and Heiser's *Systems +Benchmarking Crimes*, because the failure modes those documents warn about are +exactly the ways an automated loop can fool itself. + +## Quick start + +```bash +# ONE command: compare two git refs. Both are REQUIRED (no implicit default). +uv run python evaluate.py --baseline HEAD --candidate my-optim-branch +# exit 0 = ACCEPT, exit 1 = REJECT, exit 2 = infra error +``` + +So the loop is: commit the change on a branch, then compare it against the +baseline ref. Each ref is checked out into its **own throwaway git worktree**, +built there (`dune build`; deps come from the shared opam switch), and measured on +the same held-out instances. The caller's checkout and `_build` are untouched, and +both temp worktrees are removed afterwards. + +Everything runs through [`uv`](https://docs.astral.sh/uv/); dependencies (numpy, +scipy) are pinned in `pyproject.toml` / `uv.lock`. The harness is pure Python — +`evaluate.py` orchestrates the other modules by importing them directly. + +## The two gates + +### 1. Correctness — external oracle consensus + +Every instance's true optimum is computed by **two independent, external LP +solvers** and our solver must agree with both: + +- **GLPK** (`glpsol` 5.0) — also the primary infeasible/unbounded status oracle. +- **Gurobi** (`gurobi_cl` 12) — numeric cross-check, run with `DualReductions=0` + so it too distinguishes infeasible from unbounded. + +Why external? Grading the solver against its own frozen outputs (the cram tests +in `test/`) only catches regressions against *current* behavior — and a proposer +that edits `solver/` could also edit those expected outputs. GLPK and Gurobi are +ground truth the proposer cannot touch. This is the checklist's *appropriate +baseline* and Heiser's *don't evaluate against yourself*. + +The gate is verified to catch a silent wrong-answer bug: injecting a `2x` +objective error is flagged `[BAD]` on every instance and rejected before any +timing is even considered. + +### 2. Fitness — geometric mean of speedups, with a significance test + +`compare.py` computes, per instance, the ratio `baseline_time / candidate_time` +(>1 means the candidate is faster) and summarizes with the **geometric mean** — +the correct summary for normalized ratios (Fleming & Wallace; the checklist's +*inappropriate summary statistics* and *ratios plotted incorrectly*). + +A speedup is only called **significant** if a **95% bootstrap confidence +interval** over the per-instance log-ratios lies **entirely above 1.0**. This is +the guard against *treating noise as signal*: on a laptop, timing has real +variance, and without this test the loop would accept meaningless ±2% wobble. + +- Verified: comparing the solver **against an identical copy of itself** yields a + geomean of ~1.00 with a CI straddling 1.0 → correctly **REJECTED** (no signal). + +### Pivot count — a deterministic anchor + +`compare.py` also reports the geomean **pivot-count** ratio. Pivot count is +deterministic (zero measurement noise), so: + +- If time improved but pivots didn't, the win is in **per-pivot cost** (e.g. + cheaper rational arithmetic) — which is real and worth keeping. +- If pivots dropped but time didn't, the per-pivot cost went **up** (e.g. + rational blow-up) — a regression that a pivots-only metric would hide. + +Reporting both is Heiser's *measure all effects* / the checklist's *fails to +measure all important effects*. + +## Instance sources + +A single instance distribution invites overfitting, so the corpus spans several +**families**, each seeded for reproducibility (`gen_instances.py`): + +| Family | What it is | Why it's here | +|---|---|---| +| `rand_NxM` | dense uniform-random LPs, sizes 5×5 → 25×60 | scaling behavior; bulk of the geomean | +| `assign_K` | balanced K×K assignment LPs | real combinatorial structure, integral optimum | +| `match_LxR` | max-weight bipartite matching | sparse, decouples #vars from #constraints | +| `kleeminty_D` | Klee–Minty cube (base-100 integer form) | large-coefficient stress on exact rationals | +| `degen_N` | highly degenerate LPs | ratio-test ties, anti-cycling stress | + +Measured pivot counts differ substantially between the `bland` and `max` rules +across the assignment, matching, random, and degeneracy families — i.e. the +corpus genuinely exercises pivot-strategy differences, which is the signal an +optimizer needs. + +> Honest note on Klee–Minty: the textbook cube forces `2^D − 1` pivots under +> Dantzig's rule, but *this* solver's max-rule + lexicographic ratio test happens +> to solve it in one pivot. It is therefore **not** a worst case here; it is kept +> purely as large-coefficient arithmetic stress. This is documented in the +> generator rather than quietly relabeled, to avoid overclaiming. + +## Overfitting guard: train / held-out split + +- `instances/train/` — the proposer **may** inspect and iterate against these. +- `instances/heldout/` — used **only** for the final accept/reject verdict. + +The two splits use **disjoint seed ranges** (train: base 0, held-out: base 1000), +so no held-out instance coincides with a train instance, and both are regenerated +from those fixed seeds on every `evaluate.py` run. This is the direct analog of +ML cross-validation — the checklist's *tested on training set* crime. A change +that only wins by memorizing the train instances will not clear the held-out gate. + +## Files + +| File | Role | +|---|---| +| `gen_instances.py` | seeded generators for all instance families; `python gen_instances.py {train,heldout} --out DIR` | +| `to_lp.py` | convert a `.in` instance to CPLEX LP format for the oracles | +| `oracle.py` | run GLPK + Gurobi, return their consensus optimum (raises on disagreement) | +| `run_bench.py` | `run_corpus()`: build solver, run corpus, correctness-check vs oracle, parse the solver's `-json` metrics → report dict | +| `compare.py` | `compare()` / `format_verdict()`: baseline vs candidate → geomean + bootstrap CI + pivot ratio → verdict | +| `evaluate.py` | the single loop entry point tying it all together (pure Python, imports the modules above) | + +## Measurement hygiene + +- **The solver times itself, with warmup + multiple samples.** In + `-json --repeat N --warmup W` mode the solver runs W untimed solves (default + W=3, via `--warmup`) to reach steady state, then N timed solves (default N=15, + via `--trials`), reporting the min and median over the N samples. The timing + loop is *in-process*, so it excludes process startup and input parsing — the + metric is intrinsic solve cost, not a ~4 ms startup floor. The **minimum** is + the point estimate (least perturbed by external interference for a + deterministic CPU-bound task); the median is recorded alongside. +- **Near-threshold verdicts need more samples.** At the default `--trials 15` the + laptop noise floor is a few percent, so a *true* speedup under ~3–4 % can + occasionally flip a self-vs-self comparison to a false ACCEPT. For a change in + that range, raise `--trials` (e.g. 30) — a self-vs-self check then stays + correctly REJECTED, confirming the floor is below your signal. +- Each ref is built with `dune build` in its own temporary git worktree, isolated + from every other checkout's `_build`. +- Bootstrap uses a fixed seed (`compare.py`), so a verdict is reproducible given + the same two reports. + +## Known limitations (disclosed, not hidden) + +- **Comparisons are between committed refs.** The metric source is the solver's + own `-json` output, which must exist in *both* refs being compared. Uncommitted + working-tree edits are not measured directly — commit them on a branch first. +- **Corpus size (19 instances).** Enough to detect a solid geomean shift, but + borderline changes (±a few %) can tip on laptop noise. For a high-stakes + accept, raise `TRIALS` and/or enlarge the corpus in `gen_instances.py`. +- **Not a realistic workload.** These are synthetic/structured LPs, not + production models (e.g. Netlib). They span useful structure but a win here is + evidence, not proof, of a win on real-world LPs. +- **Platform-specific timing.** Absolute times reflect this machine (Apple M4). + The geomean-ratio design is platform-robust, but re-record the baseline on the + machine where verdicts are made. +- **Feasibility bias.** Random instances are generated feasible-at-origin to focus + on optimization work; phase-1 feasibility hunting is under-exercised outside the + degenerate family. diff --git a/bench/compare.py b/bench/compare.py new file mode 100644 index 0000000..f0917dc --- /dev/null +++ b/bench/compare.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Compare two benchmark reports and decide: real, significant improvement? + +This is the *verdict* stage of the optimization loop. Given a BASELINE report and +a CANDIDATE report (both produced by run_bench.py on the SAME instance set), it: + + 1. CORRECTNESS: if the candidate is incorrect on any instance, it is rejected + outright, regardless of speed. (The baseline is assumed already-correct.) + + 2. FITNESS: computes the geometric mean of per-instance speed ratios + baseline_time / candidate_time. A geomean > 1 means the candidate is faster + on average. The geometric mean is the correct summary for normalized ratios + (Fleming & Wallace; SIGPLAN checklist "inappropriate summary statistics"). + + 3. SIGNIFICANCE: a bootstrap 95% confidence interval over the per-instance log + ratios. We only call an improvement "significant" if the entire CI lies above + 1.0 (i.e. we are confident the geomean speedup is real, not measurement + noise). This is the guard against "treating noise as signal" -- the failure + mode that would let an automated loop accept junk changes. + + 4. PIVOTS: reports the geomean pivot ratio too. Pivots are deterministic, so a + pivot-count change is noise-free evidence; if time improved but pivots did + not, the improvement is in per-pivot cost (and vice-versa). Surfaced, never + hidden, so per-pivot regressions (e.g. rational blow-up) can't sneak through. + +`compare()` returns a verdict dict (`accepted` is the headline); `format_verdict()` +renders it. Both are imported by evaluate.py -- this module has no CLI of its own. + +Bootstrap uses a fixed seed so the verdict is reproducible run-to-run given the +same reports. +""" + +import math + +import numpy as np + +# We use the per-instance MINIMUM time as the point estimate (least perturbed by +# external interference for a deterministic CPU-bound task; see run_bench.py). +POINT = "time_min" + +BOOTSTRAP_RESAMPLES = 10000 +CI = 0.95 +SEED = 12345 + + +def _geomean(xs): + return math.exp(sum(math.log(x) for x in xs) / len(xs)) + + +def compare(baseline, candidate): + b_idx = {r["instance"]: r for r in baseline["results"]} + c_idx = {r["instance"]: r for r in candidate["results"]} + common = sorted(set(b_idx) & set(c_idx)) + if not common: + raise RuntimeError("baseline and candidate share no instances") + + # --- correctness gate -------------------------------------------------- + incorrect = [name for name in common if not c_idx[name]["correct"]] + candidate_correct = (len(incorrect) == 0) and candidate["all_correct"] + + # --- per-instance ratios ---------------------------------------------- + time_ratios = [] # baseline / candidate (>1 == candidate faster) + pivot_ratios = [] + per_instance = [] + for name in common: + bt = b_idx[name][POINT] + ct = c_idx[name][POINT] + tr = bt / ct + time_ratios.append(tr) + + bp = b_idx[name]["pivots"] + cp = c_idx[name]["pivots"] + pr = (bp / cp) if (bp and cp) else None + if pr is not None: + pivot_ratios.append(pr) + + per_instance.append({ + "instance": name, + "baseline_time": bt, + "candidate_time": ct, + "time_ratio": tr, + "baseline_pivots": bp, + "candidate_pivots": cp, + "correct": c_idx[name]["correct"], + }) + + time_geomean = _geomean(time_ratios) + pivot_geomean = _geomean(pivot_ratios) if pivot_ratios else None + + # --- bootstrap CI on the geomean of time ratios ----------------------- + # Resample the per-instance log-ratios with replacement; the geomean of a + # resample is exp(mean of its logs). One vectorized draw of shape + # (resamples, n) does all BOOTSTRAP_RESAMPLES at once. + rng = np.random.default_rng(SEED) + log_ratios = np.log(time_ratios) + resamples = rng.choice(log_ratios, size=(BOOTSTRAP_RESAMPLES, len(log_ratios)), + replace=True) + geomeans = np.exp(resamples.mean(axis=1)) + lo, hi = np.quantile(geomeans, [(1 - CI) / 2, 1 - (1 - CI) / 2]) + lo, hi = float(lo), float(hi) + + significant_speedup = candidate_correct and lo > 1.0 + significant_regression = hi < 1.0 + + return { + "candidate_correct": candidate_correct, + "incorrect_instances": incorrect, + "n_instances": len(common), + "time_geomean": time_geomean, + "time_ci_low": lo, + "time_ci_high": hi, + "pivot_geomean": pivot_geomean, + "significant_speedup": significant_speedup, + "significant_regression": significant_regression, + "accepted": significant_speedup, + "per_instance": per_instance, + } + + +def format_verdict(v, baseline_label, candidate_label): + lines = [] + lines.append(f"Baseline : {baseline_label}") + lines.append(f"Candidate: {candidate_label}") + lines.append(f"Instances: {v['n_instances']}") + lines.append("") + if not v["candidate_correct"]: + lines.append("CORRECTNESS: FAILED") + lines.append(" incorrect on: " + ", ".join(v["incorrect_instances"])) + lines.append("") + lines.append("VERDICT: REJECTED (incorrect)") + return "\n".join(lines) + + lines.append("CORRECTNESS: passed (matches GLPK+Gurobi on all instances)") + lines.append("") + pct = (v["time_geomean"] - 1) * 100 + lines.append(f"Time geomean speedup : {v['time_geomean']:.4f}x " + f"({pct:+.2f}%)") + lines.append(f" 95% bootstrap CI : [{v['time_ci_low']:.4f}, " + f"{v['time_ci_high']:.4f}]") + if v["pivot_geomean"] is not None: + ppct = (v["pivot_geomean"] - 1) * 100 + lines.append(f"Pivot geomean ratio : {v['pivot_geomean']:.4f}x " + f"({ppct:+.2f}% fewer pivots)") + lines.append("") + + # Worst and best individual instances (never summarize away the spread). + by_ratio = sorted(v["per_instance"], key=lambda r: r["time_ratio"]) + worst = by_ratio[0] + best = by_ratio[-1] + lines.append(f"Best instance: {best['instance']} " + f"{best['time_ratio']:.3f}x") + lines.append(f"Worst instance: {worst['instance']} " + f"{worst['time_ratio']:.3f}x") + lines.append("") + + if v["accepted"]: + lines.append("VERDICT: ACCEPTED " + "(correct and significantly faster: CI entirely above 1.0)") + elif v["significant_regression"]: + lines.append("VERDICT: REJECTED " + "(significant regression: CI entirely below 1.0)") + else: + lines.append("VERDICT: REJECTED " + "(no significant improvement: CI includes 1.0)") + return "\n".join(lines) diff --git a/bench/evaluate.py b/bench/evaluate.py new file mode 100644 index 0000000..5c5a3a1 --- /dev/null +++ b/bench/evaluate.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Compare two solver revisions (the loop entry point). + +Given two git refs -- a BASELINE (the revision to beat) and a CANDIDATE (the +proposed change) -- this answers: + + 1. Is the candidate CORRECT? (matches the GLPK+Gurobi oracle consensus) + 2. Is it a real, SIGNIFICANT IMPROVEMENT over the baseline? + +Both refs are checked out into their own throwaway git worktrees, built there +(`dune build`), and measured on the same held-out instances under identical +conditions. Neither the current working tree nor its `_build` is touched. + + uv run python evaluate.py --baseline HEAD --candidate my-optim-branch + +Both refs are REQUIRED -- there is no implicit default, so it is always explicit +which two revisions are being compared. + +The verdict is decided on the HELD-OUT split, which the proposer is told not to +inspect or tune against. Both splits are regenerated from fixed disjoint seeds on +every run, so they are reproducible yet mutually independent. + +Exit codes: + 0 candidate accepted (correct AND significantly faster than baseline) + 1 candidate rejected (incorrect, regression, or no significant change) + 2 usage / infrastructure error + +This module orchestrates the other bench modules by importing them directly +(no shelling out), so the whole harness is pure Python. +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from gen_instances import generate_split +from run_bench import run_corpus, _project_root +from compare import compare, format_verdict + +HERE = os.path.dirname(os.path.abspath(__file__)) +TRAIN_DIR = os.path.join(HERE, "instances", "train") +HELDOUT_DIR = os.path.join(HERE, "instances", "heldout") + + +def regenerate_splits(): + """Regenerate train + held-out corpora from their fixed seeds.""" + generate_split("train", TRAIN_DIR) + generate_split("heldout", HELDOUT_DIR) + + +def measure_ref(ref, side, rule, trials, warmup): + """Check out `ref` into a throwaway worktree, build it, and measure it. + + The ref is checked out with `git worktree add --detach`, the solver is built + there by run_corpus->build_solver (`dune build`, deps come from the shared + opam switch), and the corpus is run against it. The temp worktree and its + `_build` are removed afterwards, so nothing leaks and the caller's checkout is + untouched. `side` is a label ("baseline"/"candidate") for output. + """ + root = _project_root() + with tempfile.TemporaryDirectory(prefix="simplex-ref-") as tmp: + ref_root = os.path.join(tmp, "ref") + print(f"[{side}] checking out '{ref}' into a temp worktree...", + file=sys.stderr) + add = subprocess.run( + ["git", "worktree", "add", "--detach", ref_root, ref], + cwd=root, capture_output=True, text=True) + if add.returncode != 0: + raise RuntimeError( + f"git worktree add failed for ref '{ref}':\n{add.stderr}") + try: + print(f"[{side}] building and measuring '{ref}' (rule={rule})...", + file=sys.stderr) + return run_corpus(HELDOUT_DIR, rule, trials, f"{side}:{ref}", + root=ref_root, warmup=warmup) + finally: + # Drop the temp worktree registration (the dir is cleaned up by + # TemporaryDirectory; --force also drops the now-missing checkout). + subprocess.run(["git", "worktree", "remove", "--force", ref_root], + cwd=root, capture_output=True, text=True) + + +def do_compare(baseline_ref, candidate_ref, rule, trials, warmup): + """Compare two git refs: build+measure each in its own worktree, then judge.""" + try: + baseline = measure_ref(baseline_ref, "baseline", rule, trials, warmup) + candidate = measure_ref(candidate_ref, "candidate", rule, trials, warmup) + except RuntimeError as e: + print(str(e), file=sys.stderr) + return 2 + + verdict = compare(baseline, candidate) + print("", file=sys.stderr) + print(format_verdict(verdict, baseline_ref, candidate_ref)) + return 0 if verdict["accepted"] else 1 + + +def main(): + ap = argparse.ArgumentParser( + description="Compare two solver revisions (given as git refs) end-to-end.") + ap.add_argument("--baseline", required=True, + help="git ref for the baseline (the revision to beat)") + ap.add_argument("--candidate", required=True, + help="git ref for the candidate (the revision under test)") + ap.add_argument("--rule", default=os.environ.get("RULE", "bland"), + help="pivot rule under test (default: bland)") + ap.add_argument("--trials", type=int, + default=int(os.environ.get("TRIALS", "15")), + help="timed in-process re-solves per instance (default 15)") + ap.add_argument("--warmup", type=int, + default=int(os.environ.get("WARMUP", "3")), + help="untimed in-process warmup solves per instance " + "(default 3)") + args = ap.parse_args() + + # Always refresh the corpus so it is reproducible and never stale. + regenerate_splits() + + sys.exit(do_compare(args.baseline, args.candidate, args.rule, args.trials, + args.warmup)) + + +if __name__ == "__main__": + main() diff --git a/bench/gen_instances.py b/bench/gen_instances.py new file mode 100644 index 0000000..527389e --- /dev/null +++ b/bench/gen_instances.py @@ -0,0 +1,297 @@ +#!/usr/bin/env python3 +"""Generate benchmark instances for the simplex solver. + +Every instance is written in the solver's native `.in` format: + + n # number of variables + m # number of constraints + c_1 ... c_n # objective coefficients (MAXIMIZE c^T x) + b_1 ... b_m # constraint bounds + A_11 ... A_1n # constraint matrix, one row per constraint + ... + A_m1 ... A_mn + +The solved problem is max c^T x s.t. A x <= b, x >= 0. + +Design decisions (see bench/README.md for the rationale): + + * Multiple *families* of instances, not one. A single distribution (e.g. dense + uniform-random) lets an automated optimizer overfit to that distribution. + We include random-dense, structured-combinatorial, and adversarial families + so a change must generalize to be accepted. + + * Every family is *seeded*. Given the same seed an instance is bit-for-bit + reproducible, which is required both for a fair before/after comparison and + for the held-out set to be regenerable on demand. + + * Coefficients are integers or small rationals-as-integers so that the exact + rational solver and the floating-point oracles (GLPK, Gurobi) agree on the + optimum without rounding ambiguity. + +This module is used two ways: + * as a CLI (`python gen_instances.py --out `) to populate a corpus, + * imported, via `family_*` functions returning instance strings. +""" + +import argparse +import os +import random + + +# --------------------------------------------------------------------------- +# Serialization +# --------------------------------------------------------------------------- + +def format_instance(c, b, A): + """Serialize an LP into the solver's `.in` text format. + + c: length-n objective, b: length-m bounds, A: m x n matrix (lists of ints). + """ + n = len(c) + m = len(b) + assert all(len(row) == n for row in A), "A rows must have length n" + assert len(A) == m, "A must have m rows" + lines = [str(n), str(m)] + lines.append(" ".join(str(x) for x in c)) + lines.append(" ".join(str(x) for x in b)) + for row in A: + lines.append(" ".join(str(x) for x in row)) + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------- +# Instance families +# +# Each family is a function (rng, **params) -> instance string. Keeping the rng +# explicit (rather than the global `random`) is what makes seeding reliable. +# --------------------------------------------------------------------------- + +def family_random_dense(rng, n, m, lo=-9, hi=9, b_lo=1, b_hi=20): + """Dense LP with uniform-random integer coefficients. + + b is kept strictly positive so that x = 0 is always feasible; this guarantees + the problem is feasible and lets us focus the comparison on optimization work + rather than on phase-1 feasibility hunts. (A separate family exercises + infeasibility.) Cheap and controllable, but *not* realistic on its own -- a + known limitation, disclosed in the README. + """ + c = [rng.randint(1, hi) for _ in range(n)] # positive obj -> nontrivial optimum + b = [rng.randint(b_lo, b_hi) for _ in range(m)] + A = [[rng.randint(lo, hi) for _ in range(n)] for _ in range(m)] + # Ensure each constraint has at least one positive coeff so the feasible + # region is bounded in that direction (reduces the rate of unbounded LPs, + # which are trivial and uninteresting for timing). + for row in A: + if all(v <= 0 for v in row): + row[rng.randrange(n)] = rng.randint(1, hi) + return format_instance(c, b, A) + + +def family_assignment(rng, k): + """Balanced assignment LP: assign k agents to k tasks, maximize total value. + + Variables x_ij (k^2 of them). Constraints: each agent used <=1, each task + used <=1 (2k constraints). The LP relaxation is integral (assignment polytope), + so the optimum is a clean integer both solvers reproduce exactly. This is a + genuinely *structured*, real-world-shaped problem, unlike the random family. + """ + n = k * k + value = [[rng.randint(1, 20) for _ in range(k)] for _ in range(k)] + c = [value[i][j] for i in range(k) for j in range(k)] + + A = [] + b = [] + # each agent i assigned to at most one task: sum_j x_ij <= 1 + for i in range(k): + row = [0] * n + for j in range(k): + row[i * k + j] = 1 + A.append(row) + b.append(1) + # each task j taken by at most one agent: sum_i x_ij <= 1 + for j in range(k): + row = [0] * n + for i in range(k): + row[i * k + j] = 1 + A.append(row) + b.append(1) + return format_instance(c, b, A) + + +def family_bipartite_matching(rng, n_left, n_right, density): + """Max-weight fractional matching on a random bipartite graph. + + Variables = edges. Constraints = one per vertex (sum of incident edges <= 1). + Structured and sparse; the number of constraints (vertices) is decoupled from + the number of variables (edges), which stresses a different (n, m) regime than + the square assignment family. + """ + edges = [] + weights = [] + for u in range(n_left): + for v in range(n_right): + if rng.random() < density: + edges.append((u, v)) + weights.append(rng.randint(1, 20)) + if not edges: # guarantee a non-empty problem + edges.append((0, 0)) + weights.append(1) + n = len(edges) + c = weights + A = [] + b = [] + for u in range(n_left): + row = [1 if edges[e][0] == u else 0 for e in range(n)] + if any(row): + A.append(row) + b.append(1) + for v in range(n_right): + row = [1 if edges[e][1] == v else 0 for e in range(n)] + if any(row): + A.append(row) + b.append(1) + return format_instance(c, b, A) + + +def family_klee_minty(rng, d): + """Klee-Minty cube: the classic worst case for Dantzig's pivot rule. + + Standard integer form (Chvatal, Linear Programming, ch. 4; Klee & Minty 1972): + + max sum_{i=1..d} 10^{d-i} x_i + s.t. for i=1..d: 2 * sum_{j= 0 + + Under the *textbook* Dantzig rule from the origin this cube forces 2^d - 1 + pivots. NOTE: this solver's "max" rule pairs largest-objective-coefficient + entering with a lexicographic ratio-test leaving rule, and on this cube that + combination happens to jump to the apex in a single pivot -- so here it is + *not* a worst case for max. It is kept in the corpus for a different, still + valuable reason: the base-100 right-hand sides and factor-of-2 off-diagonals + produce large integer coefficients, which stresses the exact-rational + arithmetic (the very cost the project's Discussion.md hypothesizes about). + Real pivot-rule divergence in this corpus comes from the assignment, matching, + random, and degeneracy families; see bench/README.md. + + Exact-rational arithmetic makes coefficients grow with d, so d is kept small. + `rng` is unused (deterministic family) but kept for a uniform interface. + """ + c = [10 ** (d - i - 1) for i in range(d)] # i is 0-based here + A = [] + b = [] + for i in range(d): # constraint i, 0-based + row = [0] * d + for j in range(i): + row[j] = 2 * (10 ** (i - j)) + row[i] = 1 + A.append(row) + b.append(100 ** i) + return format_instance(c, b, A) + + +def family_degenerate(rng, n, extra): + """Highly degenerate LP: many constraints pass through the same vertex. + + Built by stacking `extra` redundant/near-redundant tight constraints at the + origin-adjacent vertex, which forces ratio-test ties and stresses anti-cycling + (Bland). Feasible at x = 0. Complements Klee-Minty with a *degeneracy* stress + rather than an *exponential-path* stress. + """ + c = [rng.randint(1, 9) for _ in range(n)] + A = [] + b = [] + # n bounding constraints + for i in range(n): + row = [0] * n + row[i] = 1 + A.append(row) + b.append(rng.randint(1, 5)) + # `extra` degenerate constraints that are tight at the same point (b=0 rows + # with mixed signs create ratio ties without making the LP infeasible). + for _ in range(extra): + row = [rng.choice([-1, 0, 1]) for _ in range(n)] + if all(v == 0 for v in row): + row[rng.randrange(n)] = 1 + A.append(row) + b.append(0) + return format_instance(c, b, A) + + +# --------------------------------------------------------------------------- +# Corpus definition +# +# A "split" is a list of (name, family_fn, params, seed). The train and heldout +# splits use DISJOINT seed ranges so that no heldout instance can coincide with a +# train instance. Sizes are chosen so the whole corpus runs in seconds-to-minutes +# on a laptop while still spanning small -> medium problems. +# --------------------------------------------------------------------------- + +def _split_spec(base_seed): + """Return the list of instances for a split, parameterized by a seed offset. + + base_seed=0 -> train, base_seed=1000 -> heldout (disjoint seeds). + """ + spec = [] + + # Random dense, a range of sizes (scaling behavior + bulk of the geomean). + sizes = [(5, 5), (8, 12), (12, 20), (16, 30), (20, 40), (25, 60)] + for k, (n, m) in enumerate(sizes): + spec.append((f"rand_{n}x{m}", family_random_dense, + dict(n=n, m=m), base_seed + k)) + + # Structured: assignment problems. + for k, size in enumerate([4, 5, 6, 7]): + spec.append((f"assign_{size}", family_assignment, + dict(k=size), base_seed + 100 + k)) + + # Structured: bipartite matching. + for k, (nl, nr) in enumerate([(6, 6), (8, 5), (10, 8)]): + spec.append((f"match_{nl}x{nr}", family_bipartite_matching, + dict(n_left=nl, n_right=nr, density=0.5), + base_seed + 200 + k)) + + # Adversarial: Klee-Minty (deterministic; seed unused but disjoint anyway). + for k, d in enumerate([4, 5, 6]): + spec.append((f"kleeminty_{d}", family_klee_minty, + dict(d=d), base_seed + 300 + k)) + + # Adversarial: degeneracy. + for k, (n, extra) in enumerate([(6, 6), (10, 10), (14, 14)]): + spec.append((f"degen_{n}", family_degenerate, + dict(n=n, extra=extra), base_seed + 400 + k)) + + return spec + + +SPLITS = { + "train": lambda: _split_spec(0), + "heldout": lambda: _split_spec(1000), +} + + +def generate_split(split, out_dir): + """Write every instance of `split` into out_dir as .in. Returns names.""" + os.makedirs(out_dir, exist_ok=True) + names = [] + for name, fn, params, seed in SPLITS[split](): + rng = random.Random(seed) + text = fn(rng, **params) + path = os.path.join(out_dir, f"{name}.in") + with open(path, "w") as f: + f.write(text) + names.append(name) + return names + + +def main(): + ap = argparse.ArgumentParser(description="Generate simplex benchmark instances.") + ap.add_argument("split", choices=list(SPLITS.keys())) + ap.add_argument("--out", required=True, help="output directory") + args = ap.parse_args() + names = generate_split(args.split, args.out) + print(f"wrote {len(names)} instances to {args.out}") + for n in names: + print(" " + n) + + +if __name__ == "__main__": + main() diff --git a/bench/oracle.py b/bench/oracle.py new file mode 100644 index 0000000..ae20914 --- /dev/null +++ b/bench/oracle.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +"""Independent ground-truth oracles for the simplex solver. + +The correctness gate must not be graded against the very solver being mutated +(the "evaluate against self" crime), and it must not be foolable by a proposer +that also edits the solver's own expected-output files. So we compute the true +optimum with two *external, independent* LP solvers and require our solver to +agree with them: + + * GLPK (glpsol 5.0) -- primary status oracle; clean infeasible/unbounded. + * Gurobi (gurobi_cl 12) -- numeric cross-check; disambiguated with + DualReductions=0 so it also separates the two + infeasible/unbounded cases. + +Each oracle returns a normalized result: + ("optimal", value_float) | ("infeasible", None) | ("unbounded", None) + +`solve` runs both and returns their consensus, raising if they disagree (which +would mean the benchmark itself is broken and no verdict can be trusted). +""" + +import os +import re +import subprocess +import sys +import tempfile + +from to_lp import to_lp + +# Objective values are compared with a relative tolerance because the oracles +# are floating point while our solver is exact rational. 1e-6 is far tighter +# than any real optimization signal but loose enough for FP round-off. +REL_TOL = 1e-6 +ABS_TOL = 1e-6 + + +def _close(a, b): + return abs(a - b) <= max(ABS_TOL, REL_TOL * max(abs(a), abs(b))) + + +# --------------------------------------------------------------------------- +# GLPK +# --------------------------------------------------------------------------- + +_GLPK_OBJ_RE = re.compile(r"Objective:\s+\S+\s*=\s*([-\d.eE+]+)") + + +def solve_glpk(lp_path): + with tempfile.NamedTemporaryFile("r", suffix=".sol", delete=False) as sol: + sol_path = sol.name + try: + proc = subprocess.run( + ["glpsol", "--lp", lp_path, "--max", "-o", sol_path], + capture_output=True, text=True, timeout=120, + ) + out = proc.stdout + proc.stderr + if "NO PRIMAL FEASIBLE SOLUTION" in out: + return ("infeasible", None) + if "NO DUAL FEASIBLE SOLUTION" in out: + return ("unbounded", None) + with open(sol_path) as f: + sol_text = f.read() + m = _GLPK_OBJ_RE.search(sol_text) + if m and "OPTIMAL" in sol_text: + return ("optimal", float(m.group(1))) + raise RuntimeError(f"glpsol: could not parse result\n{out}\n---\n{sol_text}") + finally: + if os.path.exists(sol_path): + os.unlink(sol_path) + + +# --------------------------------------------------------------------------- +# Gurobi +# --------------------------------------------------------------------------- + +_GRB_OBJ_RE = re.compile(r"Optimal objective\s+([-\d.eE+]+)") + + +def solve_gurobi(lp_path): + # LogFile="" stops gurobi_cl from dropping a gurobi.log in the cwd. + proc = subprocess.run( + ["gurobi_cl", "DualReductions=0", "LogFile=", lp_path], + capture_output=True, text=True, timeout=120, + ) + out = proc.stdout + proc.stderr + # Order matters: "Infeasible or unbounded" shouldn't occur with + # DualReductions=0, but check the specific words first. + if re.search(r"\bInfeasible model\b", out): + return ("infeasible", None) + if re.search(r"\bUnbounded model\b", out): + return ("unbounded", None) + m = _GRB_OBJ_RE.search(out) + if m: + return ("optimal", float(m.group(1))) + raise RuntimeError(f"gurobi_cl: could not parse result\n{out}") + + +# --------------------------------------------------------------------------- +# Consensus +# --------------------------------------------------------------------------- + +def solve(instance_path): + """Return the consensus oracle result for an `.in` instance. + + Raises RuntimeError if the two oracles disagree -- that means the benchmark + harness cannot be trusted for this instance, which is a louder and more + useful failure than silently picking one. + """ + with open(instance_path) as f: + lp = to_lp(f.read()) + with tempfile.NamedTemporaryFile("w", suffix=".lp", delete=False) as fh: + lp_path = fh.name + fh.write(lp) + try: + g = solve_glpk(lp_path) + r = solve_gurobi(lp_path) + finally: + os.unlink(lp_path) + + if g[0] != r[0]: + raise RuntimeError( + f"oracle status disagreement on {instance_path}: " + f"glpk={g}, gurobi={r}") + if g[0] == "optimal" and not _close(g[1], r[1]): + raise RuntimeError( + f"oracle value disagreement on {instance_path}: " + f"glpk={g[1]}, gurobi={r[1]}") + return g + + +def main(): + if len(sys.argv) < 2: + sys.exit("usage: oracle.py [instance.in ...]") + for path in sys.argv[1:]: + status, val = solve(path) + val_s = "" if val is None else f" {val}" + print(f"{os.path.basename(path)}\t{status}{val_s}") + + +if __name__ == "__main__": + main() diff --git a/bench/pyproject.toml b/bench/pyproject.toml new file mode 100644 index 0000000..7da4af6 --- /dev/null +++ b/bench/pyproject.toml @@ -0,0 +1,8 @@ +[project] +name = "bench" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = [ + "numpy>=2.5.0", + "scipy>=1.18.0", +] diff --git a/bench/run_bench.py b/bench/run_bench.py new file mode 100644 index 0000000..bc0b158 --- /dev/null +++ b/bench/run_bench.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +"""Run the simplex solver over a corpus, checking correctness and timing it. + +This is the measurement core of the optimization loop. For each instance it: + + 1. Runs the solver in `-json` metrics mode and parses the JSON object it emits + (status, objective, pivot_count, and the solver's own solve timing). + 2. CORRECTNESS GATE: compares status+objective against the external oracle + consensus (GLPK + Gurobi). A mismatch marks the instance as incorrect; + any incorrect instance fails the whole run. + +Timing comes from the solver itself: `-json --repeat N` re-solves the instance N +times in-process and reports the min and median solve time. Because the timing +loop is *inside* the program, it excludes process startup and input parsing -- +the ~4 ms floor that would otherwise swamp small instances -- so the reported +number is the intrinsic solve cost. We use the MINIMUM as the point estimate (for +a deterministic CPU-bound computation the minimum is the run least perturbed by +external interference) and keep the median alongside. + +Pivot count (also from the JSON) is deterministic, so it is a stability anchor: +if pivots are unchanged but time moved, the change is per-pivot cost (e.g. +rational blow-up), which we want to see, not hide. + +`run_corpus()` returns a report dict consumed by compare.py; it is imported by +evaluate.py, which is the harness's sole entry point. + +The solver binary is located by running `dune build` then using the known +`_build/default/bin/simplex.exe` path under the dune project root. +""" + +import json +import os +import subprocess +import sys + +from oracle import solve as oracle_solve, _close + + +def _project_root(): + """Directory containing dune-project (walk up from this file).""" + d = os.path.dirname(os.path.abspath(__file__)) + while d != "/": + if os.path.exists(os.path.join(d, "dune-project")): + return d + d = os.path.dirname(d) + raise RuntimeError("could not find dune-project above bench/") + + +def build_solver(root=None): + """Build the solver at `root` (default: this worktree) and return the exe. + + We pass `--root .` so the build is anchored at *that* dune-project and the exe + lands in its own `_build`, isolated from any other checkout. dune may exit + non-zero over a spurious alias-name warning from inside a git worktree while + still producing the exe, so we do not trust the exit code -- we check that the + exe exists. + """ + if root is None: + root = _project_root() + exe = os.path.join(root, "_build", "default", "bin", "simplex.exe") + proc = subprocess.run(["dune", "build", "--root", ".", "bin/simplex.exe"], + cwd=root, capture_output=True, text=True) + if not os.path.exists(exe): + raise RuntimeError( + f"build failed: {exe} not produced\n{proc.stdout}\n{proc.stderr}") + return exe + + +def require_json_support(exe, ref_label): + """Fail fast, with a clear message, if this solver lacks the -json flag. + + -json is the sole metric source; there is no fallback. A ref that predates the + -json commit builds a solver whose Arg.parse rejects the flag with an + "unknown option" message. Rather than let that surface later as an + unparseable-JSON error (or, worse, a bogus comparison), we detect it up front + and explain exactly what to do. + """ + proc = subprocess.run([exe, "-json", "--repeat", "1", "-q", os.devnull], + capture_output=True, text=True) + blob = proc.stdout + proc.stderr + if "unknown option" in blob and "-json" in blob: + raise RuntimeError( + f"ref '{ref_label}' has no -json support: its solver does not " + f"understand the -json flag, which is the benchmark's only metric " + f"source.\nFix: pick a baseline that already has -json, or rebase the " + f"-json commit onto '{ref_label}' first.") + + +def run_solver_json(exe, rule, instance, repeat, warmup): + """Run the solver in -json mode and return the parsed metrics dict. + + The solver runs `warmup` untimed solves, then `repeat` timed ones, and emits + one JSON object with status, objective (exact rational + float), pivots, and + min/median solve time over the timed samples. + """ + args = [exe, "-json", "--repeat", str(repeat), "--warmup", str(warmup), + "--rule", rule, instance] + proc = subprocess.run(args, capture_output=True, text=True, timeout=300) + out = proc.stdout.strip() + try: + return json.loads(out) + except json.JSONDecodeError as e: + raise RuntimeError( + f"could not parse solver JSON for {instance}:\n{out!r}\n" + f"stderr:\n{proc.stderr}") from e + + +# --------------------------------------------------------------------------- +# Per-instance measurement +# --------------------------------------------------------------------------- + +def measure_instance(exe, rule, instance, oracle_result, trials, warmup): + """Check correctness against the oracle and record the solver's own metrics. + + `trials`/`warmup` become the solver's --repeat/--warmup (its in-process timing + loop). Returns a dict with the correctness verdict and the solver's metrics. + """ + o_status, o_value = oracle_result + + m = run_solver_json(exe, rule, instance, trials, warmup) + s_status = m["status"] + s_value = m.get("value") # absent for infeasible/unbounded + + correct = (s_status == o_status) + if correct and o_status == "optimal": + correct = _close(s_value, o_value) + + return { + "instance": os.path.basename(instance), + "correct": correct, + "oracle_status": o_status, + "oracle_value": o_value, + "solver_status": s_status, + "solver_value": s_value, + "value_rat": m.get("value_rat"), + "pivots": m["pivots"], + "trials": m["trials"], + "time_min": m["time_min"], + "time_median": m["time_median"], + } + + +# --------------------------------------------------------------------------- +# Corpus driver +# --------------------------------------------------------------------------- + +def run_corpus(instances_dir, rule, trials, label, root=None, warmup=3): + """Build the solver (at `root`), then measure it over the corpus. + + `root` selects which checkout to build (defaults to this worktree); passing a + temporary git-worktree root lets a caller measure a different revision. + `trials`/`warmup` are the solver's in-process --repeat/--warmup counts. + """ + exe = build_solver(root) + require_json_support(exe, label) + instances = sorted( + os.path.join(instances_dir, f) + for f in os.listdir(instances_dir) if f.endswith(".in")) + if not instances: + raise RuntimeError(f"no .in instances in {instances_dir}") + + results = [] + all_correct = True + for inst in instances: + oracle_result = oracle_solve(inst) # (status, value) + r = measure_instance(exe, rule, inst, oracle_result, trials, warmup) + results.append(r) + if not r["correct"]: + all_correct = False + flag = "ok " if r["correct"] else "BAD" + print(f" [{flag}] {r['instance']:<16} " + f"pivots={str(r['pivots']):>5} " + f"t_min={r['time_min']*1e3:8.3f}ms " + f"t_med={r['time_median']*1e3:8.3f}ms", + file=sys.stderr) + + return { + "label": label, + "rule": rule, + "instances_dir": os.path.abspath(instances_dir), + "trials": trials, + "warmup": warmup, + "all_correct": all_correct, + "results": results, + } diff --git a/bench/to_lp.py b/bench/to_lp.py new file mode 100644 index 0000000..8a088fe --- /dev/null +++ b/bench/to_lp.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Convert a solver `.in` instance into CPLEX LP format. + +The CPLEX LP format is read by both external oracles (glpsol --lp, gurobi_cl), +so a single converter feeds both. We emit a MAXIMIZE objective to match the +solver's semantics (max c^T x s.t. A x <= b, x >= 0), which was verified against +both oracles on the course's subject instance. + +All variables are non-negative (x >= 0), matching the solver, so we do not emit a +bounds section (LP format defaults to [0, +inf)). +""" + +import sys + + +def parse_in(text): + """Parse the `.in` format into (n, m, c, b, A). + + Tokens are whitespace-separated and may span lines, but we follow the + documented line structure: line1=n, line2=m, line3=c, line4=b, then m rows. + Coefficients may be integers or `p/q` rationals (the generators emit ints, + but we accept rationals for robustness against hand-written instances). + """ + lines = [ln for ln in text.splitlines() if ln.strip() != ""] + n = int(lines[0]) + m = int(lines[1]) + c = lines[2].split() + b = lines[3].split() + A = [lines[4 + i].split() for i in range(m)] + assert len(c) == n, f"objective has {len(c)} coeffs, expected n={n}" + assert len(b) == m, f"bounds has {len(b)} values, expected m={m}" + for i, row in enumerate(A): + assert len(row) == n, f"constraint {i} has {len(row)} coeffs, expected n={n}" + return n, m, c, b, A + + +def _num(tok): + """Return a coefficient token as an exact number. + + LP format does not accept `p/q`, so a rational is converted to a float. + Integers stay ints to avoid any float rounding. + """ + if "/" in tok: + p, q = tok.split("/") + return int(p) / int(q) + return int(tok) + + +def _rhs(tok): + """Render a right-hand-side value (may be negative, printed as-is).""" + v = _num(tok) + return repr(v) if isinstance(v, float) else str(v) + + +def _linexpr(coeffs): + """Render `sum_j coeff_j * x_j` with signs folded into the operators. + + CPLEX LP format (as read by glpsol) rejects `+ -9 x2`; the sign must be a + standalone operator: `- 9 x2`. Zero coefficients are dropped. If every + coefficient is zero we emit `0 x0` so the expression is never empty. + """ + parts = [] + for j, tok in enumerate(coeffs): + v = _num(tok) + if v == 0: + continue + op = "-" if v < 0 else "+" + mag = -v if v < 0 else v + mag = repr(mag) if isinstance(mag, float) else str(mag) + parts.append(f"{op} {mag} x{j}") + if not parts: + return "0 x0" + # Drop a leading "+ " for cleanliness; keep a leading "- ". + expr = " ".join(parts) + if expr.startswith("+ "): + expr = expr[2:] + return expr + + +def to_lp(text): + n, m, c, b, A = parse_in(text) + out = [] + out.append("\\ auto-generated from .in by to_lp.py") + out.append("maximize") + out.append(" obj: " + _linexpr(c)) + out.append("subject to") + for i in range(m): + out.append(f" c{i}: " + _linexpr(A[i]) + f" <= {_rhs(b[i])}") + # x >= 0 is the LP-format default lower bound, so no bounds section needed. + out.append("end") + return "\n".join(out) + "\n" + + +def main(): + if len(sys.argv) != 2: + sys.exit("usage: to_lp.py (writes LP to stdout)") + with open(sys.argv[1]) as f: + sys.stdout.write(to_lp(f.read())) + + +if __name__ == "__main__": + main() diff --git a/bench/uv.lock b/bench/uv.lock new file mode 100644 index 0000000..6214611 --- /dev/null +++ b/bench/uv.lock @@ -0,0 +1,120 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" + +[[package]] +name = "bench" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "numpy" }, + { name = "scipy" }, +] + +[package.metadata] +requires-dist = [ + { name = "numpy", specifier = ">=2.5.0" }, + { name = "scipy", specifier = ">=1.18.0" }, +] + +[[package]] +name = "numpy" +version = "2.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/05/3d27272d30698dc0ecb7fdfaa41ad70303b444f81722bb99bce1d818638a/numpy-2.5.0.tar.gz", hash = "sha256:5a129578019311b6e56bdd714250f19b518f7dceeeb8d1af5490f4942d3f891c", size = 20652461, upload-time = "2026-06-21T20:57:51.95Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/0a/11486d02add7b1384dff7374d124b1cfbb0ee864dcc9f6a2c0380638cf84/numpy-2.5.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:489780423903667933b4ed6197b6ec3b75ea5dd17d1d8f0f38d798feb6921561", size = 16789987, upload-time = "2026-06-21T20:56:16.657Z" }, + { url = "https://files.pythonhosted.org/packages/55/b2/285f48640a181947b4587a3766d21ec1eaa7fea833d4b49957e09da467a2/numpy-2.5.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ece55976ced6bca95a03ae2839e2e5ccffe8eb6a3e7022415645eb154a81e4e6", size = 11760322, upload-time = "2026-06-21T20:56:19.813Z" }, + { url = "https://files.pythonhosted.org/packages/dd/67/b032db1eb03ca30d16eda3b0c22aaa615338b9263c2fd559d0f29451aca4/numpy-2.5.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:c83b664b0e6eee9594fa920cf0639d8af796606d3fad6cc70180c87e4b97c7be", size = 5319605, upload-time = "2026-06-21T20:56:22.173Z" }, + { url = "https://files.pythonhosted.org/packages/b9/83/03fc7300c7c6b6c84c487b1dc80d322817b95fbd1f4dd57a85e23b7198de/numpy-2.5.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:bf80333980bf37f523341ddd72c783f39d6829ec7736b9eb99086388a2d52cc2", size = 6653628, upload-time = "2026-06-21T20:56:23.914Z" }, + { url = "https://files.pythonhosted.org/packages/82/49/2ec21730bc63ccfda829323f7040a8ed4715b3852ce658689cf74ee96a8c/numpy-2.5.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a1a4874217b36d5ac8fc876f52e39df56f8182c88463e9e2dceabf7ca8b7efb8", size = 15153691, upload-time = "2026-06-21T20:56:25.631Z" }, + { url = "https://files.pythonhosted.org/packages/bb/6b/f4a3d0637692c49da8ef99d72d52526f92e0a8d6ac4f0ca9f31441b9d9ea/numpy-2.5.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:aaa760137137e8d3c920d27927748215b56014f92667dc9b6c27dfc61249255a", size = 16660066, upload-time = "2026-06-21T20:56:28.009Z" }, + { url = "https://files.pythonhosted.org/packages/3a/2f/c354ec86d1f3f5c19649463b0d39652e160736e5b0a4cd18dff0576715c4/numpy-2.5.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7174ce8265fc7f7417d171c9ea8fe905220748893ea67a2a7abe726ec331c4b0", size = 16514638, upload-time = "2026-06-21T20:56:30.26Z" }, + { url = "https://files.pythonhosted.org/packages/06/34/43efdcb319988648580f93c11f1ae82cf7e2faa74925e98e454ae3aa95f8/numpy-2.5.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b8c3daaf99de52415d20b42f8e8155c78642cb04207d02f9d317a0dcf1b3fb54", size = 18419647, upload-time = "2026-06-21T20:56:32.41Z" }, + { url = "https://files.pythonhosted.org/packages/71/e2/f5d1676b1d7fb682eb5e9a1641e7ebd2414b3216c370661d1029778908b4/numpy-2.5.0-cp312-cp312-win32.whl", hash = "sha256:6206db0af545d73d068add6d992279145f158428d1da6cc49adc4b630c5d6ee5", size = 6056688, upload-time = "2026-06-21T20:56:34.657Z" }, + { url = "https://files.pythonhosted.org/packages/8f/7c/48f115d1c58a34032facebcd51fdf2d02df2c51d4a46a81dd1197bb2ea6b/numpy-2.5.0-cp312-cp312-win_amd64.whl", hash = "sha256:6f2d6873e2940c860a309d21e25b1e69af6aaffdd80aa056b04c16380db1c4f2", size = 12419237, upload-time = "2026-06-21T20:56:36.24Z" }, + { url = "https://files.pythonhosted.org/packages/86/26/2e0882f4044d1b1a1b63e875151fb2393389032022a8b7f5657a7996d3b2/numpy-2.5.0-cp312-cp312-win_arm64.whl", hash = "sha256:a55e1eb2bca2cfd17a16b213c99dfc8502d47b0d494224d2122277d0400935ca", size = 10339912, upload-time = "2026-06-21T20:56:38.733Z" }, + { url = "https://files.pythonhosted.org/packages/8a/33/07675aaad7f26ea013d5e884d9a0d784b79c6bd7566c333f5a52fa3c610b/numpy-2.5.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:520e6b8be0a4b65840ac8090d4f51cef4bed66e2b0894d5a520f099adc24a9b2", size = 16784890, upload-time = "2026-06-21T20:56:40.799Z" }, + { url = "https://files.pythonhosted.org/packages/85/4b/953118a730ee3b35e28645e0eb4cf9beec5bdbb954e1ac2f5fcefba6bbc3/numpy-2.5.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:146b81cdd3967fdb6beca8ba25f00c58741d8f3cbd797f55af0fbe0bfec3469c", size = 11754584, upload-time = "2026-06-21T20:56:43.094Z" }, + { url = "https://files.pythonhosted.org/packages/44/9b/56dd530c367c74ae17411027cea4135ca57e1e0583bf5594cee18bd83217/numpy-2.5.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:126b88d95e8ff9b00c9e717aa540469f21d6180162f84c0caec51b16215d49cd", size = 5313904, upload-time = "2026-06-21T20:56:45.503Z" }, + { url = "https://files.pythonhosted.org/packages/ce/b0/bcd672edad27ecca7da1f7bb0ce72cd1706a4f2d79ae94990afc97c13e1c/numpy-2.5.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:d4313cef1594c5ce46c31b6e54e918338f63f16ee9322304e8c9114d6d81c8bd", size = 6648504, upload-time = "2026-06-21T20:56:47.567Z" }, + { url = "https://files.pythonhosted.org/packages/80/9e/15cdfcbd30a1544a46c9e487a00df331c4672450216538705a9e51fa6710/numpy-2.5.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:750fb097caf26fa878746d9d119f6f9da12dedcbff1eea966c3e3447647c4a9e", size = 15150086, upload-time = "2026-06-21T20:56:49.352Z" }, + { url = "https://files.pythonhosted.org/packages/32/4e/8d7656ccaab3e81e97258b8a9bc5f0c8502513a92fb4ceb0a2cbfebc17bf/numpy-2.5.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3893adc2dc7c0412ba76777db55a049215d99c9aa3113003be8f49f4f1290ab9", size = 16647250, upload-time = "2026-06-21T20:56:51.542Z" }, + { url = "https://files.pythonhosted.org/packages/3c/81/97060281b602ed07f21b12f4ec409eac1f75a2f91fbc829ed8b2becf3ad4/numpy-2.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:835e454dd99b238cdc5a3f63bce2371296f5ebc53ca1e0f8e6ddbb6d92a29aab", size = 16512864, upload-time = "2026-06-21T20:56:55.401Z" }, + { url = "https://files.pythonhosted.org/packages/33/ab/4496208146911f8d8ddb54f68a972aafa6c8d44babcb2ea03b0e5cc87c9d/numpy-2.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6f9836778081a0a3c02a6a21493f3e9f5b311f8d2541934f31f05583dc999ea4", size = 18408407, upload-time = "2026-06-21T20:56:57.75Z" }, + { url = "https://files.pythonhosted.org/packages/d4/9f/a4df67c181e4ee8b467aa3332dc2db10fd5c515136831302f3ca48bc0a01/numpy-2.5.0-cp313-cp313-win32.whl", hash = "sha256:0b525be4744b60bb0557ac872d53ef07d085b5f39622bc579c98d3809d05b988", size = 6054431, upload-time = "2026-06-21T20:57:00.016Z" }, + { url = "https://files.pythonhosted.org/packages/30/53/491e1c47c55b62ccc6a63c1c5b8635c73fc2258dddeb9bda27cae4a0ae96/numpy-2.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:44353e2878930039db472b99dc353d749826e4010bd4d2a7f835e94a97a5c748", size = 12414420, upload-time = "2026-06-21T20:57:01.815Z" }, + { url = "https://files.pythonhosted.org/packages/eb/4a/25c2906f541e9d9f4c5769764db732e6627be91a13f4724fa10634d77db4/numpy-2.5.0-cp313-cp313-win_arm64.whl", hash = "sha256:48f54b00711f83a5f796b70c518e8c2b3c5848dda03a54911f23eb68519b9b60", size = 10339533, upload-time = "2026-06-21T20:57:03.961Z" }, + { url = "https://files.pythonhosted.org/packages/86/ad/abc44aaceaf7b17ee1edde2bbb4458da591bc79574cffff50c4bb35f00d1/numpy-2.5.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f27582c55ba4c750b7c58c8faf021d2cd9324a662b466229db8a417b41368af9", size = 16783807, upload-time = "2026-06-21T20:57:06.253Z" }, + { url = "https://files.pythonhosted.org/packages/5d/39/b72e168daf9c00fb20c9fc996d00437ccecdef3102387775d29d7a62576d/numpy-2.5.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:28e7137057d551e4a83c4ae414e3451f50568409db7569aacc7f9811ee06a446", size = 11765215, upload-time = "2026-06-21T20:57:08.547Z" }, + { url = "https://files.pythonhosted.org/packages/f7/a0/8400a9c0e3625182347593f5e1f57da9a617a534794805c8df5518154ddc/numpy-2.5.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:e1da54b53e75cd9fcfc23efcc7edab2c6aecf97b6037566d8a0fe804af8ec57c", size = 5324493, upload-time = "2026-06-21T20:57:11.012Z" }, + { url = "https://files.pythonhosted.org/packages/f6/8c/0d104deaa0401c93395a629ec902891618a2eff76d19229139cb5a887bfc/numpy-2.5.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:694d8f74e156f7fd01179f1aa8faa2f648ab6ae0f70b6c3fe57a03249aea2303", size = 6645211, upload-time = "2026-06-21T20:57:12.919Z" }, + { url = "https://files.pythonhosted.org/packages/6a/d9/4a4a628c812750363786afc3d33492709a5cd64b215469c16b0f6c7bb811/numpy-2.5.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1a7569a7b53c77716f036bb28cb1c91f166a26ec7d9502cd1e4bdfe502fdec22", size = 15166004, upload-time = "2026-06-21T20:57:14.717Z" }, + { url = "https://files.pythonhosted.org/packages/a0/5e/2a902317d7fc4aa93236e80c932662dadfc459b323d758329e01775125e1/numpy-2.5.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:39a0433bd4086ebd462960cf375e19195bb07b53dc1d87dd5fcf47ad78576f03", size = 16650797, upload-time = "2026-06-21T20:57:16.906Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a0/a0090e6329f4ca5992c07847bb579c5259a19953dc57255bb08793142ffb/numpy-2.5.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:929f0c79ac38bcbd7154fe631dc907abfeddbcc5027a896bd1f7767323271e7a", size = 16524647, upload-time = "2026-06-21T20:57:19.165Z" }, + { url = "https://files.pythonhosted.org/packages/5e/7d/6caf27734c42b65837e7461ed0dbbd6b6fc835060c9714ec59d673bb383a/numpy-2.5.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:cc4f247a47bbf070bfd70be53ccdcf47b800af563535e7bbe172322197c30e21", size = 18411841, upload-time = "2026-06-21T20:57:21.638Z" }, + { url = "https://files.pythonhosted.org/packages/13/dc/26edadbd812536769a82c2e9e002234e33feb5da43061d47a044f6d309b7/numpy-2.5.0-cp314-cp314-win32.whl", hash = "sha256:5dc71423499fab3f46f7a7201155ade1669ea101f2f429d332df9e72f8161731", size = 6106361, upload-time = "2026-06-21T20:57:23.844Z" }, + { url = "https://files.pythonhosted.org/packages/f2/9e/4dd1459282229a72d92dece2ae9138e5cac94a72263a7ceb48f37434c925/numpy-2.5.0-cp314-cp314-win_amd64.whl", hash = "sha256:ebb81d9d5443e0309d6c54894c3fbed74ad7da0714352a67b6d773cd189eae73", size = 12551749, upload-time = "2026-06-21T20:57:25.945Z" }, + { url = "https://files.pythonhosted.org/packages/05/a7/6bc6384c080b86c7f6c85c5bc5b540b24f4f679cd144791d99574e90d462/numpy-2.5.0-cp314-cp314-win_arm64.whl", hash = "sha256:3b94d0d0deceebfad3e67ae5c0e5eb87371e8f7a0581cd04a779928c2450cf1e", size = 10617072, upload-time = "2026-06-21T20:57:28.175Z" }, + { url = "https://files.pythonhosted.org/packages/86/6b/4a2b71d66ada5608ae02b63f150dfad520f6940721cb7f029ad270befc0e/numpy-2.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:22f3d43e362d650bc39db1f17851302874a148ca95ba6981c1dfb5fa6862f35b", size = 11881067, upload-time = "2026-06-21T20:57:30.104Z" }, + { url = "https://files.pythonhosted.org/packages/dc/b2/d365eb40a20efb49d67e9feb90494ed8511282ee1f5fa16006675c65397d/numpy-2.5.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:243563efb4cd7528a264567e9fd206c87826457322521d06206a00bfa316c927", size = 5440290, upload-time = "2026-06-21T20:57:32.193Z" }, + { url = "https://files.pythonhosted.org/packages/fa/5e/e9c03188de5f9b767e46a8fe988bcfd3efad066a4a3fda8b9cb11a93f895/numpy-2.5.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:84881d825ca75249b189bbee875fcfe3238aa5c479e6100893cda566e8e86826", size = 6748371, upload-time = "2026-06-21T20:57:33.933Z" }, + { url = "https://files.pythonhosted.org/packages/fd/1d/68c186a38a5027bae2c4ddd5ea681fdaf8b4d30fb7301def6d8ad270390f/numpy-2.5.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cda12aa4779d42b8771180aba759c96f527d43446d8f380ab59e2b35e8489efd", size = 15214643, upload-time = "2026-06-21T20:57:35.677Z" }, + { url = "https://files.pythonhosted.org/packages/8c/67/73f67b7c7e20635baae9c4c3ead4ae7326a005900297a6110971abd62eb5/numpy-2.5.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c0121101093d2bd74981b10f8837d78e794a8ff57834eb27179f49e1ba11ac6", size = 16690128, upload-time = "2026-06-21T20:57:38.159Z" }, + { url = "https://files.pythonhosted.org/packages/eb/05/d4c1fb0c46d02a27d6b2b8b319a78c90937acec8631c1641874670b31e6f/numpy-2.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d371c92cfa09da00022f501ab67fafaea813d752eb30ac44336d45b1e5b0268a", size = 16577902, upload-time = "2026-06-21T20:57:40.447Z" }, + { url = "https://files.pythonhosted.org/packages/9e/1d/771c797d50fa26e4888989cccf1d50ee51f530d4e455ad2692dcb64fa711/numpy-2.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9990713e9c38154c6861e7547f1e3fc7a87e75ff09bab24ef1cc81d81c2835e9", size = 18452814, upload-time = "2026-06-21T20:57:42.875Z" }, + { url = "https://files.pythonhosted.org/packages/e8/46/52fc0d2a68d7643f0f149eeea5a5d8ea2a3507056ac8afa83c9212606e8b/numpy-2.5.0-cp314-cp314t-win32.whl", hash = "sha256:edadfbd4794b1086c0d822f81863e8a68fc129d132fd0bb9e31e955d7fbbbdb7", size = 6253168, upload-time = "2026-06-21T20:57:45.101Z" }, + { url = "https://files.pythonhosted.org/packages/2a/be/6c8d1118b5f13b2881dc095d5b345de19c6638b8959c17409b6eff84c8aa/numpy-2.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:f7e5fa4382967ae6548bd2f174219afb908e294b0d5f625af01166edd5f7d9aa", size = 12736286, upload-time = "2026-06-21T20:57:46.935Z" }, + { url = "https://files.pythonhosted.org/packages/fd/6a/d3a169aaf8536cf228d56a09e04bcb713a2fe4410d4e2105b9419b5a9c89/numpy-2.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:016623417bb330d719d579daf2d6b9a01ddc52e41a9ed61a47f39fde46dcd865", size = 10686451, upload-time = "2026-06-21T20:57:49.313Z" }, +] + +[[package]] +name = "scipy" +version = "1.18.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a7/25/c2700dfaf6442b4effaa91af24ebce5dc9d31bb4a69706313aae70d72cd0/scipy-1.18.0.tar.gz", hash = "sha256:67b2ad2ad54c72ca6d04975a9b2df8c3638c34ddd5b28738e94fc2b57929d378", size = 30774447, upload-time = "2026-06-19T15:01:43.456Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/19/ca10ead60b0acc80b2b833c2c4a4f2ff753d0f58b811f70d911c7e94a25c/scipy-1.18.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:7bd21faaf5a1a3b2eff922d02db5f191b99a6518db9078a8fb23169f6d22259a", size = 31056519, upload-time = "2026-06-19T14:59:45.203Z" }, + { url = "https://files.pythonhosted.org/packages/96/72/1e6442a00cd2924d361aa1b642ab6373ec35c6fabf311a760be9f76e0f13/scipy-1.18.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:265915e79107de9f946b855e50d7470d5893ec3f54b342e1aa6201cbdcd8bb6b", size = 28681889, upload-time = "2026-06-19T14:59:48.103Z" }, + { url = "https://files.pythonhosted.org/packages/9b/2d/11dd93d21e147a73ba22bd75c0b9208d3a2e0ec76d53170ce7d9029b1015/scipy-1.18.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:9ab7b758be6940954a713ee466e2043e9f6e2ed965c1fce5c91039f4be3d90a9", size = 20423580, upload-time = "2026-06-19T14:59:50.665Z" }, + { url = "https://files.pythonhosted.org/packages/9c/01/93552f75e0d2a7dd115a45e59209c51e8d514daff02fc887d2623be06fe1/scipy-1.18.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:97b6cddaaee0a779ef6b5ca83c9604b27cc16b2b8fc22c142652df8793319fb8", size = 23054441, upload-time = "2026-06-19T14:59:53.564Z" }, + { url = "https://files.pythonhosted.org/packages/3c/23/21f5e703643d66f21faa6b4c73195bfcad70c55efcb4f1ab327cd7c4101a/scipy-1.18.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:52a96e21517c7292375c0e27dd796a811f03fcea5fd4d108fdfea8145dcf17ab", size = 33968720, upload-time = "2026-06-19T14:59:56.415Z" }, + { url = "https://files.pythonhosted.org/packages/dd/aa/1b939f6c67ed68635bb538e6752d3dacc02f66535182e939a89581a44e9c/scipy-1.18.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f55797419e16e7f30cf88ffb3113ce0467f00cfe3f70d5c281730b21769bfc2", size = 35287115, upload-time = "2026-06-19T14:59:59.411Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ff/eec46be7e9234208f801062b53e1983085eddebd693f6c9bfb03b459830d/scipy-1.18.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ad033410e2e0672ffdc1042110cef20e1c46f8fd0616cee1d44d8d58fad8fc11", size = 35577989, upload-time = "2026-06-19T15:00:02.235Z" }, + { url = "https://files.pythonhosted.org/packages/84/ca/210d4759c7210bb7d269437421959b39a33434e2776b60c5cb8a763bb30a/scipy-1.18.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4a55985d54c769c872e64b7f4c8a81cc30ef700cc04296abbbf3705439c126de", size = 37421717, upload-time = "2026-06-19T15:00:05.102Z" }, + { url = "https://files.pythonhosted.org/packages/2b/54/9a9edb45345bd6744da5ddfb6628e5d5185920494c6a67ec45b6381004cb/scipy-1.18.0-cp312-cp312-win_amd64.whl", hash = "sha256:71ccc8faa2dd16ac310233203474a8b5cb67f10dedd54a3116d34943f4b19132", size = 36597428, upload-time = "2026-06-19T15:00:08.112Z" }, + { url = "https://files.pythonhosted.org/packages/99/0e/33f32a2a58987e26aec0f7df252cbbad1e90ae77bdbc76f40dd4ed0cf0ea/scipy-1.18.0-cp312-cp312-win_arm64.whl", hash = "sha256:d88363fd9d8fbd3511bd273f1a49efb2a540773ddf92a91d57498ce7dd7f3e76", size = 24351481, upload-time = "2026-06-19T15:00:11.103Z" }, + { url = "https://files.pythonhosted.org/packages/05/52/9c0136c2de7ae0779b7b366447766cec6d9f0702c56bb8ffeb04c8fd3af4/scipy-1.18.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:09143f676d157d9f546d663504ef9c1becb819824f1afc018814176411942446", size = 31036107, upload-time = "2026-06-19T15:00:14.03Z" }, + { url = "https://files.pythonhosted.org/packages/02/73/0291a64843270f4efb86cdcf2ee0f2048631b65ec6b405398b2b4dbf11bf/scipy-1.18.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5efe260f69417b97ddae455bfb5a95e8359f7f66ad7fa9522a60feb66f169520", size = 28663303, upload-time = "2026-06-19T15:00:16.819Z" }, + { url = "https://files.pythonhosted.org/packages/d3/0f/10ffa0b697a572f4e0d48b92a88895d366422f019f723e7e14a84c050dac/scipy-1.18.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:68363b7eaacd8b5dd426df56d782cc156468ac79a127a1b87ca597d6e2e82197", size = 20404960, upload-time = "2026-06-19T15:00:19.635Z" }, + { url = "https://files.pythonhosted.org/packages/7e/d2/e896cea21ba8edd6c81d4c55b1ffcc717e79698dcbebf9641b4cfb4c6622/scipy-1.18.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:c5557d8be5da8e41353fcd4d21491fdbab83b062fc579e94dc09a7c8ab4f669b", size = 23034074, upload-time = "2026-06-19T15:00:22.107Z" }, + { url = "https://files.pythonhosted.org/packages/ea/b2/e83ea34279a52c03374477c74006256ec78df65fc877baa4617d6de1d202/scipy-1.18.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0d13bca67c096d89fb95ced0d8921807300fce0275643aef9533cc63a0773468", size = 33942038, upload-time = "2026-06-19T15:00:24.964Z" }, + { url = "https://files.pythonhosted.org/packages/f6/af/e8fe5fb136f51e2b01678b92cb4106d10d8cd68ec147ead2e7cb0ac75398/scipy-1.18.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a46f9273dbd0eb1cefba61c9b8648b4dfe3cbc14a080176f9a73e44b8336dc7f", size = 35266390, upload-time = "2026-06-19T15:00:28.059Z" }, + { url = "https://files.pythonhosted.org/packages/3a/49/2c5cbb907b56695fc67517811d1db234dfd83381a84814ec220aded2794d/scipy-1.18.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5aba46108853ddfc77906b6557aac839d2b52e900c1d72a1180adaaab58d265f", size = 35551324, upload-time = "2026-06-19T15:00:31.014Z" }, + { url = "https://files.pythonhosted.org/packages/bb/73/eda39f7a2d306ff0ffc574afd13c0bbb6d10a603d9a413998ee269487a80/scipy-1.18.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b6f758e35f12757b5d95c00bc6de2438e229c2664b7a92e96f205959d9f2dfa4", size = 37404785, upload-time = "2026-06-19T15:00:34.072Z" }, + { url = "https://files.pythonhosted.org/packages/b7/d2/ae881ee28d014f38e0ccbfd974a06a919ba9af34f1f74bf42b5301891d63/scipy-1.18.0-cp313-cp313-win_amd64.whl", hash = "sha256:1afac4a847207c7ff8efd321734a50b06d0280b3b2a2c0fc2f413101747ad7c7", size = 36554943, upload-time = "2026-06-19T15:00:36.903Z" }, + { url = "https://files.pythonhosted.org/packages/70/3a/21154e2d54eb3639c6bf4dbae2e531c68356bfe95990daa30df33b30d556/scipy-1.18.0-cp313-cp313-win_arm64.whl", hash = "sha256:c5dbddf60e58c2312316d097271a8e73d40eaf2eabfa4d95ed7d3695bbf2ce7b", size = 24350911, upload-time = "2026-06-19T15:00:40.062Z" }, + { url = "https://files.pythonhosted.org/packages/78/b5/915a19b3de2f7430062b509653563db1633ddbb6f021b06731521115d4e2/scipy-1.18.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:4c256ee70c0d1a8a2ace807e199ccd4e3f57037433842abb3fb36bc17eaa9578", size = 31036253, upload-time = "2026-06-19T15:00:43.216Z" }, + { url = "https://files.pythonhosted.org/packages/d7/88/b72def7262e150d16be13fca37a96481138d624e700340bc3362a7588929/scipy-1.18.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:2ef3abc54a4ffc53765374b0d5728532dfdd2585ed23f6b11c206a1f0b1b9af8", size = 28673758, upload-time = "2026-06-19T15:00:46.663Z" }, + { url = "https://files.pythonhosted.org/packages/91/02/2e636a61a525632c373cf6a9c24442a3ffb79e364d38e98b32042964ac32/scipy-1.18.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f2a6af57bd9e4a75d70e4117e78a1bbee84f79ae3fbb6d0111005d6ebcc4cb8d", size = 20415514, upload-time = "2026-06-19T15:00:49.399Z" }, + { url = "https://files.pythonhosted.org/packages/c9/b6/2135974442f6aba159d9d39d774a1c8cb19947016725d69fecc685df45bf/scipy-1.18.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:3f1ac564d3bf6c03d861d2cd87a1bea0da2887136f7fb1bf519c05a8971452d6", size = 23034398, upload-time = "2026-06-19T15:00:51.941Z" }, + { url = "https://files.pythonhosted.org/packages/f6/e6/ba89ec5abf6ee9257c0d1ec985573f3ae32742c24bc03e016388a40b1b15/scipy-1.18.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40395a5fcd1abee49a5c7aaa98c29db393eedc835138560a588c47ec16156690", size = 33998032, upload-time = "2026-06-19T15:00:54.838Z" }, + { url = "https://files.pythonhosted.org/packages/7f/c4/bc41eb19b0fd0db868f4132920879019318d80cc522ad8f2bca4611af808/scipy-1.18.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ca01e8ae69f1b18e9a58d91afead31be3cef0dd905a10249dac559ee15460a0", size = 35283333, upload-time = "2026-06-19T15:00:58.152Z" }, + { url = "https://files.pythonhosted.org/packages/53/a4/cbdeef6eb3830a8462a9d4ada814de5fc984345cc9ecf17cbec51a036f1e/scipy-1.18.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7a7f3b01647384dbc3a711e8c6778e0aabbe93959249fef5c7393396bcac0867", size = 35610216, upload-time = "2026-06-19T15:01:01.155Z" }, + { url = "https://files.pythonhosted.org/packages/80/4d/b2b82502b65f661d1b789c1665dcdf315d5f12194e06fc0b37946294ebae/scipy-1.18.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6aa94e78ec192a30063a5e72e561c28af769dc311190b24fe91774eff1969709", size = 37418960, upload-time = "2026-06-19T15:01:04.155Z" }, + { url = "https://files.pythonhosted.org/packages/93/3e/902d836831474b0ab5a37d16404f7bc5fafd9efba632890e271ba952635f/scipy-1.18.0-cp314-cp314-win_amd64.whl", hash = "sha256:2d8bbdc6c817f5b4006a54d799d4f5bab6f910193cbb9a1ff310833d4d270f61", size = 37288845, upload-time = "2026-06-19T15:01:07.822Z" }, + { url = "https://files.pythonhosted.org/packages/b6/43/8d73b337a3bdb14daa0314f0434210747c02d79d729ce1777574a817dcf6/scipy-1.18.0-cp314-cp314-win_arm64.whl", hash = "sha256:18e9575f1569b2c54174e6159d32942e03731177f63dce7975f0a0c88d102f5b", size = 24988971, upload-time = "2026-06-19T15:01:11.076Z" }, + { url = "https://files.pythonhosted.org/packages/b4/b4/f11918b0508a2787031a0499a03fbe3546f3bb5ca05d01038c45b278c09a/scipy-1.18.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f351e0dd702687d12a402b867a1b4146a256923e1c38317cbc472f6372b94707", size = 31399325, upload-time = "2026-06-19T15:01:13.723Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d1/1f287b57c0ff0ee5185dff3946d92c8017d39b0e431f0ae79a3ff1859512/scipy-1.18.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7c7a51b33ce387193c97f228320cf8e87361daa1bba750638677729598b3e677", size = 29092110, upload-time = "2026-06-19T15:01:16.908Z" }, + { url = "https://files.pythonhosted.org/packages/ff/1a/7b74eb6c392fdcb27d414c0e7558a6d0231eb3b6d73571f479bb81ea8794/scipy-1.18.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:84031d7b052a54fae2f8632e0ec802073d385476eb9a63079bce6e23ef9283d4", size = 20833811, upload-time = "2026-06-19T15:01:20.488Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ad/f3941716320a7b9cb4d68734a903b45fe16eff5fb7da7e16f2e619304979/scipy-1.18.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:56abf29a7c067dde59be8b9a22d606a4ea1b2f2a4b756d9d903c62818f5dacce", size = 23396644, upload-time = "2026-06-19T15:01:23.364Z" }, + { url = "https://files.pythonhosted.org/packages/22/22/1446b62ffe07f9719b7d9b1b6a4e05a772833ae8f441fe4c22c34c9b250f/scipy-1.18.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ad44305cfa24b1ba5803cbbebf033590ccbac1aa5d612d727b785325ab408b0", size = 34079318, upload-time = "2026-06-19T15:01:26.002Z" }, + { url = "https://files.pythonhosted.org/packages/56/3b/b87da667098bb470fa30c7011b0ba351ee976dd395c78798c66e941665a3/scipy-1.18.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:945c1761b93f38d7f99ae81ae80c63e621471608c7eeead563f6df025585cd58", size = 35324320, upload-time = "2026-06-19T15:01:28.881Z" }, + { url = "https://files.pythonhosted.org/packages/f8/a1/c7932f91909759b0267f75fdea34e91309f96b895757534b76a90b6b4344/scipy-1.18.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:1a4441f15d620578772a49e5ab48c0ee1f7a0220e387110283062729136b2553", size = 35699541, upload-time = "2026-06-19T15:01:31.968Z" }, + { url = "https://files.pythonhosted.org/packages/f7/86/5185061a1fcc41d18c5dc2463969b3a3964b31d9ac67b2fb05d4c7ff7670/scipy-1.18.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9aac6192fac56bf2ca534389d24623f07b39ff83317d58287285e7fbd622ff76", size = 37472480, upload-time = "2026-06-19T15:01:35.136Z" }, + { url = "https://files.pythonhosted.org/packages/31/8e/f04c68e39919a010d34f2ee1367fd705b0a25a02f609d755f0bfbc0a15fc/scipy-1.18.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e40baea28ae7f5475c779741e2d90b1247c78531207b49c7030e698ff81cee3f", size = 37365390, upload-time = "2026-06-19T15:01:38.091Z" }, + { url = "https://files.pythonhosted.org/packages/d5/19/969dc072906c84dd0a3b05dcf57ea750936087d7873549e408b35cfc3f97/scipy-1.18.0-cp314-cp314t-win_arm64.whl", hash = "sha256:368e0a705903c466aa5f08eefb39e6b1b6b2d659e7352a31fd9e2438365be0f8", size = 25279661, upload-time = "2026-06-19T15:01:40.817Z" }, +] diff --git a/bin/simplex.ml b/bin/simplex.ml index e4edee6..0bb8e40 100644 --- a/bin/simplex.ml +++ b/bin/simplex.ml @@ -45,24 +45,96 @@ let output res = if !Params.debug then output_debug res else if !Params.verbose then output_verbose res else if !Params.time then () + else if !Params.json then () else if !Params.quiet then output_quiet res else output_normal res +(* JSON string escaping is trivial here since every value we emit is a number, a + bare status word, or a rational like "51457/1485" -- none contain quotes or + backslashes -- so a plain quote wrap is sufficient. *) +let json_status = function + | Finished _ -> "optimal" + | Unbounded -> "unbounded" + | Unfeasible -> "infeasible" + | Paused -> "paused" + +(* OCaml's string_of_float renders e.g. 40. and 8e-06, both invalid JSON. Use a + round-trippable %h-free format: %.17g gives full double precision, and we + ensure a decimal point / exponent is always present so the token is a valid + JSON number. *) +let json_float f = Printf.sprintf "%.17g" f + +(* Emit the metrics as one JSON object on a single line. `times` are the + per-iteration solve durations (seconds); we report their count, min and + median so the consumer sees the distribution, not just a point. *) +let output_json ~n ~m ~res ~pivots ~times = + let sorted = List.sort compare times in + let arr = Array.of_list sorted in + let k = Array.length arr in + let tmin = if k = 0 then 0. else arr.(0) in + let tmedian = if k = 0 then 0. else arr.(k / 2) in + let status = json_status res in + let value_fields = + match res with + | Finished (_, v) -> + Printf.sprintf {|, "value_rat": "%s", "value": %s|} + (Q.to_string v) + (json_float (Q.to_float v)) + | _ -> "" + in + Printf.printf + {|{"status": "%s", "rule": "%s", "n": %d, "m": %d, "pivots": %d, "trials": %d, "time_min": %s, "time_median": %s%s}|} + status !Params.rule n m pivots k + (json_float tmin) (json_float tmedian) value_fields; + print_newline () + let solve_file file_name = Params.handle (); - let tb = parse_file file_name in - reset Params.nb_pivots; - let st = Sys.time () in - if not !Params.quiet then print_tableau tb; - let res = do_simplex tb in - let et = Sys.time () in - output res; - if !Params.time then - let n = get_n tb in - let m = get_m tb in - print_endline - (string_of_int n ^ " " ^ string_of_int m ^ " " - ^ string_of_float (et -. st)) + if !Params.json then ( + (* Metrics mode: re-parse + solve `repeat` times, timing each solve in + isolation (parse excluded), so the reported time is pure solve cost. + `warmup` untimed solves run first so the timed samples reflect steady + state (cache/allocator warmed, first-touch effects excluded). *) + let repeat = max 1 !Params.repeat in + let warmup = max 0 !Params.warmup in + for _ = 1 to warmup do + let tb = parse_file file_name in + reset Params.nb_pivots; + ignore (do_simplex tb) + done; + let times = ref [] in + let last = ref None in + let n = ref 0 and m = ref 0 and pivots = ref 0 in + for _ = 1 to repeat do + let tb = parse_file file_name in + reset Params.nb_pivots; + let st = Sys.time () in + let res = do_simplex tb in + let et = Sys.time () in + times := (et -. st) :: !times; + last := Some res; + n := get_n tb; + m := get_m tb; + pivots := !Params.nb_pivots + done; + match !last with + | Some res -> output_json ~n:!n ~m:!m ~res ~pivots:!pivots ~times:!times + | None -> assert false) + else begin + let tb = parse_file file_name in + reset Params.nb_pivots; + let st = Sys.time () in + if not !Params.quiet then print_tableau tb; + let res = do_simplex tb in + let et = Sys.time () in + output res; + if !Params.time then + let n = get_n tb in + let m = get_m tb in + print_endline + (string_of_int n ^ " " ^ string_of_int m ^ " " + ^ string_of_float (et -. st)) + end let speclist = let open Arg in @@ -73,6 +145,18 @@ let speclist = ("--rule", Set_string Params.rule, "What rule is to be used"); ("-ez", Set Params.ez, "easy printing of tableau"); ("-t", Set Params.time, "Show timing insted of calcul result. Imply quiet"); + ( "-json", + Set Params.json, + "Emit metrics (status, objective, pivots, timing) as JSON. Implies quiet." + ); + ( "--repeat", + Set_int Params.repeat, + "In -json mode, re-solve this many times in-process for timing (default 1)" + ); + ( "--warmup", + Set_int Params.warmup, + "In -json mode, untimed solves to run before the timed ones (default 0)" + ); ] let usage_msg = "Simplex " diff --git a/solver/params.ml b/solver/params.ml index d0dc7a0..fc7c5ce 100644 --- a/solver/params.ml +++ b/solver/params.ml @@ -7,7 +7,15 @@ let time = ref false let ez = ref false let nb_pivots = ref 0 +(* Machine-readable metrics mode: emit a single JSON object at the end and + suppress all human-facing output. `repeat` re-runs the solve in-process so the + reported time excludes process startup + parsing overhead. `warmup` untimed + solves run first so the timed samples reflect steady state. *) +let json = ref false +let repeat = ref 1 +let warmup = ref 0 + let handle () = verbose := !verbose || !debug; - quiet := !quiet || !time; + quiet := !quiet || !time || !json; rule := String.lowercase_ascii !rule