diff --git a/README.md b/README.md index 2c42c78..b2446a0 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,18 @@ See the [full documentation](https://hydrosolutions.github.io/ctrl-freak/) for A ## Benchmarks -See [benchmarks/](benchmarks/) for the comparison suite. +ctrl-freak ships a validation benchmark suite that checks `ga()` and `nsga2()` +against pymoo and DEAP on standard problems with known optima. With the genetic +algorithm held identical across all three libraries (ported SBX, aligned mutation +and selection, identical evaluation budget), ctrl-freak's results are statistically +indistinguishable from both baselines on the single-objective error metrics (all +six functions) and on multi-objective convergence (ZDT1, ZDT2, ZDT3, and DTLZ2). On +the two hardest problems (ZDT4, ZDT6) none of the three libraries converges at this +budget, and ctrl-freak is at least as good as both. The goal is parity, not +superiority. + +See the canonical report in [benchmarks/README.md](benchmarks/README.md) and the +citable [Validation page](https://hydrosolutions.github.io/ctrl-freak/validation/). ## Links diff --git a/benchmarks/README.md b/benchmarks/README.md index dda1295..20fa6c2 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,68 +1,256 @@ -# NSGA-II Benchmarks +# Validation Benchmark Suite -Benchmark suite comparing ctrl-freak against Pymoo and DEAP on standard multi-objective optimization test problems. +This suite validates ctrl-freak's `ga()` (single-objective) and `nsga2()` +(multi-objective) against [pymoo](https://pymoo.org) and +[DEAP](https://deap.readthedocs.io) on standard problems with **known analytical +optima**. Its purpose is scientific trust, not marketing: the claim is **parity** — +*across these problems, ctrl-freak's results are statistically indistinguishable +from pymoo and DEAP* — so a manuscript that calibrates parameters with ctrl-freak +has a citable basis for the optimizer. We never claim ctrl-freak is *better*. -## Test Problems +Every table and figure on this page is regenerated from the committed results +artifact `results/benchmark_results.json` by `render.py`; the numbers trace to that +artifact. -### ZDT Suite +## Provenance -The ZDT (Zitzler-Deb-Thiele) test problems are standard benchmarks for multi-objective evolutionary algorithms: + -| Problem | Pareto Front | Characteristics | -|---------|--------------|-----------------| -| ZDT1 | Convex | Continuous, 30 variables | -| ZDT2 | Concave | Non-convex front, 30 variables | -| ZDT3 | Discontinuous | Multiple disconnected segments | +| Field | Value | +|---|---| +| Seeds | 30 (0–29) | +| ctrl-freak | 0.1.0 | +| pymoo | 0.6.1.6 | +| deap | 1.4.3 | +| numpy | 2.4.1 | +| SO budget | pop 100 × 200 gen (20100 evals) | +| MO budget | pop 100 × 250 gen (25100 evals) | +| SBX / PM eta | 15.0 / 20.0 | -All problems have: + -- Decision variables in [0, 1] -- Two objectives to minimize -- Known analytical Pareto-optimal fronts +## Problems + +| Single-objective (10-D, f\* = 0) | Multi-objective | +|---|---| +| Sphere — separable, unimodal | ZDT1 — convex front (30 vars) | +| Rosenbrock — curved valley | ZDT2 — concave front (30 vars) | +| Rastrigin — highly multimodal | ZDT3 — discontinuous front (30 vars) | +| Ackley — multimodal, one funnel | ZDT4 — convex front, multimodal space (30 vars) | +| Griewank — coupled product term | ZDT6 — biased, concave front (30 vars) | +| Schwefel — deceptive global basin | DTLZ2 — 3-objective unit-sphere octant (12 vars) | + +Single-objective problems are 10-D (multimodality bites at 10-D while a real-coded +GA can still reach the optimum, keeping the metrics informative). The +multi-objective problems carry analytical true-front samplers used as the IGD+ / GD +reference. ## Methodology -### Algorithm Configuration +**Identical algorithm, only the library differs.** Fairness is load-bearing: any +residual difference must come from the algorithm, not the configuration. So all +three libraries run ctrl-freak's exact operator stack: + +- **Crossover — ported, identical across all three.** ctrl-freak's single-child, + every-eligible-variable SBX (eta 15) is ported into a custom pymoo `Crossover` + and a custom DEAP mate; only each framework's RNG source differs. (pymoo's stock + `SBX` and DEAP's `cxSimulatedBinaryBounded` recombine only ~half the variables by + default, which alone put ctrl-freak ~70 % above both baselines on convex-ZDT + IGD+ — a configuration artifact, now eliminated.) +- **Mutation — aligned.** Polynomial mutation (eta 20) applied to every offspring + at a per-variable rate `1/n_vars` (`PM(prob=1.0, prob_var=1/n_var)` for pymoo, + `mutPolynomialBounded(indpb=1/n_var)` for DEAP). +- **Selection / survival — aligned.** Binary tournament; elitist(1) survival for + SO, NSGA-II (non-dominated sort + crowding) for MO. +- **Budget — identical.** SO: population 100 × 200 generations (20 100 + evaluations). MO: population 100 × 250 generations (25 100 evaluations). The + equivalence claim is void otherwise. + +**Statistic — overlapping variance.** Parity is an *equivalence* claim, not a +difference test (a non-significant t-test is not evidence of equivalence). For each +problem and metric, ctrl-freak and a baseline are **equivalent** when +`|mean_cf − mean_lib| < max(std_cf, std_lib)` — the cross-library gap is smaller +than the seed-to-seed spread. Each table cell pairs the per-library mean ± std with +the verdict and its margin against pymoo and DEAP. Thirty seeds. + +## Single-objective results + +Single-objective parity is led by the continuous error metrics — the absolute +objective error |f − f*| and the distance to the known optimum ‖x − x*‖ — which are +statistically indistinguishable from both baselines across all six problems +(including the multimodal Rastrigin, Ackley, and Schwefel). + +### Objective error |f − f\*| + + + +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| sphere | 4.61e-03 ± 1.99e-03 | 3.68e-03 ± 1.39e-03 | 3.63e-03 ± 1.86e-03 | equivalent (margin +1.1e-03) | equivalent (margin +1.0e-03) | +| rosenbrock | 6.58e+00 ± 1.65e+00 | 6.90e+00 ± 1.39e+00 | 6.72e+00 ± 1.46e+00 | equivalent (margin +1.3e+00) | equivalent (margin +1.5e+00) | +| rastrigin | 1.84e+00 ± 8.87e-01 | 1.65e+00 ± 9.06e-01 | 1.99e+00 ± 1.10e+00 | equivalent (margin +7.1e-01) | equivalent (margin +9.4e-01) | +| ackley | 7.10e-01 ± 2.29e-01 | 7.12e-01 ± 2.32e-01 | 8.67e-01 ± 3.01e-01 | equivalent (margin +2.3e-01) | equivalent (margin +1.4e-01) | +| griewank | 7.11e-01 ± 1.18e-01 | 7.09e-01 ± 1.19e-01 | 7.10e-01 ± 1.02e-01 | equivalent (margin +1.2e-01) | equivalent (margin +1.2e-01) | +| schwefel | 7.04e+01 ± 7.27e+01 | 3.84e+01 ± 5.47e+01 | 6.62e+01 ± 8.51e+01 | equivalent (margin +4.1e+01) | equivalent (margin +8.1e+01) | + + + +### Solution distance ‖x − x\*‖ + + + +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| sphere | 6.63e-02 ± 1.45e-02 | 5.96e-02 ± 1.12e-02 | 5.84e-02 ± 1.48e-02 | equivalent (margin +7.8e-03) | equivalent (margin +6.8e-03) | +| rosenbrock | 2.66e+00 ± 5.18e-01 | 2.75e+00 ± 2.80e-01 | 2.71e+00 ± 3.19e-01 | equivalent (margin +4.2e-01) | equivalent (margin +4.7e-01) | +| rastrigin | 6.79e-01 ± 5.17e-01 | 6.18e-01 ± 5.86e-01 | 8.53e-01 ± 6.04e-01 | equivalent (margin +5.2e-01) | equivalent (margin +4.3e-01) | +| ackley | 2.77e-01 ± 6.79e-02 | 2.77e-01 ± 6.97e-02 | 3.20e-01 ± 8.57e-02 | equivalent (margin +6.9e-02) | equivalent (margin +4.3e-02) | +| griewank | 2.63e+01 ± 7.37e+00 | 2.85e+01 ± 8.17e+00 | 2.64e+01 ± 5.95e+00 | equivalent (margin +6.0e+00) | equivalent (margin +7.3e+00) | +| schwefel | 3.84e+02 ± 3.86e+02 | 2.20e+02 ± 3.29e+02 | 3.44e+02 ± 4.01e+02 | equivalent (margin +2.2e+02) | equivalent (margin +3.6e+02) | + + + +### Success rate (fraction of seeds reaching f − f\* < ε) + +The success thresholds ε are strict (e.g. Sphere ε = 1e-6 while all three libraries +reach ~4e-3 at this budget), so the binary success rate is **`0` for every problem +and every library**. This is a documented strict-threshold non-convergence property +at this budget — not a failure and not a parity violation: the three libraries +reach the same error band (the continuous metrics above), and that is where the +single-objective parity evidence rests. The metric is reported faithfully rather +than retuned. + + + +| Problem | ctrl-freak | pymoo | deap | +|---|---|---|---| +| sphere | 0.00 | 0.00 | 0.00 | +| rosenbrock | 0.00 | 0.00 | 0.00 | +| rastrigin | 0.00 | 0.00 | 0.00 | +| ackley | 0.00 | 0.00 | 0.00 | +| griewank | 0.00 | 0.00 | 0.00 | +| schwefel | 0.00 | 0.00 | 0.00 | + + + +### Convergence -All libraries configured with identical parameters: +![Single-objective convergence: best objective error per generation, seed-mean ± std](results/figures/so_convergence.png) -| Parameter | Value | -|-----------|-------| -| Population size | 100 | -| Generations | 250 | -| Decision variables | 30 | -| Bounds | [0, 1] | -| Crossover | SBX (eta = 15) | -| Mutation | Polynomial (eta = 20, prob = 1/30) | -| Runs per config | 10 (seeds 0-9) | +## Multi-objective results -### Metrics +The primary multi-objective metrics are IGD+ and GD against the analytical true +front (convergence distance); hypervolume is kept as a secondary column. +Convergence is statistically indistinguishable from both baselines on ZDT1, ZDT2, +ZDT3, and DTLZ2 — on all three metrics (IGD+, GD, and hypervolume). -- **Hypervolume (HV)**: Volume of objective space dominated by the Pareto front approximation (higher is better) -- **Reference point**: [1.1, 1.1] for all ZDT problems -- **Time**: Wall-clock seconds per run +### IGD+ (to the analytical front) -## Results + -Results are produced by running `benchmarks/zdt/run_benchmark.py`. Regenerate benchmark results after operator changes, including crossover or mutation updates, before publishing or comparing numbers. +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| zdt1 | 5.85e-03 ± 3.75e-04 | 6.07e-03 ± 3.63e-04 | 5.94e-03 ± 4.22e-04 | equivalent (margin +1.5e-04) | equivalent (margin +3.3e-04) | +| zdt2 | 5.46e-03 ± 4.38e-04 | 5.87e-03 ± 3.77e-04 | 5.69e-03 ± 4.77e-04 | equivalent (margin +2.7e-05) | equivalent (margin +2.4e-04) | +| zdt3 | 5.91e-03 ± 1.02e-02 | 3.17e-03 ± 2.63e-04 | 3.14e-03 ± 2.22e-04 | equivalent (margin +7.4e-03) | equivalent (margin +7.4e-03) | +| zdt4 | 8.73e+00 ± 2.26e+00 | 1.59e+01 ± 4.68e+00 | 1.48e+01 ± 4.73e+00 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| zdt6 | 3.81e-01 ± 3.88e-02 | 5.08e-01 ± 6.77e-02 | 5.25e-01 ± 6.04e-02 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| dtlz2 | 5.47e-02 ± 3.67e-03 | 5.43e-02 ± 3.61e-03 | 5.47e-02 ± 3.28e-03 | equivalent (margin +3.3e-03) | equivalent (margin +3.6e-03) | -### Visual Comparison + -ZDT3 Pareto Front Comparison +### GD (to the analytical front) -*ZDT3 discontinuous Pareto front: ctrl-freak and DEAP closely track the true front, while Pymoo shows visible scatter.* + -## How to Run +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| zdt1 | 6.62e-03 ± 1.48e-03 | 7.10e-03 ± 1.18e-03 | 6.65e-03 ± 1.73e-03 | equivalent (margin +9.9e-04) | equivalent (margin +1.7e-03) | +| zdt2 | 7.11e-03 ± 3.07e-03 | 9.07e-03 ± 3.15e-03 | 8.66e-03 ± 3.68e-03 | equivalent (margin +1.2e-03) | equivalent (margin +2.1e-03) | +| zdt3 | 5.09e-03 ± 9.83e-04 | 5.68e-03 ± 1.18e-03 | 6.25e-03 ± 1.89e-03 | equivalent (margin +6.0e-04) | equivalent (margin +7.3e-04) | +| zdt4 | 1.04e+01 ± 2.89e+00 | 2.08e+01 ± 5.47e+00 | 1.97e+01 ± 5.79e+00 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| zdt6 | 4.23e-01 ± 4.09e-02 | 5.75e-01 ± 7.77e-02 | 5.95e-01 ± 7.27e-02 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| dtlz2 | 5.50e-02 ± 1.25e-02 | 5.66e-02 ± 1.33e-02 | 6.06e-02 ± 1.81e-02 | equivalent (margin +1.2e-02) | equivalent (margin +1.3e-02) | + + + +### Hypervolume (secondary; reference point [1.1, …]) + + + +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| zdt1 | 8.65e-01 ± 6.89e-04 | 8.65e-01 ± 6.63e-04 | 8.65e-01 ± 7.49e-04 | equivalent (margin +3.2e-04) | equivalent (margin +6.0e-04) | +| zdt2 | 5.32e-01 ± 9.01e-04 | 5.31e-01 ± 7.96e-04 | 5.31e-01 ± 1.02e-03 | equivalent (margin +8.8e-06) | equivalent (margin +4.5e-04) | +| zdt3 | 1.32e+00 ± 2.04e-02 | 1.32e+00 ± 1.05e-03 | 1.32e+00 ± 9.44e-04 | equivalent (margin +1.5e-02) | equivalent (margin +1.5e-02) | +| zdt4 | 0.00e+00 ± 0.00e+00 | 0.00e+00 ± 0.00e+00 | 0.00e+00 ± 0.00e+00 | not equivalent — identical (degenerate) | not equivalent — identical (degenerate) | +| zdt6 | 1.14e-01 ± 2.26e-02 | 5.34e-02 ± 2.29e-02 | 4.61e-02 ± 2.25e-02 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| dtlz2 | 6.55e-01 ± 1.09e-02 | 6.55e-01 ± 1.00e-02 | 6.55e-01 ± 9.13e-03 | equivalent (margin +1.0e-02) | equivalent (margin +1.0e-02) | + + + +### Two documented exceptions + +- **ZDT4 and ZDT6.** None of the three libraries converges on these hardest + multimodal / biased problems at this budget. The symmetric overlapping-variance + test flags the gap, but **in ctrl-freak's favour**: ctrl-freak's IGD+, GD, and + hypervolume are at least as good as both pymoo and DEAP (lower IGD+/GD; ZDT6 + higher HV). The trust claim holds (ctrl-freak is at-least-as-good); the clean + equivalence statement simply does not apply where no library reaches the front. + ZDT4's hypervolume is identically `0` for all three because every final front + lies outside the `[1.1, 1.1]` reference box. +- These are reported, not hidden: the committed JSON carries the full + per-problem × metric verdict and margin behind every cell above. + +### Convergence and fronts + +![Multi-objective convergence: IGD+ per generation, seed-mean ± std](results/figures/mo_convergence.png) + +![Final non-dominated fronts vs the analytical true fronts](results/figures/mo_pareto_fronts.png) + +The ZDT panels overlay the three libraries' final fronts on the analytical true +front; ZDT1/ZDT2/ZDT3 land on it for all three libraries, while ZDT4/ZDT6 sit off +it for all three (the documented exceptions). The DTLZ2 (3-objective) panel shows +ctrl-freak's final front against the unit-sphere octant; per-library DTLZ2 parity +is quantified in the IGD+/GD/HV tables and the convergence panel. + +> **On the earlier "ZDT3 scatter" note.** A previous version of this report +> captioned a ZDT3 figure as pymoo showing "visible scatter". That was a +> configuration artifact, not an algorithm difference: the legacy harness used +> `PM(prob=1/n_var)`, which mutated far fewer genes (~0.03 vs ~1.0 expected gene +> flips per offspring at `n_var = 30`, a **~30× under-mutation**) and slowed +> pymoo's convergence. With the correctly aligned +> `PM(prob=1.0, prob_var=1/n_var)` used here, the scatter disappears and all three +> libraries converge to nearly identical fronts on the standard ZDT/DTLZ2 problems. + +## Reproducing + +The committed JSON and every table and figure are regenerated by one command: ```bash -# Run full benchmark suite -uv run python benchmarks/zdt/run_benchmark.py +uv run python benchmarks/reproduce.py +``` -# Results saved to benchmarks/zdt/results/benchmark_results.json +This runs the full sweep (`benchmarks/run.py` → `results/benchmark_results.json`, +~20–35 min) and then renders the tables and figures (`benchmarks/render.py`, +seconds). To re-render the tables and figures from an existing JSON without +re-running the sweep: + +```bash +uv run python benchmarks/render.py ``` -## References +Library versions are pinned via `uv.lock` and re-embedded in the JSON metadata +(see Provenance). -1. Zitzler, E., Deb, K., & Thiele, L. (2000). Comparison of multiobjective evolutionary algorithms: Empirical results. *Evolutionary computation*, 8(2), 173-195. +## References -2. Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. A. M. T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. *IEEE transactions on evolutionary computation*, 6(2), 182-197. +1. Zitzler, E., Deb, K., & Thiele, L. (2000). Comparison of multiobjective + evolutionary algorithms: Empirical results. *Evolutionary Computation*, 8(2), + 173–195. +2. Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist + multiobjective genetic algorithm: NSGA-II. *IEEE Transactions on Evolutionary + Computation*, 6(2), 182–197. +3. Deb, K., Thiele, L., Laumanns, M., & Zitzler, E. (2005). Scalable test problems + for evolutionary multiobjective optimization (DTLZ). In *Evolutionary + Multiobjective Optimization*, 105–145. diff --git a/benchmarks/render.py b/benchmarks/render.py new file mode 100644 index 0000000..5ebb97d --- /dev/null +++ b/benchmarks/render.py @@ -0,0 +1,646 @@ +"""Render the committed benchmark results into report tables and figures. + +This is the s6 reporting step's renderer. It reads the committed validation +artifact ``benchmarks/results/benchmark_results.json`` (produced by +:mod:`benchmarks.run`) and emits the two human-facing products the pydrology +manuscript cites: + +* **Markdown tables** -- per-problem mean +/- std for every metric, each cell + paired with the overlapping-variance equivalence verdict and margin. The tables + are injected *in place* between ```` / ```` marker + comments in ``benchmarks/README.md`` and ``docs/validation.md``, so the prose is + hand-written but the numbers always trace to the committed JSON. +* **Figures** -- SO and MO convergence curves (seed-mean line + a +/-std band, one + panel per problem, three libraries) and a final-front Pareto scatter against the + analytical true fronts. Each figure is written to ``benchmarks/results/figures/`` + (for the GitHub-rendered report) and ``docs/assets/validation/`` (so + ``mkdocs build --strict`` resolves the image links from inside ``docs/``). + +The renderer never re-runs the sweep and never rewrites the JSON; it is a pure +JSON-to-artifacts pass. Run it via :mod:`benchmarks.reproduce` (sweep + render) or +directly:: + + uv run python benchmarks/render.py + +Examples +-------- +>>> from benchmarks.render import _fmt +>>> _fmt(0.00321, 0.000386) +'3.21e-03 ± 3.86e-04' +""" + +# ruff: noqa: E402 (sys.path bootstrap + matplotlib backend must precede imports) + +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") + +import matplotlib.pyplot as plt +import numpy as np + +# Make ``import benchmarks`` work under a plain ``python benchmarks/render.py``. +_REPO_ROOT = Path(__file__).resolve().parent.parent +if str(_REPO_ROOT) not in sys.path: + sys.path.insert(0, str(_REPO_ROOT)) + +from benchmarks.problems.multi_objective import MO_PROBLEMS + +LIBRARIES: tuple[str, ...] = ("ctrl-freak", "pymoo", "deap") +"""Reference library first, then the two baselines (display order).""" +BASELINES: tuple[str, ...] = ("pymoo", "deap") +"""Libraries ctrl-freak is adjudicated against.""" +SO_ORDER: tuple[str, ...] = ("sphere", "rosenbrock", "rastrigin", "ackley", "griewank", "schwefel") +"""Single-objective problems in display order.""" +MO_ORDER: tuple[str, ...] = ("zdt1", "zdt2", "zdt3", "zdt4", "zdt6", "dtlz2") +"""Multi-objective problems in display order (DTLZ2 last: the only 3-objective one).""" + +_REPORT_CLAIMS: dict[str, dict[str, tuple[str, ...]]] = { + # table_id -> {claim -> problems}. Encodes exactly the parity the report prose + # asserts, so the renderer fails fast if a committed verdict contradicts it. + "so_objective_error": {"indistinguishable": SO_ORDER}, + "so_solution_distance": {"indistinguishable": SO_ORDER}, + "mo_igd_plus": {"indistinguishable": ("zdt1", "zdt2", "zdt3", "dtlz2"), "at_least_as_good": ("zdt4", "zdt6")}, + "mo_gd": {"indistinguishable": ("zdt1", "zdt2", "zdt3", "dtlz2"), "at_least_as_good": ("zdt4", "zdt6")}, + "mo_hypervolume": { + "indistinguishable": ("zdt1", "zdt2", "zdt3", "dtlz2"), + "at_least_as_good": ("zdt4", "zdt6"), + }, +} +"""Parity claims the report prose makes, keyed by the table that backs each. + +``indistinguishable`` problems must render an ``equivalent`` verdict against both +baselines; ``at_least_as_good`` problems (the documented ZDT4/ZDT6 exceptions) must +never render ``ctrl-freak worse``. :func:`check_report_consistency` enforces this. +""" + +_BENCH_DIR = Path(__file__).resolve().parent +RESULTS_PATH = _BENCH_DIR / "results" / "benchmark_results.json" +FIGURES_DIR = _BENCH_DIR / "results" / "figures" +README_PATH = _BENCH_DIR / "README.md" +DOCS_FIGURES_DIR = _REPO_ROOT / "docs" / "assets" / "validation" +VALIDATION_PATH = _REPO_ROOT / "docs" / "validation.md" +_TINY = 1e-12 # log-scale floor for the lower edge of a +/-std band + + +def _fmt(mean: float, std: float) -> str: + """Format a ``mean +/- std`` pair in two-significant-figure scientific form. + + Parameters + ---------- + mean, std : float + The aggregate mean and population standard deviation of one metric. + + Returns + ------- + str + e.g. ``'3.21e-03 ± 3.86e-04'``. + + Examples + -------- + >>> _fmt(1.0, 0.5) + '1.00e+00 ± 5.00e-01' + """ + return f"{mean:.2e} ± {std:.2e}" + + +def _verdict_cell(verdict: dict, lower_is_better: bool) -> str: + """Render one overlapping-variance verdict as a compact table cell. + + Parameters + ---------- + verdict : dict + An entry of the JSON ``equivalence`` section (keys ``equivalent``, + ``margin``, ``delta_mean``, ``mean_reference``, ``mean_candidate``). + lower_is_better : bool + ``True`` for error metrics (objective_error, solution_distance, IGD+, GD) + where smaller is better; ``False`` for hypervolume. + + Returns + ------- + str + ``'equivalent (margin +M)'`` when the spreads overlap; otherwise the + direction relative to ctrl-freak (``'... ctrl-freak better/worse'``), or + ``'... identical (degenerate)'`` for the zero-gap/zero-variance case + (e.g. ZDT4 hypervolume ``0/0/0``). + + Examples + -------- + >>> _verdict_cell({"equivalent": True, "margin": 0.0123}, lower_is_better=True) + 'equivalent (margin +1.2e-02)' + >>> _verdict_cell( + ... {"equivalent": False, "delta_mean": 8.0, "mean_reference": 8.7, "mean_candidate": 19.2}, + ... lower_is_better=True, + ... ) + 'not equivalent — ctrl-freak better' + >>> _verdict_cell({"equivalent": False, "delta_mean": 0.0}, lower_is_better=False) + 'not equivalent — identical (degenerate)' + """ + if verdict["equivalent"]: + return f"equivalent (margin +{verdict['margin']:.1e})" + if verdict["delta_mean"] == 0.0: + return "not equivalent — identical (degenerate)" + better = ( + verdict["mean_reference"] < verdict["mean_candidate"] + if lower_is_better + else verdict["mean_reference"] > verdict["mean_candidate"] + ) + return "not equivalent — ctrl-freak " + ("better" if better else "worse") + + +def _metric_table(section: dict, order: tuple[str, ...], metric: str, lower_is_better: bool) -> str: + """Build a per-problem table for one metric (mean +/- std per library + verdicts). + + Parameters + ---------- + section : dict + One modality block (``results["single_objective"]`` or + ``results["multi_objective"]``); read for ``aggregated`` and + ``equivalence``. + order : tuple of str + Problem display order. + metric : str + Metric key (e.g. ``"objective_error"``, ``"igd_plus"``). + lower_is_better : bool + Passed to :func:`_verdict_cell`. + + Returns + ------- + str + A GitHub/MkDocs markdown table. + + Examples + -------- + >>> section = { + ... "aggregated": {"p": { + ... "ctrl-freak": {"m": {"mean": 1.0, "std": 0.1}}, + ... "pymoo": {"m": {"mean": 1.1, "std": 0.1}}, + ... "deap": {"m": {"mean": 1.2, "std": 0.1}}, + ... }}, + ... "equivalence": {"p": {"m": { + ... "pymoo": {"equivalent": True, "margin": 0.05}, + ... "deap": {"equivalent": True, "margin": 0.02}, + ... }}}, + ... } + >>> print(_metric_table(section, ("p",), "m", True)) # doctest: +ELLIPSIS + | Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | + |---|---|---|---|---|---| + | p | 1.00e+00 ± 1.00e-01 | ... | equivalent (margin +5.0e-02) | equivalent (margin +2.0e-02) | + """ + agg = section["aggregated"] + eq = section["equivalence"] + lines = [ + "| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap |", + "|---|---|---|---|---|---|", + ] + for p in order: + cells = [p] + for lib in LIBRARIES: + m = agg[p][lib][metric] + cells.append(_fmt(m["mean"], m["std"])) + for b in BASELINES: + cells.append(_verdict_cell(eq[p][metric][b], lower_is_better)) + lines.append("| " + " | ".join(cells) + " |") + return "\n".join(lines) + + +def _success_table(section: dict, order: tuple[str, ...]) -> str: + """Build the SO success-rate table (one library column each). + + Parameters + ---------- + section : dict + ``results["single_objective"]``. + order : tuple of str + Problem display order. + + Returns + ------- + str + A markdown table of per-library success rates. + + Examples + -------- + >>> section = {"aggregated": {"p": { + ... "ctrl-freak": {"success_rate": 0.0}, + ... "pymoo": {"success_rate": 0.0}, + ... "deap": {"success_rate": 0.0}, + ... }}} + >>> print(_success_table(section, ("p",))) + | Problem | ctrl-freak | pymoo | deap | + |---|---|---|---| + | p | 0.00 | 0.00 | 0.00 | + """ + agg = section["aggregated"] + lines = ["| Problem | ctrl-freak | pymoo | deap |", "|---|---|---|---|"] + for p in order: + cells = [p] + [f"{agg[p][lib]['success_rate']:.2f}" for lib in LIBRARIES] + lines.append("| " + " | ".join(cells) + " |") + return "\n".join(lines) + + +def _provenance_table(meta: dict) -> str: + """Build the provenance table (pinned versions, budgets, seeds) from metadata. + + Parameters + ---------- + meta : dict + The ``metadata`` block of the results document. + + Returns + ------- + str + A two-column markdown table. + + Examples + -------- + >>> meta = { + ... "n_seeds": 30, + ... "versions": {"ctrl_freak": "0.1.0", "pymoo": "0.6.1.6", "deap": "1.4.3", "numpy": "2.4.1"}, + ... "budgets": {"so_pop_size": 100, "so_n_generations": 200, "mo_pop_size": 100, + ... "mo_n_generations": 250, "sbx_eta": 15.0, "pm_eta": 20.0}, + ... } + >>> print(_provenance_table(meta)) # doctest: +ELLIPSIS + | Field | Value | + |---|---| + | Seeds | 30 (0–29) | + | ctrl-freak | 0.1.0 | + ... + | SBX / PM eta | 15.0 / 20.0 | + """ + v = meta["versions"] + b = meta["budgets"] + rows = [ + ("Seeds", f"{meta['n_seeds']} (0–{meta['n_seeds'] - 1})"), + ("ctrl-freak", v["ctrl_freak"]), + ("pymoo", v["pymoo"]), + ("deap", v["deap"]), + ("numpy", v["numpy"]), + ( + "SO budget", + f"pop {b['so_pop_size']} × {b['so_n_generations']} gen ({b['so_pop_size'] * (1 + b['so_n_generations'])} evals)", + ), + ( + "MO budget", + f"pop {b['mo_pop_size']} × {b['mo_n_generations']} gen ({b['mo_pop_size'] * (1 + b['mo_n_generations'])} evals)", + ), + ("SBX / PM eta", f"{b['sbx_eta']} / {b['pm_eta']}"), + ] + lines = ["| Field | Value |", "|---|---|"] + lines += [f"| {k} | {val} |" for k, val in rows] + return "\n".join(lines) + + +def build_tables(results: dict) -> dict[str, str]: + """Build every injectable table, keyed by its marker id. + + Parameters + ---------- + results : dict + The full results document. + + Returns + ------- + dict + Mapping ``marker_id -> markdown_table``. + + Examples + -------- + >>> tables = build_tables(results) # doctest: +SKIP + """ + so = results["single_objective"] + mo = results["multi_objective"] + return { + "provenance": _provenance_table(results["metadata"]), + "so_objective_error": _metric_table(so, SO_ORDER, "objective_error", True), + "so_solution_distance": _metric_table(so, SO_ORDER, "solution_distance", True), + "so_success_rate": _success_table(so, SO_ORDER), + "mo_igd_plus": _metric_table(mo, MO_ORDER, "igd_plus", True), + "mo_gd": _metric_table(mo, MO_ORDER, "gd", True), + "mo_hypervolume": _metric_table(mo, MO_ORDER, "hypervolume", False), + } + + +def _verdict_rows(table: str) -> dict[str, tuple[str, str]]: + """Parse a metric table into ``{problem: (vs_pymoo_cell, vs_deap_cell)}``. + + Parameters + ---------- + table : str + A markdown metric table built by :func:`_metric_table` (columns + ``Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap``). + + Returns + ------- + dict + Problem name -> its two verdict cells (the final two columns). + + Examples + -------- + >>> t = "| Problem | a | b | c | vs pymoo | vs deap |\\n|---|---|---|---|---|---|\\n| p | 1 | 1 | 1 | x | y |" + >>> _verdict_rows(t)["p"] + ('x', 'y') + """ + rows: dict[str, tuple[str, str]] = {} + for line in table.splitlines(): + parts = [c.strip() for c in line.strip().strip("|").split("|")] + if len(parts) < 6 or parts[0] == "Problem" or set(parts[0]) <= set("-"): + continue + rows[parts[0]] = (parts[-2], parts[-1]) + return rows + + +def check_report_consistency( + tables: dict[str, str], claims: dict[str, dict[str, tuple[str, ...]]] = _REPORT_CLAIMS +) -> None: + """Raise if any injected verdict cell contradicts the report's prose claims. + + Scans the verdict cells of the freshly built tables -- the exact text injected + into ``benchmarks/README.md`` and ``docs/validation.md`` -- and raises if a + problem the prose declares *indistinguishable* renders ``not equivalent`` for + either baseline, or a problem declared *at least as good* (the documented + ZDT4/ZDT6 exceptions) renders ``ctrl-freak worse``. This makes it impossible to + ship a citation page whose prose contradicts a committed verdict, and catches + any future verdict flip. + + Parameters + ---------- + tables : dict + Output of :func:`build_tables` (marker id -> markdown table). + claims : dict, optional + The prose's parity claims (defaults to :data:`_REPORT_CLAIMS`). + + Raises + ------ + ValueError + Listing every (table, problem, baseline) whose verdict contradicts the + claimed parity. + + Examples + -------- + >>> ok = {"m": "| Problem | a | b | c | vs pymoo | vs deap |\\n|---|---|---|---|---|---|\\n| p | 1 | 1 | 1 | equivalent (margin +1.0e-01) | equivalent (margin +1.0e-01) |"} + >>> check_report_consistency(ok, {"m": {"indistinguishable": ("p",)}}) is None + True + >>> bad = {"m": ok["m"].replace("equivalent (margin +1.0e-01) | equivalent", "not equivalent — ctrl-freak worse | equivalent", 1)} + >>> try: + ... check_report_consistency(bad, {"m": {"indistinguishable": ("p",)}}) + ... except ValueError: + ... print("raised") + raised + """ + violations: list[str] = [] + for table_id, by_claim in claims.items(): + rows = _verdict_rows(tables[table_id]) + for claim, problems in by_claim.items(): + for p in problems: + for base, cell in zip(BASELINES, rows[p], strict=True): + if claim == "indistinguishable" and cell.startswith("not equivalent"): + violations.append(f"{table_id}/{p} vs {base}: prose claims indistinguishable, cell is {cell!r}") + elif claim == "at_least_as_good" and "worse" in cell: + violations.append(f"{table_id}/{p} vs {base}: prose claims at-least-as-good, cell is {cell!r}") + if violations: + raise ValueError("report prose contradicts committed verdicts:\n" + "\n".join(violations)) + + +def _replace_markers(text: str, tables: dict[str, str]) -> str: + """Replace the body of every ``BEGIN/END`` marker pair found in `text`. + + Only ids present in `text` are replaced; ids absent from `text` are ignored, so + each target file declares the subset of tables it wants by including the + matching markers. + + Parameters + ---------- + text : str + Source markdown. + tables : dict + Mapping ``marker_id -> markdown_table``. + + Returns + ------- + str + Markdown with each marker body replaced by its table. + + Examples + -------- + >>> _replace_markers("a old b", {"x": "NEW"}) + 'a \\n\\nNEW\\n\\n b' + """ + for tid, table in tables.items(): + pattern = re.compile( + rf"()(.*?)()", + re.DOTALL, + ) + text = pattern.sub(rf"\1\n\n{table}\n\n\3", text) + return text + + +def inject_tables(path: Path, tables: dict[str, str]) -> int: + """Inject `tables` into the marker blocks of the markdown file at `path`. + + Parameters + ---------- + path : pathlib.Path + Target markdown file (rewritten in place). + tables : dict + Mapping ``marker_id -> markdown_table``. + + Returns + ------- + int + The number of marker blocks replaced (for a sanity print). + + Examples + -------- + >>> inject_tables(README_PATH, build_tables(results)) # doctest: +SKIP + """ + text = path.read_text() + before = len(re.findall(r" + +| Field | Value | +|---|---| +| Seeds | 30 (0–29) | +| ctrl-freak | 0.1.0 | +| pymoo | 0.6.1.6 | +| deap | 1.4.3 | +| numpy | 2.4.1 | +| SO budget | pop 100 × 200 gen (20100 evals) | +| MO budget | pop 100 × 250 gen (25100 evals) | +| SBX / PM eta | 15.0 / 20.0 | + + + +## Result + +Single-objective parity is led by the continuous error metrics |f − f*| and +‖x − x*‖: ctrl-freak is statistically indistinguishable from both baselines on all +six problems, including the multimodal Rastrigin, Ackley, and Schwefel. + + + +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| sphere | 4.61e-03 ± 1.99e-03 | 3.68e-03 ± 1.39e-03 | 3.63e-03 ± 1.86e-03 | equivalent (margin +1.1e-03) | equivalent (margin +1.0e-03) | +| rosenbrock | 6.58e+00 ± 1.65e+00 | 6.90e+00 ± 1.39e+00 | 6.72e+00 ± 1.46e+00 | equivalent (margin +1.3e+00) | equivalent (margin +1.5e+00) | +| rastrigin | 1.84e+00 ± 8.87e-01 | 1.65e+00 ± 9.06e-01 | 1.99e+00 ± 1.10e+00 | equivalent (margin +7.1e-01) | equivalent (margin +9.4e-01) | +| ackley | 7.10e-01 ± 2.29e-01 | 7.12e-01 ± 2.32e-01 | 8.67e-01 ± 3.01e-01 | equivalent (margin +2.3e-01) | equivalent (margin +1.4e-01) | +| griewank | 7.11e-01 ± 1.18e-01 | 7.09e-01 ± 1.19e-01 | 7.10e-01 ± 1.02e-01 | equivalent (margin +1.2e-01) | equivalent (margin +1.2e-01) | +| schwefel | 7.04e+01 ± 7.27e+01 | 3.84e+01 ± 5.47e+01 | 6.62e+01 ± 8.51e+01 | equivalent (margin +4.1e+01) | equivalent (margin +8.1e+01) | + + + +Multi-objective convergence (IGD+ to the analytical front) is statistically +indistinguishable from both baselines on ZDT1, ZDT2, ZDT3, and DTLZ2, on all three +metrics (IGD+, GD, hypervolume). + + + +| Problem | ctrl-freak | pymoo | deap | vs pymoo | vs deap | +|---|---|---|---|---|---| +| zdt1 | 5.85e-03 ± 3.75e-04 | 6.07e-03 ± 3.63e-04 | 5.94e-03 ± 4.22e-04 | equivalent (margin +1.5e-04) | equivalent (margin +3.3e-04) | +| zdt2 | 5.46e-03 ± 4.38e-04 | 5.87e-03 ± 3.77e-04 | 5.69e-03 ± 4.77e-04 | equivalent (margin +2.7e-05) | equivalent (margin +2.4e-04) | +| zdt3 | 5.91e-03 ± 1.02e-02 | 3.17e-03 ± 2.63e-04 | 3.14e-03 ± 2.22e-04 | equivalent (margin +7.4e-03) | equivalent (margin +7.4e-03) | +| zdt4 | 8.73e+00 ± 2.26e+00 | 1.59e+01 ± 4.68e+00 | 1.48e+01 ± 4.73e+00 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| zdt6 | 3.81e-01 ± 3.88e-02 | 5.08e-01 ± 6.77e-02 | 5.25e-01 ± 6.04e-02 | not equivalent — ctrl-freak better | not equivalent — ctrl-freak better | +| dtlz2 | 5.47e-02 ± 3.67e-03 | 5.43e-02 ± 3.61e-03 | 5.47e-02 ± 3.28e-03 | equivalent (margin +3.3e-03) | equivalent (margin +3.6e-03) | + + + +### Two documented exceptions + +The "statistically indistinguishable" sentence has two honest exceptions, both +recorded because the suite reports faithfully rather than selectively: + +- **ZDT4 and ZDT6.** None of the three libraries converges on these hardest + multimodal / biased problems at this budget; the symmetric variance test flags + the gap, but **in ctrl-freak's favour** — ctrl-freak's IGD+, GD, and hypervolume + are at least as good as both pymoo and DEAP. The parity / trust claim is not + threatened (ctrl-freak is at-least-as-good); the clean equivalence sentence + simply does not apply where no library converges. (ZDT4's hypervolume is `0` for + all three because every final front lies outside the `[1.1, 1.1]` reference box.) +- **Single-objective success rate is `0` for every problem and every library.** The + committed ε thresholds are strict (e.g. Sphere ε = 1e-6 while all three reach + ~4e-3 at this budget), so the binary success metric is uniformly zero. This is a + documented **strict-threshold non-convergence property at this budget**, not a + failure: the three libraries reach the same error band, and the parity evidence + rests on the continuous error metrics above. + +## Figures + +Seed-mean convergence with a ±std band, three libraries: + +![Single-objective convergence: best objective error per generation](assets/validation/so_convergence.png) + +![Multi-objective convergence: IGD+ per generation](assets/validation/mo_convergence.png) + +Final non-dominated fronts against the analytical true fronts (ZDT4/ZDT6 sit off +the front for all three libraries — the documented exceptions; the DTLZ2 panel +shows ctrl-freak against the unit-sphere octant, with per-library parity quantified +in the report's IGD+/GD/HV table): + +![Pareto fronts vs analytical true fronts](assets/validation/mo_pareto_fronts.png) diff --git a/mkdocs.yml b/mkdocs.yml index f0edebb..10502e1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,3 +56,4 @@ nav: - Usage Guide: usage.md - User Contracts: contracts.md - API Reference: api.md + - Validation: validation.md