This repository contains the verified, reproducible empirical benchmarks and ablation experiments for OOPPG (A Contextual Policy over Composable Optimizers).
All experiments adhere to a strict Evidence Gate: every published claim and table can be reproduced on a single machine directly from raw data in results/raw/.
Generate the entire summary report directly from raw data:
git clone https://github.com/optimization-os/ooppg-experiments.git
cd ooppg-experiments
pip install -r requirements.txt
# Autogenerate RESULTS.md and print complete summary tables
python3 scripts/summarize_results.pyThe experiment suite is organized into 5 core research questions across 12 auditable artifacts:
| Category | Script / Artifact | Core Question & Key Finding | Raw Data Source |
|---|---|---|---|
transfer/ |
cold_vs_warm.py |
Does state migration help? +52.7% to +99.5% improvement on smooth functions. |
results/raw/transfer_vs_cold_results.json |
mapped_vs_random.py |
Does it transfer specific structure? Outperforms random warm on Sphere/Rosenbrock; specificity lost on Rastrigin. |
results/raw/specificity_results.json |
|
mechanism/ |
basin_location_invariant.py |
What is the key transferable invariant? Position (mean) is primary (>700% collapse); covariance provides 23%–78% refinement. Geometric invariance rejected. |
results/raw/invariant_results.json |
basin_identity.py |
Is basin identity preserved? Transferred on smooth multi-funnel landscapes (Ackley 80%, Sphere 93%), breaks on Schwefel (0%). |
results/raw/basin_identity_results.json |
|
funnel_prediction.py |
Can funnel ratio predict transfer feasibility? Predictive on Griewank and Levy; fails on boundary optima (Michalewicz). |
results/raw/funnel_prediction_results.json |
|
scheduler/ |
contextual_vs_static.py |
Contextual vs static meta-scheduling (100 seeds) LinUCB demonstrates lowest adaptation lag in 10D (37.0 steps); in 30D Exp3.S achieves 46.1 steps vs LinUCB 49.2 steps. |
results/raw/scheduler_baseline_results.csv |
ablation_scheduling.py |
Bandit policy ablation LinUCB reacts in 37.0 steps vs 48.2–49.1 steps for standard MAB in 10D. |
results/raw/scheduler_baseline_results.csv |
|
dynamic_landscapes/ |
regime_switch.py |
Single abrupt regime shift Static DE (30.71) outperforms OOPPG (42.98); meta-scheduling offers no gain on one-off shifts. |
results/raw/dyna_switch.json |
regime_return.py |
Regime recurrence / detour OOPPG (221.96) beats DE (280.44) by +20.9% and Random (239.72) on the mean. |
results/raw/regime_return.json |
|
negative_results/ |
rugged_multimodal.py |
Multimodal ruggedness and budget sensitivity Evaluates failure modes and negative controls on deceptive landscapes. |
results/raw/rugged_multimodal_results.csv |
st_sufficiency_falsify.py |
Falsification of state sufficiency Evaluates whether adding gradient norm / diversity features outperforms 5D context. |
results/raw/st_sufficiency_results.json |
|
financial_snr_audit.md |
Financial low-SNR negative result audit Full documented audit of CPD failure and exploration drift under noisy market Sharpe objectives. |
Analytical Audit |
Each script can be run in two modes:
- Inspection Mode (default): loads raw experimental data and outputs audited statistical tables in seconds.
python3 transfer/cold_vs_warm.py python3 mechanism/basin_location_invariant.py python3 dynamic_landscapes/regime_switch.py
- Live Execution Mode (
--run): executes live optimizations and updates raw data files.python3 transfer/cold_vs_warm.py --run python3 dynamic_landscapes/regime_return.py --run
- RESULTS.md: Complete autogenerated empirical tables from raw data.
- CLAIMS.md: Audited capabilities and strict empirical boundaries.
- LIMITATIONS.md: 5-field structured failure mode and negative result registry.
If you use these benchmarks or reference data in your research:
@software{ooppg_experiments_2026,
author = {Yingjie Gao},
title = {OOPPG Reproducible Experiment Suite and Benchmarks},
url = {https://github.com/optimization-os/ooppg-experiments},
year = {2026}
}MIT License - see LICENSE for details.