TenSolver Stochastic Benchmark example

[jvpcms]

TenSolver Benchmark Example

Adds a complete Jupyter notebook benchmarking TenSolver on 23 QUBO instances from the QpLib subset of QUBOLib (instances 308–330).

What's included

• Energy distribution visualizations — heatmap showing how the sampled energy distribution evolves across DMRG iterations for a representative instance.
• Performance ratio — computed against Gurobi's best-known objectives as the reference minimum and random bitstring sampling as the baseline.
• Bootstrap analysis — performance ratio inferred for 1, 10, 100, and 1000 samples per iteration using the stochastic-benchmark framework.
• Resource analysis — iterations × samples used as a proxy for compute; interpolation, train/test split, virtual best baseline, and projection experiments follow the same pattern as the Simulated Annealing example.
• Output plots — performance.png, iteration.png, samples.png.

Data files

File	Description
results/energy_history.json	1000 energy samples per DMRG iteration for all 23 instances
results/random_baseline.json	Mean and min energy from 1000 random bitstrings per instance
results/gurobi_best.json	Best objective found by Gurobi (900 s limit) per instance

[Copilot]

Pull request overview

Adds a TenSolver stochastic-benchmark example dataset (Gurobi reference objectives + checkpoint CSVs) to support the Jupyter notebook benchmarking workflow described in the PR.

Changes:

• Added gurobi_best.json reference objectives/status for QpLib instances 308–330.
• Added performance and parameter checkpoint CSV artifacts used for plotting/projection.
• Added multiple “Projection from …” CSV outputs for training stats/results.

Reviewed changes

Copilot reviewed 7 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
examples/TenSolver/results/gurobi_best.json	Adds Gurobi best-known objectives/status per instance for performance-ratio reference.
examples/TenSolver/checkpoints/performance_plotting/baseline.csv	Adds baseline curve data used for performance plotting.
examples/TenSolver/checkpoints/performance_plotting/Projection from TrainingStats.csv	Adds projected performance curve derived from training stats.
examples/TenSolver/checkpoints/performance_plotting/Projection from TrainingResults.csv	Adds projected performance curve derived from training results.
examples/TenSolver/checkpoints/params_plotting/baseline.csv	Adds baseline resource→(iteration,samples) mapping for parameter plotting.
examples/TenSolver/checkpoints/params_plotting/Projection from TrainingStats.csv	Adds projected resource→(iteration,samples) mapping from training stats.
examples/TenSolver/checkpoints/params_plotting/Projection from TrainingResults.csv	Adds projected resource→(iteration,samples) mapping from training results.

[bernalde]

Blocking issues:

• The notebook is not CWD-independent. It fails when executed from the repository root because data files are opened through results/... paths and the source path is added as ../../src.
• The benchmark is not reproducible because the bootstrap sampling and train/test split use NumPy randomness without a fixed seed; re-executing changed the committed displayed values.

Nonblocking issues:

• The notebook works around run_baseline() by attaching a placeholder baseline object, which is brittle example code and should be replaced with a core/API fix or cleaner setup.

Questions:

• None.

Tests run and outcomes:

• git diff --check upstream/main...HEAD: passed.
• In-sandbox direct notebook code-cell smoke from repo root with the Python 3.10 CI environment: failed at cell 3 with FileNotFoundError: results/energy_history.json.
• In-sandbox direct notebook code-cell smoke from examples/TenSolver with the Python 3.10 CI environment: passed.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/ -v: 221 passed.
• jupyter nbconvert --execute could not be completed inside the sandbox because Jupyter kernels need local sockets; unsandboxed execution of an untrusted PR notebook was rejected, so I used the direct in-sandbox code-cell smoke instead.

I would not merge this until the blocking issues above are addressed.

[bernalde]

Blocking: These data loads, together with sys.path.append("../../src") below, depend on the process CWD being examples/TenSolver. Running the notebook from the repository root fails before the first plot with FileNotFoundError: results/energy_history.json, which is a common way users and CI execute notebooks by path. Define a notebook base path once and use it for all data paths, sys.path, and stochastic_benchmark(..., here=...); for example, resolve HERE from either Path.cwd() or Path.cwd() / "examples" / "TenSolver", then open HERE / "results" / ....

[bernalde]

Addressed in e4fdf9f. The notebook now defines HERE, uses it for data paths, src imports, here=str(HERE), and plot outputs. Added tests/test_tensolver_notebook.py and verified full code-cell execution from the repo root.

[bernalde]

Addressed earlier in e4fdf9f. The notebook now defines HERE and uses it for data paths, source imports, here=str(HERE), and plot outputs. The repo-root path behavior is covered by tests/test_tensolver_notebook.py::test_first_tensolver_code_cell_runs_from_repo_root and the direct notebook code-cell smoke.

[bernalde]

Blocking: This benchmark has stochastic outputs but never seeds NumPy before bootstrapping or the later train/test split. Re-executing the notebook changed the committed output for instance 326; for example, the max PerfRatio iteration for 10 samples moved from 39 to 40 in my run. Set a fixed seed before the bootstrap loop and before sb.run_Stats(...), or use a local RNG if the framework supports one, then rerun the notebook outputs so the plots and conclusions are reproducible.

[bernalde]

Addressed in e4fdf9f. Added RANDOM_SEED and seeded NumPy before the bootstrap and sb.run_Stats(...) steps. Added a notebook seed smoke check in tests/test_tensolver_notebook.py; full notebook code-cell execution from the repo root passed.

[bernalde]

Addressed earlier in e4fdf9f. The notebook now defines RANDOM_SEED and seeds NumPy before both bootstrap sampling and sb.run_Stats(...). This is covered by tests/test_tensolver_notebook.py::test_tensolver_notebook_seeds_stochastic_steps; the direct notebook code-cell smoke also reproduced the seeded instance 326 table.

[bernalde]

Nonblocking: This placeholder baseline is a brittle workaround for run_baseline() calling get_experiment_parameters(), which expects self.baseline.recalibrate to already exist. Public examples should not teach users to attach dummy internal state. Prefer fixing stochastic_benchmark.run_baseline()/get_experiment_parameters() to construct the virtual-best baseline without a preexisting baseline, or add a small public setup helper, then remove this class from the notebook.

[bernalde]

Addressed in e4fdf9f. Removed the notebook placeholder and changed stochastic_benchmark.get_experiment_parameters() to use a no-op recalibrator until the first baseline exists. Added test_run_baseline_does_not_require_existing_baseline.

[bernalde]

Addressed by e4fdf9f and completed by dc0a082. The placeholder baseline was removed, run_baseline() can construct the first baseline, and the recalibration callback is now late-bound so later baselines still recalibrate. Covered by the two runtime tests in tests/test_experiments.py.

[bernalde]

Pushed commit:

• e4fdf9f Address TenSolver example review feedback

Main changes:

• Made examples/TenSolver/TenSolver.ipynb resolve its example directory from either the repository root or examples/TenSolver, and use that path for data, source imports, benchmark checkpoints, and plot outputs.
• Added a fixed RANDOM_SEED before the notebook bootstrap and train/test split steps.
• Removed the notebook placeholder baseline workaround by making stochastic_benchmark.get_experiment_parameters() tolerate the first run_baseline() call before a baseline object exists.
• Added regression coverage for run_baseline() and TenSolver notebook path/seed behavior.

Tests run:

• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_run_baseline_does_not_require_existing_baseline tests/test_tensolver_notebook.py -v: 4 passed.
• Direct in-sandbox execution of all TenSolver notebook code cells from the repository root with the Python 3.10 CI environment: passed.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/ -v: 225 passed.
• git diff --cached --check: passed.

Comments intentionally not addressed:

• Earlier Copilot checkpoint CSV comments and earlier Anurag notebook/README comments were already resolved or outdated before this pass, so I did not make additional changes for them.

Remaining risks or follow-up:

• I did not run jupyter nbconvert --execute; it requires a local Jupyter kernel socket in this sandbox, and unsandboxed execution of an untrusted PR notebook is not appropriate. The direct in-sandbox code-cell smoke covers the notebook execution path without opening a Jupyter kernel.

[bernalde]

Blocking issues:

• get_experiment_parameters() captures a permanent no-op recalibrator when called before a baseline exists. See the inline comment.

Nonblocking issues:

• None.

Questions:

• None.

Tests run and outcomes:

• Discovered test commands from README, TESTING.md, pyproject pytest config, Makefile, and GitHub Actions CI.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_run_baseline_does_not_require_existing_baseline tests/test_tensolver_notebook.py -v: 4 passed, 1 warning.
• Direct in-sandbox execution of all TenSolver notebook code cells from the repository root with the Python 3.10 CI environment: passed and reproduced the seeded instance 326 table values.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/ -v: 225 passed, 1 warning.
• git diff --check upstream/main...HEAD: passed.
• GitHub CI on head 82a79009: test (3.10), test (3.11), test (3.12), integration-tests, and coverage-summary all succeeded.

I would not merge this until the blocking issues above are addressed.

[bernalde]

Blocking: This captures lambda df: None into every ExperimentParameters object created before self.baseline exists. That fixes the first run_baseline() call, but it also lets callers create projection/static/search experiments before run_baseline() and later call attach_runs() without recalibrating the now-existing baseline, because StaticRecommendationExperiment.attach_runs() calls the stale parent_params.baseline_recalibrate callback. That is a silent correctness regression for experiment ordering. Please make the callback resolve self.baseline at call time, for example by calling getattr(self, "baseline", None) inside baseline_recalibrate, or limit the no-op only to the ExperimentParameters used to construct VirtualBestBaseline. Add a regression test that creates experiment parameters before a baseline exists, attaches a baseline later, and verifies recalibration is invoked.

[bernalde]

Addressed in dc0a082. baseline_recalibrate now resolves self.baseline when invoked, so parameters created before a baseline exists do not keep a stale no-op. Added test_experiment_parameters_recalibrate_late_bound_baseline, which covers late baseline attachment through StaticRecommendationExperiment.attach_runs().

[bernalde]

Pushed commit:

• dc0a082 Fix late baseline recalibration

Main changes:

• Changed stochastic_benchmark.get_experiment_parameters() so baseline_recalibrate resolves self.baseline at call time instead of capturing a permanent no-op before a baseline exists.
• Added a regression test that creates experiment parameters before a baseline exists, attaches a baseline later, and verifies StaticRecommendationExperiment.attach_runs() recalibrates that late baseline.

Tests run:

• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_run_baseline_does_not_require_existing_baseline tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_experiment_parameters_recalibrate_late_bound_baseline tests/test_tensolver_notebook.py -v: 5 passed, 1 warning.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/ -v: 226 passed, 1 warning.
• Direct in-sandbox execution of all TenSolver notebook code cells from the repository root with the Python 3.10 CI environment: passed.
• git diff --check and git diff --cached --check: passed.
• GitHub CI on dc0a082: test (3.10), test (3.11), test (3.12), integration-tests, and coverage-summary passed.

Comments intentionally not addressed:

• The older CWD, seed, and placeholder-baseline notebook threads are outdated and already had fixes/replies from e4fdf9f, so I did not make additional changes for those.
• The older Copilot checkpoint CSV threads and README/notebook grammar threads are resolved or outdated from earlier commits, so I left them unchanged.

Remaining risks or follow-up:

• The direct notebook smoke leaves generated checkpoint and plot artifacts in the worktree; they remain untracked and were not committed.

[bernalde]

Blocking issues:

• None.

Nonblocking issues:

• None.

Questions:

• None.

Tests run and outcomes:

• Discovered test commands from README, TESTING.md, pyproject pytest config, Makefile, and GitHub Actions CI.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_run_baseline_does_not_require_existing_baseline tests/test_experiments.py::TestStochasticBenchmarkRuntimeErrors::test_experiment_parameters_recalibrate_late_bound_baseline tests/test_tensolver_notebook.py -v: 5 passed, 1 warning.
• PYTHONPATH=src MPLCONFIGDIR=/tmp/mpl-pr52 /home/bernalde/miniconda3/envs/stochastic-benchmark-ci-py310/bin/python -m pytest tests/ -v: 226 passed, 1 warning.
• Direct in-sandbox execution of all TenSolver notebook code cells from the repository root with the Python 3.10 CI environment: passed and reproduced the seeded instance 326 table.
• git diff --check upstream/main...HEAD: passed.
• GitHub CI on head dc0a082: test (3.10), test (3.11), test (3.12), integration-tests, and coverage-summary passed.

Merge recommendation:

• The PR can be merged as-is.