Which forecast should a building's demand-response agent trust? This is a pre-registered benchmark of classical and quantum reservoir forecasters, judged in euros. It uses a real office building's HVAC load and Estonian 15-minute day-ahead prices.
Report: paper/flexbench_report.pdf (4 pages).
| Result | |
|---|---|
| Demand response with a perfect forecast | 22.9% of HVAC energy cost on test (26.0% during COVID) |
| Best real forecaster | keeps 94.9% of that |
| Accuracy vs euros | models span 51% in RMSE but 12% in euros; naive (2× the error) still keeps 84% |
| 6-qubit quantum reservoir vs matched classical reservoir | −0.4% RMSE, p = 0.35: no significant advantage (pre-registered prediction); +3.9% worse during COVID |
| Shot noise | 1k shots: +2.5% RMSE; 10k shots: +1.6%, which erases the edge |
| Hardware cost (memory 32 h, replay at every step) | 129 QPU-seconds per building per 15 minutes (superconducting, 1k shots), so 7 buildings per processor |
| Weekly re-learning through COVID | −21% RMSE (p < 0.001) vs a frozen model; only €0.09 per week |
| With a €4/kW·month capacity charge | every model raised the monthly peak; 62–64% of the perfect-forecast value kept |
Pre-registered scorecard: 3 confirmed, 2 split, 2 failed (details in results/verdicts.json):
| Hypothesis | Verdict |
|---|---|
| H1 | passed |
| H2 | passed |
| H3 | failed: p = 0.08 |
| H4 | passed |
| H5 | split: a passed, b failed |
| H6 | split: RMSE passed, euros failed |
| H7 | failed by €1.9 out of €7,935 |
- Data (
src/flexbench/data.py):- LBNL Building 59 (CC0, Zenodo 5951008): 15-minute HVAC load and weather, 2018–2020;
- Elering EE day-ahead prices (15-minute, Oct 2025 – Sep 2026), mapped by weekday and season.
- Forecasters (
forecast.py,reservoirs.py), all refit weekly on a trailing 8-week window:- naive, ridge and gradient boosting;
- echo-state networks (84, 144 and 400 units);
- an exact density-matrix quantum reservoir: transverse-field Ising, 6 or 8 qubits, reset injection, ⟨Z⟩ and ⟨ZZ⟩ readout. It also runs memoryless, finite-shot and depolarising variants.
- Controller (
dr.py): thermal mass as a virtual battery. It re-plans hourly by LP over 24 h on the forecast, and is clipped to the actual load and the comfort band every 15 minutes. - Agent (
study/agent.py): picks a forecaster each week from realised euros (SekMo-style agentic model selection). - Hardware cost (
study/resources.py): reservoir memory length, converted to quantum-processor time.
python data/fetch.py # downloads and md5-checks the building data
pip install -r requirements.txt
pytest -q # 13 tests (physics, causality, no look-ahead, controller)
python -m study.dev # development grid, Jan-Jun 2019 -> results/chosen.json
python -m study.run # test + COVID periods, 13 models x 2 tariffs
python -m study.agent && python -m study.resources && python -m study.analyze
python -m study.figures && python paper/build_pdf.pyThe protocol was frozen in PREREGISTRATION.md at commit 388a281, before any test forecast. Its SHA-256 is recorded in results/models.json.
MIT licence. Not affiliated with R8 Technologies or LBNL.