Benchmark Suite

Benchmark Suite

Load, compare, and run Speculate emulator-quality reports across three increasingly realistic test tiers.

---

Overview

The Benchmark Suite is Speculate's quality-assurance harness. It runs in two modes from the same notebook: an interactive viewer that loads saved JSON reports out of benchmark_results/ for side-by-side comparison, and a runner that executes any of the three benchmark tiers live against the currently loaded emulator. Reports are self-describing; large reconstruction or posterior arrays are kept in memory for live inspection but excluded from the on-disk JSON to keep report sizes manageable.

The three tiers form a ladder of realism. Tier 1 asks how accurately the emulator reconstructs spectra, using analytical leave-one-out diagnostics on the training grid and direct reconstruction on an independent paired test grid. Tier 2 stresses the full MLE+MCMC inference workflow against held-out synthetic spectra with known truth. Tier 3 measures observational fit quality when no ground truth exists.

Tiers

• Benchmark Suite — Tier 1 Grid Reconstruction — Does the emulator reconstruct spectra accurately under training-grid leave-one-out and on an independent test grid?
• Benchmark Suite — Tier 2 Parameter Recovery — Can the full inference stack recover parameters from held-out synthetic spectra?
• Benchmark Suite — Tier 3 Observational Spectra — How well does the inference stack fit real observations?

Viewer Capabilities

• Load a saved report and browse its tier summaries.
• Compare Tier 1 scalar metrics and Tier 2 per-parameter RMSE across multiple reports.
• Inspect Tier 3 outputs in single-report view (Tier 3 cross-report comparison is not currently implemented).
• Inspect Tier 1 training- and test-grid per-wavelength RMSE envelopes, reconstruction overlays, and per-component diagnostics.
• Inspect Tier 2 serialised posterior payloads when present.
• Export MCMC cornerplot data bundles for Tier 2 spectra and Tier 3 observations.
• Review the shared mixed Normal/Uniform distance and inclination priors for each selected Tier 3 observation before launching a live benchmark.
• Resume interrupted Tier 3 live runs by skipping observations that already completed for the same configuration.
• Launch new benchmark runs directly from the notebook.

References

• speculate_benchmark_viewer.py — viewer notebook.
• Speculate_addons/speculate_benchmark.py — tier runners and JSON serialisation.