diff --git a/.gitignore b/.gitignore index 9b2e2a80..155c8e9e 100644 --- a/.gitignore +++ b/.gitignore @@ -178,6 +178,9 @@ benches/results/** # devcontainer files .devcontainer/devcontainer-lock.json +# `just capture-baseline` / `just diff-baseline` output (tools/capture-baseline.py) +.baseline-capture/** + # Useful when running in clusters logs/ diff --git a/docs/content/docs/testing.mdx b/docs/content/docs/testing.mdx index 51326cff..04ea8da5 100644 --- a/docs/content/docs/testing.mdx +++ b/docs/content/docs/testing.mdx @@ -98,6 +98,34 @@ one entry per rank count rather than per case, because the ranks have to reach the same collectives. Select either group with `ctest --test-dir build/editable/Release -L serial` (or `-L mpi-2`). +### Golden baselines + +A refactor that is meant to change no result is checked against a captured +baseline rather than by reading the diff. `tools/capture-baseline.py` propagates +every `tests/data/*.msgpack` fixture at a spread of cutoffs and cutoff types, +plus a few hand-written qubit problems, and dumps each run's term count, its +full `(monomial indices, coefficient)` set and its expectation value: + +```bash +just capture-baseline golden # seed, once, on an unmodified tree +just diff-baseline # rebuild, re-capture, diff against the golden +``` + +Terms are dumped in the engine's own returned order, not sorted, and +`diff-baseline` is a byte-wise `diff`. That is deliberate: the order runs from +`SplitmixHash` through probe order, MPI owner routing and insertion order into +floating-point accumulation order, so a silent reorder is a regression, not an +implementation detail. + +For a change that reorders terms *on purpose*, `--compare REF CAND --tol` holds +the term set and the term count exactly and compares coefficients and energies to +a relative tolerance instead. Reach for it only when the reordering is the point; +it is not a way to wave through a diff that was supposed to be stable. + +The capture is rank-local, so under `mpiexec` each rank writes its own +`manifest-rank.json`. Output lands in `.baseline-capture/`, which is +gitignored. + ## Adding tests diff --git a/justfile b/justfile index 9346525c..737a844b 100644 --- a/justfile +++ b/justfile @@ -27,6 +27,23 @@ test: uv run python -m pytest -m "not mpi" ctest --test-dir build/editable/Release --output-on-failure +# Output directory for `capture-baseline` / `diff-baseline` (gitignored). +baseline_dir := ".baseline-capture" + +# Capture a golden baseline snapshot into `.baseline-capture/LABEL` (default "golden") via +# tools/capture-baseline.py. Run once on an unmodified tree to seed the golden baseline, then +# `just diff-baseline` after any change that must not move a term or a coefficient. +capture-baseline LABEL='golden': + uv run --no-sync python tools/capture-baseline.py --out "{{ baseline_dir }}/{{ LABEL }}" + +# Rebuild monoprop and diff a fresh capture against a stored one (default "golden"). Byte-identical +# is the bar: term order is a regression signal, not an implementation detail. +diff-baseline AGAINST='golden': + uv sync --all-extras --group test --reinstall-package monoprop --no-cache -v + rm -rf "{{ baseline_dir }}/candidate" + just capture-baseline candidate + diff -rq "{{ baseline_dir }}/{{ AGAINST }}" "{{ baseline_dir }}/candidate" + # MPI is off by default in source builds, so build an MPI-enabled editable install # first, then run the suite under mpiexec with --no-sync (avoids a per-rank resync). # Pass RANKS as either a single integer or a semicolon-separated list (e.g. "1;2;4"). diff --git a/tools/capture-baseline.py b/tools/capture-baseline.py new file mode 100755 index 00000000..be6b9d41 --- /dev/null +++ b/tools/capture-baseline.py @@ -0,0 +1,429 @@ +#!/usr/bin/env python3 +# Copyright 2026 Algorithmiq +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Capture golden baselines for bit-identity checks across a refactor. + +For every fixture in ``tests/data/*.msgpack``, propagates through +:class:`~monoprop.MajoranaPropagator` (``Basis::Majorana``) at a handful of cutoffs and cutoff +types, and for a small set of hand-written qubit smoke problems through +:class:`~monoprop.PauliPropagator` (``Basis::Pauli``). For each run this dumps: the term count, the +full ``(monomial indices, coefficient)`` set (in the engine's own iteration order -- ordering is +itself a regression signal, see the module docstring on diffing below), and the expectation value. +Dual-basis note: the msgpack fixtures are all fermionic (Majorana). The public API only converts +Pauli -> Majorana (``PauliOperator.get_majorana_operator()``, the Jordan-Wigner image); there is no +Majorana -> Pauli operator converter to press a fixture into a qubit circuit, and hand-rolling that +transform for a test-only tool risks baking a silently-wrong transform into the "golden" baseline. +So ``Basis::Pauli`` coverage instead comes from ``_PAULI_SMOKE_CASES`` below: a few hardcoded native +``PauliOperator``/``Circuit`` problems, sized like the ones in ``tests/test_basis.py`` / +``tests/test_pauli.py``. Their values are not independently re-derived here -- as with the msgpack +fixtures' *non*-``_EXACT_FIXTURES``, they are simply frozen as-is for regression diffing. + +Ordering and diffing: terms are dumped in the engine's own returned order, not sorted -- that order is +itself load-bearing (``SplitmixHash`` -> probe order -> MPI owner routing -> insertion order -> +floating-point accumulation order), so a silent reorder is exactly the kind of regression this tool +exists to catch, and ``just diff-baseline`` stays a byte-wise diff. + +``--compare REF CAND [--tol]`` is the other mode, for a change that reorders terms *on purpose*: it +holds term membership and the term count exactly and compares coefficients and energies only to a +relative tolerance. It exists for a change that reorders on purpose -- a second row backend that +hashes rows differently, say, and so accumulates in a different order. Do not use it to wave through +a diff whose ordering was supposed to be stable. + +Usage (needs the `test` dependency group synced -- this reuses tests/cases.py, which imports +pytest-cases): + uv sync --all-extras --group test + uv run --no-sync python tools/capture-baseline.py --out .baseline-capture/