Repository navigation
Add a --regenerate-snapshots pytest option - #10
Merged
Merged
Conversation
ContentTester.verify() writes a .page snapshot only when the file is absent; when it exists and the rendered content differs it fails the test. Re-recording snapshots therefore meant deleting every .page file, rebuilding the test image, running the suite, and copying the files back. With --regenerate-snapshots, a snapshot whose content differs is rewritten in place and a SnapshotRegenerated warning naming the file is emitted instead of failing. A snapshot whose content matches is left alone, so the resulting diff is the set of pages the source change affected. Without the flag, behavior is unchanged: compare, and fail on any difference. The rewrite goes through a temporary file in the same directory and os.replace, so an interrupted run cannot leave a half-written snapshot to be committed as the new baseline. The option is registered in a top level conftest.py because pytest only calls pytest_addoption for initial conftests, and with no path argument the rootdir is the only directory searched for one. regenerate_snapshots.sh drives a full regeneration run: the test image COPYs the source in rather than bind-mounting it, so the rewritten files are copied back out of the container.
The inner sh -c ran pytest through tee and then tar, so the container's exit status was tar's. An ordinary red suite still failed the job, but only because --no-cov-on-fail suppresses .coverage.xml and tar then errored on the missing file. A run that wrote the reports and still exited non-zero -- pytest's exit 5, "no tests collected", from a -k that matches nothing -- left tar succeeding and the job green. pipefail makes the pipeline report pytest's status, it is captured before tar runs, and the container exits with it. On the host the status is captured rather than allowed to abort, so the script runs to the end, and it exits with that status ahead of the coverage gate. A suite that fails writes no coverage at all, so the gate is now skipped with a message rather than letting diff-cover fail on the missing file. The artifacts are cleared before the run so that a run which fails before writing them cannot leave the previous run's files behind to be read as current.
Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
--regenerate-snapshotspytest option so snapshots can be re-recorded in place.ContentTester.verify()writes a.pagesnapshot only when the file is absent; when it exists and the rendered content differs, it fails the test. Re-recording therefore meant deleting every.pagefile, rebuilding the test image, running the suite, and copying the files back.With the flag, a snapshot whose content differs is rewritten in place and a
SnapshotRegeneratedwarning naming the file is emitted instead of failing. A snapshot whose content matches is left alone, so the resulting diff is exactly the set of pages the source change affected. Without the flag, behavior is unchanged: compare, and fail on any difference.The rewrite goes through a temporary file in the same directory and
os.replace, so an interrupted run cannot leave a half-written snapshot to be committed as the new baseline.The option is registered in a top level
conftest.pybecause pytest only callspytest_addoptionfor initial conftests, and with no path argument the rootdir is the only directory searched for one.regenerate_snapshots.shdrives a full regeneration run: the test image COPYs the source in rather than bind-mounting it, so the rewritten files are copied back out of the container.Orphaned snapshots are not pruned — a
.pagefile whose test was renamed or removed is never rewritten and never reported, so finding those remains a manual step.run_coverage.sh exit status
The second commit is separable.
run_coverage.shran pytest throughteeand thentarinside a singlesh -c, so the container's exit status wastar's. An ordinary red suite still failed the job, but only because--no-cov-on-failsuppresses.coverage.xmlandtarthen errored on the missing file. A run that wrote the reports and still exited non-zero — pytest's exit 5, "no tests collected", from a-kthat matches nothing — lefttarsucceeding and the job green.The suite's status is now carried out of the container and reported, ahead of the coverage gate. Since a failing suite writes no coverage at all, the gate is skipped with a message rather than letting diff-cover fail on the missing file.