Skip to content

Add a --regenerate-snapshots pytest option - #10

Merged
tescalada merged 2 commits into
mainfrom
add-regenerate-snapshots-script
Jul 21, 2026
Merged

tescalada merged 2 commits into
mainfrom
add-regenerate-snapshots-script

Conversation

@tescalada

Copy link
Copy Markdown
Contributor

Adds a --regenerate-snapshots pytest option so snapshots can be re-recorded in place.

ContentTester.verify() writes a .page snapshot only when the file is absent; when it exists and the rendered content differs, it fails the test. Re-recording therefore meant deleting every .page file, rebuilding the test image, running the suite, and copying the files back.

With the flag, a snapshot whose content differs is rewritten in place and a SnapshotRegenerated warning naming the file is emitted instead of failing. A snapshot whose content matches is left alone, so the resulting diff is exactly the set of pages the source change affected. Without the flag, behavior is unchanged: compare, and fail on any difference.

The rewrite goes through a temporary file in the same directory and os.replace, so an interrupted run cannot leave a half-written snapshot to be committed as the new baseline.

The option is registered in a top level conftest.py because pytest only calls pytest_addoption for initial conftests, and with no path argument the rootdir is the only directory searched for one. regenerate_snapshots.sh drives a full regeneration run: the test image COPYs the source in rather than bind-mounting it, so the rewritten files are copied back out of the container.

Orphaned snapshots are not pruned — a .page file whose test was renamed or removed is never rewritten and never reported, so finding those remains a manual step.

run_coverage.sh exit status

The second commit is separable. run_coverage.sh ran pytest through tee and then tar inside a single sh -c, so the container's exit status was tar's. An ordinary red suite still failed the job, but only because --no-cov-on-fail suppresses .coverage.xml and tar then errored on the missing file. A run that wrote the reports and still exited non-zero — pytest's exit 5, "no tests collected", from a -k that matches nothing — left tar succeeding and the job green.

The suite's status is now carried out of the container and reported, ahead of the coverage gate. Since a failing suite writes no coverage at all, the gate is skipped with a message rather than letting diff-cover fail on the missing file.

ContentTester.verify() writes a .page snapshot only when the file is absent;
when it exists and the rendered content differs it fails the test. Re-recording
snapshots therefore meant deleting every .page file, rebuilding the test image,
running the suite, and copying the files back.

With --regenerate-snapshots, a snapshot whose content differs is rewritten in
place and a SnapshotRegenerated warning naming the file is emitted instead of
failing. A snapshot whose content matches is left alone, so the resulting diff
is the set of pages the source change affected. Without the flag, behavior is
unchanged: compare, and fail on any difference.

The rewrite goes through a temporary file in the same directory and os.replace,
so an interrupted run cannot leave a half-written snapshot to be committed as
the new baseline.

The option is registered in a top level conftest.py because pytest only calls
pytest_addoption for initial conftests, and with no path argument the rootdir is
the only directory searched for one. regenerate_snapshots.sh drives a full
regeneration run: the test image COPYs the source in rather than bind-mounting
it, so the rewritten files are copied back out of the container.
The inner sh -c ran pytest through tee and then tar, so the container's exit
status was tar's. An ordinary red suite still failed the job, but only because
--no-cov-on-fail suppresses .coverage.xml and tar then errored on the missing
file. A run that wrote the reports and still exited non-zero -- pytest's exit 5,
"no tests collected", from a -k that matches nothing -- left tar succeeding and
the job green.

pipefail makes the pipeline report pytest's status, it is captured before tar
runs, and the container exits with it. On the host the status is captured
rather than allowed to abort, so the script runs to the end, and it exits with
that status ahead of the coverage gate.

A suite that fails writes no coverage at all, so the gate is now skipped with a
message rather than letting diff-cover fail on the missing file. The artifacts
are cleared before the run so that a run which fails before writing them cannot
leave the previous run's files behind to be read as current.
@github-actions

Copy link
Copy Markdown

Coverage

Coverage report
FileStmtsMissCoverMissing
sparkmeter
   __version__.py16288%32, 38
   _wsgi_mount_smoketest.py97970%19–152
   app.py3151197%19–20, 111, 368–375, 552–554, 675, 682–683
   asgi.py74740%19–166
   cli.py42686%59–70
   cli_app.py880%4–20
   cli_prompts.py151220%8–12, 17–21, 26–28
   conftest.py181995%51–52, 85–86, 108–120, 209
   controller.py2444283%183, 207–208, 531, 585–642
   debug_memory.py32320%6–63
   periodic.py563145%60–66, 71–75, 80–84, 93–98, 103–111, 124–128
   sentry_proxy.py342235%38, 44–47, 56–65, 73–82
   servercommand.py1127236%47, 56–60, 64–69, 73–78, 82–85, 89–90, 101–103, 112–181
sparkmeter/alembic/versions
   0.32_merge_sms_tables.py46393%61–67
   0.77_snapshot_support.py128298%340, 401
sparkmeter/api
   customerviews0.py283199%301
   historyviews0.py1149219%49–55, 69–72, 81–131, 144–175, 185–216, 230–257
sparkmeter/config
   configparameter.py57296%80, 111
sparkmeter/database
   alchemy.py54493%61–63, 74
   database.py232399%497, 579–580
   ormobject.py55198%65
   session.py21210%12–62
   types.py54394%91–92, 125
sparkmeter/ground
   groundcommand.py201050%20–35
sparkmeter/history
   historyview.py351654%30–31, 41–54, 65–72
sparkmeter/meter
   meterdomain.py7722797%892, 1805–1806, 1817–1868, 1911
sparkmeter/metering
   api.py27967%66–76
   cli.py2275874%158, 174–175, 184, 193, 198–201, 203, 234, 238–244, 281–297, 301–335, 456–458
   dispatch.py741678%78–80, 85–97, 185
   events.py1053468%56, 75–77, 109–112, 118–193, 240
   lifespan.py695225%37–98, 109–125
   reconcile.py1134362%84–107, 119–120, 129, 135–138, 140–143, 169, 259–262, 266–269, 273–276, 287
sparkmeter/metering/tools
   cli_client.py825829%46–52, 56–61, 65–70, 83–107, 115–148, 153–166
sparkmeter/misc
   datetimeutils.py65395%115–117
sparkmeter/system
   systemcommand.py67593%35, 130–132, 134–135
sparkmeter/tariff
   tariffcommand.py821582%27–44
sparkmeter/transaction
   transactionview.py167199%71
sparkmeter/user
   userdomain.py1871294%416–434
sparkmeter/web
   unittestutils.py1941194%73–81, 323, 332, 398, 400
   views.py72199%71
TOTAL1266792193% 

Tests Skipped Failures Errors Time
1277 0 💤 0 ❌ 0 🔥 3m 58s ⏱️

@github-actions

Copy link
Copy Markdown

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • conftest.py (100%)
  • sparkmeter/web/unittestutils.py (100%)

Summary

  • Total: 36 lines
  • Missing: 0 lines
  • Coverage: 100%

@tescalada
tescalada merged commit b9ed0b6 into main Jul 21, 2026
5 checks passed
@tescalada
tescalada deleted the add-regenerate-snapshots-script branch September 25, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant