TEST: run docstring examples without comparing output (#866) - #920
Conversation
There was a problem hiding this comment.
Pull request overview
Adds automated execution of docstring examples as part of CI, turning doctests into continuously-validated specifications for the quantecon/ package. This aims to prevent future drift/rot in documented examples (per #866) by running pytest --doctest-modules in CI and addressing known collection/state-leak blockers.
Changes:
- Add a
quantecon/conftest.pyto (a) reset NumPy print options between tests and (b) exclude modules whose doctests can’t pass deterministically (network / wall-clock output). - Extend the existing GitHub Actions test workflow to run a doctest sweep for
quanteconon Linux.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
quantecon/conftest.py |
Adds pytest config to ignore non-deterministic doctest modules and an autouse fixture to prevent NumPy print-option leaks across doctests. |
.github/workflows/ci.yml |
Adds a Linux-only CI step to run pytest --doctest-modules quantecon while ignoring */tests/* to avoid re-running unit tests. |
|
thanks @oyamad -- this was a good idea. |
|
coverage: 93.967% (+3.1%) from 90.9% — ci/doctest-modules-866 into main |
384fcb2 to
0a7f0e7
Compare
mmcky
changed the title
There was a problem hiding this comment.
🔵 Needs a closer look
The test dynamically executes arbitrary docstring code across the whole package and manipulates global NumPy state, and its correctness is entangled with the separate, not-yet-merged #864, warranting human final review.
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Wires the docstring-example sweep from gh-866 into ci.yml so examples cannot silently rot again after the gh-864 fixes. A new quantecon/conftest.py resolves the two blockers raised in gh-866: - An autouse fixture restores NumPy print options after each test, so the game_theory examples that set precision=4 no longer leak process-global state into doctests collected later (previously 8 spurious failures in markov, optimize and random). - collect_ignore excludes util/notebooks.py (fetches over the network) and util/timing.py (prints wall-clock durations), keeping the rendered docs free of `# doctest: +SKIP` directives. The CI step runs on Linux only: one platform is enough to stop drift, and it sidesteps --ignore-glob path-separator issues on Windows. The DeprecationWarning from util/array.py::searchsorted needs no handling since no strict warning filter is configured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
qe_apidoc.py discovers Tools pages by globbing quantecon/[a-z0-9]*.py, so an in-package conftest.py would be picked up as a public module and fail the docs-drift check; the root location also keeps a module that imports pytest out of the installed wheel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the `--doctest-modules` CI step and its root `conftest.py` with a single test that executes every docstring example and checks only that it does not raise. Comparing example output is what made the previous design need machinery. Global NumPy print options leak between modules, causing nine spurious failures, and two modules could not pass verbatim at all. Neither applies once output is not compared: leaked print options are invisible, and `util/timing.py`'s wall-clock durations no longer matter, so only `util/notebooks.py`, which fetches over the network, is excluded -- and that module is deprecated for removal in v1.0. What this gives up is drift in the *text* of an example's output, the category that breaks on every NumPy repr change and would commit us to re-pasting expected output indefinitely. What it keeps is every failure that breaks a reader who copies an example: missing imports, renamed functions, changed signatures. Running as an ordinary test rather than a separate CI step means the sweep covers the whole platform matrix, which execution-only tolerates because it has no platform-sensitive output. A collection guard fails if the sweep ever finds nothing, so it cannot silently become a no-op. Closes #866. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
64848de to
1132b9f
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The test's success depends on external state — it is explicitly stacked on the unmerged #864 (20 examples fail on main) and makes every package docstring a CI gate across the full matrix, which warrants human confirmation of merge ordering.
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Balanced
A failing example was reported only by docstring name and exception: the compiled snippet had no registered source, so the traceback showed '???' and ':1' for every failure, above fifteen lines of harness code. Register each example with linecache under a '<doctest name[n]>' filename, as doctest's own runner does, and re-raise with the source file, exact line, position within the docstring and the example's source, chaining the original exception so a failure inside library code keeps its traceback. The execution loop moves into a helper so a self-test can drive it with a synthetic docstring: the reporting path only runs when an example is broken, and a mistake in its line arithmetic would otherwise surface at the worst moment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
thanks @oyamad for this suggestion. It has been updated for execution only. Will merge. |
|
Author review complete. Rebased onto |
Closes #866.
Makes docstring examples executable specifications, so they cannot silently rot again after the fixes in #864.
Note
This was stacked on #864 because the sweep only passes with those example fixes in place: on the old
mainit reported 20 failures, allNameError. #864 has now merged as squash commit 62f6715, and this branch was rebased onto it on 10 September 2026. It carries only its own three commits, its base ismain, and the net diff is the one test file.Reworked: execution-only
Following @oyamad's review — run the examples, do not compare their output. That is a smaller change than the original
--doctest-modulesdesign, and it removes the two blockers #866 was stuck on rather than working around them.pytest --doctest-modulesin a Linux-only CI stepconftest.pyutil/timing.py(blocker 2)collect_ignoreutil/notebooks.py(blocker 2)collect_ignoreconftest.py+ci.ymlconftest.pyis deleted andci.ymlis untouched: as an ordinary test the sweep runs in the existing pytest job, across the whole platform matrix rather than Linux only. Execution-only tolerates that because it has no platform-sensitive output — float reprs were the reason the original design pinned itself to one platform.What this trades away
Output comparison catches drift in the text of an example. That is the category most of #864's 31 repairs fell into — pre-NumPy-1.14 array spacing and NumPy 2 scalar reprs — and it is precisely the category that breaks again on every NumPy release, committing us to re-pasting expected output indefinitely.
What execution-only keeps is every failure that reaches a reader who copies an example: missing imports, renamed functions, changed signatures. The
lorenz()call in_inequality.pythat motivated #866 is caught either way.Verification
On the rebased branch, against the
mainthat contains #864: 52 passed — 50 docstrings, the collection guard, and the failure-report self-test. Full suite green with the sweep included: 730 passed. The sweep adds three warnings to the suite's usual two, all raised by examples that deliberately exercise deprecated or warning code paths, such asutil.array.searchsorted.Measured earlier on this branch for comparison of the two designs:
--doctest-modules, no conftest--doctest-modules+ the original conftestBoth take about 9 seconds.
The print-options guard is verified as necessary as well as sufficient: running
lemke_howson's examples unguarded leavesnp.get_printoptions()['precision']at 4, and the sweep restores it to 8.A
test_doctests_are_collectedguard fails if the collector ever finds nothing, so the sweep cannot quietly become a no-op — which is how these examples rotted in the first place.Failure reports
A failing example reports the source file and exact line of the offending
>>>call, its position within the docstring, and the exception, with the example's own source shown in the traceback rather than???. Each example is registered withlinecacheunder a<doctest name[n]>filename, as the standard library's doctest runner does, and the original exception is chained so an error raised inside library code keeps its full traceback. Verified by reverting thelorenz()fix from #864 and by injecting a wrong-signature call seven examples intocartesian_nearest_index; both reported the correct file and line.Because this path only runs when an example is broken,
test_failure_report_names_the_exampledrives it with a synthetic docstring: one example that documents a traceback and must be tolerated, followed by one that raisesNameError. It asserts the reported file, line, position and source, and that the original exception is chained. The file now collects 52 items: the collection guard, 50 docstrings, and this self-test.🤖 Generated with Claude Code