feat: treat documentation examples as real, tested code (include_docs) #261
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # ADR-028: Tested documentation examples | ||
|
|
||
| ## Context | ||
|
|
||
| Pasted Python fences in generated documentation can silently drift from the | ||
| package API: neither the test suite nor the type-checker and lint gates execute | ||
| them. The template's high coverage threshold and broad static-analysis matrix | ||
| otherwise make that inconsistency particularly easy to miss. | ||
|
|
||
| ## Decision | ||
|
|
||
| Generated projects keep complete Python examples in `docs/examples/`. Sphinx | ||
| pages render these files with `literalinclude`, so published code is the exact | ||
| source that the regular pytest suite imports and, when appropriate, calls. | ||
|
|
||
| `docs/examples/` is explicitly included in Ruff, mypy, basedpyright, ty, | ||
| pyrefly, zuban, and pylint scope. It is not part of the package or coverage | ||
| source set: examples document the package rather than constitute product code. | ||
|
|
||
| The Sphinx `doctest` extension is enabled, with a `docs-doctest` tox environment | ||
| included in the default CI run for inline `>>>` snippets that cannot use | ||
| `literalinclude`. | ||
|
|
||
| ## Consequences | ||
|
|
||
| Documentation code now fails the same local and CI checks as a stale import in | ||
| the application. Examples remain close to the docs and out of built wheels; | ||
| users who need distributable demonstrations can use the independent | ||
| `include_examples` scaffold option. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| """Look up the version of the installed distribution.""" | ||
|
|
||
| from importlib.metadata import version | ||
|
|
||
| from {{github_repo_name}}.__metadata__ import PROJECT_NAME | ||
|
|
||
|
|
||
| def version_lookup() -> str: | ||
| """Return the installed distribution version for this project. | ||
|
|
||
| Returns: | ||
| str: The installed distribution version. | ||
| """ | ||
| return version(PROJECT_NAME) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| """Keep the Python modules embedded in the documentation executable.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import importlib.util | ||
| from importlib.metadata import version | ||
| from pathlib import Path | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from {{github_repo_name}}.__metadata__ import PROJECT_NAME | ||
|
|
||
| if TYPE_CHECKING: | ||
| from types import ModuleType | ||
|
|
||
| EXAMPLES_DIR = Path(__file__).parents[1] / "docs" / "examples" | ||
|
|
||
|
|
||
| def _load_example(path: Path) -> ModuleType: | ||
| """Import one documentation example directly from its source path.""" | ||
| spec = importlib.util.spec_from_file_location(f"docs_example_{path.stem}", path) | ||
| assert spec is not None | ||
| module = importlib.util.module_from_spec(spec) | ||
| assert spec.loader is not None | ||
| spec.loader.exec_module(module) | ||
| return module | ||
|
Comment on lines
+18
to
+25
|
||
|
|
||
|
|
||
| def test_all_documentation_examples_are_importable() -> None: | ||
| """Every nested Python module in ``docs/examples`` imports successfully.""" | ||
| examples = sorted(EXAMPLES_DIR.rglob("*.py")) | ||
| assert examples, "docs/examples must contain at least one tested module" | ||
|
Comment on lines
+30
to
+31
There was a problem hiding this comment.
When Useful? React with 👍 / 👎. |
||
| for path in examples: | ||
| _load_example(path) | ||
|
|
||
|
|
||
| def test_version_lookup_example_uses_the_installed_distribution() -> None: | ||
| """The usage-page example resolves the same version as package metadata.""" | ||
| example = _load_example(EXAMPLES_DIR / "version_lookup.py") | ||
| assert example.version_lookup() == version(PROJECT_NAME) | ||
|
Comment on lines
+36
to
+39
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
fd -i 'test_docs_examples.py|pyproject.toml|ty.toml|pyrefly.toml|mypy.ini|.pre-commit-config.yaml' .
printf '%s\n' '--- template outline ---'
candidate=$(fd -i 'test_docs_examples.py' template | head -n 1 || true)
if [ -n "$candidate" ]; then
ast-grep outline "$candidate" --view compact || true
printf '%s\n' '--- candidate contents ---'
cat -n "$candidate"
fi
printf '%s\n' '--- type-checker configuration references ---'
rg -n -i '(^|[^[:alnum:]_])(ty|mypy|basedpyright|pyrefly|zuban|strict|tests)([^[:alnum:]_]|$)' \
--glob '!*.lock' --glob '!*.jinja' . | head -n 300
printf '%s\n' '--- related helper and example references ---'
rg -n '_load_example|version_lookup|PROJECT_NAME|EXAMPLES_DIR' . --glob '*.py' --glob '*.jinja' | head -n 300Repository: hasansezertasan/copier-pyproject Length of output: 43432 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- generated type-checker configuration ---'
for f in tests/test_golden_files/pyproject_library.toml tests/test_golden_files/pyproject_full.toml template/pyproject.toml.jinja; do
if [ -f "$f" ]; then
printf '\n### %s\n' "$f"
cat -n "$f" | sed -n '375,425p;515,555p;590,615p'
fi
done
printf '%s\n' '--- documentation-example ADR ---'
cat -n docs/adr/028-tested-documentation-examples.md | sed -n '1,55p'
printf '%s\n' '--- render-validity assertions ---'
cat -n tests/test_render_validity.py | sed -n '295,350p'
printf '%s\n' '--- installed checker binaries, if present ---'
command -v ty || true
command -v mypy || true
command -v basedpyright || true
command -v pyrefly || true
command -v zuban || true
printf '%s\n' '--- read-only static probe ---'
python3 - <<'PY'
import ast
import pathlib
import re
import tomllib
template = pathlib.Path("template/tests/{% if include_docs %}test_docs_examples.py{% endif %}.jinja").read_text()
generated = re.sub(r"\{\{[^{}]*\}\}", "generated_pkg", template)
generated = re.sub(r"\{%[^{}]*%\}", "", generated)
tree = ast.parse(generated)
function = next(
node for node in tree.body
if isinstance(node, ast.FunctionDef)
and node.name == "test_version_lookup_example_uses_the_installed_distribution"
)
call = next(
node for node in ast.walk(function)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Attribute)
and node.func.attr == "version_lookup"
)
load_call = next(
node for node in ast.walk(function)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "_load_example"
)
helper = next(
node for node in tree.body
if isinstance(node, ast.FunctionDef) and node.name == "_load_example"
)
return_annotation = ast.unparse(helper.returns)
print("helper_return_annotation:", return_annotation)
print("dynamic_attribute_access:", ast.unparse(call.func))
print("loaded_value_name:", ast.unparse(load_call.parent) if hasattr(load_call, "parent") else "example assignment present")
print("protocol_or_cast_in_template:", any(
isinstance(node, ast.Name) and node.id in {"Protocol", "cast"}
for node in ast.walk(tree)
))
for path in (
pathlib.Path("tests/test_golden_files/pyproject_library.toml"),
pathlib.Path("tests/test_golden_files/pyproject_full.toml"),
):
data = tomllib.loads(path.read_text())
ty = data.get("tool", {}).get("ty", {})
print(path, "ty.src.include =", ty.get("src", {}).get("include"))
print(path, "tox style commands containing checkers =",
[cmd for cmd in data.get("tool", {}).get("tox", {}).get("env", {}).get("style", {}).get("commands", [])
if any(name in cmd for name in ("mypy", "basedpyright", "ty", "pyrefly", "zuban"))])
PYRepository: hasansezertasan/copier-pyproject Length of output: 18094 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import ast
import pathlib
import re
import types
path = pathlib.Path(
"template/tests/{% if include_docs %}test_docs_examples.py{% endif %}.jinja"
)
source = path.read_text(encoding="utf-8")
generated = re.sub(r"\{\{[^{}]*\}\}", "generated_pkg", source)
generated = re.sub(r"\{%[^{}]*%\}", "", generated)
tree = ast.parse(generated)
helper = next(
node for node in tree.body
if isinstance(node, ast.FunctionDef) and node.name == "_load_example"
)
test = next(
node for node in tree.body
if isinstance(node, ast.FunctionDef)
and node.name == "test_version_lookup_example_uses_the_installed_distribution"
)
attribute = next(
node for node in ast.walk(test)
if isinstance(node, ast.Attribute) and node.attr == "version_lookup"
)
print("helper_return_annotation:", ast.unparse(helper.returns))
print("attribute_access:", ast.unparse(attribute))
print("protocol_or_cast_declared:",
bool(re.search(r"\b(?:Protocol|cast)\b", source)))
print("module_type_has_version_lookup:",
hasattr(types.ModuleType, "version_lookup"))
for config in (
"tests/test_golden_files/pyproject_library.toml",
"tests/test_golden_files/pyproject_full.toml",
):
text = pathlib.Path(config).read_text(encoding="utf-8")
ty_scope = re.search(
r"\[tool\.ty\.src\](.*?)(?=^\[|\Z)", text, re.MULTILINE | re.DOTALL
)
tox_style = re.search(
r"\[tool\.tox\.env\.style\](.*?)(?=^\[|\Z)", text, re.MULTILINE | re.DOTALL
)
print(config)
print("ty_includes_tests_and_docs:",
bool(ty_scope and 'include = ["src", "tests", "docs/examples"]' in ty_scope.group(1)))
print("style_invokes_ty_check:",
bool(tox_style and re.search(r'"ty",\s*"check"', tox_style.group(1))))
PYRepository: hasansezertasan/copier-pyproject Length of output: 534 Add a typed interface for the dynamically loaded module. 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: hasansezertasan/copier-pyproject
Length of output: 50390
🏁 Script executed:
Repository: hasansezertasan/copier-pyproject
Length of output: 19171
Gate the version lookup example on
include_docs.When
include_docs=False, the template rendersdocs/version_lookup.py. The current test does not detect this file.template/docs/examples/{% if include_docs %}version_lookup.py{% endif %}.jinja.root / "docs" / "version_lookup.py"does not exist.📍 Affects 2 files
template/docs/{% if include_docs %}examples{% endif %}/version_lookup.py.jinja#L1-L14(this comment)tests/test_render_validity.py#L335-L344🤖 Prompt for AI Agents
Source: Coding guidelines