docs: build the package reference pages in parallel - #674
Conversation
Split the per-package sphinx-build loop into a ThreadPoolExecutor fan-out with a placeholder-writing pre-pass, and inject each build's automodule directive in-memory at source-read time so parallel builds don't race on the shared rst files. Each per-package build also gets its own doctree cache and html outdir under _build/_pkg_pass/<pkg>/ to avoid clobbering. Local wall-time drops from >8 min (sequential, killed mid-run) to ~1m40s for a full `just docs html` on this box (8 cores).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019e4ws34Es9T8rKzwu3zW93
james-garner-canonical
left a comment
james-garner-canonical
left a comment
There was a problem hiding this comment.
Thanks, this should be a really nice speed-up! I'm on board with the basic approach: write out the placeholders first unconditionally, then use in-memory read interception instead of a different written tree for each pass. My main concern is with the build wrapper reaching into the extension internals to call _main -- would it be clean to refactor this into an unconditional preprocessor step instead?
Done. |
The `_packages` recipe reached into the `package_docs` extension internals, inserting the extensions directory on `sys.path` to call `_main` directly. The placeholders it writes are identical for every per-package build, so generating them is a one-time preparation of the source tree rather than per-build work. Move that logic into `.docs/scripts/package_docs_preprocessor.py`, alongside the existing `diataxis_preprocessor.py`, and run it from a new `_package_docs` recipe that `_packages` depends on. `package_docs.py` drops its `builder-inited` hook and is now purely the source-read and doctree save/restore extension. Also count failed package builds and exit with that count, and hoist `BUILD_DIR` alongside `ROOT`. Refs canonical#498 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6VNKeqxRpjmNNmjRMrPoy
james-garner-canonical
left a comment
james-garner-canonical
left a comment
There was a problem hiding this comment.
Thanks, this looks great!
2 participants
The docs build runs
sphinx-buildonce per package, one after another, because the packages don't share a dependency graph. Fan those out instead..docs/extensions/package_docs.pynow writes only the shared placeholder rst files, and appends the automodule directive in memory on Sphinx'ssource-readevent. The on-disk rst is then identical for every per-package build, so there is nothing left for concurrent builds to race on.docs.just's_packagesrecipe fans out over aThreadPoolExecutor, after a one-shot placeholder pre-pass. Each build gets its own doctree cache and html outdir, and the only shared state is the.save/pickles, which are already keyed per package.On this machine (8 cores)
just docs htmlfinishes in 49 seconds, where the sequential version had not reached halfway through the package list after 8 minutes. The win depends on core count, so the number on a CI runner will be different and is worth a look.Fixes #498