perf(dump): add --exclude so deploy-pages skips generating the game dump - #47
Merged
Merged
Conversation
`python -m app.dump` regenerates every collection. With ~962k games the Pages build spent hours writing per-record files that the assemble step then deleted before upload (the last deploy ran 5h26m). Add a repeatable `--exclude COLLECTION` flag: `generate()` already accepted a `collections` list, so this just threads it through `run()` and the CLI via a new `resolve_collections()` helper. Unknown names raise instead of being ignored, so a typo in a workflow fails loudly. deploy-pages now runs `--exclude games`, which drops the wasted generation work and makes the post-hoc `rm -rf _site/v1/games` + manifest edit unnecessary (the manifest is written without games to begin with).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
deploy-pagesregenerates the full dump (python -m app.dump), then #46 deletes_site/v1/gamesbefore upload. So the build spends hours writing ~962k per-record game files that are thrown away — the last successful deploy took 5h26m, almost all of it inGenerate static JSON dump.Change
Add a repeatable
--exclude COLLECTIONflag to the dump CLI.generate()already accepted acollectionslist, so this only threads it throughrun()andargparsevia a smallresolve_collections()helper:COLLECTIONSis preserved; the run summary reports what was skipped.deploy-pagesnow passes--exclude games, which makes the post-hocrm -rf _site/v1/games+ manifest surgery from ci(pages): exclude game dump from Pages deploy (fixes syncing_files timeout) #46 unnecessary — the manifest is written withoutgamesto begin with. Net −20/+6 lines in the workflow.Games remain available as versioned data in the TechAPI repo; all other collections are published exactly as before.
Verification
Ran locally against the engine:
python -m app.dump --helpshows the new flagresolve_collections()→ 12 collections by default, 11 with--exclude games,ValueErroron a typoruff check app tests✅ ·mypy app/dump.py✅tests/integration/test_dump.py(defaults / exclusion+order / unknown-name) passFollow-up to #45 and #46.