Repository navigation
Feat/guarded catalog refresh - #36
Merged
Merged
Conversation
…reaming budget full_refresh.py hardcoded --streaming-max-requests/--enrichment-max-titles as argparse defaults instead of reading Settings, so STREAMING_AVAILABILITY_ MAX_REQUESTS_PER_RUN and TMDB_ENRICHMENT_MAX_TITLES_PER_RUN in .env had no effect on `make catalog-refresh`. Wire both defaults through Settings, matching the pattern already used in enrich_catalog.py. Also resize the streaming budget for the intended twice-weekly cadence: the old defaults (2 requests/run, 800/month cap) were either unusable in production or slightly over budget once actually applied (100 req/run x ~8.7 runs/month ≈ 867 > 800). New defaults: 80 requests/run, 900/month cap (the real provider limit is 1000/month; 900 leaves margin). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…esh workflow manual-ingestion.yml (bounded TMDB discovery sample) and streaming-lifecycle.yml (standalone lifecycle fetch) are superseded by ingestion.full_refresh, which already runs both stages as part of one guarded, validated pipeline. Keeping streaming-lifecycle.yml around also let an ad hoc manual dispatch compete with the routine refresh for the same monthly Streaming Availability request budget. Add catalog-refresh.yml as a draft of the routine refresh flow from docs/catalog-refresh-runbook.md. workflow_dispatch is its only trigger -- no schedule -- so it never runs on its own until deliberately activated in a future release. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/clean_rebuild.sh wraps the "clean rebuild from scratch" flow from docs/catalog-refresh-runbook.md: it moves (never deletes) data/lake and the DuckDB files into a timestamped backup directory, then runs ingestion.full_refresh in backfill mode against the now-empty lake. Requires typing "rebuild" to confirm (or --yes for non-interactive use). Backfill only skips titles that already have any retained TMDB metadata, regardless of age, so backfilling on top of an existing lake would silently re-enrich nothing already touched -- moving prior state aside first is what makes this an actual from-scratch rebuild. Also documents a real caveat rather than hiding it: data/warehouse.duckdb holds the persisted monthly Streaming Availability request counter, so running this resets that counter locally without resetting real usage against the provider's cap. The script warns about this before confirming. Wired up as `make catalog-rebuild-clean [COUNTRY=xx]`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts: # Makefile # ingestion/full_refresh.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
full_refresh.py's CLI defaults for streaming request volume and enrichment title cap were hardcoded instead of reading fromSettings/.env.manual-ingestion.ymlandstreaming-lifecycle.ymlworkflows — both superseded byingestion.full_refresh, and the latter competed with the routine refresh for the same monthly budget — with a draftcatalog-refresh.yml(workflow_dispatch-only, not yet scheduled).scripts/clean_rebuild.sh(make catalog-rebuild-clean): backs up local catalog state before running a full TMDB backfill, since backfill mode permanently skips any title it's already touched once.masterresolving duplicate history left by PR feat(ingestion): add guarded catalog refresh workflow #35's squash-merge (same content, different commit hash as this branch's base) — no functional changes from the merge itself.Test plan
make test(117 passed, 79% coverage)make lintcatalog-refresh.ymlagainst a real TMDB/Streaming Availability key (not run — draft, not yet activated)