Skip to content

Pause scheduled ETL runs - #111

Merged
capituwuistgoonsquad merged 2 commits into
mainfrom
codex/manual-only-scraper
Aug 19, 2026
Merged

Pause scheduled ETL runs#111
capituwuistgoonsquad merged 2 commits into
mainfrom
codex/manual-only-scraper

Conversation

@capituwuistgoonsquad

@capituwuistgoonsquad capituwuistgoonsquad commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • remove the daily cron trigger and rename the workflow to Manual ETL Scraper
  • preserve explicit fail-closed write choices, the four-hour job bound, and the existing concurrency group
  • keep successful manual ETL runs connected to the frontend deployment workflow
  • document safe manual inputs, quota-aware cadence, and the completed migration 0037 rollout

Safety

  • House, Senate, and Congress.gov writes still default to disabled and require an explicit manual choice
  • legacy person-filtered GovTrack enrichment remains disabled by default
  • runs remain serialized, including against an older in-flight run that uses the existing concurrency group
  • this PR does not dispatch a scraper run; the first manual run should happen only after merge

Validation

  • all 374 scraper unit tests pass
  • both workflow YAML files parse successfully
  • focused workflow/runtime contract tests pass
  • git diff --check passes

Post-merge run

Use blank state LittleSis fields, enable the reviewed House, Senate, and Congress.gov modes, and keep GovTrack profile enrichment disabled. Do not overlap runs; keep full runs at least one hour apart and ordinarily to no more than three per UTC day while OpenStates is active. Monitor the run at 30-minute intervals and inspect ETL_SUMMARY_JSON for local caps, upstream quota observations, breakers, and write reconciliation.

Greptile Summary

This change pauses scheduled ETL execution and keeps the scraper available through an explicitly configured manual run. Manual write modes remain disabled by default, runs remain serialized and time-bounded, and successful scraper runs still trigger the frontend deployment.

Confidence Score: 5/5

Safe to merge: the manual workflow and deployment handoff behave as intended.

No blocking failure remains.

T-Rex T-Rex Logs

What T-Rex did

  • I ran the read-only Python validator against the pre-change parent (4dd9d41).
  • Before capture, I reproduced the retired scheduled trigger and schedule-enabled write mappings.
  • I ran the read-only validator on the current PR Pause scheduled ETL runs #111 checkout and observed all contract assertions passing with exit code 0.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (2): Last reviewed commit: "Update Senate policy contract for manual..." | Re-trigger Greptile

Signed-off-by: capituwuistgoonsquad <ultimamaster13@gmail.com>
Signed-off-by: capituwuistgoonsquad <ultimamaster13@gmail.com>
@capituwuistgoonsquad
capituwuistgoonsquad merged commit 0d537b7 into main Aug 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant