Make the live source-contract monitor trustworthy - #207
Merged
Conversation
jimdc
enabled auto-merge
July 31, 2026 01:05
Retune the daily live verifier so it fails on real upstream drift only: Checkbook Spending probes the product contract_id path and real XML fields; Legistar and ZAP use probeable machine endpoints; bot-blocked landings are classified when the product machine path is healthy; pointer-class sources skip ingest freshness; every failure line names source id and URL class. Also move the MOCS LL63 landing URL to the publisher's current path, align Spending XML parsing with live field names, and keep depot landing pages in sync with source contracts.
GitHub runners are blocked from PASSPort Public machine dumps as well as the HTML landings. Mark those contracts as bot-blocked egress so the live monitor does not report runner 403 as upstream drift; product freshness remains the Worker materialization path.
The keyboard focus gate was failing on Cloudflare Turnstile widget hosts (bare DIVs without product focus styles). Block third-party captcha loads during the walk and skip any remaining captcha hosts so the gate only checks first-party controls.
jimdc
force-pushed
the
the-daily-live-civic-data-source-contracts-monit
branch
from
July 31, 2026 01:26
857b86f to
8dcdadc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The daily live civic-data source-contract check mixed real upstream catches with false alarms (wrong probe shapes, runner bot-blocks, anonymous network errors, and pointer-class freshness gates). This change keeps hard failures for genuine drift while re-truing each probe class to the product path.
What changed
contract_id(the product path after Contracts-domain join) and require live XML fieldscontract_id,payee_name,check_amount,issue_date. Spending parser accepts those field names.Bodies?$top=1(auth via optionalLEGISTAR_API_TOKEN; unauthenticated 403 is the expected gate). ZAP probes a real sample project id, not the literal{project_id}path.source_idand URL class (no barefetch failed); concurrent probes are bounded with one retry on network/5xx flakes.Validation
node tools/verify_source_contracts.mjs(offline fixtures)node --test test/source_contracts.test.mjs test/depot_rederive.test.mjs test/capital_projects_join.test.mjscd worker && node --test test/checkbook_lifecycle.test.mjsnode tools/verify_source_contracts.mjs --live→ green (36/36) locallyNotes for reviewers
Real teeth remain: vanished datasets, schema loss on fields the product uses, broken machine endpoints, and ingest-class freshness still fail the job and open/update the drift issue.