Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ When contributing to this project, you must adhere strictly to these rules:
## 🚀 Next Steps & Outstanding Work
- The active remaining roadmap is `docs/canonical_data_and_analytics_plan.md`. Phases 1 and
2 are implemented and the scraper identity resolver in Phase 3 is complete. Applied migrations
`0022` through `0037` establish deterministic identity, atomic source-profile writes,
`0022` through `0038` establish deterministic identity, atomic source-profile writes,
provenance, person office terms, and private normalized House and Senate roll-call facts.
Both official vote paths remain bounded, identifier-only, and
isolated from legacy `voting_records`; the versioned `0031` read RPC combines presentation-safe
Expand All @@ -182,7 +182,9 @@ When contributing to this project, you must adhere strictly to these rules:
JSON, legacy vote writes, or a public read path. `0036` records the successful manual canary
and private audit before the reviewed workflow enabled that same bounded path for schedules.
`0037` adds a narrow measure-aware public voting-record RPC without opening those private
tables or changing the v2 vote contract; it is applied and live-validated.
tables or changing the v2 vote contract. `0038` adds bounded official participation summaries,
canonical same-scope alignment rankings, and inspectable pairwise evidence while keeping
state/historical legacy votes separate; both migrations are applied and live-validated.
- Monitor and resolve quarantined identity candidates instead of weakening the pre-write
boundary. A person can have federal, state, and local roles over time; those roles must
not become separate canonical people or be flattened into one office field.
Expand All @@ -207,7 +209,10 @@ When contributing to this project, you must adhere strictly to these rules:
run confirmed that the legacy person-filtered GovTrack crawl had zero attempts or failures while
the official House, Senate, and Congress.gov paths stayed healthy. Migration `0037` is now
applied: v3/v2 base-row equivalence, all 43 exact measure links across 40 roll calls, browser
ACLs, pagination, and representative House/Senate profiles were validated live. The scraper
ACLs, pagination, and representative House/Senate profiles were validated live. Migration
`0038` is also applied: anonymous-role House/Senate summary, alignment, and pair-evidence
reconciliations passed with zero ACL or contract violations, and scraper preflight now requires
the complete analytics surface. The scraper
cron is intentionally paused during active development. Use only the manual workflow, keep the
state LittleSis fields blank and GovTrack profile mode disabled for routine runs, explicitly
enable the reviewed House/Senate/Congress.gov modes when refreshing those facts, and never
Expand Down
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,13 @@ then decorates only official rows with at most 100 exact linked measures. It exp
titles, purposes, official Congress.gov links, and bounded source freshness without granting
browser roles direct access to private provenance or legislative tables.

Migration `0038` builds the first canonical analytics surface from those same reviewed official
facts. The live profile summarizes coverage and participation by chamber and Congress, ranks
same-scope peers using only shared Yea/Nay votes with a ten-vote minimum, and provides paginated
pairwise evidence behind every comparison. State and historical legacy votes remain visibly
separate, and every federal comparison links back to its official roll call and any exact
Congress.gov measure metadata without opening the private fact tables.

Migration `0030_senate_roll_call_production_enablement.sql` verifies migration `0029`'s exact
public barrier, owner-only helper, constraint, ACL, dependency, identity-index, zero-fact, and
service-role read-only contracts. It replaces the same public function OID with a thin guarded
Expand Down Expand Up @@ -394,7 +401,7 @@ only runs the ETL and `nextjs.yml` only builds.
prevents half-applied data changes. Migration `0027` is the explicit exception: use
`ON_ERROR_STOP=1` but omit `--single-transaction` so its committed cutover barrier can become
visible before its second transaction drains old callers and atomically enables both gates.
Migrations `0028` through `0037` use the normal single-transaction rule. `0028` is the private
Migrations `0028` through `0038` use the normal single-transaction rule. `0028` is the private
Senate source-review decision. `0029` installs the write-disabled Senate provenance
contract and hard preflight-only barrier. `0030` verifies that exact disabled state, installs
the guarded wrapper, and enables the database gates. `0031` installs the narrow public,
Expand All @@ -412,7 +419,9 @@ only runs the ETL and `nextjs.yml` only builds.
bounded writer. The current cron pause leaves runtime and manual-input defaults disabled, and
unknown events fail closed. `0037` creates the
measure-aware `get_canonical_voting_records_v3` read RPC without changing v2, legacy votes, or
scraper writers; it advances preflight so the new read contract cannot silently drift.
scraper writers. `0038` adds bounded official-vote summary, same-scope alignment, and
inspectable pair-comparison RPCs; it keeps legacy votes separate and advances preflight so the
complete analytics contract cannot silently drift.

Do **not** replay the full historical migration directory against an upgraded database.
Some migrations contain guarded data decisions and review-state transitions, not just
Expand Down
22 changes: 21 additions & 1 deletion docs/canonical_data_and_analytics_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -511,11 +511,19 @@ or identity-candidate queue.

Build analytics only after identity-aware reads exist.

The first complete capability is implemented by
`0038_canonical_federal_voting_analytics.sql` and the profile/Connections clients. It summarizes
official participation by chamber and Congress, ranks peers only within that exact scope using
shared Yea/Nay casts and a ten-vote minimum, and exposes paginated roll-call evidence behind each
pair. Present and Not Voting remain visible in participation but do not enter the alignment
denominator. The federal lane never reads legacy `voting_records`; state and historical
co-voting remains a clearly labeled compatibility lane until it has its own canonical model.

Candidate RPCs/views:

- shared donor overlap by person
- donor bridge view across canonical people
- voting agreement/opposition by person and jurisdiction
- state/local voting agreement by person and jurisdiction (federal official alignment is complete)
- two-hop organization/person relationship overlap
- same-state, same-chamber, and same-office cohorts
- media co-mention discovery signals
Expand Down Expand Up @@ -712,6 +720,18 @@ access, pagination, and representative House and Senate profiles. Scraper prefli
the 0037 marker, so deployment drift cannot remain silent. This slice added no source request,
credential, writer, or legacy vote mutation.

The federal voting analytics slice added and applied
`0038_canonical_federal_voting_analytics.sql` and the live Voting & Analytics/Connections
integration. Three bounded RPCs provide per-scope participation, canonical same-chamber and
same-Congress rankings, and inspectable pairwise official roll calls with exact measure metadata.
The metric uses only shared Yea/Nay votes, requires at least ten, describes recorded alignment
rather than causation, and keeps state/historical legacy co-voting in a separate labeled lane.
The internal read model and all private provenance/fact tables remain closed to browser roles;
the scraper write contract and source request budget are unchanged. Live anonymous-role checks
reconciled summary categories, rates, ranking samples, and pair evidence for representative House
and Senate profiles with zero contract or ACL failures. Scraper preflight now requires the 0038
marker and all three RPCs so this read surface cannot silently drift.

During active site development the cron trigger is intentionally paused. The ETL is dispatched
manually, remains serialized, and keeps every write-mode input disabled by default. Routine full
refreshes explicitly enable only the reviewed House, Senate, and Congress.gov paths, keep the
Expand Down
Loading
Loading