Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ EXPO_PUBLIC_API_URL=http://localhost:3000
# Get it: https://openstates.org/accounts/profile/
# OPEN_STATES_API_KEY=

# Optional per-run cap for changed Missouri General Assembly bills (default 100).
# MISSOURI_LEGISLATURE_MAX_ITEMS=100

# Vote Smart key for candidate and state-measure enrichment.
# Get it: https://votesmart.org/share/api
# VOTE_SMART_API_KEY=
Expand Down
62 changes: 48 additions & 14 deletions apps/scraper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ These data sources are registered and run by `all`:
| `ca-sos-statements` | California SOS statewide-office candidate-statement pages | Candidate statements in `CivicApiCache`; the API reads the cache and can fall back to the live source |
| `ncsbe` | Current-cycle NCSBE candidate CSV, referendum PDFs, and result ZIPs | Provider-neutral election tables; powers `civic.getNcElectionData` with exact file provenance |
| `texas-legislature` | Texas Legislative Council anonymous FTP: current-session history XML and bulk documents | State-aware `bill` rows; read through `content.texasBills` and `content.getById` |
| `missouri-legislature` | Missouri House current-session XML exports | Shared state-aware `bill` rows; read through `content.stateBills` and `content.getById` |
| `texas-current-election` | Texas SOS structured election feed and TLC amendment analyses | Current-cycle snapshots; powers `civic.getTexasCurrentElection` and measure enrichment |
| `cedar-park-council` | Cedar Park's CivicEngage City Council page and its official Municode Meetings embed | Provider-neutral local meetings, documents, agenda items, motions, outcomes, and votes |

Expand Down Expand Up @@ -62,7 +63,7 @@ work:
| `COURTLISTENER_API_KEY` | Optional | Higher CourtListener limits for `scotus`. |
| `GOOGLE_API_KEY` / `GOOGLE_SEARCH_ENGINE_ID` | Optional pair | Google Custom Search article thumbnails. |
| `GOOGLE_GENERATIVE_AI_API_KEY` | Optional | Gemini vision fallback for `scc-cvig` PDF extraction. |
| `OPEN_STATES_API_KEY` | Optional | Adds an exact Open States bill ID when Texas jurisdiction/session/identifier match. |
| `OPEN_STATES_API_KEY` | Optional | Adds an exact Open States bill ID when Texas or Missouri jurisdiction/session/identifier match. |

See [the launch environment guide](../../docs/launch.md) for the complete
per-scraper matrix, provider setup links, defaults, and production guidance.
Expand Down Expand Up @@ -127,19 +128,20 @@ pnpm --filter @acme/scraper run start congress --max-items 10
CONGRESS_MAX_ITEMS=10 pnpm --filter @acme/scraper run start congress
```

| Variable | Default | Counts |
| ------------------------------- | ------: | --------------------------------------------------- |
| `FEDERALREGISTER_MAX_ITEMS` | 20 | Presidential documents |
| `CONGRESS_MAX_ITEMS` | 100 | Bills |
| `SCOTUS_MAX_ITEMS` | 50 | CourtListener opinion clusters |
| `SCC_CVIG_MAX_ITEMS` | 10 | Voter-guide PDF documents |
| `CA_SOS_MAX_ITEMS` | 9 | Statewide-office candidate-statement pages |
| `NCSBE_MAX_ITEMS` | 4 | Current-cycle candidate/referendum/result files |
| `TEXAS_LEGISLATURE_MAX_ITEMS` | 100 | Bills from the latest Texas bulk session |
| `TX_SOS_MAX_ITEMS` | 12 | Current-cycle Texas SOS election payloads |
| `CEDAR_PARK_COUNCIL_MAX_ITEMS` | 100 | Council meetings (after the 12-month cutoff) |
| `DURHAM_BOCC_MAX_ITEMS` | 100 | Current-cycle Durham County BOCC meetings |
| `SCRAPER_MAX_NEW_ITEMS_PER_RUN` | 10 | New records receiving expensive AI/image enrichment |
| Variable | Default | Counts |
| -------------------------------- | ------: | --------------------------------------------------- |
| `FEDERALREGISTER_MAX_ITEMS` | 20 | Presidential documents |
| `CONGRESS_MAX_ITEMS` | 100 | Bills |
| `SCOTUS_MAX_ITEMS` | 50 | CourtListener opinion clusters |
| `SCC_CVIG_MAX_ITEMS` | 10 | Voter-guide PDF documents |
| `CA_SOS_MAX_ITEMS` | 9 | Statewide-office candidate-statement pages |
| `NCSBE_MAX_ITEMS` | 4 | Current-cycle candidate/referendum/result files |
| `TEXAS_LEGISLATURE_MAX_ITEMS` | 100 | Bills from the latest Texas bulk session |
| `MISSOURI_LEGISLATURE_MAX_ITEMS` | 100 | Changed bills from active Missouri sessions |
| `TX_SOS_MAX_ITEMS` | 12 | Current-cycle Texas SOS election payloads |
| `CEDAR_PARK_COUNCIL_MAX_ITEMS` | 100 | Council meetings (after the 12-month cutoff) |
| `DURHAM_BOCC_MAX_ITEMS` | 100 | Current-cycle Durham County BOCC meetings |
| `SCRAPER_MAX_NEW_ITEMS_PER_RUN` | 10 | New records receiving expensive AI/image enrichment |

These are per-run limits, not durable calendar-day quotas. Schedule one run per
day to obtain a daily cap. If the scheduler retries or runs multiple times, each
Expand Down Expand Up @@ -223,6 +225,38 @@ work deliberately does not provide historical-session browsing or backfills.

---

## Missouri General Assembly (`missouri-legislature.ts`)

Discovers the active regular and any enabled special sessions from the official
[`SessionSet.js`](https://documents.house.mo.gov/SessionSet.js); no session code
is configured or hard-coded. Each run observes
the House's no-more-than-once-per-30-minutes rule through a durable database
guard, fetches `BillList.XML`, and downloads only individual House bill XML rows
whose `LastTimeRun` changed. At most two bill XML requests run concurrently.

`SenateActList.XML` is also imported, but it is explicitly incomplete: it only
contains Senate bills with House activity. Their latest shared `bill.versions`
entry carries `changes = senate_with_house_actions_only`. Official actions, sponsors,
committees, proposed effective dates, roll-call totals, bill versions,
summaries, fiscal notes, amendments, veto letters, and witness documents are
stored in the shared `bill` contract. `OPEN_STATES_API_KEY` optionally adds an
exact identity match and is never required for official ingestion.

```bash
pnpm --filter @acme/scraper run start missouri-legislature --max-items 10
```

`content.stateBills({ stateCode: "MO" })` returns the active rows retained by
the latest official session descriptor; the scraper removes prior Missouri
sessions, so historical browsing is not provided.

Source format and coverage references: [XML export overview](https://documents.house.mo.gov/),
[BillList fields](https://documents.house.mo.gov/XMLBillList.html),
[individual bill fields](https://documents.house.mo.gov/XMLBillExports.html),
and [Senate bills with House actions](https://documents.house.mo.gov/XMLSenateBillsWithHouseActions.html).

---

## Court cases (`scotus.ts`)

Uses the [CourtListener API](https://www.courtlistener.com/api/) — free, works without a key. Fetches recent opinions and pulls in the plain-text opinion content for AI article generation.
Expand Down
2 changes: 2 additions & 0 deletions apps/scraper/src/scraper-contracts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { congressConfig } from "./scrapers/congress.config.js";
import { durhamBoccConfig } from "./scrapers/durham-bocc.config.js";
import { durhamOnBaseConfig } from "./scrapers/durham-onbase.config.js";
import { federalregisterConfig } from "./scrapers/federalregister.config.js";
import { missouriLegislatureConfig } from "./scrapers/missouri-legislature.config.js";
import { ncsbeConfig } from "./scrapers/ncsbe.config.js";
import { sccCvigConfig } from "./scrapers/scc-cvig.config.js";
import { scotusConfig } from "./scrapers/scotus.config.js";
Expand All @@ -20,6 +21,7 @@ export const scraperContracts: readonly ScraperEnvContract[] = [
sccCvigConfig,
caSosStatementsConfig,
ncsbeConfig,
missouriLegislatureConfig,
texasCurrentElectionConfig,
texasLegislatureConfig,
cedarParkCouncilConfig,
Expand Down
2 changes: 2 additions & 0 deletions apps/scraper/src/scrapers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { congress } from "./scrapers/congress.js";
import { durhamBocc } from "./scrapers/durham-bocc.js";
import { durhamOnBase } from "./scrapers/durham-onbase.js";
import { federalregister } from "./scrapers/federalregister.js";
import { missouriLegislature } from "./scrapers/missouri-legislature.js";
import { ncsbe } from "./scrapers/ncsbe.js";
import { sccCvig } from "./scrapers/scc-cvig.js";
import { scotus } from "./scrapers/scotus.js";
Expand All @@ -19,6 +20,7 @@ export const scrapers: readonly Scraper[] = [
sccCvig,
caSosStatements,
ncsbe,
missouriLegislature,
texasCurrentElection,
texasLegislature,
cedarParkCouncil,
Expand Down
4 changes: 4 additions & 0 deletions apps/scraper/src/scrapers/fixtures/missouri-bill-list.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<ROOT>
<BillXML><BillType>HB</BillType><BillNumber>42</BillNumber><SessionYear>2026</SessionYear><SessionCode>R</SessionCode><BillXMLLink>https://documents.house.mo.gov/xml/261-HB42.xml</BillXMLLink><LastTimeRun>07-22-2026 10:30:48.253</LastTimeRun></BillXML>
<BillXML><BillType>HJR</BillType><BillNumber>7</BillNumber><SessionYear>2026</SessionYear><SessionCode>R</SessionCode><BillXMLLink>https://documents.house.mo.gov/xml/261-HJR7.xml</BillXMLLink><LastTimeRun>07-21-2026 09:20:10.010</LastTimeRun></BillXML>
</ROOT>
16 changes: 16 additions & 0 deletions apps/scraper/src/scrapers/fixtures/missouri-hb42.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
<ROOT><BillInformation>
<BillNumber>HB42</BillNumber><CurrentBillString>HCS HB 42</CurrentBillString>
<Title><ShortTitle>PUBLIC SAFETY</ShortTitle><LongTitle>Modifies provisions relating to public safety</LongTitle></Title>
<ProposedEffectiveDate>2026-08-28</ProposedEffectiveDate><LastAction>04/03/2026 - Reported Do Pass (H)</LastAction>
<Action><Link>https://house.mo.gov/bill.aspx?bill=HB42&amp;year=2026&amp;code=R</Link><Description>Introduced and Read First Time (H)</Description><Guid>100</Guid><ActivitySequence>100</ActivitySequence><PubDate>2026-01-08</PubDate></Action>
<Action><Link>https://house.mo.gov/bill.aspx?bill=HB42&amp;year=2026&amp;code=R</Link><Description>Reported Do Pass (H)</Description><Guid>200</Guid><ActivitySequence>200</ActivitySequence><PubDate>2026-04-03</PubDate><RollCall><TotalYes>10</TotalYes><TotalNo>2</TotalNo><TotalPresent>1</TotalPresent></RollCall></Action>
<Sponsor><SponsorType>Sponsor</SponsorType><FullName>Jamie Example</FullName><District>12</District></Sponsor>
<Sponsor><SponsorType>Co-Sponsor</SponsorType><FullName>Alex Example</FullName><District>13</District></Sponsor>
<BillText><BillTextLink>https://documents.house.mo.gov/billtracking/HB42P.pdf</BillTextLink><LRNumber>1000H.01P</LRNumber><DocumentName>Perfected</DocumentName></BillText>
<BillSummary><SummaryTextLink>https://documents.house.mo.gov/billtracking/HB42-summary.pdf</SummaryTextLink><DocumentName>Perfected</DocumentName></BillSummary>
<FiscalNote><FiscalNoteLink>https://documents.house.mo.gov/billtracking/HB42-fiscal.pdf</FiscalNoteLink></FiscalNote>
<Amendment><AmendmentText>https://documents.house.mo.gov/billtracking/HB42-amendment.pdf</AmendmentText><AmendmentDescription>HA 1</AmendmentDescription></Amendment>
<Hearings><CommitteeName>Crime Prevention and Public Safety</CommitteeName><NoticeDate>2026-02-01</NoticeDate></Hearings>
<SubjectIndex><SubjectName>PUBLIC SAFETY</SubjectName></SubjectIndex>
<Witness><WitnessFormsLink>https://documents.house.mo.gov/billtracking/HB42-witness.pdf</WitnessFormsLink><WitnessFormsLinkDescription>Public testimony</WitnessFormsLinkDescription></Witness>
</BillInformation></ROOT>
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
<ROOT><SenateBillsWithHouseActions><BillInformation>
<BillNumber>SB9</BillNumber><ShortTitle>HEALTH CARE</ShortTitle><LongTitle>Modifies provisions relating to health care</LongTitle><ProposedEffectiveDate>2026-08-28T00:00:00</ProposedEffectiveDate>
<SponsorName>Sample, Sam</SponsorName><LastAction>04/10/2026 - Reported Do Pass (H)</LastAction>
<Action><Link>https://house.mo.gov/bill.aspx?bill=SB9&amp;year=2026&amp;code=R</Link><Description>Reported to the House and First Read (H)</Description><Guid>300</Guid><ActivitySequence>100</ActivitySequence><PubDate>2026-03-20</PubDate></Action>
<Hearings><CommitteeName>Health and Mental Health</CommitteeName></Hearings>
<BillSummary><SummaryTextLink>https://documents.house.mo.gov/billtracking/SB9-summary.pdf</SummaryTextLink><DocumentName>Committee</DocumentName></BillSummary>
</BillInformation></SenateBillsWithHouseActions></ROOT>
7 changes: 7 additions & 0 deletions apps/scraper/src/scrapers/fixtures/missouri-session-set.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
var sessionyearcode = "261";
var baseURL = "https://documents.house.mo.gov/xml/261-";
var specsessionyearcode = "263";
var specbaseURL = "https://documents.house.mo.gov/xml/263-";
var specsession2yearcode = "264";
var specbaseURL2 = "https://documents.house.mo.gov/xml/264-";
var showSpec = "true";
Loading