Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@ CONGRESS_API_KEY=
# California SOS office pages per run.
# CA_SOS_MAX_ITEMS=9

# Certified current-cycle Missouri candidates per run.
# MISSOURI_SOS_MAX_ITEMS=1000

# Scraper cost reporting
# Estimated DeepSeek V4 Flash input price.
# LLM_INPUT_PRICE=0.14
Expand Down
8 changes: 8 additions & 0 deletions apps/scraper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ These data sources are registered and run by `all`:
| `scc-cvig` | Hand-configured Santa Clara County voter-guide PDFs | Candidate statements in `CivicApiCache`; the API matches statements to candidates |
| `ca-sos-statements` | California SOS statewide-office candidate-statement pages | Candidate statements in `CivicApiCache`; the API reads the cache and can fall back to the live source |
| `ncsbe` | Current-cycle NCSBE candidate CSV, referendum PDFs, and result ZIPs | Provider-neutral election tables; powers `civic.getNcElectionData` with exact file provenance |
| `missouri-sos` | Missouri SOS 2026 certified candidates, measures, and active results | Provider-neutral current-cycle snapshot; powers `civic.getMissouriCurrentElection` |
| `texas-legislature` | Texas Legislative Council anonymous FTP: current-session history XML and bulk documents | State-aware `bill` rows; read through `content.texasBills` and `content.getById` |
| `texas-current-election` | Texas SOS structured election feed and TLC amendment analyses | Current-cycle snapshots; powers `civic.getTexasCurrentElection` and measure enrichment |
| `cedar-park-council` | Cedar Park's CivicEngage City Council page and its official Municode Meetings embed | Provider-neutral local meetings, documents, agenda items, motions, outcomes, and votes |
Expand Down Expand Up @@ -135,6 +136,7 @@ CONGRESS_MAX_ITEMS=10 pnpm --filter @acme/scraper run start congress
| `SCC_CVIG_MAX_ITEMS` | 10 | Voter-guide PDF documents |
| `CA_SOS_MAX_ITEMS` | 9 | Statewide-office candidate-statement pages |
| `NCSBE_MAX_ITEMS` | 4 | Current-cycle candidate/referendum/result files |
| `MISSOURI_SOS_MAX_ITEMS` | 1000 | Certified current-cycle Missouri candidates |
| `TEXAS_LEGISLATURE_MAX_ITEMS` | 100 | Bills from the latest Texas bulk session |
| `TX_SOS_MAX_ITEMS` | 12 | Current-cycle Texas SOS election payloads |
| `CEDAR_PARK_COUNCIL_MAX_ITEMS` | 100 | Council meetings (after the 12-month cutoff) |
Expand All @@ -153,6 +155,12 @@ history plus candidate contact/address fields. See
[`docs/ncsbe-election-data.md`](../../docs/ncsbe-election-data.md) for discovery,
idempotency, provenance, API, and deterministic Civic-matching details.

The Missouri integration is also current-cycle only. It follows the dynamic
2026 SOS election code, parses the cumulative certified list without candidate
address/contact cells, persists certified measures and result availability, and
has no voter-level or historical reader. See
[`docs/missouri-sos-election-data.md`](../../docs/missouri-sos-election-data.md).

## Cedar Park City Council (`civicengage.ts`)

Cedar Park's public site is CivicEngage, but the City Council records page now
Expand Down
36 changes: 36 additions & 0 deletions apps/scraper/src/fixtures/missouri-sos/calendar.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
<main id="main">
<table class="cal">
<tr>
<th>Official Election Day</th>
<th>Style</th>
<th>Registration</th>
<th>Filing</th>
<th>Close</th>
<th>Final Certification</th>
</tr>
<tr>
<td>April 7, 2026</td>
<td>General Municipal Election Day</td>
<td></td>
<td></td>
<td></td>
<td>January 27, 2026</td>
</tr>
<tr>
<td>August 4, 2026</td>
<td>Primary Election</td>
<td></td>
<td></td>
<td></td>
<td>May 26, 2026</td>
</tr>
<tr>
<td>November 3, 2026</td>
<td>General Election</td>
<td></td>
<td></td>
<td></td>
<td>August 25, 2026</td>
</tr>
</table>
</main>
11 changes: 11 additions & 0 deletions apps/scraper/src/fixtures/missouri-sos/candidate-index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<div id="main">
<h2>2026 Primary Election</h2>
<a
href="DisplayCandidatesPlacement.aspx?OfficeCode=20%20SE%2002&amp;ElectionCode=750006905"
>State Senator - District 2</a
>
<a
href="DisplayCandidatesPlacement.aspx?OfficeCode=20%20LE%20011&amp;ElectionCode=750006905"
>State Representative - District 11</a
>
</div>
49 changes: 49 additions & 0 deletions apps/scraper/src/fixtures/missouri-sos/candidates.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
<div id="main">
<h1>Certified Candidate List</h1>
<h2>2026 Primary Election</h2>
<h3>State Senator - District 2</h3>
<table>
<caption>
Republican
</caption>
<tr>
<th class="NameCol">Name</th>
<th>Mailing Address</th>
</tr>
<tr>
<td class="NameCol">Alice First</td>
<td class="AddressCol">101 Secret Street</td>
</tr>
<tr>
<td class="NameCol">Bob Second</td>
<td class="AddressCol">202 Private Avenue</td>
</tr>
</table>
<table>
<caption>
Democratic
</caption>
<tr>
<th class="NameCol">Name</th>
<th>Mailing Address</th>
</tr>
<tr>
<td class="NameCol">Carol Third</td>
<td class="AddressCol">303 Hidden Road</td>
</tr>
</table>
<h3>State Representative - District 11</h3>
<table>
<caption>
Libertarian
</caption>
<tr>
<th class="NameCol">Name</th>
<th>Mailing Address</th>
</tr>
<tr>
<td class="NameCol">Dana Fourth</td>
<td class="AddressCol">404 Confidential Lane</td>
</tr>
</table>
</div>
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<!-- Historical-looking layout is a parser fixture only. Production never references this path or a historical URL. -->
<div id="main">
<p>Last Updated: November 4, 2024 11:45 PM | Unofficial Results</p>
<table data-reporting-status="100% precincts reporting">
<caption>
Governor
</caption>
<tr>
<th>Candidate</th>
<th>Party</th>
<th>Total Votes</th>
<th>Percent</th>
</tr>
<tr>
<td>Example Candidate A</td>
<td>Republican</td>
<td>1,250</td>
<td>52.1%</td>
</tr>
<tr>
<td>Example Candidate B</td>
<td>Democratic</td>
<td>1,149</td>
<td>47.9%</td>
</tr>
</table>
</div>
56 changes: 56 additions & 0 deletions apps/scraper/src/fixtures/missouri-sos/measures.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
<div id="main">
<h2>
The following ballot measures have been certified for the
<em>August 4, 2026</em> primary election.
</h2>
<p>
<strong>Official Ballot Title<br /></strong>Amendment 1
</p>
<p>
<a href="/full-amendment-1.pdf" title="Amendment 1 full text">full text</a>
</p>
<p>
<a
href="/certificate-amendment-1.pdf"
title="View Certificate of Official Ballot Title for Amendment 1"
>certificate</a
>
</p>
<p><strong>Official Ballot Title:</strong></p>
<blockquote>
<p>Shall the Missouri Constitution be amended for this fixture?</p>
<p>State entities estimate no costs or savings.</p>
</blockquote>
<p><strong>Fair Ballot Language:</strong></p>
<blockquote>
<p>A “yes” vote will amend the Constitution.</p>
<p>A “no” vote will not amend it.</p>
</blockquote>
<h2>
The following ballot measures have been certified for the
<em>November 3, 2026</em> general election.
</h2>
<p>
<strong>Official Ballot Title<br /></strong>Amendment 3
</p>
<p>
<a href="/full-amendment-3.pdf" title="Amendment 3 full text">full text</a>
</p>
<p>
<a
href="/certificate-amendment-3.pdf"
title="View Certificate of Official Ballot Title for Amendment 3"
>certificate</a
>
</p>
<p><strong>Official Ballot Title:</strong></p>
<blockquote>
<p>Shall another certified proposal be adopted?</p>
<p>The fiscal impact is unknown.</p>
</blockquote>
<p><strong>Fair Ballot Language:</strong></p>
<blockquote>
<p>A “yes” vote adopts the proposal.</p>
<p>A “no” vote rejects the proposal.</p>
</blockquote>
</div>
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
<div id="main">
<h1>ShowMO Votes</h1>
<p>Election resources and voter information.</p>
<a href="/elections/results">Previous Election Results</a>
</div>
22 changes: 22 additions & 0 deletions apps/scraper/src/fixtures/missouri-sos/withdrawals.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<div id="main">
<table title="Removed Candidates">
<tr>
<th>Office</th>
<th>Candidate</th>
<th>Reason</th>
<th>Removal Date</th>
</tr>
<tr>
<td>State Representative - District 11</td>
<td><span>Kim Miller (Republican)</span><br />1816 Private Street</td>
<td>Withdrawn</td>
<td>6/9/2026<br />1:02 PM</td>
</tr>
<tr>
<td>State Senator - District 2</td>
<td><span>Robin Removed (Democratic)</span><br />PO Box 999</td>
<td>Disqualified</td>
<td>5/19/2026</td>
</tr>
</table>
</div>
2 changes: 2 additions & 0 deletions apps/scraper/src/scraper-contracts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { congressConfig } from "./scrapers/congress.config.js";
import { durhamBoccConfig } from "./scrapers/durham-bocc.config.js";
import { durhamOnBaseConfig } from "./scrapers/durham-onbase.config.js";
import { federalregisterConfig } from "./scrapers/federalregister.config.js";
import { missouriSosConfig } from "./scrapers/missouri-sos.config.js";
import { ncsbeConfig } from "./scrapers/ncsbe.config.js";
import { sccCvigConfig } from "./scrapers/scc-cvig.config.js";
import { scotusConfig } from "./scrapers/scotus.config.js";
Expand All @@ -20,6 +21,7 @@ export const scraperContracts: readonly ScraperEnvContract[] = [
sccCvigConfig,
caSosStatementsConfig,
ncsbeConfig,
missouriSosConfig,
texasCurrentElectionConfig,
texasLegislatureConfig,
cedarParkCouncilConfig,
Expand Down
2 changes: 2 additions & 0 deletions apps/scraper/src/scrapers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { congress } from "./scrapers/congress.js";
import { durhamBocc } from "./scrapers/durham-bocc.js";
import { durhamOnBase } from "./scrapers/durham-onbase.js";
import { federalregister } from "./scrapers/federalregister.js";
import { missouriSos } from "./scrapers/missouri-sos.js";
import { ncsbe } from "./scrapers/ncsbe.js";
import { sccCvig } from "./scrapers/scc-cvig.js";
import { scotus } from "./scrapers/scotus.js";
Expand All @@ -19,6 +20,7 @@ export const scrapers: readonly Scraper[] = [
sccCvig,
caSosStatements,
ncsbe,
missouriSos,
texasCurrentElection,
texasLegislature,
cedarParkCouncil,
Expand Down
Loading