Spec: Allow election admins to bulk upload ballots

[ArendPeter]

Part of #810

This spec consolidates the decisions already grilled and recorded on #810's wayfinder map, across #1598, #1599, #1600, #1601, and #1602. Nothing below has been implemented yet — this is the implementation-ready spec the map exists to produce.

Problem Statement

Election admins have no supported way to bulk-load a batch of already-cast ballots (paper ballots that were tallied elsewhere, or ballots recovered from another system) into a BetterVoting election. The one existing bulk-upload tool (Upload Elections) is a system-admin-only tool that creates whole new archived elections from rank-only CVR files inferred by filename — it can't add ballots into an election that's also collecting votes online, it can't express any voting method besides ranked-choice, and it has no place for an ordinary election owner to use it.

This blocks two needs: hybrid paper+online elections (the long-term driver — paper ballots are a real requirement for some elections), and quickly seeding test data into an election under development (the short-term driver). It's also currently impossible for an election owner to control, on a per-election basis, whether admin-submitted ("paper") ballots are allowed at all — the existing canUploadBallots role permission is an all-or-nothing gate tied only to being the election owner or a system admin, with no election-level opt-in.

Solution

• On the Manage Voters admin page, an election owner (or system admin) sees a new "Upload Ballots" button next to the existing "Add Voters" button, usable whenever the election's settings permit paper-ballot submissions.
• Clicking it opens a dialog — modeled on the existing "Add Voters" dialog — with a paste-CSV textarea stacked above a file-select button (CSV or JSON; no separate format toggle, the file extension picks the parser). Both accept the same shape the election's own ballot-data export produces, with a voter_id column/field added, so an admin can round-trip an export back in with voter_ids filled in.
• Before anything is uploaded, the dialog validates the file's structure against the election's actual races, candidates, and voting method — any mismatch rejects the whole file up front, with a clear error, before a single row is parsed.
• The dialog then previews every row's status in a single table: ready to upload, will be skipped (because that voter already voted online — closed elections only), or already invalid (bad/unmatched voter_id). The admin reviews this, then clicks "Confirm & Upload", which flips the same rows in place to their final result (done / failed) as the batch uploads, without a separate results screen.
• On the Settings admin page (editable only in draft), the owner picks which ballot submission channels are allowed for the election via a checkbox group: Online (browser), Paper ballots (admin upload), and (behind a feature flag) Discord. "Paper ballots (admin upload)" is unchecked by default, matching the issue's requirement that admins must opt in prior to running the election. At least one channel must always remain checked.
• If Online (browser) isn't an allowed channel for an election, voters see a clear banner in place of being able to cast a ballot, and the "Vote" button is hidden from the election's home page — the same treatment already used for draft/archived elections.

User Stories

1. As an election owner, I want to enable "Paper ballots (admin upload)" as a submission channel while my election is still in draft, so that I can prepare to accept ballots collected on paper.
2. As an election owner, I want "Paper ballots (admin upload)" unchecked by default, so that I never accidentally expose a way to inject ballots into an election I didn't intend to allow it for.
3. As an election owner, I want to be prevented from unchecking every submission channel at once, so that I can't accidentally lock everyone (including myself) out of voting.
4. As an election owner, I want the submission-channel settings locked once my election leaves draft, so that the rules for how ballots may be submitted can't change mid-election.
5. As an election owner or system admin, I want an "Upload Ballots" entry point next to "Add Voters" on Manage Voters, so I can find bulk ballot upload where I already manage my voter list.
6. As an election owner, I want the ability to use "Upload Ballots" to actually depend on my "Paper ballots (admin upload)" setting, so the setting I configured is genuinely enforced, not just cosmetic.
7. As an election owner, I want to paste ballot data directly into a text box, so I don't need to save a file first for a small batch.
8. As an election owner, I want to select a CSV or JSON file from my computer, so I can upload a larger batch at once.
9. As an election owner, I want the accepted upload format to be my election's own ballot-data export format plus a voter_id column, so I can round-trip an export back in without hand-reformatting it.
10. As an election owner, I want the whole file rejected up front if its columns/structure don't match my election's races and candidates, so I never end up with a partially-applied, structurally-wrong upload.
11. As an election owner, I want to see, before committing anything, which rows are ready, which will be skipped, and which are already invalid, so I can review the batch before it's applied.
12. As an election owner uploading ballots into a closed (voter-list) election, I want a row whose voter already voted online to be automatically flagged as skipped, so I don't double-count that voter.
13. As an election owner, I want to see the full list of rows that will be skipped, not just a count, so I can verify the skip list is what I expect before confirming.
14. As an election owner, I want one "Confirm & Upload" action that updates the same table in place with final results, so I'm not walked through a multi-step wizard for something this routine.
15. As an election owner uploading ballots into an open election, I want rows without a voter_id handled automatically, so I don't have to invent voter identities for ballots that never had one.
16. As an election owner, I want an invalid voter_id in a closed election to fail only that row, so one bad row doesn't block the rest of the batch.
17. As an election owner, I want a genuine (rare) race condition — where a voter submits online in the moments between my preview and my confirm click — surfaced as a normal per-row failure, so I'm not confused by a separate, unexplained error state.
18. As a system admin, I want the existing canUploadBallots role permission (system admin / election owner only) to remain the access gate on the upload endpoint, so the new setting composes with, rather than replaces, existing role-based access control.
19. As a developer, I want the CVR-parsing and batch-upload logic already built for the Upload Elections tool extracted and reused, not duplicated, so there is exactly one implementation of race-order computation, ballot encoding, and batched upload with retry/progress in the codebase.
20. As a developer, I want the Upload Elections tool itself rewired onto that same shared implementation, so no divergent copy of the upload pipeline is left behind.
21. As a developer, I want each admin-submitted ballot's origin already captured via the existing per-ballot history mechanism, so no new tracking field is needed to tell admin-uploaded ballots apart from voter-submitted ones.
22. As an election owner, I want the backend to reject an admin-uploaded ballot outright if paper-ballot submission isn't an allowed channel for my election, so the rule holds even if someone calls the upload endpoint directly.
23. As a voter, I want a clear banner and no "Vote" button when online voting isn't an allowed channel for the election I'm viewing, so I understand why I can't cast a ballot there.
24. As a developer, I want the Discord submission-channel checkbox hidden behind a feature flag, so an option for an unimplemented channel doesn't confuse admins who don't have it enabled — while Discord stays policy-allowed by default underneath, since that flag only controls UI visibility.
25. As a developer, I want the CVR-parser interface to be a plain per-format function (no runtime registry), selected by file extension, so adding a future format later doesn't require a new abstraction.
26. As an election owner, I want manual paste and file upload to use the identical parsing path, so pasting a CSV behaves exactly like uploading that same CSV as a file.

Implementation Decisions

Scope boundary: this issue specs the feature; it does not implement it. Each numbered item below is a decision already reached (via grilling, in the referenced sub-issue) about what the eventual implementation must do.

Shared domain types (packages/shared)

• BallotSubmitType ('submitted_via_browser' | 'submitted_via_admin' | 'submitted_via_discord') moves from its current backend-local definition into the shared Ballot domain model, alongside the existing BallotAction/NewBallotWithVoterID types.
• A new BallotActionType is added in the same place — today a plain alias of BallotSubmitType (nothing in the codebase pushes any other value onto a ballot's history), kept as a separate name so it's the extension point if a non-submission ballot action is ever introduced. The existing (currently untyped, bare string) action-type field on a ballot's history entries retypes to it.
• No new field is needed to record whether a ballot was admin-uploaded — that's already fully captured by the existing per-ballot history mechanism's action type ('submitted_via_admin' vs 'submitted_via_browser').
• No new type is introduced for parser output. The already-existing NewBallotWithVoterID shape ({ voter_id: string, ballot: OrderedNewBallot }) is the parser output target — it already matches what the upload endpoint expects per row, and is already the request-body type of an existing-but-currently-unused upload hook, evidently scaffolded ahead of this feature.
• The existing NewBallot type gets three more of its fields marked optional (election_id, status, date_submitted, joining the already-optional ballot_id/create_date/update_date/head) — every current constructor already supplies them, and the ballot-submission pipeline unconditionally overwrites all three server-side regardless of what's sent, so requiring a parser to fabricate them is pure busywork.

Election settings & validation

• New optional ElectionSettings field: allowed_submit_types?: BallotSubmitType[].
• Default, applied at every read site as election.settings.allowed_submit_types ?? DEFAULT_ALLOWED_SUBMIT_TYPES: DEFAULT_ALLOWED_SUBMIT_TYPES = ['submitted_via_browser', 'submitted_via_discord'] — i.e. paper-ballot admin upload is off by default, matching the issue's explicit requirement.
• No database migration: election settings is a schema-less JSON column, and adding an optional field with a runtime fallback is this repo's established pattern for new settings.
• Settings validation gains a rule rejecting a resolved allowed_submit_types array that would end up empty — the server-side backstop behind the client-side "can't uncheck the last channel" guard described below.

Backend enforcement

• The ballot-submission pipeline (the shared function all ballot writes — browser, admin-upload, and future Discord — already funnel through) gains a check that the incoming submission type is one of the election's allowed_submit_types, rejecting with a 400 naming the disallowed type if not.
• This check sits in the same already-existing bypass block that skips auth/roll/validation checks for a draft election or a prior_election-sourced election (i.e. building out a draft, or importing a prior election's history, is never blocked by this setting) — consistent with every other gate in that pipeline.
• The existing canUploadBallots role permission (system admin / election owner only) is unchanged and remains the access gate on the upload endpoint. The new setting is a second, independent gate layered on top — not a replacement. (This composition — a role permission plus an election-level setting, checked independently — is a deliberate pattern worth recording in an ADR; see Further Notes.)
• A pre-existing local constant that hardcodes the 'submitted_via_admin' string for admin-submission history bookkeeping gets typed against the new shared BallotSubmitType, closing a small gap where that value could drift from the shared union.

Frontend settings UI

• One allowed_submit_types control on the election Settings page, styled as a FormGroup of three checkboxes — not three independent toggles, since they share one array-valued setting: "Online (browser)", "Paper ballots (admin upload)", "Discord".
• Each checkbox's checked state reads (election.settings.allowed_submit_types ?? DEFAULT_ALLOWED_SUBMIT_TYPES).includes(<that type>); toggling one writes the full updated array back.
• The whole group is editable only while the election is in draft, matching the existing pattern used by every other election-settings toggle.
• The last remaining checked box is disabled client-side so it can't be unchecked (minimizing round-trips to the server-side backstop above).
• The Discord row is hidden entirely behind a new feature flag (following the existing pattern used for the precinct-grouping feature flag) — visibility only. Discord stays policy-allowed by default in the underlying array regardless of the flag; actual Discord ballot submission remains unimplemented and out of scope here.

Browser-voting gating (a consequence of allowed_submit_types, not a separate feature)

• The election home page hides the "Vote" button when submitted_via_browser isn't in the resolved allowed_submit_types — the same treatment already given to an archived election.
• The vote page shows a banner ("Online voting is not enabled for this election. Please contact the election administrator.") in that case, styled and placed like the existing draft/archived banners (alongside the ballot, not replacing the page).

CVR parser module

• Two new named parser functions are added alongside the existing (untouched in interface, though not in output type — see below) ranked-CVR parser: one for the CSV shape, one for the JSON shape. There is no runtime parser registry — the caller picks a parser directly by file extension, since auto-detecting format by trying multiple parsers is explicitly out of scope.
• Common parser signature (a type-shape decision worth inlining precisely):

  type CvrParser = (fileText: string, election: Election) => {
      ballots: (NewBallotWithVoterID | undefined)[]; // undefined = a row skipped/rejected during parsing
      errors: ParseError[];
  }

• CSV format: the election's existing ballot-data export header, with a voter_id column prepended — voter_id, ballot_id, precinct, <one column per race/candidate>, [overvote_rank, has_duplicate_rank for ranked races]. voter_id may be blank per row (see voter_id handling below).
• JSON format: { Election, Ballots: (AnonymizedBallot & { voter_id?: string })[] } — keeping the same top-level Election field a real export already contains (now used for structural validation, see below), and ignoring a Results field if present (harmless — a real export download already includes one).
• Structural validation against the target election is a whole-file reject, performed before any row is parsed:
  • CSV: match column headers against candidate names per race (the export never writes internal race/candidate IDs, only labels, so name-matching is the only option for a true round trip). An unrecognized column, or a race with zero matched candidate columns, rejects the entire file.
  • JSON: compare the embedded Election.races against the target election's races directly (exact IDs are available here, so this is a precise structural comparison, not name-matching).
• voter_id handling:
  • Open elections (with voter-ID authentication enabled): a blank voter_id is auto-generated client-side before the row is sent. This isn't cosmetic — the roll-lookup path falls back to the admin's own user id when no override is given, which would otherwise silently collapse every voter-id-less row onto one roll entry.
  • Closed elections: a voter_id must resolve to an existing roll entry; a missing/non-matching one is a per-row rejection (the upload endpoint already reports success/failure per row), not a whole-file reject.
• Each parser (including the existing ranked-CVR one, once rewired) computes race order and encodes each row into wire format internally and returns ready-to-upload rows directly — parsing and wire-encoding are not separate stages exposed to callers.

Shared upload utilities (new module, used by both this feature and the rewired Upload Elections tool)

• computeRaceOrder(election): RaceCandidateOrder[] — derived from the election's races/candidates, independent of any ballot row.
• encodeBallotRow(row, raceOrder): OrderedNewBallot — looks up each race in a row's votes by race ID (not positional index); a race missing from a row's votes encodes as all-undecided for that race.
• uploadBallotsBatched(electionId, raceOrder, ballots, onProgress, options) — the shared batch/retry/progress loop (default batch size 700, shrinking by 25% down to a floor of 10 on failure, matching today's tuning), resolving with partial results plus a synthetic failure entry for any untried remainder if the shrink floor is hit (rather than silently discarding progress on abort, which is what happens today).
• The Upload Elections tool is rewired onto this module and the new parser interface: its election-creation/inference logic and post-reupload ballot-deletion logic are untouched (public-archive-only concerns this feature never needs), but its inline race-order computation, ballot encoding, and hand-rolled batch/retry loop are deleted, leaving no duplicate implementation behind.
• The existing (currently unused) upload-ballots API hook is missing a race_order field in its typed request body relative to what the endpoint actually expects — this gets fixed as part of wiring the hook up for real use.

Online/upload conflict detection

• Closed elections only. Open elections get no voter-identity conflict check at all — every row is trusted and submitted as-is (the synthetic voter_id open elections still get, per above, exists purely for roll bookkeeping and is never checked against anything).
• No new backend endpoint. The dry run is client-side: after parsing, fetch the election's voter roll via the existing roll-listing endpoint/hook and cross-reference each row's voter_id against whether that roll entry has already submitted.
• Backend enforcement is unchanged — the upload endpoint already rejects an already-voted voter's row server-side, independent of anything the client checks. The dry run is a pre-commit UX convenience layered on top of that already-authoritative check, not a replacement for it.
• The cross-reference runs automatically right after parsing, folded into the same preview pass the admin already reviews — not a separate explicit step.
• On confirm, the client filters out rows already known to conflict before the real submit, so "skipped" (informational, pre-confirm) and "failed" (a hard error, post-submit — including the narrow race-condition case where a voter votes online between preview and confirm) stay distinct, non-overlapping statuses in the final per-row report.
• The dry run exposes the full list of conflicting rows/voter IDs (not just a count), so the dialog can show the admin exactly which rows will be skipped.

Frontend entry point & dialog

• Entry point: a new "Upload Ballots" button next to the existing "Add Voters" button on the Manage Voters admin page — gated by the existing canUploadBallots role permission and by the election's allowed_submit_types including paper-ballot admin upload. No new page, and no new Sidebar navigation item — a dedicated-page variant was prototyped and explicitly rejected in favor of this.
• Dialog layout: stacked (not toggled) paste-CSV textarea above a single file-select button, mirroring the existing "Add Voters" dialog's layout exactly. No separate CSV/JSON toggle — file extension picks the parser; pasted text always goes through the CSV parser.
• Preview → confirm, one table, not a wizard: clicking "Preview" (or selecting a file) runs parsing, structural validation, and (for closed elections) the online/upload conflict dry run, then populates a single results-table component (the same one already used elsewhere for tabular admin data) with each row's pre-upload status: ready, skipped (inline, as a normal row status — not a popup), or error. Clicking "Confirm & Upload" does not navigate to a new screen; it flips those same rows in place to their final status as the batch upload reports progress. A four-step wizard variant (select → review → confirm → results) was prototyped and explicitly rejected as unnecessary — one table covers all three states.

Testing Decisions

• Only test observable behavior through real interfaces — the upload HTTP endpoint's request/response contract, the settings validation function's accept/reject behavior, and (for the dialog) what the user can see and click — not internal call structure.
• Backend: new test coverage for
  • allowed_submit_types enforcement in the ballot-submission pipeline — an admin-submitted ballot is rejected with a 400 when the election's settings don't include submitted_via_admin, and accepted when they do (including the default-array case where the field is absent entirely).
  • electionSettingsValidation rejecting an allowed_submit_types value that resolves to an empty array.
  • The existing bulk-upload endpoint's existing already-voted/per-row-rejection behavior for closed elections (already covered, but exercise it in combination with the new setting to confirm the two checks compose rather than interfere).
  • Prior art: the existing ballot/roll test suite already exercises election settings, roll creation, and bulk/anonymized ballot flows end-to-end against a real (test) database via the shared test helper and fixture election objects — new tests should follow that same pattern (build a fixture Election with the relevant settings, drive it through the real controller, assert on the persisted/returned result) rather than mocking the pipeline internals.
• Frontend/shared: unit tests for the two new CVR parsers (CSV and JSON) — valid round-trip of an exported file, a column/structure mismatch producing a whole-file reject, a closed-election missing/invalid voter_id producing a per-row error, an open-election blank voter_id being auto-generated — plus the extracted computeRaceOrder/encodeBallotRow utilities (a race missing from a row encodes as all-undecided). Prior art: the existing ordered-vote codec has direct unit-test coverage in this style already.
• E2E (Playwright): a new spec exercising the full admin flow — enable "Paper ballots (admin upload)" in Settings (draft only, can't be the only channel disabled), open "Upload Ballots" from Manage Voters, paste a small valid CSV, confirm the preview table, upload, and verify the ballots appear in results — plus a case verifying the "Vote" button/banner behavior when "Online (browser)" is disabled. Prior art: the existing voter-list and rolls E2E specs already drive the Manage Voters page and roll-based auth flows end-to-end; follow their setup/fixture conventions.

Out of Scope

• Non-vote ballot vocabulary (real-0 vs. blank vs. abstain vs. spoiled) — a separate, unresolved policy discussion this feature explicitly should not block on.
• Multi-parser auto-detection ("try every parser and see what fits") beyond a plain CSV-vs-JSON file-type check.
• Any new per-ballot origin-tracking field — already fully satisfied by the existing ballot-history action-type mechanism.
• The actual Discord ballot-submission integration — this spec only covers the settings/enforcement scaffolding that allows it and a feature flag that can expose the option in the UI later.
• Fuzzy/approximate candidate-name matching for CSV column validation — matching is exact-name only for v1; a separate, currently-unwired name-matching/clustering utility exists elsewhere in the codebase as a prototype but isn't part of this feature.
• Per-row fine-grained control over the online/upload conflict skip behavior (e.g. letting an admin selectively force-include a flagged row) — deferred entirely, not designed here.
• Any change to the public-archive Upload Elections importer's own format inference or filename-based election creation — untouched except for being rewired onto the shared utilities/parser-output type described above.

Further Notes

• This spec is the stated destination of Allow election admins to bulk upload ballots #810's wayfinder map; implementation is intentionally a separate, later effort and is not scoped here.
• The dual-gate design — a role permission (canUploadBallots) and an election-level setting (allowed_submit_types), checked independently rather than one replacing the other — is significant enough to warrant its own ADR alongside the vocabulary this issue introduces (Submission Channel, Paper Ballots, and the deliberate distinction from the existing Ballot Source concept). Recording that doc update is expected to land as a small, separate docs-only PR alongside (or just ahead of) implementation, not as part of this spec.
• A throwaway three-variant UI prototype for the Upload Ballots dialog exists on a fork branch (prototype/1601-upload-ballots-dialog-ux) — useful as a reference for the dialog's structure, not intended to land as-is.
• Original issue task list called for a dedicated /admin/ballots page; that idea was explicitly tried and rejected during grilling in favor of the "Upload Ballots" button on the existing Manage Voters page. This spec supersedes that part of the original issue body.

---

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Lob7JDFkjxL44NfHgXZMgY