Performance: batched statements, indexes and event-loop fixes (backend campaign 13)

[theobong]

What changed

Performance work, behavior-neutral: statements that ran once per row now run once per request, independent reads run together, seven indexes back slow lookups, several admin and host queries read index ranges instead of whole tables, and four spots that blocked the server's event loop (the slur filter, the CSV export guard, certificate fonts, map pin signing) now do a small fraction of the work. Nothing a user sees should change, except that some screens answer faster.

Before you start

• Where: staging (civfix.dev, admin.civfix.dev, the staging mobile build) after this merges to main
• Sign in as: your own accounts, A (host) and B (attendee), with emailed codes; an operator session on admin.civfix.dev
• Data: an event hosted by A with two ticket types, B registered, and a group chat; B follows A and a few others
• Stacked on Repo-wide abstraction and naming (backend campaign 12) #104 (repo-wide abstraction and naming). Read only this PR's diff against that branch, commit by commit.
• Size: 4399 counted lines across 93 files. Campaign PR; the author approved PRs over the 400-line cap for this cleanup. About two thirds of it is tests; each commit is one change.

Verify

Nothing new to verify: every step below is a regression check of a flow whose queries changed.

Regression

Hosting an event — [Web]

1. On civfix.dev/dashboard click "Create", give it a title, click "Next", pick a date and times, "Next", place the pin, "Next"
2. Name a slot, set "Spots", click "Add another slot", name it, then "Publish event" — Expect: the event page lists both under "Sign-up slots"
3. Back on the dashboard open the event's actions, "Edit", rename a slot and change its spots, "Save changes" — Expect: the event page shows the change
4. In the host console click "Tickets"; on the second type click "Move up" — Expect: the order swaps and stays after reload
5. Click "Delete" on a type nobody registered for and confirm "Delete" — Expect: "Ticket type deleted"
6. Click "Delete" on the type B registered for and confirm — Expect: "People have already registered for this type. Close its sales instead of deleting it." and the row stays
7. Under "Registration questions" click "Add a question", type a prompt, tick "Required", "Save" — Expect: "Questions saved"; edit the prompt, "Save" again — the new text stays after reload
8. As B open the event, leave the answer empty and click "Register" — Expect: "Please answer the required questions."; answer it and "Register" — Expect: "You're registered."

Check-in and messages — [Web] [Mobile]

1. As A (web) open "Check-in" in the host console — Expect: "Live counters" with "Checked in" and "Registered", and the "By ticket type" donut
2. Click "Check in" on a row under "Not checked in yet" — Expect: "Checked in" rises by one and that type's segment grows
3. As A (web) open "Messages", "New message", fill it in, pick "Everyone registered", click "Preview" — Expect: "… people will receive this" matches the roster; switch to "Checked in" and "Preview" — the number changes to the checked-in count
4. As A (mobile) open "Host tools", "Make announcement" — Expect: "Counting…" then "… people"; picking another audience updates "Send to … people"

Team and organizations — [Web] [Mobile] [Admin]

1. As A open the event's team, click "Invite", enter B's handle, choose co-host, "Send the invite" — Expect: "Invite sent"
2. As B open the profile, find "Invitations", click "Accept" — Expect: the invitation disappears and B shows on A's team as co-host
3. As an organization owner (web), open the organization, "Members", "Invite", "Send invite" — Expect: "Invitation sent"; change a member's role and confirm "Change role" — Expect: "Role updated"; "Remove" and confirm — Expect: "Removed …"
4. Admin: "Organizations", pick the organization, "Verification", "Verify as …" — Expect: "… verified · …"

Host portfolio — [Web] [Mobile]

1. As A open civfix.dev/manage/ — Expect: "Hosting summary" with "Events hosted", "Upcoming", "Registrations", "Checked in" as before
2. Set the time window to "Past", then "Load more" — Expect: past events, no duplicates
3. On mobile open "Event dashboard", "Past", "Show more" — Expect: the same events

Posts, chat and polls — [Web] [Mobile]

1. As A write a post, type @ and pick B, "Post" — Expect: B's "Notifications" shows "… mentioned you"
2. As B open A's profile and choose "Block", then as A mention B in a new post — Expect: B gets no mention notification
3. In the group chat open your message's actions, "Edit", change the text and send — Expect: the new text and "(edited)" for both
4. Attach a "Poll", fill the question and options, "Create poll"; vote — Expect: "1 vote"; open the actions, "Stop this poll" — Expect: "Poll · Final results"

Followers and volunteer hours — [Web] [Mobile]

1. Open a profile with many followers, tap "Followers", scroll to the end — Expect: further pages load in order with no duplicates; the same for "Following"
2. Open a profile with credited hours — Expect: "Total volunteer hours" matches the "Service record"
3. Tap the jurisdiction chip — Expect: the leaderboard, your row marked "You"
4. Under "Official transcript" click "Prepare transcript", then "Open PDF" — Expect: "Transcript ready" and a PDF whose hours equal the total

Exports and reports — [Web] [Mobile]

1. As A (web) open "Attendees", "Export", the attendees export, then "Download" when ready — Expect: the roster CSV with the right rows
2. Open the map and zoom into a cluster of photo reports until the list opens — Expect: "… reports" with a thumbnail on each photo row

Admin — [Admin]

1. On the Dashboard — Expect: the live map pins, the Events tile ("… upcoming events", "Live now", "Attending") and the Reports tile as before
2. "See analytics" — Expect: the KPI strip ("Pins this month", "Resolved", "Cleanups planned", "Events this month", "New users") with last month's comparison
3. Open Reports and click each filter chip — Expect: each chip's count equals the rows listed after "Load more"
4. Open "Jurisdictions", try "Needs mapping", "Routed", "All", each sort, and "Load more" — Expect: same counts and order as before, no duplicates
5. Select a jurisdiction, enter a default contact and a discussion handle, "Save draft" — Expect: "Draft saved for …"; then "Save & route" — Expect: "Contacts saved for … · discovery task closed"
6. Enter a discussion handle that differs only in letter case from an existing user's handle — Expect: refused, as before
7. Open a report and click "Send to city" — Expect: "Sent to " and the email carries the photos in order

Findings addressed

Campaign PR 13 (performance). Each change carries its measurement or its algorithmic argument; none changes a result, an order, an error or a response except the failure-only differences listed under decisions.

Round trips (per-row SQL and sequential awaits)

• Event slots: up to 20 INSERTs in the create transaction become 1 (unnest).
• Ticket type reorder: up to 20 UPDATEs become 1; a repeated id still ends on its last position.
• Registration questions: up to 20 statements become 2 (one UPDATE for kept questions, one INSERT for new ones).
• Check-in counters (polled up to 120 times a minute): 3 serial reads become 1 concurrent wave.
• Team invite accept: the seated role comes back from the upsert, 2 statements become 1.
• Broadcast preview: the audience count was up to 12 paged queries at the default cap (up to 200 at the maximum), each re-running a DISTINCT keyset scan and shipping ids; now 2 counts in one wave, equal to the old total because the cap never exceeds the old page limit.
• Reminder sweep: event contexts load in one query (3N+1 round trips become 2N+2 per sweep).
• Metrics rollup: the event time zone comes with the page (3 statements per event become 2).
• Organization gates (9 endpoints): the gate read an organization row with 3 correlated subqueries plus an hours aggregate; it now reads one primary-key row with the same role subquery.
• Account deletion: host-transfer audit rows ride the transfer statements (2+N statements become 2); post-commit object deletes and notifications run four at a time.
• Post mentions: 20 mentions went through 60 serial statements; now 1 block lookup and at most 4 bells in flight. The block query binds 3 parameters instead of 2N+1, so it is prepared once.
• Chat edit returns the edited row from the UPDATE (one fewer statement, and no longer probes every monthly partition by id); poll vote/close reads its meta in 1 statement instead of 2.
• Volunteer hours totals, leaderboard and certificate entries read concurrently (3, 4 and 2 round trips become 1).
• Jurisdiction contacts save: up to 7 statements become 2.
• The report packet to the city fetches up to 3 images ahead instead of one at a time (up to 20 serial storage GETs become about n/3 waves).

Indexes (migrations 0186–0192, none on a hot table)

• Followers pages: every follower edge was fetched and sorted; now an index range of one page, and the keyset compares the edge columns so the page seek is an index condition.
• Account deletion: two sequential scans of moderation items and one of team invites inside the erasure transaction become index probes.
• Jurisdiction health, outreach and discovery: the bounced-contact check read every mail event of the jurisdiction's threads per legacy email; now one expression-index probe.
• Admin event flag state: a cleanup's whole timeline per probe becomes at most one index entry.
• Push token prune: a sequential scan becomes an index probe.
• Broadcast delivery log: every page sorted all deliveries of the broadcast; now an index range.

Query shapes

• Host analytics hosted events: an OR of correlated EXISTS forced a scan of every event; now a union of three indexed lookups.
• Jurisdiction directory: the four decoration subqueries ran for every filtered jurisdiction (up to about 35,000); now only for the page's rows. The handle clash probe uses the unique handle index instead of scanning users.
• Admin reports counts: two passes over reports become one. Admin home summaries and recent pins and the analytics KPIs read index ranges instead of full tables.
• Ticket type delete: the in-use probe is scoped to the event, so it uses the event's roster index instead of scanning.

Event loop (CPU and synchronous I/O)

• Slur filter: a long run of a doubled letter backtracked quadratically: 716 ms of blocked event loop for 8,000 characters (a page block allows several such fields per request). Now 1.1 ms. Same accepted strings: 0 differences over 808,539 inputs (30.7 million pattern comparisons), confirmed by an independent 2.05 million-input fuzz in review.
• CSV export formula guard: 1,000 adversarial answers took 2.2 s of one synchronous stretch; now 4.6 ms. 0 differences over 1,000,000 inputs.
• Certificate fonts load asynchronously (the 4.5 MiB CJK font held the loop about 3 ms on first use); PDF bytes are sha256-identical on 6 fixed inputs.
• Map pins presign one key per pin instead of two (2,000 signed pins: about 1.8 s to 0.75 s of loop time; with the public media base used on staging and production, 4 ms to 1.6 ms).

Decisions for the reviewer

• Proof that no other statement changed: the offline SQL recorder (Backend test safety net: SQL transcripts, CSRF pairing, authz and characterization tests #92, run locally) was regenerated before and after: of 930 transcripts, 907 are byte-identical, 23 changed (exactly the statements named above) and 5 were added for the new repository methods; none was removed. The concurrency changes keep byte-identical transcripts because postgres.js dispatches in subscription order.
• Failure-only differences: if the mention block lookup fails, no mention bell goes out (before, bells ahead of the failure were already sent; the same error propagates). With several concurrent reads, when two fail the error reported is whichever rejects first. A failed reminder-context read fails the sweep before creating any reminder (creation is idempotent, so a retry ends in the same state). A certificate with a missing fonts directory rejects before drawing starts instead of at the first font, with the same error.
• Review found and fixed four issues before this PR opened: mention bells failed open if the batch block lookup was not wired (it now falls back to the per-pair check); two question saves diverged from the old per-row updates (two kept entries whose ids differ only in letter case, and a lone UTF-16 surrogate in the text, which now stores U+FFFD exactly as before instead of failing); and the image prefetch kept every loaded image until the packet was built (75 MiB peak for 20 images of 4 MiB instead of 11 MiB; consumed slots are now released, pinned by a garbage-collection test).
• Pool pressure: check-in counters, volunteer hours and broadcast counts now use up to 3 or 4 of the 10 pooled connections per request for a moment (the host insights compute goes from 7 to 9). Total connection time is unchanged; watch pool wait on staging.
• mail_events index (0189): the out-of-band index doc and CLAUDE.md do not list mail tables as hot, so it builds inline; the civfix-migration skill does. The table grows only with outbound mail and the build takes a SHARE lock (inserts wait, reads do not). If staging or production has a large mail_events, build it CONCURRENTLY first; the migration is then a no-op.
• Recent pins filter: the admin recent-pins query now skips reports with no creation time. No writer can store one (the column defaults to now and nothing sets it), so the result is the same unless a row was edited by hand.
• Not done, recorded: moving certificate rendering to a worker thread (83 ms median, 220 ms at 1,000 rows, on the event loop; designed, about 200 lines), capping unread counts, caching admin counts, capping admin timelines, skipping the thread re-read after sends, the data-export early return, the Argon2 thread pool, the error-report scrubber regex and moving inbound mail parsing off the API. Each changes an observable value, timing or security posture.
• Depends on Backend test safety net: SQL transcripts, CSRF pairing, authz and characterization tests #92: new unit tests use the SQL recorder helper that Backend test safety net: SQL transcripts, CSRF pairing, authz and characterization tests #92 adds; in merge order Backend test safety net: SQL transcripts, CSRF pairing, authz and characterization tests #92 lands first.

User-visible copy changes

None.

Tests changed

• New unit tests for every batching change (statement count, text and parameters through the recorder or the fake SQL helper), the linear-time regexes, the font read failure, the mention batch (including the fallback), the organization gates (404/403/suspended/409 on all 9 endpoints), the prefetch (order, errors, in-flight bound, memory release) and the index mirrors.
• New integration tests (run in CI): the seven indexes (definitions and plans), hosted event ids, question reconcile, ticket type reorder and delete, slots with timed windows, the host-transfer audit rows, and new cases in the admin home, analytics, reports and jurisdiction directory suites.
• Changed to follow the new shapes (no assertion weakened): the metrics rollup stubs return the time zone with each event; one organization security test moves its "lost the power in between" override from the profile read to the access read the gate now uses; two chat test comments describe the batch block lookup, which every blocks repository now implements.

Verification

• Every commit typechecks on its own (api and media worker).
• At this branch: pnpm typecheck, pnpm lint, check:sql, prettier clean; api unit suite by path 439 files, 7381 passed, 3 skipped; media-worker unit 300 passed, 2 failed (the known arm64 ffprobe helper issue fixed in Backend test safety net: SQL transcripts, CSRF pairing, authz and characterization tests #92).
• The SQL transcript proof above; differential fuzzing of every regex rewrite; PDF bytes compared before and after.
• An adversarial review (five passes: registration SQL, CPU and concurrency, erasure/posts/chat, broadcast/organizations/metrics, indexes and admin SQL) found no security issue; its four correctness and memory findings are fixed in the commits they belong to.
• The integration suite was not run (it needs Docker); CI runs it once this PR is retargeted to main.

Staging checks

• https://api.civfix.dev/readyz answers ready after migrations 0186–0192.
• Before deploying to production, check SELECT pg_size_pretty(pg_total_relation_size('mail_events')); if it is large, build mail_events_bounced_recipient_idx CONCURRENTLY first.
• EXPLAIN (ANALYZE, BUFFERS) on a followers page, a jurisdiction directory page and a ticket type delete shows the new indexes.
• Pool wait during a check-in session stays flat.

Not covered

• Nightly jobs (reminders, metrics rollup) are not triggered by hand; unit and integration tests cover them.
• Account deletion's host-transfer audit rows are visible only in the database (audit log), not in a screen.

🤖 Generated with Claude Code