Ranked home feed, event announcements, event analytics, chat media serving, street-level addresses (#100, #118, #122)

docs/location-coarsening-assessment.md

@@ -22,6 +22,24 @@ The owner's own detail/list (`mine: true`) and the create-time idempotency
 snapshot also carry full precision; routing to a jurisdiction uses the precise
 `geom`.
 
+**A second stored copy exists as of migration 0176 (issue #100).** `posts.geom`
+denormalises the linked report's or event's point onto the post row so the
+ranked home feed's proximity pool is one bounded KNN scan. It is NOT a new
+class of data and NOT a new exposure: the value is copied server-side from a
+coordinate this platform already publishes at full precision on the linked
+report or event, it is never populated from a client-supplied coordinate, and
+it is never projected into any DTO — it only orders the feed, and the feed
+emits `PostDTO`, which carries no post-level coordinate at all. The only
+derived value that leaves the server is the ranking score.
+
+Consequence for any future coarsening decision: rounding the projected
+`lat`/`lng` would NOT cover `posts.geom`, which would keep ordering the feed at
+full precision. That is almost certainly the desired behaviour (proximity
+ranking is the point), but it must be a stated decision rather than an
+oversight, and a policy that requires coarsening at REST — not just in
+transit — has to cover this column, `reports.geom`, `cleanups.geom` and
+`users.last_activity_geom` together.
+
 ## Is a backend-only coarsening possible without a shared-contract change?
 
 **Technically yes.** `ReportDTO.lat/lng` and `ReportPinDTO.lat/lng` are

docs/out-of-band-indexes.md

@@ -104,6 +104,71 @@ statement through the exported `explainSuggestFollows`. The offline half —
 that the emitted SQL really is a `CROSS JOIN LATERAL` and not a same-level
 cross join — is `test/unit/social-suggest-sql.test.ts`.
 
+### `posts.geom` backfill (migration 0176, issue #100) — data, not an index
+
+`posts` is NOT a hot table, so `posts_geom_gist` and
+`posts_author_public_recent_idx` are built inline by migrations 0176 and 0177
+and need nothing here. What DOES need an out-of-band run is the **backfill**:
+0176 adds the column but deliberately populates no rows, because one `UPDATE`
+over the whole table inside the migration's single transaction is a lock
+hazard. The migration RAISEs a `WARNING` when any backfillable post is still
+unpopulated.
+
+Run it once the deploy is healthy — keyset-paged, idempotent, safe to re-run
+and safe while the API serves traffic:
+
+```sh
+sudo -n docker exec compose-api-1 node dist/db/backfill-post-geom.js
+```
+
+Nothing breaks without it: a `NULL` `posts.geom` yields a `NULL` `distance_km`,
+the ranker simply scores no proximity term for that post, and the in-network
+and recent-public pools still fill the feed. Only the *nearby* pool is degraded.
+
+Verify:
+
+```sql
+SELECT count(*)
+  FROM posts p
+  LEFT JOIN reports  r ON r.id = p.report_id
+  LEFT JOIN cleanups c ON c.id = p.event_id
+ WHERE p.geom IS NULL AND COALESCE(r.geom, c.geom) IS NOT NULL;
+```
+
+must return `0`. (The `COALESCE(...) IS NOT NULL` term matters: a post linked to
+a row whose own `geom` is null is not backfillable, and counting it would make
+this check permanently unsatisfiable. The migration's `DO` block uses the same
+predicate.)
+
+**`posts.geom` is an insert-time snapshot, not a live mirror.** It is written
+once by `createPost` and never updated: there is no trigger and no relocation
+hook, and the backfill only touches rows where `geom IS NULL`. If a report or
+cleanup is later moved to a new coordinate, every post already linked to it
+keeps ranking against the OLD point indefinitely. That is acceptable for feed
+proximity (the post was about the place as it was), but it is a deliberate
+property, not an oversight — if live tracking is ever wanted, the relocation
+paths must update the derived posts explicitly.
+
+If `posts` has grown large enough that an inline `CREATE INDEX` would be
+disruptive, build both indexes with `CONCURRENTLY` BEFORE deploying — the
+migrations' `IF NOT EXISTS` guards then no-op:
+
+```sql
+CREATE INDEX CONCURRENTLY IF NOT EXISTS posts_geom_gist
+  ON posts USING gist (geom)
+  WHERE geom IS NOT NULL AND deleted_at IS NULL AND reply_to_id IS NULL
+    AND visibility = 'public';
+
+CREATE INDEX CONCURRENTLY IF NOT EXISTS posts_author_public_recent_idx
+  ON posts (author_id, created_at DESC, id DESC)
+  WHERE deleted_at IS NULL AND reply_to_id IS NULL AND visibility = 'public';
+```
+
+The nearby pool's plan is asserted by
+`test/integration/feed-ranked-pg.test.ts`, which `EXPLAIN`s the real statement
+through the exported `explainFeedCandidates` and requires
+`posts_geom_gist` with no `Seq Scan on posts`.
+
 ### `media_assets_orphan_sweep_idx` (migration 0098, audit H13)
 
 Back the hourly orphan sweep's candidate scan (`findOrphans` /`deleteOrphan` in

docs/retention-cleanup.md

@@ -12,7 +12,8 @@ previously accumulated forever (no existing path deleted them).
 A `retention.sweep` cron in the **media-worker** (the same process that runs
 `orphan.sweep` and `chat.partition.maintenance`) deletes expired rows from the
 tables below. Every lane but `inbound_emails` deletes already-expired rows only
-and needs no schema of its own.
+and needs no schema of its own (`geocode_cache` brings its own index, created
+with the table).
 
 | Table | Rows deleted | Source schema |
 |---|---|---|
@@ -21,6 +22,7 @@ and needs no schema of its own.
 | `sessions` | `expires_at < cutoff` | `services/api/src/db/schema/sessions.ts` |
 | `idempotency_keys` | `created_at < now - 48h` (`RETENTION_IDEMPOTENCY_MS`) | `services/api/src/db/schema/idempotency.ts` |
 | `notifications` | `created_at < now - 90d` (see F088 below) | `services/api/src/db/schema/notifications.ts` |
+| `geocode_cache` | `resolved_at < now - 180d` (`GEOCODE_CACHE_TTL_MS`, see below) | `services/api/src/db/schema/geocode_cache.ts` |
 | `inbound_emails` | `archived_at < now - 180d` (see H10 below) | `services/api/src/db/schema/inbound_emails.ts` |
 
 `cutoff = now - grace`, where `grace` defaults to **1 hour** past expiry (so a
@@ -141,6 +143,34 @@ lane (`src/routes/chat-gateway-wiring.ts`) is the remaining step.
 
 ---
 
+## `geocode_cache` — bounding the reverse-geocode cache (0179)
+
+`geocode_cache` (migration `0179_address_resolution.sql`) is written by the
+address ladder on every resolve, and the write path is reachable by an
+UNAUTHENTICATED caller (`POST /map/resolve-address`, 30/min/IP). Its TTLs are
+applied on READ — an expired row is served as a miss and OVERWRITTEN in place —
+so nothing in `services/api/src/services/geocode-cache.ts` ever deletes a row,
+and a point that is resolved once and never visited again stays forever. Rows are
+tiny and derived (a public coordinate → a public address line, no user, report or
+event reference, outside the erasure lane by construction), but "tiny × forever"
+is still unbounded, so the sweep owns the deletion side.
+
+| What | Value |
+|---|---|
+| Table | `geocode_cache` |
+| Predicate | `resolved_at < now() - 180 days` (`GEOCODE_CACHE_TTL_MS`, the POSITIVE TTL — imported from `@civfix/api/geocode-cache` so the two can never drift) |
+| Index | `geocode_cache_resolved_at_idx` (created by 0179) |
+| Cron | the existing `retention.sweep` (`37 4 * * *`, daily) |
+| Batching | `DELETE … WHERE point_key IN (SELECT point_key … LIMIT n)`, drained page-wise like the other lanes |
+| Tuning | `runRetentionSweep({ geocodeCacheRetentionMs })` |
+
+The cutoff is the POSITIVE TTL deliberately, even though a non-chain row (a chain
+miss) stops being SERVED after 15 minutes: once past 180 days a row cannot be
+served on any path, so deleting it loses nothing, and one cutoff keeps the lane a
+single index scan.
+
+---
+
 ## F106 — `inbound/failed/` poison-message store (adminmail-a)
 
 The inbound-mail processor parks parse-poisoned or over-cap `.eml` objects under

docs/security/2026-07-24-full-backend-security-review.md

@@ -322,7 +322,7 @@ Everything an operator has to do by hand, in order, plus the two infra facts and
 node dist/db/migrate.js        # == pnpm --filter @civfix/api db:migrate
 ```
 
-`drizzle/` holds **158 files**, `0000_extensions.sql` … `0175_forward_template_settings.sql`. The nine rows
+`drizzle/` holds **162 files**, `0000_extensions.sql` … `0179_address_resolution.sql`. The nine rows
 below are exactly what this change set adds — `0052`–`0060`, contiguous, no gaps — and everything from
 `0000` through `0051_social_posts.sql` predates it. (`0060` arrived later than the rest, with the feed
 redesign; it is listed here because this table is the single operator runbook. `0061`–`0064` arrived
@@ -563,6 +563,14 @@ foreign key into `organizations` from `0105`.
 | `0174_mail_messages_outbound_snapshot.sql` | adds `mail_messages.html` and `mail_messages.kind`, both nullable `text`, with `ADD COLUMN IF NOT EXISTS` and nothing else: no CHECK, no index, no backfill, so it is catalog-only on a cold table and takes no scan. `html` stores the exact HTML part handed to the mailer so a resend replays the message the city actually received instead of a bare-text, 64KB-truncated copy; `kind` records what an outbound row is (`packet` | `discussion` | `followup` | `digest` | `compose` | `reply` | `resend`) and is NULL on every inbound row and on every row written before this file. No CHECK is deliberate and matches the other free-text discriminators on this table: a value the code stops writing must never turn an old row un-readable. The operator send gate reads `COALESCE(kind, 'packet') = 'packet'`: a NULL kind — every row written before this file — counts AS a packet, deliberately, so every report routed before the deploy keeps its "already routed" gate, while a citizen's `@city` discussion forward (written as `discussion`) stops counting as one | Routing a report 500s on `column "html" does not exist` — the outbound INSERT names both columns unconditionally — so Approve & send, operator replies, resends and the outreach digest all fail; the packet gate itself still refuses correctly, because a missing `kind` column errors before it can misread |
 | `0175_forward_template_settings.sql` | new `forward_template_settings`, the singleton row (`id smallint PRIMARY KEY CHECK (id = 1)`) holding the platform-wide default forwarding email template. It sits between a jurisdiction's own `forward_*_template` (0050) and the built-in default in `@civfix/shared`, so a packet resolves jurisdiction → this row → built-in. Both template columns are nullable (NULL = fall through) and `updated_by` is deliberately not a FK to `users`, since the authoritative trail is the `mail.forward_template_set` audit row written in the same transaction. `CREATE TABLE IF NOT EXISTS` on a table with no rows: no lock of consequence | The operator Mail page 500s on its default-template read and save (`relation "forward_template_settings" does not exist`), and every report forwarded to a jurisdiction without its own template 500s too, because the packet builder reads the row before it renders |
 
+| `0176_posts_geom.sql` | adds `posts.geom` (nullable `geometry(Point,4326)`, no default, so a catalog-only ADD COLUMN with no table rewrite) plus the partial GIST `posts_geom_gist` over live public rows, denormalising the linked report's or event's point onto the post so the ranked home feed's proximity pool is one bounded KNN scan instead of two joins. `posts` is NOT on the hot-table list in `docs/out-of-band-indexes.md`, so the index builds inline; build it with `CREATE INDEX CONCURRENTLY` first if `posts` has grown, and the `IF NOT EXISTS` guard makes the migration a no-op. No backfill in the file - a single UPDATE over the whole table inside one transaction is a lock hazard - so a DO block RAISEs a WARNING while attached rows are still unpopulated and `node dist/db/backfill-post-geom.js` (keyset-paged, idempotent, safe under live traffic) fills them after the deploy is healthy. PRIVACY: no new class of data, a copy of a coordinate already published at full precision on the linked report or event, never client-supplied, never served to clients | Nothing breaks: `p.geom IS NULL` simply yields a NULL `distance_km`, the ranker scores no proximity term, and the feed falls back to the in-network and recent-public pools. Skipping only the BACKFILL means existing attached posts score no proximity until it is run |
+
+| `0177_posts_author_public_recent_idx.sql` | adds the partial index `posts_author_public_recent_idx (author_id, created_at DESC, id DESC) WHERE deleted_at IS NULL AND reply_to_id IS NULL AND visibility = 'public'` - the author-keyed twin of `0071_posts_toplevel_recent_idx`, backing the ranked feed's in-network candidate pool (self + followees). Neither `posts_author_created_idx` (0051, not partial on reply_to_id/visibility, no id tiebreak) nor `posts_toplevel_recent_idx` (0071, not author-keyed) serves that shape. Same not-a-hot-table reasoning as 0176 | The in-network pool sorts instead of stopping at its LIMIT. Correct, slower; no functional change |
+
+| `0178_event_announcements.sql` | widens `broadcasts_kind_check` to accept the new `announcement` kind (event announcements ride the existing host-broadcast pipeline rather than a parallel table), adds the partial `broadcasts_announcement_public_idx (cleanup_id, created_at DESC, id DESC) WHERE kind = 'announcement'` that backs the public per-event announcements list, and re-creates `broadcasts_scrub_idx` with `kind <> 'announcement'` so the retention scrub can never NULL an announcement body - an announcement is permanent public event content rendered on the event page, not a one-shot email. `broadcasts` is NOT on the hot-table list in `docs/out-of-band-indexes.md`, so both indexes build inline; `IF NOT EXISTS` makes an out-of-band CONCURRENTLY build a no-op | Every `createEventAnnouncement` fails on the CHECK constraint. Applying the CHECK but not the scrub-index swap still works (the repository query carries the same predicate), it just scans more rows |
+| `0179_address_resolution.sql` | the address-resolution overhaul: creates `geocode_cache` (a read-through cache of reverse geocodes keyed by the shared 5-decimal point key, with its `geocode_cache_resolved_at_idx`; derived public data only, no user/report/event reference, TTL applied on read rather than by a cron), adds `cleanups.address_source` (resolved | edited | manual) and backfills it to `manual` for every event that already has an address, and adds `reports.addr_source` + `reports.addr_precision`. All three CHECK constraints are `NOT VALID` (new columns, so they hold by construction) and are listed in the outstanding-VALIDATE set below. No new index on a hot table: both `reports` columns are plain row columns | `POST /map/resolve-address` and every create path still answer, but uncached (the cache swallows its own errors) and with no provenance persisted; a new client's event create fails on the missing `cleanups.address_source` column |
+
+
 **Deferred to the NEXT release** (expand/contract, `docs/migrations-expand-contract.md`): 0.43.0
 stops every code path from reading `user_verification` but does NOT drop it — a `DROP TABLE` in the
 same release would make the previous api color raise `relation does not exist` on the feed, the
@@ -590,9 +598,10 @@ either way — `org_verifications` documents are claimed with it.
 **Deferred to a later release, out of band** (record them in the expand/contract ledger): the six
 `NOT VALID` CHECKs `0107` adds on `cleanups` and `0110`'s `media_assets_purpose_expanded` still need
 `ALTER TABLE … VALIDATE CONSTRAINT`, and so do `0164`'s
-`users_primary_organization_fk` and `posts_organization_fk`. They are correct as `NOT VALID` — the
-constraint is enforced for every new row — and validating takes a scan that does not belong in a
-deploy transaction.
+`users_primary_organization_fk` and `posts_organization_fk`, and `0179`'s
+`cleanups_address_source_chk`, `reports_addr_source_chk` and `reports_addr_precision_chk`. They are
+correct as `NOT VALID` — the constraint is enforced for every new row — and validating takes a scan
+that does not belong in a deploy transaction.
 | `0096_cleanup_guests.sql` | guest event RSVP (contract 0.38.0): `cleanup_guests` (event-scoped, contact-bearing attendance rows; SHA-256 manage-token hash only; partial unique on the active `(cleanup_id, contact_key)`), `guest_otps` (event-scoped one-time codes, hash only), `sms_opt_outs` (STOP suppression list, kept indefinitely). Three brand-new empty tables; no existing table touched | `POST /v1/cleanups/:id/guest-rsvp/*` and `GET /v1/cleanups/:id/guests` 500 on a missing relation, and `going` cannot include guests |
 | `0097_media_assets_served_key.sql` | adds `media_assets.served_key` (audit C1): the worker-owned key the processed object is published to, so the client-writable upload key is never served after `ready`. Catalog-only nullable `ADD COLUMN` on a hot table. **Post-deploy:** run `node dist/db/backfill-served-key.js` immediately and again ~20 min later; pre-existing `ready` media reads as not-found until it completes | Every existing photo 404s until the backfill runs; new uploads work |
 | `0098_sweep_predicate_indexes.sql` | idempotent guard that warns while `media_assets_orphan_sweep_idx` is missing (hot table; build it out of band with `CREATE INDEX CONCURRENTLY`, see `docs/out-of-band-indexes.md`) plus two small inline `email_otps` indexes (`expires_at`, partial `consumed_at`) backing the OTP retention lane (audit H13/M) | Both sweeps sequential-scan; correctness unaffected |