Skip to content

fix: migration 23 is failing in production and would corrupt data if it ran - #9

Merged
keonik merged 1 commit into
mainfrom
fix/migration-23-unblock
Sep 14, 2026
Merged

fix: migration 23 is failing in production and would corrupt data if it ran#9
keonik merged 1 commit into
mainfrom
fix/migration-23-unblock

Conversation

@keonik

@keonik keonik commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Migration 23 has failed on every boot since it deployed, and production is
still on schema 22:

"schema_version": 22,
"migration_error": "failed to run migration 23: refusing to backfill blank
  regions to OH: rows already exist for 0, BE, IH, IN, MI, O, ON, PA, PJ"

Two separate bugs, both mine, both in #7. This branch carries the fix that was
pushed to that branch after it had already merged, so it never reached main.

1. The guard is right, and too strict

It refuses whenever any non-OH region exists. Production has nine, so the
migration can never run. Other states on their own are fine — what is genuinely
ambiguous is other states and blanks together, because then a blank could
belong to any of them. The two conditions are now separate; other states are
logged and left alone.

2. The backfill would have relabelled every out-of-state row as Ohio

SET region = 'OH' WHERE region IS NULL OR region = '' OR region <> 'OH'

Production is not single-state and never was:

OH 5,775,659   IN 249   PA 170   MI 59
oh 2   BE 9   ON 2   O 1   IH 1   0 1   PJ 1

That statement stamps OH over 478 genuine Indiana, Pennsylvania and Michigan
addresses — permanently, with nothing in the logs. The over-strict guard is
the only reason it did not run.
Fixing the guard without fixing this would
have shipped the corruption on the next deploy.

It came from collapsing two statements into one to avoid a second table
rewrite — an optimisation that changed what the statement meant. It is two
statements again, with a comment saying why they cannot be merged.

Verified against production's actual data

before:  0:1 BE:1 IN:1 MI:1 O:1 oh:1 OH:1 ON:1 PA:1
after:   0:1 BE:1 IN:1 MI:1 O:1      OH:2 ON:1 PA:1

oh merges into OH; everything else is untouched; the migration completes.

Also removed

A pre-check for (hash, region) collisions after case normalisation. It can
never fire — the constraint being replaced makes hash unique on its own, so
no two rows share a hash in any region. Confirmed by trying to construct the
collision and being refused by the old constraint, which is now a test.

The junk codes (BE, ON, O, IH, 0, PJ — 15 rows) are left alone
deliberately. They come from the legacy loader truncating a state name to two
characters, fixed going forward in #7, but inventing a correction for them
would be the same guessing this migration refuses to do elsewhere. A follow-up
endpoint will surface them instead.

Note on current impact

Imports are currently refused with "migrations are still running", which is the
RequireSchemaVersion guard from #7 working as intended — the ingest path needs
the index migration 23 creates. Search and geocoding are unaffected.

🤖 Generated with Claude Code

Smoke-testing the new /coverage endpoint against production showed the data was
never single-state:

    OH 5,775,659   IN 249   PA 170   MI 59
    oh 2   BE 9   ON 2   O 1   IH 1   0 1   PJ 1

The backfill read:

    SET region = 'OH' WHERE region IS NULL OR region = '' OR region <> 'OH'

which stamps OH over Indiana, Pennsylvania and Michigan. Permanently, with
nothing in the logs. That came from collapsing two statements into one to avoid
a second table rewrite -- an optimisation that changed what the statement
meant. It is two statements again, and the comment says why they cannot be
merged: case normalisation touches only rows whose case differs, the backfill
touches only blanks, and neither is the full-table rewrite the single statement
appeared to avoid.

The guard added alongside it is the only reason this was caught rather than
deployed, but it was too strict in the other direction: it refused outright
whenever any non-OH region existed, so migration 23 could never have run in
production at all. The two conditions are now separate. Other states alone are
fine and are logged. Other states AND blanks is the genuinely ambiguous case,
because a blank could belong to any of them, and that still refuses.

Also removed a pre-check for (hash, region) collisions after case
normalisation. It can never fire: the constraint this migration replaces makes
hash unique on its own, so no two rows share a hash in any region. Verified by
trying to construct the collision and being refused by the old constraint --
which is now a test.

Verified against a database carrying the real region distribution: 'oh' merges
into 'OH', and IN, PA, MI and the junk codes are left exactly as they were.

The junk codes are left alone deliberately. BE, ON, O, IH, 0 and PJ come from
the legacy loader truncating a state name to two characters -- fixed going
forward in this branch -- but inventing a correction for 15 existing rows would
be the same guessing this migration refuses to do elsewhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@keonik
keonik merged commit 1a22d99 into main Sep 14, 2026
3 checks passed
@keonik
keonik deleted the fix/migration-23-unblock branch September 14, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant