Skip to content

fix: attribute 985k blank regions from coordinates, unblocking migration 23 - #10

Merged
keonik merged 1 commit into
mainfrom
fix/attribute-blank-regions
Sep 14, 2026
Merged

keonik merged 1 commit into
mainfrom
fix/attribute-blank-regions

Conversation

@keonik

@keonik keonik commented Sep 14, 2026

Copy link
Copy Markdown
Owner

With the guard corrected, migration 23 reported the real obstacle:

985,634 row(s) have no region while rows also exist for 0, BE, IH, IN, MI, O, ON, PA, PJ

About 17% of the table has no state at all.

Why not just stamp them OH

That is the same mistake as the backfill already reverted on this branch, just
quieter. Border-county extracts genuinely contain Indiana, Pennsylvania and
Michigan addresses, and a mislabel is permanent and invisible.

us_states already holds TIGER boundaries behind a GIST index — it is what
/states/lookup answers from — so each blank row is asked where it actually
is
and takes the state it sits in.

Verified against a fixture spanning three states:

before:  <blank>:4  OH:1  PA:1  oh:1
after:   OH:4  IN:1  MI:1  PA:1

Migration 23: attributed 4 of 4 blank region(s) from their coordinates

The blanks in Ohio became OH, the one in Indiana became IN, the one in
Michigan became MI. A single stamp could not have been right for all three.

Rows that cannot be attributed

If a row is still blank afterwards, its coordinates fall outside every state
boundary — the location is wrong, not the state missing. The migration names
the count and stops rather than inventing a state:

1 row(s) still have no region after attribution: their coordinates fall outside
every US state boundary, so the location is wrong rather than the state missing.
Fix or delete them, then re-run

Verified with a row in the middle of the Atlantic.

Cost

One pass of ~1M point-in-polygon lookups against a GIST-indexed boundary table,
inside the migration's existing transaction. It holds ROW EXCLUSIVE, so reads
are unaffected; the ACCESS EXCLUSIVE statements still come last.

Related

/coverage reports blank regions as Ohio — it does
COALESCE(NULLIF(region,''),'OH'). That is why nobody saw a million stateless
rows, and why the real Ohio count is nearer 4.79M than the 5.78M it shows. Next
PR fixes that and adds the data-quality endpoint that would have surfaced all
of this directly.

🤖 Generated with Claude Code

With the guard corrected, migration 23 reported the real obstacle: 985,634 rows
-- about 17% of the table -- have no region at all, alongside rows for nine
other regions.

Stamping those 'OH' because this is "the Ohio dataset" is the same mistake as
the backfill already reverted on this branch, just quieter. Border-county
extracts genuinely contain Indiana, Pennsylvania and Michigan addresses, and a
mislabel is permanent and invisible.

us_states already holds TIGER boundaries behind a GIST index -- it is what
/states/lookup answers from -- so each blank row is asked where it actually is
and takes the state it sits in. Evidence rather than assumption. Verified
against a fixture spanning three states: blanks in Ohio became OH, the one in
Indiana became IN and the one in Michigan became MI. A single stamp could not
have been right for all three.

Rows still blank afterwards sit outside every state boundary, which means the
coordinates are wrong rather than the state missing. The migration names the
count and stops rather than inventing a state for them; the message says what
to do. Verified with a row in the middle of the Atlantic.

The old unconditional backfill is gone, not just bypassed -- left in place it
would have stamped OH over rows attribution deliberately left alone.

Also worth flagging: /coverage reports blank regions as Ohio, because it does
COALESCE(NULLIF(region,''),'OH'). That is why nobody saw a million stateless
rows. It is the next thing to fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@keonik
keonik merged commit e689ad1 into main Sep 14, 2026
3 checks passed
@keonik
keonik deleted the fix/attribute-blank-regions branch September 14, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant