fix: attribute 985k blank regions from coordinates, unblocking migration 23 - #10
Merged
Merged
Conversation
With the guard corrected, migration 23 reported the real obstacle: 985,634 rows -- about 17% of the table -- have no region at all, alongside rows for nine other regions. Stamping those 'OH' because this is "the Ohio dataset" is the same mistake as the backfill already reverted on this branch, just quieter. Border-county extracts genuinely contain Indiana, Pennsylvania and Michigan addresses, and a mislabel is permanent and invisible. us_states already holds TIGER boundaries behind a GIST index -- it is what /states/lookup answers from -- so each blank row is asked where it actually is and takes the state it sits in. Evidence rather than assumption. Verified against a fixture spanning three states: blanks in Ohio became OH, the one in Indiana became IN and the one in Michigan became MI. A single stamp could not have been right for all three. Rows still blank afterwards sit outside every state boundary, which means the coordinates are wrong rather than the state missing. The migration names the count and stops rather than inventing a state for them; the message says what to do. Verified with a row in the middle of the Atlantic. The old unconditional backfill is gone, not just bypassed -- left in place it would have stamped OH over rows attribution deliberately left alone. Also worth flagging: /coverage reports blank regions as Ohio, because it does COALESCE(NULLIF(region,''),'OH'). That is why nobody saw a million stateless rows. It is the next thing to fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With the guard corrected, migration 23 reported the real obstacle:
About 17% of the table has no state at all.
Why not just stamp them OH
That is the same mistake as the backfill already reverted on this branch, just
quieter. Border-county extracts genuinely contain Indiana, Pennsylvania and
Michigan addresses, and a mislabel is permanent and invisible.
us_statesalready holds TIGER boundaries behind a GIST index — it is what/states/lookupanswers from — so each blank row is asked where it actuallyis and takes the state it sits in.
Verified against a fixture spanning three states:
The blanks in Ohio became
OH, the one in Indiana becameIN, the one inMichigan became
MI. A single stamp could not have been right for all three.Rows that cannot be attributed
If a row is still blank afterwards, its coordinates fall outside every state
boundary — the location is wrong, not the state missing. The migration names
the count and stops rather than inventing a state:
Verified with a row in the middle of the Atlantic.
Cost
One pass of ~1M point-in-polygon lookups against a GIST-indexed boundary table,
inside the migration's existing transaction. It holds ROW EXCLUSIVE, so reads
are unaffected; the ACCESS EXCLUSIVE statements still come last.
Related
/coveragereports blank regions as Ohio — it doesCOALESCE(NULLIF(region,''),'OH'). That is why nobody saw a million statelessrows, and why the real Ohio count is nearer 4.79M than the 5.78M it shows. Next
PR fixes that and adds the data-quality endpoint that would have surfaced all
of this directly.
🤖 Generated with Claude Code