feat: data-quality endpoint, and stop /coverage hiding stateless rows - #12
Merged
Conversation
Everything this endpoint reports is wrong in a way that raises no error. A search still returns rows, an import still reports success, and nothing in the logs says otherwise. Production carried eleven distinct region codes -- ON (Ontario), BE, IH, PJ, a bare 0, a lowercase oh -- and 985,634 addresses with no state at all. None of it was visible anywhere. It surfaced only because /coverage happened to group by region while being used for something else. GET /api/v1/admin/data-quality reports invalid region codes with a reason for each, blank region, county, city and postcode counts, and coordinates outside a loose envelope around the United States. Admin-only, and cached for thirty minutes: every check is an aggregate over the whole address table, and an operator refreshing a dashboard should not re-scan 5.8M rows each time. An import invalidates it, as it does coverage. Regions are classified in Go rather than SQL because the set of valid codes already lives in utils.IsUSStateCode, and a second copy inside a query is how the two drift apart. Note IsUSStateCode upper-cases before checking, so it accepts "oh" -- a value only counts as correct if it is a real code AND already canonical, since 'oh' and 'OH' are distinct keys to the uniqueness index. The first version of this missed that and reported the lowercase region as fine; the test caught it. Also fixes /coverage, which folded blank regions into the Ohio bucket with COALESCE(NULLIF(region,''),'OH'). That is the specific reason a million stateless rows went unseen and the Ohio count read about a million higher than it was. Blanks are now reported as (none), and the per-county drill-down no longer absorbs them either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything this reports is wrong in a way that raises no error. A search
still returns rows, an import still reports success, nothing appears in the
logs.
Production carried eleven distinct region codes —
ON(Ontario),BE,IH,PJ, a bare0, a lowercaseoh— and 985,634 addresses with no state at all.None of it was visible anywhere. It surfaced only because
/coveragehappenedto group by region while being used for something unrelated.
Reports invalid region codes with a reason for each, blank region / county /
city / postcode counts, and coordinates outside a loose envelope around the US
(a zero pair, a dropped minus sign, a transposed lat/lng).
Two decisions
Classified in Go, not SQL. The set of valid state codes already lives in
utils.IsUSStateCode, and a second copy inside a query is how the two driftapart.
Cached for 30 minutes. Every check is an aggregate over the whole address
table — the same per-request full-table scan the address search count query was
doing before it was fixed. An import invalidates it, as it does coverage.
A bug my own test caught
IsUSStateCodeupper-cases before checking, so it happily accepts"oh". Thefirst version therefore classified the lowercase region as correct and never
reported it. A value only counts as correct if it is a real code and already
canonical —
'oh'and'OH'are distinct keys to the uniqueness index, so acase difference is a real defect, not a cosmetic one.
Also: /coverage was hiding the problem it revealed
It did
COALESCE(NULLIF(region,''),'OH'), folding every blank region into theOhio bucket. That is the specific reason a million stateless rows went unseen,
and why the Ohio count read ~1M higher than reality (5.78M shown vs ~4.79M
actual). Blanks now report as
(none), and the per-county drill-down no longerabsorbs them either.
🤖 Generated with Claude Code