Recover CensusAPI.load() metadata from name + long data - #116
Merged
Merged
Conversation
Drop the private _morpc descriptor that load() depended on. save() now
writes only standard frictionless fields, and load() recovers all six
constructor arguments via the new _recover_metadata(name, long):
- survey/year from the uniform survey/reference_period columns
- variables from base codes re-suffixed off the value-type columns
(new _recover_variable_codes), gated on the -select-variables marker
- group from the variable codes, gated on the matching -{code} segment
- sumlevel from the geoidfq summary level, gated on the name's hierarchy
token (the name is the source of truth for whether sumlevel was given)
- scope from the SCOPES key remaining after the other markers are stripped
load() no longer raises on a missing _morpc block; it raises ValueError
only when the name/data are inconsistent with a known scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CensusAPI.load()previously depended on a private_morpcblock written into the resource descriptor. This removes that dependency:save()/create_resource()no longer write_morpc, andload()recovers all six constructor arguments from the canonical dataset name and the long-format data.Recovery map (
_recover_metadata(name, long)):survey,yearsurvey/reference_periodcolumnsvariables_recover_variable_codes); only when the name carries-select-variablesgroup_group_code_from_variableon the data, kept only when the name ends with the matching-{code}sumlevelGeoIDFQ.parse(...).sumlevel(geoidfq primary), kept only when the name carries the hierarchy tokenscopeSCOPESkey remaining after the survey/year/sumlevel/group/variable markers are strippedThe name's presence/absence of the sumlevel token and group segment is the source of truth for distinguishing the three modes (group-only / variables-only / both) — the long data alone cannot, since all three can produce identical rows. Anchored parsing recovers the other parts from the data first, so the survey's embedded dashes don't make the name parse ambiguous.
Behavior change
load()no longer raises on resources lacking a_morpcblock. It raisesValueErroronly when the name doesn't match the data's survey/year or the residual token isn't a known scope.Tests
test_resource_without_morpc_raiseswithtest_resource_has_no_morpc_block(assertssave()no longer emits_morpcand the resource still loads).test_recover_group_only_no_sumlevel,test_recover_variables_only,test_recover_group_and_variables.🤖 Generated with Claude Code