feat(custom_data): support remaining config.json fields#122
Merged
Conversation
Adds three importer-supported config.json fields the package previously rejected under extra="forbid": - dataDownloadUrl (top-level list), with set_dataDownloadUrl and add_dataDownloadUrl on CustomDataManager - verticalSpecsFile (top-level string), with set_verticalSpecsFile - observationProperties on each input file, via a typed ObservationProperties model and a new kwarg on add_explicit_schema_file. Keys are emitted verbatim with no dcid: aliasing and custom keys pass through, matching how the importer reads them. Also closes a pre-existing gap where _merge_simple_attrs only copied the config fields it enumerated, silently dropping any others on merge. Both new top-level fields are now carried through merge_config. Docs are left for the #105 rewrite. Closes #113. Co-authored-by: Claude <noreply@anthropic.com>
Adds add_vertical_spec on CustomDataManager: it appends typed VerticalSpec
entries, points the config's verticalSpecsFile at the output file (default
vertical_specs.json, leaving an explicitly set name untouched), and writes
the {"specs": [...]} JSON through a new export_vertical_specs, wired into
export_all.
Companion to the verticalSpecsFile config field in this PR, so users can
generate the specs file instead of hand-writing it.
Co-authored-by: Claude <noreply@anthropic.com>
tillywoodfield
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds three config.json fields the importer supports but the package rejected under
extra="forbid", plus a builder for the vertical-specs file, so generated bundles can carry them and round-trip cleanly.What changed
dataDownloadUrl(top-level list):set_dataDownloadUrl(urls | None)replaces or unsets;add_dataDownloadUrl(url)appends, initialising the list when unset.verticalSpecsFile(top-level string):set_verticalSpecsFile(name | None).observationProperties(per input file): a typedObservationPropertiesmodel (unit,scalingFactor,measurementMethod,observationPeriod) withextra="allow", surfaced through a newobservationProperties=kwarg onadd_explicit_schema_file. Keys are written verbatim with nodcid:aliasing and custom keys pass through, which is how the importer reads them.add_vertical_spec(verticals=..., population_type="Thing", measured_properties=...)appends typedVerticalSpecentries, pointsverticalSpecsFileat the output file, andexport_vertical_specs(called byexport_all) writes the{"specs": [...]}JSON the importer reads whengroupStatVarsByPropertyis set. Lets users generate that file instead of hand-writing it.While adding the two top-level fields I found that
_merge_simple_attrsonly copied the fields it enumerated by name, so any field not in that list was silently dropped on merge. Both new fields are now wired into the merge path (scalar forverticalSpecsFile, sequence fordataDownloadUrl).Docs are intentionally left out — the loading-data docs are being rewritten under #105, so documenting these here would just be redone.
Testing
ruff format --check,ruff check,ty checkall clean;uv run pytest→ 129 passed. New coverage: model round-trips for all three fields, custom-key preservation, nestedexclude_none, the absent-when-unset contract, bothdataDownloadUrlsetters, merge preservation, and the vertical-specs builder (config wiring,{"specs": [...]}export content, and theexport_allhook). A new golden fixture (tests/goldens/config_all_fields.json) proves a config with all three fields survives load→dump unchanged.Closes #113.
Co-authored-by: Claude noreply@anthropic.com