feat(typespec): close datapackage.json -> validate loop (item 8l) - #296
Merged
discreteds merged 2 commits intoAug 17, 2026
Merged
Conversation
…, tasks 1-2) Closes the DataResource -> TypeSpec seam the vision doc's sketch already assumes exists: to_typespec() passthrough/convert/raise over the three legitimate table_schema shapes (None/TypeSpec/dict), to_contract() as a thin delegation raising on missing schema.
…l, tasks 3-4) Proves the headline loop cross-backend (polars/narwhals/ibis): - conforming data passes on all 3 backends - duplicate primary_key raises IdentityInvalidError on all 3 (item 8j characterization holds for descriptor-sourced TypeSpec) - drifting data reports specific check IDs (score__le, status__isin, FK anti-join) on all 3 Discovered a genuine pre-existing defect during this work: the pattern constraint check (regexp_match_substring(...).is_not_null()) silently passes on narwhals (no-op stub) and ibis-duckdb (empty-string, not NULL, on no-match). Unrelated to to_typespec()/to_contract() and out of this item's scope per spec §7 -- narrowed the pattern assertion to Polars, filed backlog item 103 with probe evidence (mountainash-central).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes item 8l (backlog:
mountainash-central/01.principles/mountainash/h.backlog/active/frictionless-validation-loop-closure.md) — the vision's headline Phase-1 loop:datapackage.json → DataResource → TypeSpec → contract → validate.DataResource.to_typespec()— passthrough (already-TypeSpec) / convert (raw dict viatypespec_from_frictionless) / raise (garbage type).table_schema is NonereturnsNone(legitimately optional).DataResource.to_contract()— thin delegation toTypeSpec.to_contract(), raisesValueErrorwhen there's no schema to compile.tests/relations/dag/cross_backend/test_datapackage_validation_loop.py): a real on-diskdatapackage.json(loaded via thePathbranch ofDataPackage.from_descriptor, not the dict branch every other test uses) with two resources (primaryKey,pattern, range,foreignKeys,enum), provingdag.validate(specs, backend=...)onpolars/narwhals/ibis:primary_keyraisesIdentityInvalidErroron all 3 (confirms item 8j's raise-by-default characterization holds for a descriptor-sourcedTypeSpec, not just hand-built ones)score__le,status__isin, the FK anti-join) on all 3 backendsmountainash-central, committed directly tomainper that repo's convention):unique=Truerow was stale (n_unique().eq(len())), corrected to the shipped row-levelis_duplicated().not_(); added a footnote documenting the null-precondition guard on nullable range/pattern/enum checks.Scope note — cross-backend
patterncheck narrowed to PolarsThe drifting-data test's cross-backend
pattern-constraint assertion surfaced a genuine pre-existing defect, unrelated to this item'sto_typespec()/to_contract()seam:regexp_match_substringis a complete no-op stub (return input— never evaluates the pattern).ibis-duckdb(the defaultibis.get_backend()dialect) returns''(notNULL) fromre_extracton no-match, socompile_datacontract's.is_not_null()check is alwaysTrue.Both make the
patterncheck silently report"passed"for values that violate it. Range/enum/FK checks are confirmed correct on all 3 backends — this is isolated specifically topattern. Per the plan's explicit I2 disposition (any pre-existing cross-backend defect discovered here is out of scope —compile_datacontract/dag.validateare already-shipped, already-tested code this item does not touch): narrowed thepatternassertion to a dedicated Polars-only test (test_pattern_violation_detected_polars), and filed backlog item 103 with full probe evidence and candidate fix shapes (mountainash-central/01.principles/mountainash/h.backlog/active/pattern-constraint-check-silently-passes-narwhals-ibis-duckdb.md).Verification
hatch run test:test-quick— 22991 passed, 0 failed, 81 skipped, 3465 xfailed (full suite).hatch run ruff:check— clean.hatch run mypy:check— zero new errors (pre-existingimport-untypedbaseline noise only; none touch the changed files).Spec / Plan
mountainash-central/04.planning/mountainash/superpowers/specs/2026-08-17-frictionless-validation-loop-closure-design.md(rev 3)mountainash-central/04.planning/mountainash/superpowers/plans/2026-08-17-frictionless-validation-loop-closure.md