Skip to content

feat(typespec): close datapackage.json -> validate loop (item 8l) - #296

Merged
discreteds merged 2 commits into
developfrom
feature/8l-frictionless-validation-loop-closure
Aug 17, 2026
Merged

feat(typespec): close datapackage.json -> validate loop (item 8l)#296
discreteds merged 2 commits into
developfrom
feature/8l-frictionless-validation-loop-closure

Conversation

@discreteds

Copy link
Copy Markdown
Member

Summary

Closes item 8l (backlog: mountainash-central/01.principles/mountainash/h.backlog/active/frictionless-validation-loop-closure.md) — the vision's headline Phase-1 loop: datapackage.json → DataResource → TypeSpec → contract → validate.

  • DataResource.to_typespec() — passthrough (already-TypeSpec) / convert (raw dict via typespec_from_frictionless) / raise (garbage type). table_schema is None returns None (legitimately optional).
  • DataResource.to_contract() — thin delegation to TypeSpec.to_contract(), raises ValueError when there's no schema to compile.
  • End-to-end cross-backend test (tests/relations/dag/cross_backend/test_datapackage_validation_loop.py): a real on-disk datapackage.json (loaded via the Path branch of DataPackage.from_descriptor, not the dict branch every other test uses) with two resources (primaryKey, pattern, range, foreignKeys, enum), proving dag.validate(specs, backend=...) on polars/narwhals/ibis:
    • conforming data passes on all 3 backends
    • a duplicate primary_key raises IdentityInvalidError on all 3 (confirms item 8j's raise-by-default characterization holds for a descriptor-sourced TypeSpec, not just hand-built ones)
    • drifting data reports specific failing check IDs (score__le, status__isin, the FK anti-join) on all 3 backends
  • Vision doc fix (mountainash-central, committed directly to main per that repo's convention): unique=True row was stale (n_unique().eq(len())), corrected to the shipped row-level is_duplicated().not_(); added a footnote documenting the null-precondition guard on nullable range/pattern/enum checks.

Scope note — cross-backend pattern check narrowed to Polars

The drifting-data test's cross-backend pattern-constraint assertion surfaced a genuine pre-existing defect, unrelated to this item's to_typespec()/to_contract() seam:

  • Narwhals regexp_match_substring is a complete no-op stub (return input — never evaluates the pattern).
  • ibis-duckdb (the default ibis.get_backend() dialect) returns '' (not NULL) from re_extract on no-match, so compile_datacontract's .is_not_null() check is always True.

Both make the pattern check silently report "passed" for values that violate it. Range/enum/FK checks are confirmed correct on all 3 backends — this is isolated specifically to pattern. Per the plan's explicit I2 disposition (any pre-existing cross-backend defect discovered here is out of scope — compile_datacontract/dag.validate are already-shipped, already-tested code this item does not touch): narrowed the pattern assertion to a dedicated Polars-only test (test_pattern_violation_detected_polars), and filed backlog item 103 with full probe evidence and candidate fix shapes (mountainash-central/01.principles/mountainash/h.backlog/active/pattern-constraint-check-silently-passes-narwhals-ibis-duckdb.md).

Verification

  • hatch run test:test-quick — 22991 passed, 0 failed, 81 skipped, 3465 xfailed (full suite).
  • hatch run ruff:check — clean.
  • hatch run mypy:check — zero new errors (pre-existing import-untyped baseline noise only; none touch the changed files).

Spec / Plan

  • Spec: mountainash-central/04.planning/mountainash/superpowers/specs/2026-08-17-frictionless-validation-loop-closure-design.md (rev 3)
  • Plan: mountainash-central/04.planning/mountainash/superpowers/plans/2026-08-17-frictionless-validation-loop-closure.md

…, tasks 1-2)

Closes the DataResource -> TypeSpec seam the vision doc's sketch already
assumes exists: to_typespec() passthrough/convert/raise over the three
legitimate table_schema shapes (None/TypeSpec/dict), to_contract() as a
thin delegation raising on missing schema.
…l, tasks 3-4)

Proves the headline loop cross-backend (polars/narwhals/ibis):
- conforming data passes on all 3 backends
- duplicate primary_key raises IdentityInvalidError on all 3 (item 8j characterization holds for descriptor-sourced TypeSpec)
- drifting data reports specific check IDs (score__le, status__isin, FK anti-join) on all 3

Discovered a genuine pre-existing defect during this work: the pattern
constraint check (regexp_match_substring(...).is_not_null()) silently
passes on narwhals (no-op stub) and ibis-duckdb (empty-string, not NULL,
on no-match). Unrelated to to_typespec()/to_contract() and out of this
item's scope per spec §7 -- narrowed the pattern assertion to Polars,
filed backlog item 103 with probe evidence (mountainash-central).
@discreteds
discreteds merged commit 7e82b39 into develop Aug 17, 2026
4 checks passed
@discreteds
discreteds deleted the feature/8l-frictionless-validation-loop-closure branch August 17, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant