feat(typespec): nested LIST + categorical dtype fidelity, Pandas categorical (item 54, gaps 2-3) - #295
Merged
Conversation
… 54 dtypes (task 12)
…yArrow strings; review round (GLM-5.2) Adversarial whole-branch review (opencode-go/glm-5.2, high reasoning, job 001) verdict: DONE_WITH_CONCERNS — Approved with one Important + two Minor findings. All folded: Important 1: PyArrow regex branches propagated raw Arrow constructor ValueErrors (timestamp[badunit], decimal128(999, 10), time64[badunit]) instead of returning None, breaking the InvalidBackendTypeError contract in _resolve_field_native. Regex constructors now wrapped in try/except (KeyError, ValueError) -> None; resolver-level regression test asserts the typed error surfaces through to_arrow_schema (plan-mandated gap: spec §2.2's code omitted the wrap). Minor 1: target_narwhals redundant disjunct (isinstance(result, type) or result is not None == result is not None) simplified, intent documented. Minor 2: inline-read blast-radius test was Polars-only; parametrized cross-backend with boundary-honest categorical assertion (ibis/Arrow converts Enum -> String — documented boundary conversion, not a defect). Traceability: test_namespace_classes_present_at_pinned_floor documents task 13's floor verification (narwhals 2.20.0 wheel inspected 2026-08-17). Full suite: 22974 passed / 0 failed. ruff: clean.
discreteds
force-pushed
the
feature/typespec-list-categorical-fidelity
branch
from
August 17, 2026 04:52
a15d51c to
d5e5446
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Item 54, PR-2 (gaps 2–3), stacked on PR #294 (gap 1 + validation strictness — same develop baseline; will be rebased if #294 merges first).
This PR:
LISTinner type via the existing Frictionlessitem_typecarriage:_resolve_field_nativenow parameterizes a bare list container on all applicable targets (pl.List(Int64)/nw.List(Int64)/pa.list_(pa.int64())/"array<int64>"; Pandas stays"object"— no native parameterized list dtype). Also fixes a second latent bug: PyArrow's bare fallback silently defaulted every untyped list to astringelement.categorical_values()helper (typespec/_categorical.py) consumed by BOTH conform's stage-5b and the converters (previously copy-paste-identical inline extraction).to_polars_schemamapscategories/categoriesOrdered→pl.Enum([...])/pl.Categorical, categories-first precedence (mirrors conform stage 5 branch order — a field with bothcategoriesand an invalidbackend_typenever raises). Pandas un-deferred:to_pandas_dtypesreturn type widenedDict[str,str]→Dict[str,Any](zero production callers — verified), categorical fields now return a realpd.CategoricalDtypeinstance.empty_frameAND item 53's inline-read cast both produce the upgraded dtypes (the shared-resolver design's whole point), plus Ibis/Narwhals backend-native schema + executed-op assertions post-wrap, plus the no-raise partner regression (backend_type=None/""still falls through).backend_type="Enum"/"Categorical"bare forms) resolves as a documented consequence — no separate code.OBJECT/STRUCTinner schema remains out of scope → tracked as new item 102 (user-confirmed carve-out).Spec:
2026-08-16-converters-resolver-dtype-fidelity-design.md(rev 2, GLM-5.2 adversarial review folded). Plan tasks 8–13, RED-first TDD, one commit per task.Verification
hatch run ruff:check: cleanhatch run mypy:check: 98 errors — identical count to baseline (all pre-existingimport-untyped/fixture errors; touched files contribute none new)Array/Enum/Categorical/Decimalall present + top-level exported +DTypeClassinstantiation mechanism present at the pinned floor2.20.0— no floor change needed