From b110e5c2a6001077887f9ebd8ed09fcad2a7a5ac Mon Sep 17 00:00:00 2001 From: Nathaniel Ramm Date: Sat, 15 Aug 2026 21:52:19 +1000 Subject: [PATCH] feat(capabilities): wire extract/extract_boolean, strptime_timestamp.timezone, retire MA duplicates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backlog item 62. Two canonical Substrait datetime ops plus one options fix, landed as a single commit (the closed-by-default option-disposition matrix couples the MA-retirement drain to the same commit as the wiring — see notes). extract / extract_boolean (new): - component (23/2-member closed domains, DatetimeComponent/BooleanComponent), indexing (ONE/ZERO omission-sentinel, calendar components only), timezone (open IANA_TIMEZONE value class) wired through protocol -> API builder -> three backends -> capability facts -> disposition matrix. - Backend bodies rewritten per semantic probe (gate disabled, all 4 fixtures): deleted every silent fallback (return x.dt.year() / pl.lit(False)); in-domain-but-unsupported components now raise BackendCapabilityError, backed by UNSUPPORTED facts (polars 6, ibis 7, narwhals 9 declared components; IS_DST declared on all three). - Two additional silent-wrong bugs found and fixed: MICROSECOND/NANOSECOND were returning since-last-SECOND (0-999999/0-999999999) instead of Substrait's since-last-unit (0-999) semantics — now % 1000; narwhals QUARTER formula (month // 4 + 1) was wrong for months 7/10/11 — now (month-1)//3+1. TIMEZONE_OFFSET declared everywhere (native primitives require tz-aware input extract can't guarantee at build time — input-type-dependent, matches spec §7.3, not a raw backend exception). strptime_timestamp.timezone (end-to-end fix): - MountainAshScalarStringAPIBuilder.to_datetime gains timezone (IANA- validated), threading through the def's options tuple (now ("format","timezone")) to the protocol and all three backends. - narwhals body wired for real (dt.replace_time_zone); ibis silently drops (no timezone primitives, matching assume_timezone/to_timezone/ local_timestamp/extract.timezone) — now backed by an UNSUPPORTED fact instead of a silent discard. Mountainash-namespace retirement: - Deleted MA extract/extract_boolean (protocol + 3 backend impls) — dead code, positional-only, no caller (named wrappers call backends directly). Drains their option-matrix parks by deletion. Capability facts (new capabilities/datetime/extract.py + additions to value_classes_substrait.py): - Closed option_value= grain (component/IS_DST) lives in extract.py. - Open value_class=IANA_TIMEZONE grain (extract/extract_boolean/ strptime_timestamp timezone) lives in value_classes_substrait.py per the placement-guard's domain-module restriction (spec §3: domain modules never carry value_class facts). Disposition matrix (test_arg_types_datetime.py + option_disposition.py): full per-backend x per-value dispositions, probes (declared cells verified via gate-disabled native probe + strict xfail), invalid-sentinel rejections, and TESTED_OPTION_PARAMS entries for all 6 wired option params. Drained all 6 "operation not implemented" parks plus the 2 MA argument-channel aliases. Conformance/reachability drift entries retired (signatures/options/call- patterns that were "known divergent" now genuinely match): protocol alignment KNOWN_ASPIRATIONAL, signature/options drift dicts, unverifiable- call-pattern entries, and A2 local builders + reachability builders added for the two newly-public FKEYs. AST-identity coverage added: extract/extract_boolean node shape (test_ast_scalar_datetime.py), to_datetime timezone option presence/ omission (test_ast_str_to_date.py). Regenerated docs/reference/expression-coverage.{md,json} (3 new declarations from extract.py). Deleted tests/expressions/backends/test_narwhals_extract_capability_error.py — exercised the now-retired MA extract's inline ISO_WEEK raise; the same guarantee (ISO_WEEK on narwhals raises BackendCapabilityError, not AttributeError) is now covered by the extract disposition matrix's declared cell + strict-xfail probe. Deferred (spec §7, unchanged from plan): strptime_time + precision (needs core/dtypes work); is_dst (item 65, IS_DST stays UNSUPPORTED); input-type- dependent overload validity (error normalization, pre-existing). Verification: full suite 22519 passed / 0 failed / 81 skipped / 3734 xfailed; ruff clean; mypy — no new errors (2 touched-file hits are the pre-existing ibis import-untyped baseline noise, same class as the other 95). Smoke- tested end-to-end on all 4 fixtures (extract, extract_boolean, indexing, declared-component raise, strptime_timestamp+timezone) via the public relation API. Deviation from the written plan: the plan sliced this into PR-1 (extract/ extract_boolean) and PR-2 (strptime_timestamp.timezone + MA retirement). The option-disposition matrix's closed-by-default reason-staleness check (test_untested_option_param_custom_reasons_still_hold) ties MA retirement to the same commit as wiring extract into the API builder — draining the MA park's "not implemented in API builder" reason requires the MA methods to already be gone the moment extract becomes a builder method name, or the check fails on the newly-false reason. Landed as one commit; still two logically-separable diffs within it (extract+extract_boolean vs strptime_timestamp+retirement) per the plan's task boundaries. Backlog: substrait-datetime-missing-ops (item 62) --- docs/reference/expression-coverage-scoped.md | 51 +- docs/reference/expression-coverage.json | 1780 ++++++++++++++++- docs/reference/expression-coverage.md | 33 +- .../backends/capabilities/datetime/extract.py | 206 ++ .../datetime/value_classes_substrait.py | 31 + .../expsys_ib_ext_ma_scalar_datetime.py | 75 - .../substrait/expsys_ib_scalar_datetime.py | 58 +- .../expsys_nw_ext_ma_scalar_datetime.py | 83 - .../substrait/expsys_nw_scalar_datetime.py | 94 +- .../expsys_pl_ext_ma_scalar_datetime.py | 85 - .../substrait/expsys_pl_scalar_datetime.py | 105 +- .../expressions/core/datetime_components.py | 18 +- .../api_bldr_ext_ma_scalar_string.py | 22 +- .../api_builders/substrait/_option_domains.py | 9 + .../substrait/api_bldr_scalar_datetime.py | 98 +- .../prtcl_api_bldr_scalar_datetime.py | 27 + .../prtcl_expsys_ext_ma_scalar_datetime.py | 43 - .../substrait/prtcl_expsys_scalar_datetime.py | 6 +- .../function_mapping/definitions.py | 7 +- tests/_spine_expectation_census.md | 14 +- tests/core/test_api_reachability.py | 13 +- tests/core/test_capability_protocol_guard.py | 2 + tests/core/test_protocol_alignment.py | 3 - tests/core/test_signature_conformance.py | 37 +- .../argument_types/option_disposition.py | 12 + .../argument_types/test_arg_types_datetime.py | 672 +++++++ .../argument_types/test_coverage_guard.py | 29 - .../test_option_fact_integrity.py | 2 +- .../ast/test_ast_scalar_datetime.py | 35 + tests/expressions/ast/test_ast_str_to_date.py | 16 +- .../test_narwhals_extract_capability_error.py | 16 - 31 files changed, 3085 insertions(+), 597 deletions(-) create mode 100644 src/mountainash/expressions/backends/capabilities/datetime/extract.py delete mode 100644 tests/expressions/backends/test_narwhals_extract_capability_error.py diff --git a/docs/reference/expression-coverage-scoped.md b/docs/reference/expression-coverage-scoped.md index 1014e916..3255d05b 100644 --- a/docs/reference/expression-coverage-scoped.md +++ b/docs/reference/expression-coverage-scoped.md @@ -5,7 +5,7 @@ Scoped deviations — dialect, parameter, option, value-class; function-level coverage and matrices live in [`expression-coverage.md`](expression-coverage.md). -Declarations: 39 · Facts: 1509 · Registered operations: 324 · Implementation records: 972 +Declarations: 42 · Facts: 1558 · Registered operations: 324 · Implementation records: 972 Legend — scoped deviations: @@ -963,6 +963,50 @@ Legend — scoped deviations: | * | timezone | — | iana_timezone | unsupported | gate | build | — | assume_timezone silently drops the timezone (returns a naive timestamp) — the tz argument is ignored; only polars attaches the timezone | — | — | 2026-07-25 | — | — | | ibis-duckdb | timezone | — | iana_timezone | unsupported | gate | build | — | assume_timezone silently drops the timezone (returns a naive timestamp) — the tz argument is ignored; only polars attaches the timezone | — | — | 2026-07-25 | — | — | +### `EXTRACT` × polars (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| polars | component | MONDAY_WEEK, PICOSECOND, SUNDAY_WEEK, TIMEZONE_OFFSET, US_WEEK, US_YEAR | — | unsupported | gate | build | — | the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py) | — | — | 2026-08-15 | — | — | + +### `EXTRACT` × narwhals (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| narwhals-pandas | component | ISO_WEEK, ISO_YEAR, MONDAY_WEEK, PICOSECOND, SUNDAY_WEEK, TIMEZONE_OFFSET, UNIX_TIME, US_WEEK, US_YEAR | — | unsupported | gate | build | — | the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py) | — | — | 2026-08-15 | — | — | +| narwhals-polars | component | ISO_WEEK, ISO_YEAR, MONDAY_WEEK, PICOSECOND, SUNDAY_WEEK, TIMEZONE_OFFSET, UNIX_TIME, US_WEEK, US_YEAR | — | unsupported | gate | build | — | the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py) | — | — | 2026-08-15 | — | — | + +### `EXTRACT` × ibis (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| * | component | MONDAY_WEEK, NANOSECOND, PICOSECOND, SUNDAY_WEEK, TIMEZONE_OFFSET, US_WEEK, US_YEAR | — | unsupported | gate | build | — | the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py) | — | — | 2026-08-15 | — | — | +| * | timezone | — | iana_timezone | unsupported | gate | build | — | ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py | — | — | 2026-08-15 | — | — | +| ibis-duckdb | component | MONDAY_WEEK, NANOSECOND, PICOSECOND, SUNDAY_WEEK, TIMEZONE_OFFSET, US_WEEK, US_YEAR | — | unsupported | gate | build | — | the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py) | — | — | 2026-08-15 | — | — | +| ibis-duckdb | timezone | — | iana_timezone | unsupported | gate | build | — | ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py | — | — | 2026-08-15 | — | — | + +### `EXTRACT_BOOLEAN` × polars (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| polars | component | IS_DST | — | unsupported | gate | build | — | extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation) | — | — | 2026-08-15 | — | — | + +### `EXTRACT_BOOLEAN` × narwhals (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| narwhals-pandas | component | IS_DST | — | unsupported | gate | build | — | extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation) | — | — | 2026-08-15 | — | — | +| narwhals-polars | component | IS_DST | — | unsupported | gate | build | — | extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation) | — | — | 2026-08-15 | — | — | + +### `EXTRACT_BOOLEAN` × ibis (FKEY_SUBSTRAIT_SCALAR_DATETIME) + +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| * | component | IS_DST | — | unsupported | gate | build | — | extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation) | — | — | 2026-08-15 | — | — | +| * | timezone | — | iana_timezone | unsupported | gate | build | — | ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py | — | — | 2026-08-15 | — | — | +| ibis-duckdb | component | IS_DST | — | unsupported | gate | build | — | extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation) | — | — | 2026-08-15 | — | — | +| ibis-duckdb | timezone | — | iana_timezone | unsupported | gate | build | — | ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py | — | — | 2026-08-15 | — | — | + ### `LOCAL_TIMESTAMP` × ibis (FKEY_SUBSTRAIT_SCALAR_DATETIME) | Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | @@ -994,6 +1038,11 @@ Legend — scoped deviations: | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ibis-sqlite | * | — | — | unsupported | gate | build | — | ibis-sqlite has no compilation rule for StringToDate/StringToTimestamp (OperationNotDefinedError); format-driven parsing is unavailable on this dialect, so it is gated rather than left to fail natively | — | — | 2026-07-30 | — | whole-op gate on a WILDCARD_PARAM fact; cannot be keyed on an OpSpec param (OpSpecs are indexed by concrete argument name) — verified by the dedicated cross-backend gate tests in test_datetime_strptime_format.py | +| Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| * | timezone | — | iana_timezone | unsupported | gate | build | — | strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone | — | — | 2026-08-15 | — | — | +| ibis-duckdb | timezone | — | iana_timezone | unsupported | gate | build | — | strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone | — | — | 2026-08-15 | — | — | + ### `CAPITALIZE` × polars (FKEY_SUBSTRAIT_SCALAR_STRING) | Dialect | Param | Option values | Value class | Level | Enforcement | Boundary | Condition | Message | Workaround | Upstream | Since | Native errors | Probe-exempt | diff --git a/docs/reference/expression-coverage.json b/docs/reference/expression-coverage.json index dda5dffc..0f7d12e9 100644 --- a/docs/reference/expression-coverage.json +++ b/docs/reference/expression-coverage.json @@ -1,7 +1,7 @@ { "stamp": { - "declarations": 39, - "facts": 1509, + "declarations": 42, + "facts": 1558, "operations": 324, "implementation_records": 972 }, @@ -14,9 +14,9 @@ "not_implemented": 0, "unknown": 0 }, - "default_capable": 199, - "audited_clean": 75, - "constrained": 50, + "default_capable": 191, + "audited_clean": 81, + "constrained": 52, "audited_unknown": 0 }, "narwhals": { @@ -27,8 +27,8 @@ "unknown": 0 }, "default_capable": 97, - "audited_clean": 151, - "constrained": 76, + "audited_clean": 149, + "constrained": 78, "audited_unknown": 0 }, "ibis": { @@ -39,8 +39,8 @@ "unknown": 0 }, "default_capable": 138, - "audited_clean": 114, - "constrained": 72, + "audited_clean": 112, + "constrained": 74, "audited_unknown": 0 } }, @@ -50,19 +50,19 @@ "expr_capable": 149, "literal_only": 65, "polymorphic": 9, - "unsupported": 1286 + "unsupported": 1335 }, "facts_by_enforcement": { - "gate": 1502, + "gate": 1551, "router_metadata": 3, "materialize_residue": 4 }, "facts_by_backend": { - "polars": 258, - "narwhals": 642, - "ibis": 609 + "polars": 265, + "narwhals": 662, + "ibis": 631 }, - "facts_total": 1509 + "facts_total": 1558 }, "families": [ { @@ -30995,7 +30995,7 @@ "impl": "implemented", "impl_method": "add_intervals", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31060,7 +31060,7 @@ "impl": "implemented", "impl_method": "assume_timezone", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31191,17 +31191,114 @@ "impl": "implemented", "impl_method": "extract", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, + "params": 1, + "option_selectors": 6, "value_classes": 0, - "dialects": 0 + "dialects": 1 }, - "constraints": [], + "constraints": [ + { + "dialect": "polars", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31212,15 +31309,304 @@ "impl_protocol": "SubstraitNarwhalsScalarDatetimeExpressionSystem", "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, + "params": 1, + "option_selectors": 9, "value_classes": 0, - "dialects": 0 + "dialects": 2 }, - "constraints": [], + "constraints": [ + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "ISO_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "ISO_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "UNIX_TIME", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "ISO_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "ISO_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "UNIX_TIME", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31231,15 +31617,272 @@ "impl_protocol": "SubstraitIbisScalarDatetimeExpressionSystem", "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, - "value_classes": 0, - "dialects": 0 + "params": 2, + "option_selectors": 7, + "value_classes": 1, + "dialects": 1 }, - "constraints": [], + "constraints": [ + { + "dialect": null, + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "NANOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "NANOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31256,17 +31899,34 @@ "impl": "implemented", "impl_method": "extract_boolean", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, + "params": 1, + "option_selectors": 1, "value_classes": 0, - "dialects": 0 + "dialects": 1 }, - "constraints": [], + "constraints": [ + { + "dialect": "polars", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31277,15 +31937,48 @@ "impl_protocol": "SubstraitNarwhalsScalarDatetimeExpressionSystem", "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, + "params": 1, + "option_selectors": 1, "value_classes": 0, - "dialects": 0 + "dialects": 2 }, - "constraints": [], + "constraints": [ + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31296,15 +31989,80 @@ "impl_protocol": "SubstraitIbisScalarDatetimeExpressionSystem", "audited": true, "whole_op": null, - "constrained": false, + "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, - "option_selectors": 0, - "value_classes": 0, - "dialects": 0 + "params": 2, + "option_selectors": 1, + "value_classes": 1, + "dialects": 1 }, - "constraints": [], + "constraints": [ + { + "dialect": null, + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ], "residue": [], "routed": [], "refinements": [] @@ -31321,7 +32079,7 @@ "impl": "implemented", "impl_method": "local_timestamp", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31419,7 +32177,7 @@ "impl": "implemented", "impl_method": "strftime", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31484,7 +32242,7 @@ "impl": "implemented", "impl_method": "strptime_date", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31583,7 +32341,7 @@ "impl": "implemented", "impl_method": "strptime_timestamp", "impl_protocol": "SubstraitPolarsScalarDatetimeExpressionSystem", - "audited": false, + "audited": true, "whole_op": null, "constrained": false, "contradiction": false, @@ -31626,12 +32384,44 @@ "constrained": true, "contradiction": false, "selector_counts": { - "params": 0, + "params": 1, "option_selectors": 0, - "value_classes": 0, - "dialects": 1 + "value_classes": 1, + "dialects": 2 }, "constraints": [ + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, { "dialect": "ibis-sqlite", "param": "*", @@ -52532,6 +53322,54 @@ "native_errors": [], "probe_exempt": null }, + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, { "dialect": "ibis-duckdb", "param": "timezone", @@ -52563,6 +53401,54 @@ "since": "2026-07-29", "native_errors": [], "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "ibis has no timezone primitives; extract/extract_boolean's timezone option is silently ignored (the local component is read from the stored value, not the target zone) -- see capabilities/datetime/extract.py", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "timezone", + "option_value": null, + "value_class": "iana_timezone", + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "strptime_timestamp silently drops the timezone (returns a naive timestamp) on ibis -- ibis has no timezone primitives, matching assume_timezone/to_timezone/local_timestamp/extract.timezone", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null } ] }, @@ -52626,6 +53512,292 @@ } ] }, + { + "backend": "ibis", + "source": "substrait", + "domain": "datetime", + "evidence": { + "probe_date": "2026-08-15", + "library_versions": [ + [ + "polars", + "1.43.2" + ], + [ + "narwhals", + "2.24.0" + ], + [ + "ibis", + "12.0.0" + ] + ], + "fixtures": [ + "polars", + "ibis-duckdb", + "narwhals-polars", + "narwhals-pandas" + ] + }, + "facts": [ + { + "dialect": null, + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "NANOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": null, + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "NANOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "ibis-duckdb", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ] + }, { "backend": "ibis", "source": "substrait", @@ -62632,6 +63804,356 @@ } ] }, + { + "backend": "narwhals", + "source": "substrait", + "domain": "datetime", + "evidence": { + "probe_date": "2026-08-15", + "library_versions": [ + [ + "polars", + "1.43.2" + ], + [ + "narwhals", + "2.24.0" + ], + [ + "ibis", + "12.0.0" + ] + ], + "fixtures": [ + "polars", + "ibis-duckdb", + "narwhals-polars", + "narwhals-pandas" + ] + }, + "facts": [ + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "ISO_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "ISO_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "UNIX_TIME", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-pandas", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "ISO_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "ISO_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "UNIX_TIME", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "narwhals-polars", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ] + }, { "backend": "narwhals", "source": "substrait", @@ -68864,6 +70386,148 @@ } ] }, + { + "backend": "polars", + "source": "substrait", + "domain": "datetime", + "evidence": { + "probe_date": "2026-08-15", + "library_versions": [ + [ + "polars", + "1.43.2" + ], + [ + "narwhals", + "2.24.0" + ], + [ + "ibis", + "12.0.0" + ] + ], + "fixtures": [ + "polars", + "ibis-duckdb", + "narwhals-polars", + "narwhals-pandas" + ] + }, + "facts": [ + { + "dialect": "polars", + "param": "component", + "option_value": "IS_DST", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; deferred to backlog item 65 (is-dst-placeholder-implementation)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "MONDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "PICOSECOND", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "SUNDAY_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "TIMEZONE_OFFSET", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "US_WEEK", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + }, + { + "dialect": "polars", + "param": "component", + "option_value": "US_YEAR", + "value_class": null, + "level": "unsupported", + "enforcement": "gate", + "boundary": "build", + "condition": null, + "message": "the native backend has no primitive for this extract component (verified by semantic probe; see capabilities/datetime/extract.py)", + "workaround": null, + "upstream_ref": null, + "since": "2026-08-15", + "native_errors": [], + "probe_exempt": null + } + ] + }, { "backend": "polars", "source": "substrait", diff --git a/docs/reference/expression-coverage.md b/docs/reference/expression-coverage.md index 40e5eb5b..94493fe1 100644 --- a/docs/reference/expression-coverage.md +++ b/docs/reference/expression-coverage.md @@ -3,7 +3,7 @@ -Declarations: 39 · Facts: 1509 · Registered operations: 324 · Implementation records: 972 +Declarations: 42 · Facts: 1558 · Registered operations: 324 · Implementation records: 972 Scoped deviations (dialect/param/option/value-class) live in [`expression-coverage-scoped.md`](expression-coverage-scoped.md). @@ -50,9 +50,9 @@ Legend — cell states (by exception): | Backend | default_capable | audited_clean | constrained | NOT_IMPLEMENTED | UNKNOWN | ops_total | | --- | --- | --- | --- | --- | --- | --- | -| polars | 199 | 75 | 50 | 0 | 0 | 324 | -| narwhals | 97 | 151 | 76 | 0 | 0 | 324 | -| ibis | 138 | 114 | 72 | 0 | 0 | 324 | +| polars | 191 | 81 | 52 | 0 | 0 | 324 | +| narwhals | 97 | 149 | 78 | 0 | 0 | 324 | +| ibis | 138 | 112 | 74 | 0 | 0 | 324 | contradictions: 0 audited_unknown: 0 @@ -61,9 +61,9 @@ audited_unknown: 0 | Axis | Breakdown | | --- | --- | -| Level | expr_capable 149, literal_only 65, polymorphic 9, unsupported 1286 | -| Enforcement | gate 1502, router_metadata 3, materialize_residue 4 | -| Backend | polars 258, narwhals 642, ibis 609 | +| Level | expr_capable 149, literal_only 65, polymorphic 9, unsupported 1335 | +| Enforcement | gate 1551, router_metadata 3, materialize_residue 4 | +| Backend | polars 265, narwhals 662, ibis 631 | `pandas` / `pyarrow` are routed input types (they execute via the narwhals path) and are not independent coverage columns. @@ -80,6 +80,7 @@ audited_unknown: 0 | ibis | substrait | arithmetic | 2026-07-21 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | ibis | substrait | datetime | 2026-07-25 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | ibis | substrait | datetime | 2026-07-30 | ibis 12.0.0, narwhals 2.23.0 | polars, ibis-duckdb, ibis-polars, ibis-sqlite, narwhals-polars, narwhals-pandas | +| ibis | substrait | datetime | 2026-08-15 | polars 1.43.2, narwhals 2.24.0, ibis 12.0.0 | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | ibis | substrait | string | 2026-07-05 | | | | ibis | substrait | string | 2026-07-23 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | ibis | substrait | string | 2026-08-12 | ibis 12.0.0 | ibis-sqlite, ibis-duckdb | @@ -97,6 +98,7 @@ audited_unknown: 0 | narwhals | substrait | arithmetic | 2026-07-21 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | narwhals | substrait | datetime | 2026-07-25 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | narwhals | substrait | datetime | 2026-07-30 | ibis 12.0.0, narwhals 2.23.0 | polars, ibis-duckdb, ibis-polars, ibis-sqlite, narwhals-polars, narwhals-pandas | +| narwhals | substrait | datetime | 2026-08-15 | polars 1.43.2, narwhals 2.24.0, ibis 12.0.0 | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | narwhals | substrait | string | 2026-07-05 | | | | narwhals | substrait | string | 2026-07-05 | narwhals 2.19.0 | | | narwhals | substrait | string | 2026-07-23 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | @@ -106,6 +108,7 @@ audited_unknown: 0 | polars | mountainash | set | — | — | — | | polars | mountainash | ternary | — | — | — | | polars | substrait | arithmetic | 2026-07-21 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | +| polars | substrait | datetime | 2026-08-15 | polars 1.43.2, narwhals 2.24.0, ibis 12.0.0 | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | polars | substrait | string | 2026-07-05 | | polars | | polars | substrait | string | 2026-07-23 | | polars, ibis-duckdb, narwhals-polars, narwhals-pandas | | polars | substrait | string | 2026-08-12 | ibis 12.0.0, narwhals 2.24.0 | polars, ibis-duckdb, ibis-polars, ibis-sqlite, narwhals-polars, narwhals-pandas | @@ -315,14 +318,14 @@ audited_unknown: 0 | Operation | polars | narwhals | ibis | | --- | --- | --- | --- | -| `ADD_INTERVALS` | ✓ | ✓ audited | ✓ audited | -| `ASSUME_TIMEZONE` | ✓ | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 2 dialects) | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 1 dialects) | -| `EXTRACT` | ✓ | ✓ audited | ✓ audited | -| `EXTRACT_BOOLEAN` | ✓ | ✓ audited | ✓ audited | -| `LOCAL_TIMESTAMP` | ✓ | ✓ audited | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 1 dialects) | -| `STRFTIME` | ✓ | ✓ audited | ✓ audited | -| `STRPTIME_DATE` | ✓ | ◐ partial (0 params, 0 option-selectors, 0 value-classes, 1 dialects) · unsupported on narwhals-pandas | ◐ partial (0 params, 0 option-selectors, 0 value-classes, 1 dialects) · unsupported on ibis-sqlite | -| `STRPTIME_TIMESTAMP` | ✓ | ✓ audited | ◐ partial (0 params, 0 option-selectors, 0 value-classes, 1 dialects) · unsupported on ibis-sqlite | +| `ADD_INTERVALS` | ✓ audited | ✓ audited | ✓ audited | +| `ASSUME_TIMEZONE` | ✓ audited | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 2 dialects) | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 1 dialects) | +| `EXTRACT` | ◐ partial (1 params, 6 option-selectors, 0 value-classes, 1 dialects) | ◐ partial (1 params, 9 option-selectors, 0 value-classes, 2 dialects) | ◐ partial (2 params, 7 option-selectors, 1 value-classes, 1 dialects) | +| `EXTRACT_BOOLEAN` | ◐ partial (1 params, 1 option-selectors, 0 value-classes, 1 dialects) | ◐ partial (1 params, 1 option-selectors, 0 value-classes, 2 dialects) | ◐ partial (2 params, 1 option-selectors, 1 value-classes, 1 dialects) | +| `LOCAL_TIMESTAMP` | ✓ audited | ✓ audited | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 1 dialects) | +| `STRFTIME` | ✓ audited | ✓ audited | ✓ audited | +| `STRPTIME_DATE` | ✓ audited | ◐ partial (0 params, 0 option-selectors, 0 value-classes, 1 dialects) · unsupported on narwhals-pandas | ◐ partial (0 params, 0 option-selectors, 0 value-classes, 1 dialects) · unsupported on ibis-sqlite | +| `STRPTIME_TIMESTAMP` | ✓ audited | ✓ audited | ◐ partial (1 params, 0 option-selectors, 1 value-classes, 2 dialects) · unsupported on ibis-sqlite | ### `FKEY_SUBSTRAIT_SCALAR_SET` (substrait / set) diff --git a/src/mountainash/expressions/backends/capabilities/datetime/extract.py b/src/mountainash/expressions/backends/capabilities/datetime/extract.py new file mode 100644 index 00000000..c432da04 --- /dev/null +++ b/src/mountainash/expressions/backends/capabilities/datetime/extract.py @@ -0,0 +1,206 @@ +"""Value-scoped capability facts for Substrait ``extract`` / ``extract_boolean``. + +Backlog item 62 (substrait-datetime-missing-ops). Both ops are now API-reachable; +their closed ``component`` domain is enforced at the api-builder and the option +values reaching the visitor are exactly the set enumerated here (gate-domain == +production-domain). + +Only the closed ``component`` grain (``option_value=``) lives here — one +UNSUPPORTED fact per probe-confirmed (backend, component) cell the native +backend cannot produce. The backend body raises ``BackendCapabilityError`` +for these; the fact is the registry's authoritative record and drives the +visitor gate. The open ``timezone`` grain (``value_class=ValueClass.IANA_TIMEZONE``) +is a placement-guard domain module restriction (spec §3: domain modules carry +option-value / WILDCARD_PARAM / value-agnostic grains only, never value_class) +— those facts live in ``value_classes_substrait.py`` alongside +``assume_timezone``/``local_timestamp``/``strptime_timestamp``. + +Probe matrix — per-backend × per-component (2026-08-15, polars 1.43.2, +narwhals 2.24.0, ibis 12.0.0/duckdb), verified natively with gate disabled: + +| component | polars | ibis | narwhals (both) | +|--------------------|--------|------|-----------------| +| YEAR | hon | hon | hon | +| ISO_YEAR | hon | hon | UNSUPPORTED | +| US_YEAR | UNS | UNS | UNS | +| QUARTER | hon | hon | hon | +| MONTH | hon | hon | hon | +| DAY | hon | hon | hon | +| DAY_OF_YEAR | hon | hon | hon | +| MONDAY_DAY_OF_WEEK | hon | hon | hon | +| SUNDAY_DAY_OF_WEEK | hon | hon | hon | +| MONDAY_WEEK | UNS | UNS | UNS | +| SUNDAY_WEEK | UNS | UNS | UNS | +| ISO_WEEK | hon | hon | UNS | +| US_WEEK | UNS | UNS | UNS | +| HOUR | hon | hon | hon | +| MINUTE | hon | hon | hon | +| SECOND | hon | hon | hon | +| MILLISECOND | hon | hon | hon | +| MICROSECOND | hon | hon | hon | +| NANOSECOND | hon | UNS | hon | +| PICOSECOND | UNS | UNS | UNS | +| SUBSECOND | hon | hon | hon | +| UNIX_TIME | hon | hon | UNS | +| TIMEZONE_OFFSET | UNS | UNS | UNS | + +``TIMEZONE_OFFSET`` is declared everywhere: its native primitives +(``base_utc_offset``/``dst_offset``) require a timezone-aware timestamp, which +``extract`` cannot guarantee at build time — surfaced as input-type-dependent +overload validity (spec §7.3), not a raw backend error. + +``extract_boolean.IS_DST`` is declared on all three backends (item 65's +placeholder); ``IS_LEAP_YEAR`` is honored on all three. + +Family/dialect discipline: ibis gets a family-default (``dialect=None``) fact +plus a concrete ``ibis-duckdb`` refinement; narwhals gets per-dialect facts +only; polars gets a single ``dialect="polars"`` fact (matching the arithmetic +capabilities precedent). +""" +from __future__ import annotations + +from mountainash.core.capabilities import ( + CapabilityFact, + CapabilityLevel, +) +from mountainash.core.constants import CONST_BACKEND +from mountainash.expressions.core.expression_system.function_keys.enums import ( + FKEY_SUBSTRAIT_SCALAR_DATETIME as FK_SUB_DT, +) + +_SINCE = "2026-08-15" + +# Per-backend closed-domain components that are DECLARED UNSUPPORTED (the +# complement of each backend's honored set — the probe matrix above). +_POLARS_UNSUPPORTED = frozenset( + {"US_YEAR", "MONDAY_WEEK", "SUNDAY_WEEK", "US_WEEK", "PICOSECOND", "TIMEZONE_OFFSET"} +) +_IBIS_UNSUPPORTED = frozenset( + { + "US_YEAR", + "MONDAY_WEEK", + "SUNDAY_WEEK", + "US_WEEK", + "NANOSECOND", + "PICOSECOND", + "TIMEZONE_OFFSET", + } +) +_NARWHALS_UNSUPPORTED = frozenset( + { + "ISO_YEAR", + "US_YEAR", + "MONDAY_WEEK", + "SUNDAY_WEEK", + "ISO_WEEK", + "US_WEEK", + "PICOSECOND", + "UNIX_TIME", + "TIMEZONE_OFFSET", + } +) + +_COMPONENT_MSG = ( + "the native backend has no primitive for this extract component " + "(verified by semantic probe; see capabilities/datetime/extract.py)" +) + + +def _component_fact( + backend: CONST_BACKEND, dialect: str | None, component: str +) -> CapabilityFact: + return CapabilityFact( + operation_key=FK_SUB_DT.EXTRACT, + param="component", + option_value=component, + level=CapabilityLevel.UNSUPPORTED, + backend=backend, + dialect=dialect, + message=_COMPONENT_MSG, + since=_SINCE, + ) + + +_POLARS_FACTS = tuple( + _component_fact(CONST_BACKEND.POLARS, "polars", c) for c in sorted(_POLARS_UNSUPPORTED) +) +_IBIS_FACTS = tuple( + _component_fact(CONST_BACKEND.IBIS, None, c) for c in sorted(_IBIS_UNSUPPORTED) +) + tuple( + _component_fact(CONST_BACKEND.IBIS, "ibis-duckdb", c) for c in sorted(_IBIS_UNSUPPORTED) +) +_NARWHALS_FACTS = tuple( + fact + for dialect in ("narwhals-polars", "narwhals-pandas") + for fact in ( + tuple( + _component_fact(CONST_BACKEND.NARWHALS, dialect, c) + for c in sorted(_NARWHALS_UNSUPPORTED) + ) + ) +) + +_IS_DST_MSG = ( + "extract_boolean(IS_DST) is a placeholder (constant False) on all backends; " + "deferred to backlog item 65 (is-dst-placeholder-implementation)" +) + + +def _isdst_fact(backend: CONST_BACKEND, dialect: str | None) -> CapabilityFact: + return CapabilityFact( + operation_key=FK_SUB_DT.EXTRACT_BOOLEAN, + param="component", + option_value="IS_DST", + level=CapabilityLevel.UNSUPPORTED, + backend=backend, + dialect=dialect, + message=_IS_DST_MSG, + since=_SINCE, + ) + + +_IS_DST_FACTS = ( + _isdst_fact(CONST_BACKEND.POLARS, "polars"), + _isdst_fact(CONST_BACKEND.IBIS, None), + _isdst_fact(CONST_BACKEND.IBIS, "ibis-duckdb"), + _isdst_fact(CONST_BACKEND.NARWHALS, "narwhals-polars"), + _isdst_fact(CONST_BACKEND.NARWHALS, "narwhals-pandas"), +) + + +from mountainash.core.capabilities.declarations import ( # noqa: E402 + CapabilityDeclaration, + Domain, + FactSource, + ProbeEvidence, +) + +_EVIDENCE = ProbeEvidence( + probe_date=_SINCE, + library_versions=(("polars", "1.43.2"), ("narwhals", "2.24.0"), ("ibis", "12.0.0")), + fixtures=("polars", "ibis-duckdb", "narwhals-polars", "narwhals-pandas"), +) + +DECLARATIONS = ( + CapabilityDeclaration( + backend=CONST_BACKEND.POLARS, + domain=Domain.DATETIME, + source=FactSource.SUBSTRAIT, + facts=_POLARS_FACTS + (_IS_DST_FACTS[0],), + evidence=_EVIDENCE, + ), + CapabilityDeclaration( + backend=CONST_BACKEND.IBIS, + domain=Domain.DATETIME, + source=FactSource.SUBSTRAIT, + facts=_IBIS_FACTS + _IS_DST_FACTS[1:3], + evidence=_EVIDENCE, + ), + CapabilityDeclaration( + backend=CONST_BACKEND.NARWHALS, + domain=Domain.DATETIME, + source=FactSource.SUBSTRAIT, + facts=_NARWHALS_FACTS + _IS_DST_FACTS[3:5], + evidence=_EVIDENCE, + ), +) diff --git a/src/mountainash/expressions/backends/capabilities/datetime/value_classes_substrait.py b/src/mountainash/expressions/backends/capabilities/datetime/value_classes_substrait.py index 35b34128..8f4f13e9 100644 --- a/src/mountainash/expressions/backends/capabilities/datetime/value_classes_substrait.py +++ b/src/mountainash/expressions/backends/capabilities/datetime/value_classes_substrait.py @@ -60,6 +60,18 @@ "17:30 for Asia/Kolkata)" ) +_STRPTIME_TS_MSG = ( + "strptime_timestamp silently drops the timezone (returns a naive " + "timestamp) on ibis -- ibis has no timezone primitives, matching " + "assume_timezone/to_timezone/local_timestamp/extract.timezone" +) + +_EXTRACT_TZ_MSG = ( + "ibis has no timezone primitives; extract/extract_boolean's timezone " + "option is silently ignored (the local component is read from the " + "stored value, not the target zone) -- see capabilities/datetime/extract.py" +) + def _fact( op_key, message: str, backend, dialect: str | None, since: str = "2026-07-29" @@ -94,6 +106,25 @@ def _fact( since="2026-07-29", ) for dialect in (None, "ibis-duckdb") +) + tuple( + _fact( + FK_SUB_DT.STRPTIME_TIMESTAMP, + _STRPTIME_TS_MSG, + CONST_BACKEND.IBIS, + dialect, + since="2026-08-15", + ) + for dialect in (None, "ibis-duckdb") +) + tuple( + _fact( + op_key, + _EXTRACT_TZ_MSG, + CONST_BACKEND.IBIS, + dialect, + since="2026-08-15", + ) + for op_key in (FK_SUB_DT.EXTRACT, FK_SUB_DT.EXTRACT_BOOLEAN) + for dialect in (None, "ibis-duckdb") ) _NARWHALS_FACTS = tuple( diff --git a/src/mountainash/expressions/backends/expression_systems/ibis/extensions_mountainash/expsys_ib_ext_ma_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/ibis/extensions_mountainash/expsys_ib_ext_ma_scalar_datetime.py index e37a16e4..6708cc7a 100644 --- a/src/mountainash/expressions/backends/expression_systems/ibis/extensions_mountainash/expsys_ib_ext_ma_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/ibis/extensions_mountainash/expsys_ib_ext_ma_scalar_datetime.py @@ -6,10 +6,6 @@ from __future__ import annotations from datetime import date -from mountainash.expressions.core.datetime_components import ( - BooleanComponent, - DatetimeComponent, -) from typing import TYPE_CHECKING, Optional import ibis @@ -57,77 +53,6 @@ class MountainAshIbisScalarDatetimeExpressionSystem(IbisBaseExpressionSystem, Mo Plus convenience methods for common operations. """ - # ========================================================================= - # Core Extraction Methods - # ========================================================================= - - def extract( - self, - x: IbisTemporalExpr, - component: str, - timezone: Optional[str] = None, - /, - ) -> IbisValueExpr: - """Extract portion of a date/time value. - - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. - """ - comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() - - component_map = { - "YEAR": lambda e: e.year(), - "QUARTER": lambda e: e.quarter(), - "MONTH": lambda e: e.month(), - "DAY": lambda e: e.day(), - "DAY_OF_YEAR": lambda e: e.day_of_year(), - "MONDAY_DAY_OF_WEEK": lambda e: e.day_of_week.index() + ibis.literal(1), # 1-indexed - "ISO_WEEK": lambda e: e.week_of_year(), - "HOUR": lambda e: e.hour(), - "MINUTE": lambda e: e.minute(), - "SECOND": lambda e: e.second(), - "MILLISECOND": lambda e: e.millisecond(), - "MICROSECOND": lambda e: e.microsecond(), - "UNIX_TIME": lambda e: e.epoch_seconds(), - } - - if comp in component_map: - return component_map[comp](x) - - return x.year() - - def extract_boolean( - self, - x: IbisValueExpr, - /, - component: str, - ) -> IbisValueExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ - comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() - - if comp == "IS_LEAP_YEAR": - year = x.year() - return ((year % ibis.literal(4) == ibis.literal(0)) & - (year % ibis.literal(100) != ibis.literal(0))) | (year % ibis.literal(400) == ibis.literal(0)) - - if comp == "IS_DST": - return ibis.literal(False) - - return ibis.literal(False) - # ========================================================================= # Convenience Extraction Methods # ========================================================================= diff --git a/src/mountainash/expressions/backends/expression_systems/ibis/substrait/expsys_ib_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/ibis/substrait/expsys_ib_scalar_datetime.py index 353ba448..fed3bddf 100644 --- a/src/mountainash/expressions/backends/expression_systems/ibis/substrait/expsys_ib_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/ibis/substrait/expsys_ib_scalar_datetime.py @@ -7,6 +7,7 @@ from mountainash.expressions.core.datetime_components import ( BooleanComponent, + CALENDAR_COMPONENTS, DatetimeComponent, ) from typing import TYPE_CHECKING, Optional @@ -48,56 +49,55 @@ def extract( x: IbisValueExpr, /, component: str, + indexing: str = None, timezone: str = None, ) -> IbisValueExpr: - """Extract portion of a date/time value. + """Extract a date/time component (Substrait: extract). - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. + ``timezone`` is not honored on ibis (no timezone primitives); the + capability gate raises before this body is reached in production. """ comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() component_map = { "YEAR": lambda e: e.year(), + "ISO_YEAR": lambda e: e.iso_year(), "QUARTER": lambda e: e.quarter(), "MONTH": lambda e: e.month(), "DAY": lambda e: e.day(), "DAY_OF_YEAR": lambda e: e.day_of_year(), - "MONDAY_DAY_OF_WEEK": lambda e: e.day_of_week.index() + ibis.literal(1), # 1-indexed + "MONDAY_DAY_OF_WEEK": lambda e: e.day_of_week.index() + ibis.literal(1), + "SUNDAY_DAY_OF_WEEK": lambda e: (e.day_of_week.index() + ibis.literal(1)) % ibis.literal(7) + ibis.literal(1), "ISO_WEEK": lambda e: e.week_of_year(), "HOUR": lambda e: e.hour(), "MINUTE": lambda e: e.minute(), "SECOND": lambda e: e.second(), "MILLISECOND": lambda e: e.millisecond(), - "MICROSECOND": lambda e: e.microsecond(), + "MICROSECOND": lambda e: e.microsecond() % ibis.literal(1000), + "SUBSECOND": lambda e: e.microsecond(), "UNIX_TIME": lambda e: e.epoch_seconds(), } - if comp in component_map: - return component_map[comp](x) + if comp not in component_map: + raise BackendCapabilityError( + f"extract component {comp!r} is not supported on ibis", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT, + ) - return x.year() + result = component_map[comp](x) + if indexing == "ZERO" and comp in CALENDAR_COMPONENTS: + result = result - ibis.literal(1) + return result def extract_boolean( self, x: IbisValueExpr, /, component: str, + timezone: str = None, ) -> IbisValueExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ + """Extract a boolean date/time component (Substrait: extract_boolean).""" comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() if comp == "IS_LEAP_YEAR": @@ -105,10 +105,11 @@ def extract_boolean( return ((year % ibis.literal(4) == ibis.literal(0)) & (year % ibis.literal(100) != ibis.literal(0))) | (year % ibis.literal(400) == ibis.literal(0)) - if comp == "IS_DST": - return ibis.literal(False) - - return ibis.literal(False) + raise BackendCapabilityError( + f"extract_boolean component {comp!r} is not supported on ibis", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN, + ) @@ -367,7 +368,10 @@ def strptime_timestamp( """ # `.as_timestamp()` returns timestamp('UTC'); the recast restores the # naive wall-clock dtype the previous `cast("timestamp")` produced, so - # the fix changes the parse and nothing else. + # the fix changes the parse and nothing else. `timezone` is ignored -- + # ibis has no timezone primitives (matches assume_timezone/to_timezone/ + # local_timestamp/extract.timezone); a declared_unsupported fact gates + # this in production. return x.as_timestamp(format).cast("timestamp") # ========================================================================= diff --git a/src/mountainash/expressions/backends/expression_systems/narwhals/extensions_mountainash/expsys_nw_ext_ma_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/narwhals/extensions_mountainash/expsys_nw_ext_ma_scalar_datetime.py index 6dfdeff3..cb03b425 100644 --- a/src/mountainash/expressions/backends/expression_systems/narwhals/extensions_mountainash/expsys_nw_ext_ma_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/narwhals/extensions_mountainash/expsys_nw_ext_ma_scalar_datetime.py @@ -6,10 +6,6 @@ from __future__ import annotations from datetime import date, datetime -from mountainash.expressions.core.datetime_components import ( - BooleanComponent, - DatetimeComponent, -) from typing import TYPE_CHECKING, Optional import narwhals as nw @@ -36,85 +32,6 @@ class MountainAshNarwhalsScalarDatetimeExpressionSystem(NarwhalsBaseExpressionSy use workarounds or simplified implementations. """ - # ========================================================================= - # Core Extraction Methods - # ========================================================================= - - def extract( - self, - x: NarwhalsExpr, - component: str, - timezone: Optional[str] = None, - /, - ) -> NarwhalsExpr: - """Extract portion of a date/time value. - - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. - """ - comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() - - component_map = { - "YEAR": lambda e: e.dt.year(), - "QUARTER": lambda e: e.dt.month() // nw.lit(4) + nw.lit(1), - "MONTH": lambda e: e.dt.month(), - "DAY": lambda e: e.dt.day(), - "DAY_OF_YEAR": lambda e: e.dt.ordinal_day(), - "MONDAY_DAY_OF_WEEK": lambda e: e.dt.weekday(), - "HOUR": lambda e: e.dt.hour(), - "MINUTE": lambda e: e.dt.minute(), - "SECOND": lambda e: e.dt.second(), - "MILLISECOND": lambda e: e.dt.millisecond(), - "MICROSECOND": lambda e: e.dt.microsecond(), - "NANOSECOND": lambda e: e.dt.nanosecond(), - } - - if comp == "ISO_WEEK": - from mountainash.core.types import BackendCapabilityError - from mountainash.expressions.core.expression_system.function_keys.enums import FKEY_MOUNTAINASH_SCALAR_DATETIME - raise BackendCapabilityError( - "Narwhals does not support ISO week extraction", - backend="narwhals", - # no generic EXTRACT key exists yet; PR-C adds it (item 62) - function_key=FKEY_MOUNTAINASH_SCALAR_DATETIME.EXTRACT_WEEK, - ) - - if comp in component_map: - return component_map[comp](x) - - return x.dt.year() - - def extract_boolean( - self, - x: NarwhalsExpr, - /, - component: str, - ) -> NarwhalsExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ - comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() - - if comp == "IS_LEAP_YEAR": - year = x.dt.year() - return ((year % nw.lit(4) == nw.lit(0)) & (year % nw.lit(100) != nw.lit(0))) | (year % nw.lit(400) == nw.lit(0)) - - if comp == "IS_DST": - return nw.lit(False) - - return nw.lit(False) - # ========================================================================= # Convenience Extraction Methods # ========================================================================= diff --git a/src/mountainash/expressions/backends/expression_systems/narwhals/substrait/expsys_nw_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/narwhals/substrait/expsys_nw_scalar_datetime.py index 4a0d66b2..20e8afb6 100644 --- a/src/mountainash/expressions/backends/expression_systems/narwhals/substrait/expsys_nw_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/narwhals/substrait/expsys_nw_scalar_datetime.py @@ -7,6 +7,7 @@ from mountainash.expressions.core.datetime_components import ( BooleanComponent, + CALENDAR_COMPONENTS, DatetimeComponent, ) from typing import TYPE_CHECKING, Optional @@ -43,66 +44,72 @@ def extract( x: NarwhalsExpr, /, component: str, + indexing: str = None, timezone: str = None, ) -> NarwhalsExpr: - """Extract portion of a date/time value. - - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. - """ + """Extract a date/time component (Substrait: extract).""" comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() + e = x + if timezone is not None: + e = e.dt.convert_time_zone(timezone) + component_map = { - "YEAR": lambda e: e.dt.year(), - "QUARTER": lambda e: e.dt.month() // nw.lit(4) + nw.lit(1), - "MONTH": lambda e: e.dt.month(), - "DAY": lambda e: e.dt.day(), - "DAY_OF_YEAR": lambda e: e.dt.ordinal_day(), - "MONDAY_DAY_OF_WEEK": lambda e: e.dt.weekday(), - "ISO_WEEK": lambda e: e.dt.week(), - "HOUR": lambda e: e.dt.hour(), - "MINUTE": lambda e: e.dt.minute(), - "SECOND": lambda e: e.dt.second(), - "MILLISECOND": lambda e: e.dt.millisecond(), - "MICROSECOND": lambda e: e.dt.microsecond(), - "NANOSECOND": lambda e: e.dt.nanosecond(), + "YEAR": lambda d: d.dt.year(), + "QUARTER": lambda d: (d.dt.month() - nw.lit(1)) // nw.lit(3) + nw.lit(1), + "MONTH": lambda d: d.dt.month(), + "DAY": lambda d: d.dt.day(), + "DAY_OF_YEAR": lambda d: d.dt.ordinal_day(), + "MONDAY_DAY_OF_WEEK": lambda d: d.dt.weekday(), + "SUNDAY_DAY_OF_WEEK": lambda d: (d.dt.weekday() % nw.lit(7)) + nw.lit(1), + "HOUR": lambda d: d.dt.hour(), + "MINUTE": lambda d: d.dt.minute(), + "SECOND": lambda d: d.dt.second(), + "MILLISECOND": lambda d: d.dt.millisecond(), + "MICROSECOND": lambda d: d.dt.microsecond() % nw.lit(1000), + "NANOSECOND": lambda d: d.dt.nanosecond() % nw.lit(1000), + "SUBSECOND": lambda d: d.dt.microsecond(), } - if comp in component_map: - return component_map[comp](x) + if comp not in component_map: + from mountainash.core.types import BackendCapabilityError + from mountainash.expressions.core.expression_system.function_keys.enums import FKEY_SUBSTRAIT_SCALAR_DATETIME + raise BackendCapabilityError( + f"extract component {comp!r} is not supported on narwhals", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT, + ) - return x.dt.year() + result = component_map[comp](e) + if indexing == "ZERO" and comp in CALENDAR_COMPONENTS: + result = result - nw.lit(1) + return result def extract_boolean( self, x: NarwhalsExpr, /, component: str, + timezone: str = None, ) -> NarwhalsExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ + """Extract a boolean date/time component (Substrait: extract_boolean).""" comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() + e = x + if timezone is not None: + e = e.dt.convert_time_zone(timezone) + if comp == "IS_LEAP_YEAR": - year = x.dt.year() + year = e.dt.year() return ((year % nw.lit(4) == nw.lit(0)) & (year % nw.lit(100) != nw.lit(0))) | (year % nw.lit(400) == nw.lit(0)) - if comp == "IS_DST": - return nw.lit(False) - - return nw.lit(False) + from mountainash.core.types import BackendCapabilityError + from mountainash.expressions.core.expression_system.function_keys.enums import FKEY_SUBSTRAIT_SCALAR_DATETIME + raise BackendCapabilityError( + f"extract_boolean component {comp!r} is not supported on narwhals", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN, + ) @@ -357,9 +364,10 @@ def strptime_timestamp( Returns: Parsed timestamp expression. """ - # `timezone` is not in the def's options tuple, so it never arrives; - # see spec 2026-07-28 section 3.2 (retained park). - return x.str.to_datetime(format=format) + result = x.str.to_datetime(format=format) + if timezone is not None: + result = result.dt.replace_time_zone(timezone) + return result # ========================================================================= # Formatting Methods diff --git a/src/mountainash/expressions/backends/expression_systems/polars/extensions_mountainash/expsys_pl_ext_ma_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/polars/extensions_mountainash/expsys_pl_ext_ma_scalar_datetime.py index ff24b71e..20ca38b8 100644 --- a/src/mountainash/expressions/backends/expression_systems/polars/extensions_mountainash/expsys_pl_ext_ma_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/polars/extensions_mountainash/expsys_pl_ext_ma_scalar_datetime.py @@ -6,10 +6,6 @@ from __future__ import annotations from datetime import date, datetime -from mountainash.expressions.core.datetime_components import ( - BooleanComponent, - DatetimeComponent, -) from typing import TYPE_CHECKING, Optional import polars as pl @@ -38,87 +34,6 @@ class MountainAshPolarsScalarDatetimeExpressionSystem(PolarsBaseExpressionSystem - Formatting: strftime """ - # ========================================================================= - # Core Extraction Methods - # ========================================================================= - - def extract( - self, - x: PolarsExpr, - component: str, - timezone: Optional[str] = None, - /, - ) -> PolarsExpr: - """Extract portion of a date/time value. - - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. - """ - # Handle component as string or enum - comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() - - # Map component to Polars extraction method - component_map = { - "YEAR": lambda e: e.dt.year(), - "ISO_YEAR": lambda e: e.dt.iso_year(), - "QUARTER": lambda e: e.dt.quarter(), - "MONTH": lambda e: e.dt.month(), - "DAY": lambda e: e.dt.day(), - "DAY_OF_YEAR": lambda e: e.dt.ordinal_day(), - "MONDAY_DAY_OF_WEEK": lambda e: e.dt.weekday(), # 1=Monday to 7=Sunday - "SUNDAY_DAY_OF_WEEK": lambda e: (e.dt.weekday() % 7) + 1, # 1=Sunday to 7=Saturday - "ISO_WEEK": lambda e: e.dt.week(), - "MONDAY_WEEK": lambda e: e.dt.week(), - "HOUR": lambda e: e.dt.hour(), - "MINUTE": lambda e: e.dt.minute(), - "SECOND": lambda e: e.dt.second(), - "MILLISECOND": lambda e: e.dt.millisecond(), - "MICROSECOND": lambda e: e.dt.microsecond(), - "NANOSECOND": lambda e: e.dt.nanosecond(), - "SUBSECOND": lambda e: e.dt.microsecond(), # Microseconds since last second - "UNIX_TIME": lambda e: e.dt.epoch("s"), - } - - if comp in component_map: - return component_map[comp](x) - - # Fallback for unhandled components - return x.dt.year() # Default to year - - def extract_boolean( - self, - x: PolarsExpr, - /, - component: str, - ) -> PolarsExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ - comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() - - if comp == "IS_LEAP_YEAR": - # A year is a leap year if divisible by 4, except centuries unless divisible by 400 - year = x.dt.year() - return ((year % 4 == 0) & (year % 100 != 0)) | (year % 400 == 0) - - if comp == "IS_DST": - # Polars doesn't have direct DST detection - # Return a placeholder - would need timezone-aware implementation - return pl.lit(False) - - return pl.lit(False) - # ========================================================================= # Convenience Extraction Methods # ========================================================================= diff --git a/src/mountainash/expressions/backends/expression_systems/polars/substrait/expsys_pl_scalar_datetime.py b/src/mountainash/expressions/backends/expression_systems/polars/substrait/expsys_pl_scalar_datetime.py index c8c735a8..ee21b515 100644 --- a/src/mountainash/expressions/backends/expression_systems/polars/substrait/expsys_pl_scalar_datetime.py +++ b/src/mountainash/expressions/backends/expression_systems/polars/substrait/expsys_pl_scalar_datetime.py @@ -7,6 +7,7 @@ from mountainash.expressions.core.datetime_components import ( BooleanComponent, + CALENDAR_COMPONENTS, DatetimeComponent, ) from typing import TYPE_CHECKING, Optional @@ -44,78 +45,80 @@ def extract( x: PolarsExpr, /, component: str, + indexing: str = None, timezone: str = None, ) -> PolarsExpr: - """Extract portion of a date/time value. + """Extract a date/time component (Substrait: extract). - Args: - x: Datetime expression. - component: Component to extract (YEAR, MONTH, DAY, etc.). - timezone: Timezone string (IANA format). - - Returns: - Extracted component as integer. + ``indexing`` (ONE/ZERO) applies only to calendar components; the + builder rejects it otherwise. ``timezone`` converts the value to the + target IANA zone before the component lookup. """ - # Handle component as string or enum comp = component.value if isinstance(component, DatetimeComponent) else str(component).upper() - # Map component to Polars extraction method + e = x + if timezone is not None: + e = e.dt.convert_time_zone(timezone) + component_map = { - "YEAR": lambda e: e.dt.year(), - "ISO_YEAR": lambda e: e.dt.iso_year(), - "QUARTER": lambda e: e.dt.quarter(), - "MONTH": lambda e: e.dt.month(), - "DAY": lambda e: e.dt.day(), - "DAY_OF_YEAR": lambda e: e.dt.ordinal_day(), - "MONDAY_DAY_OF_WEEK": lambda e: e.dt.weekday(), # 1=Monday to 7=Sunday - "SUNDAY_DAY_OF_WEEK": lambda e: (e.dt.weekday() % 7) + 1, # 1=Sunday to 7=Saturday - "ISO_WEEK": lambda e: e.dt.week(), - "MONDAY_WEEK": lambda e: e.dt.week(), - "HOUR": lambda e: e.dt.hour(), - "MINUTE": lambda e: e.dt.minute(), - "SECOND": lambda e: e.dt.second(), - "MILLISECOND": lambda e: e.dt.millisecond(), - "MICROSECOND": lambda e: e.dt.microsecond(), - "NANOSECOND": lambda e: e.dt.nanosecond(), - "SUBSECOND": lambda e: e.dt.microsecond(), # Microseconds since last second - "UNIX_TIME": lambda e: e.dt.epoch("s"), + "YEAR": lambda d: d.dt.year(), + "ISO_YEAR": lambda d: d.dt.iso_year(), + "QUARTER": lambda d: d.dt.quarter(), + "MONTH": lambda d: d.dt.month(), + "DAY": lambda d: d.dt.day(), + "DAY_OF_YEAR": lambda d: d.dt.ordinal_day(), + "MONDAY_DAY_OF_WEEK": lambda d: d.dt.weekday(), + "SUNDAY_DAY_OF_WEEK": lambda d: (d.dt.weekday() % 7) + 1, + "ISO_WEEK": lambda d: d.dt.week(), + "HOUR": lambda d: d.dt.hour(), + "MINUTE": lambda d: d.dt.minute(), + "SECOND": lambda d: d.dt.second(), + "MILLISECOND": lambda d: d.dt.millisecond(), + "MICROSECOND": lambda d: d.dt.microsecond() % 1000, + "NANOSECOND": lambda d: d.dt.nanosecond() % 1000, + "SUBSECOND": lambda d: d.dt.microsecond(), + "UNIX_TIME": lambda d: d.dt.epoch("s"), } - if comp in component_map: - return component_map[comp](x) - - # Fallback for unhandled components - return x.dt.year() # Default to year + if comp not in component_map: + from mountainash.core.types import BackendCapabilityError + from mountainash.expressions.core.expression_system.function_keys.enums import FKEY_SUBSTRAIT_SCALAR_DATETIME + raise BackendCapabilityError( + f"extract component {comp!r} is not supported on polars", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT, + ) + + result = component_map[comp](e) + if indexing == "ZERO" and comp in CALENDAR_COMPONENTS: + result = result - 1 + return result def extract_boolean( self, x: PolarsExpr, /, component: str, + timezone: str = None, ) -> PolarsExpr: - """Extract boolean values of a date/time value. - - Args: - x: Datetime expression. - component: Boolean component (IS_LEAP_YEAR, IS_DST). - - Returns: - Boolean expression. - """ + """Extract a boolean date/time component (Substrait: extract_boolean).""" comp = component.value if isinstance(component, BooleanComponent) else str(component).upper() + e = x + if timezone is not None: + e = e.dt.convert_time_zone(timezone) + if comp == "IS_LEAP_YEAR": - # A year is a leap year if divisible by 4, except centuries unless divisible by 400 - year = x.dt.year() + year = e.dt.year() return ((year % 4 == 0) & (year % 100 != 0)) | (year % 400 == 0) - if comp == "IS_DST": - # Polars doesn't have direct DST detection - # Return a placeholder - would need timezone-aware implementation - return pl.lit(False) - - return pl.lit(False) - + from mountainash.core.types import BackendCapabilityError + from mountainash.expressions.core.expression_system.function_keys.enums import FKEY_SUBSTRAIT_SCALAR_DATETIME + raise BackendCapabilityError( + f"extract_boolean component {comp!r} is not supported on polars", + backend=self.BACKEND_NAME, + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN, + ) # ========================================================================= diff --git a/src/mountainash/expressions/core/datetime_components.py b/src/mountainash/expressions/core/datetime_components.py index dec53976..ab17093a 100644 --- a/src/mountainash/expressions/core/datetime_components.py +++ b/src/mountainash/expressions/core/datetime_components.py @@ -45,4 +45,20 @@ class BooleanComponent(Enum): IS_DST = "IS_DST" -__all__ = ["DatetimeComponent", "BooleanComponent"] +CALENDAR_COMPONENTS: frozenset[str] = frozenset( + { + "QUARTER", + "MONTH", + "DAY", + "DAY_OF_YEAR", + "MONDAY_DAY_OF_WEEK", + "SUNDAY_DAY_OF_WEEK", + "MONDAY_WEEK", + "SUNDAY_WEEK", + "ISO_WEEK", + "US_WEEK", + } +) + + +__all__ = ["DatetimeComponent", "BooleanComponent", "CALENDAR_COMPONENTS"] diff --git a/src/mountainash/expressions/core/expression_api/api_builders/extensions_mountainash/api_bldr_ext_ma_scalar_string.py b/src/mountainash/expressions/core/expression_api/api_builders/extensions_mountainash/api_bldr_ext_ma_scalar_string.py index f8d71dc3..cf909ea5 100644 --- a/src/mountainash/expressions/core/expression_api/api_builders/extensions_mountainash/api_bldr_ext_ma_scalar_string.py +++ b/src/mountainash/expressions/core/expression_api/api_builders/extensions_mountainash/api_bldr_ext_ma_scalar_string.py @@ -160,12 +160,28 @@ def to_date(self, format: str) -> BaseExpressionAPI: ) return self._build(node) - def to_datetime(self, format: str) -> BaseExpressionAPI: - """Parse string to datetime using format string.""" + def to_datetime(self, format: str, timezone: str = None) -> BaseExpressionAPI: + """Parse string to datetime using format string. + + Args: + format: strptime format string. + timezone: Optional IANA timezone attached to the parsed timestamp. + """ + options = {"format": format} + if timezone is not None: + from ..api_builder_base import _reject_expression + from mountainash.core.capabilities.schema import ValueClass + from ._ma_option_domains import validate_open_value + + _reject_expression("timezone", timezone, "strptime_timestamp") + timezone = validate_open_value( + ValueClass.IANA_TIMEZONE, "timezone", timezone, "strptime_timestamp" + ) + options["timezone"] = timezone node = ScalarFunctionNode( function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_TIMESTAMP, arguments=[self._node], - options={"format": format}, + options=options, ) return self._build(node) diff --git a/src/mountainash/expressions/core/expression_api/api_builders/substrait/_option_domains.py b/src/mountainash/expressions/core/expression_api/api_builders/substrait/_option_domains.py index 8c0c39df..176aaf09 100644 --- a/src/mountainash/expressions/core/expression_api/api_builders/substrait/_option_domains.py +++ b/src/mountainash/expressions/core/expression_api/api_builders/substrait/_option_domains.py @@ -5,6 +5,10 @@ from typing import Any from mountainash.core.errors import InvalidOptionValueError +from mountainash.expressions.core.datetime_components import ( + BooleanComponent, + DatetimeComponent, +) _OVERFLOW = frozenset({"SILENT", "SATURATE", "ERROR"}) @@ -101,6 +105,11 @@ ("swapcase", "char_set"): _CHAR_SET, ("title", "char_set"): _CHAR_SET, ("upper", "char_set"): _CHAR_SET, + # Datetime extraction closed domains. The enums mirror the upstream + # overload union (23 datetime + 2 boolean members) — no new members. + ("extract", "component"): frozenset(c.value for c in DatetimeComponent), + ("extract", "indexing"): frozenset({"ONE", "ZERO"}), + ("extract_boolean", "component"): frozenset(c.value for c in BooleanComponent), } diff --git a/src/mountainash/expressions/core/expression_api/api_builders/substrait/api_bldr_scalar_datetime.py b/src/mountainash/expressions/core/expression_api/api_builders/substrait/api_bldr_scalar_datetime.py index b7ef4fd6..eb80c39a 100644 --- a/src/mountainash/expressions/core/expression_api/api_builders/substrait/api_bldr_scalar_datetime.py +++ b/src/mountainash/expressions/core/expression_api/api_builders/substrait/api_bldr_scalar_datetime.py @@ -1,8 +1,8 @@ """Substrait DateTime operations APIBuilder. -Substrait-aligned implementation for timezone and formatting operations only. -All other datetime operations (extraction, arithmetic, truncation, etc.) -are in MountainAshScalarDatetimeAPIBuilder (the extension builder). +Substrait-aligned implementation for datetime extraction, timezone, and +formatting operations. Arithmetic and truncation live in +MountainAshScalarDatetimeAPIBuilder (the extension builder). """ from __future__ import annotations @@ -14,6 +14,7 @@ from mountainash.expressions.core.expression_system.function_keys.enums import ( FKEY_SUBSTRAIT_SCALAR_DATETIME, ) +from mountainash.expressions.core.datetime_components import CALENDAR_COMPONENTS from mountainash.expressions.core.expression_nodes import ScalarFunctionNode from mountainash.expressions.core.expression_protocols.api_builders.substrait import ( SubstraitScalarDatetimeAPIBuilderProtocol, @@ -29,12 +30,97 @@ class SubstraitScalarDatetimeAPIBuilder( BaseExpressionAPIBuilder, SubstraitScalarDatetimeAPIBuilderProtocol, ): - """Substrait datetime operations (timezone and formatting only). + """Substrait datetime operations (extraction, timezone, formatting). - Substrait defines: local_timestamp, assume_timezone, strftime. - All other datetime operations are in MountainAshScalarDatetimeAPIBuilder. + Substrait defines: extract, extract_boolean, local_timestamp, + assume_timezone, strftime. Arithmetic and truncation are in + MountainAshScalarDatetimeAPIBuilder. """ + def extract( + self, + component: str, + indexing: str = None, + timezone: str = None, + ) -> BaseExpressionAPI: + """Extract a date/time component (Substrait: extract). + + Args: + component: Closed ``DatetimeComponent`` member (e.g. ``"YEAR"``). + indexing: ``"ONE"``/``"ZERO"`` (calendar components only; ``None`` + is the omission sentinel and is behaviour-preserving). + timezone: Optional IANA timezone; converts the value before the + component lookup. + """ + from mountainash.core.errors import InvalidOptionValueError + from ._option_domains import validate_option + from mountainash.core.capabilities.schema import ValueClass + from ..extensions_mountainash._ma_option_domains import validate_open_value + + _reject_expression("component", component, "extract") + component = validate_option("extract", "component", component) + if indexing is not None: + indexing = validate_option("extract", "indexing", indexing) + if component.upper() not in CALENDAR_COMPONENTS: + raise InvalidOptionValueError( + f"extract indexing is only valid on calendar components, " + f"got {component!r}" + ) + if timezone is not None: + _reject_expression("timezone", timezone, "extract") + timezone = validate_open_value( + ValueClass.IANA_TIMEZONE, "timezone", timezone, "extract" + ) + options = {"component": component} + if indexing is not None: + options["indexing"] = indexing + if timezone is not None: + options["timezone"] = timezone + node = ScalarFunctionNode( + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT, + arguments=[self._node], + options=options, + ) + return self._build(node) + + def extract_boolean( + self, + component: str, + timezone: str = None, + ) -> BaseExpressionAPI: + """Extract a boolean date/time component (Substrait: extract_boolean). + + Args: + component: Closed ``BooleanComponent`` member (``IS_LEAP_YEAR`` / + ``IS_DST``). + timezone: Optional IANA timezone. Required for ``IS_DST``. + """ + from mountainash.core.errors import InvalidOptionValueError + from ._option_domains import validate_option + from mountainash.core.capabilities.schema import ValueClass + from ..extensions_mountainash._ma_option_domains import validate_open_value + + _reject_expression("component", component, "extract_boolean") + component = validate_option("extract_boolean", "component", component) + if component.upper() == "IS_DST" and timezone is None: + raise InvalidOptionValueError( + "extract_boolean(IS_DST) requires a timezone" + ) + if timezone is not None: + _reject_expression("timezone", timezone, "extract_boolean") + timezone = validate_open_value( + ValueClass.IANA_TIMEZONE, "timezone", timezone, "extract_boolean" + ) + options = {"component": component} + if timezone is not None: + options["timezone"] = timezone + node = ScalarFunctionNode( + function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN, + arguments=[self._node], + options=options, + ) + return self._build(node) + def local_timestamp(self, timezone: str) -> BaseExpressionAPI: """Get current timestamp in the specified timezone. diff --git a/src/mountainash/expressions/core/expression_protocols/api_builders/substrait/prtcl_api_bldr_scalar_datetime.py b/src/mountainash/expressions/core/expression_protocols/api_builders/substrait/prtcl_api_bldr_scalar_datetime.py index 3d3839a0..0d2fd448 100644 --- a/src/mountainash/expressions/core/expression_protocols/api_builders/substrait/prtcl_api_bldr_scalar_datetime.py +++ b/src/mountainash/expressions/core/expression_protocols/api_builders/substrait/prtcl_api_bldr_scalar_datetime.py @@ -49,6 +49,33 @@ def assume_timezone( """ ... + # ============================================================ + # Extraction (Direct Substrait functions) + # ============================================================ + + def extract( + self, + component: str, + indexing: str | None = None, + timezone: str | None = None, + ) -> BaseExpressionAPI: + """Extract a date/time component. + + Substrait: extract + """ + ... + + def extract_boolean( + self, + component: str, + timezone: str | None = None, + ) -> BaseExpressionAPI: + """Extract a boolean date/time component. + + Substrait: extract_boolean + """ + ... + # ============================================================ # Formatting / Parsing (Direct Substrait functions) # ============================================================ diff --git a/src/mountainash/expressions/core/expression_protocols/expression_systems/extensions_mountainash/prtcl_expsys_ext_ma_scalar_datetime.py b/src/mountainash/expressions/core/expression_protocols/expression_systems/extensions_mountainash/prtcl_expsys_ext_ma_scalar_datetime.py index dbfbbbef..eccb772a 100644 --- a/src/mountainash/expressions/core/expression_protocols/expression_systems/extensions_mountainash/prtcl_expsys_ext_ma_scalar_datetime.py +++ b/src/mountainash/expressions/core/expression_protocols/expression_systems/extensions_mountainash/prtcl_expsys_ext_ma_scalar_datetime.py @@ -402,49 +402,6 @@ def offset_by( """ ... - # ========================================================================= - # Core Dispatch Methods (used by visitor) - # ========================================================================= - - def extract( - self, - x: ExpressionT, - component: str, - timezone: Optional[str] = None, - /, - ) -> ExpressionT: - """Extract a datetime component by name. - - Dispatches to backend-specific extraction (year, month, day, etc.). - - Args: - x: Datetime expression. - component: Component identifier. - - Returns: - Extracted component value. - """ - ... - - def extract_boolean( - self, - x: ExpressionT, - /, - component: str, - ) -> ExpressionT: - """Extract a boolean datetime property. - - Dispatches to backend-specific boolean extraction (is_leap_year, is_dst). - - Args: - x: Datetime expression. - component: Boolean component identifier. - - Returns: - Boolean expression. - """ - ... - # ========================================================================= # Timezone Operations # ========================================================================= diff --git a/src/mountainash/expressions/core/expression_protocols/expression_systems/substrait/prtcl_expsys_scalar_datetime.py b/src/mountainash/expressions/core/expression_protocols/expression_systems/substrait/prtcl_expsys_scalar_datetime.py index 26ae9f48..fa4a3af4 100644 --- a/src/mountainash/expressions/core/expression_protocols/expression_systems/substrait/prtcl_expsys_scalar_datetime.py +++ b/src/mountainash/expressions/core/expression_protocols/expression_systems/substrait/prtcl_expsys_scalar_datetime.py @@ -20,7 +20,7 @@ class SubstraitScalarDatetimeExpressionSystemProtocol(Protocol[ExpressionT]): Function type: scalar """ - def extract(self, x: ExpressionT, /, component: str, timezone: str) -> ExpressionT: + def extract(self, x: ExpressionT, /, component: str, indexing: Optional[str] = None, timezone: Optional[str] = None) -> ExpressionT: """Extract portion of a date/time value. * YEAR Return the year. * ISO_YEAR Return the ISO 8601 week-numbering year. First week of an ISO year has the majority (4 or more) of its days in January. * US_YEAR Return the US epidemiological year. First week of US epidemiological year has the majority (4 or more) @@ -50,7 +50,7 @@ def extract(self, x: ExpressionT, /, component: str, timezone: str) -> Expressio """ ... - def extract_boolean(self, x: ExpressionT, /, component: str) -> ExpressionT: + def extract_boolean(self, x: ExpressionT, /, component: str, timezone: Optional[str] = None) -> ExpressionT: """Extract boolean values of a date/time value. * IS_LEAP_YEAR Return true if year of the given value is a leap year and false otherwise. * IS_DST Return true if DST (Daylight Savings Time) is observed at the given value in the given timezone. @@ -164,7 +164,7 @@ def strptime_date(self, x: ExpressionT, /, format: str) -> ExpressionT: """ ... - def strptime_timestamp(self, x: ExpressionT, /, format: str, timezone: str) -> ExpressionT: + def strptime_timestamp(self, x: ExpressionT, /, format: str, timezone: Optional[str] = None) -> ExpressionT: """Parse string into timestamp using provided format, see https://man7.org/linux/man-pages/man3/strptime.3.html for reference. If timezone is present in timestamp and provided as parameter an error is thrown. Timezone strings must be as defined by IANA timezone database (https://www.iana.org/time-zones). Examples: "Pacific/Marquesas", "Etc/GMT+1". If timezone is supplied as parameter and present in the parsed string the parsed timezone is used. If parameter supplied timezone is invalid an error is thrown. diff --git a/src/mountainash/expressions/core/expression_system/function_mapping/definitions.py b/src/mountainash/expressions/core/expression_system/function_mapping/definitions.py index da80c8cf..ae30e1b9 100644 --- a/src/mountainash/expressions/core/expression_system/function_mapping/definitions.py +++ b/src/mountainash/expressions/core/expression_system/function_mapping/definitions.py @@ -773,7 +773,7 @@ def register_all_functions() -> None: substrait_uri=SubstraitExtension.SCALAR_DATETIME, substrait_name="strptime_timestamp", protocol_method=SubstraitScalarDatetimeExpressionSystemProtocol.strptime_timestamp, - options=("format",), + options=("format", "timezone"), ), ExpressionFunctionDef( function_key=FKEY_MOUNTAINASH_SCALAR_STRING.TO_TIME, @@ -871,13 +871,14 @@ def register_all_functions() -> None: function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT, substrait_uri=SubstraitExtension.SCALAR_DATETIME, substrait_name="extract", - options=("component",), # component="YEAR" + options=("component", "indexing", "timezone"), protocol_method=SubstraitScalarDatetimeExpressionSystemProtocol.extract, ), ExpressionFunctionDef( function_key=FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN, substrait_uri=SubstraitExtension.SCALAR_DATETIME, - substrait_name="extract", + substrait_name="extract_boolean", + options=("component", "timezone"), protocol_method=SubstraitScalarDatetimeExpressionSystemProtocol.extract_boolean, ), ExpressionFunctionDef( diff --git a/tests/_spine_expectation_census.md b/tests/_spine_expectation_census.md index eca47537..fc6311aa 100644 --- a/tests/_spine_expectation_census.md +++ b/tests/_spine_expectation_census.md @@ -172,16 +172,16 @@ _none_ | site | kind | op | backend | param | option | reason | | --- | --- | --- | --- | --- | --- | --- | -| tests/core/test_api_reachability.py:120 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: API reachability gap (fkey not emitted by any public API entry point) — an emission/wiring gap, not a backend capability the spine gates | +| tests/core/test_api_reachability.py:113 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: API reachability gap (fkey not emitted by any public API entry point) — an emission/wiring gap, not a backend capability the spine gates | | tests/core/test_compile_smoke.py:712 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: _KNOWN_SMOKE_FAILURES park — the harness expects a NATIVE (non-BackendCapabilityError) compile failure here; not a spine-gated backend capability | | tests/core/test_compile_smoke.py:750 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: the dynamic compile-smoke catch-all absorber (inventory_has lookup); the absorbed gaps are the runtime found_via=catch-all rows, not an imperative-drain target | | tests/core/test_rel_signature_conformance.py:126 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: relation protocol conformance divergence (signature/dispatch/unhandled-node) — not a backend capability gate | | tests/core/test_rel_signature_conformance.py:236 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: relation protocol conformance divergence (signature/dispatch/unhandled-node) — not a backend capability gate | | tests/core/test_rel_signature_conformance.py:321 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: relation protocol conformance divergence (signature/dispatch/unhandled-node) — not a backend capability gate | -| tests/core/test_signature_conformance.py:249 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | -| tests/core/test_signature_conformance.py:624 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | -| tests/core/test_signature_conformance.py:633 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | -| tests/core/test_signature_conformance.py:641 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | -| tests/core/test_signature_conformance.py:809 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | -| tests/core/test_signature_conformance.py:834 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:235 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:601 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:610 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:618 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:780 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | +| tests/core/test_signature_conformance.py:805 | imperative-xfail | UNRESOLVED | UNRESOLVED | UNRESOLVED | None | non-capability: protocol signature/options/call-pattern conformance divergence — not a backend capability gate | diff --git a/tests/core/test_api_reachability.py b/tests/core/test_api_reachability.py index d3b554db..a53bfa7f 100644 --- a/tests/core/test_api_reachability.py +++ b/tests/core/test_api_reachability.py @@ -35,6 +35,7 @@ def _builders() -> dict[Enum, Callable[[], Any]]: FKEY_MOUNTAINASH_SCALAR_LIST as ML, FKEY_MOUNTAINASH_SCALAR_STRUCT as MS, FKEY_SUBSTRAIT_SCALAR_AGGREGATE as SA, + FKEY_SUBSTRAIT_SCALAR_DATETIME as SD, SUBSTRAIT_ARITHMETIC_WINDOW as SW, ) @@ -58,6 +59,8 @@ def _builders() -> dict[Enum, Callable[[], Any]]: SA.MEDIAN: lambda: ma.median(0, c), # Default method= emits the MA alias key, not the Substrait canonical. SW.RANK: lambda: c.rank(method="min").over("b"), + SD.EXTRACT: lambda: c.dt.extract("YEAR"), + SD.EXTRACT_BOOLEAN: lambda: c.dt.extract_boolean("IS_LEAP_YEAR"), } @@ -73,16 +76,6 @@ def _builders() -> dict[Enum, Callable[[], Any]]: "FKEY_MOUNTAINASH_SCALAR_TERNARY.COLLECT_VALUES": ( "AST-internal marker node, not a compilable expression" ), - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT": ( - "no public builder method emits the canonical Substrait EXTRACT key -- " - "dt.year() and friends emit their own MA keys; wired by PR-C " - "(backlog: substrait-datetime-missing-ops)" - ), - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN": ( - "no public builder method emits the canonical Substrait EXTRACT_BOOLEAN key -- " - "dt.is_leap_year() emits FKEY_MOUNTAINASH_SCALAR_DATETIME.IS_LEAP_YEAR; " - "wired by PR-C (backlog: substrait-datetime-missing-ops)" - ), } diff --git a/tests/core/test_capability_protocol_guard.py b/tests/core/test_capability_protocol_guard.py index 6c194e2b..3491ffd5 100644 --- a/tests/core/test_capability_protocol_guard.py +++ b/tests/core/test_capability_protocol_guard.py @@ -28,6 +28,7 @@ EXPECTED_DECLARATION_MODULES = ( # expressions/backends/capabilities/ "mountainash.expressions.backends.capabilities.arithmetic", + "mountainash.expressions.backends.capabilities.datetime.extract", "mountainash.expressions.backends.capabilities.datetime.options", "mountainash.expressions.backends.capabilities.datetime.strptime", "mountainash.expressions.backends.capabilities.datetime.value_classes_ma", @@ -124,6 +125,7 @@ def _domain_predicate(leaf: str): domains = { "string": Domain.STRING, "arithmetic": Domain.ARITHMETIC, "options": Domain.DATETIME, "strptime": Domain.DATETIME, + "extract": Domain.DATETIME, } want = domains[leaf] return lambda f: ( diff --git a/tests/core/test_protocol_alignment.py b/tests/core/test_protocol_alignment.py index 483683df..eaabbbb3 100644 --- a/tests/core/test_protocol_alignment.py +++ b/tests/core/test_protocol_alignment.py @@ -748,9 +748,6 @@ def _alignment_category(registry_category: str, prefix: str) -> str: # Substrait Field Reference / Literal — special node types, not ScalarFunctionNode (SubstraitFieldReferenceExpressionSystemProtocol, "col"): KnownGap(gap_kind=GapKind.ASPIRATIONAL, reason="Special node type (FieldReferenceNode), not dispatched via function registry", since="2026-05-12"), (SubstraitLiteralExpressionSystemProtocol, "lit"): KnownGap(gap_kind=GapKind.ASPIRATIONAL, reason="Special node type (LiteralNode), not dispatched via function registry", since="2026-05-12"), - # Mountainash Scalar Datetime — methods not yet in function registry - (MountainAshScalarDatetimeExpressionSystemProtocol, "extract"): KnownGap(gap_kind=GapKind.ASPIRATIONAL, reason="No function mapping registered", since="2026-05-12"), - (MountainAshScalarDatetimeExpressionSystemProtocol, "extract_boolean"): KnownGap(gap_kind=GapKind.ASPIRATIONAL, reason="No function mapping registered", since="2026-05-12"), } KNOWN_ASPIRATIONAL_AND_TESTED: dict[tuple[type, str], KnownGap] = { diff --git a/tests/core/test_signature_conformance.py b/tests/core/test_signature_conformance.py index a256c1ed..c3351973 100644 --- a/tests/core/test_signature_conformance.py +++ b/tests/core/test_signature_conformance.py @@ -128,20 +128,6 @@ def test_divide_invokes_legacy_positional_options_with_keyword_rounding( "Protocol string_agg(input, separator, ordering) 3-arg vs Ibis string_agg(x) 1-arg. Since 2026-05-18.", ("SubstraitAggregateStringExpressionSystemProtocol", "string_agg", "narwhals"): "Protocol string_agg(input, separator, ordering) 3-arg vs Narwhals string_agg(x) 1-arg. Since 2026-05-18.", - # extract: protocol extract(component, input) vs backends extract(input) — component is an option - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract", "polars"): - "Protocol extract(component, input) vs Polars extract(input) — component passed as option. Since 2026-05-18.", - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract", "ibis"): - "Protocol extract(component, input) vs Ibis extract(input) — component passed as option. Since 2026-05-18.", - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract", "narwhals"): - "Protocol extract(component, input) vs Narwhals extract(input) — component passed as option. Since 2026-05-18.", - # strptime_timestamp: protocol strptime_timestamp(input, format, timezone) vs backends strptime_timestamp(input) - ("SubstraitScalarDatetimeExpressionSystemProtocol", "strptime_timestamp", "polars"): - "Protocol strptime_timestamp(input, format, timezone) 3-arg vs Polars strptime_timestamp(input) 1-arg. Since 2026-05-18.", - ("SubstraitScalarDatetimeExpressionSystemProtocol", "strptime_timestamp", "ibis"): - "Protocol strptime_timestamp(input, format, timezone) 3-arg vs Ibis strptime_timestamp(input) 1-arg. Since 2026-05-18.", - ("SubstraitScalarDatetimeExpressionSystemProtocol", "strptime_timestamp", "narwhals"): - "Protocol strptime_timestamp(input, format, timezone) 3-arg vs Narwhals strptime_timestamp(input) 1-arg. Since 2026-05-18.", # round: protocol round(x, s) vs backends round(x) — s passed as option ("SubstraitScalarRoundingExpressionSystemProtocol", "round", "polars"): "Protocol round(x, s) vs Polars round(x) — s passed as option. Since 2026-05-18.", @@ -396,6 +382,8 @@ def _init_a2_local_builders() -> dict: FKEY_SUBSTRAIT_SCALAR_LOGARITHMIC.LOGB: lambda: c.log(base=10), FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_DATE: lambda: s.str.to_date("%Y-%m-%d"), FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_TIMESTAMP: lambda: s.str.to_datetime("%Y-%m-%d"), + FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT: lambda: c.dt.extract("YEAR"), + FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN: lambda: c.dt.extract_boolean("IS_LEAP_YEAR"), SUBSTRAIT_ARITHMETIC_WINDOW.PERCENT_RANK: lambda: c.percent_rank().over("b"), SUBSTRAIT_ARITHMETIC_WINDOW.CUME_DIST: lambda: c.cume_dist().over("b"), # Constructs successfully but always 0-ary (order_by_col unreachable @@ -490,19 +478,8 @@ def _collect_a2_cases() -> list[tuple[str, int]]: # (fkey_str) → "reason. Since YYYY-MM-DD." # Escape hatch for A2 arg-count verification: an entry here means the FKEY's # expression truly cannot be constructed with the correct identity via any -# current public API surface (not merely "the generic resolver doesn't find -# it" — check the A2-local builder overlay above first). Expected to be -# small; see f.development-practices/closed-by-default-verification.md. -_KNOWN_UNVERIFIABLE_CALL_PATTERNS: dict[str, str] = { - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT": - "No public builder emits the canonical Substrait EXTRACT key — dt.year() and " - "friends emit their own MA-specific keys (e.g. EXTRACT_YEAR). Matches " - "test_api_reachability.py's own _UNREACHABLE_FKEYS finding. Since 2026-08-11.", - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN": - "No public builder emits the canonical Substrait EXTRACT_BOOLEAN key — " - "dt.is_leap_year() emits FKEY_MOUNTAINASH_SCALAR_DATETIME.IS_LEAP_YEAR. Matches " - "test_api_reachability.py's own _UNREACHABLE_FKEYS finding. Since 2026-08-11.", -} +# current public API surface. +_KNOWN_UNVERIFIABLE_CALL_PATTERNS: dict[str, str] = {} class _A2ConstructionError(Exception): @@ -774,12 +751,6 @@ def _collect_a3_cases() -> list[tuple[str, tuple[str, ...], list[str], list[str] _KNOWN_OPTIONS_DRIFT: dict[str, str] = { "FKEY_SUBSTRAIT_SCALAR_STRING.SUBSTRING": "Registry has 'start','length' but protocol has 'negative_start' — options not aligned. Since 2026-05-18.", - "FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_TIMESTAMP": - "Protocol has required 'timezone' option not in registry — timezone handled by API builder. Since 2026-05-18.", - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT": - "Protocol extract(component, input) — component is a required positional option not in registry. Since 2026-05-18.", - "FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN": - "Protocol extract_boolean(component, input) — component is a required positional option not in registry. Since 2026-05-18.", "FKEY_MOUNTAINASH_NAME.ALIAS": "Protocol has required 'name' kwarg not in registry — handled by API builder directly. Since 2026-05-18.", "FKEY_MOUNTAINASH_NAME.PREFIX": diff --git a/tests/expressions/argument_types/option_disposition.py b/tests/expressions/argument_types/option_disposition.py index 77c69202..820e2427 100644 --- a/tests/expressions/argument_types/option_disposition.py +++ b/tests/expressions/argument_types/option_disposition.py @@ -287,6 +287,14 @@ class InvalidOptionRejection(NamedTuple): # OPTION_DTYPES carries the input column dtype (str), not the option's own type. ("strptime_date", "format"): ("str",), ("strptime_timestamp", "format"): ("str",), + ("strptime_timestamp", "timezone"): ("str",), + # Datetime extraction (item 62): closed component/indexing domains and + # open IANA-timezone value class operate on the datetime operand column. + ("extract", "component"): ("datetime",), + ("extract", "indexing"): ("datetime",), + ("extract", "timezone"): ("datetime",), + ("extract_boolean", "component"): ("datetime",), + ("extract_boolean", "timezone"): ("datetime",), } # Representative legal values for open-integer options that have NO finite @@ -328,6 +336,10 @@ class InvalidOptionRejection(NamedTuple): # so the result differs when input has day > 12. ("strptime_date", "format"): ("%Y-%m-%d", "%Y-%d-%m"), ("strptime_timestamp", "format"): ("%Y-%m-%d %H:%M:%S", "%Y-%d-%m %H:%M:%S"), + # IANA-timezone value class on extract / extract_boolean (item 62). + ("extract", "timezone"): ("UTC", "Australia/Sydney", "America/New_York"), + ("extract_boolean", "timezone"): ("UTC", "Australia/Sydney", "America/New_York"), + ("strptime_timestamp", "timezone"): ("UTC", "Australia/Sydney", "America/New_York"), } diff --git a/tests/expressions/argument_types/test_arg_types_datetime.py b/tests/expressions/argument_types/test_arg_types_datetime.py index 37f528ad..6b0939e1 100644 --- a/tests/expressions/argument_types/test_arg_types_datetime.py +++ b/tests/expressions/argument_types/test_arg_types_datetime.py @@ -28,6 +28,10 @@ import mountainash as ma from mountainash.core.errors import InvalidOptionValueError from mountainash.core.types import BackendCapabilityError +from mountainash.expressions.core.datetime_components import ( + BooleanComponent, + DatetimeComponent, +) from mountainash.expressions.core.expression_system.function_keys.enums import ( FKEY_SUBSTRAIT_SCALAR_DATETIME as FK_DT, FKEY_MOUNTAINASH_SCALAR_DATETIME as FK_MA_DT, @@ -1505,6 +1509,638 @@ def _strptime_ts_probe(fmt: str, backend: str) -> OptionSpec: for fmt in _STRPTIME_TS_DOMAIN ) +# 7. strptime_timestamp.timezone (item 62 — end-to-end wiring) +_STRPTIME_TS_TZ_DOMAIN = ("UTC", "Australia/Sydney", "America/New_York") +_STRPTIME_TS_TZ_DATA = {"s": ["2024-01-05 06:07:08"]} + + +def _strptime_ts_tz_expr(tz: str): + return ma.col("s").str.to_datetime("%Y-%m-%d %H:%M:%S", timezone=tz) + + +def _strptime_ts_tz_ref_expr(tz: str): + ref = "UTC" if tz != "UTC" else "Australia/Sydney" + return _strptime_ts_tz_expr(ref) + + +def _strptime_ts_tz_disposition(backend: str) -> str: + return "declared_unsupported" if backend == "ibis" else "honored" + + +def _strptime_ts_tz_probe(tz: str, backend: str) -> OptionSpec: + if _strptime_ts_tz_disposition(backend) == "honored": + return OptionSpec( + FK_DT.STRPTIME_TIMESTAMP, "timezone", tz, "str", + lambda t=tz: _strptime_ts_tz_expr(t), + lambda t=tz: _strptime_ts_tz_ref_expr(t), + _STRPTIME_TS_TZ_DATA, expected_discriminates=True, + ) + return OptionSpec( + FK_DT.STRPTIME_TIMESTAMP, "timezone", tz, "str", + lambda t=tz: _strptime_ts_tz_expr(t), + lambda t=tz: _strptime_ts_tz_expr(t), + _STRPTIME_TS_TZ_DATA, expected_discriminates=True, + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.STRPTIME_TIMESTAMP, + _SUBSTRAIT_DT_PROTOCOL, + "strptime_timestamp", + "timezone", + backend, + tz, + "str", + _strptime_ts_tz_disposition(backend), + ( + "ibis has no timezone primitives; the timezone option is silently ignored" + if backend == "ibis" + else "native backend attaches the parsed timezone" + ), + "class" if backend == "ibis" else "absence", + ) + for backend in ALL_BACKENDS + for tz in _STRPTIME_TS_TZ_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _strptime_ts_tz_probe(tz, backend), + backend, + _strptime_ts_tz_disposition(backend), + OptionProbeDidNotDiscriminateError + if _strptime_ts_tz_disposition(backend) == "declared_unsupported" + else None, + ) + for backend in ALL_BACKENDS + for tz in _STRPTIME_TS_TZ_DOMAIN +) + +_STRPTIME_TS_TZ_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.STRPTIME_TIMESTAMP, + _SUBSTRAIT_DT_PROTOCOL, + "strptime_timestamp", + "timezone", + INVALID_OPTION_VALUE, + "str", + lambda: _strptime_ts_tz_expr(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_STRPTIME_TS_TZ_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _STRPTIME_TS_TZ_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _STRPTIME_TS_TZ_INVALID_REJECTIONS) +def test_strptime_timestamp_timezone_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + + + +# ============================================================================ +# extract / extract_boolean (item 62) +# ============================================================================ + +_EXTRACT_COMPONENT_DOMAIN = tuple(c.value for c in DatetimeComponent) +_EXTRACT_BOOL_COMPONENT_DOMAIN = tuple(c.value for c in BooleanComponent) +_EXTRACT_INDEXING_DOMAIN = ("ONE", "ZERO") +_EXTRACT_TZ_DOMAIN = ("UTC", "Australia/Sydney", "America/New_York") +# Boundary data: Jan 1 00:30 interpreted as UTC flips to Dec 31 of the prior +# year in America/New_York, so IS_LEAP_YEAR(timezone=...) discriminates across +# zones (2024 leap vs 2023 non-leap). +_EXTRACT_BOOL_TZ_DATA = {"ts": [datetime(2024, 1, 1, 0, 30, 0)]} + +# Per-backend component sets the native backend cannot produce (probe-authoritative). +_EXTRACT_DECLARED = { + "polars": frozenset( + {"US_YEAR", "MONDAY_WEEK", "SUNDAY_WEEK", "US_WEEK", "PICOSECOND", "TIMEZONE_OFFSET"} + ), + "ibis": frozenset( + { + "US_YEAR", "MONDAY_WEEK", "SUNDAY_WEEK", "US_WEEK", + "NANOSECOND", "PICOSECOND", "TIMEZONE_OFFSET", + } + ), + "narwhals-polars": frozenset( + { + "ISO_YEAR", "US_YEAR", "MONDAY_WEEK", "SUNDAY_WEEK", "ISO_WEEK", + "US_WEEK", "PICOSECOND", "UNIX_TIME", "TIMEZONE_OFFSET", + } + ), + "narwhals-pandas": frozenset( + { + "ISO_YEAR", "US_YEAR", "MONDAY_WEEK", "SUNDAY_WEEK", "ISO_WEEK", + "US_WEEK", "PICOSECOND", "UNIX_TIME", "TIMEZONE_OFFSET", + } + ), +} + + +def _extract_expr(comp: str): + return ma.col("ts").dt.extract(comp) + + +def _extract_ref_expr(comp: str): + # Reference component chosen to differ from every honored component on the + # probe fixture (month=7/day=21 never collide with year/week/… values). + return _extract_expr("MONTH" if comp != "MONTH" else "DAY") + + +def _extract_disposition(backend: str, comp: str) -> str: + return "declared_unsupported" if comp in _EXTRACT_DECLARED[backend] else "honored" + + +def _extract_component_probe(comp: str, backend: str) -> OptionSpec: + if _extract_disposition(backend, comp) == "honored": + return OptionSpec( + FK_DT.EXTRACT, "component", comp, "datetime", + lambda c=comp: _extract_expr(c), + lambda c=comp: _extract_ref_expr(c), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + return OptionSpec( + FK_DT.EXTRACT, "component", comp, "datetime", + lambda c=comp: _extract_expr(c), + lambda c=comp: _extract_expr(c), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "component", + backend, + comp, + "datetime", + _extract_disposition(backend, comp), + ( + "native backend has no primitive for this extract component " + "(probe-authoritative)" + if _extract_disposition(backend, comp) == "declared_unsupported" + else "native backend honors this extract component" + ), + "absence", + ) + for backend in ALL_BACKENDS + for comp in _EXTRACT_COMPONENT_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _extract_component_probe(comp, backend), + backend, + _extract_disposition(backend, comp), + BackendCapabilityError + if _extract_disposition(backend, comp) == "declared_unsupported" + else None, + ) + for backend in ALL_BACKENDS + for comp in _EXTRACT_COMPONENT_DOMAIN +) + +OPTION_FAMILY_DEFAULT_FACT_KEYS.update( + (FK_DT.EXTRACT, "component", comp, CONST_BACKEND.IBIS, None) + for comp in sorted(_EXTRACT_DECLARED["ibis"]) +) +OPTION_FAMILY_DEFAULT_FACT_KEYS.add( + (FK_DT.EXTRACT_BOOLEAN, "component", "IS_DST", CONST_BACKEND.IBIS, None) +) + +_EXTRACT_COMPONENT_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "component", + INVALID_OPTION_VALUE, + "datetime", + lambda: _extract_expr(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_EXTRACT_COMPONENT_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _EXTRACT_COMPONENT_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _EXTRACT_COMPONENT_INVALID_REJECTIONS) +def test_extract_component_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + + +def _extract_indexing_expr(idx: str): + return ma.col("ts").dt.extract("MONTH", indexing=idx) + + +def _extract_indexing_probe(idx: str) -> OptionSpec: + return OptionSpec( + FK_DT.EXTRACT, "indexing", idx, "datetime", + lambda i=idx: _extract_indexing_expr(i), + lambda: ma.col("ts").dt.extract("MONTH"), + _DATETIME_UNIT_DATA, + expected_discriminates=(idx == "ZERO"), + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "indexing", + backend, + idx, + "datetime", + "honored", + "ONE is the native 1-based default; ZERO subtracts one on calendar components", + "absence", + ) + for backend in ALL_BACKENDS + for idx in _EXTRACT_INDEXING_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _extract_indexing_probe(idx), + backend, + "honored", + None, + ) + for backend in ALL_BACKENDS + for idx in _EXTRACT_INDEXING_DOMAIN +) + +_EXTRACT_INDEXING_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "indexing", + INVALID_OPTION_VALUE, + "datetime", + lambda: _extract_indexing_expr(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_EXTRACT_INDEXING_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _EXTRACT_INDEXING_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _EXTRACT_INDEXING_INVALID_REJECTIONS) +def test_extract_indexing_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + + +def _extract_tz_expr(tz: str): + return ma.col("ts").dt.extract("HOUR", timezone=tz) + + +def _extract_tz_ref_expr(tz: str): + ref = "UTC" if tz != "UTC" else "Australia/Sydney" + return _extract_tz_expr(ref) + + +def _extract_tz_disposition(backend: str) -> str: + return "declared_unsupported" if backend == "ibis" else "honored" + + +def _extract_tz_probe(tz: str, backend: str) -> OptionSpec: + if _extract_tz_disposition(backend) == "honored": + return OptionSpec( + FK_DT.EXTRACT, "timezone", tz, "datetime", + lambda t=tz: _extract_tz_expr(t), + lambda t=tz: _extract_tz_ref_expr(t), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + return OptionSpec( + FK_DT.EXTRACT, "timezone", tz, "datetime", + lambda t=tz: _extract_tz_expr(t), + lambda t=tz: _extract_tz_expr(t), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "timezone", + backend, + tz, + "datetime", + _extract_tz_disposition(backend), + ( + "ibis has no timezone primitives; the timezone option is silently ignored" + if backend == "ibis" + else "native backend converts to the target zone before the component lookup" + ), + "class" if backend == "ibis" else "absence", + ) + for backend in ALL_BACKENDS + for tz in _EXTRACT_TZ_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _extract_tz_probe(tz, backend), + backend, + _extract_tz_disposition(backend), + OptionProbeDidNotDiscriminateError + if _extract_tz_disposition(backend) == "declared_unsupported" + else None, + ) + for backend in ALL_BACKENDS + for tz in _EXTRACT_TZ_DOMAIN +) + +_EXTRACT_TZ_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.EXTRACT, + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "timezone", + INVALID_OPTION_VALUE, + "datetime", + lambda: _extract_tz_expr(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_EXTRACT_TZ_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _EXTRACT_TZ_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _EXTRACT_TZ_INVALID_REJECTIONS) +def test_extract_timezone_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + + +def _extract_boolean_expr(comp: str): + if comp == "IS_DST": + return ma.col("ts").dt.extract_boolean("IS_DST", timezone="UTC") + return ma.col("ts").dt.extract_boolean(comp) + + +def _extract_boolean_disposition(comp: str) -> str: + return "declared_unsupported" if comp == "IS_DST" else "honored" + + +def _extract_boolean_component_probe(comp: str, backend: str) -> OptionSpec: + if _extract_boolean_disposition(comp) == "honored": + return OptionSpec( + FK_DT.EXTRACT_BOOLEAN, "component", comp, "datetime", + lambda c=comp: _extract_boolean_expr(c), + lambda: ma.col("ts").dt.extract("MONTH"), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + return OptionSpec( + FK_DT.EXTRACT_BOOLEAN, "component", comp, "datetime", + lambda c=comp: _extract_boolean_expr(c), + lambda c=comp: _extract_boolean_expr(c), + _DATETIME_UNIT_DATA, expected_discriminates=True, + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.EXTRACT_BOOLEAN, + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "component", + backend, + comp, + "datetime", + _extract_boolean_disposition(comp), + ( + "IS_DST is a placeholder (constant False); deferred to backlog item 65" + if comp == "IS_DST" + else "native backend honors IS_LEAP_YEAR" + ), + "absence", + ) + for backend in ALL_BACKENDS + for comp in _EXTRACT_BOOL_COMPONENT_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _extract_boolean_component_probe(comp, backend), + backend, + _extract_boolean_disposition(comp), + BackendCapabilityError + if _extract_boolean_disposition(comp) == "declared_unsupported" + else None, + ) + for backend in ALL_BACKENDS + for comp in _EXTRACT_BOOL_COMPONENT_DOMAIN +) + +_EXTRACT_BOOL_COMPONENT_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.EXTRACT_BOOLEAN, + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "component", + INVALID_OPTION_VALUE, + "datetime", + lambda: ma.col("ts").dt.extract_boolean(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_EXTRACT_BOOL_COMPONENT_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _EXTRACT_BOOL_COMPONENT_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _EXTRACT_BOOL_COMPONENT_INVALID_REJECTIONS) +def test_extract_boolean_component_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + + +def _extract_bool_tz_expr(tz: str): + return ma.col("ts").dt.extract_boolean("IS_LEAP_YEAR", timezone=tz) + + +def _extract_bool_tz_ref_expr(tz: str): + ref = "America/New_York" if tz != "America/New_York" else "UTC" + return _extract_bool_tz_expr(ref) + + +def _extract_bool_tz_probe(tz: str, backend: str) -> OptionSpec: + if backend == "ibis": + return OptionSpec( + FK_DT.EXTRACT_BOOLEAN, "timezone", tz, "datetime", + lambda t=tz: _extract_bool_tz_expr(t), + lambda t=tz: _extract_bool_tz_expr(t), + _EXTRACT_BOOL_TZ_DATA, expected_discriminates=True, + ) + return OptionSpec( + FK_DT.EXTRACT_BOOLEAN, "timezone", tz, "datetime", + lambda t=tz: _extract_bool_tz_expr(t), + lambda t=tz: _extract_bool_tz_ref_expr(t), + _EXTRACT_BOOL_TZ_DATA, expected_discriminates=True, + ) + + +OPTION_DISPOSITIONS.extend( + OptionCell( + FK_DT.EXTRACT_BOOLEAN, + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "timezone", + backend, + tz, + "datetime", + _extract_tz_disposition(backend), + ( + "ibis has no timezone primitives; the timezone option is silently ignored" + if backend == "ibis" + else "native backend converts to the target zone before the boolean lookup" + ), + "class" if backend == "ibis" else "absence", + ) + for backend in ALL_BACKENDS + for tz in _EXTRACT_TZ_DOMAIN +) + +REGISTERED_OPTION_PROBES.extend( + OptionProbeRegistration( + _extract_bool_tz_probe(tz, backend), + backend, + _extract_tz_disposition(backend), + OptionProbeDidNotDiscriminateError + if _extract_tz_disposition(backend) == "declared_unsupported" + else None, + ) + for backend in ALL_BACKENDS + for tz in _EXTRACT_TZ_DOMAIN +) + +_EXTRACT_BOOL_TZ_INVALID_REJECTIONS = [ + InvalidOptionRejection( + FK_DT.EXTRACT_BOOLEAN, + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "timezone", + INVALID_OPTION_VALUE, + "datetime", + lambda: _extract_bool_tz_expr(INVALID_OPTION_VALUE), + ) +] +REGISTERED_INVALID_OPTION_REJECTIONS.extend(_EXTRACT_BOOL_TZ_INVALID_REJECTIONS) +OPTION_DISPOSITIONS.extend( + OptionCell( + rejection.fkey, + rejection.protocol, + rejection.op, + rejection.param, + backend, + rejection.value, + rejection.dtype, + "invalid", + "canonical build-time rejection sentinel; invalid strings are unbounded", + "absence", + ) + for rejection in _EXTRACT_BOOL_TZ_INVALID_REJECTIONS + for backend in ALL_BACKENDS +) + + +@pytest.mark.parametrize("rejection", _EXTRACT_BOOL_TZ_INVALID_REJECTIONS) +def test_extract_boolean_timezone_invalid_option_rejected_at_build_time( + rejection: InvalidOptionRejection, +) -> None: + with pytest.raises(InvalidOptionValueError): + rejection.build_expr() + TESTED_OPTION_PARAMS: list[tuple] = [] TESTED_OPTION_PARAMS.extend( @@ -1559,6 +2195,42 @@ def _strptime_ts_probe(fmt: str, backend: str) -> OptionSpec: "format", param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "strptime_timestamp", "format"), ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "strptime_timestamp", + "timezone", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "strptime_timestamp", "timezone"), + ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "component", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "extract", "component"), + ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "indexing", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "extract", "indexing"), + ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "extract", + "timezone", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "extract", "timezone"), + ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "component", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "extract_boolean", "component"), + ), + ( + _SUBSTRAIT_DT_PROTOCOL, + "extract_boolean", + "timezone", + param_taxonomy(_SUBSTRAIT_DT_PROTOCOL, "extract_boolean", "timezone"), + ), ]) diff --git a/tests/expressions/argument_types/test_coverage_guard.py b/tests/expressions/argument_types/test_coverage_guard.py index ec3373cb..d7a1d9b4 100644 --- a/tests/expressions/argument_types/test_coverage_guard.py +++ b/tests/expressions/argument_types/test_coverage_guard.py @@ -822,12 +822,6 @@ def test_collect_option_param_taxonomy_rejects_conflicting_duplicates(monkeypatc since="2026-07-25", ) for key in { - ("MountainAshScalarDatetimeExpressionSystemProtocol", "extract", "component"), - ("MountainAshScalarDatetimeExpressionSystemProtocol", "extract", "timezone"), - ("MountainAshScalarDatetimeExpressionSystemProtocol", "extract_boolean", "component"), - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract", "component"), - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract", "timezone"), - ("SubstraitScalarDatetimeExpressionSystemProtocol", "extract_boolean", "component"), ("SubstraitScalarDatetimeExpressionSystemProtocol", "round_calendar", "multiple"), ("SubstraitScalarDatetimeExpressionSystemProtocol", "round_calendar", "origin"), ("SubstraitScalarDatetimeExpressionSystemProtocol", "round_calendar", "rounding"), @@ -852,19 +846,6 @@ def test_collect_option_param_taxonomy_rejects_conflicting_duplicates(monkeypatc ), since="2026-07-25", ) -_KNOWN_UNTESTED_OPTION_PARAMS[ - ("SubstraitScalarDatetimeExpressionSystemProtocol", "strptime_timestamp", "timezone") -] = KnownGap( - gap_kind=GapKind.UNTESTED_OPTION, - reason=( - "the op IS API-reachable via str.to_datetime, but its ExpressionFunctionDef " - "declares only options=('format',), so the timezone argument never reaches " - "the backend; deferred with `precision` as parameters of the same upstream " - "overload family — see backlog: substrait-datetime-missing-ops section 4.4" - ), - since="2026-07-30", -) - for _case_sensitivity_op in ( "contains", @@ -1322,16 +1303,6 @@ def test_collect_option_param_taxonomy_rejects_conflicting_duplicates(monkeypatc tuple[str, str, str], tuple[str, str, str], ] = { - ( - "SubstraitScalarDatetimeExpressionSystemProtocol", - "extract", - "x", - ): ("MountainAshScalarDatetimeExpressionSystemProtocol", "extract", "x"), - ( - "SubstraitScalarDatetimeExpressionSystemProtocol", - "extract_boolean", - "x", - ): ("MountainAshScalarDatetimeExpressionSystemProtocol", "extract_boolean", "x"), ( "SubstraitScalarSetExpressionSystemProtocol", "is_in", diff --git a/tests/expressions/argument_types/test_option_fact_integrity.py b/tests/expressions/argument_types/test_option_fact_integrity.py index f745bf7b..9f93d191 100644 --- a/tests/expressions/argument_types/test_option_fact_integrity.py +++ b/tests/expressions/argument_types/test_option_fact_integrity.py @@ -613,7 +613,7 @@ def test_representative_dtype_policy_exactly_covers_option_domain_owners() -> No ("modulus", "on_domain_error"), } else ("datetime",) - if key[1] in {"unit", "timezone", "offset", "format"} + if key[1] in {"unit", "timezone", "offset", "format", "component", "indexing"} and key[0] not in {"strptime_date", "strptime_timestamp"} else ("str",) if key[0] in {"strptime_date", "strptime_timestamp"} diff --git a/tests/expressions/ast/test_ast_scalar_datetime.py b/tests/expressions/ast/test_ast_scalar_datetime.py index 4115916c..48acd3fe 100644 --- a/tests/expressions/ast/test_ast_scalar_datetime.py +++ b/tests/expressions/ast/test_ast_scalar_datetime.py @@ -127,6 +127,41 @@ def test_strftime(self): assert node.function_key == FKEY_SUBSTRAIT_SCALAR_DATETIME.STRFTIME +class TestSubstraitExtract: + """item 62: canonical Substrait extract/extract_boolean AST identity.""" + + def test_extract_component_only(self): + expr = ma.col("ts").dt.extract("YEAR") + node = expr._node + assert isinstance(node, ScalarFunctionNode) + assert node.function_key == FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT + assert len(node.arguments) == 1 + assert node.options == {"component": "YEAR"} + + def test_extract_with_indexing(self): + expr = ma.col("ts").dt.extract("MONTH", indexing="ZERO") + node = expr._node + assert node.options == {"component": "MONTH", "indexing": "ZERO"} + + def test_extract_with_timezone(self): + expr = ma.col("ts").dt.extract("HOUR", timezone="UTC") + node = expr._node + assert node.options == {"component": "HOUR", "timezone": "UTC"} + + def test_extract_boolean_component_only(self): + expr = ma.col("ts").dt.extract_boolean("IS_LEAP_YEAR") + node = expr._node + assert isinstance(node, ScalarFunctionNode) + assert node.function_key == FKEY_SUBSTRAIT_SCALAR_DATETIME.EXTRACT_BOOLEAN + assert len(node.arguments) == 1 + assert node.options == {"component": "IS_LEAP_YEAR"} + + def test_extract_boolean_with_timezone(self): + expr = ma.col("ts").dt.extract_boolean("IS_DST", timezone="UTC") + node = expr._node + assert node.options == {"component": "IS_DST", "timezone": "UTC"} + + class TestDatetimeComponents: def test_date(self): expr = ma.col("ts").dt.date() diff --git a/tests/expressions/ast/test_ast_str_to_date.py b/tests/expressions/ast/test_ast_str_to_date.py index b7400f15..83e3b754 100644 --- a/tests/expressions/ast/test_ast_str_to_date.py +++ b/tests/expressions/ast/test_ast_str_to_date.py @@ -25,4 +25,18 @@ def test_to_datetime_creates_node(self): assert node.function_key == FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_TIMESTAMP assert len(node.arguments) == 1 assert isinstance(node.arguments[0], FieldReferenceNode) - assert node.options.get("format") == "%Y-%m-%d %H:%M:%S" + assert node.options == {"format": "%Y-%m-%d %H:%M:%S"} + + def test_to_datetime_with_timezone_includes_option(self): + """item 62: timezone reaches the node end-to-end (options dict), not + just the builder signature.""" + expr = ma.col("dt_str").str.to_datetime("%Y-%m-%d %H:%M:%S", timezone="UTC") + node = expr._node + assert node.function_key == FKEY_SUBSTRAIT_SCALAR_DATETIME.STRPTIME_TIMESTAMP + assert node.options == {"format": "%Y-%m-%d %H:%M:%S", "timezone": "UTC"} + + def test_to_datetime_without_timezone_omits_option(self): + """Omission is distinct from an explicit value at the AST level.""" + expr = ma.col("dt_str").str.to_datetime("%Y-%m-%d %H:%M:%S") + node = expr._node + assert "timezone" not in node.options diff --git a/tests/expressions/backends/test_narwhals_extract_capability_error.py b/tests/expressions/backends/test_narwhals_extract_capability_error.py deleted file mode 100644 index ce914e36..00000000 --- a/tests/expressions/backends/test_narwhals_extract_capability_error.py +++ /dev/null @@ -1,16 +0,0 @@ -import pytest -from mountainash.core.types import BackendCapabilityError -from mountainash.expressions.backends.expression_systems.narwhals.extensions_mountainash.expsys_nw_ext_ma_scalar_datetime import ( - MountainAshNarwhalsScalarDatetimeExpressionSystem, -) - - -def test_narwhals_iso_week_raises_capability_error_not_attribute_error(): - """The ISO_WEEK branch must raise BackendCapabilityError, not AttributeError.""" - import narwhals as nw - - system = MountainAshNarwhalsScalarDatetimeExpressionSystem() - # component is POSITIONAL_ONLY -- passing it as a keyword raises TypeError - # before the ISO_WEEK branch is reached. - with pytest.raises(BackendCapabilityError): - system.extract(nw.col("x"), "ISO_WEEK")