Skip to content

fix(relations,fixtures): ibis-sqlite null date/timestamp binding crash (item 112, IB-DT-19) - #303

Merged
discreteds merged 1 commit into
developfrom
discreteds/item112
Aug 19, 2026
Merged

fix(relations,fixtures): ibis-sqlite null date/timestamp binding crash (item 112, IB-DT-19)#303
discreteds merged 1 commit into
developfrom
discreteds/item112

Conversation

@discreteds

Copy link
Copy Markdown
Member

Summary

Fixes item 112 — cross-backend test fixture factory crashes on null date/timestamp for ibis-sqlite — with scope expanded (see backlog doc, mountainash-central commit 425f6e7) to also cover mountainash's own production code, since the same crash is reachable independent of the original test-infra-only framing.

Root cause: ibis.backends.sqlite.Backend._register_in_memory_table always stages an in-memory table via a pandas roundtrip (op.data.to_frame()) before binding rows through stdlib sqlite3. A null date/timestamp value becomes pandas NaT, and sqlite3 has no adapter for NaTType — raises sqlite3.ProgrammingError before Mountainash's own visitor/compile machinery ever runs. Verified empirically (ibis 12.0.0) that explicit PyArrow schemas, ibis.memtable(schema=...), and object-dtype pandas columns all still crash — ibis re-normalises internally before the roundtrip regardless of input shape.

Fix: relations/backends/relation_systems/ibis/_sqlite_compat.py::ensure_sqlite_nat_adapter() — one idempotent, process-global sqlite3.register_adapter(NaTType, ...) call, invoked lazily wherever Mountainash builds an ibis.memtable()/create_table() from data that could contain a null temporal value:

  • Production (new scope, found during implementation): relation_visitor.py's cross-type join coercion, relsys_ib_ext_ma_util.py's resource-ingestion fallback (2 call sites). These crash independent of whether mountainash-data ships its own DEBT-13 fix, since Mountainash builds ibis.memtable() directly, not through that layer.
  • Test fixtures: the 4 duplicated ibis-sqlite table-creation call sites across backend_registry.py/backend_helpers.py/conftest.py now delegate to one shared helper instead of re-registering the workaround.

Rejected alternatives:

  • pandas.DataFrame.to_sql() bypass avoids the crash but silently degrades Boolean columns to Int64 — an observable regression for every existing non-null fixture.
  • Eager top-level adapter registration at package-import time — ibis-framework has no unconditional pandas dependency (only specific backend extras do), so this would regress every pandas-free ibis backend user.

Tests

  • tests/fixtures/test_backend_registry.py, test_backend_helpers.py (new file): null date + null datetime regression fixture, parametrized over ALL_BACKENDS, for REGISTRY.build and BackendDataFrameFactory.create/create_pair.
  • test_backend_registry.py::test_raw_ibis_sqlite_null_temporal_upstream_bug_ib_dt_19: xfail(strict=True) upstream-fix monitor — reproduces the raw crash in a subprocess isolated from Mountainash's own process-global adapter patch, so it flips to XPASS(strict) and fails CI the moment ibis fixes IB-DT-19 upstream.
  • backend_helpers.py::get_count: fixed a pre-existing narwhals-lazy gap (missing .collect()) that blocked the new ALL_BACKENDS parametrization — unrelated to the main fix, found while writing the regression test.
  • Manually verified the exact production scenario end-to-end (ma.relation(ibis_sqlite_table).join(polars_df_with_null_datetime)) before/after the fix.

Verification run

tests/fixtures/ + tests/relations/: 2685 passed, 11 skipped, 36 xfailed, 0 failed
ruff check: all checks passed (touched + new files)
mypy check: 9 pre-existing errors, unchanged — confirmed identical against develop baseline (none newly introduced)

Not in this PR

  • mountainash-data's own DEBT-13 (IbisBackend.create_table, separate repo, separate write path) — explicitly out of scope, not shipped as of 2026-08-19.
  • Filing the upstream ibis issue — drafted only (docs/upstream-issue-drafts/IB-DT-19-sqlite-nat-binding.md), per explicit instruction not to submit it.

Backlog

mountainash-central h.backlog/active/ibis-sqlite-fixture-factory-nat-crash.md updated (commit 425f6e7) to record the scope expansion and resolved fix shape.

🤖 Generated with assistance from an AI coding agent.

…h (item 112, IB-DT-19)

ibis's SQLite backend stages every in-memory table via a pandas roundtrip
(_register_in_memory_table -> op.data.to_frame()); a null date/timestamp
value becomes pandas NaT, and stdlib sqlite3 has no adapter for NaTType,
raising ProgrammingError before mountainash's own visitor/compile machinery
ever runs. Verified empirically (ibis 12.0.0) that explicit PyArrow schemas,
ibis.memtable(schema=...), and object-dtype pandas columns all still crash --
ibis re-normalises internally before the pandas roundtrip regardless of
input shape.

Fix: relations/backends/relation_systems/ibis/_sqlite_compat.py registers a
single idempotent sqlite3 adapter (NaT -> NULL), invoked lazily wherever
Mountainash builds an ibis.memtable()/create_table() from data that could
contain a null temporal value:

- Production: relation_visitor.py cross-type join coercion,
  relsys_ib_ext_ma_util.py resource ingestion (2 call sites). Scope expanded
  beyond the original test-infra-only backlog item after confirming these
  crash independent of mountainash-data's own DEBT-13 (mountainash builds
  ibis.memtable() directly, not through that layer).
- Test fixtures: backend_registry.py/backend_helpers.py/conftest.py's 4
  duplicated ibis-sqlite table-creation call sites now delegate to one
  shared helper instead of re-registering the workaround.

Rejected: a pandas.DataFrame.to_sql() bypass avoids the crash but silently
degrades Boolean columns to Int64 -- an observable regression for every
existing non-null fixture. Rejected eager top-level adapter registration at
package-import time -- ibis-framework has no unconditional pandas
dependency (only specific backend extras do), so it would regress every
pandas-free ibis backend user.

Tests:
- tests/fixtures/test_backend_registry.py, test_backend_helpers.py (new):
  null date + null datetime regression fixture, parametrized over
  ALL_BACKENDS, for REGISTRY.build and BackendDataFrameFactory.create/
  create_pair.
- test_backend_registry.py: xfail(strict=True) upstream-fix monitor --
  reproduces the raw crash in a subprocess isolated from mountainash's own
  process-global adapter patch, so it flips to XPASS(strict) and fails CI
  the moment ibis fixes IB-DT-19 upstream.
- backend_helpers.py::get_count: fixed a pre-existing narwhals-lazy gap
  (missing .collect()) blocking the new ALL_BACKENDS parametrization.

docs/upstream-issue-drafts/IB-DT-19-sqlite-nat-binding.md: drafted, not
filed -- ready-to-file ibis issue text for when someone decides to submit
it.

Backlog: mountainash-central h.backlog/active/ibis-sqlite-fixture-factory-nat-crash.md
updated (commit 425f6e7) to record the scope expansion and resolved fix
shape.
@discreteds
discreteds merged commit a369002 into develop Aug 19, 2026
4 checks passed
@discreteds
discreteds deleted the discreteds/item112 branch August 19, 2026 03:48
discreteds added a commit that referenced this pull request Aug 19, 2026
…303) (#304)

PR #302 (adding this entry) and PR #303 (shipping the fix) merged in the
opposite order (303 landed first), leaving our_workaround: none stale on
develop even though ensure_sqlite_nat_adapter() already shipped in the same
history. Update our_workaround to fallback_impl and document the shim,
its call sites, the rejected alternatives, and the xfail(strict=True)
upstream-fix monitor that will flag when ibis fixes this natively.

Verified: scripts/validate_upstream_registry.py passes (127 entries, 0
errors); tests/core/test_upstream_registry_join.py and
tests/scripts/test_validate_upstream_registry.py pass (25 passed).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant