fix(relations,fixtures): ibis-sqlite null date/timestamp binding crash (item 112, IB-DT-19) - #303
Merged
Merged
Conversation
…h (item 112, IB-DT-19) ibis's SQLite backend stages every in-memory table via a pandas roundtrip (_register_in_memory_table -> op.data.to_frame()); a null date/timestamp value becomes pandas NaT, and stdlib sqlite3 has no adapter for NaTType, raising ProgrammingError before mountainash's own visitor/compile machinery ever runs. Verified empirically (ibis 12.0.0) that explicit PyArrow schemas, ibis.memtable(schema=...), and object-dtype pandas columns all still crash -- ibis re-normalises internally before the pandas roundtrip regardless of input shape. Fix: relations/backends/relation_systems/ibis/_sqlite_compat.py registers a single idempotent sqlite3 adapter (NaT -> NULL), invoked lazily wherever Mountainash builds an ibis.memtable()/create_table() from data that could contain a null temporal value: - Production: relation_visitor.py cross-type join coercion, relsys_ib_ext_ma_util.py resource ingestion (2 call sites). Scope expanded beyond the original test-infra-only backlog item after confirming these crash independent of mountainash-data's own DEBT-13 (mountainash builds ibis.memtable() directly, not through that layer). - Test fixtures: backend_registry.py/backend_helpers.py/conftest.py's 4 duplicated ibis-sqlite table-creation call sites now delegate to one shared helper instead of re-registering the workaround. Rejected: a pandas.DataFrame.to_sql() bypass avoids the crash but silently degrades Boolean columns to Int64 -- an observable regression for every existing non-null fixture. Rejected eager top-level adapter registration at package-import time -- ibis-framework has no unconditional pandas dependency (only specific backend extras do), so it would regress every pandas-free ibis backend user. Tests: - tests/fixtures/test_backend_registry.py, test_backend_helpers.py (new): null date + null datetime regression fixture, parametrized over ALL_BACKENDS, for REGISTRY.build and BackendDataFrameFactory.create/ create_pair. - test_backend_registry.py: xfail(strict=True) upstream-fix monitor -- reproduces the raw crash in a subprocess isolated from mountainash's own process-global adapter patch, so it flips to XPASS(strict) and fails CI the moment ibis fixes IB-DT-19 upstream. - backend_helpers.py::get_count: fixed a pre-existing narwhals-lazy gap (missing .collect()) blocking the new ALL_BACKENDS parametrization. docs/upstream-issue-drafts/IB-DT-19-sqlite-nat-binding.md: drafted, not filed -- ready-to-file ibis issue text for when someone decides to submit it. Backlog: mountainash-central h.backlog/active/ibis-sqlite-fixture-factory-nat-crash.md updated (commit 425f6e7) to record the scope expansion and resolved fix shape.
This was referenced Aug 19, 2026
discreteds
added a commit
that referenced
this pull request
Aug 19, 2026
…303) (#304) PR #302 (adding this entry) and PR #303 (shipping the fix) merged in the opposite order (303 landed first), leaving our_workaround: none stale on develop even though ensure_sqlite_nat_adapter() already shipped in the same history. Update our_workaround to fallback_impl and document the shim, its call sites, the rejected alternatives, and the xfail(strict=True) upstream-fix monitor that will flag when ibis fixes this natively. Verified: scripts/validate_upstream_registry.py passes (127 entries, 0 errors); tests/core/test_upstream_registry_join.py and tests/scripts/test_validate_upstream_registry.py pass (25 passed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes item 112 — cross-backend test fixture factory crashes on null date/timestamp for
ibis-sqlite— with scope expanded (see backlog doc,mountainash-centralcommit425f6e7) to also cover mountainash's own production code, since the same crash is reachable independent of the original test-infra-only framing.Root cause:
ibis.backends.sqlite.Backend._register_in_memory_tablealways stages an in-memory table via a pandas roundtrip (op.data.to_frame()) before binding rows through stdlibsqlite3. A nulldate/timestampvalue becomes pandasNaT, andsqlite3has no adapter forNaTType— raisessqlite3.ProgrammingErrorbefore Mountainash's own visitor/compile machinery ever runs. Verified empirically (ibis 12.0.0) that explicit PyArrow schemas,ibis.memtable(schema=...), and object-dtype pandas columns all still crash — ibis re-normalises internally before the roundtrip regardless of input shape.Fix:
relations/backends/relation_systems/ibis/_sqlite_compat.py::ensure_sqlite_nat_adapter()— one idempotent, process-globalsqlite3.register_adapter(NaTType, ...)call, invoked lazily wherever Mountainash builds anibis.memtable()/create_table()from data that could contain a null temporal value:relation_visitor.py's cross-type join coercion,relsys_ib_ext_ma_util.py's resource-ingestion fallback (2 call sites). These crash independent of whethermountainash-dataships its own DEBT-13 fix, since Mountainash buildsibis.memtable()directly, not through that layer.backend_registry.py/backend_helpers.py/conftest.pynow delegate to one shared helper instead of re-registering the workaround.Rejected alternatives:
pandas.DataFrame.to_sql()bypass avoids the crash but silently degradesBooleancolumns toInt64— an observable regression for every existing non-null fixture.ibis-frameworkhas no unconditionalpandasdependency (only specific backend extras do), so this would regress every pandas-free ibis backend user.Tests
tests/fixtures/test_backend_registry.py,test_backend_helpers.py(new file): null date + null datetime regression fixture, parametrized overALL_BACKENDS, forREGISTRY.buildandBackendDataFrameFactory.create/create_pair.test_backend_registry.py::test_raw_ibis_sqlite_null_temporal_upstream_bug_ib_dt_19:xfail(strict=True)upstream-fix monitor — reproduces the raw crash in a subprocess isolated from Mountainash's own process-global adapter patch, so it flips toXPASS(strict)and fails CI the moment ibis fixes IB-DT-19 upstream.backend_helpers.py::get_count: fixed a pre-existingnarwhals-lazygap (missing.collect()) that blocked the newALL_BACKENDSparametrization — unrelated to the main fix, found while writing the regression test.ma.relation(ibis_sqlite_table).join(polars_df_with_null_datetime)) before/after the fix.Verification run
Not in this PR
mountainash-data's own DEBT-13 (IbisBackend.create_table, separate repo, separate write path) — explicitly out of scope, not shipped as of 2026-08-19.docs/upstream-issue-drafts/IB-DT-19-sqlite-nat-binding.md), per explicit instruction not to submit it.Backlog
mountainash-centralh.backlog/active/ibis-sqlite-fixture-factory-nat-crash.mdupdated (commit425f6e7) to record the scope expansion and resolved fix shape.🤖 Generated with assistance from an AI coding agent.