fix(relations,fixtures): ibis-sqlite null date/timestamp binding crash (item 112, IB-DT-19)

docs/upstream-issue-drafts/IB-DT-19-sqlite-nat-binding.md

@@ -0,0 +1,111 @@
+<!--
+DRAFT ONLY — NOT FILED.
+
+Tracked internally as IB-DT-19 in registry/upstream-issues.yaml
+(status: needs_filing). This file is a ready-to-file draft for
+https://github.com/ibis-project/issues/new — copy the section below the
+divider into a new issue when someone decides to file it. Do not open the
+upstream issue as part of landing this PR.
+-->
+
+# Draft: `ibis.sqlite` `create_table`/in-memory registration crashes on a null date/timestamp value (`NaTType` binding error)
+
+---
+
+## Bug
+
+`Backend.create_table()` (and any expression execution that registers an
+in-memory table, e.g. `ibis.memtable(...)` compiled against a SQLite
+connection) crashes with `sqlite3.ProgrammingError` when the input contains a
+null `date`/`timestamp` value, for **every** input shape I tried — a plain
+dict, an explicit-schema PyArrow table, `ibis.memtable(..., schema=...)`, and
+an object-dtype pandas column all hit the identical crash.
+
+### Reproduction
+
+```python
+import ibis
+import datetime as dt
+
+data = {
+    "id": [1, 2],
+    "ts": [None, dt.datetime(2024, 1, 1, 12, 0, 0)],
+}
+
+conn = ibis.sqlite.connect(":memory:")
+t = conn.create_table("t", data, overwrite=True)
+```
+
+```
+Traceback (most recent call last):
+  ...
+  File ".../ibis/backends/sqlite/__init__.py", line 382, in _register_in_memory_table
+    cur.executemany(insert_stmt, data)
+sqlite3.ProgrammingError: Error binding parameter 2: type 'NaTType' is not supported
+```
+
+A null `date32` value happens to work — but only incidentally, via PyArrow's
+`Table.to_pandas(date_as_object=True)` default converting it to Python `None`
+before the pandas roundtrip. A null `timestamp` has no such incidental
+escape hatch: PyArrow's `to_pandas()` always maps a null timestamp to
+`datetime64[ns]`'s `NaT`, and `sqlite3` has no adapter for `NaTType`.
+
+### Root cause
+
+`ibis.backends.sqlite.Backend._register_in_memory_table` always stages the
+in-memory table via a pandas roundtrip before binding rows:
+
+```python
+def _register_in_memory_table(self, op: ops.InMemoryTable) -> None:
+    ...
+    df = op.data.to_frame()
+    data = df.itertuples(index=False)
+    ...
+    with self.begin() as cur:
+        cur.execute(create_stmt)
+        cur.executemany(insert_stmt, data)
+```
+
+`op.data.to_frame()` always produces a pandas `datetime64` column for a
+timestamp field, and pandas has no way to represent a missing value in a
+`datetime64` column other than `NaT`. Because the roundtrip happens
+internally, no input shape avoids it — I confirmed all of the following still
+crash the same way:
+
+- Explicit-schema `pyarrow.Table` (`pa.schema([("ts", pa.timestamp("us"))])`)
+- `ibis.memtable(data, schema=ibis.schema({"ts": "timestamp"}))`
+- A pandas `DataFrame` with the timestamp column forced to `dtype=object`
+  before being passed in (ibis re-normalises it to `datetime64` internally
+  regardless)
+
+### Expected behavior
+
+A null timestamp/date value should round-trip through `create_table`/table
+registration the same way it does for every other ibis-supported backend —
+binding as SQL `NULL`, not raising `ProgrammingError`.
+
+### Suggested fix direction
+
+Register a `sqlite3.register_adapter` for the pandas `NaTType` (or,
+upstream, whatever internal NaT-like sentinel `_register_in_memory_table`
+produces) that converts it to `None` before binding — this is the smallest
+fix, doesn't change any other type's binding behavior, and matches how every
+other backend already treats a missing temporal value.
+
+### Environment
+
+- ibis: 12.0.0
+- python: 3.12.12
+- pandas: 3.0.5
+- sqlite3 (stdlib): module 2.6.0, libsqlite3 3.50.4
+- platform: macOS-26.4-arm64-arm-64bit
+
+### Downstream impact (context, not required for the upstream report)
+
+Found via [mountainash](https://github.com/mountainash-io/mountainash)'s
+cross-backend test suite and confirmed to also affect mountainash's own
+production code (any relation that cross-type-joins a null-timestamp frame
+against, or ingests one into, an ibis-sqlite-backed relation). Worked around
+downstream with the `sqlite3.register_adapter` technique described above
+(`relations/backends/relation_systems/ibis/_sqlite_compat.py`), tracked
+internally as `IB-DT-19` pending this upstream report.

src/mountainash/relations/backends/relation_systems/ibis/_sqlite_compat.py

@@ -0,0 +1,59 @@
+"""Compatibility shim for Ibis's SQLite backend.
+
+Ibis's SQLite backend (``ibis.backends.sqlite.Backend._register_in_memory_table``)
+always stages an in-memory table via a pandas roundtrip (``op.data.to_frame()``)
+before binding rows through the stdlib ``sqlite3`` module -- regardless of
+whether the table was built from a dict, a PyArrow table, or an
+``ibis.memtable(..., schema=...)`` call with an explicit temporal schema. A null
+``date``/``timestamp`` value becomes pandas ``NaT`` during that roundtrip, and
+``sqlite3`` has no adapter for ``NaTType`` -- ``cur.executemany()`` raises
+``sqlite3.ProgrammingError("Error binding parameter N: type 'NaTType' is not
+supported")`` before Mountainash's own visitor/compile machinery ever runs.
+
+Verified empirically against ibis 12.0.0 (2026-08-18): explicit PyArrow
+schemas, ``ibis.memtable(schema=...)``, and object-dtype pandas columns all
+still crash, because Ibis re-normalises the input internally before reaching
+the pandas roundtrip. Tracked upstream as ``IB-DT-19`` in
+``registry/upstream-issues.yaml`` (status: ``needs_filing`` as of 2026-08-18 --
+no upstream ibis issue exists yet).
+
+This module registers a single process-global ``sqlite3`` adapter that binds
+``NaT`` as ``NULL``, matching how every other backend already treats a missing
+temporal value. It does not touch Ibis's own type inference, so schema
+fidelity (including ``Boolean`` columns, which a naive ``pandas.to_sql``
+bypass would silently degrade to ``Int64``) is unaffected.
+"""
+from __future__ import annotations
+
+_NAT_ADAPTER_INSTALLED = False
+
+
+def ensure_sqlite_nat_adapter() -> None:
+    """Register a ``sqlite3`` adapter so pandas ``NaT`` binds as ``NULL``.
+
+    Idempotent and process-global: ``sqlite3.register_adapter`` just
+    overwrites one dict entry, so repeated calls are a cheap no-op. Must be
+    called before any Ibis-SQLite table is created/executed with data that
+    might contain a null temporal value -- call sites include every place
+    Mountainash builds an ``ibis.memtable()``/``create_table()`` from
+    caller- or file-supplied data that could reach a SQLite-backed
+    connection (cross-type join coercion, resource ingestion, and the
+    cross-backend test-fixture factory).
+
+    Silently returns if pandas is not importable: Ibis's own SQLite roundtrip
+    requires pandas too (``ibis-framework[sqlite]`` depends on it), so if
+    pandas is missing here, the crash this guards against cannot occur either.
+    """
+    global _NAT_ADAPTER_INSTALLED
+    if _NAT_ADAPTER_INSTALLED:
+        return
+
+    import sqlite3
+
+    try:
+        import pandas as pd
+    except ImportError:
+        return
+
+    sqlite3.register_adapter(type(pd.NaT), lambda _: None)
+    _NAT_ADAPTER_INSTALLED = True

src/mountainash/relations/backends/relation_systems/ibis/extensions_mountainash/relsys_ib_ext_ma_util.py

@@ -10,6 +10,9 @@
 from mountainash.relations.core.relation_protocols.relation_systems.extensions_mountainash import (
     MountainashExtensionRelationSystemProtocol,
 )
+from mountainash.relations.backends.relation_systems.ibis._sqlite_compat import (
+    ensure_sqlite_nat_adapter,
+)
 
 
 class MountainashIbisExtensionRelationSystem(MountainashExtensionRelationSystemProtocol[ir.Table]):
@@ -130,6 +133,7 @@ def read_resource(self, resource: Any) -> ir.Table:
                 MountainashPolarsExtensionRelationSystem,
             )
             lf = MountainashPolarsExtensionRelationSystem()._read_inline(resource)
+            ensure_sqlite_nat_adapter()
             return ibis.memtable(lf.collect().to_arrow())
 
         fmt = self._detect_format_name(resource)
@@ -158,6 +162,7 @@ def read_resource(self, resource: Any) -> ir.Table:
 
         # Fallback: mountainash-files -> Arrow -> memtable (no pandas). The files
         # reader honours the full CSV dialect via CsvSpec (>=26.7.1).
+        ensure_sqlite_nat_adapter()
         return ibis.memtable(rf.parse_resource_to_arrow(resource))
 
     @staticmethod

src/mountainash/relations/core/unified_visitor/relation_visitor.py

@@ -657,6 +657,10 @@ def _coerce_to_match(target: Any, value: Any) -> Any:
             source_type = type(value).__name__
             try:
                 import ibis
+                from mountainash.relations.backends.relation_systems.ibis._sqlite_compat import (
+                    ensure_sqlite_nat_adapter,
+                )
+                ensure_sqlite_nat_adapter()
                 if is_narwhals_lazyframe(value):
                     value = value.to_native()
                 return ibis.memtable(value)

tests/conftest.py

@@ -33,6 +33,7 @@
 from fixtures.backend_registry import (
     REGISTRY as BACKEND_REGISTRY,
     ALL_BACKENDS,
+    create_ibis_sqlite_table,
 )
 TEMPORAL_BACKENDS = [
     "polars",
@@ -239,7 +240,7 @@ def ibis_polars_df(sample_data) -> Any:
 def ibis_sqlite_df(sample_data) -> Any:
     """Create Ibis Table with SQLite backend from sample data."""
     conn = ibis.sqlite.connect(":memory:")
-    return conn.create_table("sample", sample_data, overwrite=True)
+    return create_ibis_sqlite_table(conn, "sample", sample_data)
 
 
 @pytest.fixture

tests/fixtures/backend_helpers.py

@@ -12,6 +12,8 @@
 import narwhals as nw
 import ibis
 
+from .backend_registry import create_ibis_sqlite_table
+
 
 class BackendResultHelper:
     """
@@ -48,7 +50,7 @@ def get_count(df: Any, backend_name: str) -> int:
         """
         if backend_name.startswith("ibis-"):
             return df.count().execute()
-        elif backend_name == "polars-lazy":
+        elif backend_name in ("polars-lazy", "narwhals-lazy"):
             return df.collect().shape[0]
         elif backend_name in ["polars", "pandas", "narwhals", "narwhals-polars", "narwhals-pandas"]:
             return df.shape[0]
@@ -299,7 +301,7 @@ def create(data: Dict[str, List], backend_name: str) -> Any:
             return conn.create_table("test_table", pl_df, overwrite=True)
         elif backend_name == "ibis-sqlite":
             conn = ibis.sqlite.connect(":memory:")
-            return conn.create_table("test_table", data, overwrite=True)
+            return create_ibis_sqlite_table(conn, "test_table", data)
         else:
             raise ValueError(f"Unknown backend: {backend_name}")
 
@@ -340,8 +342,8 @@ def create_pair(
             return left, right
         elif backend_name == "ibis-sqlite":
             conn = ibis.sqlite.connect(":memory:")
-            left = conn.create_table(left_name, left_data, overwrite=True)
-            right = conn.create_table(right_name, right_data, overwrite=True)
+            left = create_ibis_sqlite_table(conn, left_name, left_data)
+            right = create_ibis_sqlite_table(conn, right_name, right_data)
             return left, right
         else:
             # Non-ibis backends: independent DataFrames, no shared connection needed.

tests/fixtures/backend_registry.py

@@ -68,9 +68,25 @@ def _build_ibis_polars(data: DataDict, table_name: str):
     return conn.create_table(table_name, pl.DataFrame(data), overwrite=True)
 
 
+def create_ibis_sqlite_table(conn, name: str, data: DataDict, *, overwrite: bool = True):
+    """Create an ibis-sqlite table, safe against null date/timestamp values.
+
+    Single source of truth for every ibis-sqlite table-creation call site in
+    the cross-backend test suite (this module, backend_helpers.py, and
+    conftest.py's ibis_sqlite_df fixture) -- delegates the NaT-binding
+    workaround to mountainash's own production fix (item 112 / IB-DT-19)
+    rather than re-registering a second copy of it.
+    """
+    from mountainash.relations.backends.relation_systems.ibis._sqlite_compat import (
+        ensure_sqlite_nat_adapter,
+    )
+    ensure_sqlite_nat_adapter()
+    return conn.create_table(name, data, overwrite=overwrite)
+
+
 def _build_ibis_sqlite(data: DataDict, table_name: str):
     conn = ibis.sqlite.connect(":memory:")
-    return conn.create_table(table_name, data, overwrite=True)
+    return create_ibis_sqlite_table(conn, table_name, data)
 
 
 def _build_ibis_duckdb(data: DataDict, table_name: str):

tests/fixtures/test_backend_helpers.py

@@ -0,0 +1,50 @@
+# tests/fixtures/test_backend_helpers.py
+"""Self-tests for BackendDataFrameFactory (backend_helpers.py)."""
+from __future__ import annotations
+import datetime as dt
+
+import pytest
+
+from .backend_helpers import BackendDataFrameFactory, BackendResultHelper
+from .backend_registry import ALL_BACKENDS
+
+
+# Same shape as test_backend_registry.py's NULL_TEMPORAL_DATA -- mixed
+# null/non-null date AND datetime columns. Regression for item 112 / IB-DT-19:
+# BackendDataFrameFactory.create/create_pair duplicate the same ibis-sqlite
+# construction pattern as the REGISTRY-driven factory and were independently
+# confirmed to crash the same way before the fix.
+NULL_TEMPORAL_DATA = {
+    "id": [1, 2, 3],
+    "when_date": [None, dt.date(2024, 1, 1), dt.date(2024, 1, 2)],
+    "when_ts": [
+        None,
+        dt.datetime(2024, 1, 1, 12, 0, 0),
+        dt.datetime(2024, 1, 2, 8, 30, 0),
+    ],
+}
+
+NULL_TEMPORAL_DATA_RIGHT = {
+    "id": [1, 2, 3],
+    "other_date": [dt.date(2024, 2, 1), None, dt.date(2024, 2, 2)],
+}
+
+
+@pytest.mark.parametrize("backend_name", ALL_BACKENDS)
+def test_create_survives_null_date_and_null_datetime(backend_name):
+    """BackendDataFrameFactory.create must not crash on a null date/datetime
+    mixed with non-null rows, for every backend (named explicitly in item 112
+    required work #1, alongside _build_ibis_sqlite)."""
+    df = BackendDataFrameFactory.create(NULL_TEMPORAL_DATA, backend_name)
+    assert BackendResultHelper.get_count(df, backend_name) == 3
+
+
+@pytest.mark.parametrize("backend_name", ALL_BACKENDS)
+def test_create_pair_survives_null_date_and_null_datetime(backend_name):
+    """BackendDataFrameFactory.create_pair must not crash when either side of
+    the pair has a null date/datetime column, for every backend."""
+    left, right = BackendDataFrameFactory.create_pair(
+        NULL_TEMPORAL_DATA, NULL_TEMPORAL_DATA_RIGHT, backend_name
+    )
+    assert BackendResultHelper.get_count(left, backend_name) == 3
+    assert BackendResultHelper.get_count(right, backend_name) == 3