fix(ibis,sqlite): DEBT-13 - null date/timestamp binding crash (IB-DT-19) - #104
Merged
Merged
Conversation
ibis's SQLite backend (_register_in_memory_table) always stages an in-memory table via a pandas roundtrip (op.data.to_frame()) before binding rows through stdlib sqlite3. A null date/timestamp value becomes pandas NaT, and sqlite3 has no adapter for NaTType - crashes create_table, insert, and upsert (both UPDATE and NOTHING renderers, via compiled_source's shared memtable staging) for any frame containing a null temporal value. Fix: _sqlite_compat.ensure_sqlite_nat_adapter() registers a single process-global sqlite3 adapter binding NaT as NULL, matching how every other backend already treats a missing temporal value. Called unconditionally (cheap, idempotent) at the top of create_table, insert, and compiled_source (the shared staging step behind every upsert renderer) - covers the entire portable write surface with one mechanism, no per-frame-type dispatch, no ibis internals monkeypatch. Same fix shape as the mountainash sibling package's item 112 (PR #303), ported and adapted as a self-contained module here rather than a hard mountainash dependency. Tracked upstream as IB-DT-19 in mountainash/registry/upstream-issues.yaml (status: needs_filing - no upstream ibis issue exists yet, not filed this round per explicit instruction). Regression coverage: tests/test_unit/backends/ibis/test_sqlite_nat_binding.py - null date + null datetime via create_table, insert, upsert (both conflict_action styles) - subprocess-isolated upstream-fix monitor: reproduces the raw ibis crash independent of this workaround, flips to a failure the moment ibis fixes IB-DT-19 upstream
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Fixes DEBT-13 — sqlite write ops (
create_table/insert/upsert) crash on nulldate/timestampvalues. Same root cause as mountainash's item 112 (upstream ibis bugIB-DT-19, PR mountainash#303) — this PR ports the same fix shape intomountainash_data's own write path.Root cause:
ibis.backends.sqlite.Backend._register_in_memory_tablealways stages an in-memory table via a pandas roundtrip (op.data.to_frame()) before binding rows through stdlibsqlite3. A nulldate/timestampvalue becomes pandasNaT, andsqlite3has no adapter forNaTType— crashescreate_table,insert, andupsert(bothUPDATEandNOTHINGrenderers, viacompiled_source's shared memtable staging) for any frame containing a null temporal value, before mountainash_data's own visitor/render machinery ever runs.Fix:
backends/ibis/_sqlite_compat.py::ensure_sqlite_nat_adapter()— a single process-global, idempotentsqlite3.register_adapter(NaTType, ...)call bindingNaTasNULL, matching how every other backend already treats a missing temporal value. Called unconditionally (cheap no-op after first call, no dialect check needed) at the top of:IbisBackend.create_table(backend.py)IbisBackend.insert(backend.py)compiled_source()(_render.py) — the shared memtable-staging step behind everyupsertrenderer (UPDATE,NOTHING, and the MySQL preflight path)One mechanism covers the entire portable write surface. No per-frame-type (polars/pandas/pyarrow) dispatch logic, no monkeypatching ibis internals — this is the same technique validated and shipped in mountainash#303.
Not a hard dependency on
mountainash: kept as a self-contained module here (mountainash_data currently treatsmountainashas optional, seeoperations.py::_coerce_dtype's try/except bridge) rather than importing the sibling package's_sqlite_compatmodule.Tests
tests/test_unit/backends/ibis/test_sqlite_nat_binding.py(new):create_tablecreate_table,insert,upsert(bothconflict_action="UPDATE"and"NOTHING")test_raw_ibis_sqlite_null_temporal_upstream_bug_ib_dt_19: subprocess-isolated upstream-fix monitor — reproduces the raw ibis crash independent of this workaround, so it flips to a failure the moment ibis fixesIB-DT-19upstream (signal to remove the workaround and close DEBT-13 for good)Verification run
Not in this PR
needs_filinginmountainash/registry/upstream-issues.yaml(IB-DT-19), not filed this round per explicit instruction.feature/bronzestore-portabilitytask 6 (parity suite unpark) — separate repo, separate PR. This PR unblocks it but does not complete it.Backlog
mountainash-central04.planning/mountainash-data/a.backlog/2026-08-18-sqlite-null-temporal-binding.mdupdated to reflect this fix.🤖 Generated with assistance from an AI coding agent.