Skip to content

fix(ibis,sqlite): DEBT-13 - null date/timestamp binding crash (IB-DT-19) - #104

Merged
discreteds merged 1 commit into
developfrom
bugfix/debt-13-sqlite-nat-binding
Aug 19, 2026
Merged

fix(ibis,sqlite): DEBT-13 - null date/timestamp binding crash (IB-DT-19)#104
discreteds merged 1 commit into
developfrom
bugfix/debt-13-sqlite-nat-binding

Conversation

@discreteds

Copy link
Copy Markdown
Member

Summary

Fixes DEBT-13 — sqlite write ops (create_table/insert/upsert) crash on null date/timestamp values. Same root cause as mountainash's item 112 (upstream ibis bug IB-DT-19, PR mountainash#303) — this PR ports the same fix shape into mountainash_data's own write path.

Root cause: ibis.backends.sqlite.Backend._register_in_memory_table always stages an in-memory table via a pandas roundtrip (op.data.to_frame()) before binding rows through stdlib sqlite3. A null date/timestamp value becomes pandas NaT, and sqlite3 has no adapter for NaTType — crashes create_table, insert, and upsert (both UPDATE and NOTHING renderers, via compiled_source's shared memtable staging) for any frame containing a null temporal value, before mountainash_data's own visitor/render machinery ever runs.

Fix: backends/ibis/_sqlite_compat.py::ensure_sqlite_nat_adapter() — a single process-global, idempotent sqlite3.register_adapter(NaTType, ...) call binding NaT as NULL, matching how every other backend already treats a missing temporal value. Called unconditionally (cheap no-op after first call, no dialect check needed) at the top of:

  • IbisBackend.create_table (backend.py)
  • IbisBackend.insert (backend.py)
  • compiled_source() (_render.py) — the shared memtable-staging step behind every upsert renderer (UPDATE, NOTHING, and the MySQL preflight path)

One mechanism covers the entire portable write surface. No per-frame-type (polars/pandas/pyarrow) dispatch logic, no monkeypatching ibis internals — this is the same technique validated and shipped in mountainash#303.

Not a hard dependency on mountainash: kept as a self-contained module here (mountainash_data currently treats mountainash as optional, see operations.py::_coerce_dtype's try/except bridge) rather than importing the sibling package's _sqlite_compat module.

Tests

tests/test_unit/backends/ibis/test_sqlite_nat_binding.py (new):

  • Null date column via create_table
  • Null datetime column via create_table, insert, upsert (both conflict_action="UPDATE" and "NOTHING")
  • test_raw_ibis_sqlite_null_temporal_upstream_bug_ib_dt_19: subprocess-isolated upstream-fix monitor — reproduces the raw ibis crash independent of this workaround, so it flips to a failure the moment ibis fixes IB-DT-19 upstream (signal to remove the workaround and close DEBT-13 for good)

Verification run

tests/test_unit/backends/ibis/: 336 passed
tests/test_unit/: 722 passed, 5 skipped (pre-existing, unrelated)
tests/test_integration/: 9 passed, 18 skipped (live-db, correctly gated — no postgres/mysql in this environment)
ruff check ./src: All checks passed
mypy src/mountainash_data tests: Success, no issues found in 132 source files

Not in this PR

  • Upstream ibis issue filing — tracked as needs_filing in mountainash/registry/upstream-issues.yaml (IB-DT-19), not filed this round per explicit instruction.
  • Wearables feature/bronzestore-portability task 6 (parity suite unpark) — separate repo, separate PR. This PR unblocks it but does not complete it.
  • DEBT-13 backlog doc is not archived — full resume path (wearables task 6 + close) still pending per the doc's own resume steps.

Backlog

mountainash-central 04.planning/mountainash-data/a.backlog/2026-08-18-sqlite-null-temporal-binding.md updated to reflect this fix.

🤖 Generated with assistance from an AI coding agent.

ibis's SQLite backend (_register_in_memory_table) always stages an
in-memory table via a pandas roundtrip (op.data.to_frame()) before
binding rows through stdlib sqlite3. A null date/timestamp value
becomes pandas NaT, and sqlite3 has no adapter for NaTType - crashes
create_table, insert, and upsert (both UPDATE and NOTHING renderers,
via compiled_source's shared memtable staging) for any frame
containing a null temporal value.

Fix: _sqlite_compat.ensure_sqlite_nat_adapter() registers a single
process-global sqlite3 adapter binding NaT as NULL, matching how
every other backend already treats a missing temporal value. Called
unconditionally (cheap, idempotent) at the top of create_table,
insert, and compiled_source (the shared staging step behind every
upsert renderer) - covers the entire portable write surface with one
mechanism, no per-frame-type dispatch, no ibis internals monkeypatch.

Same fix shape as the mountainash sibling package's item 112 (PR #303),
ported and adapted as a self-contained module here rather than a hard
mountainash dependency.

Tracked upstream as IB-DT-19 in mountainash/registry/upstream-issues.yaml
(status: needs_filing - no upstream ibis issue exists yet, not filed
this round per explicit instruction).

Regression coverage: tests/test_unit/backends/ibis/test_sqlite_nat_binding.py
- null date + null datetime via create_table, insert, upsert (both
  conflict_action styles)
- subprocess-isolated upstream-fix monitor: reproduces the raw ibis
  crash independent of this workaround, flips to a failure the moment
  ibis fixes IB-DT-19 upstream
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
11.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

@discreteds
discreteds merged commit 00268ab into develop Aug 19, 2026
5 of 6 checks passed
@discreteds
discreteds deleted the bugfix/debt-13-sqlite-nat-binding branch August 19, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant