Skip to content

fix(relations): order Ibis join_asof result deterministically (IB-REL-11) - #299

Merged
discreteds merged 1 commit into
developfrom
bugfix/ibis-asof-join-nondeterministic-row-order
Aug 18, 2026
Merged

fix(relations): order Ibis join_asof result deterministically (IB-REL-11)#299
discreteds merged 1 commit into
developfrom
bugfix/ibis-asof-join-nondeterministic-row-order

Conversation

@discreteds

Copy link
Copy Markdown
Member

Summary

Root-caused the PR #297 CI failure: [XPASS(strict)] IB-REL-11 on test_asof_backward_strategy[ibis-duckdb], unrelated to that PR's own diff.

Root cause: DuckDB gives no row-order guarantee for ASOF JOIN output absent an explicit ORDER BY. mountainash's Ibis join_asof backend never applied one, so the test's positional assertions (result[0], result[1], ...) flipped strict-xfail/XPASS ~50/50 across independent runs — confirmed via 8 isolated process reruns both before and after the fix. Join values were always correct; only row order flapped. Same incident class as backlog item 101 (PR #293, 2026-08-16), which flagged this exact bug but did not act on it at the time.

Changes

  • relsys_ib_ext_ma_util.py::join_asof — orders the Ibis asof_join result by the join key(s) before returning, matching polars/narwhals' left-order preservation for asof joins. Deterministic across 8/8 verification runs post-fix (was ~50/50 pre-fix).
  • IB-REL-11 split — the ibis-duckdb portion (this bug) closed resolved_in_mountainash; the distinct, still-genuine ibis-sqlite raise (no ASOF JOIN translation in Ibis) carved out to new IB-REL-13. Updated in lockstep: divergences.py, registry/upstream-issues.yaml, docs/known-divergences.md, the test's xfail_divergence marker.
  • Regenerated docs/reference/expression-coverage.{md,json} and tests/_spine_expectation_census.md (both generated files) to reflect the split.
  • AGENTS.md — removed a stale 'local Ibis fork' dependency note; CI and dev both resolve ibis-framework from PyPI (>=9.0.0), no path dependency exists.

Backlog

  • Filed mountainash-central item 108: a distinct, unrelated latent bug found in the same function while fixing this — join_asof's strategy parameter is silently dropped on the Ibis backend (forward/nearest behave as backward, untested anywhere). Not fixed here — needs its own per-strategy emulation design.
  • Closed item 101's flagged sub-issue (item 4 in its open-questions list); its own strategic pinning/drift-detection questions (1–3) remain open.

Verification

  • test_asof_backward_strategy[ibis-duckdb]: 8/8 deterministic pass, isolated process reruns (was ~5/8 flaky pre-fix).
  • test_asof_backward_strategy[ibis-sqlite]: still raises UnsupportedOperationError as expected (3/3 reruns), unaffected.
  • tests/relations/cross_backend/ + registry-integrity suite (test_upstream_registry_join.py, test_divergence_facts.py, test_capability_census.py): 1211 passed, 1 skipped, 26 xfailed, 0 failed.
  • hatch run ruff:check on touched files: clean.
  • hatch run mypy:check on the touched Ibis backend file: 5 pre-existing errors, none on touched lines (confirmed unchanged baseline).

…-11)

Root cause (PR #297 CI investigation): DuckDB gives no row-order
guarantee for ASOF JOIN output absent an explicit ORDER BY. Ibis's
join_asof backend (relsys_ib_ext_ma_util.py) never applied one, so
test_asof_backward_strategy[ibis-duckdb] flipped strict-xfail/XPASS
~50/50 across independent runs (confirmed via 8 isolated process
reruns pre/post fix). Values were always correct; only row order
backing the test's positional assertions flapped. Same incident class
as backlog item 101 (PR #293, 2026-08-16, flagged but not fixed then).

Fix: order the join_asof result by the join key(s) before returning,
matching polars/narwhals' left-order preservation for asof joins.
Deterministic across 8/8 verification runs post-fix.

IB-REL-11 split: ibis-duckdb portion closed resolved_in_mountainash;
distinct, still-genuine ibis-sqlite raise (no ASOF JOIN translation)
carved out to new IB-REL-13. Updated divergences.py, upstream-issues.yaml,
known-divergences.md, and the test's xfail marker in lockstep; regenerated
expression-coverage.{md,json} and _spine_expectation_census.md.

AGENTS.md: removed the stale 'local Ibis fork' dependency note — CI/dev
both resolve ibis-framework from PyPI, no path dependency exists.

Backlog: filed mountainash-central item 108 for a distinct, unrelated
latent bug found in the same function (join_asof silently drops the
strategy= parameter on Ibis — forward/nearest behave as backward) and
closed item 101's flagged sub-issue.
@discreteds
discreteds merged commit 33c95d2 into develop Aug 18, 2026
4 checks passed
@discreteds
discreteds deleted the bugfix/ibis-asof-join-nondeterministic-row-order branch August 18, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant