fix(relations): order Ibis join_asof result deterministically (IB-REL-11) - #299
Merged
discreteds merged 1 commit intoAug 18, 2026
Merged
Conversation
…-11) Root cause (PR #297 CI investigation): DuckDB gives no row-order guarantee for ASOF JOIN output absent an explicit ORDER BY. Ibis's join_asof backend (relsys_ib_ext_ma_util.py) never applied one, so test_asof_backward_strategy[ibis-duckdb] flipped strict-xfail/XPASS ~50/50 across independent runs (confirmed via 8 isolated process reruns pre/post fix). Values were always correct; only row order backing the test's positional assertions flapped. Same incident class as backlog item 101 (PR #293, 2026-08-16, flagged but not fixed then). Fix: order the join_asof result by the join key(s) before returning, matching polars/narwhals' left-order preservation for asof joins. Deterministic across 8/8 verification runs post-fix. IB-REL-11 split: ibis-duckdb portion closed resolved_in_mountainash; distinct, still-genuine ibis-sqlite raise (no ASOF JOIN translation) carved out to new IB-REL-13. Updated divergences.py, upstream-issues.yaml, known-divergences.md, and the test's xfail marker in lockstep; regenerated expression-coverage.{md,json} and _spine_expectation_census.md. AGENTS.md: removed the stale 'local Ibis fork' dependency note — CI/dev both resolve ibis-framework from PyPI, no path dependency exists. Backlog: filed mountainash-central item 108 for a distinct, unrelated latent bug found in the same function (join_asof silently drops the strategy= parameter on Ibis — forward/nearest behave as backward) and closed item 101's flagged sub-issue.
discreteds
deleted the
bugfix/ibis-asof-join-nondeterministic-row-order
branch
August 18, 2026 00:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-caused the PR #297 CI failure:
[XPASS(strict)] IB-REL-11ontest_asof_backward_strategy[ibis-duckdb], unrelated to that PR's own diff.Root cause: DuckDB gives no row-order guarantee for
ASOF JOINoutput absent an explicitORDER BY. mountainash's Ibisjoin_asofbackend never applied one, so the test's positional assertions (result[0],result[1], ...) flipped strict-xfail/XPASS ~50/50 across independent runs — confirmed via 8 isolated process reruns both before and after the fix. Join values were always correct; only row order flapped. Same incident class as backlog item 101 (PR #293, 2026-08-16), which flagged this exact bug but did not act on it at the time.Changes
relsys_ib_ext_ma_util.py::join_asof— orders the Ibisasof_joinresult by the join key(s) before returning, matching polars/narwhals' left-order preservation for asof joins. Deterministic across 8/8 verification runs post-fix (was ~50/50 pre-fix).IB-REL-11split — theibis-duckdbportion (this bug) closedresolved_in_mountainash; the distinct, still-genuineibis-sqliteraise (no ASOF JOIN translation in Ibis) carved out to newIB-REL-13. Updated in lockstep:divergences.py,registry/upstream-issues.yaml,docs/known-divergences.md, the test'sxfail_divergencemarker.docs/reference/expression-coverage.{md,json}andtests/_spine_expectation_census.md(both generated files) to reflect the split.AGENTS.md— removed a stale 'local Ibis fork' dependency note; CI and dev both resolveibis-frameworkfrom PyPI (>=9.0.0), no path dependency exists.Backlog
join_asof'sstrategyparameter is silently dropped on the Ibis backend (forward/nearestbehave asbackward, untested anywhere). Not fixed here — needs its own per-strategy emulation design.Verification
test_asof_backward_strategy[ibis-duckdb]: 8/8 deterministic pass, isolated process reruns (was ~5/8 flaky pre-fix).test_asof_backward_strategy[ibis-sqlite]: still raisesUnsupportedOperationErroras expected (3/3 reruns), unaffected.tests/relations/cross_backend/+ registry-integrity suite (test_upstream_registry_join.py,test_divergence_facts.py,test_capability_census.py): 1211 passed, 1 skipped, 26 xfailed, 0 failed.hatch run ruff:checkon touched files: clean.hatch run mypy:checkon the touched Ibis backend file: 5 pre-existing errors, none on touched lines (confirmed unchanged baseline).