fix(index): keep overlays masked when updating a non-indexed column#7926
Open
wjones127 wants to merge 2 commits into
Open
fix(index): keep overlays masked when updating a non-indexed column#7926wjones127 wants to merge 2 commits into
wjones127 wants to merge 2 commits into
Conversation
A rewrite-rows update (under stable row ids) that sets only a non-indexed column moves the matched rows to a new fragment. Because the update did not touch the scalar index's field, `register_pure_rewrite_rows_update_frags_in_indices` extends that index's coverage onto the new fragment and reuses its existing entries, on the assumption that the moved rows' indexed value is unchanged. That assumption breaks when a moved row carried a data overlay on the indexed field: the update materializes the overlay's current value into the new fragment, but the reused index entry still holds the stale pre-overlay value, and marking the new fragment covered removes the flat-path re-evaluation that overlay masking relied on. The overlaid row then vanishes from index-path queries (a range scan returns 99 rows where 100 are expected). Skip extending coverage for an index whose fields are touched by an overlay committed after the index was built on any of the moved rows' original fragments, so those new fragments fall to the flat path against current values. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: QUIET Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis change prevents pure row rewrites from reusing index coverage when source fragment overlays may expose stale values. A new integration test verifies scalar index masking after updating a non-indexed column on an overlaid row. ChangesOverlay index masking
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant UpdateBuilder
participant build_manifest
participant collect_overlay_stale_frags
participant ScalarIndex
UpdateBuilder->>build_manifest: RewriteRows update
build_manifest->>collect_overlay_stale_frags: Check original overlaid fragments
collect_overlay_stale_frags-->>build_manifest: Stale fragment set
build_manifest->>ScalarIndex: Extend bitmap only when safe
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
wjones127
marked this pull request as ready for review
July 23, 2026 16:40
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance/src/dataset/transaction.rs`:
- Around line 2695-2730: Refactor the fragment_bitmap handling in the
index-update loop to avoid the is_none-then-unwrap pattern. Bind
index.fragment_bitmap once with let ... else or if let, continue when absent,
and reuse that binding for both the coverage check and subsequent fragment
insertions, removing the as_ref().unwrap() and as_mut().unwrap() calls while
preserving the existing skip behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: dee9e8bb-0c14-4918-8483-eb13ee195a40
📒 Files selected for processing (2)
rust/lance/src/dataset/tests/dataset_overlay_index_masking.rsrust/lance/src/dataset/transaction.rs
Addresses CodeRabbit review: replace the check-then-unwrap pattern (`is_none()` followed by `as_ref().unwrap()`/`as_mut().unwrap()`) with a `let ... else` binding and an `if let`, per the crate's coding guidelines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
RewriteRowsupdate (under stable row ids) that sets only a non-indexed column moves the matched rows to a new fragment. Because the update didn't touch the scalar index's field,register_pure_rewrite_rows_update_frags_in_indicesextends that index's coverage onto the new fragment and reuses its existing entries — assuming the moved rows' indexed value is unchanged.That assumption breaks when a moved row carried a data overlay on the indexed field. The update reads the row through overlay resolution, so it materializes the overlay's current value into the new fragment, but:
The overlaid row then vanishes from index-path queries. Minimal repro: BTree on
age, overlayage 10 → 999on a live row, thenUPDATE SET id=100 WHERE id=1(touching only the non-indexedid). Afterwardsage = 10wrongly returns the row andage = 999returns nothing — the row is lost from the index path (a range scan returns 99 where 100 are expected).This is a distinct root cause from the
OptimizeIndicesun-masking (#7924): no version-gate flip and no merge involved — the trigger is the coverage-extension reusing stale entries.Fix
In
register_pure_rewrite_rows_update_frags_in_indices, before extending an index's coverage to the moved-rows fragments, skip any index whose fields are touched by an overlay committed after the index was built on any of the moved rows' original fragments (reusingcollect_overlay_stale_frags). Those new fragments are then left unindexed and fall to the flat path against current values.Tests
Adds
test_update_nonindexed_column_preserves_overlay_masking(fails before, passes after). The existing update/transaction suites — includingtest_update_affects_index_fragment_bitmap, which covers the non-overlay coverage-extension optimization — still pass, so the optimization is unchanged when no overlay is present.🤖 Generated with Claude Code