diff --git a/docs/src/format/table/.pages b/docs/src/format/table/.pages
index 16c20058608..5b0cb0e95e6 100644
--- a/docs/src/format/table/.pages
+++ b/docs/src/format/table/.pages
@@ -6,4 +6,5 @@ nav:
- Layout: layout.md
- Branch & Tag: branch_tag.md
- Row ID & Lineage: row_id_lineage.md
+ - Data Overlay Files: data_overlay_file.md
- MemTable & WAL: mem_wal.md
diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md
new file mode 100644
index 00000000000..5540f860018
--- /dev/null
+++ b/docs/src/format/table/data_overlay_file.md
@@ -0,0 +1,390 @@
+# Data Overlay Files
+
+!!! note "Overlay files require feature flag 64 (data overlay files)"
+
+ A reader or writer that does not understand overlay files must refuse a
+ dataset that uses them. Silently ignoring an overlay would return stale base
+ values, which is a correctness bug rather than a degraded experience.
+
+Overlay files supply new values for a subset of `(row offset, field)` cells
+within a fragment **without rewriting the fragment's base data files**. They make
+updates cheap when only a small fraction of rows and/or columns change: instead
+of rewriting whole columns or moving rows to a new fragment, a writer appends a
+small file carrying just the changed cells.
+
+This is Lance's third mechanism for changing data in place, alongside
+[deletion files](index.md#deletion-files) (which remove rows) and
+[data evolution](index.md#data-evolution) (which adds or rewrites whole columns).
+An overlay changes individual cells.
+
+## Concepts
+
+### Coverage and resolution
+
+Each overlay declares which cells it provides through a **coverage** bitmap (or,
+for sparse overlays, one bitmap per field). The bitmaps index **physical row
+offsets** — positions in the base data files, counting deleted rows — so they are
+stable across deletions, exactly like deletion vectors.
+
+To resolve a cell `(offset, field)` on read, walk the fragment's overlays from
+**newest to oldest**. The first overlay that covers `(offset, field)` wins; its
+value is used. If no overlay covers the cell, the value falls through to the base
+data file (or is `NULL` if no base data file holds that field).
+
+Precedence among overlays is determined by:
+
+1. `committed_version` — higher wins (see [Versioning](#versioning-and-ordering)).
+2. Position in `DataFragment.overlays` as a tiebreaker — a later entry is newer.
+
+A covered offset whose value is `NULL` overrides the cell **to** `NULL`. This is
+distinct from an offset that is simply absent from the bitmap, which falls
+through to the base. Coverage, not value-nullness, decides whether an overlay
+applies.
+
+### Interaction with deletions
+
+Deletions take precedence over overlays. If a row offset is marked deleted in the
+fragment's deletion file, any overlay value for that offset is dead and is
+ignored, regardless of commit order. This keeps the invariant simple: a deletion
+is the final word on a row, so a concurrent overlay against a row that was
+deleted needs no special conflict handling — its values are merely inert.
+
+### Physical layout
+
+An overlay's data file stores **one value column per field**, in the order of
+`data_file.fields`. It does **not** store a row-offset key column. The position of
+a covered offset's value within its column is the **rank** of that offset in the
+field's coverage bitmap — the number of set bits below it. For a Roaring bitmap
+this is an O(1) operation, so random access to any cell is a rank computation
+followed by a single value fetch, with no offset column to read and no binary
+search.
+
+Because different fields may cover different offset sets, the value columns of a
+single sparse overlay may have **different lengths**. The Lance file format
+permits columns of differing item counts within one file, so a sparse overlay is
+representable as a single file. (See [Writer support](#writer-support) for the
+current implementation status.)
+
+### Dense vs. sparse overlays
+
+A single overlay is one of two shapes:
+
+- **Dense (rectangular).** One `shared_offset_bitmap` applies to every field. Every
+ covered offset has a value for every field. This is the common case for a plain
+ `UPDATE`, where one `SET` list is applied to one set of rows.
+- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different
+ fields cover different offset sets — for example a `MERGE` with multiple
+ `WHEN MATCHED` branches, where different rows update different columns. A dense
+ overlay would have to widen to the bounding rectangle and fill the untouched
+ cells with their current values (post-images), which for wide columns such as
+ embeddings means re-storing data that did not change. A sparse overlay stores
+ exactly the changed cells.
+
+A writer may always express a non-rectangular update as **multiple dense overlays
+in one transaction** (one per coverage group) instead of a single sparse overlay.
+
+## Protobuf
+
+
+DataOverlayFile protobuf message
+
+```protobuf
+%%% proto.message.DataOverlayFile %%%
+```
+
+
+
+
+FieldCoverage protobuf message
+
+```protobuf
+%%% proto.message.FieldCoverage %%%
+```
+
+
+
+## Versioning and ordering
+
+Overlays reuse the dataset version as their ordering clock rather than
+introducing a separate generation counter.
+
+`committed_version` is the dataset version at which an overlay **became
+effective** — the version of the commit that introduced it, **not** the version
+it was read from. It is stamped at commit time and re-stamped if the commit is
+retried, in the same way as the created-at / last-updated-at version sequences.
+
+This single value drives every ordering decision:
+
+- **Overlay vs. overlay** (read precedence): higher `committed_version` wins.
+- **Overlay vs. index** (query correctness): an index records the
+ `dataset_version` it was built from. An index whose `dataset_version >=
+ committed_version` already incorporates the overlay. An overlay whose
+ `committed_version > index.dataset_version` is newer than the index and its
+ cells must be excluded from index results and re-evaluated.
+- **Scheduler signal**: the gap between an overlay's `committed_version` and an
+ index's `dataset_version`, or between an overlay and the base, is a staleness
+ measure the compaction scheduler can use.
+
+!!! note "Why effective version, not read version"
+
+ Suppose an overlay reads version 5 and commits at version 6, while an index
+ is built reading version 5 (before the overlay) and commits at version 7 with
+ `dataset_version = 5`. If the overlay stored its *read* version (5), the test
+ `5 > 5` is false, the row would not be excluded, and the index — which never
+ saw the overlay — would return a stale result. Storing the *effective*
+ version (6) makes `6 > 5` true, the cell is excluded and re-evaluated, and the
+ result is correct.
+
+## Index integration
+
+Building an index over a fragment that has overlays does **not** require dropping
+the fragment from the index's coverage. The fragment stays indexed, and the query
+path reconciles overlays at query time using an **exclusion set**.
+
+The exclusion set for an index on field `F` is the union of the coverage bitmaps,
+restricted to field `F`, of every overlay whose `committed_version >
+index.dataset_version`. The exclusion is **field-aware**: an overlay that touches
+only unrelated columns does not exclude anything from the index on `F`.
+
+The query then proceeds as:
+
+1. Run the index search as usual, producing candidate rows.
+2. Remove any candidate in the exclusion set. (Its indexed value may be stale.)
+3. **Re-evaluate** the excluded rows against their current values — the same flat
+ path already used for the unindexed tail of fragments. For a scalar predicate
+ this re-applies the filter; for a vector query it re-scores the row's current
+ vector. Rows that still match are added back to the result.
+
+Step 3 is what makes exclusion correct rather than merely safe: removing a row
+from index candidates without re-evaluating it would silently drop a row that
+should match under its new value.
+
+### Correctness invariant
+
+> For every indexed field `F` and every row offset `o` in a fragment the index
+> covers, the index's entry for `(o, F)` is trusted unless `o` is excluded.
+> `o` is excluded iff some overlay with `committed_version > index.dataset_version`
+> covers `(o, F)`.
+
+The write and compaction paths together preserve this:
+
+- **Writes** change a cell only by adding an overlay, and that overlay's
+ `committed_version` exceeds the version of any pre-existing index — so the
+ change is always covered by an exclusion.
+- **Compaction** may remove an overlay only if the index no longer relies on it
+ (see below).
+
+## Compaction
+
+Overlays accumulate read cost — every overlay is a bitmap to test and a possible
+file to open. Compaction bounds that cost in two modes:
+
+- **Overlay → overlay.** Merge several overlays into fewer, computing the
+ post-image per `(offset, field)` by walking the merged overlays newest-first.
+ The merged overlay takes the **maximum** `committed_version` of its inputs, so
+ the exclusion semantics are preserved. Indexes are unaffected. This is cheap and
+ does not touch the base.
+- **Overlay → base.** Fold overlays into a fresh base data file, computing the
+ post-image for every covered cell, then clear the overlays. The base is
+ complete, so every post-image is well defined. Overlay offsets are physical, so
+ they cannot survive a rewrite that reorders rows; folding therefore materializes
+ values rather than carrying overlays forward.
+
+!!! warning "Folding an indexed field must update its index"
+
+ An overlay→base fold removes the overlay, which removes the exclusion signal
+ that kept an index correct. Folding an overlay that covers an indexed field
+ `F` is therefore equivalent to a column rewrite of `F` and must, in the same
+ commit, either rebuild the index to a `dataset_version` at least the folded
+ overlay's `committed_version`, or remove the fragment from the index's
+ coverage so the rows fall to the flat path. Otherwise the index would serve
+ stale values with no overlay to exclude them. This is the same rule that
+ already governs rewriting a column that an index is built on.
+
+When a fragment with overlays is compacted by a row-rewriting operation
+(`RewriteRows`, which produces new fragments with new row addresses), the
+overlays are folded into the new base as part of the rewrite, and existing
+[fragment-reuse remapping](row_id_lineage.md) handles the row-address changes as
+it does today.
+
+## Row lineage
+
+An overlay write updates the `last_updated_at_version` of every covered row, so
+change-data-feed and time-travel queries observe the update. Because overlays are
+addressed by physical offset, they do **not** require stable row IDs to be
+enabled; lineage updates apply only when those features are on.
+
+## Worked example
+
+A table `users` with stable row IDs enabled and these fields:
+
+| field id | name | type |
+|----------|-----------|-------------------------|
+| 1 | id | `int32` (primary key) |
+| 2 | name | `utf8` |
+| 3 | age | `int32` |
+| 4 | embedding | `fixed_size_list`|
+
+Created at version 1 as a single fragment `0` with one base data file
+`data/file0.lance` holding all four columns. `physical_rows = 4`:
+
+| offset | id | name | age | embedding |
+|--------|----|-------|-----|------------------|
+| 0 | 1 | Alice | 30 | … |
+| 1 | 2 | Bob | 25 | … |
+| 2 | 3 | Carol | 40 | … |
+| 3 | 4 | Dave | 22 | … |
+
+A BTree scalar index on `age` is built at version 1, covering fragment `0`
+(`dataset_version = 1`).
+
+### Step 1 — write an overlay
+
+```sql
+UPDATE users SET age = 26 WHERE id = 2; -- Bob, offset 1
+```
+
+This touches one field (`age`) for one row, so the writer emits a dense overlay
+and commits it as version 2. Fragment `0` gains:
+
+```text
+DataOverlayFile {
+ data_file: { path: "data/overlay-.lance", fields: [3], column_indices: [0] }
+ coverage: shared_offset_bitmap = {1}
+ committed_version: 2
+}
+```
+
+The overlay file stores a single `age` column with one value, `[26]`, at
+rank `{1}.rank(1) = 0`. `last_updated_at_version[1]` is set to 2.
+
+### Step 2 — read
+
+`SELECT id, age FROM users` reads base ages `[30, 25, 40, 22]`. For `age`
+(field 3), the overlay covers offset 1, so `age[1]` is replaced with the overlay
+value at position `{1}.rank(1) = 0` → `26`. Result ages: `[30, 26, 40, 22]`.
+
+### Step 3 — index query
+
+```sql
+SELECT * FROM users WHERE age = 26;
+```
+
+The `age` index was built at `dataset_version = 1`; the overlay's
+`committed_version` is 2. Since `2 > 1`, the overlay's coverage for `age`, `{1}`,
+is the exclusion set for this query.
+
+- The index (built at v1) holds Bob's *old* `age = 25`, so a lookup for `26`
+ returns nothing from the index.
+- Offset 1 is in the exclusion set, so it is re-evaluated on the flat path. Its
+ current `age` (26, via the overlay) matches `age = 26`, so Bob is returned.
+
+The mirror case `WHERE age = 25` shows exclusion preventing a stale hit: the index
+returns offset 1 (stale `25`), but offset 1 is excluded, re-evaluated to `26`, and
+correctly dropped.
+
+### Step 4 — a second, non-rectangular write
+
+```sql
+MERGE INTO users USING staged ON users.id = staged.id
+WHEN MATCHED AND staged.kind = 'rename' THEN UPDATE SET name = staged.name -- Carol(2), Dave(3)
+WHEN MATCHED AND staged.kind = 'revec' THEN UPDATE SET embedding = staged.embedding -- Bob(1)
+```
+
+`name` is updated for offsets `{2, 3}` and `embedding` for offset `{1}` — different
+fields over different rows. This is a sparse overlay, committed as version 3:
+
+```text
+DataOverlayFile {
+ data_file: { path: "data/overlay-.lance", fields: [2, 4], column_indices: [0, 1] }
+ coverage: field_coverage { offset_bitmaps: [ {2,3}, {1} ] }
+ // name (field 2) ^ ^ embedding (field 4)
+ committed_version: 3
+}
+```
+
+The file's `name` column has **two** values (`["Caroline", "David"]`, at
+ranks 0 and 1 of `{2,3}`) and its `embedding` column has **one** value (at rank 0
+of `{1}`) — columns of different lengths in one file.
+
+### Step 5 — read after the second write
+
+`SELECT name, age, embedding FROM users` resolves each field independently,
+newest overlay first:
+
+- `name`: the v3 overlay covers `{2,3}` → `["Alice", "Bob", "Caroline", "David"]`.
+- `age`: the v3 overlay does not cover `age`; the v2 overlay still applies at
+ offset 1 → `[30, 26, 40, 22]`.
+- `embedding`: the v3 overlay covers `{1}` → Bob's vector is the new one, others
+ from base.
+
+Overlays from different versions coexist and apply per field.
+
+### Step 6 — compaction (overlay → base)
+
+The scheduler folds both overlays into fragment `0` at version 4, computing
+post-images for `age`, `name`, and `embedding`, and writing a new base data file
+`data/file1.lance` with those columns. In the old file, fields 2, 3, and 4 are
+tombstoned (`-2`); field 1 (`id`) remains. The fragment's `overlays` list is
+cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so
+stable row IDs and the deletion vector are untouched.
+
+Because the fold removed the overlay that was excluding offset 1 from the `age`
+index, the same commit must reconcile that index: either rebuild it at
+`dataset_version >= 2`, or drop fragment `0` from its coverage so `age` queries
+fall to the flat path. After a rebuild at version 4, no overlay remains and the
+`age` index directly returns `26` for Bob with no exclusion needed.
+
+## Guidance
+
+!!! note "This section is a stub."
+
+ The following are implementation considerations, not part of the on-disk
+ specification.
+
+### When to overlay vs. rewrite a column vs. move rows
+
+*(To be expanded.)* The choice between appending an overlay, rewriting a full
+column (data evolution), and moving updated rows to a new fragment depends on the
+fraction of rows changed, the fraction of columns changed, column width, the
+presence of indexes on the changed columns, and the accumulated overlay read
+cost. Roughly: few rows changed favors overlays; most rows in a few columns
+favors a column rewrite; most columns changed favors moving rows to a new
+fragment.
+
+### Writer support
+
+*(To be expanded.)* Dense (rectangular) overlays write with the existing
+equal-length file writer today. Sparse overlays stored as a **single** file
+require the writer to emit columns of independent lengths, which the current v2
+writer does not yet do (it advances all columns from one global row counter).
+Until that support lands, a writer can express a sparse update as multiple dense
+overlays in one transaction.
+
+### Scheduling compaction
+
+*(To be expanded.)* The overlay→overlay and overlay→base modes have very
+different costs; a cost/benefit scheduler decides when each is worthwhile, using
+the version gap as a staleness signal.
+
+### Open questions
+
+*(To be resolved.)*
+
+- **Per-fragment vs. per-table overlays.** Overlays are attached per fragment.
+ Should there be a table-level overlay concept, and how would it interact with
+ fragment-level row addressing?
+- **Relationship to LSM.** Overlays plus compaction resemble an LSM tree (newest
+ layer wins, periodic merge). How far should that analogy be taken, and what do
+ we deliberately do differently given Lance's random-access requirements?
+- **Coverage bitmap spill.** Coverage bitmaps live inline in the manifest. Very
+ large coverage (an overlay touching many rows) may warrant external spill, as
+ the row-ID and last-updated-at sequences already do above a size threshold.
+
+## Related specifications
+
+- [Table format overview](index.md)
+- [Transactions](transaction.md)
+- [Row ID & Lineage](row_id_lineage.md)
+- [Index Formats](../index/index.md)
+- [Format Versioning](versioning.md)
diff --git a/docs/src/format/table/index.md b/docs/src/format/table/index.md
index 94ea4b90dc9..ce4d0b26613 100644
--- a/docs/src/format/table/index.md
+++ b/docs/src/format/table/index.md
@@ -168,6 +168,35 @@ However, this invalidates row addresses and requires rebuilding indices, which c
+## Data Overlay Files
+
+!!! note "Overlay files require feature flag 64 (data overlay files)"
+
+Overlay files supply new values for a subset of `(row offset, field)` cells within
+a fragment without rewriting the base data files. They make updates cheap when only
+a small percentage of rows and/or columns change: a writer appends a small file
+carrying just the changed cells instead of rewriting whole columns or moving rows
+to a new fragment.
+
+On read, each cell is resolved by consulting the fragment's overlays from newest to
+oldest; the first overlay covering that `(offset, field)` wins, otherwise the value
+falls through to the base data file. Indices keep covering the fragment and reconcile
+overlays at query time through a field-aware exclusion set.
+
+For the full specification — coverage and resolution rules, dense vs. sparse layout,
+versioning, index integration, compaction, and a worked example — see the
+[Data Overlay Files Specification](data_overlay_file.md).
+
+
+DataOverlayFile protobuf message
+
+```protobuf
+%%% proto.message.DataOverlayFile %%%
+```
+
+
+
+
## Related Specifications
### Storage Layout
diff --git a/protos/table.proto b/protos/table.proto
index d298809d5d8..cc8b477a6a6 100644
--- a/protos/table.proto
+++ b/protos/table.proto
@@ -113,6 +113,11 @@ message Manifest {
// * 2: row ids are stable and stored as part of the fragment metadata.
// * 4: use v2 format (deprecated)
// * 8: table config is present
+ // * 16: data files use multiple base paths (shallow clone / multi-base)
+ // * 32: the transaction file under _transactions is not written (inline only)
+ // * 64: data overlay files are present (see DataOverlayFile). Readers that do
+ // not understand overlays must refuse the dataset, since ignoring an overlay
+ // would silently return stale base values.
uint64 reader_feature_flags = 9;
// Feature flags for writers.
@@ -311,6 +316,15 @@ message DataFragment {
repeated DataFile files = 2;
+ // Optional overlay files for this fragment, which supply new values for a
+ // subset of cells without rewriting the base data files. This MUST be empty
+ // if the data overlay files feature flag (64) is not set in the manifest.
+ //
+ // Order is significant: a later entry is newer than an earlier one. When two
+ // overlays cover the same (offset, field) and share a `committed_version`, the
+ // later entry wins. See DataOverlayFile for the full resolution rules.
+ repeated DataOverlayFile overlays = 11;
+
// File that indicates which rows, if any, should be considered deleted.
DeletionFile deletion_file = 3;
@@ -433,6 +447,66 @@ message DataFile {
optional uint32 base_id = 7;
} // DataFile
+// An overlay file supplies new values for a subset of (row offset, field) cells
+// within a fragment, without rewriting the fragment's base data files. It is
+// used for efficient updates when only a small fraction of rows and/or columns
+// change.
+//
+// On read, a cell is resolved by consulting the fragment's overlays from newest
+// to oldest: the first overlay that covers that (offset, field) wins; if none
+// cover it, the value falls through to the base data file. Because deletions
+// take precedence over overlays, an overlay value for an offset that is also
+// marked deleted is dead and is ignored.
+//
+// The overlay's data file does NOT store a row-offset key column. Within a value
+// column, the position of a covered offset's value is the rank (0-based count of
+// set bits below it) of that offset within the field's coverage bitmap. Because
+// fields may cover different offset sets, the value columns of a single overlay
+// data file may have different lengths (which the Lance file format permits).
+message DataOverlayFile {
+ // The data file storing the overlay's new cell values, one value column per
+ // field in `data_file.fields`. No row-offset key column is stored.
+ DataFile data_file = 1;
+
+ // Which (offset, field) cells this overlay provides values for.
+ oneof coverage {
+ // A single 32-bit Roaring bitmap of physical row offsets that applies to
+ // every field in `data_file.fields` (a "dense" / rectangular overlay).
+ // Every covered offset has a value for every field. This is the common case
+ // for a plain UPDATE, where one SET list is applied to one set of rows.
+ bytes shared_offset_bitmap = 2;
+ // Per-field coverage for a "sparse" overlay, used when different fields cover
+ // different offset sets (e.g. a MERGE with multiple WHEN MATCHED branches).
+ FieldCoverage field_coverage = 4;
+ }
+
+ // The dataset version at which this overlay became effective: the version of
+ // the commit that introduced it, NOT the version it was read from. It is
+ // stamped at commit time and re-stamped if the commit is retried, in the same
+ // way as the created-at / last-updated-at version sequences.
+ //
+ // This drives two orderings:
+ // * Versus index builds: an index whose `dataset_version` >= this value
+ // already incorporates this overlay. Otherwise the overlay's covered cells
+ // are excluded from index results for the affected fields and re-evaluated
+ // against their current values (see the Data Overlay Files specification).
+ // * Versus other overlays: when two overlays cover the same (offset, field),
+ // the one with the higher `committed_version` wins. Overlays that share a
+ // `committed_version` are ordered by their position in
+ // `DataFragment.overlays`, where a later entry is newer and wins.
+ uint64 committed_version = 3;
+}
+
+// Per-field coverage for a sparse overlay.
+message FieldCoverage {
+ // One entry per field in the overlay's `data_file.fields`, in the same order.
+ // Each is a 32-bit Roaring bitmap of the physical row offsets covered for that
+ // field. An offset present in a field's bitmap but mapped to a NULL value
+ // means the cell is overridden to NULL (distinct from an offset that is absent,
+ // which falls through to the base data file).
+ repeated bytes offset_bitmaps = 1;
+}
+
// Deletion File
//
// The path of the deletion file is constructed as:
diff --git a/protos/transaction.proto b/protos/transaction.proto
index e72e95025a4..bfc0eee354b 100644
--- a/protos/transaction.proto
+++ b/protos/transaction.proto
@@ -315,6 +315,44 @@ message Transaction {
repeated DataReplacementGroup replacements = 1;
}
+ // Overlay files to append to a single fragment, in order (the last entry is
+ // newest). The overlays are appended to the fragment's existing `overlays`
+ // list; they do not replace it, so overlays written by concurrent commits are
+ // preserved.
+ message DataOverlayGroup {
+ uint64 fragment_id = 1;
+ // Each DataOverlayFile.committed_version is left 0 by the writer and stamped
+ // to the new dataset version at commit time (re-stamped on retry), in the
+ // same way as the created-at / last-updated-at version sequences. The fields
+ // touched are read from each overlay's `data_file.fields`.
+ repeated DataOverlayFile overlays = 2;
+ }
+
+ // Attach overlay files to fragments, supplying new values for a subset of
+ // (row offset, field) cells without rewriting the fragments' base data files.
+ // See the DataOverlayFile message in table.proto and the Data Overlay Files
+ // specification for resolution, coverage, and versioning rules.
+ //
+ // Conflict semantics (intentionally permissive, like DataReplacement). Against
+ // a concurrent operation that touches one of the same fragments:
+ // * Another DataOverlay (any fields): COMPATIBLE. Overlays stack; when two
+ // overlays cover the same (offset, field) the one with the higher
+ // `committed_version` wins, so independent backfills never conflict.
+ // * Append / new fragments: COMPATIBLE.
+ // * Delete: COMPATIBLE. A deletion takes precedence over an overlay, so an
+ // overlay value for a deleted offset is inert (no special handling needed).
+ // * DataReplacement or column-rewrite (Update with REWRITE_COLUMNS) of the
+ // same field: COMPATIBLE. Both preserve physical row addresses, so overlay
+ // offsets stay valid; the overlay is newer and wins its covered cells, and
+ // the version gate excludes those cells from any rebuilt index.
+ // * Row-rewrite, compaction, or an overlay->base fold of the fragment:
+ // CONFLICT. These change physical row addresses or consume the overlays, so
+ // the overlay's offsets are no longer valid. The writer must re-read the new
+ // fragment, recompute, and retry.
+ message DataOverlay {
+ repeated DataOverlayGroup groups = 1;
+ }
+
// Update the merged generations in MemWAL index.
// This operation is used during merge-insert to atomically record which
// generations have been merged to the base table.
@@ -346,6 +384,7 @@ message Transaction {
UpdateMemWalState update_mem_wal_state = 112;
Clone clone = 113;
UpdateBases update_bases = 114;
+ DataOverlay data_overlay = 115;
}
// Fields 200/202 (`blob_append` / `blob_overwrite`) previously represented blob dataset ops.
diff --git a/rust/lance-file/src/reader.rs b/rust/lance-file/src/reader.rs
index c454f73819e..98135b60f7e 100644
--- a/rust/lance-file/src/reader.rs
+++ b/rust/lance-file/src/reader.rs
@@ -491,6 +491,87 @@ impl FileReader {
self.num_rows
}
+ /// The number of rows stored in a single physical column.
+ ///
+ /// For ordinary (rectangular) files every column has the same length, equal
+ /// to [`num_rows`](Self::num_rows). Files written with
+ /// [`FileWriter::write_column`](crate::writer::FileWriter::write_column)
+ /// may have columns of differing lengths; this returns the length of one
+ /// such column, derived by summing its pages' row counts. Returns `None` if
+ /// `column_index` is out of bounds.
+ pub fn column_num_rows(&self, column_index: usize) -> Option {
+ self.metadata
+ .column_metadatas
+ .get(column_index)
+ .map(|col| col.pages.iter().map(|page| page.length).sum())
+ }
+
+ // Number of physical columns a field (and its descendants) contributes to a
+ // projection, mirroring `ReaderProjection::from_field_ids_helper`. Used to
+ // partition a projection's flat `column_indices` by top-level field.
+ fn count_projected_columns(field: &Field, is_structural: bool) -> usize {
+ let contributes = !is_structural
+ || field.children.is_empty()
+ || field.is_blob()
+ || field.is_packed_struct();
+ let recurse = !is_structural || (!field.is_blob() && !field.is_packed_struct());
+ let mut count = contributes as usize;
+ if recurse {
+ for child in &field.children {
+ count += Self::count_projected_columns(child, is_structural);
+ }
+ }
+ count
+ }
+
+ // The top-level row count of each projected top-level field, in projection
+ // order. A field's length is the page-row sum of its root (first) physical
+ // column, which equals the field's top-level row count for primitive,
+ // struct, and list fields in both v2.0 and v2.1. Ordinary files have equal
+ // lengths across columns; files written with `write_column` may not.
+ fn projected_field_lengths(&self, projection: &ReaderProjection) -> Vec {
+ let is_structural = self.metadata.version() >= LanceFileVersion::V2_1;
+ let mut lengths = Vec::with_capacity(projection.schema.fields.len());
+ let mut cursor = 0usize;
+ for field in &projection.schema.fields {
+ let n = Self::count_projected_columns(field, is_structural);
+ if n == 0 {
+ continue;
+ }
+ let Some(&root_column) = projection.column_indices.get(cursor) else {
+ break;
+ };
+ lengths.push(self.column_num_rows(root_column as usize).unwrap_or(0));
+ cursor += n;
+ }
+ lengths
+ }
+
+ // The reader combines a projection's columns into rectangular batches, so
+ // they must all have the same length. Returns that common length, or a
+ // descriptive error (naming the per-column lengths) when they differ.
+ // Ordinary files always pass; only files written with `write_column` whose
+ // columns ended up unequal can fail, and those must be read separately.
+ fn verify_uniform_lengths(projection: &ReaderProjection, field_lengths: &[u64]) -> Result {
+ let first = field_lengths.first().copied().unwrap_or(0);
+ if field_lengths.iter().all(|&len| len == first) {
+ return Ok(first);
+ }
+ let columns = projection
+ .schema
+ .fields
+ .iter()
+ .map(|f| f.name.as_str())
+ .zip(field_lengths.iter().copied())
+ .map(|(name, len)| format!("{name}={len}"))
+ .collect::>()
+ .join(", ");
+ Err(Error::invalid_input(format!(
+ "cannot read columns of differing lengths together ({columns}); \
+ read each column (or equal-length group) separately"
+ )))
+ }
+
pub fn metadata(&self) -> &Arc {
&self.metadata
}
@@ -1221,11 +1302,18 @@ impl FileReader {
) -> Result + Send>>> {
let projection = projection.unwrap_or_else(|| self.base_projection.clone());
Self::validate_projection(&projection, &self.metadata)?;
+ // All projected columns must share a length: the reader combines them
+ // into rectangular batches. Ordinary files satisfy this (every column
+ // has `self.num_rows` rows); files written with `FileWriter::write_column`
+ // may not, and such columns must be read separately. `read_len` is that
+ // common length, which `RangeFull`/`RangeFrom` resolve against (rather
+ // than `self.num_rows`, the file's longest column).
+ let field_lengths = self.projected_field_lengths(&projection);
+ let read_len = Self::verify_uniform_lengths(&projection, &field_lengths)?;
let verify_bound = |params: &ReadBatchParams, bound: u64, inclusive: bool| {
- if bound > self.num_rows || bound == self.num_rows && inclusive {
+ if bound > read_len || (bound == read_len && inclusive) {
Err(Error::invalid_input(format!(
- "cannot read {:?} from file with {} rows",
- params, self.num_rows
+ "cannot read {params:?} from columns with {read_len} rows"
)))
} else {
Ok(())
@@ -1267,13 +1355,8 @@ impl FileReader {
}
ReadBatchParams::RangeFrom(range) => {
verify_bound(¶ms, range.start as u64, true)?;
- self.read_range(
- range.start as u64..self.num_rows,
- batch_size,
- projection,
- filter,
- )
- .await
+ self.read_range(range.start as u64..read_len, batch_size, projection, filter)
+ .await
}
ReadBatchParams::RangeTo(range) => {
verify_bound(¶ms, range.end as u64, false)?;
@@ -1281,7 +1364,7 @@ impl FileReader {
.await
}
ReadBatchParams::RangeFull => {
- self.read_range(0..self.num_rows, batch_size, projection, filter)
+ self.read_range(0..read_len, batch_size, projection, filter)
.await
}
}
@@ -1476,11 +1559,18 @@ impl FileReader {
) -> Result> {
let projection = projection.unwrap_or_else(|| self.base_projection.clone());
Self::validate_projection(&projection, &self.metadata)?;
+ // All projected columns must share a length: the reader combines them
+ // into rectangular batches. Ordinary files satisfy this (every column
+ // has `self.num_rows` rows); files written with `FileWriter::write_column`
+ // may not, and such columns must be read separately. `read_len` is that
+ // common length, which `RangeFull`/`RangeFrom` resolve against (rather
+ // than `self.num_rows`, the file's longest column).
+ let field_lengths = self.projected_field_lengths(&projection);
+ let read_len = Self::verify_uniform_lengths(&projection, &field_lengths)?;
let verify_bound = |params: &ReadBatchParams, bound: u64, inclusive: bool| {
- if bound > self.num_rows || bound == self.num_rows && inclusive {
+ if bound > read_len || (bound == read_len && inclusive) {
Err(Error::invalid_input(format!(
- "cannot read {:?} from file with {} rows",
- params, self.num_rows
+ "cannot read {params:?} from columns with {read_len} rows"
)))
} else {
Ok(())
@@ -1521,7 +1611,7 @@ impl FileReader {
ReadBatchParams::RangeFrom(range) => {
verify_bound(¶ms, range.start as u64, true)?;
self.read_range_blocking(
- range.start as u64..self.num_rows,
+ range.start as u64..read_len,
batch_size,
projection,
filter,
@@ -1532,7 +1622,7 @@ impl FileReader {
self.read_range_blocking(0..range.end as u64, batch_size, projection, filter)
}
ReadBatchParams::RangeFull => {
- self.read_range_blocking(0..self.num_rows, batch_size, projection, filter)
+ self.read_range_blocking(0..read_len, batch_size, projection, filter)
}
}
}
diff --git a/rust/lance-file/src/writer.rs b/rust/lance-file/src/writer.rs
index 12bd50df6fe..96d282e978b 100644
--- a/rust/lance-file/src/writer.rs
+++ b/rust/lance-file/src/writer.rs
@@ -6,7 +6,7 @@ use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::AtomicBool;
-use arrow_array::RecordBatch;
+use arrow_array::{ArrayRef, RecordBatch};
use arrow_data::ArrayData;
use bytes::{Buf, BufMut, Bytes, BytesMut};
@@ -221,6 +221,11 @@ pub struct FileWriter {
field_id_to_column_indices: Vec<(u32, u32)>,
num_columns: u32,
rows_written: u64,
+ // The number of rows written for each top-level field (i.e. each entry in
+ // `column_writers`). With `write_batch` every field advances together and
+ // these are all equal, but `write_column` advances one field at a time, so
+ // a single file may end up with columns of differing item counts.
+ field_rows_written: Vec,
global_buffers: Vec<(u64, u64)>,
schema_metadata: HashMap,
options: FileWriterOptions,
@@ -277,6 +282,7 @@ impl FileWriter {
column_metadata: Vec::new(),
num_columns: 0,
rows_written: 0,
+ field_rows_written: Vec::new(),
field_id_to_column_indices: Vec::new(),
global_buffers: Vec::new(),
schema_metadata: HashMap::new(),
@@ -467,6 +473,7 @@ impl FileWriter {
BatchEncoder::try_new(&schema, encoding_strategy.as_ref(), &encoding_options)?;
self.num_columns = encoder.num_columns();
+ self.field_rows_written = vec![0; encoder.field_encoders.len()];
self.column_writers = encoder.field_encoders;
self.column_metadata = vec![initial_column_metadata(); self.num_columns as usize];
self.field_id_to_column_indices = encoder.field_id_to_column_index;
@@ -490,13 +497,14 @@ impl FileWriter {
batch: &RecordBatch,
external_buffers: &mut OutOfLineBuffers,
) -> Result>> {
- self.schema
+ let items = self
+ .schema
.as_ref()
.unwrap()
.fields
.iter()
- .zip(self.column_writers.iter_mut())
- .map(|(field, column_writer)| {
+ .enumerate()
+ .map(|(field_idx, field)| {
let array =
batch
.column_by_name(&field.name)
@@ -507,19 +515,53 @@ impl FileWriter {
)
.into(),
))?;
+ Ok((field_idx, array.clone()))
+ })
+ .collect::>>()?;
+ self.encode_columns(&items, external_buffers)
+ }
+
+ // Encode a set of `(field index, array)` pairs, each advancing only its own
+ // column. The returned tasks must be written before the per-field row
+ // counters are advanced (see `advance_columns`).
+ fn encode_columns(
+ &mut self,
+ items: &[(usize, ArrayRef)],
+ external_buffers: &mut OutOfLineBuffers,
+ ) -> Result>> {
+ // Snapshot the starting row number of each field before borrowing the
+ // column writers mutably below.
+ let row_numbers = items
+ .iter()
+ .map(|(field_idx, _)| self.field_rows_written[*field_idx])
+ .collect::>();
+ items
+ .iter()
+ .zip(row_numbers)
+ .map(|((field_idx, array), row_number)| {
let repdef = RepDefBuilder::default();
let num_rows = array.len() as u64;
- column_writer.maybe_encode(
+ self.column_writers[*field_idx].maybe_encode(
array.clone(),
external_buffers,
repdef,
- self.rows_written,
+ row_number,
num_rows,
)
})
.collect::>>()
}
+ // Advance the per-field row counters after a set of columns has been
+ // written, keeping `rows_written` (the file's logical length) in sync as the
+ // maximum column length.
+ fn advance_columns(&mut self, items: &[(usize, ArrayRef)]) {
+ for (field_idx, array) in items {
+ self.field_rows_written[*field_idx] += array.len() as u64;
+ }
+ self.rows_written = self.field_rows_written.iter().copied().max().unwrap_or(0);
+ }
+
/// Schedule a batch of data to be written to the file
///
/// Note: the future returned by this method may complete before the data has been fully
@@ -557,18 +599,93 @@ impl FileWriter {
.flatten()
.collect::>();
- self.rows_written = match self.rows_written.checked_add(batch.num_rows() as u64) {
- Some(rows_written) => rows_written,
- None => {
- return Err(Error::invalid_input_source(format!("cannot write batch with {} rows because {} rows have already been written and Lance files cannot contain more than 2^64 rows", num_rows, self.rows_written).into()));
- }
- };
+ // `write_batch` advances every field by the same amount, keeping all
+ // columns equal length. Guard against overflowing the row counter.
+ if self.rows_written.checked_add(num_rows).is_none() {
+ return Err(Error::invalid_input_source(format!("cannot write batch with {} rows because {} rows have already been written and Lance files cannot contain more than 2^64 rows", num_rows, self.rows_written).into()));
+ }
+ for field_rows in self.field_rows_written.iter_mut() {
+ *field_rows += num_rows;
+ }
+ self.rows_written = self.field_rows_written.iter().copied().max().unwrap_or(0);
self.write_pages(encoding_tasks).await?;
Ok(())
}
+ /// Write a single column, advancing only that column's row counter.
+ ///
+ /// Unlike [`write_batch`](Self::write_batch), which advances every column
+ /// from a single shared row counter, this method advances one column
+ /// independently. Used across calls it produces a single file whose columns
+ /// may have different item counts — the physical layout used by sparse data
+ /// overlay files, where each field covers a different set of rows.
+ ///
+ /// `column_index` refers to a top-level field in the writer's schema (the
+ /// same order as the schema's fields). A column may be written across
+ /// multiple calls; its values are appended. A field that is never written
+ /// ends up as a zero-length column. The writer must have been created with
+ /// an explicit schema (via [`try_new`](Self::try_new)); a lazy schema cannot
+ /// be inferred here because individual calls need not cover every field.
+ ///
+ /// ```
+ /// # use arrow_array::{ArrayRef, Int32Array};
+ /// # use std::sync::Arc;
+ /// # use lance_file::writer::FileWriter;
+ /// # async fn example(writer: &mut FileWriter) -> lance_core::Result<()> {
+ /// // Field 0 gets three values, field 1 gets one — a non-rectangular file.
+ /// writer.write_column(0, Arc::new(Int32Array::from(vec![1, 2, 3]))).await?;
+ /// writer.write_column(1, Arc::new(Int32Array::from(vec![10]))).await?;
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub async fn write_column(&mut self, column_index: usize, array: ArrayRef) -> Result<()> {
+ let schema = self.schema.as_ref().ok_or_else(|| {
+ Error::invalid_input_source(
+ "write_column requires the writer to be created with an explicit schema".into(),
+ )
+ })?;
+ let field = schema.fields.get(column_index).ok_or_else(|| {
+ Error::invalid_input_source(
+ format!(
+ "write_column: field index {} is out of bounds (schema has {} fields)",
+ column_index,
+ schema.fields.len()
+ )
+ .into(),
+ )
+ })?;
+ if array.len() as u64 > u32::MAX as u64 {
+ return Err(Error::invalid_input_source(
+ "cannot write Lance files with more than 2^32 rows".into(),
+ ));
+ }
+ Self::verify_field_nullability(&array.to_data(), field)?;
+
+ // A never-advanced field simply remains a zero-length column, which the
+ // encoders handle at `finish` time.
+ if array.is_empty() {
+ return Ok(());
+ }
+
+ let columns = [(column_index, array)];
+ let mut external_buffers =
+ OutOfLineBuffers::new(self.tell().await?, PAGE_BUFFER_ALIGNMENT as u64);
+ let encoding_tasks = self.encode_columns(&columns, &mut external_buffers)?;
+ for external_buffer in external_buffers.take_buffers() {
+ Self::do_write_buffer(&mut self.writer, &external_buffer).await?;
+ }
+ let encoding_tasks = encoding_tasks
+ .into_iter()
+ .flatten()
+ .collect::>();
+
+ self.advance_columns(&columns);
+ self.write_pages(encoding_tasks).await?;
+ Ok(())
+ }
+
async fn write_column_metadata(
&mut self,
metadata: pbfile::ColumnMetadata,
@@ -974,11 +1091,11 @@ mod tests {
use std::collections::HashMap;
use std::sync::Arc;
- use crate::reader::{FileReader, FileReaderOptions, describe_encoding};
+ use crate::reader::{FileReader, FileReaderOptions, ReaderProjection, describe_encoding};
use crate::testing::FsFixture;
use crate::writer::{ENV_LANCE_FILE_WRITER_MAX_PAGE_BYTES, FileWriter, FileWriterOptions};
use arrow_array::builder::{Float32Builder, Int32Builder};
- use arrow_array::{Int32Array, RecordBatch, UInt64Array};
+ use arrow_array::{ArrayRef, Int32Array, RecordBatch, UInt64Array};
use arrow_array::{RecordBatchReader, StringArray, types::Float64Type};
use arrow_schema::{DataType, Field, Field as ArrowField, Schema, Schema as ArrowSchema};
use lance_core::cache::LanceCache;
@@ -990,6 +1107,7 @@ mod tests {
use lance_encoding::version::LanceFileVersion;
use lance_io::object_store::ObjectStore;
use lance_io::utils::CachedFileSize;
+ use rstest::rstest;
#[tokio::test]
async fn test_basic_write() {
@@ -1040,6 +1158,326 @@ mod tests {
file_writer.finish().await.unwrap();
}
+ // Read a single column back at an explicit range/index set, returning its
+ // `Int32` values. Reading one column (or an equal-length group) at a time is
+ // how unequal-length files are consumed: a full scan across columns of
+ // differing lengths cannot form a single rectangular batch.
+ async fn read_int32_column(
+ reader: &FileReader,
+ schema: &LanceSchema,
+ version: LanceFileVersion,
+ name: &str,
+ params: lance_io::ReadBatchParams,
+ ) -> Vec> {
+ use futures::TryStreamExt;
+ use lance_encoding::decoder::FilterExpression;
+
+ let projection = ReaderProjection::from_column_names(version, schema, &[name]).unwrap();
+ let batches: Vec = reader
+ .read_stream_projected(params, 1024, 16, projection, FilterExpression::no_filter())
+ .await
+ .unwrap()
+ .try_collect()
+ .await
+ .unwrap();
+ batches
+ .iter()
+ .flat_map(|b| {
+ b.column(0)
+ .as_any()
+ .downcast_ref::()
+ .unwrap()
+ .iter()
+ .collect::>()
+ })
+ .collect()
+ }
+
+ /// A single file may hold columns of differing item counts (no shared global
+ /// row counter). This is the physical layout used by sparse data overlay
+ /// files, where each field covers a different set of rows.
+ #[rstest]
+ #[tokio::test]
+ async fn test_write_columns_unequal_lengths(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ use lance_io::ReadBatchParams;
+
+ let arrow_schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("a", DataType::Int32, true),
+ ArrowField::new("b", DataType::Int32, true),
+ ArrowField::new("c", DataType::Int32, true),
+ ]));
+ let lance_schema = LanceSchema::try_from(arrow_schema.as_ref()).unwrap();
+
+ let fs = FsFixture::default();
+ let options = FileWriterOptions {
+ format_version: Some(version),
+ ..Default::default()
+ };
+ let mut writer = FileWriter::try_new(
+ fs.object_store.create(&fs.tmp_path).await.unwrap(),
+ lance_schema.clone(),
+ options,
+ )
+ .unwrap();
+
+ // Field "a" gets 5 values across two calls (appending), field "b" gets a
+ // single value, and field "c" is never written (a zero-length column).
+ let a1: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
+ let b: ArrayRef = Arc::new(Int32Array::from(vec![10]));
+ writer.write_column(0, a1).await.unwrap();
+ writer.write_column(1, b).await.unwrap();
+ let a2: ArrayRef = Arc::new(Int32Array::from(vec![4, 5]));
+ writer.write_column(0, a2).await.unwrap();
+ // A zero-length array for an otherwise-unwritten field is a no-op.
+ let c_empty: ArrayRef = Arc::new(Int32Array::from(Vec::::new()));
+ writer.write_column(2, c_empty).await.unwrap();
+
+ let summary = writer.finish().await.unwrap();
+ // The file's logical length is the longest column.
+ assert_eq!(summary.num_rows, 5);
+
+ let file_scheduler = fs
+ .scheduler
+ .open_file(&fs.tmp_path, &CachedFileSize::unknown())
+ .await
+ .unwrap();
+ let reader = FileReader::try_open(
+ file_scheduler,
+ None,
+ Arc::::default(),
+ &LanceCache::no_cache(),
+ FileReaderOptions::default(),
+ )
+ .await
+ .unwrap();
+
+ // Per-column row counts are recorded in / derivable from file metadata.
+ assert_eq!(reader.num_rows(), 5);
+ assert_eq!(reader.column_num_rows(0), Some(5));
+ assert_eq!(reader.column_num_rows(1), Some(1));
+ assert_eq!(reader.column_num_rows(2), Some(0));
+ assert_eq!(reader.column_num_rows(3), None);
+
+ // Each column reads back independently at its own length.
+ assert_eq!(
+ read_int32_column(
+ &reader,
+ &lance_schema,
+ version,
+ "a",
+ ReadBatchParams::Range(0..5)
+ )
+ .await,
+ vec![Some(1), Some(2), Some(3), Some(4), Some(5)],
+ );
+ assert_eq!(
+ read_int32_column(
+ &reader,
+ &lance_schema,
+ version,
+ "b",
+ ReadBatchParams::Range(0..1)
+ )
+ .await,
+ vec![Some(10)],
+ );
+
+ // Random access by position within the longer column returns the right
+ // value even though other columns are shorter. (The take path requires
+ // strictly increasing indices.)
+ assert_eq!(
+ read_int32_column(
+ &reader,
+ &lance_schema,
+ version,
+ "a",
+ ReadBatchParams::Indices(arrow_array::UInt32Array::from(vec![0, 2, 4])),
+ )
+ .await,
+ vec![Some(1), Some(3), Some(5)],
+ );
+ }
+
+ /// Reading an unequal-length file:
+ /// - a projection whose columns are equal length full-scans normally;
+ /// - a full scan across columns of differing length is rejected up front,
+ /// before any batch is produced (even though a prefix would be rectangular);
+ /// - a bounded read is valid as long as every projected column covers it;
+ /// - a single-column `RangeFull` resolves to that column's own length, not
+ /// the file's (maximum) length.
+ #[rstest]
+ #[tokio::test]
+ async fn test_read_unequal_length_projection(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ use futures::TryStreamExt;
+ use lance_encoding::decoder::FilterExpression;
+ use lance_io::ReadBatchParams;
+
+ let arrow_schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("a", DataType::Int32, true),
+ ArrowField::new("b", DataType::Int32, true),
+ ArrowField::new("c", DataType::Int32, true),
+ ]));
+ let lance_schema = LanceSchema::try_from(arrow_schema.as_ref()).unwrap();
+ let fs = FsFixture::default();
+ let options = FileWriterOptions {
+ format_version: Some(version),
+ ..Default::default()
+ };
+ let mut writer = FileWriter::try_new(
+ fs.object_store.create(&fs.tmp_path).await.unwrap(),
+ lance_schema.clone(),
+ options,
+ )
+ .unwrap();
+ // "a" and "b" are equal length (5); "c" is shorter (1).
+ writer
+ .write_column(0, Arc::new(Int32Array::from(vec![1, 2, 3, 4, 5])))
+ .await
+ .unwrap();
+ writer
+ .write_column(1, Arc::new(Int32Array::from(vec![6, 7, 8, 9, 10])))
+ .await
+ .unwrap();
+ writer
+ .write_column(2, Arc::new(Int32Array::from(vec![100])))
+ .await
+ .unwrap();
+ writer.finish().await.unwrap();
+
+ let file_scheduler = fs
+ .scheduler
+ .open_file(&fs.tmp_path, &CachedFileSize::unknown())
+ .await
+ .unwrap();
+ let reader = FileReader::try_open(
+ file_scheduler,
+ None,
+ Arc::::default(),
+ &LanceCache::no_cache(),
+ FileReaderOptions::default(),
+ )
+ .await
+ .unwrap();
+
+ let read = |names: &'static [&'static str], params: ReadBatchParams| {
+ let projection =
+ ReaderProjection::from_column_names(version, &lance_schema, names).unwrap();
+ async {
+ match reader
+ .read_stream_projected(
+ params,
+ 1024,
+ 16,
+ projection,
+ FilterExpression::no_filter(),
+ )
+ .await
+ {
+ Ok(stream) => stream.try_collect::>().await,
+ Err(e) => Err(e),
+ }
+ }
+ };
+ let col_values = |batches: &[RecordBatch], idx: usize| -> Vec> {
+ batches
+ .iter()
+ .flat_map(|b| {
+ b.column(idx)
+ .as_any()
+ .downcast_ref::()
+ .unwrap()
+ .iter()
+ .collect::>()
+ })
+ .collect()
+ };
+
+ // Equal-length projection [a, b] full-scans into rectangular batches.
+ let batches = read(&["a", "b"], ReadBatchParams::RangeFull).await.unwrap();
+ assert_eq!(
+ col_values(&batches, 0),
+ vec![Some(1), Some(2), Some(3), Some(4), Some(5)]
+ );
+ assert_eq!(
+ col_values(&batches, 1),
+ vec![Some(6), Some(7), Some(8), Some(9), Some(10)]
+ );
+
+ // A mismatched-length projection [a, c] (5 vs 1) is rejected before any
+ // batch is yielded, regardless of the read params — its columns cannot
+ // be combined into rectangular batches.
+ assert!(
+ read(&["a", "c"], ReadBatchParams::RangeFull).await.is_err(),
+ "full scan across unequal-length columns must error"
+ );
+ assert!(
+ read(&["a", "c"], ReadBatchParams::Range(0..1))
+ .await
+ .is_err(),
+ "even a common-prefix read of unequal-length columns must error"
+ );
+
+ // A single-column RangeFull resolves to that column's own length.
+ let batches = read(&["c"], ReadBatchParams::RangeFull).await.unwrap();
+ assert_eq!(col_values(&batches, 0), vec![Some(100)]);
+ let batches = read(&["a"], ReadBatchParams::RangeFull).await.unwrap();
+ assert_eq!(
+ col_values(&batches, 0),
+ vec![Some(1), Some(2), Some(3), Some(4), Some(5)]
+ );
+ }
+
+ /// Files written the ordinary (rectangular) way keep equal column lengths,
+ /// so the unequal-length support is backwards compatible.
+ #[tokio::test]
+ async fn test_write_batch_keeps_equal_lengths() {
+ let arrow_schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("a", DataType::Int32, true),
+ ArrowField::new("b", DataType::Int32, true),
+ ]));
+ let lance_schema = LanceSchema::try_from(arrow_schema.as_ref()).unwrap();
+
+ let fs = FsFixture::default();
+ let mut writer = FileWriter::try_new(
+ fs.object_store.create(&fs.tmp_path).await.unwrap(),
+ lance_schema,
+ FileWriterOptions::default(),
+ )
+ .unwrap();
+ let batch = RecordBatch::try_new(
+ arrow_schema.clone(),
+ vec![
+ Arc::new(Int32Array::from(vec![1, 2, 3])),
+ Arc::new(Int32Array::from(vec![4, 5, 6])),
+ ],
+ )
+ .unwrap();
+ writer.write_batch(&batch).await.unwrap();
+ let summary = writer.finish().await.unwrap();
+ assert_eq!(summary.num_rows, 3);
+
+ let file_scheduler = fs
+ .scheduler
+ .open_file(&fs.tmp_path, &CachedFileSize::unknown())
+ .await
+ .unwrap();
+ let reader = FileReader::try_open(
+ file_scheduler,
+ None,
+ Arc::::default(),
+ &LanceCache::no_cache(),
+ FileReaderOptions::default(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(reader.column_num_rows(0), Some(3));
+ assert_eq!(reader.column_num_rows(1), Some(3));
+ }
+
#[tokio::test]
async fn test_max_page_bytes_enforced() {
let arrow_field = Field::new("data", DataType::UInt64, false);
diff --git a/rust/lance-table/benches/manifest_intern.rs b/rust/lance-table/benches/manifest_intern.rs
index 78b7e352207..81bd57c1a22 100644
--- a/rust/lance-table/benches/manifest_intern.rs
+++ b/rust/lance-table/benches/manifest_intern.rs
@@ -59,6 +59,7 @@ fn make_uniform_pb_fragments(n: u64, num_fields: usize) -> Vec
file_size_bytes: 0,
base_id: None,
}],
+ overlays: vec![],
deletion_file: None,
row_id_sequence: None,
physical_rows: 1000,
@@ -135,6 +136,7 @@ fn make_diverse_pb_fragments(
file_size_bytes: 0,
base_id: None,
}],
+ overlays: vec![],
deletion_file: None,
row_id_sequence: None,
physical_rows: 1000,
diff --git a/rust/lance-table/src/feature_flags.rs b/rust/lance-table/src/feature_flags.rs
index 096f0da79e5..369c880c062 100644
--- a/rust/lance-table/src/feature_flags.rs
+++ b/rust/lance-table/src/feature_flags.rs
@@ -20,8 +20,22 @@ pub const FLAG_TABLE_CONFIG: u64 = 8;
pub const FLAG_BASE_PATHS: u64 = 16;
/// Disable writing transaction file under _transaction/, this flag is set when we only want to write inline transaction in manifest
pub const FLAG_DISABLE_TRANSACTION_FILE: u64 = 32;
+/// Fragments contain data overlay files, which supply new values for a subset of
+/// cells without rewriting base data files. A reader that does not understand
+/// overlays must refuse the dataset, since ignoring an overlay would silently
+/// return stale base values.
+///
+/// Data overlay files are not yet a released feature: in release builds this flag
+/// is treated as unknown (so a release reader/writer refuses an overlay dataset)
+/// unless [`ENABLE_DATA_OVERLAY_FILES_ENV`] is set, which lets benchmarks opt in.
+/// Debug builds always understand it so tests exercise the path.
+pub const FLAG_DATA_OVERLAY_FILES: u64 = 64;
/// The first bit that is unknown as a feature flag
-pub const FLAG_UNKNOWN: u64 = 64;
+pub const FLAG_UNKNOWN: u64 = 128;
+
+/// Environment variable that opts a release build into reading and writing data
+/// overlay files before the feature is generally released.
+pub const ENABLE_DATA_OVERLAY_FILES_ENV: &str = "LANCE_ENABLE_DATA_OVERLAY_FILES";
/// Set the reader and writer feature flags in the manifest based on the contents of the manifest.
pub fn apply_feature_flags(
@@ -71,18 +85,51 @@ pub fn apply_feature_flags(
manifest.writer_feature_flags |= FLAG_BASE_PATHS;
}
+ // Overlay files change cell values on read, so a reader that ignores them
+ // would return stale base values. Both readers and writers must understand
+ // them.
+ let has_overlays = manifest
+ .fragments
+ .iter()
+ .any(|frag| !frag.overlays.is_empty());
+ if has_overlays {
+ manifest.reader_feature_flags |= FLAG_DATA_OVERLAY_FILES;
+ manifest.writer_feature_flags |= FLAG_DATA_OVERLAY_FILES;
+ }
+
if disable_transaction_file {
manifest.writer_feature_flags |= FLAG_DISABLE_TRANSACTION_FILE;
}
Ok(())
}
+/// Whether this build understands data overlay files: always in debug builds,
+/// and in release builds only when [`ENABLE_DATA_OVERLAY_FILES_ENV`] is set.
+fn data_overlay_files_enabled() -> bool {
+ cfg!(debug_assertions) || std::env::var_os(ENABLE_DATA_OVERLAY_FILES_ENV).is_some()
+}
+
+/// The feature-flag bits this build understands, given whether overlay support
+/// is enabled. Split out from [`supported_flags`] so the policy is testable
+/// without toggling the build profile or environment.
+fn supported_flags_when(overlay_enabled: bool) -> u64 {
+ let mut supported = FLAG_UNKNOWN - 1;
+ if !overlay_enabled {
+ supported &= !FLAG_DATA_OVERLAY_FILES;
+ }
+ supported
+}
+
+fn supported_flags() -> u64 {
+ supported_flags_when(data_overlay_files_enabled())
+}
+
pub fn can_read_dataset(reader_flags: u64) -> bool {
- reader_flags < FLAG_UNKNOWN
+ reader_flags & !supported_flags() == 0
}
pub fn can_write_dataset(writer_flags: u64) -> bool {
- writer_flags < FLAG_UNKNOWN
+ writer_flags & !supported_flags() == 0
}
pub fn has_deprecated_v2_feature_flag(writer_flags: u64) -> bool {
@@ -103,6 +150,7 @@ mod tests {
assert!(can_read_dataset(super::FLAG_TABLE_CONFIG));
assert!(can_read_dataset(super::FLAG_BASE_PATHS));
assert!(can_read_dataset(super::FLAG_DISABLE_TRANSACTION_FILE));
+ assert!(can_read_dataset(super::FLAG_DATA_OVERLAY_FILES));
assert!(can_read_dataset(
super::FLAG_DELETION_FILES
| super::FLAG_STABLE_ROW_IDS
@@ -111,6 +159,53 @@ mod tests {
assert!(!can_read_dataset(super::FLAG_UNKNOWN));
}
+ #[test]
+ fn test_data_overlay_flag_release_gating() {
+ // Release default (overlays disabled): the overlay flag is treated as
+ // unknown so the dataset is refused, while other known flags still pass.
+ let supported = supported_flags_when(false);
+ assert_eq!(supported & FLAG_DATA_OVERLAY_FILES, 0);
+ assert_eq!(FLAG_DELETION_FILES & !supported, 0);
+ assert_ne!(FLAG_DATA_OVERLAY_FILES & !supported, 0);
+ // Enabled (debug or env opt-in): the overlay flag is understood.
+ let supported = supported_flags_when(true);
+ assert_eq!(FLAG_DATA_OVERLAY_FILES & !supported, 0);
+ }
+
+ #[test]
+ fn test_apply_feature_flags_sets_overlay_flag() {
+ use crate::format::{
+ DataFile, DataOverlayFile, DataStorageFormat, Fragment, OverlayCoverage,
+ };
+ use arrow_schema::{Field as ArrowField, Schema as ArrowSchema};
+ use lance_core::datatypes::Schema;
+ use roaring::RoaringBitmap;
+ use std::collections::HashMap;
+ use std::sync::Arc;
+
+ let arrow_schema = ArrowSchema::new(vec![ArrowField::new(
+ "id",
+ arrow_schema::DataType::Int64,
+ false,
+ )]);
+ let schema = Schema::try_from(&arrow_schema).unwrap();
+ let mut fragment = Fragment::new(0);
+ fragment.overlays = vec![DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", vec![0], None),
+ coverage: OverlayCoverage::dense(RoaringBitmap::from_iter([0u32])),
+ committed_version: 1,
+ }];
+ let mut manifest = Manifest::new(
+ schema,
+ Arc::new(vec![fragment]),
+ DataStorageFormat::default(),
+ HashMap::new(),
+ );
+ apply_feature_flags(&mut manifest, false, false).unwrap();
+ assert_ne!(manifest.reader_feature_flags & FLAG_DATA_OVERLAY_FILES, 0);
+ assert_ne!(manifest.writer_feature_flags & FLAG_DATA_OVERLAY_FILES, 0);
+ }
+
#[test]
fn test_write_check() {
assert!(can_write_dataset(0));
@@ -120,12 +215,14 @@ mod tests {
assert!(can_write_dataset(super::FLAG_TABLE_CONFIG));
assert!(can_write_dataset(super::FLAG_BASE_PATHS));
assert!(can_write_dataset(super::FLAG_DISABLE_TRANSACTION_FILE));
+ assert!(can_write_dataset(super::FLAG_DATA_OVERLAY_FILES));
assert!(can_write_dataset(
super::FLAG_DELETION_FILES
| super::FLAG_STABLE_ROW_IDS
| super::FLAG_USE_V2_FORMAT_DEPRECATED
| super::FLAG_TABLE_CONFIG
| super::FLAG_BASE_PATHS
+ | super::FLAG_DATA_OVERLAY_FILES
));
assert!(!can_write_dataset(super::FLAG_UNKNOWN));
}
diff --git a/rust/lance-table/src/format/fragment.rs b/rust/lance-table/src/format/fragment.rs
index 431e466dbd4..ef5166728ab 100644
--- a/rust/lance-table/src/format/fragment.rs
+++ b/rust/lance-table/src/format/fragment.rs
@@ -11,6 +11,7 @@ use lance_file::format::{MAJOR_VERSION, MINOR_VERSION};
use lance_file::version::LanceFileVersion;
use lance_io::utils::CachedFileSize;
use object_store::path::Path;
+use roaring::RoaringBitmap;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use crate::format::pb;
@@ -233,6 +234,210 @@ impl TryFrom for DataFile {
}
}
+/// Which `(physical offset, field)` cells a [`DataOverlayFile`] provides values
+/// for.
+///
+/// The coverage bitmaps index **physical** row offsets (positions in the base
+/// data files, counting deleted rows), so they are stable across deletions, like
+/// deletion vectors. Bitmaps are parsed from their 32-bit Roaring encoding once
+/// when the fragment is loaded and held behind an `Arc` so cloning a fragment is
+/// cheap; use [`DataOverlayFile::coverage_for_field`] to obtain the one that
+/// applies to a given field.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(into = "OverlayCoverageBytes", try_from = "OverlayCoverageBytes")]
+pub enum OverlayCoverage {
+ /// A single bitmap that applies to every field in the overlay's
+ /// `data_file.fields` (a dense / rectangular overlay): every covered offset
+ /// has a value for every field.
+ Shared(Arc),
+ /// One bitmap per field, in the same order as the overlay's
+ /// `data_file.fields` (a sparse overlay): different fields may cover
+ /// different offset sets.
+ PerField(Vec>),
+}
+
+/// Serialized form of [`OverlayCoverage`] — each bitmap as its 32-bit Roaring
+/// byte encoding. The in-memory form parses these once at load.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+enum OverlayCoverageBytes {
+ Shared(Vec),
+ PerField(Vec>),
+}
+
+fn deserialize_roaring(bytes: &[u8]) -> Result {
+ RoaringBitmap::deserialize_from(bytes).map_err(|e| {
+ Error::invalid_input(format!(
+ "failed to deserialize overlay coverage bitmap: {e}"
+ ))
+ })
+}
+
+fn serialize_roaring(bitmap: &RoaringBitmap) -> Vec {
+ let mut bytes = Vec::with_capacity(bitmap.serialized_size());
+ // Writing to a Vec is infallible.
+ bitmap.serialize_into(&mut bytes).unwrap();
+ bytes
+}
+
+impl From for OverlayCoverageBytes {
+ fn from(coverage: OverlayCoverage) -> Self {
+ match coverage {
+ OverlayCoverage::Shared(bitmap) => Self::Shared(serialize_roaring(&bitmap)),
+ OverlayCoverage::PerField(bitmaps) => {
+ Self::PerField(bitmaps.iter().map(|b| serialize_roaring(b)).collect())
+ }
+ }
+ }
+}
+
+impl TryFrom for OverlayCoverage {
+ type Error = Error;
+
+ fn try_from(bytes: OverlayCoverageBytes) -> Result {
+ Ok(match bytes {
+ OverlayCoverageBytes::Shared(b) => Self::Shared(Arc::new(deserialize_roaring(&b)?)),
+ OverlayCoverageBytes::PerField(bs) => Self::PerField(
+ bs.iter()
+ .map(|b| deserialize_roaring(b).map(Arc::new))
+ .collect::>()?,
+ ),
+ })
+ }
+}
+
+impl DeepSizeOf for OverlayCoverage {
+ fn deep_size_of_children(&self, _context: &mut lance_core::deepsize::Context) -> usize {
+ // RoaringBitmap does not expose its allocation size; its serialized size
+ // is a cheap, close proxy for the heap it holds.
+ let bitmap_heap = |bitmap: &RoaringBitmap| {
+ std::mem::size_of::() + bitmap.serialized_size()
+ };
+ match self {
+ Self::Shared(bitmap) => bitmap_heap(bitmap),
+ Self::PerField(bitmaps) => {
+ bitmaps.capacity() * std::mem::size_of::>()
+ + bitmaps.iter().map(|b| bitmap_heap(b)).sum::()
+ }
+ }
+ }
+}
+
+impl OverlayCoverage {
+ /// Build a dense coverage from a single bitmap shared across every field.
+ pub fn dense(bitmap: RoaringBitmap) -> Self {
+ Self::Shared(Arc::new(bitmap))
+ }
+
+ /// Build a sparse coverage from one bitmap per field.
+ pub fn sparse(bitmaps: Vec) -> Self {
+ Self::PerField(bitmaps.into_iter().map(Arc::new).collect())
+ }
+}
+
+/// An overlay file supplies new values for a subset of `(physical offset, field)`
+/// cells within a fragment, without rewriting the fragment's base data files. See
+/// the Data Overlay Files specification for the full resolution, coverage, and
+/// versioning rules.
+///
+/// The overlay's `data_file` stores one value column per field in
+/// `data_file.fields`, with **no** row-offset key column. Within a value column,
+/// the position of a covered offset's value is the **rank** (0-based count of set
+/// bits below it) of that offset in the field's coverage bitmap.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, DeepSizeOf)]
+pub struct DataOverlayFile {
+ /// The data file storing the overlay's new cell values.
+ pub data_file: DataFile,
+ /// Which cells this overlay provides values for.
+ pub coverage: OverlayCoverage,
+ /// The dataset version at which this overlay became effective (the version of
+ /// the commit that introduced it, stamped at commit time and re-stamped on
+ /// retry). Higher wins when two overlays cover the same `(offset, field)`.
+ pub committed_version: u64,
+}
+
+impl DataOverlayFile {
+ /// The parsed coverage bitmap that applies to the field stored at
+ /// `field_pos` within `data_file.fields`.
+ ///
+ /// For a dense overlay the same shared bitmap is returned for every field;
+ /// for a sparse overlay the per-field bitmap at `field_pos` is returned. The
+ /// bitmap is already parsed, so this is a cheap `Arc` clone.
+ pub fn coverage_for_field(&self, field_pos: usize) -> Result> {
+ match &self.coverage {
+ OverlayCoverage::Shared(bitmap) => Ok(bitmap.clone()),
+ OverlayCoverage::PerField(bitmaps) => {
+ bitmaps.get(field_pos).cloned().ok_or_else(|| {
+ Error::invalid_input(format!(
+ "overlay field_coverage has {} bitmaps but field position {} was requested",
+ bitmaps.len(),
+ field_pos
+ ))
+ })
+ }
+ }
+ }
+}
+
+/// Overlays are stored newest-last: a later list position is newer, and ties in
+/// `committed_version` are broken by position. Loading stable-sorts by
+/// `committed_version` so resolution can rely on the ordering without
+/// re-checking; the stable sort preserves the position tiebreak for equal
+/// versions.
+fn sort_overlays_newest_last(overlays: &mut [DataOverlayFile]) {
+ overlays.sort_by_key(|overlay| overlay.committed_version);
+}
+
+impl From<&DataOverlayFile> for pb::DataOverlayFile {
+ fn from(overlay: &DataOverlayFile) -> Self {
+ let coverage = match &overlay.coverage {
+ OverlayCoverage::Shared(bitmap) => {
+ pb::data_overlay_file::Coverage::SharedOffsetBitmap(serialize_roaring(bitmap))
+ }
+ OverlayCoverage::PerField(bitmaps) => {
+ pb::data_overlay_file::Coverage::FieldCoverage(pb::FieldCoverage {
+ offset_bitmaps: bitmaps.iter().map(|b| serialize_roaring(b)).collect(),
+ })
+ }
+ };
+ Self {
+ data_file: Some(pb::DataFile::from(&overlay.data_file)),
+ coverage: Some(coverage),
+ committed_version: overlay.committed_version,
+ }
+ }
+}
+
+impl TryFrom for DataOverlayFile {
+ type Error = Error;
+
+ fn try_from(proto: pb::DataOverlayFile) -> Result {
+ let data_file = proto
+ .data_file
+ .ok_or_else(|| Error::invalid_input("DataOverlayFile is missing its data_file"))?;
+ let coverage = match proto.coverage {
+ Some(pb::data_overlay_file::Coverage::SharedOffsetBitmap(bytes)) => {
+ OverlayCoverage::Shared(Arc::new(deserialize_roaring(&bytes)?))
+ }
+ Some(pb::data_overlay_file::Coverage::FieldCoverage(fc)) => OverlayCoverage::PerField(
+ fc.offset_bitmaps
+ .iter()
+ .map(|b| deserialize_roaring(b).map(Arc::new))
+ .collect::>()?,
+ ),
+ None => {
+ return Err(Error::invalid_input(
+ "DataOverlayFile is missing its coverage",
+ ));
+ }
+ };
+ Ok(Self {
+ data_file: DataFile::try_from(data_file)?,
+ coverage,
+ committed_version: proto.committed_version,
+ })
+ }
+}
+
/// Interns repeated data so that fragments with identical content share a
/// single heap allocation via `Arc`.
///
@@ -375,6 +580,15 @@ impl DataFileFieldInterner {
.into_iter()
.map(|f| self.intern_data_file(f))
.collect::>()?,
+ overlays: {
+ let mut overlays = p
+ .overlays
+ .into_iter()
+ .map(DataOverlayFile::try_from)
+ .collect::>>()?;
+ sort_overlays_newest_last(&mut overlays);
+ overlays
+ },
deletion_file: p.deletion_file.map(DeletionFile::try_from).transpose()?,
row_id_meta: p.row_id_sequence.map(RowIdMeta::try_from).transpose()?,
physical_rows,
@@ -483,6 +697,12 @@ pub struct Fragment {
/// Files within the fragment.
pub files: Vec,
+ /// Overlay files supplying new values for a subset of cells without
+ /// rewriting the base data files. Order is significant: a later entry is
+ /// newer than an earlier one. See [`DataOverlayFile`] for resolution rules.
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub overlays: Vec,
+
/// Optional file with deleted local row offsets.
#[serde(skip_serializing_if = "Option::is_none")]
pub deletion_file: Option,
@@ -510,6 +730,7 @@ impl Fragment {
Self {
id,
files: vec![],
+ overlays: vec![],
deletion_file: None,
row_id_meta: None,
physical_rows: None,
@@ -549,6 +770,7 @@ impl Fragment {
Self {
id,
files: vec![DataFile::new_legacy(path, schema, None, None)],
+ overlays: vec![],
deletion_file: None,
physical_rows,
row_id_meta: None,
@@ -669,6 +891,15 @@ impl TryFrom for Fragment {
.into_iter()
.map(DataFile::try_from)
.collect::>()?,
+ overlays: {
+ let mut overlays = p
+ .overlays
+ .into_iter()
+ .map(DataOverlayFile::try_from)
+ .collect::>>()?;
+ sort_overlays_newest_last(&mut overlays);
+ overlays
+ },
deletion_file: p.deletion_file.map(DeletionFile::try_from).transpose()?,
row_id_meta: p.row_id_sequence.map(RowIdMeta::try_from).transpose()?,
physical_rows,
@@ -716,6 +947,7 @@ impl From<&Fragment> for pb::DataFragment {
Self {
id: f.id,
files: f.files.iter().map(pb::DataFile::from).collect(),
+ overlays: f.overlays.iter().map(pb::DataOverlayFile::from).collect(),
deletion_file,
row_id_sequence,
physical_rows: f.physical_rows.unwrap_or_default() as u64,
@@ -734,6 +966,161 @@ mod tests {
use object_store::path::Path;
use serde_json::{Value, json};
+ #[test]
+ fn test_data_overlay_roundtrip() {
+ // A fragment carrying a dense overlay round-trips through protobuf and
+ // back, and the parsed coverage bitmap is recovered per field.
+ let mut bitmap = RoaringBitmap::new();
+ bitmap.insert(1);
+ bitmap.insert(3);
+
+ let overlay = DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("overlay-0.lance", vec![3], None),
+ coverage: OverlayCoverage::dense(bitmap.clone()),
+ committed_version: 7,
+ };
+ let mut fragment = Fragment::new(0);
+ fragment.files = vec![DataFile::new_legacy_from_fields(
+ "base.lance",
+ vec![1, 3],
+ None,
+ )];
+ fragment.overlays = vec![overlay];
+
+ let proto = pb::DataFragment::from(&fragment);
+ assert_eq!(proto.overlays.len(), 1);
+ let round_tripped = Fragment::try_from(proto).unwrap();
+ assert_eq!(round_tripped, fragment);
+
+ // Dense coverage applies to every field.
+ let recovered = round_tripped.overlays[0].coverage_for_field(0).unwrap();
+ assert_eq!(*recovered, bitmap);
+ assert_eq!(
+ *round_tripped.overlays[0].coverage_for_field(5).unwrap(),
+ bitmap
+ );
+ }
+
+ #[test]
+ fn test_data_overlay_sparse_per_field_coverage() {
+ // A sparse overlay carries one bitmap per field, recovered by position.
+ let name_coverage = RoaringBitmap::from_iter([2u32, 3]);
+ let embedding_coverage = RoaringBitmap::from_iter([1u32]);
+ let overlay = DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("overlay-1.lance", vec![2, 4], None),
+ coverage: OverlayCoverage::sparse(vec![
+ name_coverage.clone(),
+ embedding_coverage.clone(),
+ ]),
+ committed_version: 3,
+ };
+ let mut fragment = Fragment::new(1);
+ fragment.overlays = vec![overlay];
+
+ let round_tripped = Fragment::try_from(pb::DataFragment::from(&fragment)).unwrap();
+ assert_eq!(
+ *round_tripped.overlays[0].coverage_for_field(0).unwrap(),
+ name_coverage
+ );
+ assert_eq!(
+ *round_tripped.overlays[0].coverage_for_field(1).unwrap(),
+ embedding_coverage
+ );
+ }
+
+ #[test]
+ fn test_data_overlay_missing_fields_error() {
+ // A DataOverlayFile proto missing its coverage or data_file is rejected.
+ let no_coverage = pb::DataOverlayFile {
+ data_file: Some(pb::DataFile::from(&DataFile::new_legacy_from_fields(
+ "overlay.lance",
+ vec![3],
+ None,
+ ))),
+ coverage: None,
+ committed_version: 1,
+ };
+ let err = DataOverlayFile::try_from(no_coverage).unwrap_err();
+ assert!(err.to_string().contains("missing its coverage"), "{err}");
+
+ let no_data_file = pb::DataOverlayFile {
+ data_file: None,
+ coverage: Some(pb::data_overlay_file::Coverage::SharedOffsetBitmap(
+ serialize_roaring(&RoaringBitmap::from_iter([0u32])),
+ )),
+ committed_version: 1,
+ };
+ let err = DataOverlayFile::try_from(no_data_file).unwrap_err();
+ assert!(err.to_string().contains("missing its data_file"), "{err}");
+ }
+
+ #[test]
+ fn test_overlays_sorted_newest_last_on_load() {
+ // Overlays load stable-sorted by committed_version (newest last), with
+ // list position preserved as the tiebreak for equal versions.
+ let mk = |version: u64, field: i32| DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", vec![field], None),
+ coverage: OverlayCoverage::dense(RoaringBitmap::from_iter([0u32])),
+ committed_version: version,
+ };
+ let mut fragment = Fragment::new(0);
+ // Written out of order: v5, v2, v2 (second), v3.
+ fragment.overlays = vec![mk(5, 1), mk(2, 2), mk(2, 3), mk(3, 4)];
+
+ let loaded = Fragment::try_from(pb::DataFragment::from(&fragment)).unwrap();
+ let versions: Vec = loaded
+ .overlays
+ .iter()
+ .map(|o| o.committed_version)
+ .collect();
+ assert_eq!(versions, vec![2, 2, 3, 5]);
+ // Stable: the two v2 overlays keep their original relative order (field 2
+ // before field 3).
+ assert_eq!(
+ loaded.overlays[0].data_file.fields.as_ref(),
+ [2i32].as_slice()
+ );
+ assert_eq!(
+ loaded.overlays[1].data_file.fields.as_ref(),
+ [3i32].as_slice()
+ );
+ }
+
+ #[test]
+ fn test_overlay_coverage_serde_json_roundtrip() {
+ // The custom serde impl round-trips through JSON for dense/sparse,
+ // including empty bitmaps and a zero-bitmap sparse coverage.
+ for coverage in [
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1u32, 5, 100])),
+ OverlayCoverage::dense(RoaringBitmap::new()),
+ OverlayCoverage::sparse(vec![
+ RoaringBitmap::from_iter([2u32, 3]),
+ RoaringBitmap::new(),
+ ]),
+ OverlayCoverage::sparse(vec![]),
+ ] {
+ let json = serde_json::to_string(&coverage).unwrap();
+ let back: OverlayCoverage = serde_json::from_str(&json).unwrap();
+ assert_eq!(back, coverage);
+ }
+ }
+
+ #[test]
+ fn test_coverage_for_field_out_of_bounds() {
+ let overlay = DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", vec![2, 4], None),
+ coverage: OverlayCoverage::sparse(vec![
+ RoaringBitmap::from_iter([1u32]),
+ RoaringBitmap::from_iter([2u32]),
+ ]),
+ committed_version: 1,
+ };
+ assert!(overlay.coverage_for_field(0).is_ok());
+ assert!(overlay.coverage_for_field(1).is_ok());
+ let err = overlay.coverage_for_field(5).unwrap_err();
+ assert!(err.to_string().contains("field position"), "{err}");
+ }
+
#[test]
fn test_new_fragment() {
let path = "foobar.lance";
diff --git a/rust/lance-table/src/format/manifest.rs b/rust/lance-table/src/format/manifest.rs
index 9845061b7e4..cd0a403621f 100644
--- a/rust/lance-table/src/format/manifest.rs
+++ b/rust/lance-table/src/format/manifest.rs
@@ -1316,6 +1316,7 @@ mod tests {
vec![0, 1, 2],
None,
)],
+ overlays: vec![],
deletion_file: None,
row_id_meta: None,
physical_rows: None,
@@ -1328,6 +1329,7 @@ mod tests {
DataFile::new_legacy_from_fields("path2", vec![0, 1, 43], None),
DataFile::new_legacy_from_fields("path3", vec![2], None),
],
+ overlays: vec![],
deletion_file: None,
row_id_meta: None,
physical_rows: None,
diff --git a/rust/lance/src/dataset.rs b/rust/lance/src/dataset.rs
index 3e0d77704da..448feb961d7 100644
--- a/rust/lance/src/dataset.rs
+++ b/rust/lance/src/dataset.rs
@@ -79,6 +79,7 @@ pub mod index;
pub mod mem_wal;
mod metadata;
pub mod optimize;
+pub(crate) mod overlay;
pub mod progress;
pub mod refs;
pub(crate) mod rowids;
diff --git a/rust/lance/src/dataset/files.rs b/rust/lance/src/dataset/files.rs
index 848add7e4a8..2214822ed90 100644
--- a/rust/lance/src/dataset/files.rs
+++ b/rust/lance/src/dataset/files.rs
@@ -1036,6 +1036,7 @@ mod tests {
// No base_id -> falls back to the dataset base_uri.
mk_file("c.lance", None),
],
+ overlays: vec![],
// Deletion files also carry a base_id when they originate from a
// shallow clone, and must resolve against base_paths too.
deletion_file: Some(DeletionFile {
diff --git a/rust/lance/src/dataset/fragment.rs b/rust/lance/src/dataset/fragment.rs
index eb165e5f612..108c2debcf9 100644
--- a/rust/lance/src/dataset/fragment.rs
+++ b/rust/lance/src/dataset/fragment.rs
@@ -15,7 +15,8 @@ use arrow::compute::concat_batches;
use arrow_array::cast::as_primitive_array;
use arrow_array::types::UInt64Type;
use arrow_array::{
- Array, RecordBatch, RecordBatchReader, StructArray, UInt32Array, UInt64Array, new_null_array,
+ Array, ArrayRef, RecordBatch, RecordBatchReader, StructArray, UInt32Array, UInt64Array,
+ new_null_array,
};
use arrow_schema::Schema as ArrowSchema;
use datafusion::logical_expr::Expr;
@@ -47,7 +48,7 @@ use lance_table::format::{DataFile, DeletionFile, Fragment};
use lance_table::io::deletion::{deletion_file_path, write_deletion_file};
use lance_table::rowids::RowIdSequence;
use lance_table::utils::stream::{
- ReadBatchFutStream, ReadBatchTask, ReadBatchTaskStream, RowIdAndDeletesConfig,
+ ReadBatchFut, ReadBatchFutStream, ReadBatchTask, ReadBatchTaskStream, RowIdAndDeletesConfig,
wrap_with_row_id_and_delete,
};
use roaring::RoaringBitmap;
@@ -62,6 +63,9 @@ use super::updater::Updater;
use super::{NewColumnTransform, WriteParams, schema_evolution};
use crate::dataset::Dataset;
use crate::dataset::fragment::session::FragmentSession;
+use crate::dataset::overlay::{
+ assemble_overlay_column, overlay_indices_newest_first, route_overlays,
+};
use crate::io::deletion::read_dataset_deletion_file;
/// Result of [`FileFragment::update_columns_with_offsets`]: updated fragment metadata, modified field ids,
@@ -911,11 +915,24 @@ impl FileFragment {
futures::future::Either::Right(futures::future::ready(Ok(None)))
};
- let (opened_files, deletion_vec, row_id_sequence) =
- join!(open_files, deletion_vec_load, row_id_load);
+ // Open overlay value-column readers concurrently with the base files
+ // (no value bytes are read yet — that happens per batch on read).
+ let overlay_plan_load = if self.metadata.overlays.is_empty() {
+ futures::future::Either::Left(futures::future::ready(Ok(Vec::new())))
+ } else {
+ futures::future::Either::Right(self.load_overlay_plan(projection, &read_config))
+ };
+
+ let (opened_files, deletion_vec, row_id_sequence, overlay_plans) = join!(
+ open_files,
+ deletion_vec_load,
+ row_id_load,
+ overlay_plan_load
+ );
let opened_files = opened_files?;
let deletion_vec = deletion_vec?;
let row_id_sequence = row_id_sequence?;
+ let overlay_plans = overlay_plans?;
if opened_files.is_empty() && !read_config.has_system_cols() {
return Err(Error::not_found(format!(
@@ -938,6 +955,10 @@ impl FileFragment {
Arc::new(self.metadata.clone()),
)?;
+ if !overlay_plans.is_empty() {
+ reader.overlay_plans = Arc::new(overlay_plans);
+ }
+
if read_config.with_row_id {
reader.with_row_id();
}
@@ -1113,6 +1134,91 @@ impl FileFragment {
Ok(opened_files)
}
+ /// Open the overlay value-column readers for the projected fields, ordered
+ /// newest-first, ready to be merged into base batches on read.
+ ///
+ /// Each contributing overlay *file* is opened once (its metadata loaded),
+ /// projected to the fields it covers that the read also projects. The value
+ /// columns themselves are **not** read here — the per-batch merge fetches
+ /// only the coverage ranks it needs (see [`FragmentReader::merge_overlays`]),
+ /// so a `take` of a few rows no longer reads a whole overlay column.
+ ///
+ /// For each projected (top-level) field, the fragment's overlays are walked
+ /// newest-first; an overlay contributes if its `data_file.fields` includes
+ /// the field. Overlays on nested (non-top-level) fields are not yet supported
+ /// and are simply not matched here.
+ async fn load_overlay_plan(
+ &self,
+ projection: &Schema,
+ read_config: &FragReadConfig,
+ ) -> Result> {
+ let order = overlay_indices_newest_first(&self.metadata.overlays);
+
+ // Open each contributing overlay file once, concurrently. The reader is
+ // shared (via `Arc`) by every field that file covers.
+ //
+ // TODO(overlay perf): these reads use the default reader priority. Once
+ // we benchmark take/scan over overlays, decide whether overlay value
+ // reads should inherit `read_config.reader_priority` (or get a dedicated
+ // priority) so they schedule alongside the base reads.
+ let opened: Vec>> =
+ futures::future::try_join_all(order.iter().map(|&overlay_idx| async move {
+ let overlay = &self.metadata.overlays[overlay_idx];
+ let covered: Vec = projection
+ .fields
+ .iter()
+ .filter(|f| overlay.data_file.fields.contains(&f.id))
+ .cloned()
+ .collect();
+ if covered.is_empty() {
+ return Ok::<_, Error>(None);
+ }
+ let schema = Schema {
+ fields: covered,
+ metadata: Default::default(),
+ };
+ Ok(self
+ .open_reader(&overlay.data_file, Some(&schema), read_config)
+ .await?
+ .map(Arc::from))
+ }))
+ .await?;
+
+ let mut plans = Vec::new();
+ for field in &projection.fields {
+ let mut overlays_newest_first = Vec::new();
+ for (slot, &overlay_idx) in order.iter().enumerate() {
+ let overlay = &self.metadata.overlays[overlay_idx];
+ let Some(field_pos) = overlay
+ .data_file
+ .fields
+ .iter()
+ .position(|&id| id == field.id)
+ else {
+ continue;
+ };
+ let Some(reader) = &opened[slot] else {
+ continue;
+ };
+ overlays_newest_first.push(LoadedFieldOverlay {
+ coverage: overlay.coverage_for_field(field_pos)?,
+ reader: reader.clone(),
+ field_projection: Arc::new(Schema {
+ fields: vec![field.clone()],
+ metadata: Default::default(),
+ }),
+ });
+ }
+ if !overlays_newest_first.is_empty() {
+ plans.push(FieldOverlayPlan {
+ field_name: field.name.clone(),
+ overlays_newest_first,
+ });
+ }
+ }
+ Ok(plans)
+ }
+
/// Count the rows in this fragment.
pub async fn count_rows(&self, filter: Option) -> Result {
match filter {
@@ -1989,6 +2095,106 @@ impl From for Fragment {
}
}
+/// One overlay's contribution to one projected field: which physical offsets it
+/// covers, and an opened (but unread) reader over the overlay file from which the
+/// field's value column is fetched by coverage rank at merge time.
+#[derive(Debug, Clone)]
+struct LoadedFieldOverlay {
+ /// Physical offsets this overlay covers for the field.
+ coverage: Arc,
+ /// Reader over the overlay data file, projected to this field; shared across
+ /// the fields that the same file covers.
+ reader: Arc,
+ /// Single-field projection used when fetching the value column.
+ field_projection: Arc,
+}
+
+/// The overlays that apply to a single projected field, ordered newest-first.
+/// `field_name` is the top-level read-batch column name the plan applies to.
+#[derive(Debug, Clone)]
+struct FieldOverlayPlan {
+ field_name: String,
+ overlays_newest_first: Vec,
+}
+
+/// Resolve overlays for one base batch: route each projected field against the
+/// batch's physical `offsets`, fetch only the coverage ranks the batch touches
+/// (concurrently with the base read), and assemble the merged columns. Fields
+/// with no plan, and the row-id/row-address system columns, pass through.
+async fn merge_overlay_batch(
+ base: ReadBatchFut,
+ offsets: &[u32],
+ plans: &[FieldOverlayPlan],
+) -> Result {
+ let field_work = futures::future::try_join_all(plans.iter().map(|plan| async move {
+ let coverages: Vec<&RoaringBitmap> = plan
+ .overlays_newest_first
+ .iter()
+ .map(|overlay| overlay.coverage.as_ref())
+ .collect();
+ let routing = route_overlays(offsets, &coverages);
+ if routing.all_fall_through() {
+ return Ok::<_, Error>((plan.field_name.as_str(), None));
+ }
+ let fetched = futures::future::try_join_all(
+ plan.overlays_newest_first
+ .iter()
+ .zip(routing.needed_ranks())
+ .map(|(overlay, ranks)| {
+ fetch_overlay_ranks(
+ overlay.reader.as_ref(),
+ overlay.field_projection.clone(),
+ ranks,
+ )
+ }),
+ )
+ .await?;
+ Ok((plan.field_name.as_str(), Some((routing, fetched))))
+ }));
+
+ // The base read and every overlay value read proceed concurrently.
+ let (batch, resolved) = futures::future::try_join(base, field_work).await?;
+
+ let schema = batch.schema();
+ let mut columns = batch.columns().to_vec();
+ for (field_name, work) in resolved {
+ let Some((routing, fetched)) = work else {
+ continue;
+ };
+ let Some(idx) = schema.index_of(field_name).ok() else {
+ // The plan's field is not in this batch's projection; skip it.
+ continue;
+ };
+ columns[idx] = assemble_overlay_column(&columns[idx], &routing, &fetched)?;
+ }
+ Ok(RecordBatch::try_new(schema, columns)?)
+}
+
+/// Fetch one overlay's value column at the given coverage `ranks` (sorted and
+/// unique). Returns a column of `ranks.len()` values aligned with `ranks`; an
+/// empty `ranks` reads nothing and yields an empty column.
+async fn fetch_overlay_ranks(
+ reader: &dyn GenericFileReader,
+ projection: Arc,
+ ranks: &[u32],
+) -> Result {
+ if ranks.is_empty() {
+ return Ok(arrow_array::new_empty_array(
+ &projection.fields[0].data_type(),
+ ));
+ }
+ let mut tasks = reader
+ .take_all_tasks(ranks, ranks.len() as u32, projection, None)
+ .await?;
+ let mut chunks: Vec = Vec::new();
+ while let Some(task) = tasks.next().await {
+ let batch = task.task.await?;
+ chunks.push(batch.column(0).clone());
+ }
+ let chunk_refs: Vec<&dyn arrow_array::Array> = chunks.iter().map(|a| a.as_ref()).collect();
+ Ok(arrow_select::concat::concat(&chunk_refs)?)
+}
+
/// [`FragmentReader`] is an abstract reader for a [`FileFragment`].
///
/// It opens the data files that contains the columns of the projection schema, and
@@ -2042,6 +2248,11 @@ pub struct FragmentReader {
// total number of physical rows in the fragment (all rows, ignoring deletions)
num_physical_rows: usize,
+
+ /// Overlay value columns for the projected fields, loaded newest-first.
+ /// Empty when the fragment has no data overlay files. Merged into base
+ /// batches (by physical offset) before deletion filtering on every read.
+ overlay_plans: Arc>,
}
// Custom clone impl needed because it is not easy to clone Box
@@ -2070,6 +2281,7 @@ impl Clone for FragmentReader {
created_at_sequence: self.created_at_sequence.clone(),
num_rows: self.num_rows,
num_physical_rows: self.num_physical_rows,
+ overlay_plans: self.overlay_plans.clone(),
}
}
}
@@ -2137,6 +2349,7 @@ impl FragmentReader {
created_at_sequence: None,
num_rows,
num_physical_rows,
+ overlay_plans: Arc::new(Vec::new()),
})
}
@@ -2394,6 +2607,50 @@ impl FragmentReader {
Ok(result.project_by_schema(&output_schema)?)
}
+ /// Merge data overlay values into a stream of base batches.
+ ///
+ /// Runs on physical rows in read order, *before* deletion filtering, so each
+ /// row can be addressed by its physical offset (from `params`) and deletions
+ /// take precedence naturally (an overlay value for a deleted row is dropped
+ /// with the row downstream). A no-op when the fragment has no overlays.
+ ///
+ /// Each batch's physical offsets are known from `params` and the task's
+ /// `num_rows` without reading any data, so the overlay value reads (only the
+ /// coverage ranks the batch actually touches) are issued concurrently with
+ /// the base read rather than after it.
+ fn merge_overlays(
+ &self,
+ merged: ReadBatchTaskStream,
+ params: &ReadBatchParams,
+ total_num_rows: u32,
+ ) -> ReadBatchTaskStream {
+ if self.overlay_plans.is_empty() {
+ return merged;
+ }
+ let offsets: Arc> =
+ Arc::new(params.to_offsets_total(total_num_rows).values().to_vec());
+ let plans = self.overlay_plans.clone();
+ let mut row_start = 0usize;
+ merged
+ .map(move |task| {
+ let num_rows = task.num_rows;
+ let start = row_start;
+ row_start += num_rows as usize;
+ let offsets = offsets.clone();
+ let plans = plans.clone();
+ let inner = task.task;
+ ReadBatchTask {
+ num_rows,
+ task: async move {
+ let slice = &offsets[start..start + num_rows as usize];
+ merge_overlay_batch(inner, slice, &plans).await
+ }
+ .boxed(),
+ }
+ })
+ .boxed()
+ }
+
async fn new_read_impl<'a, F>(
&'a self,
params: ReadBatchParams,
@@ -2461,6 +2718,8 @@ impl FragmentReader {
lance_table::utils::stream::merge_streams(read_streams)
};
+ let merged = self.merge_overlays(merged, ¶ms, total_num_rows);
+
// Add the row id column (if needed) and delete rows (if a deletion
// vector is present).
let config = RowIdAndDeletesConfig {
@@ -2631,6 +2890,9 @@ impl FragmentReader {
lance_table::utils::stream::merge_streams(read_streams)
};
+ let params = ReadBatchParams::Ranges(ranges);
+ let merged_stream = self.merge_overlays(merged_stream, ¶ms, total_num_rows);
+
// Add the row id column (if needed) and delete rows (if a deletion
// vector is present).
let config = RowIdAndDeletesConfig {
@@ -2643,7 +2905,7 @@ impl FragmentReader {
with_row_created_at_version: self.with_row_created_at_version,
last_updated_at_sequence: self.last_updated_at_sequence.clone(),
created_at_sequence: self.created_at_sequence.clone(),
- params: ReadBatchParams::Ranges(ranges),
+ params,
total_num_rows,
};
let output_schema = Arc::new(self.output_schema.clone());
@@ -2839,6 +3101,656 @@ mod tests {
Dataset::open(test_uri).await.unwrap()
}
+ /// End-to-end tests for reading data overlay files (OSS-1324): overlays are
+ /// written, committed via the `DataOverlay` transaction, and then resolved on
+ /// the `take` and scan read paths.
+ mod overlay_read {
+ use std::sync::Arc;
+
+ use arrow_array::{
+ Array, ArrayRef, Int32Array, RecordBatch, RecordBatchIterator, UInt64Array,
+ };
+ use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema};
+ use lance_core::datatypes::Schema;
+ use lance_file::version::LanceFileVersion;
+ use lance_file::writer::{FileWriter, FileWriterOptions};
+ use lance_io::utils::CachedFileSize;
+ use lance_table::format::{DataFile, DataOverlayFile, OverlayCoverage};
+ use object_store::path::Path;
+ use roaring::RoaringBitmap;
+ use rstest::rstest;
+
+ use crate::dataset::transaction::{DataOverlayGroup, Operation};
+ use crate::dataset::{Dataset, WriteDestination, WriteParams};
+
+ fn bitmap(offsets: impl IntoIterator- ) -> RoaringBitmap {
+ RoaringBitmap::from_iter(offsets)
+ }
+
+ fn i32_array(values: impl IntoIterator
- >) -> ArrayRef {
+ Arc::new(Int32Array::from_iter(values))
+ }
+
+ /// Two-fragment Int32 dataset: `id` (field 0) = 0..12 and `val` (field 1)
+ /// = id * 10, written 6 rows per file (fragments 0 and 1).
+ ///
+ /// Uses an in-memory store so the test can write overlay files with a
+ /// store-relative `data/.lance` path and commit against the returned
+ /// dataset directly.
+ async fn create_base_dataset(version: LanceFileVersion) -> Dataset {
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, true),
+ ArrowField::new("val", DataType::Int32, true),
+ ]));
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..12)),
+ Arc::new(Int32Array::from_iter_values((0..12).map(|v| v * 10))),
+ ],
+ )
+ .unwrap();
+ let write_params = WriteParams {
+ max_rows_per_file: 6,
+ max_rows_per_group: 6,
+ data_storage_version: Some(version),
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
+ Dataset::write(reader, "memory://", Some(write_params))
+ .await
+ .unwrap()
+ }
+
+ /// Write an overlay file covering `fields` (dataset field ids) of
+ /// `fragment_id` with the given coverage and per-field value columns, then
+ /// commit it as a `DataOverlay` transaction. `name` makes the file unique.
+ #[allow(clippy::too_many_arguments)]
+ async fn commit_overlay(
+ dataset: Dataset,
+ name: &str,
+ fragment_id: u64,
+ fields: &[i32],
+ coverage: OverlayCoverage,
+ columns: Vec,
+ version: LanceFileVersion,
+ ) -> Dataset {
+ let read_version = dataset.version().version;
+ let overlay_schema = dataset.schema().project_by_ids(fields, true);
+
+ let filename = format!("{name}.lance");
+ let path = Path::from(format!("data/{filename}"));
+ let obj_writer = dataset.object_store.create(&path).await.unwrap();
+ let mut writer = FileWriter::try_new(
+ obj_writer,
+ overlay_schema,
+ FileWriterOptions {
+ format_version: Some(version),
+ ..Default::default()
+ },
+ )
+ .unwrap();
+ let (major, minor) = writer.version().to_numbers();
+ for (column_index, array) in columns.into_iter().enumerate() {
+ writer.write_column(column_index, array).await.unwrap();
+ }
+ let summary = writer.finish().await.unwrap();
+
+ let mut data_file = DataFile::new_unstarted(filename, major, minor);
+ data_file.fields = writer
+ .field_id_to_column_indices()
+ .iter()
+ .map(|(field_id, _)| *field_id as i32)
+ .collect::>()
+ .into();
+ data_file.column_indices = writer
+ .field_id_to_column_indices()
+ .iter()
+ .map(|(_, column_index)| *column_index as i32)
+ .collect::>()
+ .into();
+ data_file.file_size_bytes = CachedFileSize::new(summary.size_bytes);
+
+ let overlay = DataOverlayFile {
+ data_file,
+ coverage,
+ committed_version: 0,
+ };
+ Dataset::commit(
+ WriteDestination::Dataset(Arc::new(dataset)),
+ Operation::DataOverlay {
+ groups: vec![DataOverlayGroup {
+ fragment_id,
+ overlays: vec![overlay],
+ }],
+ },
+ Some(read_version),
+ None,
+ None,
+ Arc::new(Default::default()),
+ false,
+ )
+ .await
+ .unwrap()
+ }
+
+ fn full_schema(dataset: &Dataset) -> Schema {
+ dataset.schema().clone()
+ }
+
+ fn col(batch: &RecordBatch, name: &str) -> Int32Array {
+ let idx = batch.schema().index_of(name).unwrap();
+ batch
+ .column(idx)
+ .as_any()
+ .downcast_ref::()
+ .unwrap()
+ .clone()
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_covered_and_uncovered(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ // Overlay fragment 0's `val` at physical offsets {1, 4}.
+ let dataset = commit_overlay(
+ dataset,
+ "ov",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1, 4])),
+ vec![i32_array([Some(111), Some(444)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag
+ .take(&[0, 1, 2, 4], &full_schema(&dataset))
+ .await
+ .unwrap();
+ // Offsets 1 and 4 take overlay values; 0 and 2 fall through to base.
+ assert_eq!(col(&batch, "val").values(), &[0, 111, 20, 444]);
+ // The unrelated `id` column is untouched.
+ assert_eq!(col(&batch, "id").values(), &[0, 1, 2, 4]);
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_newest_overlay_wins(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "older",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1, 4])),
+ vec![i32_array([Some(111), Some(444)])],
+ version,
+ )
+ .await;
+ // A newer overlay (later commit -> higher committed_version) re-covers
+ // offset 1.
+ let dataset = commit_overlay(
+ dataset,
+ "newer",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1])),
+ vec![i32_array([Some(999)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[1, 4], &full_schema(&dataset)).await.unwrap();
+ // Offset 1 -> newest overlay (999); offset 4 -> only older covers it.
+ assert_eq!(col(&batch, "val").values(), &[999, 444]);
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_per_field_coverage(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ // Sparse overlay: `id` covers {2}, `val` covers {2, 3} — different
+ // offset sets and therefore unequal-length value columns.
+ let dataset = commit_overlay(
+ dataset,
+ "sparse",
+ 0,
+ &[0, 1],
+ OverlayCoverage::sparse(vec![bitmap([2]), bitmap([2, 3])]),
+ vec![i32_array([Some(777)]), i32_array([Some(220), Some(330)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[2, 3], &full_schema(&dataset)).await.unwrap();
+ // id: offset 2 covered (777), offset 3 falls through (3).
+ assert_eq!(col(&batch, "id").values(), &[777, 3]);
+ // val: both offsets covered (220, 330).
+ assert_eq!(col(&batch, "val").values(), &[220, 330]);
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_null_override(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "nullov",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([None])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[0, 1], &full_schema(&dataset)).await.unwrap();
+ let val = col(&batch, "val");
+ // Offset 0 is covered with a NULL value -> resolves to NULL; offset 1
+ // falls through to the base value.
+ assert!(val.is_null(0));
+ assert_eq!(val.value(1), 10);
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_overlay_on_deleted_row_is_inert(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let mut dataset = create_base_dataset(version).await;
+ // Delete global row 1 (fragment 0, physical offset 1).
+ dataset.delete("id = 1").await.unwrap();
+ // Overlay covers the deleted offset 1 and the live offset 4.
+ let dataset = commit_overlay(
+ dataset,
+ "delov",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1, 4])),
+ vec![i32_array([Some(111), Some(444)])],
+ version,
+ )
+ .await;
+
+ // Scan fragment 0: row 1 is gone, and offset 4's overlay value survives
+ // even though the deletion shifts logical positions — coverage is keyed
+ // by physical offset.
+ let frag = dataset.get_fragment(0).unwrap();
+ let mut scanner = frag.scan();
+ let batch = scanner
+ .project(&["id", "val"])
+ .unwrap()
+ .try_into_batch()
+ .await
+ .unwrap();
+ assert_eq!(col(&batch, "id").values(), &[0, 2, 3, 4, 5]);
+ assert_eq!(col(&batch, "val").values(), &[0, 20, 30, 444, 50]);
+ }
+
+ #[rstest]
+ #[tokio::test]
+ async fn test_scan_multi_fragment_overlays(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ // Overlay fragment 0 at offset 0 and fragment 1 at offset 0 (global
+ // row 6). Each fragment's coverage is independent.
+ let dataset = commit_overlay(
+ dataset,
+ "frag0",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([Some(1000)])],
+ version,
+ )
+ .await;
+ let dataset = commit_overlay(
+ dataset,
+ "frag1",
+ 1,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([Some(6000)])],
+ version,
+ )
+ .await;
+
+ let batch = dataset
+ .scan()
+ .project(&["id", "val"])
+ .unwrap()
+ .try_into_batch()
+ .await
+ .unwrap();
+ assert_eq!(batch.num_rows(), 12);
+ let expected: Vec = (0..12)
+ .map(|i| match i {
+ 0 => 1000,
+ 6 => 6000,
+ other => other * 10,
+ })
+ .collect();
+ assert_eq!(col(&batch, "val").values(), &expected);
+ }
+
+ /// A `take` of a few rows must read only the overlay value-column ranks
+ /// those rows touch — not the whole column. Uses v2.1 (which slices pages
+ /// on read) and an incompressible, all-covering overlay, so reading the
+ /// full column would be far more bytes than reading a couple of ranks.
+ /// This is the regression guard for the lazy, rank-pushdown overlay read.
+ #[tokio::test]
+ async fn test_take_reads_only_needed_overlay_ranks() {
+ let version = LanceFileVersion::V2_1;
+ const N: usize = 100_000;
+
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, true),
+ ArrowField::new("val", DataType::Int32, true),
+ ]));
+ let base = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..N as i32)),
+ Arc::new(Int32Array::from_iter_values((0..N as i32).map(|v| v * 10))),
+ ],
+ )
+ .unwrap();
+ let write_params = WriteParams {
+ max_rows_per_file: N,
+ max_rows_per_group: N,
+ data_storage_version: Some(version),
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(base)], schema.clone());
+ let dataset = Dataset::write(reader, "memory://", Some(write_params))
+ .await
+ .unwrap();
+
+ // Overlay `val` over ALL N offsets with incompressible values, so the
+ // value column is ~N*4 bytes on disk.
+ let values: Vec = (0..N as u64)
+ .map(|i| {
+ let mut x = i;
+ x ^= x >> 33;
+ x = x.wrapping_mul(0xff51_afd7_ed55_8ccd);
+ x ^= x >> 33;
+ x as i32
+ })
+ .collect();
+ let dataset = commit_overlay(
+ dataset,
+ "big",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap(0..N as u32)),
+ vec![Arc::new(Int32Array::from(values.clone())) as ArrayRef],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let val_only = dataset.schema().project_by_ids(&[1], true);
+
+ // Measure only the reads that resolve the take.
+ dataset.object_store.io_stats_incremental();
+ let batch = frag.take(&[0, 1], &val_only).await.unwrap();
+ let io = dataset.object_store.io_stats_incremental();
+
+ // The overlay's `val` column alone is N*4 bytes; resolving two adjacent
+ // offsets must read only a small fraction of it.
+ let full_column_bytes = (N * std::mem::size_of::()) as u64;
+ assert!(
+ io.read_bytes > 0 && io.read_bytes < full_column_bytes / 4,
+ "take read {} bytes; expected far less than the {}-byte overlay \
+ column (a take must not read the whole value column)",
+ io.read_bytes,
+ full_column_bytes,
+ );
+
+ // ...and it still resolves correctly.
+ let val = col(&batch, "val");
+ assert_eq!(val.value(0), values[0]);
+ assert_eq!(val.value(1), values[1]);
+ }
+
+ /// The overlay merge runs before `wrap_with_row_id_and_delete`, so the
+ /// `_rowid` system column must coexist with overlay-resolved data columns:
+ /// the row ids are unaffected by the merge and the overlay value still wins.
+ #[rstest]
+ #[tokio::test]
+ async fn test_scan_with_row_id_alongside_overlay(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "rowidov",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([Some(1000)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag
+ .scan()
+ .with_row_id()
+ .project(&["id", "val"])
+ .unwrap()
+ .try_into_batch()
+ .await
+ .unwrap();
+ // Overlay value resolves...
+ assert_eq!(col(&batch, "val").values()[0], 1000);
+ assert_eq!(&col(&batch, "val").values()[1..], &[10, 20, 30, 40, 50]);
+ // ...and the row ids for fragment 0 are the untouched physical offsets.
+ let row_ids = batch
+ .column(batch.schema().index_of("_rowid").unwrap())
+ .as_any()
+ .downcast_ref::()
+ .unwrap();
+ assert_eq!(row_ids.values(), &[0, 1, 2, 3, 4, 5]);
+ }
+
+ /// When the newest overlay covers every requested offset, an older overlay
+ /// in the same plan is routed zero ranks and its value column must not be
+ /// read (the empty-ranks branch of `fetch_overlay_ranks`). The result still
+ /// resolves to the newest overlay.
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_older_overlay_contributes_no_ranks(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ // Older covers {1, 4}; newer re-covers {1}. A take of only offset 1
+ // routes entirely to the newer overlay, leaving the older one with no
+ // ranks to fetch even though it is part of the field's plan.
+ let dataset = commit_overlay(
+ dataset,
+ "older",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1, 4])),
+ vec![i32_array([Some(111), Some(444)])],
+ version,
+ )
+ .await;
+ let dataset = commit_overlay(
+ dataset,
+ "newer",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1])),
+ vec![i32_array([Some(999)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[1], &full_schema(&dataset)).await.unwrap();
+ assert_eq!(col(&batch, "val").values(), &[999]);
+ }
+
+ /// A newest overlay whose value is NULL must shadow an older overlay's
+ /// non-null value at the same offset — the merge resolves to NULL, it does
+ /// not fall back to the older overlay.
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_newest_null_shadows_older(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "older",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1])),
+ vec![i32_array([Some(111)])],
+ version,
+ )
+ .await;
+ let dataset = commit_overlay(
+ dataset,
+ "newer_null",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1])),
+ vec![i32_array([None])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[1], &full_schema(&dataset)).await.unwrap();
+ let val = col(&batch, "val");
+ assert!(val.is_null(0), "newest NULL must win over older 111");
+ }
+
+ /// Newest-wins is resolved independently per field across multiple sparse
+ /// overlays: for the same offset, `id` can resolve to one overlay while
+ /// `val` resolves to the other, depending on which overlay newly covers
+ /// that field at that offset.
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_multi_sparse_per_field_newest_wins(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ // Older: id covers {3}, val covers {2}.
+ let dataset = commit_overlay(
+ dataset,
+ "older",
+ 0,
+ &[0, 1],
+ OverlayCoverage::sparse(vec![bitmap([3]), bitmap([2])]),
+ vec![i32_array([Some(7773)]), i32_array([Some(2772)])],
+ version,
+ )
+ .await;
+ // Newer: id covers {2}, val covers {3} — the mirror image.
+ let dataset = commit_overlay(
+ dataset,
+ "newer",
+ 0,
+ &[0, 1],
+ OverlayCoverage::sparse(vec![bitmap([2]), bitmap([3])]),
+ vec![i32_array([Some(9992)]), i32_array([Some(9993)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ let batch = frag.take(&[2, 3], &full_schema(&dataset)).await.unwrap();
+ // id: offset 2 -> newer (9992), offset 3 -> older (7773).
+ assert_eq!(col(&batch, "id").values(), &[9992, 7773]);
+ // val: offset 2 -> older (2772), offset 3 -> newer (9993).
+ assert_eq!(col(&batch, "val").values(), &[2772, 9993]);
+ }
+
+ /// A fragment with an overlay plan, but a take that touches only uncovered
+ /// offsets, must fall entirely through to the base values (the
+ /// `routing.all_fall_through()` early-return with a plan present).
+ #[rstest]
+ #[tokio::test]
+ async fn test_take_plan_present_all_offsets_uncovered(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "ov",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([1, 4])),
+ vec![i32_array([Some(111), Some(444)])],
+ version,
+ )
+ .await;
+
+ let frag = dataset.get_fragment(0).unwrap();
+ // None of {0, 2, 5} are covered: the plan exists but contributes nothing.
+ let batch = frag.take(&[0, 2, 5], &full_schema(&dataset)).await.unwrap();
+ assert_eq!(col(&batch, "val").values(), &[0, 20, 50]);
+ assert_eq!(col(&batch, "id").values(), &[0, 2, 5]);
+ }
+
+ /// A dataset-level `take` spanning multiple fragments, each with its own
+ /// overlay, routes every global row index to the right fragment's overlay.
+ #[rstest]
+ #[tokio::test]
+ async fn test_dataset_take_multi_fragment_overlays(
+ #[values(LanceFileVersion::V2_0, LanceFileVersion::V2_1)] version: LanceFileVersion,
+ ) {
+ let dataset = create_base_dataset(version).await;
+ let dataset = commit_overlay(
+ dataset,
+ "frag0",
+ 0,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([Some(1000)])],
+ version,
+ )
+ .await;
+ let dataset = commit_overlay(
+ dataset,
+ "frag1",
+ 1,
+ &[1],
+ OverlayCoverage::dense(bitmap([0])),
+ vec![i32_array([Some(6000)])],
+ version,
+ )
+ .await;
+
+ // Global rows 0 and 6 are the overlaid offset-0 rows of fragments 0 and
+ // 1; rows 1 and 7 fall through to base.
+ let batch = dataset
+ .take(&[0, 1, 6, 7], full_schema(&dataset))
+ .await
+ .unwrap();
+ assert_eq!(col(&batch, "id").values(), &[0, 1, 6, 7]);
+ assert_eq!(col(&batch, "val").values(), &[1000, 10, 6000, 70]);
+ }
+ }
+
#[rstest]
#[tokio::test]
async fn test_fragment_scan(
diff --git a/rust/lance/src/dataset/optimize.rs b/rust/lance/src/dataset/optimize.rs
index 87dda8e7e57..25168b54780 100644
--- a/rust/lance/src/dataset/optimize.rs
+++ b/rust/lance/src/dataset/optimize.rs
@@ -2125,6 +2125,7 @@ mod tests {
let fragment = Fragment {
id: 0,
files: vec![],
+ overlays: vec![],
deletion_file: None,
row_id_meta: None,
physical_rows: Some(0),
diff --git a/rust/lance/src/dataset/overlay.rs b/rust/lance/src/dataset/overlay.rs
new file mode 100644
index 00000000000..0da44fa8c91
--- /dev/null
+++ b/rust/lance/src/dataset/overlay.rs
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright The Lance Authors
+
+//! Resolution of data overlay files on read.
+//!
+//! An overlay supplies new values for a subset of `(physical offset, field)`
+//! cells. To resolve a field's values for a set of physical row offsets, the
+//! overlays that cover that field are walked **newest to oldest**: the first
+//! overlay that covers an offset wins, and its value is taken at the offset's
+//! **rank** (the 0-based count of set bits below it) in the field's coverage
+//! bitmap. An offset that no overlay covers falls through to the base value.
+//!
+//! The offsets are supplied explicitly (one per base row), so a single code path
+//! serves both the scan (a contiguous physical range) and `take` (arbitrary
+//! physical offsets) read paths.
+//!
+//! Deletions take precedence over overlays, but that is handled downstream: the
+//! merge runs on physical rows *before* the deletion filter, so an overlay value
+//! for a deleted offset is computed and then dropped with the row — making it
+//! inert, exactly as the specification requires, with no special handling here.
+
+use std::collections::{BTreeSet, HashMap};
+
+use arrow_array::{Array, ArrayRef};
+use arrow_select::interleave::interleave;
+use lance_core::{Error, Result};
+use roaring::RoaringBitmap;
+
+use lance_table::format::DataOverlayFile;
+
+/// Order a fragment's overlays from newest to oldest for read resolution.
+///
+/// Precedence is by `committed_version` (higher is newer); ties are broken by
+/// position in the fragment's `overlays` list, where a later entry is newer.
+/// Returns indices into `overlays`.
+pub fn overlay_indices_newest_first(overlays: &[DataOverlayFile]) -> Vec {
+ let mut indices: Vec = (0..overlays.len()).collect();
+ indices.sort_by(|&a, &b| {
+ overlays[b]
+ .committed_version
+ .cmp(&overlays[a].committed_version)
+ .then(b.cmp(&a))
+ });
+ indices
+}
+
+/// The physical offsets within a fragment whose value for an indexed field may be
+/// stale relative to an index built at `index_version`, and so must be excluded
+/// from that index's results and re-evaluated against current values on the flat
+/// path.
+///
+/// The set is the union, over every overlay whose `committed_version` is newer
+/// than `index_version`, of that overlay's coverage **restricted to the indexed
+/// fields**. The restriction makes exclusion field-aware: an overlay that touches
+/// only non-indexed fields contributes nothing. An overlay whose
+/// `committed_version <= index_version` is already incorporated by the index and
+/// is ignored.
+pub fn overlay_exclusion_offsets(
+ overlays: &[DataOverlayFile],
+ indexed_field_ids: &[i32],
+ index_version: u64,
+) -> Result {
+ let mut excluded = RoaringBitmap::new();
+ for overlay in overlays {
+ if overlay.committed_version <= index_version {
+ continue;
+ }
+ for (field_pos, field_id) in overlay.data_file.fields.iter().enumerate() {
+ if indexed_field_ids.contains(field_id) {
+ excluded |= &*overlay.coverage_for_field(field_pos)?;
+ }
+ }
+ }
+ Ok(excluded)
+}
+
+/// Resolve a single field's values for the rows whose physical offsets are given
+/// by `offsets` (one per base row, in the same order as `base`), merging the
+/// overlays that cover the field (which must be supplied newest-first).
+///
+/// Produced by [`route_overlays`] from the coverage bitmaps alone — before any
+/// value column is read — so the caller can fetch only the ranks it actually
+/// needs (see [`OverlayRouting::needed_ranks`]) instead of the whole column, and
+/// then assemble the merged column with [`assemble_overlay_column`].
+pub struct OverlayRouting {
+ /// `interleave` source/position pairs, one per output row. Source `0` is the
+ /// base column (position = the row's index); source `k + 1` is overlay `k`'s
+ /// fetched values (position = the row's index within `needed_ranks[k]`).
+ indices: Vec<(usize, usize)>,
+ /// `needed_ranks[k]` is the sorted, deduplicated set of coverage ranks that
+ /// overlay `k` must supply for this batch — the indices to fetch from its
+ /// value column.
+ needed_ranks: Vec>,
+ /// Whether any row routes to an overlay at all (false ⇒ pure fall-through).
+ any_overlay: bool,
+}
+
+impl OverlayRouting {
+ /// The ranks each overlay (newest-first) must fetch from its value column.
+ pub fn needed_ranks(&self) -> &[Vec] {
+ &self.needed_ranks
+ }
+
+ /// True when no row is covered by any overlay, so the base column is the
+ /// answer unchanged and no value-column reads are needed.
+ pub fn all_fall_through(&self) -> bool {
+ !self.any_overlay
+ }
+}
+
+/// Decide, for each physical offset in `offsets`, which source supplies its
+/// value: the newest overlay whose coverage contains it (taken at the offset's
+/// 0-based rank in that coverage), or the base column if none covers it.
+///
+/// Reads only the coverage bitmaps (newest-first), so it can run before the
+/// value columns are fetched and tells the caller exactly which ranks to fetch.
+///
+/// A scan reads a contiguous physical range, so when `offsets` is contiguous
+/// ascending we take a bitmap-major fast path that visits only each coverage's
+/// in-range bits — `O(covered + K)` — instead of probing every offset against
+/// every coverage. `take` supplies arbitrary offsets and uses the general path.
+pub fn route_overlays(
+ offsets: &[u32],
+ coverages_newest_first: &[&RoaringBitmap],
+) -> OverlayRouting {
+ match contiguous_base(offsets) {
+ Some(base) => route_contiguous(base, offsets.len(), coverages_newest_first),
+ None => route_arbitrary(offsets, coverages_newest_first),
+ }
+}
+
+/// The starting offset if `offsets` is a contiguous ascending run
+/// `[base, base + 1, ...]`, else `None` (including when empty).
+fn contiguous_base(offsets: &[u32]) -> Option {
+ let base = *offsets.first()?;
+ offsets
+ .iter()
+ .enumerate()
+ .all(|(i, &offset)| offset as u64 == base as u64 + i as u64)
+ .then_some(base)
+}
+
+/// Fast path for a contiguous batch: offset `o` is output row `o - base`, so a
+/// coverage's bits route to rows directly without per-offset probing.
+///
+/// For each coverage we intersect with the batch's offset range, which is a
+/// container-level operation that drops a non-overlapping batch (e.g. a scan
+/// batch past a contiguous coverage's bits) in `O(containers)` without touching
+/// individual cells. The in-range bits then carry **consecutive** coverage ranks
+/// starting at the count of bits below `base` (one `rank` lookup) — no bits lie
+/// between them by construction — so ranks need no running count. Coverages are
+/// processed newest-first with a "first claim wins" guard for precedence.
+fn route_contiguous(
+ base: u32,
+ len: usize,
+ coverages_newest_first: &[&RoaringBitmap],
+) -> OverlayRouting {
+ let mut needed_ranks: Vec> = vec![Vec::new(); coverages_newest_first.len()];
+ let mut routed: Vec> = vec![None; len];
+ let range_end = (base as u64 + len as u64).min(u32::MAX as u64) as u32;
+ let mut batch_range = RoaringBitmap::new();
+ batch_range.insert_range(base..range_end);
+
+ for (k, coverage) in coverages_newest_first.iter().enumerate() {
+ let in_range = *coverage & &batch_range;
+ if in_range.is_empty() {
+ continue;
+ }
+ // 0-based rank of the first in-range cell: the coverage bits below `base`.
+ let base_rank = if base == 0 {
+ 0
+ } else {
+ coverage.rank(base - 1) as u32
+ };
+ for (i, offset) in in_range.iter().enumerate() {
+ let row = (offset - base) as usize;
+ if routed[row].is_none() {
+ routed[row] = Some((k, needed_ranks[k].len()));
+ needed_ranks[k].push(base_rank + i as u32);
+ }
+ }
+ }
+
+ let mut any_overlay = false;
+ let indices = routed
+ .into_iter()
+ .enumerate()
+ .map(|(i, routed)| match routed {
+ None => (0, i),
+ Some((k, pos)) => {
+ any_overlay = true;
+ (k + 1, pos)
+ }
+ })
+ .collect();
+
+ OverlayRouting {
+ indices,
+ needed_ranks,
+ any_overlay,
+ }
+}
+
+/// General path for arbitrary (e.g. `take`) offsets: probe each offset against
+/// the coverages newest-first. `take` batches are small, so the `O(N * K)`
+/// probing here is not a bottleneck.
+fn route_arbitrary(offsets: &[u32], coverages_newest_first: &[&RoaringBitmap]) -> OverlayRouting {
+ let mut rank_sets: Vec> = vec![BTreeSet::new(); coverages_newest_first.len()];
+ let mut raw: Vec> = Vec::with_capacity(offsets.len());
+ for &offset in offsets {
+ let mut routed = None;
+ for (k, coverage) in coverages_newest_first.iter().enumerate() {
+ if coverage.contains(offset) {
+ // 0-based rank: number of set bits strictly below `offset`.
+ let rank = coverage.rank(offset) as u32 - 1;
+ rank_sets[k].insert(rank);
+ routed = Some((k, rank));
+ break;
+ }
+ }
+ raw.push(routed);
+ }
+
+ let needed_ranks: Vec> = rank_sets
+ .iter()
+ .map(|ranks| ranks.iter().copied().collect())
+ .collect();
+ let rank_positions: Vec> = needed_ranks
+ .iter()
+ .map(|ranks| ranks.iter().enumerate().map(|(pos, &r)| (r, pos)).collect())
+ .collect();
+
+ let mut any_overlay = false;
+ let indices = raw
+ .into_iter()
+ .enumerate()
+ .map(|(i, routed)| match routed {
+ None => (0, i),
+ Some((k, rank)) => {
+ any_overlay = true;
+ (k + 1, rank_positions[k][&rank])
+ }
+ })
+ .collect();
+
+ OverlayRouting {
+ indices,
+ needed_ranks,
+ any_overlay,
+ }
+}
+
+/// Assemble the merged column from `base` and the per-overlay values fetched for
+/// the ranks [`route_overlays`] asked for.
+///
+/// `fetched_newest_first[k]` holds overlay `k`'s values for `routing`'s
+/// `needed_ranks[k]`, in that order. The result has the same length and data
+/// type as `base`; a covered offset whose overlay value is NULL resolves **to**
+/// NULL (distinct from a fall-through, which keeps its base value).
+pub fn assemble_overlay_column(
+ base: &ArrayRef,
+ routing: &OverlayRouting,
+ fetched_newest_first: &[ArrayRef],
+) -> Result {
+ if routing.all_fall_through() {
+ return Ok(base.clone());
+ }
+ if fetched_newest_first.len() != routing.needed_ranks.len() {
+ return Err(Error::invalid_input(format!(
+ "overlay assembly got {} value columns but routing expects {}",
+ fetched_newest_first.len(),
+ routing.needed_ranks.len()
+ )));
+ }
+ for (k, values) in fetched_newest_first.iter().enumerate() {
+ if values.len() != routing.needed_ranks[k].len() {
+ return Err(Error::invalid_input(format!(
+ "overlay value column {} has {} values but {} ranks were requested",
+ k,
+ values.len(),
+ routing.needed_ranks[k].len()
+ )));
+ }
+ }
+
+ let mut sources: Vec<&dyn Array> = Vec::with_capacity(fetched_newest_first.len() + 1);
+ sources.push(base.as_ref());
+ for values in fetched_newest_first {
+ sources.push(values.as_ref());
+ }
+ interleave(&sources, &routing.indices).map_err(Error::from)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::{Int32Array, StringArray, UInt32Array};
+ use std::sync::Arc;
+
+ fn i32_array(values: impl IntoIterator
- >) -> ArrayRef {
+ Arc::new(Int32Array::from_iter(values))
+ }
+
+ fn bitmap(offsets: impl IntoIterator
- ) -> RoaringBitmap {
+ RoaringBitmap::from_iter(offsets)
+ }
+
+ /// Physical offsets for a contiguous range `[start, start + len)`.
+ fn offsets(start: u32, len: usize) -> Vec {
+ (start..start + len as u32).collect()
+ }
+
+ /// Drive the production flow purely in memory: route against the coverage
+ /// bitmaps, then fetch just the requested ranks from each overlay's *full*
+ /// value column (exactly what the rank-pushdown `take` does on disk), then
+ /// assemble. `overlays_newest_first` holds each overlay's `(coverage, full
+ /// value column indexed by rank)`.
+ fn resolve(
+ base: &ArrayRef,
+ offsets: &[u32],
+ overlays_newest_first: &[(RoaringBitmap, ArrayRef)],
+ ) -> ArrayRef {
+ let coverages: Vec<&RoaringBitmap> = overlays_newest_first.iter().map(|(c, _)| c).collect();
+ let routing = route_overlays(offsets, &coverages);
+ let fetched: Vec = overlays_newest_first
+ .iter()
+ .zip(routing.needed_ranks())
+ .map(|((_, full), ranks)| {
+ let indices = UInt32Array::from(ranks.clone());
+ arrow_select::take::take(full.as_ref(), &indices, None).unwrap()
+ })
+ .collect();
+ assemble_overlay_column(base, &routing, &fetched).unwrap()
+ }
+
+ fn assert_i32_eq(actual: &ArrayRef, expected: impl IntoIterator
- >) {
+ let actual = actual.as_any().downcast_ref::().unwrap();
+ assert_eq!(actual, &Int32Array::from_iter(expected));
+ }
+
+ #[test]
+ fn test_no_overlays_returns_base() {
+ let base = i32_array([Some(1), Some(2), Some(3)]);
+ let resolved = resolve(&base, &offsets(0, 3), &[]);
+ assert_i32_eq(&resolved, [Some(1), Some(2), Some(3)]);
+ }
+
+ #[test]
+ fn test_single_overlay_rank_addressing() {
+ // Base ages [30, 25, 40, 22]; overlay sets offset 1 -> 26 (rank 0).
+ let base = i32_array([Some(30), Some(25), Some(40), Some(22)]);
+ let overlay = (bitmap([1]), i32_array([Some(26)]));
+ let resolved = resolve(&base, &offsets(0, 4), &[overlay]);
+ assert_i32_eq(&resolved, [Some(30), Some(26), Some(40), Some(22)]);
+ }
+
+ #[test]
+ fn test_rank_addressing_multiple_offsets() {
+ // Coverage {0, 2, 3} -> values at ranks 0,1,2.
+ let base = i32_array([Some(10), Some(11), Some(12), Some(13)]);
+ let overlay = (
+ bitmap([0, 2, 3]),
+ i32_array([Some(100), Some(120), Some(130)]),
+ );
+ let resolved = resolve(&base, &offsets(0, 4), &[overlay]);
+ assert_i32_eq(&resolved, [Some(100), Some(11), Some(120), Some(130)]);
+ }
+
+ #[test]
+ fn test_newest_overlay_wins() {
+ // Two overlays both cover offset 1; the newest (first in the slice) wins.
+ let base = i32_array([Some(0), Some(1), Some(2)]);
+ let newest = (bitmap([1]), i32_array([Some(999)]));
+ let older = (bitmap([1, 2]), i32_array([Some(111), Some(222)]));
+ let resolved = resolve(&base, &offsets(0, 3), &[newest, older]);
+ // offset 1 -> newest (999); offset 2 -> only older covers it (222).
+ assert_i32_eq(&resolved, [Some(0), Some(999), Some(222)]);
+ }
+
+ #[test]
+ fn test_null_override_vs_fall_through() {
+ // A covered offset with a NULL value overrides the cell to NULL; an
+ // absent offset falls through to the base.
+ let base = i32_array([Some(1), Some(2), Some(3)]);
+ let overlay = (bitmap([0]), i32_array([None]));
+ let resolved = resolve(&base, &offsets(0, 3), &[overlay]);
+ assert_i32_eq(&resolved, [None, Some(2), Some(3)]);
+ }
+
+ #[test]
+ fn test_physical_start_offset() {
+ // The batch covers physical rows [10, 13); the overlay covers offset 11.
+ let base = i32_array([Some(0), Some(0), Some(0)]);
+ let overlay = (bitmap([11]), i32_array([Some(7)]));
+ let resolved = resolve(&base, &offsets(10, 3), &[overlay]);
+ assert_i32_eq(&resolved, [Some(0), Some(7), Some(0)]);
+ }
+
+ #[test]
+ fn test_string_column_merge() {
+ let base: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c"]));
+ let overlay = (
+ bitmap([0, 2]),
+ Arc::new(StringArray::from(vec!["A", "C"])) as ArrayRef,
+ );
+ let resolved = resolve(&base, &offsets(0, 3), &[overlay]);
+ let expected: ArrayRef = Arc::new(StringArray::from(vec!["A", "b", "C"]));
+ assert_eq!(&resolved, &expected);
+ }
+
+ #[test]
+ fn test_non_contiguous_offsets() {
+ // `take` supplies arbitrary, non-contiguous physical offsets. The base
+ // rows correspond to offsets 5, 1, 8 (in that order); the overlay covers
+ // offsets {1, 8} with values at ranks 0, 1.
+ let base = i32_array([Some(50), Some(10), Some(80)]);
+ let overlay = (bitmap([1, 8]), i32_array([Some(11), Some(88)]));
+ let resolved = resolve(&base, &[5, 1, 8], &[overlay]);
+ // offset 5 uncovered -> base 50; offset 1 -> rank 0 (11); offset 8 -> rank 1 (88).
+ assert_i32_eq(&resolved, [Some(50), Some(11), Some(88)]);
+ }
+
+ #[test]
+ fn test_routing_dedups_repeated_ranks() {
+ // A `take` may request the same offset twice; both rows must route to the
+ // same rank, and that rank is fetched only once.
+ let coverage = bitmap([2, 5]);
+ let routing = route_overlays(&[5, 2, 5], &[&coverage]);
+ // Offset 5 is rank 1, offset 2 is rank 0: distinct ranks {0, 1}, sorted.
+ assert_eq!(routing.needed_ranks(), &[vec![0, 1]]);
+ let full = i32_array([Some(20), Some(50)]); // values at ranks 0, 1
+ let fetched = vec![
+ arrow_select::take::take(
+ full.as_ref(),
+ &UInt32Array::from(routing.needed_ranks()[0].clone()),
+ None,
+ )
+ .unwrap(),
+ ];
+ let base = i32_array([Some(0), Some(0), Some(0)]);
+ let resolved = assemble_overlay_column(&base, &routing, &fetched).unwrap();
+ assert_i32_eq(&resolved, [Some(50), Some(20), Some(50)]);
+ }
+
+ #[test]
+ fn test_assemble_value_count_mismatch_errors() {
+ let coverage = bitmap([0, 1]);
+ let routing = route_overlays(&[0, 1], &[&coverage]);
+ let base = i32_array([Some(1), Some(2)]);
+ // One value supplied for two requested ranks is a caller bug.
+ let fetched = vec![i32_array([Some(9)])];
+ assert!(assemble_overlay_column(&base, &routing, &fetched).is_err());
+ }
+
+ #[test]
+ fn test_contiguous_fast_path_matches_general() {
+ // The contiguous fast path must produce byte-for-byte identical routing
+ // to the general offset-major path for any contiguous batch. Fuzz a range
+ // of bases, lengths, overlay counts, and coverage densities — including
+ // bits outside the batch range — and compare both fields.
+ let mut state = 0x9e3779b97f4a7c15u64;
+ let mut next = || {
+ state = state
+ .wrapping_mul(6364136223846793005)
+ .wrapping_add(1442695040888963407);
+ (state >> 33) as u32
+ };
+ for _ in 0..500 {
+ let base = next() % 64;
+ let len = (next() % 48 + 1) as usize;
+ let num_overlays = (next() % 5) as usize;
+ let coverages: Vec = (0..num_overlays)
+ .map(|_| {
+ let density = next() % 101;
+ let mut b = RoaringBitmap::new();
+ for off in base.saturating_sub(3)..base + len as u32 + 3 {
+ if next() % 100 < density {
+ b.insert(off);
+ }
+ }
+ b
+ })
+ .collect();
+ let refs: Vec<&RoaringBitmap> = coverages.iter().collect();
+ let contiguous_offsets: Vec = (base..base + len as u32).collect();
+
+ let fast = route_contiguous(base, len, &refs);
+ let general = route_arbitrary(&contiguous_offsets, &refs);
+ assert_eq!(fast.indices, general.indices, "indices differ");
+ assert_eq!(
+ fast.needed_ranks, general.needed_ranks,
+ "needed_ranks differ"
+ );
+ assert_eq!(fast.any_overlay, general.any_overlay, "any_overlay differs");
+ }
+ }
+
+ #[test]
+ fn test_overlay_ordering_newest_first() {
+ use lance_table::format::{DataFile, OverlayCoverage};
+ let mk = |version: u64| DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", vec![1], None),
+ coverage: OverlayCoverage::dense(RoaringBitmap::new()),
+ committed_version: version,
+ };
+ // List order [v2, v5, v3]; newest-first should be v5(idx1), v3(idx2), v2(idx0).
+ let overlays = vec![mk(2), mk(5), mk(3)];
+ assert_eq!(overlay_indices_newest_first(&overlays), vec![1, 2, 0]);
+
+ // Equal versions: later list position is newer.
+ let overlays = vec![mk(4), mk(4)];
+ assert_eq!(overlay_indices_newest_first(&overlays), vec![1, 0]);
+ }
+
+ /// A dense overlay covering `offsets` for `field_ids`, committed at `version`.
+ fn dense_overlay(
+ field_ids: Vec,
+ offsets: impl IntoIterator
- ,
+ version: u64,
+ ) -> lance_table::format::DataOverlayFile {
+ use lance_table::format::{DataFile, OverlayCoverage};
+ lance_table::format::DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", field_ids, None),
+ coverage: OverlayCoverage::dense(bitmap(offsets)),
+ committed_version: version,
+ }
+ }
+
+ #[test]
+ fn test_exclusion_offsets_version_gate() {
+ // index built at version 5; only overlays committed > 5 are excluded.
+ let overlays = vec![
+ dense_overlay(vec![3], [0, 1], 4),
+ dense_overlay(vec![3], [2, 7], 6),
+ ];
+ let excluded = overlay_exclusion_offsets(&overlays, &[3], 5).unwrap();
+ assert_eq!(excluded, bitmap([2, 7]));
+ // An overlay exactly at the index version is already incorporated.
+ let overlays = vec![dense_overlay(vec![3], [9], 5)];
+ assert!(
+ overlay_exclusion_offsets(&overlays, &[3], 5)
+ .unwrap()
+ .is_empty()
+ );
+ }
+
+ #[test]
+ fn test_exclusion_offsets_is_field_aware() {
+ // An overlay touching only an unrelated field excludes nothing.
+ let overlays = vec![dense_overlay(vec![2], [0, 1, 2], 9)];
+ assert!(
+ overlay_exclusion_offsets(&overlays, &[3], 1)
+ .unwrap()
+ .is_empty()
+ );
+ // The union spans only the indexed fields the overlay actually carries.
+ let overlays = vec![dense_overlay(vec![2, 3], [4], 9)];
+ assert_eq!(
+ overlay_exclusion_offsets(&overlays, &[3], 1).unwrap(),
+ bitmap([4])
+ );
+ }
+
+ #[test]
+ fn test_exclusion_offsets_sparse_per_field() {
+ use lance_table::format::{DataFile, OverlayCoverage};
+ // Sparse overlay: field 2 covers {2,3}, field 4 covers {1}.
+ let overlay = DataOverlayFile {
+ data_file: DataFile::new_legacy_from_fields("o.lance", vec![2, 4], None),
+ coverage: OverlayCoverage::sparse(vec![bitmap([2, 3]), bitmap([1])]),
+ committed_version: 9,
+ };
+ let overlays = vec![overlay];
+ // Only the bitmap for the indexed field (4) contributes.
+ assert_eq!(
+ overlay_exclusion_offsets(&overlays, &[4], 1).unwrap(),
+ bitmap([1])
+ );
+ assert_eq!(
+ overlay_exclusion_offsets(&overlays, &[2], 1).unwrap(),
+ bitmap([2, 3])
+ );
+ }
+
+ #[test]
+ fn test_exclusion_offsets_unions_multiple_overlays() {
+ let overlays = vec![
+ dense_overlay(vec![3], [1], 6),
+ dense_overlay(vec![3], [4, 5], 7),
+ ];
+ assert_eq!(
+ overlay_exclusion_offsets(&overlays, &[3], 1).unwrap(),
+ bitmap([1, 4, 5])
+ );
+ }
+}
diff --git a/rust/lance/src/dataset/scanner.rs b/rust/lance/src/dataset/scanner.rs
index d4b58e4783f..ff1cc52f59d 100644
--- a/rust/lance/src/dataset/scanner.rs
+++ b/rust/lance/src/dataset/scanner.rs
@@ -12,12 +12,12 @@ use std::task::{Context, Poll};
use crate::index::DatasetIndexExt;
use arrow::array::AsArray;
-use arrow_array::{Array, Float32Array, Int64Array, RecordBatch};
+use arrow_array::{Array, Float32Array, Int64Array, RecordBatch, UInt64Array};
use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema, SchemaRef, SortOptions};
use arrow_select::concat::concat_batches;
use async_recursion::async_recursion;
use chrono::Utc;
-use datafusion::common::{DFSchema, JoinType, NullEquality, SchemaExt, exec_datafusion_err};
+use datafusion::common::{DFSchema, JoinType, NullEquality, exec_datafusion_err};
use datafusion::functions_aggregate;
use datafusion::logical_expr::{Expr, ScalarUDF, col, lit};
use datafusion::physical_expr::PhysicalSortExpr;
@@ -83,11 +83,12 @@ use tracing::{Span, info_span, instrument};
use uuid::Uuid;
use super::Dataset;
+use crate::dataset::overlay::overlay_exclusion_offsets;
use crate::dataset::row_offsets_to_row_addresses;
use crate::dataset::utils::SchemaAdapter;
use crate::index::DatasetIndexInternalExt;
use crate::index::scalar::inverted::{load_segment_details, load_segments};
-use crate::index::scalar_logical::scalar_index_fragment_bitmap;
+use crate::index::scalar_logical::{load_named_scalar_segments, scalar_index_fragment_bitmap};
use crate::index::vector::utils::{
default_distance_type_for, get_vector_dim, get_vector_type, validate_distance_type_for,
};
@@ -2897,6 +2898,23 @@ impl Scanner {
read_options = read_options.with_only_indexed_fragments();
}
+ // Mask data overlay files: a fragment with an overlay committed after an index it relies
+ // on touched an indexed field can no longer be trusted to that index. Drop such fragments
+ // from the index's covered set so they are re-evaluated on the flat path (OSS-1325).
+ if let Some(index_query) = filter_plan.index_query.as_ref() {
+ let candidate_frags = read_options
+ .fragments
+ .clone()
+ .unwrap_or_else(|| self.dataset.fragments().clone());
+ let stale_rows = self
+ .overlay_stale_index_rows(index_query, &candidate_frags)
+ .await?;
+ if !stale_rows.is_empty() {
+ let stale_frags: RoaringBitmap = stale_rows.keys().copied().collect();
+ read_options = read_options.with_overlay_stale_fragments(stale_frags);
+ }
+ }
+
let result_format = self.index_expr_result_format();
let index_input = filter_plan.index_query.clone().map(|index_query| {
Arc::new(ScalarIndexExec::new(
@@ -3277,6 +3295,10 @@ impl Scanner {
self.fragments_covered_by_fts_query(&query).await?,
)
.await?;
+ // TODO(OSS-1325): FTS does not yet mask data overlay files. A fragment with an overlay
+ // on an FTS-indexed field committed after the index was built may return stale hits. The
+ // fix requires identifying stale FTS segments (analogous to `overlay_stale_index_frags`)
+ // and routing their fragments to the flat-text fallback path.
let fts_exec = self
.plan_fts(&query, ¶ms, filter_plan, &prefilter_source)
.await?;
@@ -3509,30 +3531,62 @@ impl Scanner {
let unindexed_fragments = self
.retain_target_fragments(self.dataset.unindexed_fragments(&index.name).await?);
- // If all target fragments are unindexed, skip index entirely
- if unindexed_fragments.len() == target_fragments.len() {
+ // Fragments whose FTS index entries may be stale due to a newer data overlay.
+ // These are excluded from the indexed path and re-evaluated on the flat path.
+ let (stale_flat_frag_ids, fresh_segments) = self
+ .fts_stale_frags_and_fresh_segments(&column, &target_fragments)
+ .await?;
+
+ // Fragments that need flat evaluation: unindexed + stale (deduplicated).
+ let flat_fragments: Vec = {
+ let mut seen: std::collections::HashSet = std::collections::HashSet::new();
+ let mut frags = Vec::new();
+ for f in unindexed_fragments.iter().chain(
+ target_fragments
+ .iter()
+ .filter(|f| stale_flat_frag_ids.contains(f.id as u32)),
+ ) {
+ if seen.insert(f.id as u32) {
+ frags.push(f.clone());
+ }
+ }
+ frags
+ };
+
+ // If all target fragments need flat evaluation, skip the indexed path.
+ if flat_fragments.len() == target_fragments.len() {
if self.fast_search {
return Ok(Arc::new(EmptyExec::new(FTS_SCHEMA.clone())));
}
let flat_match_plan = self
- .plan_flat_match_query(unindexed_fragments, query, params, filter_plan)
+ .plan_flat_match_query(flat_fragments, query, params, filter_plan)
.await?;
return Ok(flat_match_plan);
}
- // Mixed case: use index + flat search for unindexed
- let match_plan: Arc = Arc::new(MatchQueryExec::new(
- self.dataset.clone(),
- query.clone(),
- params.clone(),
- prefilter_source.clone(),
- ));
+ // Build the indexed path. When overlays made some segments stale we use
+ // `new_with_segments` to restrict the search to fresh segments only.
+ let match_plan: Arc = match fresh_segments {
+ Some(segs) => Arc::new(MatchQueryExec::new_with_segments(
+ self.dataset.clone(),
+ query.clone(),
+ params.clone(),
+ prefilter_source.clone(),
+ segs,
+ )),
+ None => Arc::new(MatchQueryExec::new(
+ self.dataset.clone(),
+ query.clone(),
+ params.clone(),
+ prefilter_source.clone(),
+ )),
+ };
- if self.fast_search || unindexed_fragments.is_empty() {
+ if self.fast_search || flat_fragments.is_empty() {
(Some(match_plan), None)
} else {
let flat_match_plan = self
- .plan_flat_match_query(unindexed_fragments, query, params, filter_plan)
+ .plan_flat_match_query(flat_fragments, query, params, filter_plan)
.await?;
(Some(match_plan), Some(flat_match_plan))
}
@@ -3787,9 +3841,32 @@ impl Scanner {
"Refine factor cannot be zero".to_string(),
));
}
+ // Mask data overlay files: compute which row addresses within each segment have
+ // been updated by a newer overlay so their ANN entries may be stale (OSS-1325).
+ // These stale rows are blocked from ANN results via the prefilter and re-scored
+ // on the targeted flat path below — only the specific stale rows, not the whole
+ // fragment, so sparse overlays incur near-zero overhead.
+ let stale_rows = self.mask_overlay_stale_rows(&index_segments)?;
+ // Build a prefilter block mask for stale rows (empty = no-op fast path).
+ let overlay_block: Option = if stale_rows.is_empty() {
+ None
+ } else {
+ let mut tree_map = RowAddrTreeMap::new();
+ for (&frag_id, offsets) in &stale_rows {
+ tree_map.insert_bitmap(frag_id, offsets.clone());
+ }
+ Some(RowAddrMask::from_block(tree_map))
+ };
+
let ann_node = match vector_type {
- DataType::FixedSizeList(_, _) => self.ann(&q, &index_segments, filter_plan).await?,
- DataType::List(_) => self.multivec_ann(&q, &index_segments, filter_plan).await?,
+ DataType::FixedSizeList(_, _) => {
+ self.ann(&q, &index_segments, filter_plan, overlay_block.clone())
+ .await?
+ }
+ DataType::List(_) => {
+ self.multivec_ann(&q, &index_segments, filter_plan, overlay_block.clone())
+ .await?
+ }
_ => unreachable!(),
};
@@ -3807,7 +3884,14 @@ impl Scanner {
if !self.fast_search {
knn_node = self
- .knn_combined(&q, &index_name, &index_segments, knn_node, filter_plan)
+ .knn_combined(
+ &q,
+ &index_name,
+ &index_segments,
+ &stale_rows,
+ knn_node,
+ filter_plan,
+ )
.await?;
}
@@ -3942,6 +4026,7 @@ impl Scanner {
q: &Query,
index_name: &str,
indexed_segments: &[IndexMetadata],
+ stale_rows: &std::collections::HashMap,
mut knn_node: Arc,
filter_plan: &ExprFilterPlan,
) -> Result> {
@@ -3958,30 +4043,42 @@ impl Scanner {
self.dataset.unindexed_fragments(index_name).await?
};
- if !fallback_fragments.is_empty() {
- let q = q.clone();
- debug_assert!(q.metric_type.is_some());
+ let has_fallback = !fallback_fragments.is_empty();
+ let has_stale = !stale_rows.is_empty();
- // If the vector column is not present, we need to take the vector column, so
- // that the distance value is comparable with the flat search ones.
- if knn_node.schema().column_with_name(&q.column).is_none() {
- let vector_projection = self
- .dataset
- .empty_projection()
- .union_column(&q.column, OnMissing::Error)
- .unwrap();
- knn_node = self.take(knn_node, vector_projection)?;
- }
+ if !has_fallback && !has_stale {
+ return Ok(knn_node);
+ }
- let mut columns = vec![q.column.clone()];
- if let Some(expr) = filter_plan.full_expr.as_ref() {
- let filter_columns = Planner::column_names_in_expr(expr);
- columns.extend(filter_columns);
- }
+ let q = q.clone();
+ debug_assert!(q.metric_type.is_some());
+
+ // Ensure the vector column is present for distance computation.
+ if knn_node.schema().column_with_name(&q.column).is_none() {
+ let vector_projection = self
+ .dataset
+ .empty_projection()
+ .union_column(&q.column, OnMissing::Error)
+ .unwrap();
+ knn_node = self.take(knn_node, vector_projection)?;
+ }
+
+ let mut columns = vec![q.column.clone()];
+ if let Some(expr) = filter_plan.full_expr.as_ref() {
+ let filter_columns = Planner::column_names_in_expr(expr);
+ columns.extend(filter_columns);
+ }
+
+ // Collect flat-path plans; union order matches original (flat before ANN) so test snapshots
+ // and downstream plan analyses remain stable.
+ let mut flat_inputs: Vec> = Vec::new();
+
+ // Flat KNN for unindexed (new-data) fragments.
+ if has_fallback {
let vector_scan_projection = Arc::new(self.dataset.schema().project(&columns).unwrap());
- // Note: we could try and use the scalar indices here to reduce the scope of this scan but the
- // most common case is that fragments that are newer than the vector index are going to be newer
- // than the scalar indices anyways
+ // Note: we could try and use the scalar indices here to reduce the scope of this scan
+ // but the most common case is that fragments newer than the vector index are also
+ // newer than the scalar indices.
let mut scan_node = self.scan_fragments(
true,
false,
@@ -3990,41 +4087,67 @@ impl Scanner {
false,
vector_scan_projection,
Arc::new(fallback_fragments),
- // Can't pushdown limit/offset in an ANN search
None,
- // We are re-ordering anyways, so no need to get data in data
- // in a deterministic order.
false,
);
-
if let Some(expr) = filter_plan.full_expr.as_ref() {
- // If there is a prefilter we need to manually apply it to the new data
scan_node = Arc::new(LanceFilterExec::try_new(expr.clone(), scan_node)?);
}
- // first we do flat search on just the new data
- let topk_appended = self.flat_knn(scan_node, &q)?;
+ let topk_fallback = self.flat_knn(scan_node, &q)?;
+ let topk_fallback: Arc =
+ Arc::new(project(topk_fallback, knn_node.schema().as_ref())?);
+ flat_inputs.push(topk_fallback);
+ }
- // To do a union, we need to make the schemas match. Right now
- // knn_node: _distance, _rowid, vector
- // topk_appended: vector, , _rowid, _distance
- let topk_appended = project(topk_appended, knn_node.schema().as_ref())?;
- assert!(
- topk_appended
- .schema()
- .equivalent_names_and_types(&knn_node.schema())
- );
- // union
- let unioned = UnionExec::try_new(vec![Arc::new(topk_appended), knn_node])?;
- // Enforce only 1 partition.
- let unioned = RepartitionExec::try_new(
- unioned,
- datafusion::physical_plan::Partitioning::RoundRobinBatch(1),
+ // Flat KNN for stale rows only (row-level precision, OSS-1325).
+ // Only specific row addresses need re-scoring, not the whole fragment, so sparse overlays
+ // incur near-zero overhead.
+ if has_stale {
+ let stale_addrs: Vec = stale_rows
+ .iter()
+ .flat_map(|(&frag_id, offsets)| {
+ offsets
+ .iter()
+ .map(move |offset| ((frag_id as u64) << 32) | offset as u64)
+ })
+ .collect();
+ let batch = RecordBatch::try_new(
+ Arc::new(ArrowSchema::new(vec![ArrowField::new(
+ ROW_ID,
+ DataType::UInt64,
+ true,
+ )])),
+ vec![Arc::new(UInt64Array::from(stale_addrs))],
)?;
- // then we do a flat search on KNN(new data) + ANN(indexed data)
- return self.flat_knn(Arc::new(unioned), &q);
+ let stale_id_plan = Arc::new(OneShotExec::from_batch(batch));
+
+ // Fetch vector + filter columns for the stale rows.
+ let mut take_proj = self
+ .dataset
+ .empty_projection()
+ .union_column(&q.column, OnMissing::Error)?;
+ if let Some(expr) = filter_plan.full_expr.as_ref() {
+ let filter_columns = Planner::column_names_in_expr(expr);
+ take_proj = take_proj.union_columns(filter_columns, OnMissing::Error)?;
+ }
+ let mut stale_node = self.take(stale_id_plan, take_proj)?;
+ if let Some(expr) = filter_plan.full_expr.as_ref() {
+ stale_node = Arc::new(LanceFilterExec::try_new(expr.clone(), stale_node)?);
+ }
+ let topk_stale = self.flat_knn(stale_node, &q)?;
+ let topk_stale: Arc =
+ Arc::new(project(topk_stale, knn_node.schema().as_ref())?);
+ flat_inputs.push(topk_stale);
}
- Ok(knn_node)
+ // Union: flat paths first (matching original order), then ANN results.
+ flat_inputs.push(knn_node);
+ let unioned = UnionExec::try_new(flat_inputs)?;
+ let unioned = RepartitionExec::try_new(
+ unioned,
+ datafusion::physical_plan::Partitioning::RoundRobinBatch(1),
+ )?;
+ self.flat_knn(Arc::new(unioned), &q)
}
#[async_recursion]
@@ -4055,29 +4178,177 @@ impl Scanner {
}
}
- /// Given an index query, split the fragments into two sets
+ /// Given an index query, split the fragments into two groups and collect per-row stale data.
///
- /// The first set is the relevant fragments, which are covered by ALL indices in the query
- /// The second set is the missing fragments, which are missed by at least one index
+ /// - `relevant_frags`: covered by ALL indices. Stale rows within them are returned separately
+ /// so callers can block them from `MaterializeIndexExec` and re-score via a targeted take.
+ /// - `missing_frags`: not covered by at least one index; fall back to full scan + filter.
+ /// - `stale_rows`: per-fragment row offsets whose indexed values are stale due to a data
+ /// overlay committed after the index was built (field-aware, version-gated). Empty when no
+ /// overlays are present.
///
- /// There is no point in handling the case where a fragment is covered by some (but not all)
- /// of the indices. If we have to do a full scan of the fragment then we do it
+ /// There is no point in partially indexing a fragment (some indices cover it, others do not).
+ /// If we have to do a full scan of a fragment for any reason, we do it entirely.
async fn partition_frags_by_coverage(
&self,
index_expr: &ScalarIndexExpr,
fragments: Arc>,
- ) -> Result<(Vec, Vec)> {
+ ) -> Result<(
+ Vec,
+ Vec,
+ std::collections::HashMap,
+ )> {
let covered_frags = self.fragments_covered_by_index_query(index_expr).await?;
+ let stale_rows = self
+ .overlay_stale_index_rows(index_expr, &fragments)
+ .await?;
let mut relevant_frags = Vec::with_capacity(fragments.len());
let mut missing_frags = Vec::with_capacity(fragments.len());
for fragment in fragments.iter() {
if covered_frags.contains(fragment.id as u32) {
+ // Indexed fragments stay on the indexed path. Stale rows within them are blocked
+ // from the index result and re-evaluated separately via a targeted take.
relevant_frags.push(fragment.clone());
} else {
missing_frags.push(fragment.clone());
}
}
- Ok((relevant_frags, missing_frags))
+ Ok((relevant_frags, missing_frags, stale_rows))
+ }
+
+ /// Per-row stale offsets for each fragment whose indexed values may be stale because an
+ /// overlay committed *after* an index was built touches a field that index covers.
+ ///
+ /// The check is field-aware (an overlay touching only unindexed fields excludes nothing) and
+ /// version-gated (an overlay with `committed_version <= index.dataset_version` is already
+ /// incorporated by the index), via [`overlay_exclusion_offsets`].
+ async fn overlay_stale_index_rows(
+ &self,
+ index_expr: &ScalarIndexExpr,
+ fragments: &[Fragment],
+ ) -> Result> {
+ // Overlays are rare; skip all index loading when none of the candidate fragments has one.
+ if fragments
+ .iter()
+ .all(|fragment| fragment.overlays.is_empty())
+ {
+ return Ok(std::collections::HashMap::new());
+ }
+ let frag_by_id: std::collections::HashMap =
+ fragments.iter().map(|f| (f.id as u32, f)).collect();
+
+ // Walk the (boolean) index expression tree to collect leaf searches.
+ let mut searches = Vec::new();
+ let mut stack = vec![index_expr];
+ while let Some(expr) = stack.pop() {
+ match expr {
+ ScalarIndexExpr::Not(inner) => stack.push(inner),
+ ScalarIndexExpr::And(lhs, rhs) | ScalarIndexExpr::Or(lhs, rhs) => {
+ stack.push(lhs);
+ stack.push(rhs);
+ }
+ ScalarIndexExpr::Query(search) => searches.push(search),
+ }
+ }
+
+ // `load_named_scalar_segments` returns cached index metadata — no disk I/O on the hot
+ // path. Even without the cache, this code is only reached when at least one fragment has
+ // overlays (rare), so the per-leaf cost is acceptable.
+ let mut stale: std::collections::HashMap =
+ std::collections::HashMap::new();
+ for search in searches {
+ let segments = load_named_scalar_segments(
+ self.dataset.as_ref(),
+ &search.column,
+ &search.index_name,
+ )
+ .await?;
+ for segment in &segments {
+ collect_overlay_stale_rows_for_segment(segment, &frag_by_id, &mut stale)?;
+ }
+ }
+ Ok(stale)
+ }
+
+ /// Compute per-row stale data for a vector index's segments.
+ ///
+ /// Returns a map from fragment_id to the set of row offsets within that fragment that are stale
+ /// (their vector values have been updated by a newer overlay since the index was built). An
+ /// empty map means no stale rows — the fast path where no masking is needed. See OSS-1325.
+ fn mask_overlay_stale_rows(
+ &self,
+ segments: &[IndexMetadata],
+ ) -> Result> {
+ let fragments = self.dataset.fragments();
+ if fragments
+ .iter()
+ .all(|fragment| fragment.overlays.is_empty())
+ {
+ return Ok(std::collections::HashMap::new());
+ }
+ let frag_by_id: std::collections::HashMap =
+ fragments.iter().map(|f| (f.id as u32, f)).collect();
+ let mut stale: std::collections::HashMap =
+ std::collections::HashMap::new();
+ for segment in segments {
+ collect_overlay_stale_rows_for_segment(segment, &frag_by_id, &mut stale)?;
+ }
+ Ok(stale)
+ }
+
+ /// Compute which FTS segments are stale due to data overlay files committed after the
+ /// index was built, and which fragments must therefore fall back to the flat text path.
+ ///
+ /// Returns `(flat_frag_ids, Some(fresh_segments))` when overlays are present:
+ /// - `flat_frag_ids`: fragment IDs that must be scanned flat (stale fragments, plus any
+ /// other fragments co-located in a segment that covers a stale one — the whole segment is
+ /// excluded, so all fragments it covered must move to flat).
+ /// - `fresh_segments`: the subset of FTS segments that cover no stale fragment; safe to
+ /// pass to `MatchQueryExec::new_with_segments`.
+ ///
+ /// Returns `(empty, None)` on the fast path (no overlays, or no segments load).
+ async fn fts_stale_frags_and_fresh_segments(
+ &self,
+ column: &str,
+ target_fragments: &[Fragment],
+ ) -> Result<(RoaringBitmap, Option>)> {
+ // Fast path: no overlays on any target fragment.
+ if target_fragments.iter().all(|f| f.overlays.is_empty()) {
+ return Ok((RoaringBitmap::new(), None));
+ }
+
+ let Some(segments) = load_segments(&self.dataset, column).await? else {
+ return Ok((RoaringBitmap::new(), None));
+ };
+
+ let frag_by_id: std::collections::HashMap =
+ target_fragments.iter().map(|f| (f.id as u32, f)).collect();
+ let mut stale_frag_ids = RoaringBitmap::new();
+ for seg in &segments {
+ collect_stale_overlay_frags(seg, &frag_by_id, &mut stale_frag_ids)?;
+ }
+
+ if stale_frag_ids.is_empty() {
+ // Overlays exist but none are on this FTS column or predate the index.
+ return Ok((stale_frag_ids, None));
+ }
+
+ // Any segment covering a stale fragment is excluded from the indexed path.
+ // All fragments covered by that segment (stale + co-located fresh ones) must
+ // fall to the flat path, since the indexed path no longer covers them.
+ let mut flat_frag_ids = stale_frag_ids.clone();
+ let mut fresh_segments = Vec::with_capacity(segments.len());
+ for seg in segments {
+ match &seg.fragment_bitmap {
+ Some(bm) if !bm.is_disjoint(&stale_frag_ids) => {
+ flat_frag_ids |= bm;
+ // exclude this segment from the indexed path
+ }
+ _ => fresh_segments.push(seg),
+ }
+ }
+
+ Ok((flat_frag_ids, Some(fresh_segments)))
}
// First perform a lookup in a scalar index for ids and then perform a take on the
@@ -4099,16 +4370,29 @@ impl Scanner {
let needs_recheck = index_expr.needs_recheck();
- // Figure out which fragments are covered by ALL indices
- let (relevant_frags, missing_frags) = self
+ // Figure out which fragments are covered by ALL indices, and which rows within
+ // covered fragments are stale due to data overlay files (OSS-1325).
+ let (relevant_frags, missing_frags, stale_rows) = self
.partition_frags_by_coverage(index_expr, fragments)
.await?;
- let mut plan: Arc = Arc::new(MaterializeIndexExec::new(
+ // Build the MaterializeIndexExec, blocking stale row addresses so the index never
+ // emits them. Stale rows are re-scored separately via a targeted take below.
+ let mat_exec = MaterializeIndexExec::new(
self.dataset.clone(),
index_expr.clone(),
Arc::new(relevant_frags),
- ));
+ );
+ let mat_exec = if stale_rows.is_empty() {
+ mat_exec
+ } else {
+ let mut tree_map = RowAddrTreeMap::new();
+ for (&frag_id, offsets) in &stale_rows {
+ tree_map.insert_bitmap(frag_id, offsets.clone());
+ }
+ mat_exec.with_overlay_block(RowAddrMask::from_block(tree_map))
+ };
+ let mut plan: Arc = Arc::new(mat_exec);
let refine_expr = filter_plan.refine_expr.as_ref();
@@ -4154,6 +4438,21 @@ impl Scanner {
plan = Arc::new(AddRowAddrExec::try_new(plan, self.dataset.clone(), 0)?);
}
+ // Both the missing-fragments path (full scan) and the stale-rows path (targeted take)
+ // need the user's projection extended with any filter columns. Compute it once.
+ let fallback_projection: Option =
+ if !missing_frags.is_empty() || !stale_rows.is_empty() {
+ let filter = filter_plan.full_expr.as_ref().unwrap();
+ let filter_cols = Planner::column_names_in_expr(filter);
+ Some(
+ projection
+ .clone()
+ .union_columns(filter_cols, OnMissing::Error)?,
+ )
+ } else {
+ None
+ };
+
let new_data_path: Option> = if !missing_frags.is_empty() {
log::trace!(
"scalar_indexed_scan will need full scan of {} missing fragments",
@@ -4172,10 +4471,8 @@ impl Scanner {
// If there were no extra columns then we still need the project
// because Materialize -> Take puts the row id at the left and
// Scan puts the row id at the right
+ let scan_projection = fallback_projection.clone().unwrap();
let filter = filter_plan.full_expr.as_ref().unwrap();
- let filter_cols = Planner::column_names_in_expr(filter);
- let scan_projection = projection.union_columns(filter_cols, OnMissing::Error)?;
-
let scan_schema = Arc::new(scan_projection.to_bare_schema());
let scan_arrow_schema = Arc::new(scan_schema.as_ref().into());
let planner = Planner::new(scan_arrow_schema);
@@ -4204,16 +4501,54 @@ impl Scanner {
None
};
- if let Some(new_data_path) = new_data_path {
- let unioned = UnionExec::try_new(vec![plan, new_data_path])?;
- // Enforce only 1 partition.
- let unioned = Arc::new(RepartitionExec::try_new(
- unioned,
- datafusion::physical_plan::Partitioning::RoundRobinBatch(1),
- )?);
- Ok(unioned)
+ // Stale-Take path: re-evaluate only the stale row addresses against the full filter
+ // (OSS-1325 row-level optimization). These rows were blocked from the index result above;
+ // here we take their current (overlay-merged) values and re-apply the predicate.
+ // The schema matches `plan` via `project(…, plan.schema())`.
+ let stale_take_path: Option> = if stale_rows.is_empty() {
+ None
} else {
+ let filter = filter_plan.full_expr.as_ref().unwrap();
+ let take_projection = fallback_projection.unwrap();
+
+ let stale_addrs: Vec = stale_rows
+ .iter()
+ .flat_map(|(&frag_id, offsets)| {
+ offsets
+ .iter()
+ .map(move |offset| ((frag_id as u64) << 32) | offset as u64)
+ })
+ .collect();
+ let batch = RecordBatch::try_new(
+ Arc::new(ArrowSchema::new(vec![ArrowField::new(
+ ROW_ID,
+ DataType::UInt64,
+ true,
+ )])),
+ vec![Arc::new(UInt64Array::from(stale_addrs))],
+ )?;
+ let stale_id_plan = Arc::new(OneShotExec::from_batch(batch));
+ let stale_node = self.take(stale_id_plan, take_projection)?;
+
+ let planner = Planner::new(stale_node.schema());
+ let optimized_filter = planner.optimize_expr(filter.clone())?;
+ let filtered = Arc::new(LanceFilterExec::try_new(optimized_filter, stale_node)?);
+ Some(Arc::new(project(filtered, plan.schema().as_ref())?))
+ };
+
+ let extra_paths: Vec> = [new_data_path, stale_take_path]
+ .into_iter()
+ .flatten()
+ .collect();
+ if extra_paths.is_empty() {
Ok(plan)
+ } else {
+ let all_paths = std::iter::once(plan).chain(extra_paths).collect();
+ let unioned = UnionExec::try_new(all_paths)?;
+ Ok(Arc::new(RepartitionExec::try_new(
+ unioned,
+ datafusion::physical_plan::Partitioning::RoundRobinBatch(1),
+ )?))
}
}
@@ -4645,11 +4980,18 @@ impl Scanner {
q: &Query,
index: &[IndexMetadata],
filter_plan: &ExprFilterPlan,
+ overlay_block: Option,
) -> Result> {
let prefilter_source = self
.prefilter_source(filter_plan, self.get_indexed_frags(index))
.await?;
- let inner_fanout_search = new_knn_exec(self.dataset.clone(), index, q, prefilter_source)?;
+ let inner_fanout_search = new_knn_exec(
+ self.dataset.clone(),
+ index,
+ q,
+ prefilter_source,
+ overlay_block,
+ )?;
let sort_expr = PhysicalSortExpr {
expr: expressions::col(DIST_COL, inner_fanout_search.schema().as_ref())?,
options: SortOptions {
@@ -4676,6 +5018,7 @@ impl Scanner {
q: &Query,
index: &[IndexMetadata],
filter_plan: &ExprFilterPlan,
+ overlay_block: Option,
) -> Result> {
// we split the query procedure into two steps:
// 1. collect the candidates by vector searching on each query vector
@@ -4708,6 +5051,7 @@ impl Scanner {
index,
&query,
prefilter_source.clone(),
+ overlay_block.clone(),
)?;
let sort_expr = PhysicalSortExpr {
expr: expressions::col(DIST_COL, ann_node.schema().as_ref())?,
@@ -4788,7 +5132,7 @@ impl Scanner {
// are not in the fragments we are scanning.
if filter_plan.is_exact_index_search() && self.fragments.is_none() {
let index_query = filter_plan.index_query.as_ref().expect_ok()?;
- let (_, missing_frags) = self
+ let (_, missing_frags, _) = self
.partition_frags_by_coverage(index_query, fragments.clone())
.await?;
@@ -12594,3 +12938,985 @@ full_filter=name LIKE Utf8(\"test%2\"), refine_filter=name LIKE Utf8(\"test%2\")
.await;
}
}
+
+/// Insert into `stale` the ids of fragments covered by `segment` whose index entries may be
+/// stale because an overlay committed after the segment was built touches a field the segment
+/// indexes. Field-aware and version-gated via [`overlay_exclusion_offsets`]. See OSS-1325.
+fn collect_stale_overlay_frags(
+ segment: &IndexMetadata,
+ frag_by_id: &std::collections::HashMap,
+ stale: &mut RoaringBitmap,
+) -> Result<()> {
+ let Some(coverage) = segment.fragment_bitmap.as_ref() else {
+ return Ok(());
+ };
+ for frag_id in coverage.iter() {
+ if stale.contains(frag_id) {
+ continue;
+ }
+ let Some(fragment) = frag_by_id.get(&frag_id) else {
+ continue;
+ };
+ if fragment.overlays.is_empty() {
+ continue;
+ }
+ // Cheap version gate: skip the field/bitmap work if every overlay on this fragment
+ // predates the segment (already incorporated by the index).
+ if fragment
+ .overlays
+ .iter()
+ .all(|o| o.committed_version <= segment.dataset_version)
+ {
+ continue;
+ }
+ if !overlay_exclusion_offsets(&fragment.overlays, &segment.fields, segment.dataset_version)?
+ .is_empty()
+ {
+ stale.insert(frag_id);
+ }
+ }
+ Ok(())
+}
+
+/// Like [`collect_stale_overlay_frags`] but with row-level granularity: instead of marking the
+/// whole fragment stale, it computes exactly which row offsets within each covered fragment are
+/// stale and accumulates them into `stale` (fragment_id → stale row offsets).
+///
+/// Used by the vector ANN path (OSS-1325) to block only the affected rows from ANN results and
+/// re-score only those rows on the flat path, keeping overhead proportional to the number of
+/// overlaid rows rather than the whole fragment size.
+fn collect_overlay_stale_rows_for_segment(
+ segment: &IndexMetadata,
+ frag_by_id: &std::collections::HashMap,
+ stale: &mut std::collections::HashMap,
+) -> Result<()> {
+ let Some(coverage) = segment.fragment_bitmap.as_ref() else {
+ return Ok(());
+ };
+ for frag_id in coverage.iter() {
+ let Some(fragment) = frag_by_id.get(&frag_id) else {
+ continue;
+ };
+ if fragment.overlays.is_empty() {
+ continue;
+ }
+ if fragment
+ .overlays
+ .iter()
+ .all(|o| o.committed_version <= segment.dataset_version)
+ {
+ continue;
+ }
+ let excluded = overlay_exclusion_offsets(
+ &fragment.overlays,
+ &segment.fields,
+ segment.dataset_version,
+ )?;
+ if !excluded.is_empty() {
+ *stale.entry(frag_id).or_default() |= &excluded;
+ }
+ }
+ Ok(())
+}
+
+/// End-to-end tests for OSS-1325: a scalar index masks data overlay files so that
+/// queries stay correct while overlays remain (stale index hits are dropped and new
+/// matches are added by re-evaluating overlay-covered rows on the flat path).
+#[cfg(test)]
+mod overlay_index_masking {
+ use std::sync::Arc;
+
+ use arrow_array::cast::AsArray;
+ use arrow_array::types::Int32Type;
+ use arrow_array::{ArrayRef, Int32Array, RecordBatch, RecordBatchIterator};
+ use arrow_schema::{DataType, Field as ArrowField, Schema as ArrowSchema};
+ use lance_index::IndexType;
+ use lance_index::scalar::FullTextSearchQuery;
+ use lance_index::scalar::ScalarIndexParams;
+ use lance_io::utils::CachedFileSize;
+ use lance_table::format::{DataFile, DataOverlayFile, OverlayCoverage};
+ use roaring::RoaringBitmap;
+
+ use lance_file::writer::{FileWriter, FileWriterOptions};
+
+ use crate::Dataset;
+ use crate::dataset::transaction::{DataOverlayGroup, Operation};
+ use crate::dataset::{WriteDestination, WriteParams};
+ use crate::index::DatasetIndexExt;
+
+ /// Two-fragment Int32 dataset: `id` (field 0) = 0..12 and `age` (field 1) = id * 10,
+ /// six rows per file (fragments 0 and 1). In-memory store so overlay files can be written
+ /// with a store-relative `data/.lance` path and committed against the dataset.
+ async fn create_base_dataset() -> Dataset {
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, true),
+ ArrowField::new("age", DataType::Int32, true),
+ ]));
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..12)),
+ Arc::new(Int32Array::from_iter_values((0..12).map(|v| v * 10))),
+ ],
+ )
+ .unwrap();
+ let write_params = WriteParams {
+ max_rows_per_file: 6,
+ max_rows_per_group: 6,
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
+ Dataset::write(reader, "memory://", Some(write_params))
+ .await
+ .unwrap()
+ }
+
+ async fn build_age_index(dataset: &mut Dataset) {
+ dataset
+ .create_index(
+ &["age"],
+ IndexType::BTree,
+ None,
+ &ScalarIndexParams::default(),
+ true,
+ )
+ .await
+ .unwrap();
+ }
+
+ /// Write an overlay file covering `fields` of `fragment_id` with `coverage` and the given
+ /// per-field value columns, then commit it as a `DataOverlay` transaction. `name` makes
+ /// the overlay file unique.
+ async fn commit_overlay(
+ dataset: Dataset,
+ name: &str,
+ fragment_id: u64,
+ fields: &[i32],
+ coverage: OverlayCoverage,
+ columns: Vec,
+ ) -> Dataset {
+ let read_version = dataset.version().version;
+ let overlay_schema = dataset.schema().project_by_ids(fields, true);
+
+ let filename = format!("{name}.lance");
+ // Use dataset.base so the path is absolute for file:// stores.
+ // to_local_path() prepends '/' to the object_store path, so a bare
+ // "data/foo.lance" would resolve to /data/foo.lance (root fs). With
+ // base we get e.g. tmp/lance-bench/data/foo.lance → /tmp/lance-bench/data/foo.lance.
+ // For memory:// stores base is empty so the result is the same as before.
+ let path = dataset.base.clone().join("data").join(filename.as_str());
+ let obj_writer = dataset.object_store.create(&path).await.unwrap();
+ let mut writer =
+ FileWriter::try_new(obj_writer, overlay_schema, FileWriterOptions::default()).unwrap();
+ let (major, minor) = writer.version().to_numbers();
+ for (i, array) in columns.into_iter().enumerate() {
+ writer.write_column(i, array).await.unwrap();
+ }
+ let summary = writer.finish().await.unwrap();
+
+ let mut data_file = DataFile::new_unstarted(filename, major, minor);
+ data_file.fields = writer
+ .field_id_to_column_indices()
+ .iter()
+ .map(|(field_id, _)| *field_id as i32)
+ .collect::>()
+ .into();
+ data_file.column_indices = writer
+ .field_id_to_column_indices()
+ .iter()
+ .map(|(_, column_index)| *column_index as i32)
+ .collect::>()
+ .into();
+ data_file.file_size_bytes = CachedFileSize::new(summary.size_bytes);
+
+ let overlay = DataOverlayFile {
+ data_file,
+ coverage,
+ committed_version: 0,
+ };
+ Dataset::commit(
+ WriteDestination::Dataset(Arc::new(dataset)),
+ Operation::DataOverlay {
+ groups: vec![DataOverlayGroup {
+ fragment_id,
+ overlays: vec![overlay],
+ }],
+ },
+ Some(read_version),
+ None,
+ None,
+ Arc::new(Default::default()),
+ false,
+ )
+ .await
+ .unwrap()
+ }
+
+ /// Sorted `id` values returned by a filtered scan.
+ async fn ids_matching(dataset: &Dataset, filter: &str) -> Vec {
+ let batch = dataset
+ .scan()
+ .filter(filter)
+ .unwrap()
+ .project(&["id"])
+ .unwrap()
+ .try_into_batch()
+ .await
+ .unwrap();
+ if batch.num_rows() == 0 {
+ return Vec::new();
+ }
+ let mut ids = batch
+ .column_by_name("id")
+ .unwrap()
+ .as_primitive::()
+ .values()
+ .to_vec();
+ ids.sort_unstable();
+ ids
+ }
+
+ fn i32_array(values: impl IntoIterator
- >) -> ArrayRef {
+ Arc::new(Int32Array::from_iter(values))
+ }
+
+ fn fsl(rows: Vec>, dim: i32) -> ArrayRef {
+ let flat: Vec = rows.into_iter().flatten().collect();
+ let item = Arc::new(ArrowField::new("item", DataType::Float32, true));
+ Arc::new(
+ arrow_array::FixedSizeListArray::try_new(
+ item,
+ dim,
+ Arc::new(arrow_array::Float32Array::from(flat)),
+ None,
+ )
+ .unwrap(),
+ )
+ }
+
+ /// A newer overlay on the indexed field drops stale index hits (the old value no longer
+ /// matches) and surfaces new matches (the new value is found even though the index never
+ /// saw it). Mirrors the spec's Bob 25 -> 26 worked example.
+ #[tokio::test]
+ async fn test_overlay_stale_drop_and_new_match() {
+ let mut dataset = create_base_dataset().await;
+ build_age_index(&mut dataset).await;
+
+ // Fragment 0, offset 1 is id=1, age=10. The overlay (committed after the index)
+ // changes its age to 999.
+ let dataset = commit_overlay(
+ dataset,
+ "age_overlay",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(999)])],
+ )
+ .await;
+
+ // Stale-drop: the index still holds age=10 for id=1, but its current value is 999,
+ // so it must not be returned.
+ assert_eq!(ids_matching(&dataset, "age = 10").await, Vec::::new());
+ // New-match: the index never saw age=999, but re-evaluation finds it.
+ assert_eq!(ids_matching(&dataset, "age = 999").await, vec![1]);
+ // An untouched indexed value is unaffected.
+ assert_eq!(ids_matching(&dataset, "age = 20").await, vec![2]);
+ }
+
+ /// Row-level BTree precision: when one row in a covered fragment is stale, only that row is
+ /// blocked from the index result and re-evaluated on the stale-Take path. Non-stale rows in
+ /// the same fragment (including one that matches the predicate) remain on the indexed path.
+ ///
+ /// Setup: fragment 0 has id=5 → age=50 (not stale). Overlay id=1 → age=50 (stale).
+ /// After the overlay two rows in fragment 0 have age=50. The row-level optimization must
+ /// return both: id=5 from the index and id=1 from the stale-Take path.
+ #[tokio::test]
+ async fn test_btree_overlay_row_level_precision() {
+ let mut dataset = create_base_dataset().await;
+ build_age_index(&mut dataset).await;
+
+ // Fragment 0: ids 0-5, ages 0,10,20,30,40,50. Overlay offset 1 (id=1): age 10→50.
+ // After this both id=1 and id=5 have age=50, in the same fragment.
+ let dataset = commit_overlay(
+ dataset,
+ "age_row_level",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(50)])],
+ )
+ .await;
+
+ // Stale drop: id=1's old age=10 entry must not appear.
+ assert_eq!(ids_matching(&dataset, "age = 10").await, Vec::::new());
+
+ // id=5 via index + id=1 via stale-Take path — both in fragment 0.
+ assert_eq!(ids_matching(&dataset, "age = 50").await, vec![1, 5]);
+
+ // Non-stale rows in the same fragment still return correctly.
+ assert_eq!(ids_matching(&dataset, "age = 20").await, vec![2]);
+ assert_eq!(ids_matching(&dataset, "age = 30").await, vec![3]);
+ }
+
+ /// An overlay touching only a non-indexed field excludes nothing from the index on `age`.
+ #[tokio::test]
+ async fn test_overlay_on_unrelated_field_excludes_nothing() {
+ let mut dataset = create_base_dataset().await;
+ build_age_index(&mut dataset).await;
+
+ // Overlay field 0 (`id`), not the indexed `age`. The age index stays fully trusted.
+ let dataset = commit_overlay(
+ dataset,
+ "id_overlay",
+ 0,
+ &[0],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(777)])],
+ )
+ .await;
+
+ // The age index is still trusted: age=10 finds the offset-1 row, whose id now reads
+ // through the overlay as 777. The fragment was not routed to the flat path on account
+ // of an overlay that touches no indexed field.
+ assert_eq!(ids_matching(&dataset, "age = 10").await, vec![777]);
+ // An untouched row is unaffected.
+ assert_eq!(ids_matching(&dataset, "age = 20").await, vec![2]);
+ // The overlaid id is the new value on read, and the old one is gone.
+ assert_eq!(ids_matching(&dataset, "id = 777").await, vec![777]);
+ assert_eq!(ids_matching(&dataset, "id = 1").await, Vec::::new());
+ }
+
+ /// An overlay whose `committed_version <= index.dataset_version` is already incorporated by
+ /// the index (the index was built reading merged values) and is not excluded.
+ #[tokio::test]
+ async fn test_overlay_older_than_index_not_excluded() {
+ let dataset = create_base_dataset().await;
+
+ // Commit the overlay first (age of id=1 becomes 999), then build the index on top.
+ let mut dataset = commit_overlay(
+ dataset,
+ "age_overlay_old",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(999)])],
+ )
+ .await;
+ build_age_index(&mut dataset).await;
+
+ // The index incorporates the overlay, so it returns the merged value directly.
+ assert_eq!(ids_matching(&dataset, "age = 999").await, vec![1]);
+ assert_eq!(ids_matching(&dataset, "age = 10").await, Vec::::new());
+ }
+
+ /// A covered offset whose overlay value is NULL overrides the cell to NULL, so the stale
+ /// index hit for its old value is dropped.
+ #[tokio::test]
+ async fn test_overlay_null_override() {
+ let mut dataset = create_base_dataset().await;
+ build_age_index(&mut dataset).await;
+
+ // id=1 (age=10) is overridden to NULL.
+ let dataset = commit_overlay(
+ dataset,
+ "age_overlay_null",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([None])],
+ )
+ .await;
+
+ assert_eq!(ids_matching(&dataset, "age = 10").await, Vec::::new());
+ assert_eq!(ids_matching(&dataset, "age IS NULL").await, vec![1]);
+ }
+
+ /// Overlays on a non-first fragment are masked correctly, and a query spanning both
+ /// fragments returns the right rows.
+ #[tokio::test]
+ async fn test_overlay_multi_fragment() {
+ let mut dataset = create_base_dataset().await;
+ build_age_index(&mut dataset).await;
+
+ // Fragment 1 holds ids 6..12 (ages 60..110). Offset 2 within fragment 1 is id=8,
+ // age=80; change it to 60 (a value that also legitimately exists at id=6).
+ let dataset = commit_overlay(
+ dataset,
+ "age_overlay_frag1",
+ 1,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([2])),
+ vec![i32_array([Some(60)])],
+ )
+ .await;
+
+ // id=8 no longer has age=80 (stale-drop on fragment 1).
+ assert_eq!(ids_matching(&dataset, "age = 80").await, Vec::::new());
+ // Both id=6 (base) and id=8 (overlay) now have age=60 (new-match added to base hit).
+ assert_eq!(ids_matching(&dataset, "age = 60").await, vec![6, 8]);
+ // A value in the untouched fragment 0 is still served correctly.
+ assert_eq!(ids_matching(&dataset, "age = 30").await, vec![3]);
+ }
+
+ /// A vector index masks overlays: a row whose vector was moved (by a newer overlay) away
+ /// from the query is dropped from results, and a row moved *onto* the query is found by
+ /// re-scoring its current vector on the flat path — even though the index never saw it.
+ #[tokio::test]
+ async fn test_vector_index_rescore_on_overlay() {
+ use arrow_array::cast::AsArray;
+ use futures::TryStreamExt;
+ use lance_index::IndexType;
+ use lance_linalg::distance::MetricType;
+
+ use crate::index::vector::VectorIndexParams;
+
+ const DIM: i32 = 8;
+ let query = vec![1.0_f32, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0];
+ let far = vec![0.0_f32, 100.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0];
+
+ // 64 rows over two fragments. Every base vector is orthogonal to the query except id=35,
+ // which equals the query (so a stale index ranks it first). All sit in fragment 1's range
+ // (32..64) for the rows we overlay; fragment 0 (0..32) holds far, never-overlaid rows.
+ let mut vectors: Vec> = Vec::with_capacity(64);
+ for i in 0..64 {
+ if i == 35 {
+ vectors.push(query.clone());
+ } else {
+ let mut v = vec![0.0_f32; DIM as usize];
+ v[1] = (i + 2) as f32; // orthogonal to the query, distinct, far
+ vectors.push(v);
+ }
+ }
+
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, true),
+ ArrowField::new(
+ "vec",
+ DataType::FixedSizeList(
+ Arc::new(ArrowField::new("item", DataType::Float32, true)),
+ DIM,
+ ),
+ true,
+ ),
+ ]));
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..64)),
+ fsl(vectors, DIM),
+ ],
+ )
+ .unwrap();
+ let write_params = WriteParams {
+ max_rows_per_file: 32,
+ max_rows_per_group: 32,
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
+ let mut dataset = Dataset::write(reader, "memory://", Some(write_params))
+ .await
+ .unwrap();
+
+ // Single-partition IVF_FLAT: the ANN searches every indexed row with exact distances.
+ let params = VectorIndexParams::ivf_flat(1, MetricType::L2);
+ dataset
+ .create_index(&["vec"], IndexType::Vector, None, ¶ms, true)
+ .await
+ .unwrap();
+
+ // Overlay fragment 1 (ids 32..64): move id=35 (offset 3) onto `far`, and id=40
+ // (offset 8) onto the query. The index, built before the overlay, still believes id=35
+ // is the query and has never seen id=40 near it.
+ let dataset = commit_overlay(
+ dataset,
+ "vec_overlay",
+ 1,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([3, 8])),
+ vec![fsl(vec![far.clone(), query.clone()], DIM)],
+ )
+ .await;
+
+ let results = dataset
+ .scan()
+ .nearest("vec", &arrow_array::Float32Array::from(query.clone()), 3)
+ .unwrap()
+ .minimum_nprobes(1)
+ .project(&["id"])
+ .unwrap()
+ .try_into_stream()
+ .await
+ .unwrap()
+ .try_collect::>()
+ .await
+ .unwrap();
+
+ let ids: Vec = results
+ .iter()
+ .flat_map(|b| {
+ b.column_by_name("id")
+ .unwrap()
+ .as_primitive::()
+ .values()
+ .to_vec()
+ })
+ .collect();
+
+ // id=40 was moved onto the query and is found by re-scoring (new-match recall).
+ assert!(
+ ids.contains(&40),
+ "expected id=40 (re-scored to query) in {ids:?}"
+ );
+ // id=35's stale index entry (the query) must not resurface: its current vector is far.
+ assert!(
+ !ids.contains(&35),
+ "stale vector for id=35 should be dropped, got {ids:?}"
+ );
+ }
+
+ /// A compound boolean predicate (age AND id) exercises the ScalarIndexExpr tree-walk in
+ /// `overlay_stale_index_frags`. An overlay on `age` marks fragment 0 stale from the `age`
+ /// index's perspective, so the compound query must re-evaluate fragment 0 on the flat path.
+ #[tokio::test]
+ async fn test_overlay_stale_with_compound_index_expression() {
+ let mut dataset = create_base_dataset().await;
+ // Build BTree indexes on both columns so a compound filter can use both.
+ build_age_index(&mut dataset).await;
+ dataset
+ .create_index(
+ &["id"],
+ IndexType::BTree,
+ None,
+ &ScalarIndexParams::default(),
+ true,
+ )
+ .await
+ .unwrap();
+
+ // Fragment 0 covers id=0..5, age=0..50. Overlay changes id=1's age from 10 to 999.
+ let dataset = commit_overlay(
+ dataset,
+ "age_compound",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(999)])],
+ )
+ .await;
+
+ // Compound query: both the `age` and `id` index are involved. The overlay on `age`
+ // makes fragment 0 stale for the `age` index; it falls to the flat path, which uses
+ // the merged (overlay) value. Result: the stale age=10 hit is gone, age=999 appears.
+ assert_eq!(ids_matching(&dataset, "age = 10").await, Vec::::new());
+ assert_eq!(ids_matching(&dataset, "age = 999").await, vec![1]);
+ // A pure `id` query on an unaffected fragment still works correctly.
+ assert_eq!(ids_matching(&dataset, "id = 2").await, vec![2]);
+ }
+
+ /// Text dataset: two fragments, 6 rows each. Schema: id (Int32), text (Utf8).
+ /// Texts are unique tokens so each row can be identified by its term.
+ async fn create_text_dataset() -> Dataset {
+ use arrow_array::StringArray;
+
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, true),
+ ArrowField::new("text", DataType::Utf8, true),
+ ]));
+ let texts: Vec<&str> = vec![
+ "apple pie",
+ "apple banana", // row 1, fragment 0 — will be overlaid in tests
+ "cherry cake",
+ "banana split",
+ "orange juice",
+ "grape vine",
+ "mango sorbet", // fragment 1 starts here
+ "pear tart",
+ "lemon curd",
+ "peach cobbler",
+ "plum pudding",
+ "fig newton",
+ ];
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..12)),
+ Arc::new(StringArray::from(texts)),
+ ],
+ )
+ .unwrap();
+ let write_params = WriteParams {
+ max_rows_per_file: 6,
+ max_rows_per_group: 6,
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
+ Dataset::write(reader, "memory://", Some(write_params))
+ .await
+ .unwrap()
+ }
+
+ async fn build_text_fts_index(dataset: &mut Dataset) {
+ use lance_index::scalar::inverted::InvertedIndexParams;
+
+ dataset
+ .create_index(
+ &["text"],
+ IndexType::Inverted,
+ None,
+ &InvertedIndexParams::default(),
+ true,
+ )
+ .await
+ .unwrap();
+ }
+
+ /// Collect sorted IDs of rows returned by an FTS query on `text`.
+ async fn fts_ids_matching(dataset: &Dataset, term: &str) -> Vec {
+ use arrow_array::cast::AsArray;
+ use futures::TryStreamExt;
+
+ let results = dataset
+ .scan()
+ .full_text_search(FullTextSearchQuery::new(term.to_owned()))
+ .unwrap()
+ .project(&["id"])
+ .unwrap()
+ .try_into_stream()
+ .await
+ .unwrap()
+ .try_collect::>()
+ .await
+ .unwrap();
+ let mut ids: Vec = results
+ .iter()
+ .flat_map(|b| {
+ b.column_by_name("id")
+ .unwrap()
+ .as_primitive::()
+ .values()
+ .to_vec()
+ })
+ .collect();
+ ids.sort_unstable();
+ ids
+ }
+
+ /// An overlay committed after the FTS index is built replaces a row's text. Searching for
+ /// the old term must not return the stale row; searching for the new term must find it.
+ #[tokio::test]
+ async fn test_fts_overlay_stale_drop_and_new_match() {
+ use arrow_array::StringArray;
+
+ let mut dataset = create_text_dataset().await;
+ build_text_fts_index(&mut dataset).await;
+
+ // fragment 0, row offset 1 (id=1): "apple banana" → "cherry mango"
+ // field ID 1 is the `text` column.
+ let dataset = commit_overlay(
+ dataset,
+ "text_overlay",
+ 0,
+ &[1],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![Arc::new(StringArray::from(vec![Some("cherry mango")]))],
+ )
+ .await;
+
+ // "apple" now matches only id=0 ("apple pie"); id=1's stale index entry must be dropped.
+ assert_eq!(fts_ids_matching(&dataset, "apple").await, vec![0]);
+
+ // "banana" matched id=1 and id=3 before; after overlay id=1's stale entry must be gone.
+ assert_eq!(fts_ids_matching(&dataset, "banana").await, vec![3]);
+
+ // "cherry" now matches id=1 (via flat path on stale fragment) and id=2 ("cherry cake").
+ let cherry_ids = fts_ids_matching(&dataset, "cherry").await;
+ assert!(
+ cherry_ids.contains(&1),
+ "id=1 overlay→cherry mango should be found: {cherry_ids:?}"
+ );
+ assert!(
+ cherry_ids.contains(&2),
+ "id=2 cherry cake should still be found: {cherry_ids:?}"
+ );
+
+ // "mango" now matches id=1 (overlay) and id=6 ("mango sorbet" in fragment 1).
+ let mango_ids = fts_ids_matching(&dataset, "mango").await;
+ assert!(
+ mango_ids.contains(&1),
+ "id=1 overlay→cherry mango should be found: {mango_ids:?}"
+ );
+ assert!(
+ mango_ids.contains(&6),
+ "id=6 mango sorbet should still be found: {mango_ids:?}"
+ );
+ }
+
+ /// An overlay on a field the FTS index does NOT cover must not exclude anything.
+ #[tokio::test]
+ async fn test_fts_overlay_unrelated_field_not_excluded() {
+ let mut dataset = create_text_dataset().await;
+ build_text_fts_index(&mut dataset).await;
+
+ // Overlay field 0 (id) — not covered by the FTS index on `text`.
+ let dataset = commit_overlay(
+ dataset,
+ "id_overlay_for_fts",
+ 0,
+ &[0],
+ OverlayCoverage::dense(RoaringBitmap::from_iter([1])),
+ vec![i32_array([Some(999)])],
+ )
+ .await;
+
+ // FTS coverage must be unchanged — both rows containing "apple" are still returned.
+ // The `id` overlay changes row offset 1's id from 1 to 999, so the projected id column
+ // reflects the overlay even though the FTS index correctly returned that row.
+ assert_eq!(fts_ids_matching(&dataset, "apple").await, vec![0, 999]);
+ assert_eq!(fts_ids_matching(&dataset, "banana").await, vec![3, 999]);
+ }
+
+ /// Benchmark: measure query latency for BTree, FTS, and vector ANN with 0/4/16 overlay layers.
+ ///
+ /// Run with: cargo test -p lance --lib --release -- overlay_index_masking::bench --ignored --nocapture
+ #[tokio::test]
+ #[ignore = "benchmark"]
+ #[allow(clippy::print_stdout)]
+ async fn bench_index_query_overlay_overhead() {
+ use std::time::Instant;
+
+ use arrow_array::Float32Array;
+ use lance_linalg::distance::MetricType;
+
+ use crate::index::vector::VectorIndexParams;
+
+ const DIM: i32 = 32;
+ const ROWS: i32 = 1_000_000;
+ const ROWS_PER_FRAG: i32 = 100_000; // 10 fragments
+ const ITERS: u32 = 10; // large scans — 10 is enough for stable averages
+
+ // Fixed disk path so timings are comparable across runs. Deleted and recreated fresh.
+ let uri = "/tmp/lance-bench-overlay-oss1325";
+ if std::path::Path::new(uri).exists() {
+ std::fs::remove_dir_all(uri).unwrap();
+ }
+
+ // --- Build 1M-row dataset on local disk --------------------------------
+ // Schema: id(0), age(1), vec(2) — 3 top-level fields.
+ // Lance field IDs (depth-first): id=0, age=1, vec=2, vec.item=3.
+
+ println!("Building {ROWS}-row dataset at {uri} (this takes ~30 s)...");
+
+ let schema = Arc::new(ArrowSchema::new(vec![
+ ArrowField::new("id", DataType::Int32, false),
+ ArrowField::new("age", DataType::Int32, false),
+ ArrowField::new(
+ "vec",
+ DataType::FixedSizeList(
+ Arc::new(ArrowField::new("item", DataType::Float32, true)),
+ DIM,
+ ),
+ false,
+ ),
+ ]));
+
+ let row_ids: Vec = (0..ROWS).collect();
+ let ages: Vec = row_ids.iter().map(|&i| i * 10).collect();
+ // Build the 128 MB flat float array directly (avoids 1M per-row Vec allocations).
+ let flat_vecs: Vec = (0..(ROWS as usize * DIM as usize))
+ .map(|j| (j / DIM as usize) as f32 % 1000.0)
+ .collect();
+ let vec_col = Arc::new(
+ arrow_array::FixedSizeListArray::try_new(
+ Arc::new(ArrowField::new("item", DataType::Float32, true)),
+ DIM,
+ Arc::new(Float32Array::from(flat_vecs)),
+ None,
+ )
+ .unwrap(),
+ );
+
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![
+ Arc::new(Int32Array::from(row_ids)),
+ Arc::new(Int32Array::from(ages)),
+ vec_col,
+ ],
+ )
+ .unwrap();
+
+ let write_params = WriteParams {
+ max_rows_per_file: ROWS_PER_FRAG as usize,
+ max_rows_per_group: ROWS_PER_FRAG as usize,
+ ..Default::default()
+ };
+ let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
+ let mut dataset = Dataset::write(reader, uri, Some(write_params))
+ .await
+ .unwrap();
+
+ println!("Building BTree index on age...");
+ dataset
+ .create_index(
+ &["age"],
+ IndexType::BTree,
+ None,
+ &ScalarIndexParams::default(),
+ true,
+ )
+ .await
+ .unwrap();
+
+ println!("Building IVF_FLAT(1 partition) index on vec...");
+ dataset
+ .create_index(
+ &["vec"],
+ IndexType::Vector,
+ None,
+ &VectorIndexParams::ivf_flat(1, MetricType::L2),
+ true,
+ )
+ .await
+ .unwrap();
+
+ println!("Indexes built.\n");
+
+ // --- Timing helper ---------------------------------------------------
+
+ async fn timeit(iters: u32, mut f: F) -> f64
+ where
+ F: FnMut() -> Fut,
+ Fut: std::future::Future