From aab9f18c1b07a2599c99079106f3e2ec1ab2bd99 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 19 Jun 2026 15:55:38 -0700 Subject: [PATCH 1/9] docs: specify data overlay files for the table format Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/table/.pages | 1 + docs/src/format/table/data_overlay_file.md | 390 +++++++++++++++++++++ docs/src/format/table/index.md | 29 ++ protos/table.proto | 72 ++++ 4 files changed, 492 insertions(+) create mode 100644 docs/src/format/table/data_overlay_file.md diff --git a/docs/src/format/table/.pages b/docs/src/format/table/.pages index 16c20058608..5b0cb0e95e6 100644 --- a/docs/src/format/table/.pages +++ b/docs/src/format/table/.pages @@ -6,4 +6,5 @@ nav: - Layout: layout.md - Branch & Tag: branch_tag.md - Row ID & Lineage: row_id_lineage.md + - Data Overlay Files: data_overlay_file.md - MemTable & WAL: mem_wal.md diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md new file mode 100644 index 00000000000..5540f860018 --- /dev/null +++ b/docs/src/format/table/data_overlay_file.md @@ -0,0 +1,390 @@ +# Data Overlay Files + +!!! note "Overlay files require feature flag 64 (data overlay files)" + + A reader or writer that does not understand overlay files must refuse a + dataset that uses them. Silently ignoring an overlay would return stale base + values, which is a correctness bug rather than a degraded experience. + +Overlay files supply new values for a subset of `(row offset, field)` cells +within a fragment **without rewriting the fragment's base data files**. They make +updates cheap when only a small fraction of rows and/or columns change: instead +of rewriting whole columns or moving rows to a new fragment, a writer appends a +small file carrying just the changed cells. + +This is Lance's third mechanism for changing data in place, alongside +[deletion files](index.md#deletion-files) (which remove rows) and +[data evolution](index.md#data-evolution) (which adds or rewrites whole columns). +An overlay changes individual cells. + +## Concepts + +### Coverage and resolution + +Each overlay declares which cells it provides through a **coverage** bitmap (or, +for sparse overlays, one bitmap per field). The bitmaps index **physical row +offsets** — positions in the base data files, counting deleted rows — so they are +stable across deletions, exactly like deletion vectors. + +To resolve a cell `(offset, field)` on read, walk the fragment's overlays from +**newest to oldest**. The first overlay that covers `(offset, field)` wins; its +value is used. If no overlay covers the cell, the value falls through to the base +data file (or is `NULL` if no base data file holds that field). + +Precedence among overlays is determined by: + +1. `committed_version` — higher wins (see [Versioning](#versioning-and-ordering)). +2. Position in `DataFragment.overlays` as a tiebreaker — a later entry is newer. + +A covered offset whose value is `NULL` overrides the cell **to** `NULL`. This is +distinct from an offset that is simply absent from the bitmap, which falls +through to the base. Coverage, not value-nullness, decides whether an overlay +applies. + +### Interaction with deletions + +Deletions take precedence over overlays. If a row offset is marked deleted in the +fragment's deletion file, any overlay value for that offset is dead and is +ignored, regardless of commit order. This keeps the invariant simple: a deletion +is the final word on a row, so a concurrent overlay against a row that was +deleted needs no special conflict handling — its values are merely inert. + +### Physical layout + +An overlay's data file stores **one value column per field**, in the order of +`data_file.fields`. It does **not** store a row-offset key column. The position of +a covered offset's value within its column is the **rank** of that offset in the +field's coverage bitmap — the number of set bits below it. For a Roaring bitmap +this is an O(1) operation, so random access to any cell is a rank computation +followed by a single value fetch, with no offset column to read and no binary +search. + +Because different fields may cover different offset sets, the value columns of a +single sparse overlay may have **different lengths**. The Lance file format +permits columns of differing item counts within one file, so a sparse overlay is +representable as a single file. (See [Writer support](#writer-support) for the +current implementation status.) + +### Dense vs. sparse overlays + +A single overlay is one of two shapes: + +- **Dense (rectangular).** One `shared_offset_bitmap` applies to every field. Every + covered offset has a value for every field. This is the common case for a plain + `UPDATE`, where one `SET` list is applied to one set of rows. +- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different + fields cover different offset sets — for example a `MERGE` with multiple + `WHEN MATCHED` branches, where different rows update different columns. A dense + overlay would have to widen to the bounding rectangle and fill the untouched + cells with their current values (post-images), which for wide columns such as + embeddings means re-storing data that did not change. A sparse overlay stores + exactly the changed cells. + +A writer may always express a non-rectangular update as **multiple dense overlays +in one transaction** (one per coverage group) instead of a single sparse overlay. + +## Protobuf + +
+DataOverlayFile protobuf message + +```protobuf +%%% proto.message.DataOverlayFile %%% +``` + +
+ +
+FieldCoverage protobuf message + +```protobuf +%%% proto.message.FieldCoverage %%% +``` + +
+ +## Versioning and ordering + +Overlays reuse the dataset version as their ordering clock rather than +introducing a separate generation counter. + +`committed_version` is the dataset version at which an overlay **became +effective** — the version of the commit that introduced it, **not** the version +it was read from. It is stamped at commit time and re-stamped if the commit is +retried, in the same way as the created-at / last-updated-at version sequences. + +This single value drives every ordering decision: + +- **Overlay vs. overlay** (read precedence): higher `committed_version` wins. +- **Overlay vs. index** (query correctness): an index records the + `dataset_version` it was built from. An index whose `dataset_version >= + committed_version` already incorporates the overlay. An overlay whose + `committed_version > index.dataset_version` is newer than the index and its + cells must be excluded from index results and re-evaluated. +- **Scheduler signal**: the gap between an overlay's `committed_version` and an + index's `dataset_version`, or between an overlay and the base, is a staleness + measure the compaction scheduler can use. + +!!! note "Why effective version, not read version" + + Suppose an overlay reads version 5 and commits at version 6, while an index + is built reading version 5 (before the overlay) and commits at version 7 with + `dataset_version = 5`. If the overlay stored its *read* version (5), the test + `5 > 5` is false, the row would not be excluded, and the index — which never + saw the overlay — would return a stale result. Storing the *effective* + version (6) makes `6 > 5` true, the cell is excluded and re-evaluated, and the + result is correct. + +## Index integration + +Building an index over a fragment that has overlays does **not** require dropping +the fragment from the index's coverage. The fragment stays indexed, and the query +path reconciles overlays at query time using an **exclusion set**. + +The exclusion set for an index on field `F` is the union of the coverage bitmaps, +restricted to field `F`, of every overlay whose `committed_version > +index.dataset_version`. The exclusion is **field-aware**: an overlay that touches +only unrelated columns does not exclude anything from the index on `F`. + +The query then proceeds as: + +1. Run the index search as usual, producing candidate rows. +2. Remove any candidate in the exclusion set. (Its indexed value may be stale.) +3. **Re-evaluate** the excluded rows against their current values — the same flat + path already used for the unindexed tail of fragments. For a scalar predicate + this re-applies the filter; for a vector query it re-scores the row's current + vector. Rows that still match are added back to the result. + +Step 3 is what makes exclusion correct rather than merely safe: removing a row +from index candidates without re-evaluating it would silently drop a row that +should match under its new value. + +### Correctness invariant + +> For every indexed field `F` and every row offset `o` in a fragment the index +> covers, the index's entry for `(o, F)` is trusted unless `o` is excluded. +> `o` is excluded iff some overlay with `committed_version > index.dataset_version` +> covers `(o, F)`. + +The write and compaction paths together preserve this: + +- **Writes** change a cell only by adding an overlay, and that overlay's + `committed_version` exceeds the version of any pre-existing index — so the + change is always covered by an exclusion. +- **Compaction** may remove an overlay only if the index no longer relies on it + (see below). + +## Compaction + +Overlays accumulate read cost — every overlay is a bitmap to test and a possible +file to open. Compaction bounds that cost in two modes: + +- **Overlay → overlay.** Merge several overlays into fewer, computing the + post-image per `(offset, field)` by walking the merged overlays newest-first. + The merged overlay takes the **maximum** `committed_version` of its inputs, so + the exclusion semantics are preserved. Indexes are unaffected. This is cheap and + does not touch the base. +- **Overlay → base.** Fold overlays into a fresh base data file, computing the + post-image for every covered cell, then clear the overlays. The base is + complete, so every post-image is well defined. Overlay offsets are physical, so + they cannot survive a rewrite that reorders rows; folding therefore materializes + values rather than carrying overlays forward. + +!!! warning "Folding an indexed field must update its index" + + An overlay→base fold removes the overlay, which removes the exclusion signal + that kept an index correct. Folding an overlay that covers an indexed field + `F` is therefore equivalent to a column rewrite of `F` and must, in the same + commit, either rebuild the index to a `dataset_version` at least the folded + overlay's `committed_version`, or remove the fragment from the index's + coverage so the rows fall to the flat path. Otherwise the index would serve + stale values with no overlay to exclude them. This is the same rule that + already governs rewriting a column that an index is built on. + +When a fragment with overlays is compacted by a row-rewriting operation +(`RewriteRows`, which produces new fragments with new row addresses), the +overlays are folded into the new base as part of the rewrite, and existing +[fragment-reuse remapping](row_id_lineage.md) handles the row-address changes as +it does today. + +## Row lineage + +An overlay write updates the `last_updated_at_version` of every covered row, so +change-data-feed and time-travel queries observe the update. Because overlays are +addressed by physical offset, they do **not** require stable row IDs to be +enabled; lineage updates apply only when those features are on. + +## Worked example + +A table `users` with stable row IDs enabled and these fields: + +| field id | name | type | +|----------|-----------|-------------------------| +| 1 | id | `int32` (primary key) | +| 2 | name | `utf8` | +| 3 | age | `int32` | +| 4 | embedding | `fixed_size_list`| + +Created at version 1 as a single fragment `0` with one base data file +`data/file0.lance` holding all four columns. `physical_rows = 4`: + +| offset | id | name | age | embedding | +|--------|----|-------|-----|------------------| +| 0 | 1 | Alice | 30 | … | +| 1 | 2 | Bob | 25 | … | +| 2 | 3 | Carol | 40 | … | +| 3 | 4 | Dave | 22 | … | + +A BTree scalar index on `age` is built at version 1, covering fragment `0` +(`dataset_version = 1`). + +### Step 1 — write an overlay + +```sql +UPDATE users SET age = 26 WHERE id = 2; -- Bob, offset 1 +``` + +This touches one field (`age`) for one row, so the writer emits a dense overlay +and commits it as version 2. Fragment `0` gains: + +```text +DataOverlayFile { + data_file: { path: "data/overlay-.lance", fields: [3], column_indices: [0] } + coverage: shared_offset_bitmap = {1} + committed_version: 2 +} +``` + +The overlay file stores a single `age` column with one value, `[26]`, at +rank `{1}.rank(1) = 0`. `last_updated_at_version[1]` is set to 2. + +### Step 2 — read + +`SELECT id, age FROM users` reads base ages `[30, 25, 40, 22]`. For `age` +(field 3), the overlay covers offset 1, so `age[1]` is replaced with the overlay +value at position `{1}.rank(1) = 0` → `26`. Result ages: `[30, 26, 40, 22]`. + +### Step 3 — index query + +```sql +SELECT * FROM users WHERE age = 26; +``` + +The `age` index was built at `dataset_version = 1`; the overlay's +`committed_version` is 2. Since `2 > 1`, the overlay's coverage for `age`, `{1}`, +is the exclusion set for this query. + +- The index (built at v1) holds Bob's *old* `age = 25`, so a lookup for `26` + returns nothing from the index. +- Offset 1 is in the exclusion set, so it is re-evaluated on the flat path. Its + current `age` (26, via the overlay) matches `age = 26`, so Bob is returned. + +The mirror case `WHERE age = 25` shows exclusion preventing a stale hit: the index +returns offset 1 (stale `25`), but offset 1 is excluded, re-evaluated to `26`, and +correctly dropped. + +### Step 4 — a second, non-rectangular write + +```sql +MERGE INTO users USING staged ON users.id = staged.id +WHEN MATCHED AND staged.kind = 'rename' THEN UPDATE SET name = staged.name -- Carol(2), Dave(3) +WHEN MATCHED AND staged.kind = 'revec' THEN UPDATE SET embedding = staged.embedding -- Bob(1) +``` + +`name` is updated for offsets `{2, 3}` and `embedding` for offset `{1}` — different +fields over different rows. This is a sparse overlay, committed as version 3: + +```text +DataOverlayFile { + data_file: { path: "data/overlay-.lance", fields: [2, 4], column_indices: [0, 1] } + coverage: field_coverage { offset_bitmaps: [ {2,3}, {1} ] } + // name (field 2) ^ ^ embedding (field 4) + committed_version: 3 +} +``` + +The file's `name` column has **two** values (`["Caroline", "David"]`, at +ranks 0 and 1 of `{2,3}`) and its `embedding` column has **one** value (at rank 0 +of `{1}`) — columns of different lengths in one file. + +### Step 5 — read after the second write + +`SELECT name, age, embedding FROM users` resolves each field independently, +newest overlay first: + +- `name`: the v3 overlay covers `{2,3}` → `["Alice", "Bob", "Caroline", "David"]`. +- `age`: the v3 overlay does not cover `age`; the v2 overlay still applies at + offset 1 → `[30, 26, 40, 22]`. +- `embedding`: the v3 overlay covers `{1}` → Bob's vector is the new one, others + from base. + +Overlays from different versions coexist and apply per field. + +### Step 6 — compaction (overlay → base) + +The scheduler folds both overlays into fragment `0` at version 4, computing +post-images for `age`, `name`, and `embedding`, and writing a new base data file +`data/file1.lance` with those columns. In the old file, fields 2, 3, and 4 are +tombstoned (`-2`); field 1 (`id`) remains. The fragment's `overlays` list is +cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so +stable row IDs and the deletion vector are untouched. + +Because the fold removed the overlay that was excluding offset 1 from the `age` +index, the same commit must reconcile that index: either rebuild it at +`dataset_version >= 2`, or drop fragment `0` from its coverage so `age` queries +fall to the flat path. After a rebuild at version 4, no overlay remains and the +`age` index directly returns `26` for Bob with no exclusion needed. + +## Guidance + +!!! note "This section is a stub." + + The following are implementation considerations, not part of the on-disk + specification. + +### When to overlay vs. rewrite a column vs. move rows + +*(To be expanded.)* The choice between appending an overlay, rewriting a full +column (data evolution), and moving updated rows to a new fragment depends on the +fraction of rows changed, the fraction of columns changed, column width, the +presence of indexes on the changed columns, and the accumulated overlay read +cost. Roughly: few rows changed favors overlays; most rows in a few columns +favors a column rewrite; most columns changed favors moving rows to a new +fragment. + +### Writer support + +*(To be expanded.)* Dense (rectangular) overlays write with the existing +equal-length file writer today. Sparse overlays stored as a **single** file +require the writer to emit columns of independent lengths, which the current v2 +writer does not yet do (it advances all columns from one global row counter). +Until that support lands, a writer can express a sparse update as multiple dense +overlays in one transaction. + +### Scheduling compaction + +*(To be expanded.)* The overlay→overlay and overlay→base modes have very +different costs; a cost/benefit scheduler decides when each is worthwhile, using +the version gap as a staleness signal. + +### Open questions + +*(To be resolved.)* + +- **Per-fragment vs. per-table overlays.** Overlays are attached per fragment. + Should there be a table-level overlay concept, and how would it interact with + fragment-level row addressing? +- **Relationship to LSM.** Overlays plus compaction resemble an LSM tree (newest + layer wins, periodic merge). How far should that analogy be taken, and what do + we deliberately do differently given Lance's random-access requirements? +- **Coverage bitmap spill.** Coverage bitmaps live inline in the manifest. Very + large coverage (an overlay touching many rows) may warrant external spill, as + the row-ID and last-updated-at sequences already do above a size threshold. + +## Related specifications + +- [Table format overview](index.md) +- [Transactions](transaction.md) +- [Row ID & Lineage](row_id_lineage.md) +- [Index Formats](../index/index.md) +- [Format Versioning](versioning.md) diff --git a/docs/src/format/table/index.md b/docs/src/format/table/index.md index 94ea4b90dc9..ce4d0b26613 100644 --- a/docs/src/format/table/index.md +++ b/docs/src/format/table/index.md @@ -168,6 +168,35 @@ However, this invalidates row addresses and requires rebuilding indices, which c +## Data Overlay Files + +!!! note "Overlay files require feature flag 64 (data overlay files)" + +Overlay files supply new values for a subset of `(row offset, field)` cells within +a fragment without rewriting the base data files. They make updates cheap when only +a small percentage of rows and/or columns change: a writer appends a small file +carrying just the changed cells instead of rewriting whole columns or moving rows +to a new fragment. + +On read, each cell is resolved by consulting the fragment's overlays from newest to +oldest; the first overlay covering that `(offset, field)` wins, otherwise the value +falls through to the base data file. Indices keep covering the fragment and reconcile +overlays at query time through a field-aware exclusion set. + +For the full specification — coverage and resolution rules, dense vs. sparse layout, +versioning, index integration, compaction, and a worked example — see the +[Data Overlay Files Specification](data_overlay_file.md). + +
+DataOverlayFile protobuf message + +```protobuf +%%% proto.message.DataOverlayFile %%% +``` + +
+ + ## Related Specifications ### Storage Layout diff --git a/protos/table.proto b/protos/table.proto index 8d0cb249fda..7d4fb19d50b 100644 --- a/protos/table.proto +++ b/protos/table.proto @@ -115,6 +115,9 @@ message Manifest { // * 1 << 3: table config is present // * 1 << 4: dataset uses multiple base paths // * 1 << 5: transaction file writes are disabled + // * 1 << 6: data overlay files are present (see DataOverlayFile). Readers that do + // not understand overlays must refuse the dataset, since ignoring an overlay + // would silently return stale base values. uint64 reader_feature_flags = 9; // Feature flags for writers. @@ -313,6 +316,15 @@ message DataFragment { repeated DataFile files = 2; + // Optional overlay files for this fragment, which supply new values for a + // subset of cells without rewriting the base data files. This MUST be empty + // if the data overlay files feature flag (64) is not set in the manifest. + // + // Order is significant: a later entry is newer than an earlier one. When two + // overlays cover the same (offset, field) and share a `committed_version`, the + // later entry wins. See DataOverlayFile for the full resolution rules. + repeated DataOverlayFile overlays = 11; + // File that indicates which rows, if any, should be considered deleted. DeletionFile deletion_file = 3; @@ -435,6 +447,66 @@ message DataFile { optional uint32 base_id = 7; } // DataFile +// An overlay file supplies new values for a subset of (row offset, field) cells +// within a fragment, without rewriting the fragment's base data files. It is +// used for efficient updates when only a small fraction of rows and/or columns +// change. +// +// On read, a cell is resolved by consulting the fragment's overlays from newest +// to oldest: the first overlay that covers that (offset, field) wins; if none +// cover it, the value falls through to the base data file. Because deletions +// take precedence over overlays, an overlay value for an offset that is also +// marked deleted is dead and is ignored. +// +// The overlay's data file does NOT store a row-offset key column. Within a value +// column, the position of a covered offset's value is the rank (0-based count of +// set bits below it) of that offset within the field's coverage bitmap. Because +// fields may cover different offset sets, the value columns of a single overlay +// data file may have different lengths (which the Lance file format permits). +message DataOverlayFile { + // The data file storing the overlay's new cell values, one value column per + // field in `data_file.fields`. No row-offset key column is stored. + DataFile data_file = 1; + + // Which (offset, field) cells this overlay provides values for. + oneof coverage { + // A single 32-bit Roaring bitmap of physical row offsets that applies to + // every field in `data_file.fields` (a "dense" / rectangular overlay). + // Every covered offset has a value for every field. This is the common case + // for a plain UPDATE, where one SET list is applied to one set of rows. + bytes shared_offset_bitmap = 2; + // Per-field coverage for a "sparse" overlay, used when different fields cover + // different offset sets (e.g. a MERGE with multiple WHEN MATCHED branches). + FieldCoverage field_coverage = 4; + } + + // The dataset version at which this overlay became effective: the version of + // the commit that introduced it, NOT the version it was read from. It is + // stamped at commit time and re-stamped if the commit is retried, in the same + // way as the created-at / last-updated-at version sequences. + // + // This drives two orderings: + // * Versus index builds: an index whose `dataset_version` >= this value + // already incorporates this overlay. Otherwise the overlay's covered cells + // are excluded from index results for the affected fields and re-evaluated + // against their current values (see the Data Overlay Files specification). + // * Versus other overlays: when two overlays cover the same (offset, field), + // the one with the higher `committed_version` wins. Overlays that share a + // `committed_version` are ordered by their position in + // `DataFragment.overlays`, where a later entry is newer and wins. + uint64 committed_version = 3; +} + +// Per-field coverage for a sparse overlay. +message FieldCoverage { + // One entry per field in the overlay's `data_file.fields`, in the same order. + // Each is a 32-bit Roaring bitmap of the physical row offsets covered for that + // field. An offset present in a field's bitmap but mapped to a NULL value + // means the cell is overridden to NULL (distinct from an offset that is absent, + // which falls through to the base data file). + repeated bytes offset_bitmaps = 1; +} + // Deletion File // // The path of the deletion file is constructed as: From 88a47e47ce246330cf06b1c709c92acdf2bac347 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Mon, 22 Jun 2026 12:47:14 -0700 Subject: [PATCH 2/9] feat: add DataOverlay transaction operation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) --- protos/transaction.proto | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/protos/transaction.proto b/protos/transaction.proto index e72e95025a4..bfc0eee354b 100644 --- a/protos/transaction.proto +++ b/protos/transaction.proto @@ -315,6 +315,44 @@ message Transaction { repeated DataReplacementGroup replacements = 1; } + // Overlay files to append to a single fragment, in order (the last entry is + // newest). The overlays are appended to the fragment's existing `overlays` + // list; they do not replace it, so overlays written by concurrent commits are + // preserved. + message DataOverlayGroup { + uint64 fragment_id = 1; + // Each DataOverlayFile.committed_version is left 0 by the writer and stamped + // to the new dataset version at commit time (re-stamped on retry), in the + // same way as the created-at / last-updated-at version sequences. The fields + // touched are read from each overlay's `data_file.fields`. + repeated DataOverlayFile overlays = 2; + } + + // Attach overlay files to fragments, supplying new values for a subset of + // (row offset, field) cells without rewriting the fragments' base data files. + // See the DataOverlayFile message in table.proto and the Data Overlay Files + // specification for resolution, coverage, and versioning rules. + // + // Conflict semantics (intentionally permissive, like DataReplacement). Against + // a concurrent operation that touches one of the same fragments: + // * Another DataOverlay (any fields): COMPATIBLE. Overlays stack; when two + // overlays cover the same (offset, field) the one with the higher + // `committed_version` wins, so independent backfills never conflict. + // * Append / new fragments: COMPATIBLE. + // * Delete: COMPATIBLE. A deletion takes precedence over an overlay, so an + // overlay value for a deleted offset is inert (no special handling needed). + // * DataReplacement or column-rewrite (Update with REWRITE_COLUMNS) of the + // same field: COMPATIBLE. Both preserve physical row addresses, so overlay + // offsets stay valid; the overlay is newer and wins its covered cells, and + // the version gate excludes those cells from any rebuilt index. + // * Row-rewrite, compaction, or an overlay->base fold of the fragment: + // CONFLICT. These change physical row addresses or consume the overlays, so + // the overlay's offsets are no longer valid. The writer must re-read the new + // fragment, recompute, and retry. + message DataOverlay { + repeated DataOverlayGroup groups = 1; + } + // Update the merged generations in MemWAL index. // This operation is used during merge-insert to atomically record which // generations have been merged to the base table. @@ -346,6 +384,7 @@ message Transaction { UpdateMemWalState update_mem_wal_state = 112; Clone clone = 113; UpdateBases update_bases = 114; + DataOverlay data_overlay = 115; } // Fields 200/202 (`blob_append` / `blob_overwrite`) previously represented blob dataset ops. From 2ef4635dc93cc10b51af516bdf841e3779761024 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Mon, 22 Jun 2026 14:29:29 -0700 Subject: [PATCH 3/9] chore: compile against DataOverlay proto, reject unsupported overlays The table/transaction proto changes generate new fields and an Operation variant. This wires the minimum needed to compile without implementing overlay support: - Emit empty `overlays` when converting fragments to proto. - Reject the `DataOverlay` transaction operation with NotSupported on read. Datasets that use overlays set reader feature flag 64, which already falls in the unknown-flag range rejected by `can_read_dataset`, so the library refuses them at the feature-flag layer. Co-Authored-By: Claude Opus 4.8 (1M context) --- rust/lance-table/benches/manifest_intern.rs | 2 ++ rust/lance-table/src/format/fragment.rs | 4 +++ rust/lance/src/dataset/transaction.rs | 28 +++++++++++++++++++++ 3 files changed, 34 insertions(+) diff --git a/rust/lance-table/benches/manifest_intern.rs b/rust/lance-table/benches/manifest_intern.rs index 78b7e352207..81bd57c1a22 100644 --- a/rust/lance-table/benches/manifest_intern.rs +++ b/rust/lance-table/benches/manifest_intern.rs @@ -59,6 +59,7 @@ fn make_uniform_pb_fragments(n: u64, num_fields: usize) -> Vec file_size_bytes: 0, base_id: None, }], + overlays: vec![], deletion_file: None, row_id_sequence: None, physical_rows: 1000, @@ -135,6 +136,7 @@ fn make_diverse_pb_fragments( file_size_bytes: 0, base_id: None, }], + overlays: vec![], deletion_file: None, row_id_sequence: None, physical_rows: 1000, diff --git a/rust/lance-table/src/format/fragment.rs b/rust/lance-table/src/format/fragment.rs index 431e466dbd4..e9d9ce036ee 100644 --- a/rust/lance-table/src/format/fragment.rs +++ b/rust/lance-table/src/format/fragment.rs @@ -716,6 +716,10 @@ impl From<&Fragment> for pb::DataFragment { Self { id: f.id, files: f.files.iter().map(pb::DataFile::from).collect(), + // Overlay files are not produced by this version of the library; a + // dataset that uses them sets reader feature flag 64, which is + // rejected at the feature-flag layer (see lance-table feature_flags). + overlays: vec![], deletion_file, row_id_sequence, physical_rows: f.physical_rows.unwrap_or_default() as u64, diff --git a/rust/lance/src/dataset/transaction.rs b/rust/lance/src/dataset/transaction.rs index 3261f9300c4..a3649b323da 100644 --- a/rust/lance/src/dataset/transaction.rs +++ b/rust/lance/src/dataset/transaction.rs @@ -3348,6 +3348,16 @@ impl TryFrom for Transaction { })) => Operation::UpdateBases { new_bases: new_bases.into_iter().map(BasePath::from).collect(), }, + Some(pb::transaction::Operation::DataOverlay(_)) => { + // Overlay files are not supported by this version of the library. + // A dataset that uses them sets reader feature flag 64, which is + // already rejected at the feature-flag layer; reject here too so a + // transaction referencing the operation can never be applied. + return Err(Error::not_supported( + "data overlay files are not supported by this version of Lance \ + (reader feature flag 64)", + )); + } None => { return Err(Error::internal( "Transaction message did not contain an operation".to_string(), @@ -6186,4 +6196,22 @@ mod tests { assert!(!left.modifies_same_metadata(&different_key)); assert!(left.modifies_same_metadata(&replace)); } + + #[test] + fn test_data_overlay_operation_rejected() { + // Overlay files are not supported by this version of the library. A + // transaction carrying the DataOverlay operation must be rejected rather + // than silently ignored, mirroring the feature-flag-64 rejection. + let message = pb::Transaction { + read_version: 1, + uuid: Uuid::new_v4().to_string(), + operation: Some(pb::transaction::Operation::DataOverlay( + pb::transaction::DataOverlay { groups: vec![] }, + )), + ..Default::default() + }; + + let result = Transaction::try_from(message); + assert!(matches!(result, Err(Error::NotSupported { .. }))); + } } From 16a9bea70e7ca7cad52650c26969522d278480fb Mon Sep 17 00:00:00 2001 From: Will Jones Date: Wed, 24 Jun 2026 10:55:39 -0700 Subject: [PATCH 4/9] docs: mark data overlay files as experimental Add an experimental admonition to the data overlay file spec and the table-format overview, with a TODO to name the first released version that supports the feature once implementation lands. Add TODO comments on the guidance subsections noting they will be filled in as benchmarking and implementation progress. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/table/data_overlay_file.md | 18 ++++++++++++++++++ docs/src/format/table/index.md | 7 +++++++ 2 files changed, 25 insertions(+) diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index 5540f860018..fb868f09e99 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -1,5 +1,13 @@ # Data Overlay Files +!!! warning "Experimental" + + This feature is currently experimental and not yet supported in any library. + + + !!! note "Overlay files require feature flag 64 (data overlay files)" A reader or writer that does not understand overlay files must refuse a @@ -344,6 +352,10 @@ fall to the flat path. After a rebuild at version 4, no overlay remains and the ### When to overlay vs. rewrite a column vs. move rows + + *(To be expanded.)* The choice between appending an overlay, rewriting a full column (data evolution), and moving updated rows to a new fragment depends on the fraction of rows changed, the fraction of columns changed, column width, the @@ -354,6 +366,9 @@ fragment. ### Writer support + + *(To be expanded.)* Dense (rectangular) overlays write with the existing equal-length file writer today. Sparse overlays stored as a **single** file require the writer to emit columns of independent lengths, which the current v2 @@ -363,6 +378,9 @@ overlays in one transaction. ### Scheduling compaction + + *(To be expanded.)* The overlay→overlay and overlay→base modes have very different costs; a cost/benefit scheduler decides when each is worthwhile, using the version gap as a staleness signal. diff --git a/docs/src/format/table/index.md b/docs/src/format/table/index.md index ce4d0b26613..f28e615649e 100644 --- a/docs/src/format/table/index.md +++ b/docs/src/format/table/index.md @@ -170,6 +170,13 @@ However, this invalidates row addresses and requires rebuilding indices, which c ## Data Overlay Files +!!! warning "Experimental" + + This feature is currently experimental and not yet supported in any library. + + + !!! note "Overlay files require feature flag 64 (data overlay files)" Overlay files supply new values for a subset of `(row offset, field)` cells within From e8341c34b0ed1ce87f74ae6ef0e2ec1ba2299399 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Tue, 7 Jul 2026 14:27:54 -0700 Subject: [PATCH 5/9] Apply suggestions from code review Co-authored-by: Weston Pace Co-authored-by: Will Jones --- docs/src/format/table/data_overlay_file.md | 40 ++++++++++------------ docs/src/format/table/index.md | 15 +------- 2 files changed, 20 insertions(+), 35 deletions(-) diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index fb868f09e99..9c2c46a8cf2 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -30,9 +30,8 @@ An overlay changes individual cells. ### Coverage and resolution Each overlay declares which cells it provides through a **coverage** bitmap (or, -for sparse overlays, one bitmap per field). The bitmaps index **physical row -offsets** — positions in the base data files, counting deleted rows — so they are -stable across deletions, exactly like deletion vectors. +for sparse overlays, one bitmap per field). The bitmaps index **physical row +offsets**. They include deleted rows and are stable even as deletion vectors change. To resolve a cell `(offset, field)` on read, walk the fragment's overlays from **newest to oldest**. The first overlay that covers `(offset, field)` wins; its @@ -80,16 +79,14 @@ A single overlay is one of two shapes: - **Dense (rectangular).** One `shared_offset_bitmap` applies to every field. Every covered offset has a value for every field. This is the common case for a plain `UPDATE`, where one `SET` list is applied to one set of rows. -- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different - fields cover different offset sets — for example a `MERGE` with multiple - `WHEN MATCHED` branches, where different rows update different columns. A dense - overlay would have to widen to the bounding rectangle and fill the untouched - cells with their current values (post-images), which for wide columns such as - embeddings means re-storing data that did not change. A sparse overlay stores - exactly the changed cells. - -A writer may always express a non-rectangular update as **multiple dense overlays -in one transaction** (one per coverage group) instead of a single sparse overlay. +- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different + fields cover different offset sets — for example a `MERGE` with multiple + `WHEN MATCHED` branches, where different rows update different columns. A dense + overlay would have to widen to the bounding rectangle and fill the untouched + cells with their current values (post-images), which for wide columns such as + embeddings means re-storing data that did not change. A sparse overlay stores + exactly the changed cells. + ## Protobuf @@ -184,14 +181,15 @@ The write and compaction paths together preserve this: ## Compaction -Overlays accumulate read cost — every overlay is a bitmap to test and a possible -file to open. Compaction bounds that cost in two modes: +Overlays accumulate read cost — every overlay is a bitmap to test, a possible +file to open, and additional work to interleave values. Compaction bounds that cost in two modes: - **Overlay → overlay.** Merge several overlays into fewer, computing the post-image per `(offset, field)` by walking the merged overlays newest-first. The merged overlay takes the **maximum** `committed_version` of its inputs, so - the exclusion semantics are preserved. Indexes are unaffected. This is cheap and - does not touch the base. + the exclusion semantics are preserved. Indexes can still be re-used, but they + may now need to exclude more rows. This is cheap to write and does not touch + the base. - **Overlay → base.** Fold overlays into a fresh base data file, computing the post-image for every covered cell, then clear the overlays. The base is complete, so every post-image is well defined. Overlay offsets are physical, so @@ -224,6 +222,8 @@ enabled; lineage updates apply only when those features are on. ## Worked example +The following example illustrates how overlays function across their lifecycle, to make the rules above concrete. + A table `users` with stable row IDs enabled and these fields: | field id | name | type | @@ -338,10 +338,8 @@ cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so stable row IDs and the deletion vector are untouched. Because the fold removed the overlay that was excluding offset 1 from the `age` -index, the same commit must reconcile that index: either rebuild it at -`dataset_version >= 2`, or drop fragment `0` from its coverage so `age` queries -fall to the flat path. After a rebuild at version 4, no overlay remains and the -`age` index directly returns `26` for Bob with no exclusion needed. +index, the commit must drop fragment `0` from its coverage so `age` queries +fall to the flat path. ## Guidance diff --git a/docs/src/format/table/index.md b/docs/src/format/table/index.md index f28e615649e..f9da132cf3b 100644 --- a/docs/src/format/table/index.md +++ b/docs/src/format/table/index.md @@ -179,29 +179,16 @@ However, this invalidates row addresses and requires rebuilding indices, which c !!! note "Overlay files require feature flag 64 (data overlay files)" -Overlay files supply new values for a subset of `(row offset, field)` cells within +Overlay files supply new values for a subset of cells within a fragment without rewriting the base data files. They make updates cheap when only a small percentage of rows and/or columns change: a writer appends a small file carrying just the changed cells instead of rewriting whole columns or moving rows to a new fragment. -On read, each cell is resolved by consulting the fragment's overlays from newest to -oldest; the first overlay covering that `(offset, field)` wins, otherwise the value -falls through to the base data file. Indices keep covering the fragment and reconcile -overlays at query time through a field-aware exclusion set. - For the full specification — coverage and resolution rules, dense vs. sparse layout, versioning, index integration, compaction, and a worked example — see the [Data Overlay Files Specification](data_overlay_file.md). -
-DataOverlayFile protobuf message - -```protobuf -%%% proto.message.DataOverlayFile %%% -``` - -
## Related Specifications From 0ffaac86b7a21353a1e1a645d4266fb6a135af5e Mon Sep 17 00:00:00 2001 From: Will Jones Date: Tue, 7 Jul 2026 15:09:19 -0700 Subject: [PATCH 6/9] docs: relocate data overlay conflict and index rules to canonical docs Move the DataOverlay conflict-resolution rules out of the transaction.proto comment into transaction.md as a DataOverlay operation + Compatibility section, matching the pattern used for every other operation. Consolidate the reader-side overlay handling in the index doc and make re-evaluation explicit. Mark the feature experimental, drop the open-questions section, and cross-link the three docs. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/index/index.md | 13 +++++- docs/src/format/table/data_overlay_file.md | 51 ++++++++-------------- docs/src/format/table/transaction.md | 47 ++++++++++++++++++++ protos/transaction.proto | 22 ++-------- 4 files changed, 80 insertions(+), 53 deletions(-) diff --git a/docs/src/format/index/index.md b/docs/src/format/index/index.md index 4a2e33c60a4..af8fdf595b4 100644 --- a/docs/src/format/index/index.md +++ b/docs/src/format/index/index.md @@ -177,7 +177,7 @@ or updated. These should be filtered out during query execution. -There are three situations to consider: +There are four situations to consider: 1. **A fragment has some deleted rows.** A few of the rows in the fragment have been marked as deleted, but some of the rows are still present. The row addresses from the deletion @@ -188,6 +188,17 @@ There are three situations to consider: 3. **A fragment has had the indexed column updated in place.** This cannot be detected just by examining metadata. To prevent reading invalid data, the engine should filter out any row addresses that are not in the index's current `fragment_bitmap`. +4. **A fragment has an updated value in an [overlay file](../table/data_overlay_file.md).** + This can be detected by checking if any of the fragments in the index's `fragment_bitmap` + have overlay files. For each overlay whose `committed_version` is greater than the index + segment's `dataset_version`, the overlay carries updated values not reflected in the index, + so its covered rows must be excluded from index results. Excluded rows are re-evaluated + against their current (overlaid) values on the flat path — dropping them without + re-evaluation would silently lose rows that match under the new value. Exclusion is + field-aware: only overlays covering the indexed field matter. You may exclude just the + affected rows or the whole fragment; the latter is simpler and safer but re-evaluates more + rows than necessary. See [Data Overlay Files](../table/data_overlay_file.md#index-integration) + for the exclusion set, re-evaluation, and correctness invariant. ## Compaction and remapping diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index 9c2c46a8cf2..f86d4df1c9c 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -30,8 +30,8 @@ An overlay changes individual cells. ### Coverage and resolution Each overlay declares which cells it provides through a **coverage** bitmap (or, -for sparse overlays, one bitmap per field). The bitmaps index **physical row -offsets**. They include deleted rows and are stable even as deletion vectors change. +for sparse overlays, one bitmap per field). The bitmaps index **physical row +offsets**. They include deleted rows and are stable even as deletion vectors change. To resolve a cell `(offset, field)` on read, walk the fragment's overlays from **newest to oldest**. The first overlay that covers `(offset, field)` wins; its @@ -52,9 +52,7 @@ applies. Deletions take precedence over overlays. If a row offset is marked deleted in the fragment's deletion file, any overlay value for that offset is dead and is -ignored, regardless of commit order. This keeps the invariant simple: a deletion -is the final word on a row, so a concurrent overlay against a row that was -deleted needs no special conflict handling — its values are merely inert. +ignored, regardless of commit order. ### Physical layout @@ -79,14 +77,14 @@ A single overlay is one of two shapes: - **Dense (rectangular).** One `shared_offset_bitmap` applies to every field. Every covered offset has a value for every field. This is the common case for a plain `UPDATE`, where one `SET` list is applied to one set of rows. -- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different - fields cover different offset sets — for example a `MERGE` with multiple - `WHEN MATCHED` branches, where different rows update different columns. A dense - overlay would have to widen to the bounding rectangle and fill the untouched - cells with their current values (post-images), which for wide columns such as - embeddings means re-storing data that did not change. A sparse overlay stores - exactly the changed cells. - +- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different + fields cover different offset sets — for example a `MERGE` with multiple + `WHEN MATCHED` branches, where different rows update different columns. A dense + overlay would have to widen to the bounding rectangle and fill the untouched + cells with their current values (post-images), which for wide columns such as + embeddings means re-storing data that did not change. A sparse overlay stores + exactly the changed cells. + ## Protobuf @@ -181,8 +179,8 @@ The write and compaction paths together preserve this: ## Compaction -Overlays accumulate read cost — every overlay is a bitmap to test, a possible -file to open, and additional work to interleave values. Compaction bounds that cost in two modes: +Overlays accumulate read cost — every overlay is a bitmap to test, a possible +file to open, and additional work to interleave values. Compaction bounds that cost in two modes: - **Overlay → overlay.** Merge several overlays into fewer, computing the post-image per `(offset, field)` by walking the merged overlays newest-first. @@ -296,7 +294,7 @@ correctly dropped. ```sql MERGE INTO users USING staged ON users.id = staged.id WHEN MATCHED AND staged.kind = 'rename' THEN UPDATE SET name = staged.name -- Carol(2), Dave(3) -WHEN MATCHED AND staged.kind = 'revec' THEN UPDATE SET embedding = staged.embedding -- Bob(1) +WHEN MATCHED AND staged.kind = 'embed' THEN UPDATE SET embedding = staged.embedding -- Bob(1) ``` `name` is updated for offsets `{2, 3}` and `embedding` for offset `{1}` — different @@ -333,7 +331,7 @@ Overlays from different versions coexist and apply per field. The scheduler folds both overlays into fragment `0` at version 4, computing post-images for `age`, `name`, and `embedding`, and writing a new base data file `data/file1.lance` with those columns. In the old file, fields 2, 3, and 4 are -tombstoned (`-2`); field 1 (`id`) remains. The fragment's `overlays` list is +marked with a tombstone (`-2`); field 1 (`id`) remains. The fragment's `overlays` list is cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so stable row IDs and the deletion vector are untouched. @@ -383,24 +381,11 @@ overlays in one transaction. different costs; a cost/benefit scheduler decides when each is worthwhile, using the version gap as a staleness signal. -### Open questions - -*(To be resolved.)* - -- **Per-fragment vs. per-table overlays.** Overlays are attached per fragment. - Should there be a table-level overlay concept, and how would it interact with - fragment-level row addressing? -- **Relationship to LSM.** Overlays plus compaction resemble an LSM tree (newest - layer wins, periodic merge). How far should that analogy be taken, and what do - we deliberately do differently given Lance's random-access requirements? -- **Coverage bitmap spill.** Coverage bitmaps live inline in the manifest. Very - large coverage (an overlay touching many rows) may warrant external spill, as - the row-ID and last-updated-at sequences already do above a size threshold. - ## Related specifications - [Table format overview](index.md) -- [Transactions](transaction.md) +- [Transactions: DataOverlay operation](transaction.md#dataoverlay) — write path + and conflict semantics - [Row ID & Lineage](row_id_lineage.md) -- [Index Formats](../index/index.md) +- [Index Formats: handling overlay rows](../index/index.md#handling-deleted-and-invalidated-rows) - [Format Versioning](versioning.md) diff --git a/docs/src/format/table/transaction.md b/docs/src/format/table/transaction.md index 78dd5301fb8..85e4ca43ed4 100644 --- a/docs/src/format/table/transaction.md +++ b/docs/src/format/table/transaction.md @@ -466,6 +466,53 @@ The following operations are retryable conflicts with DataReplacement: A concurrent Delete or Update that only adds a deletion vector to a target fragment (without removing it) is compatible: the positional column file stays aligned and the rebase preserves the deletion vector. +### DataOverlay + +Attaches [overlay files](data_overlay_file.md) to fragments, supplying new values +for a subset of `(row offset, field)` cells without rewriting the fragments' base +data files. The overlays are appended to each fragment's existing `overlays` list, +so overlays written by concurrent commits are preserved. Each overlay's +`committed_version` is stamped to the new dataset version at commit time (and +re-stamped on retry), like the created-at / last-updated-at version sequences. + +
+DataOverlay protobuf message + +```protobuf +%%% proto.message.DataOverlay %%% + +%%% proto.message.DataOverlayGroup %%% +``` + +
+ +#### DataOverlay Compatibility + +A DataOverlay operation only changes cells within existing fragments and preserves +physical row addresses, so — like DataReplacement — it is intentionally permissive. +Because overlays stack and the higher `committed_version` wins each covered cell, +independent backfills never conflict, and a concurrent Delete simply makes the +overlay value for a deleted offset inert. Here are the operations that conflict +with DataOverlay: + +- Overwrite +- Restore +- UpdateMemWalState + +The following operations are retryable conflicts with DataOverlay: + +- Rewrite (only if overlapping fragments) — row-rewriting compaction or an + overlay→base fold changes physical row addresses or consumes the overlays, so + the overlay's offsets are no longer valid; the writer must re-read the new + fragment, recompute, and retry. +- Merge (always) + +DataOverlay is compatible with another DataOverlay (any fields), Append, Delete, +and DataReplacement or a column rewrite (Update with `REWRITE_COLUMNS`) of the same +field, because all of these preserve physical row addresses: overlay offsets stay +valid, the overlay is newer and wins its covered cells, and the version gate +excludes those cells from any rebuilt index. + ### UpdateMemWalState Updates the state of MemWal indices (write-ahead log based indices). diff --git a/protos/transaction.proto b/protos/transaction.proto index bfc0eee354b..13d11915af4 100644 --- a/protos/transaction.proto +++ b/protos/transaction.proto @@ -330,25 +330,9 @@ message Transaction { // Attach overlay files to fragments, supplying new values for a subset of // (row offset, field) cells without rewriting the fragments' base data files. - // See the DataOverlayFile message in table.proto and the Data Overlay Files - // specification for resolution, coverage, and versioning rules. - // - // Conflict semantics (intentionally permissive, like DataReplacement). Against - // a concurrent operation that touches one of the same fragments: - // * Another DataOverlay (any fields): COMPATIBLE. Overlays stack; when two - // overlays cover the same (offset, field) the one with the higher - // `committed_version` wins, so independent backfills never conflict. - // * Append / new fragments: COMPATIBLE. - // * Delete: COMPATIBLE. A deletion takes precedence over an overlay, so an - // overlay value for a deleted offset is inert (no special handling needed). - // * DataReplacement or column-rewrite (Update with REWRITE_COLUMNS) of the - // same field: COMPATIBLE. Both preserve physical row addresses, so overlay - // offsets stay valid; the overlay is newer and wins its covered cells, and - // the version gate excludes those cells from any rebuilt index. - // * Row-rewrite, compaction, or an overlay->base fold of the fragment: - // CONFLICT. These change physical row addresses or consume the overlays, so - // the overlay's offsets are no longer valid. The writer must re-read the new - // fragment, recompute, and retry. + // See the DataOverlayFile message in table.proto for resolution, coverage, and + // versioning rules, and the Data Overlay Files and Transactions specifications + // for the (intentionally permissive) conflict semantics. message DataOverlay { repeated DataOverlayGroup groups = 1; } From ccdf731ca2a8889776c3d8dffdd1bcdfcabff07e Mon Sep 17 00:00:00 2001 From: Will Jones Date: Tue, 7 Jul 2026 15:10:30 -0700 Subject: [PATCH 7/9] docs: fold correctness invariant into index integration prose Remove the standalone Correctness invariant section (redundant with the operational description and the compaction warning) and keep its one meaningful claim: a write's overlay committed_version always exceeds a pre-existing index's dataset_version, so exclusion is always sufficient. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/table/data_overlay_file.md | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index f86d4df1c9c..e62b00056bf 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -85,7 +85,6 @@ A single overlay is one of two shapes: embeddings means re-storing data that did not change. A sparse overlay stores exactly the changed cells. - ## Protobuf
@@ -162,20 +161,12 @@ Step 3 is what makes exclusion correct rather than merely safe: removing a row from index candidates without re-evaluating it would silently drop a row that should match under its new value. -### Correctness invariant - -> For every indexed field `F` and every row offset `o` in a fragment the index -> covers, the index's entry for `(o, F)` is trusted unless `o` is excluded. -> `o` is excluded iff some overlay with `committed_version > index.dataset_version` -> covers `(o, F)`. - -The write and compaction paths together preserve this: - -- **Writes** change a cell only by adding an overlay, and that overlay's - `committed_version` exceeds the version of any pre-existing index — so the - change is always covered by an exclusion. -- **Compaction** may remove an overlay only if the index no longer relies on it - (see below). +Exclusion is always *sufficient* because a write changes a cell only by adding an +overlay, and that overlay's `committed_version` — the version of the commit that +adds it — necessarily exceeds the `dataset_version` of any pre-existing index. So +every cell a write changes is guaranteed to fall in that index's exclusion set. +Compaction may remove an overlay only if no index still relies on it for exclusion +(see [Compaction](#compaction)). ## Compaction From 61fd5445263158ffdd141ba3e4f1d0c18eba8f81 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Tue, 7 Jul 2026 15:29:42 -0700 Subject: [PATCH 8/9] docs: correct overlay rank-lookup cost claim The Physical layout note claimed rank on a Roaring bitmap is O(1) with no binary search. roaring-rs rank is O(number of containers) plus a within-container lookup that does binary-search (array) or popcount (bitmap), so neither claim holds. Drop the complexity claim and keep the mechanism: rank plus one value fetch, no stored offset column. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/table/data_overlay_file.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index e62b00056bf..1ef861e332a 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -59,9 +59,8 @@ ignored, regardless of commit order. An overlay's data file stores **one value column per field**, in the order of `data_file.fields`. It does **not** store a row-offset key column. The position of a covered offset's value within its column is the **rank** of that offset in the -field's coverage bitmap — the number of set bits below it. For a Roaring bitmap -this is an O(1) operation, so random access to any cell is a rank computation -followed by a single value fetch, with no offset column to read and no binary +field's coverage bitmap — the number of set bits below it. Resolving a cell is a +rank lookup plus one value fetch, with no separate offset column to store or search. Because different fields may cover different offset sets, the value columns of a From f079f3991953fc091f3212ca756a7b7509b470c3 Mon Sep 17 00:00:00 2001 From: Will Jones Date: Fri, 10 Jul 2026 12:25:01 -0700 Subject: [PATCH 9/9] docs: require contiguous overlay merges; widen worked example Overlay->overlay compaction stamps the merged overlay with the maximum committed_version of its inputs, which only preserves precedence when the merged overlays are contiguous in committed_version. State that constraint with an example. Also widen the worked example's first UPDATE to touch two rows so the dense-vs-sparse choice is motivated rather than arbitrary, and let it demonstrate that the full exclusion set is re-evaluated. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/src/format/table/data_overlay_file.md | 41 +++++++++++++--------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/docs/src/format/table/data_overlay_file.md b/docs/src/format/table/data_overlay_file.md index 1ef861e332a..9da739a29e3 100644 --- a/docs/src/format/table/data_overlay_file.md +++ b/docs/src/format/table/data_overlay_file.md @@ -175,9 +175,12 @@ file to open, and additional work to interleave values. Compaction bounds that c - **Overlay → overlay.** Merge several overlays into fewer, computing the post-image per `(offset, field)` by walking the merged overlays newest-first. The merged overlay takes the **maximum** `committed_version` of its inputs, so - the exclusion semantics are preserved. Indexes can still be re-used, but they - may now need to exclude more rows. This is cheap to write and does not touch - the base. + the exclusion semantics are preserved. The merged overlays must be **contiguous + in `committed_version`** — with overlays at v10, v30, and v50 you cannot merge + just v10 and v50, because stamping the result v50 would incorrectly promote + v10's values above the intervening v30 for any cell v30 also covers. Indexes can + still be re-used, but they may now need to exclude more rows. This is cheap to + write and does not touch the base. - **Overlay → base.** Fold overlays into a fresh base data file, computing the post-image for every covered cell, then clear the overlays. The base is complete, so every post-image is well defined. Overlay offsets are physical, so @@ -237,28 +240,31 @@ A BTree scalar index on `age` is built at version 1, covering fragment `0` ### Step 1 — write an overlay ```sql -UPDATE users SET age = 26 WHERE id = 2; -- Bob, offset 1 +UPDATE users SET age = age + 1 WHERE id IN (2, 4); -- Bob (offset 1), Dave (offset 3) ``` -This touches one field (`age`) for one row, so the writer emits a dense overlay -and commits it as version 2. Fragment `0` gains: +This touches one field (`age`) for two rows, so the writer emits a dense overlay — +one shared bitmap covering both offsets — and commits it as version 2. Fragment +`0` gains: ```text DataOverlayFile { data_file: { path: "data/overlay-.lance", fields: [3], column_indices: [0] } - coverage: shared_offset_bitmap = {1} + coverage: shared_offset_bitmap = {1, 3} committed_version: 2 } ``` -The overlay file stores a single `age` column with one value, `[26]`, at -rank `{1}.rank(1) = 0`. `last_updated_at_version[1]` is set to 2. +The overlay file stores a single `age` column with two values, `[26, 23]`, at +ranks `{1,3}.rank(1) = 0` and `{1,3}.rank(3) = 1`. `last_updated_at_version` is +set to 2 for offsets 1 and 3. ### Step 2 — read `SELECT id, age FROM users` reads base ages `[30, 25, 40, 22]`. For `age` -(field 3), the overlay covers offset 1, so `age[1]` is replaced with the overlay -value at position `{1}.rank(1) = 0` → `26`. Result ages: `[30, 26, 40, 22]`. +(field 3), the overlay covers offsets 1 and 3, so `age[1]` is replaced with the +overlay value at rank `{1,3}.rank(1) = 0` → `26`, and `age[3]` with the value at +rank `{1,3}.rank(3) = 1` → `23`. Result ages: `[30, 26, 40, 23]`. ### Step 3 — index query @@ -267,13 +273,14 @@ SELECT * FROM users WHERE age = 26; ``` The `age` index was built at `dataset_version = 1`; the overlay's -`committed_version` is 2. Since `2 > 1`, the overlay's coverage for `age`, `{1}`, +`committed_version` is 2. Since `2 > 1`, the overlay's coverage for `age`, `{1, 3}`, is the exclusion set for this query. - The index (built at v1) holds Bob's *old* `age = 25`, so a lookup for `26` returns nothing from the index. -- Offset 1 is in the exclusion set, so it is re-evaluated on the flat path. Its - current `age` (26, via the overlay) matches `age = 26`, so Bob is returned. +- The whole exclusion set is re-evaluated on the flat path, not just the rows the + index returned. Offset 1's current `age` (26, via the overlay) matches, so Bob + is returned; offset 3's current `age` (23) does not match and is dropped. The mirror case `WHERE age = 25` shows exclusion preventing a stale hit: the index returns offset 1 (stale `25`), but offset 1 is excluded, re-evaluated to `26`, and @@ -310,7 +317,7 @@ newest overlay first: - `name`: the v3 overlay covers `{2,3}` → `["Alice", "Bob", "Caroline", "David"]`. - `age`: the v3 overlay does not cover `age`; the v2 overlay still applies at - offset 1 → `[30, 26, 40, 22]`. + offsets 1 and 3 → `[30, 26, 40, 23]`. - `embedding`: the v3 overlay covers `{1}` → Bob's vector is the new one, others from base. @@ -325,8 +332,8 @@ marked with a tombstone (`-2`); field 1 (`id`) remains. The fragment's `overlays cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so stable row IDs and the deletion vector are untouched. -Because the fold removed the overlay that was excluding offset 1 from the `age` -index, the commit must drop fragment `0` from its coverage so `age` queries +Because the fold removed the overlay that was excluding offsets 1 and 3 from the +`age` index, the commit must drop fragment `0` from its coverage so `age` queries fall to the flat path. ## Guidance