test(python): add data overlay benchmark suite by wjones127 · Pull Request #7544 · lance-format/lance

wjones127 · 2026-06-30T20:04:37Z

Adds a Python benchmark suite for data overlay files in python/ci_benchmarks/, stacked on #7540 (which exposes the DataOverlay commit binding these benchmarks use to build fixtures).

The benchmarks generate their own base datasets and overlays into a temp dir, so no pre-generation step is needed. Three areas:

test_overlay_manifest.py — manifest growth vs. number of overlays and coverage size/pattern.
test_overlay_write.py — bytes written updating 1% of a column via an overlay vs. update / merge_insert / full rewrite.
test_overlay_read.py — take and scan wall time (pytest-benchmark) plus cold read IO (io_stats_incremental) as overlay layers, coverage fraction, fragmentation (contiguous vs. strided), and data type (int32 vs. fixed-size-list embedding) vary, across v2.0 and v2.1.

Shared dataset/overlay construction and measurement helpers (/proc/self/io, on-disk size, manifest size) live in ci_benchmarks/overlays.py.

Notes:

The update-via-overlay path does not exist yet, so the write benchmark's overlay arm is hand-rolled and writes fresh values without reading the base column first; the headline metric is bytes written, latency secondary.
Stacked on feat(python): expose DataOverlay commit operation #7540 → feat: resolve data overlay files on the take (and scan) read path #7536 → feat: data overlay files — model, feature flag, and write/commit path #7535. Review the last commit.

🤖 Generated with Claude Code

github-actions · 2026-07-15T23:01:22Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

coderabbitai · 2026-07-21T00:25:02Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 2ec8abc1-95ae-439c-9950-9bc5250e80fc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch will/overlay-benchmarks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

wjones127 · 2026-07-21T00:29:31Z

Benchmark results

Rebased onto latest main and stacked on #7549 (now itself rebased onto current main — was 28 commits behind and conflicting). Ran the full suite below after fixing two API drifts the benchmark helpers had picked up (DataOverlayFile(offsets=...) not shared_offsets=, and the release-build opt-in env var is LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES). All 65 tests pass.

Setup: 1M rows, local disk (NVMe), release-no-lto build (full LTO release hit a reproducible rustc/LLVM crash on the build machine, unrelated to this PR).

Manifest growth (`test_overlay_manifest.py`)

Bytes added to the manifest per overlay layer — roaring compresses contiguous coverage noticeably better than strided:

overlays	1% contiguous	1% stride	10% stride
1	16.4 KB/ea	40.2 KB/ea	253 KB/ea
4	10.3 KB/ea	25.2 KB/ea	158 KB/ea
16	8.8 KB/ea	21.4 KB/ea	134 KB/ea
64	8.4 KB/ea	20.5 KB/ea	128 KB/ea

(64 layers @ 10% stride → ~8.2 MB manifest total.)

Write cost (`test_overlay_write.py`)

Updating 1% of rows via an overlay vs. the existing rewrite-based mechanisms. narrow = id + val; wide = id + val + 64-d float32 payload:

approach	narrow: read / persisted	wide: read / persisted
update	6.2 MB / 100 KB	8.7 MB / 2.66 MB
merge_insert	2.4 MB / 101 KB	2.4 MB / 2.66 MB
merge_insert (indexed)	12.7 MB / 100 KB	15.3 MB / 2.66 MB
overlay	143 B / 105 KB	143 B / 105 KB
full_rewrite	6.2 MB / 6.17 MB	262 MB / 262 MB

The rewrite-based approaches delete-and-reinsert whole rows, so their cost scales with row width. An overlay only ever touches the changed column: read bytes stay ~0 and persisted bytes stay constant (~105 KB, dominated by the coverage-bitmap + manifest overhead) regardless of how wide the row is — the advantage over update/merge_insert grows directly with row width, and vs. full_rewrite it's already ~60-2500x smaller at this width.

Read scaling (`test_overlay_read.py`)

1M-row int32 scan/take, strided coverage, v2.0 format:

overlays	take (mean)	scan (mean)
0	~0.25 ms	~1.7 ms
4	~0.7 ms	~7-8 ms
16	~1.2 ms	~40 ms

take stays roughly flat (rank pushdown means only the requested rows are touched).
scan degrades close to linearly with overlay-layer count (~24x at 16 layers) — this is the existing route_overlays cost profile (offset-major bitmap probing), not something this PR changes. It's the strongest argument for overlay compaction.
Wide embeddings (3072-d): scan is ~600-870 ms regardless of overlay count, since base I/O volume dominates over overlay-routing cost at that width.

cc for the demo this week: the standout numbers are the overlay write cost (143 B read, constant persisted bytes independent of row width) and the scan-cost-vs-layer-count curve as the motivating case for compaction.

codecov · 2026-07-21T00:58:20Z

Codecov Report

❌ Patch coverage is 83.44710% with 97 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/scanner.rs	74.55%	39 Missing and 46 partials ⚠️
rust/lance/src/io/exec/scalar_index.rs	55.55%	4 Missing ⚠️
.../lance/src/dataset/mem_wal/scanner/point_lookup.rs	0.00%	0 Missing and 3 partials ⚠️
rust/lance/src/dataset/overlay.rs	98.43%	0 Missing and 3 partials ⚠️
rust/lance/src/io/exec/filtered_read.rs	90.47%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Adds a Python benchmark suite for data overlay files in python/ci_benchmarks/, covering three areas: - manifest size growth vs. number of overlays and coverage, - bytes written updating 1% of a column via an overlay vs. update / merge_insert / full rewrite, across narrow and wide rows, - take and scan time + cold read IO as overlay layers, coverage fraction, fragmentation, and data type vary. Shared dataset/overlay construction and measurement helpers live in ci_benchmarks/overlays.py and build fixtures through the public lance.LanceOperation.DataOverlay binding. The write benchmark verifies each approach actually performs the update and splits persisted bytes into data vs. metadata, since the rewrite mechanisms are delete-and- reinsert (whole rows, all columns) while an overlay writes only the changed column. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The overlay read benchmark's take/scan scaling sweep only covered a 4-byte int32 value column, where the base read is trivial and per-layer merge cost dominates. ML overlays typically target wide vector columns, so add test_overlay_read_wide: the same overlays x coverage-pattern grid on a 3072-d float32 embedding (12 KiB/row). Thread an embedding_dim through the overlay helpers so the value width is configurable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… API `DataOverlayFile` takes a single `offsets` param, not `shared_offsets`, and the release-build opt-in env var is `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES` (not `LANCE_ENABLE_DATA_OVERLAY_FILES`). Both diverged from the benchmark helper after PR #7540 landed the real Python bindings; commit()'s returned handle skipped flag validation so this only surfaced on a fresh dataset open.

Self-review pass on the overlay benchmark suite: - Remove the unused `base_path` parameter of `commit_overlay_layers` (the body derives its data dir from the dataset URI) and the dead `dir_size` helper; both had no effect and misled the reader. - Rename `value_type` -> `_value_type` to match the module's private helpers. - Document the `{u64::MAX - version}.manifest` naming in `manifest_size`. - Guard the read and manifest benchmark fixtures: assert a covered cell reads back the overlaid value and that overlays grow the manifest, so a silently-broken overlay can't let the benchmark pass while measuring plain base reads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wjones127 · 2026-07-23T17:13:29Z

+def _gen_values(
+    dtype: str,
+    n: int,
+    rng: np.random.Generator,
+    embedding_dim: int = EMBEDDING_DIM,
+) -> pa.Array:
+    if dtype == "int32":
+        return pa.array(rng.integers(0, 1 << 30, size=n, dtype=np.int32))
+    if dtype == "embedding":
+        flat = rng.random(n * embedding_dim, dtype=np.float32)
+        return pa.FixedSizeListArray.from_arrays(pa.array(flat), embedding_dim)
+    raise ValueError(f"unknown overlay benchmark dtype {dtype!r}")


question: other tests use from lance._datagen import rand_batches, can we use that here?

I looked at it, but rand_batches is byte-budgeted (num_batches/batch_size_bytes) with no exact row-count control, and it randomizes every column. These fixtures need an exact num_rows with rows_per_file == num_rows (single fragment, so the take indices and per-fragment coverage offsets are well-defined), so I kept the small custom generator. It also avoids the datagen-feature build requirement (is_datagen_supported()). Happy to switch if you'd prefer the dependency — let me know.

Done — switched to it (ed64118, 1be2bec). rand_batches only sized batches by bytes, but the underlying lance-datagen builder already has into_reader_rows, so I exposed a rows_per_batch option on the binding (mutually exclusive with batch_size_bytes) and make_base_dataset now uses it for an exact row count. Overlay values keep their own generator so each layer stays distinct from the base (the read-visibility guard relies on that). Built with --features datagen and verified: exact row counts (single + multi-fragment), the int32/embedding schemas, overlay-visible-on-read, and the 1M-row manifest case all pass.

…t read benchmark - README: remove the per-benchmark descriptions; keep it to general suite advice. - Manifest/read benchmarks: report via record_property only, no printing. - Drop test_overlay_read_dtype: its int32 and embedding read costs are already covered by test_overlay_read_scaling (int32) and test_overlay_read_wide (embedding) across the full overlay-layer sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The write-cost comparison (overlay vs. update / merge_insert / full rewrite) is a useful one-off measurement but not something we need to keep re-running in CI for regressions. Remove it, along with the helpers it exclusively used (file_sizes, written_breakdown, proc_io_counters, and the make_base_dataset payload column). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The datagen binding could only size generated batches by bytes. The underlying lance-datagen builder already supports an exact row count (into_reader_rows), so expose it: rand_batches gains a rows_per_batch option, mutually exclusive with batch_size_bytes. This lets callers request an exact number of rows per batch rather than approximating via a byte budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the hand-rolled base-dataset generator in make_base_dataset with lance._datagen.rand_batches (row-count mode), matching the other ci_benchmarks suites. One batch per fragment; num_rows must be a multiple of rows_per_file. Overlay values keep their own generator so each layer stays distinct from the base (the read-visibility guard depends on that). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions Bot added A-python Python bindings chore labels Jun 30, 2026

wjones127 force-pushed the will/overlay-benchmarks branch 2 times, most recently from 1156dd2 to 687507f Compare June 30, 2026 20:26

wjones127 mentioned this pull request Jun 30, 2026

feat: data overlay files — model, feature flag, and write/commit path #7535

Merged

3 tasks

wjones127 force-pushed the will/overlay-benchmarks branch from 687507f to 92734fa Compare June 30, 2026 21:25

wjones127 force-pushed the will/overlay-python-commit branch from d1e3be6 to d1fa12b Compare June 30, 2026 23:12

wjones127 force-pushed the will/overlay-benchmarks branch from 92734fa to 9a09276 Compare June 30, 2026 23:12

wjones127 mentioned this pull request Jul 2, 2026

feat: resolve data overlay files on the take (and scan) read path #7536

Merged

wjones127 force-pushed the will/overlay-python-commit branch 3 times, most recently from 91b3987 to ecfb29c Compare July 8, 2026 22:13

wjones127 force-pushed the will/overlay-python-commit branch from 6d52845 to 968d024 Compare July 15, 2026 19:52

Base automatically changed from will/overlay-python-commit to main July 15, 2026 23:01

wjones127 force-pushed the will/overlay-benchmarks branch from 40a4a5c to c3837ce Compare July 21, 2026 00:24

wjones127 and others added 4 commits July 23, 2026 09:42

wjones127 force-pushed the will/overlay-benchmarks branch from c3837ce to 8204329 Compare July 23, 2026 16:53

wjones127 commented Jul 23, 2026

View reviewed changes

wjones127 and others added 4 commits July 23, 2026 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(python): add data overlay benchmark suite#7544

test(python): add data overlay benchmark suite#7544
wjones127 wants to merge 8 commits into
mainfrom
will/overlay-benchmarks

wjones127 commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

Review skipped

Uh oh!

wjones127 commented Jul 21, 2026

Uh oh!

codecov Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 Jul 23, 2026

Uh oh!

wjones127 Jul 23, 2026

Uh oh!

wjones127 Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wjones127 commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

wjones127 commented Jul 21, 2026

Benchmark results

Manifest growth (test_overlay_manifest.py)

Write cost (test_overlay_write.py)

Read scaling (test_overlay_read.py)

Uh oh!

codecov Bot commented Jul 21, 2026

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

Manifest growth (`test_overlay_manifest.py`)

Write cost (`test_overlay_write.py`)

Read scaling (`test_overlay_read.py`)