feat: data overlay files — model, feature flag, and write/commit path #7535
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -825,6 +825,9 @@ impl FromPyObject<'_, '_> for PyLance<Fragment> { | |
| row_id_meta, | ||
| last_updated_at_version_meta, | ||
| created_at_version_meta, | ||
| // Overlays are not exposed to Python yet, and the reverse conversion | ||
| // does not export them, so this round-trip is overlay-free. | ||
| overlays: vec![], | ||
|
Comment on lines
+828
to
+830
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
rg -n -C5 \
'PyLance<Fragment>|FromPyObject.*Fragment|extract.*Fragment|Fragment.*(write|commit|append|update)' \
python rust/lanceRepository: lance-format/lance Length of output: 27436 🏁 Script executed: #!/bin/bash
set -eu
sed -n '800,910p' python/src/fragment.rs
printf '\n---\n'
rg -n -C3 'struct Fragment|overlays' rust/lance/src python/src | head -n 80Repository: lance-format/lance Length of output: 9915 🏁 Script executed: #!/bin/bash
set -eu
# Inspect the Rust Fragment definition and Python conversion code paths.
rg -n -C4 'pub struct Fragment|overlays:' rust/lance/src python/src
printf '\n=== fragment.rs around conversions ===\n'
sed -n '800,910p' python/src/fragment.rsRepository: lance-format/lance Length of output: 50374 🏁 Script executed: #!/bin/bash
set -eu
# Find where Fragment metadata is converted back to Python and whether overlays are included.
sed -n '876,930p' python/src/fragment.rs
printf '\n---\n'
sed -n '1,140p' rust/lance/src/dataset/fragment.rs
printf '\n---\n'
rg -n -C3 'overlay|overlays|DataOverlay|OverlayCoverage' rust/lance/src/lance_table rust/lance/src | head -n 120Repository: lance-format/lance Length of output: 17299 🏁 Script executed: #!/bin/bash
set -eu
# Check whether the Python package exposes overlays on Fragment at all.
rg -n -C3 'class Fragment|_Fragment|overlays|Fragment =' python/python/lance/fragment.py python/src/fragment.rs
printf '\n---\n'
sed -n '1,120p' python/python/lance/fragment.py
printf '\n---\n'
sed -n '876,930p' python/src/fragment.rsRepository: lance-format/lance Length of output: 9905 🌐 Web query:
💡 Result: The term "fragment overlays" is not a concept or feature within the Microsoft Pylance extension for Visual Studio Code [1][2]. Based on your query, it is highly likely that you are encountering a misunderstanding of terminology related to the LanceDB project. In the context of the Lance data format, "fragment overlays" (often referred to as "Data Overlay Files") are a technical feature used for data versioning, modifications, and storage [3]. Research indicates that the LanceDB project uses a Python interface, frequently referred to in repository code as Citations:
🏁 Script executed: #!/bin/bash
set -eu
# Inspect the public Python fragment API around FragmentMetadata and LanceFragment.
sed -n '59,140p' python/python/lance/fragment.py
printf '\n---\n'
sed -n '286,470p' python/python/lance/fragment.py
printf '\n---\n'
rg -n -C2 'overlays|FragmentMetadata|_Fragment' python/python/lance/fragment.py python/src/fragment.rsRepository: lance-format/lance Length of output: 21405 🏁 Script executed: #!/bin/bash
set -eu
# Inspect Python entry points that consume fragment collections for commit/write paths.
rg -n -C4 '_write_fragments_transaction|_write_fragments|fragments:|FragmentMetadata|LanceFragment' python/python/lance/fragment.py python/python/lance/dataset.py python/python/lance/transaction.py python/src/transaction.rsRepository: lance-format/lance Length of output: 41740 Preserve overlays in 🤖 Prompt for AI Agents |
||
| })) | ||
| } | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify that the row-moving Update conflict is row-level, not fragment-level.
As written, this reads as "any row-moving Update touching an overlaid fragment conflicts." In practice the conflict resolver only retries when the specific rows the update relocated intersect the overlay's covered offsets (checked by diffing deletion vectors at commit-finish time); a row-moving update on the same fragment whose moved rows are disjoint from the overlay's coverage succeeds without conflict.
As per coding guidelines: "Describe algorithms in full detail, including parameters, precision, ordering, normalization bounds, and implementation steps; do not reference an algorithm by name alone."
✏️ Suggested wording
📝 Committable suggestion
🤖 Prompt for AI Agents
Source: Coding guidelines