Follow-up from #7549 (approved). Overlay/invalidation masking works today but is pieced together across many places, which makes it easy to get subtly wrong — e.g. the address↔row-id domain composition had to be threaded to every masking site to be correct under stable row ids.
The concept currently spans:
- 3 granularities: row-offset (
overlay_exclusion_offsets), fragment, and FTS segment.
- 4 per-index-type scanner code paths (vector ANN, scalar V1
MaterializeIndexExec, scalar V2 FilteredRead, FTS) — each computes stale rows, builds a block, threads it into the node, and assembles a separate flat-scan + take "re-add" path.
- 4 node-side representations:
RowAddrMask block AND-ed into DatasetPreFilter.final_mask (ANN + scalar V1) vs. fragment subtraction (EvaluatedIndex::without_fragments, scalar V2) vs. segment selection (FTS).
Proposal: a single OverlayInvalidation component per index that (a) computes stale rows once in one canonical domain, (b) exposes a uniform {block, re-add} pair, and (c) does the address↔row-id translation in exactly one place. That collapses the per-type scanner orchestration + the node-side knobs and makes the domain-composition bug impossible by construction.
Also fold in the DatasetPreFilter rename — it now assembles the full index-result mask at plan time (deletions + filter + overlay block), so "PreFilter" is misleading (noted in review).
Note: deletions (dropped everywhere; computed/applied by the nodes) and invalidations (moved index→brute-force; computed by the scanner) legitimately use different mechanisms — this is about consolidating the invalidation half, not merging the two concepts.
Follow-up from #7549 (approved). Overlay/invalidation masking works today but is pieced together across many places, which makes it easy to get subtly wrong — e.g. the address↔row-id domain composition had to be threaded to every masking site to be correct under stable row ids.
The concept currently spans:
overlay_exclusion_offsets), fragment, and FTS segment.MaterializeIndexExec, scalar V2FilteredRead, FTS) — each computes stale rows, builds a block, threads it into the node, and assembles a separate flat-scan + take "re-add" path.RowAddrMaskblock AND-ed intoDatasetPreFilter.final_mask(ANN + scalar V1) vs. fragment subtraction (EvaluatedIndex::without_fragments, scalar V2) vs. segment selection (FTS).Proposal: a single
OverlayInvalidationcomponent per index that (a) computes stale rows once in one canonical domain, (b) exposes a uniform{block, re-add}pair, and (c) does the address↔row-id translation in exactly one place. That collapses the per-type scanner orchestration + the node-side knobs and makes the domain-composition bug impossible by construction.Also fold in the
DatasetPreFilterrename — it now assembles the full index-result mask at plan time (deletions + filter + overlay block), so "PreFilter" is misleading (noted in review).Note: deletions (dropped everywhere; computed/applied by the nodes) and invalidations (moved index→brute-force; computed by the scanner) legitimately use different mechanisms — this is about consolidating the invalidation half, not merging the two concepts.