tp: let INTERVAL INTERSECTION's PER keys be of any one type - #7689
Merged
Merged
Conversation
LalitMaganti
added this pull request to stack #7680
September 29, 2026 23:53
🎨 Perfetto UI Builds & Tests
|
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 00:19
279f412 to
7f24939
Compare
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
3 times, most recently
from
October 1, 2026 00:46
709afef to
31a4667
Compare
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 00:52
31a4667 to
55597bf
Compare
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 00:57
55597bf to
714d4c1
Compare
LalitMaganti
marked this pull request as ready for review
October 1, 2026 01:24
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 01:29
714d4c1 to
9ce7550
Compare
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
2 times, most recently
from
October 1, 2026 04:14
24de7b9 to
99fc465
Compare
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 04:30
99fc465 to
402ccc8
Compare
FlexVector::resize allocated exactly the size asked for, so growing a buffer a batch at a time with resize() reallocated and copied it every time: quadratic. BitVector::resize, built on it, did the same. Code worked around it with hand-rolled doubling, or by calling reserve(), which already grew geometrically, before every resize(). resize() now grows capacity as reserve() does: to at least 1.5x when it has to reallocate. A resize of an empty vector still allocates exactly what is asked for. The workarounds in the tree operators go, as does BitVector::reserve(), which existed only for them.
PER keys had to be integers and were widened to Int64 on every batch. They are now laid out as-is by a KeyEncoder, built on RowLayout, and compared as bytes. Integers of every width, doubles and strings can all be keys; integers of different widths agree, and nulls agree as under GROUP BY. A key must hold the same type in every operand. KeyEncoder works out each key column's type on the first batch and picks a writer for it; later batches only check that the columns are shaped as before. RowLayout::Slot is 8 bytes and passed by value, and rows are written through a PERFETTO_RESTRICT pointer, so the slot stays in registers while a column is written (2048 Int64 keys: 970 -> 708 ns; one key: 6.1 -> 4.3 ns). FlatColumnReader, beside ColumnView, reads a flat column through its selection and validity, for INTERVAL INTERSECTION's ts and dur and for string keys.
LalitMaganti
force-pushed
the
dev/lalitm/intersect-any-keys
branch
from
October 1, 2026 04:42
402ccc8 to
80ada08
Compare
LalitMaganti
removed this pull request from stack #7680
October 1, 2026 04:43
LalitMaganti
changed the base branch from
main
to
dev/lalitm/flex-vector-resize
October 1, 2026 04:43
LalitMaganti
added this pull request to stack #7716
October 1, 2026 04:43
sashwinbalaji
approved these changes
Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets INTERVAL INTERSECTION's PER keys be of any one type: integers of every width, doubles and strings, not just integers.
Why
PER keys had to be integers, and were widened to Int64 on every batch. Grouping intervals by a name or any other string column was impossible.
Changes
KeyEncoder(core/exec) lays out each row's keys as bytes withRowLayout, so equal keys have equal bytes and are compared as such. Integers of different widths agree, and nulls agree with each other, as under GROUP BY.FlatColumnReader<T>, besideColumnView, reads a flat column through its selection and validity. INTERVAL INTERSECTION's ts and dur and string keys use it, as does INTERVAL FLATTEN (tp: add INTERVAL FLATTEN to pipelines #7692).RowLayout::Slotshrinks to 8 bytes and is passed by value, and rows are written through a newPERFETTO_RESTRICTpointer, so the slot stays in registers while a column is written. The macro is__restrict__because the macOS SDK defines__restrictaway to nothing in C++.Example
Benchmarks
KeyEncoder, against writing keys with a type switch per column on every batch (best of 10, release build):BM_KeyEncoderInt64(2048 rows)BM_KeyEncoderInt64Nulls(1 in 8 null)BM_KeyEncoderInt64OneRowTesting
New
key_encoder_unittest.cc, and connection tests for keys of every type, nulls and mismatched types.