Skip to content

tp: let INTERVAL INTERSECTION's PER keys be of any one type - #7689

Merged
LalitMaganti merged 3 commits into
mainfrom
dev/lalitm/intersect-any-keys
Oct 2, 2026
Merged

LalitMaganti merged 3 commits into
mainfrom
dev/lalitm/intersect-any-keys

Conversation

@LalitMaganti

@LalitMaganti LalitMaganti commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Lets INTERVAL INTERSECTION's PER keys be of any one type: integers of every width, doubles and strings, not just integers.

Why

PER keys had to be integers, and were widened to Int64 on every batch. Grouping intervals by a name or any other string column was impossible.

Changes

  • KeyEncoder (core/exec) lays out each row's keys as bytes with RowLayout, so equal keys have equal bytes and are compared as such. Integers of different widths agree, and nulls agree with each other, as under GROUP BY.
  • A key must hold the same type in every operand; otherwise the query fails, naming the key.
  • The encoder works out each column's type on the first batch and picks a writer for it. Later batches only check that the columns are shaped as before.
  • FlatColumnReader<T>, beside ColumnView, reads a flat column through its selection and validity. INTERVAL INTERSECTION's ts and dur and string keys use it, as does INTERVAL FLATTEN (tp: add INTERVAL FLATTEN to pipelines #7692).
  • RowLayout::Slot shrinks to 8 bytes and is passed by value, and rows are written through a new PERFETTO_RESTRICT pointer, so the slot stays in registers while a column is written. The macro is __restrict__ because the macOS SDK defines __restrict away to nothing in C++.

Example

INTERVAL INTERSECTION OF (a AS a, b AS b) PER name
|> SELECT ts, dur, name

Benchmarks

KeyEncoder, against writing keys with a type switch per column on every batch (best of 10, release build):

Benchmark Before After Ratio
BM_KeyEncoderInt64 (2048 rows) 970 ns 708 ns 0.73x
BM_KeyEncoderInt64Nulls (1 in 8 null) 1571 ns 1036 ns 0.66x
BM_KeyEncoderInt64OneRow 6.10 ns 4.26 ns 0.70x

Testing

New key_encoder_unittest.cc, and connection tests for keys of every type, nulls and mismatched types.

@LalitMaganti
LalitMaganti added this pull request to stack #7680 September 29, 2026 23:53
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 279f412 to 7f24939 Compare October 1, 2026 00:19
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch 3 times, most recently from 709afef to 31a4667 Compare October 1, 2026 00:46
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 31a4667 to 55597bf Compare October 1, 2026 00:52
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 55597bf to 714d4c1 Compare October 1, 2026 00:57
@LalitMaganti
LalitMaganti marked this pull request as ready for review October 1, 2026 01:24
@LalitMaganti
LalitMaganti requested a review from a team as a code owner October 1, 2026 01:24
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 714d4c1 to 9ce7550 Compare October 1, 2026 01:29
Base automatically changed from dev/lalitm/row-layout to main October 1, 2026 01:30
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch 2 times, most recently from 24de7b9 to 99fc465 Compare October 1, 2026 04:14
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 99fc465 to 402ccc8 Compare October 1, 2026 04:30
FlexVector::resize allocated exactly the size asked for, so growing a
buffer a batch at a time with resize() reallocated and copied it every
time: quadratic. BitVector::resize, built on it, did the same. Code
worked around it with hand-rolled doubling, or by calling reserve(),
which already grew geometrically, before every resize().

resize() now grows capacity as reserve() does: to at least 1.5x when it
has to reallocate. A resize of an empty vector still allocates exactly
what is asked for. The workarounds in the tree operators go, as does
BitVector::reserve(), which existed only for them.
PER keys had to be integers and were widened to Int64 on every batch.
They are now laid out as-is by a KeyEncoder, built on RowLayout, and
compared as bytes. Integers of every width, doubles and strings can all
be keys; integers of different widths agree, and nulls agree as under
GROUP BY. A key must hold the same type in every operand.

KeyEncoder works out each key column's type on the first batch and picks
a writer for it; later batches only check that the columns are shaped
as before. RowLayout::Slot is 8 bytes and passed by value, and rows are
written through a PERFETTO_RESTRICT pointer, so the slot stays in
registers while a column is written (2048 Int64 keys: 970 -> 708 ns;
one key: 6.1 -> 4.3 ns).

FlatColumnReader, beside ColumnView, reads a flat column through its
selection and validity, for INTERVAL INTERSECTION's ts and dur and for
string keys.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/intersect-any-keys branch from 402ccc8 to 80ada08 Compare October 1, 2026 04:42
@LalitMaganti
LalitMaganti removed this pull request from stack #7680 October 1, 2026 04:43
@LalitMaganti
LalitMaganti changed the base branch from main to dev/lalitm/flex-vector-resize October 1, 2026 04:43
@LalitMaganti
LalitMaganti added this pull request to stack #7716 October 1, 2026 04:43
Base automatically changed from dev/lalitm/flex-vector-resize to main October 1, 2026 22:12
Comment thread src/trace_processor/core/exec/column_view.h
Comment thread src/trace_processor/core/exec/key_encoder.cc Outdated
@LalitMaganti
LalitMaganti merged commit 6757a30 into main Oct 2, 2026
28 checks passed
@LalitMaganti
LalitMaganti deleted the dev/lalitm/intersect-any-keys branch October 2, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants