Today — crates/oxidelake-device/src/cuda/ops.rs:392-409 allocates and builds the device hash table (capacity = 2 × right.num_rows, oxide_hj_build) inside hash_join, i.e. once per probe batch, although device_build (crates/oxidelake-compute/src/operator.rs:422-443) already uploads the build columns once. The aggregate sizes its group table by input rows (ops.rs:773, capacity = 2 × input.num_rows) and reads back cap entries every call (:540-544); with the whole-input aggregate a 10M-row GROUP BY allocates 2²⁵ slots × 12 B plus cap × 8 B per aggregate and copies it all back.
Why it is worth fixing — join cost on device is O(probe batches × build rows) and aggregate transfer is O(rows) regardless of cardinality; both are the audit's "P1 (rest)" and they dominate once the aggregate streams.
Fix — GpuBackend::prepare_join_build returning an opaque device table reused across batches (dropped with the operator); size the group table from a distinct-key estimate (HyperLogLog on the host over the first batch, or a two-pass count) and read back only occupied slots.
Done when — the on-demand GPU job records before/after timings for the conformance join and a 10M-row GROUP BY k (100 keys); bytes_d2h for the aggregate is proportional to groups, not rows.
Today —
crates/oxidelake-device/src/cuda/ops.rs:392-409allocates and builds the device hash table (capacity = 2 × right.num_rows,oxide_hj_build) insidehash_join, i.e. once per probe batch, althoughdevice_build(crates/oxidelake-compute/src/operator.rs:422-443) already uploads the build columns once. The aggregate sizes its group table by input rows (ops.rs:773,capacity = 2 × input.num_rows) and reads backcapentries every call (:540-544); with the whole-input aggregate a 10M-rowGROUP BYallocates 2²⁵ slots × 12 B pluscap × 8 Bper aggregate and copies it all back.Why it is worth fixing — join cost on device is O(probe batches × build rows) and aggregate transfer is O(rows) regardless of cardinality; both are the audit's "P1 (rest)" and they dominate once the aggregate streams.
Fix —
GpuBackend::prepare_join_buildreturning an opaque device table reused across batches (dropped with the operator); size the group table from a distinct-key estimate (HyperLogLog on the host over the first batch, or a two-pass count) and read back only occupied slots.Done when — the on-demand GPU job records before/after timings for the conformance join and a 10M-row
GROUP BY k(100 keys);bytes_d2hfor the aggregate is proportional to groups, not rows.