Skip to content

Optimize filterless group contribution caps - #23

Merged
ila merged 2 commits into
mainfrom
codex/filterless-group-cap-topk
Aug 31, 2026
Merged

Optimize filterless group contribution caps#23
ila merged 2 commits into
mainfrom
codex/filterless-group-cap-topk

Conversation

@ila

@ila ila commented Aug 31, 2026

Copy link
Copy Markdown
Member

Thanks for reviewing.

Summary

  • Replace the filterless duplicate-aware DENSE_RANK group cap with ROW_NUMBER after the unique (PU, group) pre-aggregation.
  • Order by a stable hash(PU, group), matching the bounded-selection strategy already used by ordinary standard DP and SASS aggregates.
  • Verify that DuckDB rewrites the cap to a bounded per-PU top-k aggregate.

Why this is safe

The lower aggregate emits exactly one row per logical (PU, group), so duplicate-aware ranking is unnecessary. ROW_NUMBER <= C_u still retains at most C_u groups per PU. A hash collision can change which row wins, but cannot admit more than the cap. Distinct-value paths retain DENSE_RANK, where duplicate logical items must share a rank.

Validation

  • clean Ninja build from current main
  • build/release/test/unittest test/sql/dp_filterless.test (140 assertions)
  • C_u=1 and C_u=2 correctness coverage
  • C_u=3 with 100 PUs and 10 candidate groups: exactly 300 contributions retained
  • physical-plan regex requires UNNEST over arg_min_nulls_last for the C_u>1 top-k shape
  • real 1,000,000-row grouped C_u=3 query also retained exactly 300,000 contributions

@ila
ila merged commit 437af00 into main Aug 31, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant