Skip to content

perf: place broadcast runtime filters before costly predicates - #2909

Draft
wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:runtime-filter-broadcast-placement
Draft

wirybeaver wants to merge 1 commit into
apache:mainfrom
wirybeaver:runtime-filter-broadcast-placement

Conversation

@wirybeaver

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Implements Task 5 of #2700.

Rationale for this change

In the TPC-DS SF100 results for #2749, runtime filtering increased median latency for Q37 and Q82. In Q82, the broadcast probe hashed every inventory row before a cheap quantity range and a join against 70 build rows.

What changes are included in this PR?

For a same-stage broadcast probe, AQE places RuntimeFilterApplyExec immediately below the lowest costly FilterExec on the eligible scan path. Cheaper filters below that point run first. A broadcast probe with only cheap predicates skips the apply. The planner uses six non-leaf expression nodes as a heuristic for predicate cost.

An undecided staged build can still produce a summary for partitioned fallback, where filtering the probe saves shuffle work.

Are there any user-facing changes?

With ballista.planner.runtime_filter.enabled = true, the placement of runtime filters in broadcast probe plans can change.

Test

The distributed join test compares results with filtering enabled and disabled, including NULL-safe equality. Its execution plan confirms that a filter was bound and pruned rows. A local TPC-H Q19 SF10 check also matched results and reported 59.60 million rows pruned. The SF100 latency impact on Q37 and Q82 is still open.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant