Skip to content

feat: derive eager scan statistics from selected files - #125

Merged
gabotechs merged 5 commits into
apache:mainfrom
alexanderbianchi:codex/eager-file-statistics
Nov 20, 2026
Merged

gabotechs merged 5 commits into
apache:mainfrom
alexanderbianchi:codex/eager-file-statistics

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Eager scans retained whole-snapshot row estimates even after file pruning, while individual output partitions reported unknown counts. A taxi scan selecting two of seven daily files therefore reported 175,000 rows instead of 50,000.

Follow up on #113 and #114

Derive row estimates from the retained FileScanTasks for the whole assignment or the requested partition. The two-file scan now reports 50,000 rows total and 25,000 per partition. This also works for explicitly supplied file groups.

Adapts the statistics portion of DFD #787:

Using Stats

alexanderbianchi and others added 5 commits November 19, 2026 20:06
Adapt selected-file statistics from datafusion-contrib/datafusion-distributed#787 while retaining conservative row-count precision and projected output-byte estimates.

Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
Signed-off-by: alex.bianchi <alex.bianchi@datadoghq.com>
@gabotechs
gabotechs merged commit 06d4faa into apache:main Nov 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants