Skip to content

Replace 8KB data page size recommendation with tradeoff guidance - #223

Open
alkis wants to merge 4 commits into
apache:productionfrom
alkis:page-size-guidance
Open

alkis wants to merge 4 commits into
apache:productionfrom
alkis:page-size-guidance

Conversation

@alkis

@alkis alkis commented Nov 19, 2026 •

Copy link
Copy Markdown

The configuration guide recommends 8KB data pages. parquet-java, Arrow C++, Arrow Rust, Iceberg, DuckDB, and Polars all default to 1MB. The dev list thread "[DISCUSS] Remove the 8KB page size recommendation" agreed to remove the 8KB number.

This change replaces the number with the tradeoffs of small and large pages. It tells readers to start with the default of their writer and to benchmark their own workload.

This pull request and its description were written by Isaac.

alkis and others added 4 commits November 19, 2026 16:47
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@JigaoLuo

Copy link
Copy Markdown

Hi Alkis,

Thanks for taking this on. I was just rereading the original page: https://parquet.apache.org/docs/file-format/configurations/
(so even without the change you made

One thing that's not clear to me, and probably won't be clear to other readers either: how does the rowgroup size configuration relate to page size? I'm not sure it makes sense to mention it here, since it could get complicated.

It's also an open question: DuckDB uses (or used to use) a single page per column chunk, since I assume they didn't see any benefit in having both pages and rowgroups.

at a time; this is not the IO chunk. We recommend 8KB for page sizes.
### Data Page Size
Data page size is a tradeoff. Small pages are better for point lookups, because a
reader decompresses and decodes less data to read one value. Large pages need less

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit: fundamentally, a small read size would reduce read amplification, and the following compute amplification from decompression and decoding too.

@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

See also related mailing list discussion: https://lists.apache.org/thread.html/4g8ng2mbrzj01g43dp8fcx7dz49rs529

@alamb alamb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alkis -- this makes sense to me

Data page size is a tradeoff. Small pages are better for point lookups, because a
reader decompresses and decodes less data to read one value. Large pages need less
metadata, because a column chunk has fewer page headers and fewer page index
entries. Large pages also compress and encode better.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it might help to qualify this like

Suggested change
entries. Large pages also compress and encode better.
entries. Large pages also typically compress and encode better.

@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

One thing that's not clear to me, and probably won't be clear to other readers either: how does the rowgroup size configuration relate to page size? I'm not sure it makes sense to mention it here, since it could get complicated.

Maybe we can do a follow up PR trying to explain the use for row groups and the tradeoffs

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating this @alkis 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants