Repository navigation
Conversation
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Hi Alkis, Thanks for taking this on. I was just rereading the original page: https://parquet.apache.org/docs/file-format/configurations/ One thing that's not clear to me, and probably won't be clear to other readers either: how does the rowgroup size configuration relate to page size? I'm not sure it makes sense to mention it here, since it could get complicated. It's also an open question: DuckDB uses (or used to use) a single page per column chunk, since I assume they didn't see any benefit in having both pages and rowgroups. |
| at a time; this is not the IO chunk. We recommend 8KB for page sizes. | ||
| ### Data Page Size | ||
| Data page size is a tradeoff. Small pages are better for point lookups, because a | ||
| reader decompresses and decodes less data to read one value. Large pages need less |
There was a problem hiding this comment.
One nit: fundamentally, a small read size would reduce read amplification, and the following compute amplification from decompression and decoding too.
|
See also related mailing list discussion: https://lists.apache.org/thread.html/4g8ng2mbrzj01g43dp8fcx7dz49rs529 |
| Data page size is a tradeoff. Small pages are better for point lookups, because a | ||
| reader decompresses and decodes less data to read one value. Large pages need less | ||
| metadata, because a column chunk has fewer page headers and fewer page index | ||
| entries. Large pages also compress and encode better. |
There was a problem hiding this comment.
nit: it might help to qualify this like
| entries. Large pages also compress and encode better. | |
| entries. Large pages also typically compress and encode better. |
Maybe we can do a follow up PR trying to explain the use for row groups and the tradeoffs |
The configuration guide recommends 8KB data pages. parquet-java, Arrow C++, Arrow Rust, Iceberg, DuckDB, and Polars all default to 1MB. The dev list thread "[DISCUSS] Remove the 8KB page size recommendation" agreed to remove the 8KB number.
This change replaces the number with the tradeoffs of small and large pages. It tells readers to start with the default of their writer and to benchmark their own workload.
This pull request and its description were written by Isaac.