Repository navigation
Add "Changing the Format" to Developer Guidelines - #220
julienledem wants to merge 3 commits into
Conversation
Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
|
Can rationalize this with what is in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md (the content seems to be duplicate with what is there) |
This new page is going in more details to what's mentioned in the first three points here:
The rest of that section is then on how we roll it out. |
alamb
left a comment
There was a problem hiding this comment.
I think it is looking pretty good. Thank you @julienledem and @emkornfield
| This document defines categories of changes to the Parquet Format, and proposes | ||
| a set of criteria to evaluate future potential changes. | ||
|
|
||
| Changes which are forward compatible are always welcome. Partially compatible |
There was a problem hiding this comment.
I think it would be useful here to give rationale for these preferences
Instead of "welcome" I would suggest a term more like "typically are less challenging to standardize" and explain why
Maybe something like
Changes which are forward compatible are typically less chanllenging to standardize, as
they can not adversely impact older readers by definition. Partially compatible
|
|
||
| #### Reader behavior | ||
|
|
||
| Ordered by desirability: |
| | **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". | | ||
| | **Incorrect Read** | Reads data but unknowingly misinterprets it (ex: semantic changes to an existing field like stats). | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.<br>**This should never happen.** | We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | | ||
|
|
||
| ### Proposal: Evaluation system for changes to Parquet |
There was a problem hiding this comment.
is this still a proposal? Or would we change this to "Evaluation system ...." before merge?
There was a problem hiding this comment.
It seems very similar to I tihnk this https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#releases
I think @emkornfield 's idea of combining the section (maybe take the wording from README here, and leave a back link?) would be good
There was a problem hiding this comment.
Yes, I was thinking of updating that section in the other repo after we merge this one. I would link to this page at the beginning of the "Compatibility and Feature Enablement" section inviting people who want more details to read it while leaving the three points (backward/Forward compatible and incompatible) which are a good summary of this doc for people who just want the jist.
The text that follows focuses on how implementations can turn things on and off which is not covered in this page.
There was a problem hiding this comment.
This seems OK but perhaps suboptimal. Having all the context in one doc makes sense to me and makes things easier to find. I already find myself forgetting where certain parts of contributing live (e.g. the propsoal process).
Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
|
I have updated the PR per feedback and proposed a follow up to Micah's comment. |
emkornfield
left a comment
There was a problem hiding this comment.
I'm OK with the content but would really prefer a single source of truth for this content in one repo (format).
|
@emkornfield We could move this to a Markdown file in the parquet-format repo. Would that be better? |
As I understood it, @emkornfield was concerned about the fact that this page has some non trivial overlap with the stuff already in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement One problem with https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement is that it does not appear on the parquet.apache.org site. Perhaps instead of moving the content in this PR to paruqet-format, we could instead consolidate the the content in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement into this page and then leave a pointer to the new page on the parquet site |
Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
|
This change is now in apache/parquet-format#645 |
|
@alamb with the goal of not making CONTRIBUTING.md too bloated with details and since the existing statement in it is a good summary.
|
alamb
left a comment
There was a problem hiding this comment.
Thanks @julienledem this makes sense to me and I think consolidating the advice into a single page that is then rendered on the website makes a huge amount of sense.
Before merging this PR we should merge the one to parquet-format (apache/parquet-format#645) and then update the parquet-format git submodule pin here
I tested this change locally and it looks good 👍
| linkTitle: "Changing the Format" | ||
| weight: 5 | ||
| description: > | ||
| Guidance for proposing changes to the Parquet format specification |
There was a problem hiding this comment.
👍 I like this idea (of keeping all the content in parquet-format and including it in the same way)


I created a page reflecting the content of the guidelines we have been discussing.
https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a#heading=h.lkllwrkuv6vx