Skip to content

Add "Changing the Format" to Developer Guidelines - #220

Open
julienledem wants to merge 3 commits into
apache:productionfrom
julienledem:add_change_guidelines
Open

julienledem wants to merge 3 commits into
apache:productionfrom
julienledem:add_change_guidelines

Conversation

@julienledem

@julienledem julienledem commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

I created a page reflecting the content of the guidelines we have been discussing.
https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a#heading=h.lkllwrkuv6vx

format-changes

Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
Comment thread package.json Resolved
Comment thread package.json Resolved
@emkornfield

Copy link
Copy Markdown
Contributor

Can rationalize this with what is in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md (the content seems to be duplicate with what is there)

@julienledem

julienledem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member Author

Can rationalize this with what is in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md (the content seems to be duplicate with what is there)

This new page is going in more details to what's mentioned in the first three points here:

https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement

  1. Backward compatible. A file written under an older version of the format should be readable under a newer version of the format.
  2. Forward compatible. A file written under a newer version of the format with the feature enabled can be read under an older version of the format, but some metadata might be missing or performance might be suboptimal. Simply phrased, forward compatible means all data can be read back in an older version of the format. New logical types and new combinations of existing logical/physical types are considered forward compatible despite the loss of semantic meaning.
  3. Forward incompatible. A file written under a newer version of the format with the feature enabled cannot be read under an older version of the format (e.g. adding and using a new compression algorithm). It is expected any feature in this category will provide a signal to older readers, so they can unambiguously determine that they cannot properly read the file (e.g. via adding a new value to an existing enum).

The rest of that section is then on how we roll it out.
I'll add a link to the new page there to direct people who want more details. (but it's not in the same repo so I'll do that afterwards.

@alamb alamb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is looking pretty good. Thank you @julienledem and @emkornfield

Comment thread content/en/docs/format-changes.md Outdated
This document defines categories of changes to the Parquet Format, and proposes
a set of criteria to evaluate future potential changes.

Changes which are forward compatible are always welcome. Partially compatible

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be useful here to give rationale for these preferences

Instead of "welcome" I would suggest a term more like "typically are less challenging to standardize" and explain why

Maybe something like

Changes which are forward compatible are typically less chanllenging to standardize, as
they can not adversely impact older readers by definition. Partially compatible

Comment thread content/en/docs/format-changes.md Resolved
Comment thread content/en/docs/format-changes.md Resolved
Comment thread content/en/docs/format-changes.md Resolved
Comment thread content/en/docs/format-changes.md Resolved

#### Reader behavior

Ordered by desirability:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rendered Version:

Screenshot 2026-11-11 at 11 41 42 PM

Comment thread content/en/docs/format-changes.md Resolved
Comment thread content/en/docs/format-changes.md Resolved
Comment thread content/en/docs/format-changes.md Outdated
| **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". |
| **Incorrect Read** | Reads data but unknowingly misinterprets it (ex: semantic changes to an existing field like stats). | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.<br>**This should never happen.** | We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. |

### Proposal: Evaluation system for changes to Parquet

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still a proposal? Or would we change this to "Evaluation system ...." before merge?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems very similar to I tihnk this https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#releases

I think @emkornfield 's idea of combining the section (maybe take the wording from README here, and leave a back link?) would be good

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2026-11-11 at 11 51 01 PM

@julienledem julienledem Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking of updating that section in the other repo after we merge this one. I would link to this page at the beginning of the "Compatibility and Feature Enablement" section inviting people who want more details to read it while leaving the three points (backward/Forward compatible and incompatible) which are a good summary of this doc for people who just want the jist.
The text that follows focuses on how implementations can turn things on and off which is not covered in this page.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems OK but perhaps suboptimal. Having all the context in one doc makes sense to me and makes things easier to find. I already find myself forgetting where certain parts of contributing live (e.g. the propsoal process).

Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
@julienledem

Copy link
Copy Markdown
Member Author

I have updated the PR per feedback and proposed a follow up to Micah's comment.

@emkornfield emkornfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with the content but would really prefer a single source of truth for this content in one repo (format).

@julienledem

Copy link
Copy Markdown
Member Author

@emkornfield We could move this to a Markdown file in the parquet-format repo. Would that be better?
Then it would show up under https://parquet.apache.org/docs/file-format/ instead of https://parquet.apache.org/docs/contribution-guidelines/ (CC @alamb )

@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Then it would show up under https://parquet.apache.org/docs/file-format/ instead of https://parquet.apache.org/docs/contribution-guidelines/ (CC @alamb )

As I understood it, @emkornfield was concerned about the fact that this page has some non trivial overlap with the stuff already in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement

One problem with https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement is that it does not appear on the parquet.apache.org site. Perhaps instead of moving the content in this PR to paruqet-format, we could instead consolidate the the content in https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement into this page and then leave a pointer to the new page on the parquet site

Signed-off-by: Julien Le Dem <julien.ledem@datadoghq.com>
@julienledem

Copy link
Copy Markdown
Member Author

This change is now in apache/parquet-format#645

@julienledem

Copy link
Copy Markdown
Member Author

@alamb with the goal of not making CONTRIBUTING.md too bloated with details and since the existing statement in it is a good summary.
I did the following:

  • moved this page to parquet-format where it is linked from that section in CONTRIBUTING for details.
  • added a shell here so it is visible in the site in the same place.

@alamb alamb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @julienledem this makes sense to me and I think consolidating the advice into a single page that is then rendered on the website makes a huge amount of sense.

Before merging this PR we should merge the one to parquet-format (apache/parquet-format#645) and then update the parquet-format git submodule pin here

I tested this change locally and it looks good 👍

Screenshot 2026-11-21 at 5 19 28 AM Screenshot 2026-11-21 at 5 19 48 AM

linkTitle: "Changing the Format"
weight: 5
description: >
Guidance for proposing changes to the Parquet format specification

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I like this idea (of keeping all the content in parquet-format and including it in the same way)

@alamb alamb changed the title adding changes guidelines Add "Changing the Format" to Developer Guidelines Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants