Repository navigation
MINOR: adding changes guidelines #645
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
julienledem
wants to merge
2
commits into
apache:master
Choose a base branch
from
julienledem:add_change_guidelines
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| <!-- | ||
| - Licensed to the Apache Software Foundation (ASF) under one | ||
| - or more contributor license agreements. See the NOTICE file | ||
| - distributed with this work for additional information | ||
| - regarding copyright ownership. The ASF licenses this file | ||
| - to you under the Apache License, Version 2.0 (the | ||
| - "License"); you may not use this file except in compliance | ||
| - with the License. You may obtain a copy of the License at | ||
| - | ||
| - http://www.apache.org/licenses/LICENSE-2.0 | ||
| - | ||
| - Unless required by applicable law or agreed to in writing, | ||
| - software distributed under the License is distributed on an | ||
| - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| - KIND, either express or implied. See the License for the | ||
| - specific language governing permissions and limitations | ||
| - under the License. | ||
| --> | ||
|
|
||
| Changing the Format | ||
| ==== | ||
|
|
||
| ## Scope | ||
|
|
||
| When making a change to the Parquet Format, we have to manage problems that | ||
| arise from the reader and the writer implementing a different version of the | ||
| spec. | ||
|
|
||
| This document defines categories of changes to the Parquet Format, and proposes | ||
| a set of criteria to evaluate future potential changes. | ||
|
|
||
| Changes which are forward compatible do not adversely impact the user and are easier to standardize. | ||
| Partially compatible changes are encouraged with a preference for consumers to error out when | ||
| un-parsable data is observed. Changes which would produce an incorrect read | ||
| require a PAR magic number version change and should be done sparingly. | ||
|
|
||
| ## Definitions | ||
|
|
||
| A change to the Parquet format can have the following properties: | ||
|
|
||
| **Compatibility**: We narrow this down to producing the exact same tabular data | ||
| while possibly ignoring secondary data structures (like bloom filters, indices, | ||
| stats, …). Compatible implies Correct. | ||
|
|
||
| - *Backward compatibility*: new readers can read files written by older writers. | ||
| - *Forward compatibility*: old readers can read files written by newer writers. | ||
|
|
||
| **Correctness**: Avoid reading incorrect data unknowingly. | ||
|
|
||
| - *Incorrect read*: a reader producing different data than the writer intended. | ||
| - *Correct failed read*: a reader producing an error when it can not read the | ||
| data in the file. | ||
|
|
||
| **Partial Compatibility**: when the reader can read a subset of the file (and | ||
| knows which subset). | ||
|
|
||
| - If only the subset of columns supported is accessed, then the read is | ||
| successful. | ||
| - The reader can elect to read only the subset of the data it understands | ||
| depending on the use case. | ||
|
|
||
|  | ||
|
|
||
| ## Dealing with incompatible changes | ||
|
|
||
| ### Backward compatibility | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we mention something about deprecation here? If we intend to drop a field, we should likely be deprecating it in the current version so readers/writers have time to change if needed and then dropping it in the next version. |
||
|
|
||
| New readers are aware of all the past specs and therefore backward compatibility is usually required. | ||
|
|
||
| Occasionally, we may drop support of a feature for technical reasons (ex: | ||
| dropping LZO support for read because of the additional complexity of linking | ||
| libraries). This is usually for features that are practically not used anymore | ||
| (or close). In such a case, new readers should fail with a clear error and not | ||
| produce correctness issues. | ||
|
|
||
| ### Forward compatibility | ||
|
|
||
| Old readers, however, are not aware of future spec changes. | ||
|
|
||
| #### Reader behavior | ||
|
|
||
| Ordered by desirability: | ||
|
|
||
| | Name | Possible behaviors of the reader | When used | Example | User experience | | ||
| |------|----------------------------------|-----------|---------|-----------------| | ||
| | **Correct Read** | Reads data correctly.<br>Sometimes ignores supplemental data structures it doesn't know about. | This is the most frequent case for forward compatibility. We add new features that do not prevent old readers from reading the data correctly even though they might not take advantage of those new features. | You can read the data correctly and not understand bloom filters or statistics. | **Forward compatible**: The format evolves seamlessly. | | ||
| | **Correct Partial Read** | Reads correctly a subset of the columns as long as the unsupported columns don't need to be accessed and correctly fails otherwise. | When a feature cannot be introduced without breaking forward compatibility (ex: new encoding), old readers are able to read files and columns that do not use that new feature and know they can not read the ones that do and why. | [ALP Encoding](Encodings.md#ALP): Old readers can read all the columns that don't use this encoding that applies only to floating point columns. | **Forward incompatible but correct**: Old readers will explain failed reads and preserve correctness.<br>New writers can be configured to hold off on using new features until all readers support it. | | ||
| | **Correct Failure to Read** | Can't read data at all but understands why. | When a feature cannot be introduced without breaking file-level compatibility, old readers know they can not read the file. | Changing the footer format. The reader has no granular information and fails on the magic number change. | Same as Correct Partial Read: **Forward incompatible but correct**. | | ||
| | **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". | | ||
| | **Incorrect Read** | Reads data but unknowingly misinterprets it. | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.<br>**This should never happen.** | Semantic changes to an existing field like stats. We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | | ||
|
|
||
| ### Evaluation system for changes to Parquet | ||
|
|
||
| When making a change to the format we optimize for that user experience. | ||
|
|
||
| Making changes to the Parquet format falls in 3 categories: | ||
|
|
||
| - **Forward compatible**: Correct Read | ||
| - **Forward incompatible**: | ||
| - The metadata describes what can be read and not read: the reader decides | ||
| between Correct Partial Read and Best Effort Read. | ||
| - The metadata doesn't describe what can be read and not read (this happens | ||
| for example with the new footer format): Correct Failure to Read. | ||
| - We don't make semantic changes that would cause old readers to produce an | ||
| Incorrect Read. | ||
|
|
||
|  | ||
|
|
||
| In particular that means that we avoid changing the semantics of an existing | ||
| field. For example changing a sort order that changes the min-max statistics. | ||
| Instead it can be made forward compatible by adding new metadata fields. Making | ||
| a forward compatible change like this is usually preferable to breaking | ||
| correctness or forcing a magic number increment. | ||
|
|
||
| See also [Extensibility](README.md#extensibility) for the places in | ||
| the format designed for compatible extensions. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| %% Licensed to the Apache Software Foundation (ASF) under one | ||
| %% or more contributor license agreements. See the NOTICE file | ||
| %% distributed with this work for additional information | ||
| %% regarding copyright ownership. The ASF licenses this file | ||
| %% to you under the Apache License, Version 2.0 (the | ||
| %% "License"); you may not use this file except in compliance | ||
| %% with the License. You may obtain a copy of the License at | ||
| %% http://www.apache.org/licenses/LICENSE-2.0 | ||
| %% Unless required by applicable law or agreed to in writing, | ||
| %% software distributed under the License is distributed on an | ||
| %% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| %% KIND, either express or implied. See the License for the | ||
| %% specific language governing permissions and limitations | ||
| %% under the License. | ||
| %%{init: {"look": "classic", "flowchart": {"useMaxWidth": false, "diagramPadding": 4}}}%% | ||
| flowchart LR | ||
| subgraph nested["Set of all Changes"] | ||
| direction TB | ||
| nC["Compatible"] | ||
| subgraph nI["Incompatible"] | ||
| direction TB | ||
| nIR["Incorrect Read"] | ||
| subgraph nCF["Correct Failed Read"] | ||
| direction TB | ||
| nPC["Partial Compatibility"] | ||
| nFI["Fully Incompatible"] | ||
| nPC ~~~ nFI | ||
| end | ||
| nIR ~~~ nCF | ||
| end | ||
| nC ~~~ nI | ||
| end | ||
|
|
||
| subgraph tree[" "] | ||
| direction TB | ||
| all(["Set of all Changes"]) --- C(["Compatible"]) | ||
| all --- I(["Incompatible"]) | ||
| I --- CF(["Correct failed Read"]) | ||
| I --- IR(["Incorrect Read"]) | ||
| CF --- PC(["Partial Compatibility"]) | ||
| CF --- FI(["Fully Incompatible"]) | ||
| end | ||
|
|
||
| nested ~~~ tree | ||
|
|
||
| classDef root fill:#cfe2f3,stroke:#555,color:#000 | ||
| classDef compat fill:#d9ead3,stroke:#555,color:#000 | ||
| classDef incompat fill:#fff2cc,stroke:#555,color:#000 | ||
| classDef incorrect fill:#f4cccc,stroke:#555,color:#000 | ||
| classDef failed fill:#fce5cd,stroke:#555,color:#000 | ||
| classDef partial fill:#f1c232,stroke:#555,color:#000 | ||
| classDef full fill:#bf9000,stroke:#555,color:#000 | ||
| class all,nested root | ||
| class C,nC compat | ||
| class I,nI incompat | ||
| class IR,nIR incorrect | ||
| class CF,nCF failed | ||
| class PC,nPC partial | ||
| class FI,nFI full | ||
| style tree fill:none,stroke:none |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps as a follow on PR we can consolidate this text with the text in FormatChanges.md