From 0bb58acf9187ee94acd281aa905e923a6ddf02e4 Mon Sep 17 00:00:00 2001 From: Julien Le Dem Date: Mon, 9 Nov 2026 18:05:46 -0800 Subject: [PATCH 1/3] adding changes guidelines Signed-off-by: Julien Le Dem --- README.md | 7 +- assets/scss/_styles_project.scss | 21 ++++ content/en/docs/format-changes.md | 124 +++++++++++++++++++ package.json | 8 +- static/images/FormatChangeCategories.mermaid | 60 +++++++++ static/images/FormatChangeCategories.svg | 99 +++++++++++++++ static/images/FormatChangeDecisionTree.svg | 93 ++++++++++++++ 7 files changed, 408 insertions(+), 4 deletions(-) create mode 100644 content/en/docs/format-changes.md create mode 100644 static/images/FormatChangeCategories.mermaid create mode 100644 static/images/FormatChangeCategories.svg create mode 100644 static/images/FormatChangeDecisionTree.svg diff --git a/README.md b/README.md index 317b28ca..2d918963 100644 --- a/README.md +++ b/README.md @@ -69,9 +69,9 @@ To preview this website locally, run the following in the root of the directory: hugo server ``` -### Building metadata diagrams +### Building diagrams -To build the metadata svg diagrams, you need mermaid.js installed. You can install it using npm: +To build the svg diagrams, you need mermaid.js installed. You can install it using npm: ``` npm install -D @mermaid-js/mermaid-cli @@ -83,8 +83,11 @@ Then you can build the diagrams using the following command: cd static/images npx mmdc -i FileMetaData.mermaid -o FileMetaData.svg npx mmdc -i PageHeader.mermaid -o PageHeader.svg +npx mmdc -b white -i FormatChangeCategories.mermaid -o FormatChangeCategories.svg ``` +`FormatChangeDecisionTree.svg` is hand-written SVG (Mermaid cannot produce its compact layout); edit it directly. + ## Building and Running in Docker If you don't want to install `hugo` and its dependencies on your local machine, diff --git a/assets/scss/_styles_project.scss b/assets/scss/_styles_project.scss index e1cfe9b8..7fbeee46 100644 --- a/assets/scss/_styles_project.scss +++ b/assets/scss/_styles_project.scss @@ -59,3 +59,24 @@ visibility: visible; } } + +// Tables in documentation pages: visible borders and emphasized headers. +// Matches Docsy's `.td-content table:not(.td-initial)` selectors so these +// rules take precedence, and uses Bootstrap color variables. + +.td-content table:not(.td-initial) { + border-collapse: collapse; + + > :not(caption) > * > * { + border: 1px solid var(--bs-border-color); + padding: 0.5rem 0.75rem; + vertical-align: top; + } + + > thead > tr > th { + --bs-table-bg: var(--bs-secondary-bg); + color: var(--bs-emphasis-color); + font-weight: 700; + border-bottom: 2px solid var(--bs-secondary-color); + } +} diff --git a/content/en/docs/format-changes.md b/content/en/docs/format-changes.md new file mode 100644 index 00000000..4685787d --- /dev/null +++ b/content/en/docs/format-changes.md @@ -0,0 +1,124 @@ +--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +title: "Changing the Format" +linkTitle: "Changing the Format" +weight: 6 +description: > + Guidance for proposing changes to the Parquet format specification +--- + +## Scope + +When making a change to the Parquet Format, we have to manage problems that +arise from the reader and the writer implementing a different version of the +spec. + +This document defines categories of changes to the Parquet Format, and proposes +a set of criteria to evaluate future potential changes. + +Changes which are forward compatible are always welcome. Partially compatible +changes are encouraged with a preference for consumers to error out when +un-parsable data is observed. Changes which would produce an incorrect read +require a PAR version change and should be done sparingly. + +## Definitions + +A change to the Parquet format can have the following properties: + +**Compatibility**: We narrow this down to producing the exact same tabular data +while possibly ignoring secondary data structures (like bloom filters, indices, +stats, …). Compatible implies Correct. + +- *Backward compatibility*: new readers can read files written by older writers. +- *Forward compatibility*: old readers can read files written by newer writers. + +**Correctness**: Not unknowingly reading data incorrectly. + +- *Incorrect read*: a reader producing different data than the writer intended. +- *Correct failed read*: a reader producing an error when it can not read the + data in the file. + +**Partial Compatibility**: when the reader can read a subset of the file (and +knows which subset). + +- If only the subset of columns supported is accessed, then the read is + successful. +- The reader can elect to read only the subset of the data it understands + depending on the use case. + +{{< figure src="/images/FormatChangeCategories.svg" alt="Categories of changes: compatible, or incompatible. Incompatible changes cause either an incorrect read or a correct failed read, and a correct failed read is either partially compatible or fully incompatible." >}} + +## Dealing with incompatible changes + +### Backward compatibility + +When making a change to the spec, the new readers are aware of all the past +specs and therefore backward compatibility is usually required. + +Occasionally, we may drop support of a feature for technical reasons (ex: +dropping LZO support for read because of the additional complexity of linking +libraries). This is usually for features that are practically not used anymore +(or close). In such a case, new readers will fail with a clear error and not +produce correctness issues. + +In conclusion, a change is either always backward compatible or occasionally +backward incompatible and causes failure with an error and no correctness +issues. + +### Forward compatibility + +Old readers, however, are not aware of future spec changes. + +#### Reader behavior + +Ordered by desirability: + +| Name | Possible behaviors of the reader | When used | Example | User experience | +|------|----------------------------------|-----------|---------|-----------------| +| **Correct Read** | Reads data correctly.
Sometimes ignores supplemental data structures it doesn't know about. | This is the most frequent case for forward compatibility. We add new features that do not prevent old readers from reading the data correctly even though they might not take advantage of those new features. | You can read the data correctly and not understand bloom filters or statistics. | **Forward compatible**: The format evolves seamlessly. | +| **Correct Partial Read** | Reads correctly a subset of the columns as long as the unsupported columns don't need to be accessed and correctly fails otherwise. | When a feature cannot be introduced without breaking forward compatibility (ex: new encoding), old readers are able to read files and columns that do not use that new feature and know they can not read the ones that do and why. | ALP: Old readers can read all the columns that don't use this encoding that applies only to floating point columns. | **Forward incompatible but correct**: Old readers will explain failed reads and preserve correctness.
New writers can be configured to hold off on using new features until all readers support it. | +| **Correct Failure to Read** | Can't read data at all but understands why. | When a feature cannot be introduced without breaking file-level compatibility, old readers know they can not read the file. | Changing the footer format. The reader has no granular information and fails on the magic number change. | Same as Correct Partial Read: **Forward incompatible but correct**. | +| **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". | +| **Incorrect Read** | Reads data but unknowingly misinterprets it (ex: semantic changes to an existing field like stats). | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.
**This should never happen.** | We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | + +### Proposal: Evaluation system for changes to Parquet + +When making a change to the format we optimize for that user experience. + +Making changes to the Parquet format falls in 3 categories: + +- **Forward compatible**: Correct Read +- **Forward incompatible**: + - The metadata describes what can be read and not read: the reader decides + between Correct Partial Read and Best Effort Read. + - The metadata doesn't describe what can be read and not read (this happens + for example with the new footer format): Correct Failure to Read. +- We don't make semantic changes that would cause old readers to produce an + Incorrect Read. + +{{< figure src="/images/FormatChangeDecisionTree.svg" alt="Decision flow for a change: if it can be made forward compatible, it results in a Correct Read. Otherwise, if the metadata can tell old readers what and why they can not read, it results in a Correct Partial Read or Best Effort Read. Otherwise the magic number must be incremented, resulting in a Correct Failure to Read. If not, the result is an Incorrect Read. Except for a footer format change, we will prefer finding a way to either make it forward compatible or described by the metadata." >}} + +In particular that means that we avoid changing the semantics of an existing +field. For example changing a sort order that changes the min-max statistics. +Instead it can be made forward compatible by adding new metadata fields. Making +a forward compatible change like this is usually preferable to breaking +correctness or forcing a magic number increment. + +See also [Extensibility](/docs/file-format/extensibility/) for the places in +the format designed for compatible extensions. diff --git a/package.json b/package.json index 56b7ca78..ce5e3625 100644 --- a/package.json +++ b/package.json @@ -18,9 +18,13 @@ "homepage": "https://github.com/apache/parquet-site#readme", "devDependencies": { "@mermaid-js/mermaid-cli": "^12.0.0", - "autoprefixer": "^10.4.18", + "autoprefixer": "^10.6.2", "bootstrap": "^5.3.6", - "postcss": "^8.4.35", + "hugo-installer": "^4.0.1", + "postcss": "^8.5.32", "postcss-cli": "^12.0.0" + }, + "allowScripts": { + "puppeteer@25.16.0": true } } diff --git a/static/images/FormatChangeCategories.mermaid b/static/images/FormatChangeCategories.mermaid new file mode 100644 index 00000000..482765ab --- /dev/null +++ b/static/images/FormatChangeCategories.mermaid @@ -0,0 +1,60 @@ +%% Licensed to the Apache Software Foundation (ASF) under one +%% or more contributor license agreements. See the NOTICE file +%% distributed with this work for additional information +%% regarding copyright ownership. The ASF licenses this file +%% to you under the Apache License, Version 2.0 (the +%% "License"); you may not use this file except in compliance +%% with the License. You may obtain a copy of the License at +%% http://www.apache.org/licenses/LICENSE-2.0 +%% Unless required by applicable law or agreed to in writing, +%% software distributed under the License is distributed on an +%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +%% KIND, either express or implied. See the License for the +%% specific language governing permissions and limitations +%% under the License. +%%{init: {"look": "classic", "flowchart": {"useMaxWidth": false, "diagramPadding": 4}}}%% +flowchart LR + subgraph nested["Set of all Changes"] + direction TB + nC["Compatible"] + subgraph nI["Incompatible"] + direction TB + nIR["Incorrect Read"] + subgraph nCF["Correct Failed Read"] + direction TB + nPC["Partial Compatibility"] + nFI["Fully Incompatible"] + nPC ~~~ nFI + end + nIR ~~~ nCF + end + nC ~~~ nI + end + + subgraph tree[" "] + direction TB + all(["Set of all Changes"]) --- C(["Compatible"]) + all --- I(["Incompatible"]) + I --- CF(["Correct failed Read"]) + I --- IR(["Incorrect Read"]) + CF --- PC(["Partial Compatibility"]) + CF --- FI(["Fully Incompatible"]) + end + + nested ~~~ tree + + classDef root fill:#cfe2f3,stroke:#555,color:#000 + classDef compat fill:#d9ead3,stroke:#555,color:#000 + classDef incompat fill:#fff2cc,stroke:#555,color:#000 + classDef incorrect fill:#f4cccc,stroke:#555,color:#000 + classDef failed fill:#fce5cd,stroke:#555,color:#000 + classDef partial fill:#f1c232,stroke:#555,color:#000 + classDef full fill:#bf9000,stroke:#555,color:#000 + class all,nested root + class C,nC compat + class I,nI incompat + class IR,nIR incorrect + class CF,nCF failed + class PC,nPC partial + class FI,nFI full + style tree fill:none,stroke:none diff --git a/static/images/FormatChangeCategories.svg b/static/images/FormatChangeCategories.svg new file mode 100644 index 00000000..46dcfdfc --- /dev/null +++ b/static/images/FormatChangeCategories.svg @@ -0,0 +1,99 @@ +

Set of all Changes

Incompatible

Correct Failed Read

Compatible

Incorrect Read

Partial Compatibility

Fully Incompatible

Set of all Changes

Compatible

Incompatible

Correct failed Read

Incorrect Read

Partial Compatibility

Fully Incompatible

\ No newline at end of file diff --git a/static/images/FormatChangeDecisionTree.svg b/static/images/FormatChangeDecisionTree.svg new file mode 100644 index 00000000..6fa12926 --- /dev/null +++ b/static/images/FormatChangeDecisionTree.svg @@ -0,0 +1,93 @@ + + + + Decision flow for making a change to the Parquet format + + + + + + + + + + I want to make + a change + + + + + Can it be made + forward + compatible? + + Yes + + Correct Read + + No + + + + Can the metadata + tell old readers + what and why they + can not read? + + Yes + + Correct Partial + Read or + Best Effort Read + + The reader knows + what subset of + the file it can + read and decides + what to do + + No + + + + Then you must + increment the + magic number + + Yes + + Correct Failure + to Read + + No + + + + Incorrect Read + + Except for a footer format change, + we will prefer finding a way to + either make it forward compatible + or described by the metadata + From 6902aee904e5bee188c107c7718f14224bb662f7 Mon Sep 17 00:00:00 2001 From: Julien Le Dem Date: Fri, 13 Nov 2026 15:10:56 -0800 Subject: [PATCH 2/3] review feedback Signed-off-by: Julien Le Dem --- .../format-changes.md | 25 ++++++++----------- package.json | 8 ++---- 2 files changed, 12 insertions(+), 21 deletions(-) rename content/en/docs/{ => Contribution Guidelines}/format-changes.md (83%) diff --git a/content/en/docs/format-changes.md b/content/en/docs/Contribution Guidelines/format-changes.md similarity index 83% rename from content/en/docs/format-changes.md rename to content/en/docs/Contribution Guidelines/format-changes.md index 4685787d..d2dfb060 100644 --- a/content/en/docs/format-changes.md +++ b/content/en/docs/Contribution Guidelines/format-changes.md @@ -18,7 +18,7 @@ title: "Changing the Format" linkTitle: "Changing the Format" -weight: 6 +weight: 5 description: > Guidance for proposing changes to the Parquet format specification --- @@ -32,10 +32,10 @@ spec. This document defines categories of changes to the Parquet Format, and proposes a set of criteria to evaluate future potential changes. -Changes which are forward compatible are always welcome. Partially compatible -changes are encouraged with a preference for consumers to error out when +Changes which are forward compatible do not adversely impact the user and are easier to standardize. +Partially compatible changes are encouraged with a preference for consumers to error out when un-parsable data is observed. Changes which would produce an incorrect read -require a PAR version change and should be done sparingly. +require a PAR magic number version change and should be done sparingly. ## Definitions @@ -48,7 +48,7 @@ stats, …). Compatible implies Correct. - *Backward compatibility*: new readers can read files written by older writers. - *Forward compatibility*: old readers can read files written by newer writers. -**Correctness**: Not unknowingly reading data incorrectly. +**Correctness**: Avoid reading incorrect data unknowingly. - *Incorrect read*: a reader producing different data than the writer intended. - *Correct failed read*: a reader producing an error when it can not read the @@ -68,19 +68,14 @@ knows which subset). ### Backward compatibility -When making a change to the spec, the new readers are aware of all the past -specs and therefore backward compatibility is usually required. +New readers are aware of all the past specs and therefore backward compatibility is usually required. Occasionally, we may drop support of a feature for technical reasons (ex: dropping LZO support for read because of the additional complexity of linking libraries). This is usually for features that are practically not used anymore -(or close). In such a case, new readers will fail with a clear error and not +(or close). In such a case, new readers should fail with a clear error and not produce correctness issues. -In conclusion, a change is either always backward compatible or occasionally -backward incompatible and causes failure with an error and no correctness -issues. - ### Forward compatibility Old readers, however, are not aware of future spec changes. @@ -92,12 +87,12 @@ Ordered by desirability: | Name | Possible behaviors of the reader | When used | Example | User experience | |------|----------------------------------|-----------|---------|-----------------| | **Correct Read** | Reads data correctly.
Sometimes ignores supplemental data structures it doesn't know about. | This is the most frequent case for forward compatibility. We add new features that do not prevent old readers from reading the data correctly even though they might not take advantage of those new features. | You can read the data correctly and not understand bloom filters or statistics. | **Forward compatible**: The format evolves seamlessly. | -| **Correct Partial Read** | Reads correctly a subset of the columns as long as the unsupported columns don't need to be accessed and correctly fails otherwise. | When a feature cannot be introduced without breaking forward compatibility (ex: new encoding), old readers are able to read files and columns that do not use that new feature and know they can not read the ones that do and why. | ALP: Old readers can read all the columns that don't use this encoding that applies only to floating point columns. | **Forward incompatible but correct**: Old readers will explain failed reads and preserve correctness.
New writers can be configured to hold off on using new features until all readers support it. | +| **Correct Partial Read** | Reads correctly a subset of the columns as long as the unsupported columns don't need to be accessed and correctly fails otherwise. | When a feature cannot be introduced without breaking forward compatibility (ex: new encoding), old readers are able to read files and columns that do not use that new feature and know they can not read the ones that do and why. | [ALP Encoding](https://parquet.apache.org/docs/file-format/data-pages/encodings/#ALP): Old readers can read all the columns that don't use this encoding that applies only to floating point columns. | **Forward incompatible but correct**: Old readers will explain failed reads and preserve correctness.
New writers can be configured to hold off on using new features until all readers support it. | | **Correct Failure to Read** | Can't read data at all but understands why. | When a feature cannot be introduced without breaking file-level compatibility, old readers know they can not read the file. | Changing the footer format. The reader has no granular information and fails on the magic number change. | Same as Correct Partial Read: **Forward incompatible but correct**. | | **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". | -| **Incorrect Read** | Reads data but unknowingly misinterprets it (ex: semantic changes to an existing field like stats). | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.
**This should never happen.** | We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | +| **Incorrect Read** | Reads data but unknowingly misinterprets it. | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.
**This should never happen.** | Semantic changes to an existing field like stats. We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | -### Proposal: Evaluation system for changes to Parquet +### Evaluation system for changes to Parquet When making a change to the format we optimize for that user experience. diff --git a/package.json b/package.json index ce5e3625..56b7ca78 100644 --- a/package.json +++ b/package.json @@ -18,13 +18,9 @@ "homepage": "https://github.com/apache/parquet-site#readme", "devDependencies": { "@mermaid-js/mermaid-cli": "^12.0.0", - "autoprefixer": "^10.6.2", + "autoprefixer": "^10.4.18", "bootstrap": "^5.3.6", - "hugo-installer": "^4.0.1", - "postcss": "^8.5.32", + "postcss": "^8.4.35", "postcss-cli": "^12.0.0" - }, - "allowScripts": { - "puppeteer@25.16.0": true } } From 47c404d45e810595aad99c40bfd53041cc11aec1 Mon Sep 17 00:00:00 2001 From: Julien Le Dem Date: Fri, 20 Nov 2026 16:19:34 -0800 Subject: [PATCH 3/3] moving to parquet-format Signed-off-by: Julien Le Dem --- README.md | 7 +- assets/parquet-format | 2 +- .../Contribution Guidelines/format-changes.md | 95 +----------------- static/images/FormatChangeCategories.mermaid | 60 ----------- static/images/FormatChangeCategories.svg | 99 ------------------- static/images/FormatChangeDecisionTree.svg | 93 ----------------- 6 files changed, 4 insertions(+), 352 deletions(-) delete mode 100644 static/images/FormatChangeCategories.mermaid delete mode 100644 static/images/FormatChangeCategories.svg delete mode 100644 static/images/FormatChangeDecisionTree.svg diff --git a/README.md b/README.md index 2d918963..317b28ca 100644 --- a/README.md +++ b/README.md @@ -69,9 +69,9 @@ To preview this website locally, run the following in the root of the directory: hugo server ``` -### Building diagrams +### Building metadata diagrams -To build the svg diagrams, you need mermaid.js installed. You can install it using npm: +To build the metadata svg diagrams, you need mermaid.js installed. You can install it using npm: ``` npm install -D @mermaid-js/mermaid-cli @@ -83,11 +83,8 @@ Then you can build the diagrams using the following command: cd static/images npx mmdc -i FileMetaData.mermaid -o FileMetaData.svg npx mmdc -i PageHeader.mermaid -o PageHeader.svg -npx mmdc -b white -i FormatChangeCategories.mermaid -o FormatChangeCategories.svg ``` -`FormatChangeDecisionTree.svg` is hand-written SVG (Mermaid cannot produce its compact layout); edit it directly. - ## Building and Running in Docker If you don't want to install `hugo` and its dependencies on your local machine, diff --git a/assets/parquet-format b/assets/parquet-format index 219e3f12..0eea7503 160000 --- a/assets/parquet-format +++ b/assets/parquet-format @@ -1 +1 @@ -Subproject commit 219e3f12a62f9476e830c21e26d030d231f7c017 +Subproject commit 0eea7503c721e29276117745e1c6a5ecf05447c3 diff --git a/content/en/docs/Contribution Guidelines/format-changes.md b/content/en/docs/Contribution Guidelines/format-changes.md index d2dfb060..2de3e252 100644 --- a/content/en/docs/Contribution Guidelines/format-changes.md +++ b/content/en/docs/Contribution Guidelines/format-changes.md @@ -23,97 +23,4 @@ description: > Guidance for proposing changes to the Parquet format specification --- -## Scope - -When making a change to the Parquet Format, we have to manage problems that -arise from the reader and the writer implementing a different version of the -spec. - -This document defines categories of changes to the Parquet Format, and proposes -a set of criteria to evaluate future potential changes. - -Changes which are forward compatible do not adversely impact the user and are easier to standardize. -Partially compatible changes are encouraged with a preference for consumers to error out when -un-parsable data is observed. Changes which would produce an incorrect read -require a PAR magic number version change and should be done sparingly. - -## Definitions - -A change to the Parquet format can have the following properties: - -**Compatibility**: We narrow this down to producing the exact same tabular data -while possibly ignoring secondary data structures (like bloom filters, indices, -stats, …). Compatible implies Correct. - -- *Backward compatibility*: new readers can read files written by older writers. -- *Forward compatibility*: old readers can read files written by newer writers. - -**Correctness**: Avoid reading incorrect data unknowingly. - -- *Incorrect read*: a reader producing different data than the writer intended. -- *Correct failed read*: a reader producing an error when it can not read the - data in the file. - -**Partial Compatibility**: when the reader can read a subset of the file (and -knows which subset). - -- If only the subset of columns supported is accessed, then the read is - successful. -- The reader can elect to read only the subset of the data it understands - depending on the use case. - -{{< figure src="/images/FormatChangeCategories.svg" alt="Categories of changes: compatible, or incompatible. Incompatible changes cause either an incorrect read or a correct failed read, and a correct failed read is either partially compatible or fully incompatible." >}} - -## Dealing with incompatible changes - -### Backward compatibility - -New readers are aware of all the past specs and therefore backward compatibility is usually required. - -Occasionally, we may drop support of a feature for technical reasons (ex: -dropping LZO support for read because of the additional complexity of linking -libraries). This is usually for features that are practically not used anymore -(or close). In such a case, new readers should fail with a clear error and not -produce correctness issues. - -### Forward compatibility - -Old readers, however, are not aware of future spec changes. - -#### Reader behavior - -Ordered by desirability: - -| Name | Possible behaviors of the reader | When used | Example | User experience | -|------|----------------------------------|-----------|---------|-----------------| -| **Correct Read** | Reads data correctly.
Sometimes ignores supplemental data structures it doesn't know about. | This is the most frequent case for forward compatibility. We add new features that do not prevent old readers from reading the data correctly even though they might not take advantage of those new features. | You can read the data correctly and not understand bloom filters or statistics. | **Forward compatible**: The format evolves seamlessly. | -| **Correct Partial Read** | Reads correctly a subset of the columns as long as the unsupported columns don't need to be accessed and correctly fails otherwise. | When a feature cannot be introduced without breaking forward compatibility (ex: new encoding), old readers are able to read files and columns that do not use that new feature and know they can not read the ones that do and why. | [ALP Encoding](https://parquet.apache.org/docs/file-format/data-pages/encodings/#ALP): Old readers can read all the columns that don't use this encoding that applies only to floating point columns. | **Forward incompatible but correct**: Old readers will explain failed reads and preserve correctness.
New writers can be configured to hold off on using new features until all readers support it. | -| **Correct Failure to Read** | Can't read data at all but understands why. | When a feature cannot be introduced without breaking file-level compatibility, old readers know they can not read the file. | Changing the footer format. The reader has no granular information and fails on the magic number change. | Same as Correct Partial Read: **Forward incompatible but correct**. | -| **Best Effort Read** | Reads the subset of data it understands and ignores the rest. | Similar to Correct Partial Reads. In cases where this does not lead to data loss, readers can elect to read only the subset of the file they can read correctly. This is different from a correct partial read as a "select \*" would return only the columns it understands instead of failing for not reading a subset of the columns. | "Select \*" ignoring columns encoded in ALP and returning the subset of the file that is readable. | **Forward incompatible best effort**: Old readers can elect to read in best effort the subset of the data. Not reading incorrect data but would lead to dropping data if the output was saved without understanding the difference with "correct partial read". | -| **Incorrect Read** | Reads data but unknowingly misinterprets it. | When an old reader reads incorrect data because a semantic change is indistinguishable and they have no way to know their read is incorrect.
**This should never happen.** | Semantic changes to an existing field like stats. We avoid this case at all costs. | **Incorrect**: Silently reading the data incorrectly. | - -### Evaluation system for changes to Parquet - -When making a change to the format we optimize for that user experience. - -Making changes to the Parquet format falls in 3 categories: - -- **Forward compatible**: Correct Read -- **Forward incompatible**: - - The metadata describes what can be read and not read: the reader decides - between Correct Partial Read and Best Effort Read. - - The metadata doesn't describe what can be read and not read (this happens - for example with the new footer format): Correct Failure to Read. -- We don't make semantic changes that would cause old readers to produce an - Incorrect Read. - -{{< figure src="/images/FormatChangeDecisionTree.svg" alt="Decision flow for a change: if it can be made forward compatible, it results in a Correct Read. Otherwise, if the metadata can tell old readers what and why they can not read, it results in a Correct Partial Read or Best Effort Read. Otherwise the magic number must be incremented, resulting in a Correct Failure to Read. If not, the result is an Incorrect Read. Except for a footer format change, we will prefer finding a way to either make it forward compatible or described by the metadata." >}} - -In particular that means that we avoid changing the semantics of an existing -field. For example changing a sort order that changes the min-max statistics. -Instead it can be made forward compatible by adding new metadata fields. Making -a forward compatible change like this is usually preferable to breaking -correctness or forcing a magic number increment. - -See also [Extensibility](/docs/file-format/extensibility/) for the places in -the format designed for compatible extensions. +{{< parquet-format "FormatChanges.md" >}} diff --git a/static/images/FormatChangeCategories.mermaid b/static/images/FormatChangeCategories.mermaid deleted file mode 100644 index 482765ab..00000000 --- a/static/images/FormatChangeCategories.mermaid +++ /dev/null @@ -1,60 +0,0 @@ -%% Licensed to the Apache Software Foundation (ASF) under one -%% or more contributor license agreements. See the NOTICE file -%% distributed with this work for additional information -%% regarding copyright ownership. The ASF licenses this file -%% to you under the Apache License, Version 2.0 (the -%% "License"); you may not use this file except in compliance -%% with the License. You may obtain a copy of the License at -%% http://www.apache.org/licenses/LICENSE-2.0 -%% Unless required by applicable law or agreed to in writing, -%% software distributed under the License is distributed on an -%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -%% KIND, either express or implied. See the License for the -%% specific language governing permissions and limitations -%% under the License. -%%{init: {"look": "classic", "flowchart": {"useMaxWidth": false, "diagramPadding": 4}}}%% -flowchart LR - subgraph nested["Set of all Changes"] - direction TB - nC["Compatible"] - subgraph nI["Incompatible"] - direction TB - nIR["Incorrect Read"] - subgraph nCF["Correct Failed Read"] - direction TB - nPC["Partial Compatibility"] - nFI["Fully Incompatible"] - nPC ~~~ nFI - end - nIR ~~~ nCF - end - nC ~~~ nI - end - - subgraph tree[" "] - direction TB - all(["Set of all Changes"]) --- C(["Compatible"]) - all --- I(["Incompatible"]) - I --- CF(["Correct failed Read"]) - I --- IR(["Incorrect Read"]) - CF --- PC(["Partial Compatibility"]) - CF --- FI(["Fully Incompatible"]) - end - - nested ~~~ tree - - classDef root fill:#cfe2f3,stroke:#555,color:#000 - classDef compat fill:#d9ead3,stroke:#555,color:#000 - classDef incompat fill:#fff2cc,stroke:#555,color:#000 - classDef incorrect fill:#f4cccc,stroke:#555,color:#000 - classDef failed fill:#fce5cd,stroke:#555,color:#000 - classDef partial fill:#f1c232,stroke:#555,color:#000 - classDef full fill:#bf9000,stroke:#555,color:#000 - class all,nested root - class C,nC compat - class I,nI incompat - class IR,nIR incorrect - class CF,nCF failed - class PC,nPC partial - class FI,nFI full - style tree fill:none,stroke:none diff --git a/static/images/FormatChangeCategories.svg b/static/images/FormatChangeCategories.svg deleted file mode 100644 index 46dcfdfc..00000000 --- a/static/images/FormatChangeCategories.svg +++ /dev/null @@ -1,99 +0,0 @@ -

Set of all Changes

Incompatible

Correct Failed Read

Compatible

Incorrect Read

Partial Compatibility

Fully Incompatible

Set of all Changes

Compatible

Incompatible

Correct failed Read

Incorrect Read

Partial Compatibility

Fully Incompatible

\ No newline at end of file diff --git a/static/images/FormatChangeDecisionTree.svg b/static/images/FormatChangeDecisionTree.svg deleted file mode 100644 index 6fa12926..00000000 --- a/static/images/FormatChangeDecisionTree.svg +++ /dev/null @@ -1,93 +0,0 @@ - - - - Decision flow for making a change to the Parquet format - - - - - - - - - - I want to make - a change - - - - - Can it be made - forward - compatible? - - Yes - - Correct Read - - No - - - - Can the metadata - tell old readers - what and why they - can not read? - - Yes - - Correct Partial - Read or - Best Effort Read - - The reader knows - what subset of - the file it can - read and decides - what to do - - No - - - - Then you must - increment the - magic number - - Yes - - Correct Failure - to Read - - No - - - - Incorrect Read - - Except for a footer format change, - we will prefer finding a way to - either make it forward compatible - or described by the metadata -