Repository navigation
Conversation
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Before we support schema tracking log, we should disallow all non-additive schema change in dsv2. Column Mapping related code change is pure refactoring that for v2 to reuse the utilities. ## How was this patch tested? Unit test: - New test case for drop column in a nested struct - Modify existed tests for non-additive schema change to throws the same error as v1 Integration test: - drop column: should fail with non-additive schema change error - drop column: should succeed with unsafe column mapping schema change flag enabled - rename column: should fail with non-additive schema change error - rename column: should throw schema change error with unsafe flag enabled - type widening: should fail with non-additive schema change error when enable schema tracking <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…ty (delta-io#6227) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6227/files) to review incremental changes. - [**stack/type-nullability-e2e-test**](delta-io#6227) [[Files changed](https://github.com/delta-io/delta/pull/6227/files)] - [stack/forbid-schema-evolution-with-staleDF-v2](delta-io#6228) [[Files changed](https://github.com/delta-io/delta/pull/6228/files/cafe746303c3e32bdcf5499c8e99fd6a59c033c4..94b77dc62d21e896e9f4fdafd3ff17fafddd606f)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Added schema evolution e2e tests for type widening and relax nullability for both refresh dataframe and stale dataframe cases. Refresh dataframe cases work for both v1 & v2 connector, while stale dataframe cases only work for v1 connector. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? test-only PR <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…m 50000 to 100000 (delta-io#6240) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Double the default value of `DELTA_STREAMING_INITIAL_SNAPSHOT_MAX_FILES` from 50,000 to 100,000 to better support larger tables for V2 streaming initial snapshots out of the box. ## How was this patch tested? Existing tests continue to pass since this is a default-value-only change. Tests that explicitly set this config (e.g., `SparkMicroBatchStreamTest`) are unaffected. ## Does this PR introduce _any_ user-facing changes? Yes. The default maximum number of files allowed in the initial snapshot for V2 streaming is increased from 50,000 to 100,000. Users with tables between 50k–100k files will no longer hit the limit without explicitly raising the config. This change only affects the unreleased master branch and the future release.
…ype widening and relaxing nullability (delta-io#6228) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055) to review incremental changes. - [stack/type-nullability-e2e-test](delta-io#6227) [[Files changed](https://github.com/delta-io/delta/pull/6227/files)] - [**stack/forbid-schema-evolution-with-staleDF-v2**](delta-io#6228) [[Files changed](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add schema validation during stream restart e2e tests for type widening and relaxing nullability to block user from adopt these two schema change without refresh the dataframe in v2 connector <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Test-only PR <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…a-io#5837) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add the DataFrame basic test coverage for preventing future regression. ## How was this patch tested? Do the locally unit tests and remotely integration tests. ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: openinx <openinx@gmail.com>
… repartition the table's columns (delta-io#6238) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description - Updated partition column change tests in `OptimisticTransactionSuite` to use `checkError` instead of calling `ex.getErrorClass` directly - Each `checkError` call now verifies the error condition (`DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE`), SQLSTATE (`42P10`), and named parameter values (`operation`, `oldPartitionColumns`, `newPartitionColumns`) - Added expected operation name to the parameterized blocked-operations test cases to support per-case parameter verification - Collapsed the `DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE` error message from three lines into a single line in `delta-error-classes.json`, `databricks-error-classes.json`, and `delta-error-classes-data-classification.json` (the multi-line split was not justified as the column lists render as single-line comma-separated strings) ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> No new tests added. Existing tests in `OptimisticTransactionSuite` covering partition column change validation were updated to use `checkError` as required by the error message review guidelines. ## Does this PR introduce _any_ user-facing changes? N/A <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
delta-io#5775) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5775/files) to review incremental changes. - [**stack/dv_pr3_phase2_vectorized**](delta-io#5775) [[Files changed](https://github.com/delta-io/delta/pull/5775/files)] - [stack/dv_pr4_phase3_file_splitting](delta-io#5776) [[Files changed](https://github.com/delta-io/delta/pull/5776/files/463011af2fae489f7f3fe8c39fe86acbba06277b..0e534ed9c0257735523532c8c6f8e15ea33ae114)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Enable vectorized (columnar batch) reading for DV tables in the Spark V2 connector. ### Changes: - `ColumnVectorWithFilter`: Wraps `ColumnVector` with row ID mapping for zero-copy filtering (follows Iceberg pattern) - `DeletionVectorReadFunction`: Extended to handle `ColumnarBatch` in addition to `InternalRow` - Enable vectorized reader for DV tables (remove `!isTableSupportDv` condition) ### How it works: 1. For `ColumnarBatch`: Build live row mapping, wrap columns with `ColumnVectorWithFilter` 2. For `InternalRow`: Filter deleted rows, project using `ProjectingInternalRow` ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> - `ColumnVectorWithFilterTest`: Unit tests for various data types and filtering scenarios - `DeletionVectorVectorizedReaderTest`: Integration tests verifying vectorized path is used - All DV golden table tests pass with vectorized reading ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…a-io#6104) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Spark's StringStartsWith(attribute, value) filter was not converted to a Kernel predicate, meaning WHERE col LIKE 'prefix%' queries had no filter pushdown in the V2 connector. The Kernel's STARTS_WITH predicate maps directly: STARTS_WITH(column, string_literal). ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…sts (delta-io#6210) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add the DataFrame streaming integration tests to cover the spark + uc case. ## How was this patch tested? Run in both locally UC server and remote UC server. ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: openinx <openinx@gmail.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (sharing) ## Description Upgrade delta sharing client to 1.3.10 ## How was this patch tested? CI pipeline ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR blocks writes with NullType in UDT columns, similar to how we block NullType in arrays/maps. NullType is typically used as a temporary placeholder, which is later evolved into some other type. However, UDTs cannot evolve over time, hence NullType inside UDT remains as NullType forever. Additionally, our old check missed arrays/maps inside UDT and didn't check whether they contain NullType or not. When trying to write such Parquet files, the Parquet writer failed. This PR fixes these problems. ## How was this patch tested? New unit tests. ## Does this PR introduce _any_ user-facing changes? No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR adds support to excludeRegex read option in deltaDSv2 <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Unit tests: Create table partitioned by category with 4 versions of data. ``` Version 0: CREATE TABLE (empty) Version 1: ids {1,2} in category=alpha Version 2: ids {3,4} in category=beta Version 3: ids {5,6} in category=alpha Version 4: ids {7,8} in category=gamma ``` Test cases: "No excludeRegex - all versions", "Exclude alpha category", "Exclude beta category", "Exclude gamma category", "Exclude alpha and beta categories", "Regex matching no files", "Regex matching all files", "Exclude alpha versions 2 to 4", "Exclude beta version 2 only", "Initial snapshot - no excludeRegex", "Initial snapshot - exclude alpha". Enable integration test: "excludeRegex works and doesn't mess up offsets across restarts - parquet version" ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6204) #### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Send table schema to UC server when creating managed tables ## How was this patch tested? Integration test ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: Yi Li <yi.li@databricks.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] General repository ## Description Adds a note regarding the required Java version into the README.md. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Inapplicable. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Co-authored-by: Leonid Lygin <leonid.lygin@databricks.com> Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Iceberg) ## Description We have the following NPE: ``` java.lang.NullPointerException at java.base/java.util.Objects.requireNonNull(Objects.java:209) at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:296) at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:144) at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:129) at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:138) at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114) ``` Referring to this line: https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/core/src/main/java/org/apache/iceberg/SchemaParser.java#L296 This comes from the following line: https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L144 The field `schema` is only set in V1 fields. I believe the reference implementation (Java OSS) also sets this field for V2/V3 to maximize compatibility, but this might be different for other writers. By passing in the spec, we don't execute the branch and we avoid the NPE: https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L126-L130 <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6203) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Convert CatalogStatistics to V2 connector Statistics and thread them through SparkTable → SparkScanBuilder → SparkScan.estimateStatistics(). Gate numRows/columnStats on cboEnabled || planStatsEnabled to match V1 LogicalRelation.computeStats() behavior. Use per-column avgLen from catalog stats in size estimation instead of defaultSize(), mirroring EstimationUtils.getSizePerRow() with UTF8String overhead for StringType columns. Resolves delta-io#5952 ## How was this patch tested? Unit test ## Does this PR introduce _any_ user-facing changes? No
…a-io#5888) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR adds a new Spark flag that will make all code paths in DataSkippingReader to collect stats. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> New tests. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No.
…iveCatalog for Delta UniForm (delta-io#6189) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Use UnityCatalog as Iceberg committing catalog instead of HiveCatalog for Delta UniForm ## How was this patch tested? Add UTs ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite" ```
…flictSuiteBase (delta-io#6277) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Add an overridable `sqlWithTotalRpcBound` method to `IdentityColumnConflictSuiteBase` that defaults to calling `sql()`. This is a no-op in OSS - it allows internal subclasses to override the method with custom assertions on the winning transaction's SQL execution path. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> No behavioral change - the new method defaults to `sql()`, identical to the existing behavior. All existing `IdentityColumnConflictSuite` tests pass unchanged. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No. Co-authored-by: Shlok Jhawar <shlok.jhawar@databricks.com>
…o#6271) ## Problem In `buildTransactionInternal()`, when `!needsMetadataOrProtocolUpdate`, a new `TransactionImpl` is constructed but never returned. Execution falls through to the full metadata/protocol update pipeline, doing unnecessary work and constructing a second `TransactionImpl`. Fixes delta-io#4713. ## Fix Add `return` before `new TransactionImpl(` so the early-exit path actually returns. The early return skips `TransactionMetadataFactory`, which also runs `validateRowTrackingConfigsNotMissing` on existing tables. To preserve this validation, it is moved into `validateWriteToExistingTable()` so it runs on every write regardless of whether metadata/protocol updates are needed. ## Validation - Confirmed by reading lines 230-247. - New unit test verifies the early-return path (uses a fake `delta.feature.*` property so the factory throws if the early return is bypassed). - Verified `LegacyRowTrackingSuite` passes (45/45 tests), including the row tracking config validation test. ## Regression Prevention - `TransactionBuilderImplSuite`: tests the no-update transaction path. - Existing `LegacyRowTrackingSuite`: covers the row tracking validation that was moved. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…lta-io#6278) ## Problem Spark's CheckDeletionVectorDisabled checks both newestProtocol and prevSnapshot to guard against concurrent writers enabling DVs. The Kernel's CHECK_HAS_NO_DELETION_VECTORS only checks newProtocol, missing the concurrent writer guard. ## Fix Add the previous protocol check to the Kernel's DV validation, matching the Spark implementation's defense-in-depth approach. ## Validation - Confirmed by comparing Spark IcebergCompat.scala with Kernel IcebergCompatMetadataValidatorAndUpdater.java. - New test verifies the check catches DV enablement from a previous protocol. ## Regression Prevention New test case explicitly validates the previous-protocol DV check. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…n in dsv2 (delta-io#6246) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613) to review incremental changes. - [stack/ignoreDeletesV2](delta-io#6245) [[Files changed](https://github.com/delta-io/delta/pull/6245/files)] - [**stack/skipChangeCommitsV2**](delta-io#6246) [[Files changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)] - [stack/ignoreChangesV2](delta-io#6249) [[Files changed](https://github.com/delta-io/delta/pull/6249/files/c1c942553a20800e318a5d2d384f869a3c0c5d3c..7209f24b489535e57a8396d712f0f1da577520d2)] - [stack/ignoreFileDeletionV2](delta-io#6250) [[Files changed](https://github.com/delta-io/delta/pull/6250/files/7209f24b489535e57a8396d712f0f1da577520d2..8690f78ca1bae137706f46ed1e05cdb05702ee6b)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Support - skipChangeCommits read option is DSv2, which skip a commit that contains remove file actions regardless of if there is an add file action or not. - ignoreDeletes read option is DSv2, which skip a commit that contains only remove file actions but no add file action. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Unit tests that test parity between v1 and v2 connector on both pure deletes commit (only remove) and change commit (add + remove). Integration tests: - streaming with skipChangeCommits = true skips both delete and change commits <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6267) ## Problem `generateIcebergCompatWriterV3RemoveAction()` accepts `baseRowId` and `defaultRowCommitVersion` parameters but silently discards them, hardcoding `Optional.empty()` instead. This breaks row tracking for V3 writer compat remove actions. ## Fix Forward the caller-provided `baseRowId` and `defaultRowCommitVersion` to `convertRemoveDataFileStatus()` instead of passing `Optional.empty()`. ## Validation - Confirmed by reading the method: params accepted at lines 266-274, discarded at lines 308-310. - New test verifies non-empty values are preserved in the output Row. ## Regression Prevention New test case explicitly checks that baseRowId and defaultRowCommitVersion are forwarded through to the RemoveFile row. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description During data-skipping, we add one millisecond to timestamp max stats to compensate for the fact that JSON serialization will truncate them to milliseconds. This has the consequence that if the data contains a timestamp within the range (Long.MaxValue - 1000, Long.MaxValue], then the addition will throw an overflow exception. This PR fixes this by adding a check for if the stats is within that range, then we saturate the stats to Long.MaxValue instead of adding 1000 to it. ## How was this patch tested? Unit test ## Does this PR introduce _any_ user-facing changes? No
…io#6268) ## Problem `IcebergRESTCatalogPlanningClient` implements `AutoCloseable` but the `ServerSidePlanningClient` trait does not extend it, and `ServerSidePlannedTable` never calls `close()`. This leaks HTTP client connections. ## Fix Add proper lifecycle management so the HTTP client is closed when no longer needed. ## Validation - Confirmed by reading the trait hierarchy and table construction code. - New test verifies the client is properly closed. ## Regression Prevention New test verifies client lifecycle management. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Deflake PartitionLikeDataSkippingSuite by making result comparison insensitive to nondeterministic record ordering. ## How was this patch tested? See test change. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Block metadata changes on UC-managed tables Without this, metadata mutations on a managed table succeed locally but leave the UC catalog unaware, causing metadata drift. Two kill switches in OptimisticTransaction: - updateMetadata(): blocks schema, partition, description, and configuration changes. Covers INSERT with autoMerge, MERGE with schema evolution, and DataFrame writes with mergeSchema. - commitLarge(): blocks clustering DomainMetadata changes for RESTORE TABLE, which bypasses updateMetadata(). ALTER TABLE is already blocked by UCSingleCatalog before reaching Delta. ## How was this patch tested? New tests added. ## Does this PR introduce _any_ user-facing changes? Yes. Table schema evolution with DML is blocked for UC managed tables. --------- Signed-off-by: Yi Li <yi.li@databricks.com>
…in CRC file (delta-io#6282) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Added a conf to make delta-spark write fileSizeHistogram (and not histogramOpt) as field name in CRC file. ## How was this patch tested? UTs ## Does this PR introduce _any_ user-facing changes? No
…elta-io#6281) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description This PR updates Kernel's `CRCInfo` to **read** both `fileSizeHistogram` and `histogramOpt` fields to ensure backward compatibility with Delta-Spark. While the [Delta spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#version-checksum-file-schema) officially uses `fileSizeHistogram` in the Version Checksum file, Delta-Spark has historically used histogramOpt ([code](https://github.com/delta-io/delta/blob/5ba861fa6d6999d7079706a307a7e293b59d4880/spark/src/main/scala/org/apache/spark/sql/delta/Checksum.scala#L87)). Following [a recent fix](delta-io@021755f) that allows Delta-Spark to **read** both fields, this PR mirrors that compatibility in Kernel. This ensures smooth read operations as we fully transition Delta-Spark to write the correct `fileSizeHistogram` field. ## How was this patch tested? New unit tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…lta-io#6155) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6155/files) to review all changes. **Stack:** - **[[UC Commit Metrics] Add skeleton transport wiring and smoke tests](delta-io#6155 [[Files changed](https://github.com/delta-io/delta/pull/6155/files)] ⬅️ _This PR_ - [[UC Commit Metrics] Add full payload construction and schema tests](delta-io#6156) [[Files changed](https://github.com/delta-io/delta/pull/6156/files/4b5eadb28ab152d9ac180423bfd51720b6219b5b..1f39302986a7b6138cd358f9515386fc283810a8)] - [[UC Commit Metrics] Add feature flag and async dispatch](delta-io#6333) [[Files changed](https://github.com/delta-io/delta/pull/6333/files/1f39302986a7b6138cd358f9515386fc283810a8..7b6d633980419060404203551bb1577ccfe922aa)] --- ## Summary - Self-contained post-commit hook in `hooks/metrics/UpdateMetricsHook.scala` (case class + companion object with payload builder, HTTP client, and constants). - Registered only for UC-managed tables (`CatalogTableUtils.isUnityCatalogManagedTable` check at registration time). - Derives endpoint from `spark.sql.catalog.<name>.uri`, authenticates via `UCCommitCoordinatorBuilder` + `TokenProvider`. - Best-effort: failures logged as warnings, never fail the commit. ## Test plan - [x] `build/sbt "spark/testOnly *metrics.UpdateMetricsHookSuite"` -- 6 tests pass --------- Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com> Signed-off-by: foo.bar <foo.bar@foobar.com> Co-authored-by: foo.bar <foo.bar@foobar.com>
…ta-io#6377) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6377/files) to review incremental changes. - [**stack/ddl-1-foundations**](delta-io#6377) [[Files changed](https://github.com/delta-io/delta/pull/6377/files)] - [stack/ddl-2-builder](delta-io#6378) [[Files changed](https://github.com/delta-io/delta/pull/6378/files/b8c478ac19da5afb5aea0d8ce25dfa744b689e4f..a652586290caadbe79201d594ecf0e95f129b328)] - [stack/ddl-2b-builder](delta-io#6442) [[Files changed](https://github.com/delta-io/delta/pull/6442/files/a652586290caadbe79201d594ecf0e95f129b328..88c17741e4c8f17a56c4a0523c1b744c36789c95)] - [stack/ddl-3-wiring](delta-io#6379) [[Files changed](https://github.com/delta-io/delta/pull/6379/files/88c17741e4c8f17a56c4a0523c1b744c36789c95..113ebc006446bd7645be6a75c900104b0b0ed8a4)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? ## Description In this stack of PRs we are implementing the foundation for DDL operations (metadata-only CREATE, CTAS, RTAS). For context, CREATE TABLE in this DSv2 world is 3 main steps: 1. pre-register the table to the catalog (UCProxy.createTable or HMS.createTable) 2. build a transaction for table creation (Delta commit 0) 3. commit (actually commit the transaction above) **We also want to make implementing future DDL operations easier in the future**. That's why in this PR we create a unified DDL request POJO. This POJO/request will be fed into the transaction builder. **Significance**: CreateTableBuilder.prepare() does all the Spark → Kernel conversion - DDLContext is a normalized request for downstream Kernel operations. > By the time DDLContext is constructed, everything is Kernel-native. buildTransaction() just passes it straight through to TableManager or UCCatalogManagedClient with no further conversion. ``` SQL: CREATE TABLE t (id INT, name STRING) │ ▼ ┌──────────────────────────────────────────┐ │ UCSingleCatalog.createTable( │ │ ident, schema, partitions, props) │ │ │ │ Spark's UC catalog plugin. Delegates │ │ to DeltaCatalog for Delta tables. │ └─────────────────────┬────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────────────────────────────┐ │ DeltaCatalog.createTable(ident, schema, partitions, properties) │ │ │ │ 1. DeltaV2Mode routing check │ │ STRICT → always Kernel path │ │ AUTO → Kernel only if UC + catalogManaged feature flag │ │ NONE → V1 path (super.createTable) │ │ │ │ 2. If UC: preRegisterIfNeeded(ident, schema, parts, props) │ │ └→ Optional<CatalogTable> │ │ then injectStorageCredentials(ct, hadoopConf) │ │ then UCUtils.extractTableInfo(ct, spark) → Optional<UCTableInfo> │ │ │ │ 3. CreateTableBuilder.prepare(ident, schema, parts, props, hadoopConf, ucTableInfo) │ │ └→ DDLContext │ │ │ │ 4. CreateTableBuilder.buildTransaction(ddlContext) → Transaction │ │ │ │ 5. txn.commit(engine, emptyIterable()) — metadata-only │ │ │ │ 6. return new SparkTable(ident, tablePath) — V2 connector │ └──────┬─────────────────────┬──────────────────────────┬──────────────────────┬───────────┘ │ │ │ │ ▼ ▼ ▼ ▼ STEP 2 STEP 3 STEP 4 STEP 5 Pre-register Prepare Build Txn Commit UC-only. Converts Spark Routes to Kernel txn.commit( Calls delegate types to Kernel: API based on engine, .createTable() ucTableInfo: emptyIterable() via UCProxy. sparkSchema → ) kernelSchema ┌── UC present? ───┐ UC assigns: (SchemaUtils) │ │ Kernel handles • tableId │ YES │ conflict • managed Transform[] → │ UCCatalog │ resolution, location DataLayoutSpec │ ManagedClient │ protocol, • ucUri (identity or │ .buildCreate │ and log write. • authConfig cluster-by) │ TableTxn( │ │ tableId, │ For UC: commit Returns properties → │ path, │ goes through Optional filtered props │ schema, │ UC committer. <CatalogTable> (strip DSv2 keys: │ engineInfo) │ location, provider, │ + UC committer │ For path-based: Non-UC: returns comment, owner, │ + UC properties │ writes directly Optional.empty() external, path, │ │ to filesystem. option.path) │ NO │ On failure: │ TableManager │ Returns drops pre- hadoopConf → │ .buildCreate │ TransactionCommit registered Engine │ TableTxn( │ Result with: catalog entry (DefaultEngine) │ path, │ • version (0) │ schema, │ • postCommit path resolved: │ engineInfo) │ Snapshot UC → ucTableInfo └───────┬──────────┘ .tablePath non-UC → props Then applies: ["location"] • .withTableProperties(filtered) (throws if • .withDataLayoutSpec(layout) none found) • .build(engine) → Transaction Returns: DDLContext POJO (ident, tablePath, kernelSchema, filteredProps, comment, dataLayoutSpec, engine, ucTableInfo, engineInfo) ``` <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…6438) ## Description Add `DeltaV2TableManager` trait to `v2/interop/` that abstracts `DeltaLog`'s role as the entry point for Delta table operations. `DeltaLog` extends this trait. This is the foundation for migrating Delta protocol reading from `DeltaLog` to Kernel. The trait defines the minimal surface that commands and `OptimisticTransaction` need from the table manager: - Table identity: `logPath`, `dataPath`, `clock`, `options`, `tableId` - Configuration: `newDeltaHadoopConf()` - Snapshot access: `snapshot`, `unsafeVolatileSnapshot`, `tableExists` - Setup: `createLogDirectoriesIfNotExists()` With this trait in place, a future `KernelTableManager` can implement the same interface via Kernel instead of log replay, and all existing business logic (commands, `OptimisticTransaction`) works with either backend. **This PR has zero behavior change** — `DeltaLog` already implements every method defined in the trait, so `with DeltaV2TableManager` is a no-op. ## How was this patch tested? All existing `DeltaLog` tests exercise the new trait. No test changes needed. ## Does this PR introduce _any_ user-facing changes? No.
… test (delta-io#6348) ## Description Fixes a flaky test in `DeltaRetentionWithCatalogOwnedBatch1Suite` that fails intermittently with: ``` 20 did not equal 0 Delta files before the last checkpoint version should have been deleted (DeltaRetentionSuite.scala:152) ``` ### Root Cause The test `"log files being already deleted shouldn't fail log deletion job"` fails **only when executed between 00:00-01:00 UTC**. The mechanism: 1. `getStartTimeForRetentionTest()` returns today at 01:00 UTC — which is a **future** timestamp if the test runs before 01:00 UTC. 2. Delta files get `setLastModified` to this future time (01:00:xx UTC). 3. V2 checkpoint files (introduced by CatalogOwned in delta-io#5841) use UUID-based names (`*.checkpoint.<uuid>.json`) that don't match `checkpointFileSingular` format (`*.checkpoint.parquet`), so the test never sets their timestamps — they keep their real wallclock time (~00:04 UTC). 4. `BufferingLogDeletionIterator` checks `needsTimeAdjustment()` **before** `isCheckpointFile()`. When a V2 checkpoint (modTime=00:04) follows a delta file (modTime=01:01), the checkpoint's earlier real timestamp triggers time adjustment, causing it to be treated as a regular file instead of a checkpoint boundary. The buffer is never flushed, and zero files get deleted. ### Fix Replace the `checkpointFileSingular` lookup (lines 133-136) with the `getCheckpointFiles` + `checkpointVersion` pattern already used and working correctly elsewhere in the same file (lines 724-734). This finds all checkpoint file formats including V2. ### Verified Reproduced the failure by forcing `getStartTimeForRetentionTest` to return a future timestamp: - **Without fix**: `FAILED` — `20 did not equal 0` (identical to CI failures) - **With fix**: `PASSED` Confirmed all CI failures occurred in the 00:00-01:00 UTC window: 00:04:33, 00:13:10, 00:21:48, 00:04:47 UTC. ## How was this patch tested? Existing test `DeltaRetentionWithCatalogOwnedBatch1Suite."log files being already deleted shouldn't fail log deletion job"` — this PR fixes it. ## Does this PR introduce _any_ user-facing changes? No. Test-only change. This pull request was AI-assisted by Isaac.
…-io#6443) Extract the file name construction logic from DelayedCommitProtocol.getFileName into a shared companion object method DelayedCommitProtocol.buildFileName. This enables reuse of the standard Delta file naming convention (part-NNNNN-uuid.ext / cdc-NNNNN-uuid.ext) by both the V1 write path (DelayedCommitProtocol) and the DSv2 write path (DataWriter), ensuring consistent file naming across both code paths. Changes: - Move file name building logic to DelayedCommitProtocol.buildFileName companion method - Simplify getFileName to delegate to the new shared method - Preserve all existing behavior (test prefix, CDC prefix, split formatting)
…leanup race (delta-io#6371) ## Summary Fixes a flaky test failure in `V2StreamingReadTest.testStreamingQueryStopDoesNotSurfaceException` that occurs intermittently in CI. ### Error ``` org.junit.platform.commons.JUnitException: Failed to close extension context, took 6.353s Caused by: java.io.IOException: Failed to delete temp directory ``` ### Root Cause The static `DeltaLog` cache holds references to snapshot files in `@TempDir` directories. When a streaming query stops, these cached instances keep file handles open. JUnit's `@TempDir` cleanup then races against these open handles during extension context closure. Under CI resource pressure (**4 parallel JVMs on 2 vCPUs**), this race is wide enough for the recursive deletion to block and exceed JUnit's ~5s extension context close timeout, producing a flaky `JUnitException`. ### Fix Call `DeltaLog.clearCache()` in `V2TestBase` at two points: - **`tearDown()`**: Before `spark.stop()`, release cached DeltaLog instances so the class-level `@TempDir` can be cleaned up. - **`processStreamingQuery()`**: After `query.stop()`, release cached instances so the method-level `@TempDir` can be cleaned up. This is the same approach already used in `testStreamingQueryStopDoesNotSurfaceException` itself (which calls `DeltaLog.clearCache()` at line 299), but applied to the shared test infrastructure so all test methods benefit. ### Reproduction The flakiness was **reproduced locally** by simulating CI conditions: ```bash # Unfixed: 1 out of 4 instances FAILS taskset -c 0-1 ./build/sbt "sparkV2/testOnly ...V2StreamingReadTest" & # x4 parallel # → StreamingQueryException: Thread was interrupted # Fixed: all instances PASS across 3 rounds of 2-parallel runs taskset -c 0-1 ./build/sbt "sparkV2/testOnly ...V2StreamingReadTest" & # x2 parallel, 3 rounds # → Passed: Total 8, Failed 0 (every run) ``` ### CI Evidence - **Failed run**: https://github.com/delta-io/delta/actions/runs/23449372021 (attempt 1, Shard 1) - **Passed re-run**: Same run, attempt 2 — zero code changes, same SHA `f864a12c` ## Test plan - [x] `sparkV2/testOnly ...V2StreamingReadTest` — all 8 tests pass - [x] Reproduction under CI-like conditions (parallel JVMs, 2 cores): unfixed fails, fixed passes - [x] No changes to production code — test infrastructure only This pull request was AI-assisted by Isaac.
#### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Change master version to 4.2.0-SNAPSHOT ## How was this patch tested? CI. ## Does this PR introduce _any_ user-facing changes? No Signed-off-by: Yi Li <yi.li@databricks.com>
) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The Delta protocol specifies that `offset` is optional for UUID and path-based Deletion Vector descriptors. Early implementations stored one DV per file (no offset needed); later implementations combined multiple DVs into a single file for performance, making offset necessary in practice. The protocol only specifies how to read DVs — a missing offset means read from offset 0. The factory methods (`onDiskWithRelativePath`, `onDiskWithAbsolutePath`) and downstream usage sites already treat `offset` as optional (defaulting to `None` / `getOrElse(0)`). However, the `serializeToBase64` helper introduced in PR #98496 asserts that `offset.isDefined` for non-inline DVs, which contradicts the protocol and is inconsistent with the rest of the codebase. This PR: - Removes the assertion in `serializeToBase64()` that required offset to be present for non-inline DVs - Uses uppercase storage type markers (`U`, `P`) in the base64 format to signal that no offset bytes follow, preserving the `None` vs `Some(0)` distinction through round-trips - Updates `deserializeFromBase64()` to decode the new markers back to the canonical lowercase storage type with `offset = None` ## How was this patch tested? New base64 round-trip tests for inline DVs and on-disk DVs with both `offset=None` and `offset=Some(25)`, plus existing UTs. ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR provides implementation to `DeltaTable` and `DeltaCatalog`. We add two new classes: `UnityCatalog` and `CatalogManagedTable` `UnityCatalog` implements `DeltaCatalog` and supports * getTable * listTable * createTable * credential vending `CatalogManagedTable` implements `DeltaTable` and supports * commit to a catalog support catalog-managed Delta tables For completeness, we also add `HadoopCatalog` and `HadoopTable` that supports path-based Delta table access with static credentials. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UT/Int <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
… transitions (delta-io#6439) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Sharing) ## Description This PR adds the back-conversion path for the "unlucky case" where a legacy checkpoint is mid-version: - Add convertDeltaSourceOffsetToLegacyOffset helper that converts a DeltaSourceOffset back to a legacy DeltaSharingSourceOffset. This is needed when the stream restarts from a legacy checkpoint mid-version (index != -1) and cannot finish the current version in a single batch. - In latestOffset, after computing the end offset, check whether both start and end offsets are still mid-version during a legacy transition. If so, convert the end offset back to legacy format so the next micro-batch continues using the legacy file-id hash (MD5). Once the stream reaches a version boundary, the conversion stops and the stream fully transitions to DeltaSourceOffset. ## How was this patch tested? New test case ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
…a write codes (delta-io#6386) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description We have this pattern of create a table relation from the current txn, in order to create a `LogicalPlan` and execute that `LogicalPlan`. We have an incoming PR that will add additional logic to this pattern, so best to refactor them. ## How was this patch tested? Existing UTs, just a refactoring. ## Does this PR introduce _any_ user-facing changes? No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add implementation of Flink Writer and Committers <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UT <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…-io#6315) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Support converting uncommitted txn for Delta UniForm. This is part of external Delta UniForm write support. ## How was this patch tested? Add UTs ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite" ```
…BigDecimal (delta-io#6259) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description `Literal.ofDecimal(BigDecimal value, int precision, int scale)` creates a `DecimalType(precision, scale)` directly from the caller-provided arguments. However, Java's `BigDecimal.precision()` returns the count of significant digits in the unscaled value, not the SQL precision. This can be less than `scale()` -- for example, `BigDecimal.valueOf(0, 18)` has `precision()=1` and `scale()=18`. When a caller passes `bd.precision()` as the precision argument (as Spark's `ExpressionUtils.convertValueToKernelLiteral` does), `ofDecimal` creates `DecimalType(1, 18)` which violates the Kernel constraint `scale <= precision` and throws `IllegalArgumentException: Invalid precision and scale combo`. This surfaces during data-skipping in Spark's DSv2 path, where predicate literals may have a small unscaled value with a large scale after Spark's type widening. Changes: - `ofDecimal()`: Adjust precision upward to `max(precision, scale)` before constructing the `DecimalType`, so the invariant `precision >= scale` always holds. Related: delta-io#6257 (fixes a different Decimal bug in the expression evaluator where casts between Decimal types with different precisions were unsupported) ## How was this patch tested? New unit tests in `ExpressionsSuite`: - `ofDecimal: adjusts precision when scale exceeds caller-provided precision` -- the core bug repro (`BigDecimal.valueOf(0, 18)`) - `ofDecimal: normal case with precision >= scale is unchanged` -- no regression for normal inputs - `ofDecimal: rejects value that exceeds adjusted precision` -- validation still fires when the value genuinely overflows ## Does this PR introduce _any_ user-facing changes? No. --------- Co-authored-by: rapoth <rapoth@users.noreply.github.com>
…s and validation (delta-io#6445) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description `replaceOn` and `replaceUsing` are new Selective Overwrite options for Delta tables, complementing the existing `replaceWhere`: - **`replaceOn`**: Accepts a matching condition (e.g. `.option("replaceOn", "t.id = s.id")`) to determine which existing rows to replace with incoming data. - **`replaceUsing`**: Accepts a comma-separated list of columns (e.g. `.option("replaceUsing", "id")`) to match existing rows by equality on those columns. Both provide row-level replacement semantics — matched rows are deleted and replaced by the new data, while unmatched existing rows are preserved. This PR introduces the option definitions, feature gate confs, and validation logic that prevents invalid option combinations. The actual execution paths will follow in a subsequent PR. ## How was this patch tested? - `DeltaInsertReplaceOnOrUsingDFOptionCombinationSuite` — tests all invalid option combinations with both `replaceOn` and `replaceUsing` - `DeltaInsertReplaceOnOrUsingDFv2CreateReplaceBlockingSuite` — tests DFv2 `create()`/`replace()`/`createOrReplace()` blocking - `DeltaSuite` — tests `replaceOn`/`replaceUsing` blocked when feature confs are disabled ## Does this PR introduce any user-facing changes? Yes. Two new DataFrame writer options `replaceOn` and `replaceUsing` are now recognized by Delta, along with their feature gate confs. When the confs are disabled (default: enabled), using these options will throw `DELTA_OPERATION_NOT_ALLOWED`. When enabled, invalid combinations with other options (e.g. `overwriteSchema`, `dataChange=false`) will throw descriptive errors. The actual replace execution is not yet implemented in this PR.
… eviction (delta-io#6334) ## What changes are proposed in this pull request? CI builds on PRs may be compiling dependencies from scratch even though `build.sbt` rarely changes and `actions/cache` should be providing warm caches. **Root cause:** GitHub Actions caches are scoped by git ref. A PR's cache can only be read by that same PR, never by sibling PRs. Every PR writes cache entries (across 5 active workflows with 20+ matrix jobs) that no other PR can use. This fills the 10GB cache budget, LRU-evicts master's shared entries, and forces every PR to recompile from scratch. **Fix:** Split `actions/cache` into `actions/cache/restore` (runs on all events) and `actions/cache/save` (only on `push` to `master`). PRs still read from master's cache (exact key hit when dependencies match), but no longer write their own entries. Master's cache stays alive, and every PR benefits from it. The save condition checks both `event_name == 'push'` and `ref == refs/heads/master` to prevent accidental cache writes from non-master push events. All cache actions are SHA-pinned to `actions/cache@0057852b...` (v4.3.0), consistent with the repo's supply-chain hardening convention. Also normalizes cache paths in `kernel_test.yaml` and `flink_test.yaml` to match other workflows (`~/.ivy2` instead of `~/.ivy2/cache`, drops redundant `~/.coursier/cache`). ### Workflows modified (5 active): - `build.yaml` — 1 cache key - `spark_test.yaml` — N cache keys (per Spark version) - `spark_examples_test.yaml` — N cache keys (per Spark version) - `kernel_test.yaml` — 1 cache key - `flink_test.yaml` — 1 cache key Disabled workflows (`disabled_iceberg_test.yaml`, `disabled_spark_python_test.yaml`, `disabled_publish_docs.yaml`) are not modified. ## How was this change tested? All CI checks passed on a previous iteration of this PR (before rebase). The `cache/restore` step works identically to the old `cache` step for reads. ### When will the benefit be visible? **This PR itself does not show dramatic speed improvements.** The optimization is about **preventing future degradation**, not speeding up individual runs. 1. **Today**, this PR reads from master's existing cache via `actions/cache/restore` — same as before. The change is that it **does not write** cache entries (by design). 2. **The real benefit comes after merging.** Currently, every PR writes cache entries that evict master's caches via LRU. After this merges: - Master pushes write caches (and keep them warm) - PRs only read, never write - Master's caches survive LRU eviction indefinitely - **All future PRs** get consistent cache hits instead of random misses 3. **Where savings will show:** The SBT dependency resolution + compilation phase is where cache hits save time. On a cold cache miss, SBT dependency resolution alone can add **5-15 minutes per job**. With 20+ matrix jobs per PR, this prevents significant wasted aggregate CI time when cache misses occur. ### Post-merge verification plan: 1. Observe master's CI run writes caches (`"Saved cache"` in post-action logs) 2. Open a new PR and verify it hits master's cache and does NOT write its own 3. Compare compile times on new PRs against pre-merge baseline Inspired by [delta-io/delta-kernel-rs#2130](delta-io/delta-kernel-rs#2130).
…letion Vectors enabled (delta-io#6329) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description In [this PR](delta-io#6328), we blocked preservation of variant stats until the preservation of stats during DML commands is improved. This PR adds Z85 encoding of variant stats during writes when deletion vectors are involved and it doing so, it re-enables the config controlling the preservation of variant stats. ## How was this patch tested? Test making sure variant stats are preserved after DML commands. ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (CI / Build Infrastructure) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> **Supply chain hardening: replace pip/pipenv with UV, add hash-verified lock files, and harden the Dockerfile.** This PR migrates all Python package management in CI workflows and the Docker build from pip/pipenv to [uv](https://github.com/astral-sh/uv), with cryptographic integrity verification and temporal lockdown to mitigate supply chain attacks targeting PyPI. ### Changes **1. UV-ify disabled CI workflows (`disabled_iceberg_test.yaml`, `disabled_spark_python_test.yaml`)** - Replace `pyenv install` + `pipenv` with `uv python install` + `uv venv` + `uv pip install` - Install uv via inline SHA256-verified tarball download (no GitHub Action dependency) - Install all Python packages from hash-verified lock files using `--require-hashes` - Add `UV_EXCLUDE_NEWER="2026-03-10T00:00:00Z"` workflow-level env as a temporal firewall — uv refuses any package version published after this date - pyspark installed separately with `--no-deps` (its only transitive dep, py4j, is included in the lock file) - Workflows remain disabled (`workflow_dispatch` + `if: false`); the UV changes are staged so that re-enabling is just removing the disable guards - Add SHA256 checksum verification for the `buf` binary download **2. Add hash-verified lock files for CI (`spark4.0.lock`, `spark4.1.lock`)** - `.github/ci-requirements/spark-python/spark4.0.in` and `spark4.1.in` — direct dependency declarations (extracted from the original per-package `pipenv run pip install` lines) - `.github/ci-requirements/spark-python/spark4.0.lock` and `spark4.1.lock` — full transitive closure with SHA256 hashes, generated by `uv pip compile --generate-hashes` with `UV_EXCLUDE_NEWER` - Spark 4.0 and 4.1 have separate lock files due to differing grpcio/protobuf version requirements - These lock files were validated end-to-end in CI on the [seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta) staging repo before being ported here - To add a new dependency to the spark_python_test set of python dependencies, update the corresponding cross-spark version's `.in` file and regenerate the lock file using the command: ``` UV_EXCLUDE_NEWER="2026-03-10T00:00:00Z" uv pip compile \ --python-version 3.10 \ --python-platform linux \ --generate-hashes \ -o sparkx.x.lock \ sparkx.x.in ``` **3. Lock developer tooling (`dev/requirements.lock`)** - Compiled from `dev/requirements.txt` (mypy, flake8, black, grpcio, mypy-protobuf) using `uv pip compile --generate-hashes` with `UV_EXCLUDE_NEWER` - Pins the previously-unpinned grpcio/googleapis-common-protos ranges to specific versions with hashes - 19 packages, 249 lines **4. Harden `disabled_publish_docs.yaml`** - `pnpm install` → `pnpm install --frozen-lockfile` (enforces lock file in CI, prevents drift) - Pin `netlify-cli` to `23.15.1` (published 2026-02-04, pre-attack baseline; was previously unpinned) **5. Harden Dockerfile** - Pin base image by SHA256 digest: `ubuntu:focal-20221019@sha256:450e06...` - Replace `python3-pip` + 14 bare `pip3 install` commands with uv (SHA256-verified musl binary) - All packages installed from `docker-build-requirements.lock` with `--require-hashes` (43 packages, full transitive closure) - Add `UV_EXCLUDE_NEWER="2026-02-10T00:00:00Z"` as temporal firewall - Isolated `/opt/venv` instead of system-wide pip install - Collapse 8 `RUN apt-get` layers into single layer with `--no-install-recommends` and cache cleanup - Fix legacy `ENV key value` → `ENV key=value` format - `ARG UV_INDEX_URL` build arg for optional PyPI mirror/proxy ### Files changed (11 files, +2684/−112) | File | Change | |------|--------| | `disabled_iceberg_test.yaml` | UV-ified (pyenv/pipenv → uv), still disabled | | `disabled_spark_python_test.yaml` | UV-ified (pyenv/pipenv → uv), still disabled | | `spark4.0.in` / `spark4.0.lock` | New — CI lock file for Spark 4.0 test variant | | `spark4.1.in` / `spark4.1.lock` | New — CI lock file for Spark 4.1 test variant | | `dev/requirements.lock` | New — hash-verified dev tooling lock | | `disabled_publish_docs.yaml` | `--frozen-lockfile`, pin netlify-cli | | `Dockerfile` | Full rewrite: digest-pinned base, uv, lock file, venv | | `docker-build-requirements.in` / `.lock` | New — Dockerfile dependency lock | ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> 1. **CI workflow changes validated end-to-end in sandbox repo**: The UV-ified `iceberg_test.yaml` and `spark_python_test.yaml` were pushed to [seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta) with the disabled guards removed, and all CI checks passed (uv install, venv creation, hash-verified package installation, sbt test execution). 2. **Lock file generation verified**: All lock files were generated with `UV_EXCLUDE_NEWER` set to the pre-attack baseline date. Every package version in the lock files pre-dates the cutoff. 3. **Byte-identical verification**: The workflow files in this PR were diffed against the CI-validated sandboxed-delta versions — the only differences are the disable guards (`workflow_dispatch`, `if: false`, `[DISABLED]` name prefix). 4. **Dockerfile build tested locally**: `docker build . -t test_delta_spark --platform=linux/amd64 --build-arg UV_INDEX_URL=<proxy>` — uv SHA verification passes, venv creation succeeds, hash-verified package installation succeeds. 5. **netlify-cli version audit**: `23.15.1` confirmed via npm registry API as published 2026-02-04, before the Feb 10 cutoff. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No. All changes are to CI infrastructure, the Docker build image, and developer tooling. The published `delta_spark` Python package, its `install_requires`, and all Scala/Java code are unchanged.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR adds support to Sink SQL API <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UT/Int <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Provide README and Docker Compose for easy testing <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? No code change <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…lta-io#6263) ## Summary - Adds a **non-blocking** CI workflow (`spark_test_uc_master.yaml`) that runs UC integration tests (`sparkUnityCatalog/test` and `kernelUnityCatalog/test`) against Unity Catalog built from source (main branch) - Makes the UC version configurable at build time via `-DunityCatalogVersion` system property (defaults to the current pinned release `0.4.0`); the new workflow derives the version dynamically from UC's `version.sbt` after building from source - Includes a `setup_unitycatalog_main.sh` script that clones UC, builds it, and publishes jars to the local Maven cache <img width="993" height="145" alt="Screenshot 2026-03-31 at 6 39 07 PM" src="https://github.com/user-attachments/assets/976718b0-ce0a-4adf-b682-364b5053db39" /> ## Motivation Today, Delta CI pins to the last UC release. This means: 1. **Incompatibilities are discovered in bulk** at version-bump time near release — costly to fix under pressure 2. **No way to distinguish transient UC bugs from real Delta issues** — when UC has a bug already fixed on main but not yet released, Delta CI is red on every PR with no signal that it's transient This non-blocking job gives a second signal on every PR: | UC Release | UC Main | Interpretation | |---|---|---| | Pass | Pass | All good | | Pass | Fail | Heads-up: upcoming UC change will require a Delta patch | | Fail | Pass | UC bug already fixed on main — transient, safe to land | | Fail | Fail | Real issue in Delta code | The job is **not a required status check**, so it cannot block PR merges. Reviewers see the signal and can choose to investigate or not. ## CI cost impact **Zero impact on critical path.** The existing CI takes ~81 min end-to-end (dominated by the Delta Spark sharded tests). The new UC master job completes in ~13 min — well before the existing jobs finish — so total wall-clock time remains ~81 min. Total compute cost increase is ~1.3% (13 runner-minutes on top of ~1040 existing runner-minutes). ## Design choices - **Targeted tests only**: Runs only the UC integration test suites (`sparkUnityCatalog/test`, `kernelUnityCatalog/test`) rather than the full sharded test suite. This keeps the workflow to a **single CI job** while still catching UC API/behavior breakages. - **Default Spark version**: Lets SBT use its own default Spark version rather than testing all versions, since we're testing UC compatibility, not Spark version compatibility. - **No hardcoded UC version**: The setup script extracts the version from UC's `version.sbt` after cloning, so the workflow automatically adapts when UC bumps versions. ## Changes | File | What | |------|------| | `build.sbt` | `unityCatalogVersion` now reads from `-DunityCatalogVersion` system property, defaulting to `0.4.0` | | `project/scripts/setup_unitycatalog_main.sh` | New script: clones UC repo, builds from source, publishes to `~/.m2`, outputs the built version | | `.github/workflows/spark_test_uc_master.yaml` | New single-job workflow: builds UC from main, runs UC integration tests | ## Test plan - [ ] Verify the existing `spark_test.yaml` workflow is unchanged and still passes - [ ] Verify the new `spark_test_uc_master.yaml` workflow runs and reports status without blocking merge - [ ] Verify `setup_unitycatalog_main.sh` successfully builds UC from source - [ ] Confirm the new workflow is NOT listed as a required status check in branch protection --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…elta-io#6319) ## Description Support for schema evolution in INSERT landed in Spark master: apache/spark#54488 A few changes are required to maintain compatibility in Delta: - V2Write commands (AppendData, OverwriteByExpression, ..) now have an extra parameter `withSchemaEvolution` and introduce a few methods to implement for schema evolution for V2Write commands (e.p. `writePrivileges`) - Spark doesn't set delta's writer option "mergeSchema" anymore, which was used as workaround until a `withSchemaEvolution` was introduced on write plan nodes. Instead, Spark now sets `withSchemaEvolution` directly, and a pre-resolution rule `PropagateSchemaEvolutionWriteOption` is added in Delta to set the writer option `mergeSchema` when `withSchemaEvolution=true` if it's not already explicitly set by the user. ## How was this patch tested? Existing tests ## Does this PR introduce _any_ user-facing changes? No
… ran it (delta-io#6270) ## Problem `performIcebergCompatUpgradesAndValidation()` runs IcebergCompatV3 validation twice when WriterCompatV3 is enabled: once nested as a post-processor, and once standalone. This causes redundant schema traversals and feature checks. ## Fix Track whether V3 was already validated by WriterCompatV3 and skip the standalone call if so. Same optimization applied for V2/V1. ## Validation - Confirmed by reading TransactionMetadataFactory.java lines 480-531. - Tests verify both paths: WriterCompatV3+CompatV3 (runs once) and CompatV3-only (still runs). ## Regression Prevention Test cases cover both the optimized and standalone validation paths. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…in IcebergConverterSuite (delta-io#6430) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Support verifying read result on Iceberg metadata location inside Iceberg test ## How was this patch tested? Test-only change ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite" ```
…6468) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (test infrastructure) ## Description Pass the `EXTRA_MAVEN_REPO` environment variable into the Docker container when running tests via the Docker path in `run-tests.py`, so that Python tests inside Docker can resolve artifacts from a customized Maven repository (e.g., a staging repo for release verification). Currently, when running Python tests inside Docker (`USE_DOCKER=1 ./run-tests.py --group spark-python`), the `EXTRA_MAVEN_REPO` env var is not forwarded into the container. This means the PyPi packaging test (`using_with_pip.py`) inside Docker cannot pick up the custom Maven repo. ### Where is `EXTRA_MAVEN_REPO` consumed? `EXTRA_MAVEN_REPO` is read by several downstream scripts to configure `spark.jars.repositories`: | File | How it's used | |------|--------------| | `examples/python/using_with_pip.py` | `builder.config("spark.jars.repositories", os.getenv('EXTRA_MAVEN_REPO'))` | | `examples/python/image_storage.py` | Same as above | | `examples/scala/build.sbt` | `sys.env.get("EXTRA_MAVEN_REPO")` added as a resolver | | `run-integration-tests.py` | Sets `env["EXTRA_MAVEN_REPO"]` for integration test subprocesses | ### Call stack: Python tests in Docker ``` [Host] USE_DOCKER=1 ./run-tests.py --group spark-python │ ├─ pull_or_build_docker_image() # run-tests.py:288 │ └─ run_tests_in_docker(image_tag, "spark-python") # run-tests.py:289 Forwards env vars into Docker container: JENKINS_URL, SBT_1_5_5_MIRROR_JAR_URL, SCALA_VERSION, TEST_PARALLELISM_COUNT, DISABLE_UNIDOC, EXTRA_MAVEN_REPO ← this PR │ └─ docker run ... ./run-tests.py --group spark-python │ [Inside Docker — USE_DOCKER is NOT forwarded, prevents recursion] │ └─ run_python_tests(root_dir) # run-tests.py:291→101 │ └─ python3 python/run-tests.py # run-tests.py:105 │ ├─ prepare() # python/run-tests.py:65 │ └─ sbt clean publishM2 │ (builds Delta jars → ~/.m2/repository) │ ├─ run_python_style_checks() # lint ├─ run_mypy_tests() # type checking │ ├─ run_pypi_packaging_tests() # python/run-tests.py:165 │ ├─ pip install delta-spark wheel │ └─ python3 examples/python/using_with_pip.py │ └─ reads EXTRA_MAVEN_REPO ← consumed here │ builder.config("spark.jars.repositories", ...) │ SparkSession starts with custom Maven repo │ └─ test(root_dir, "delta", ...) # python/run-tests.py:26 └─ spark-submit \ --repositories file://~/.m2/repo,<public repos> \ --packages io.delta:delta-spark_2.13:X.Y.Z \ <test_file.py> ``` ## How was this patch tested? Manually verified the env var is correctly forwarded into the `docker run` command. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [x] Spark - [x] Standalone - [x] Flink - [x] Kernel - [x] Other (CI / GitHub Actions) ## Description Gate all active CI workflows to only trigger on pushes to `master` and pull requests targeting `master`. This is a follow-up to delta-io#6390 (supply chain attack mitigation) and delta-io#6372 (SHA-pinning GitHub Actions). **What changed**: Added `branches: [master]` to both `push` and `pull_request` triggers in all 7 active workflow files: | Workflow | Trigger style | |---|---| | `build.yaml` | Added `branches: [master]` to existing `push`/`pull_request` | | `flink_test.yaml` | Expanded `on: [push, pull_request]` → long form with `branches: [master]` | | `kernel_test.yaml` | Added `branches: [master]` to existing `push`/`pull_request` | | `kernel_unitycatalog_test.yaml` | Added `branches: [master]` to existing `push`/`pull_request` | | `spark_examples_test.yaml` | Added `branches: [master]` to existing `push`/`pull_request` | | `spark_test.yaml` | Added `branches: [master]` to existing `push`/`pull_request` | | `unidoc.yaml` | Expanded `on: [push, pull_request]` → long form with `branches: [master]` | The 3 disabled workflows (`disabled_iceberg_test.yaml`, `disabled_publish_docs.yaml`, `disabled_spark_python_test.yaml`) already use `workflow_dispatch` only and require no changes. **Why**: Without branch gating, every push to any branch and every PR targeting any branch triggers the full CI suite. This wastes GitHub Actions minutes on feature branches, forks, and experimental work that will eventually be validated when merged to `master` anyway. More importantly, in the context of the ongoing supply chain attack targeting OSS package ecosystems, restricting CI execution surface reduces the window for potential abuse via workflow triggers on non-default branches. **Effect**: Only pushes to `master` and PRs targeting `master` will trigger CI. All other branch activity is silenced. Existing `paths`/`paths-ignore` filters are preserved and compose with the branch filter (both conditions must match). ## How was this patch tested? These changes were validated in a mirror fork ([seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta)) before being ported here: 1. **Positive test**: Pushed a commit to `master` — all 7 workflows triggered as expected. 2. **Negative test**: Opened a PR from a non-master feature branch targeting a non-master base — zero workflows triggered (confirmed via the PR's Checks tab showing no status checks). 3. **PR-to-master test**: Opened a PR targeting `master` — all applicable workflows triggered correctly. 4. **Byte-level parity**: `diff -r` of `.github/workflows/` between this branch and the validated sandbox confirms zero differences. ## Does this PR introduce _any_ user-facing changes? No. This change only affects when CI workflows run (branch-scoped gating). It does not change any build outputs, library behavior, or APIs. Contributors opening PRs against `master` (the standard workflow) will see no change in CI behavior.
…o#6456) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Change `build.sbt` to add `releaseSettings` to the flink project. Add a Flink runner to `run-integration-tests.py` <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Build change and test only change. Manually tested <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
… data skipping (delta-io#6444) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> In this PR we are adding geospatial table feature as per spec: This is **PR 2 of 3** for geospatial data skipping support. It adds the `StGeometryBoxesIntersect` public expression and its internal translation to `ST_GEOMETRY_BOEXES_INTERSECT_ON_STATS` for data skipping, including expression evaluation and end-to-end data skipping tests. To avoid introducing complex geospatial calculations into kernel and default kernel, we don't allow evaluating ST_GEOMETRY_BOXES_INTERSECT predicate onto rows, we only use it to to translate it to stats predicate which is trivial to implement and does not require complex geometric algorithms. Geography boxes intersection is more invovled and requires accounting for antimeridian and the fact that min can bigger than max, so we leave that for other PRs. This is following delta protocol delta-io#4726 ### Changes - **StIntersectsBoxes** (new): Public predicate taking column + WKT query bounds (queryMin, queryMax) - **DataSkippingUtils**: Translate `ST_INTERSECTS_BOXES` to `ST_INTERSECTS_BOXES_ON_STATS` with min/max stats columns - **ExpressionVisitor**: Add `visitStIntersectsBoxesOnStats` dispatch - **DefaultExpressionEvaluator**: Transform (type validation) and eval (AABB intersection via WKT parsing with GeometryUtils) ## How was this patch tested? - `GeometryDataSkippingSuite`: 5 end-to-end tests covering 4-quadrant layout, null stats handling (conservative inclusion), and combined AND predicates with geometry + scalar columns. Tests bypass Parquet writer by injecting `DataFileStatistics` with WKT string literals directly. ## Does this PR introduce _any_ user-facing changes? New expression
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
In this PR we are adding geospatial table feature as per spec: delta-io#4726
This is PR 3 of 3 for geospatial data skipping support. It adds Parquet read/write support for geometry/geography columns (as WKB binary), including Parquet schema mapping with GEOMETRY/GEOGRAPHY logical type annotations, and reading geospatial bounding box statistics from Parquet file metadata.
Depends on: PR A - Geospatial stats parsing (will rebase to master after PR A merges)
Changes
GeometryType/GeographyTypein constructor andgetBinary()BinaryColumnReaderBinaryWriterGeometryType-> BINARY withgeometryType(srid)annotation;GeographyType-> BINARY withgeographyType(srid, algorithm)annotationGeospatialStatisticsbounding box from Parquet metadata; convert to WKT string literal viaGeometryUtils.formatPointWKT()PR dependency graph
How was this patch tested?
GeometryParquetReaderSuite: 8 tests covering WKB roundtrip, null rows, all-null batch, getString throws for both GeometryType and GeographyTypeParquetSchemaUtilsSuite: 9 new tests covering GeometryType/GeographyType Parquet schema conversion, algorithm mapping, and invalid algorithm rejectionDoes this PR introduce any user-facing changes?
Adds new table feature