Repository navigation
Conversation
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Before we support schema tracking log, we should disallow all non-additive schema change in dsv2. Column Mapping related code change is pure refactoring that for v2 to reuse the utilities. ## How was this patch tested? Unit test: - New test case for drop column in a nested struct - Modify existed tests for non-additive schema change to throws the same error as v1 Integration test: - drop column: should fail with non-additive schema change error - drop column: should succeed with unsafe column mapping schema change flag enabled - rename column: should fail with non-additive schema change error - rename column: should throw schema change error with unsafe flag enabled - type widening: should fail with non-additive schema change error when enable schema tracking <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…ty (delta-io#6227) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6227/files) to review incremental changes. - [**stack/type-nullability-e2e-test**](delta-io#6227) [[Files changed](https://github.com/delta-io/delta/pull/6227/files)] - [stack/forbid-schema-evolution-with-staleDF-v2](delta-io#6228) [[Files changed](https://github.com/delta-io/delta/pull/6228/files/cafe746303c3e32bdcf5499c8e99fd6a59c033c4..94b77dc62d21e896e9f4fdafd3ff17fafddd606f)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Added schema evolution e2e tests for type widening and relax nullability for both refresh dataframe and stale dataframe cases. Refresh dataframe cases work for both v1 & v2 connector, while stale dataframe cases only work for v1 connector. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? test-only PR <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…m 50000 to 100000 (delta-io#6240) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Double the default value of `DELTA_STREAMING_INITIAL_SNAPSHOT_MAX_FILES` from 50,000 to 100,000 to better support larger tables for V2 streaming initial snapshots out of the box. ## How was this patch tested? Existing tests continue to pass since this is a default-value-only change. Tests that explicitly set this config (e.g., `SparkMicroBatchStreamTest`) are unaffected. ## Does this PR introduce _any_ user-facing changes? Yes. The default maximum number of files allowed in the initial snapshot for V2 streaming is increased from 50,000 to 100,000. Users with tables between 50k–100k files will no longer hit the limit without explicitly raising the config. This change only affects the unreleased master branch and the future release.
…ype widening and relaxing nullability (delta-io#6228) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055) to review incremental changes. - [stack/type-nullability-e2e-test](delta-io#6227) [[Files changed](https://github.com/delta-io/delta/pull/6227/files)] - [**stack/forbid-schema-evolution-with-staleDF-v2**](delta-io#6228) [[Files changed](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add schema validation during stream restart e2e tests for type widening and relaxing nullability to block user from adopt these two schema change without refresh the dataframe in v2 connector <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Test-only PR <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…a-io#5837) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add the DataFrame basic test coverage for preventing future regression. ## How was this patch tested? Do the locally unit tests and remotely integration tests. ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: openinx <openinx@gmail.com>
… repartition the table's columns (delta-io#6238) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description - Updated partition column change tests in `OptimisticTransactionSuite` to use `checkError` instead of calling `ex.getErrorClass` directly - Each `checkError` call now verifies the error condition (`DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE`), SQLSTATE (`42P10`), and named parameter values (`operation`, `oldPartitionColumns`, `newPartitionColumns`) - Added expected operation name to the parameterized blocked-operations test cases to support per-case parameter verification - Collapsed the `DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE` error message from three lines into a single line in `delta-error-classes.json`, `databricks-error-classes.json`, and `delta-error-classes-data-classification.json` (the multi-line split was not justified as the column lists render as single-line comma-separated strings) ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> No new tests added. Existing tests in `OptimisticTransactionSuite` covering partition column change validation were updated to use `checkError` as required by the error message review guidelines. ## Does this PR introduce _any_ user-facing changes? N/A <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
delta-io#5775) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5775/files) to review incremental changes. - [**stack/dv_pr3_phase2_vectorized**](delta-io#5775) [[Files changed](https://github.com/delta-io/delta/pull/5775/files)] - [stack/dv_pr4_phase3_file_splitting](delta-io#5776) [[Files changed](https://github.com/delta-io/delta/pull/5776/files/463011af2fae489f7f3fe8c39fe86acbba06277b..0e534ed9c0257735523532c8c6f8e15ea33ae114)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Enable vectorized (columnar batch) reading for DV tables in the Spark V2 connector. ### Changes: - `ColumnVectorWithFilter`: Wraps `ColumnVector` with row ID mapping for zero-copy filtering (follows Iceberg pattern) - `DeletionVectorReadFunction`: Extended to handle `ColumnarBatch` in addition to `InternalRow` - Enable vectorized reader for DV tables (remove `!isTableSupportDv` condition) ### How it works: 1. For `ColumnarBatch`: Build live row mapping, wrap columns with `ColumnVectorWithFilter` 2. For `InternalRow`: Filter deleted rows, project using `ProjectingInternalRow` ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> - `ColumnVectorWithFilterTest`: Unit tests for various data types and filtering scenarios - `DeletionVectorVectorizedReaderTest`: Integration tests verifying vectorized path is used - All DV golden table tests pass with vectorized reading ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…a-io#6104) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Spark's StringStartsWith(attribute, value) filter was not converted to a Kernel predicate, meaning WHERE col LIKE 'prefix%' queries had no filter pushdown in the V2 connector. The Kernel's STARTS_WITH predicate maps directly: STARTS_WITH(column, string_literal). ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…sts (delta-io#6210) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add the DataFrame streaming integration tests to cover the spark + uc case. ## How was this patch tested? Run in both locally UC server and remote UC server. ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: openinx <openinx@gmail.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (sharing) ## Description Upgrade delta sharing client to 1.3.10 ## How was this patch tested? CI pipeline ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR blocks writes with NullType in UDT columns, similar to how we block NullType in arrays/maps. NullType is typically used as a temporary placeholder, which is later evolved into some other type. However, UDTs cannot evolve over time, hence NullType inside UDT remains as NullType forever. Additionally, our old check missed arrays/maps inside UDT and didn't check whether they contain NullType or not. When trying to write such Parquet files, the Parquet writer failed. This PR fixes these problems. ## How was this patch tested? New unit tests. ## Does this PR introduce _any_ user-facing changes? No.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR adds support to excludeRegex read option in deltaDSv2 <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Unit tests: Create table partitioned by category with 4 versions of data. ``` Version 0: CREATE TABLE (empty) Version 1: ids {1,2} in category=alpha Version 2: ids {3,4} in category=beta Version 3: ids {5,6} in category=alpha Version 4: ids {7,8} in category=gamma ``` Test cases: "No excludeRegex - all versions", "Exclude alpha category", "Exclude beta category", "Exclude gamma category", "Exclude alpha and beta categories", "Regex matching no files", "Regex matching all files", "Exclude alpha versions 2 to 4", "Exclude beta version 2 only", "Initial snapshot - no excludeRegex", "Initial snapshot - exclude alpha". Enable integration test: "excludeRegex works and doesn't mess up offsets across restarts - parquet version" ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6204) #### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Send table schema to UC server when creating managed tables ## How was this patch tested? Integration test ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: Yi Li <yi.li@databricks.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] General repository ## Description Adds a note regarding the required Java version into the README.md. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Inapplicable. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Co-authored-by: Leonid Lygin <leonid.lygin@databricks.com> Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Iceberg) ## Description We have the following NPE: ``` java.lang.NullPointerException at java.base/java.util.Objects.requireNonNull(Objects.java:209) at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:296) at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:144) at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:129) at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:138) at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114) ``` Referring to this line: https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/core/src/main/java/org/apache/iceberg/SchemaParser.java#L296 This comes from the following line: https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L144 The field `schema` is only set in V1 fields. I believe the reference implementation (Java OSS) also sets this field for V2/V3 to maximize compatibility, but this might be different for other writers. By passing in the spec, we don't execute the branch and we avoid the NPE: https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L126-L130 <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6203) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Convert CatalogStatistics to V2 connector Statistics and thread them through SparkTable → SparkScanBuilder → SparkScan.estimateStatistics(). Gate numRows/columnStats on cboEnabled || planStatsEnabled to match V1 LogicalRelation.computeStats() behavior. Use per-column avgLen from catalog stats in size estimation instead of defaultSize(), mirroring EstimationUtils.getSizePerRow() with UTF8String overhead for StringType columns. Resolves delta-io#5952 ## How was this patch tested? Unit test ## Does this PR introduce _any_ user-facing changes? No
…a-io#5888) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR adds a new Spark flag that will make all code paths in DataSkippingReader to collect stats. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> New tests. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No.
…iveCatalog for Delta UniForm (delta-io#6189) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Use UnityCatalog as Iceberg committing catalog instead of HiveCatalog for Delta UniForm ## How was this patch tested? Add UTs ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite" ```
…flictSuiteBase (delta-io#6277) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Add an overridable `sqlWithTotalRpcBound` method to `IdentityColumnConflictSuiteBase` that defaults to calling `sql()`. This is a no-op in OSS - it allows internal subclasses to override the method with custom assertions on the winning transaction's SQL execution path. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> No behavioral change - the new method defaults to `sql()`, identical to the existing behavior. All existing `IdentityColumnConflictSuite` tests pass unchanged. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No. Co-authored-by: Shlok Jhawar <shlok.jhawar@databricks.com>
…o#6271) ## Problem In `buildTransactionInternal()`, when `!needsMetadataOrProtocolUpdate`, a new `TransactionImpl` is constructed but never returned. Execution falls through to the full metadata/protocol update pipeline, doing unnecessary work and constructing a second `TransactionImpl`. Fixes delta-io#4713. ## Fix Add `return` before `new TransactionImpl(` so the early-exit path actually returns. The early return skips `TransactionMetadataFactory`, which also runs `validateRowTrackingConfigsNotMissing` on existing tables. To preserve this validation, it is moved into `validateWriteToExistingTable()` so it runs on every write regardless of whether metadata/protocol updates are needed. ## Validation - Confirmed by reading lines 230-247. - New unit test verifies the early-return path (uses a fake `delta.feature.*` property so the factory throws if the early return is bypassed). - Verified `LegacyRowTrackingSuite` passes (45/45 tests), including the row tracking config validation test. ## Regression Prevention - `TransactionBuilderImplSuite`: tests the no-update transaction path. - Existing `LegacyRowTrackingSuite`: covers the row tracking validation that was moved. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…lta-io#6278) ## Problem Spark's CheckDeletionVectorDisabled checks both newestProtocol and prevSnapshot to guard against concurrent writers enabling DVs. The Kernel's CHECK_HAS_NO_DELETION_VECTORS only checks newProtocol, missing the concurrent writer guard. ## Fix Add the previous protocol check to the Kernel's DV validation, matching the Spark implementation's defense-in-depth approach. ## Validation - Confirmed by comparing Spark IcebergCompat.scala with Kernel IcebergCompatMetadataValidatorAndUpdater.java. - New test verifies the check catches DV enablement from a previous protocol. ## Regression Prevention New test case explicitly validates the previous-protocol DV check. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
…n in dsv2 (delta-io#6246) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613) to review incremental changes. - [stack/ignoreDeletesV2](delta-io#6245) [[Files changed](https://github.com/delta-io/delta/pull/6245/files)] - [**stack/skipChangeCommitsV2**](delta-io#6246) [[Files changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)] - [stack/ignoreChangesV2](delta-io#6249) [[Files changed](https://github.com/delta-io/delta/pull/6249/files/c1c942553a20800e318a5d2d384f869a3c0c5d3c..7209f24b489535e57a8396d712f0f1da577520d2)] - [stack/ignoreFileDeletionV2](delta-io#6250) [[Files changed](https://github.com/delta-io/delta/pull/6250/files/7209f24b489535e57a8396d712f0f1da577520d2..8690f78ca1bae137706f46ed1e05cdb05702ee6b)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Support - skipChangeCommits read option is DSv2, which skip a commit that contains remove file actions regardless of if there is an add file action or not. - ignoreDeletes read option is DSv2, which skip a commit that contains only remove file actions but no add file action. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Unit tests that test parity between v1 and v2 connector on both pure deletes commit (only remove) and change commit (add + remove). Integration tests: - streaming with skipChangeCommits = true skips both delete and change commits <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6267) ## Problem `generateIcebergCompatWriterV3RemoveAction()` accepts `baseRowId` and `defaultRowCommitVersion` parameters but silently discards them, hardcoding `Optional.empty()` instead. This breaks row tracking for V3 writer compat remove actions. ## Fix Forward the caller-provided `baseRowId` and `defaultRowCommitVersion` to `convertRemoveDataFileStatus()` instead of passing `Optional.empty()`. ## Validation - Confirmed by reading the method: params accepted at lines 266-274, discarded at lines 308-310. - New test verifies non-empty values are preserved in the output Row. ## Regression Prevention New test case explicitly checks that baseRowId and defaultRowCommitVersion are forwarded through to the RemoveFile row. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description During data-skipping, we add one millisecond to timestamp max stats to compensate for the fact that JSON serialization will truncate them to milliseconds. This has the consequence that if the data contains a timestamp within the range (Long.MaxValue - 1000, Long.MaxValue], then the addition will throw an overflow exception. This PR fixes this by adding a check for if the stats is within that range, then we saturate the stats to Long.MaxValue instead of adding 1000 to it. ## How was this patch tested? Unit test ## Does this PR introduce _any_ user-facing changes? No
…io#6268) ## Problem `IcebergRESTCatalogPlanningClient` implements `AutoCloseable` but the `ServerSidePlanningClient` trait does not extend it, and `ServerSidePlannedTable` never calls `close()`. This leaks HTTP client connections. ## Fix Add proper lifecycle management so the HTTP client is closed when no longer needed. ## Validation - Confirmed by reading the trait hierarchy and table construction code. - New test verifies the client is properly closed. ## Regression Prevention New test verifies client lifecycle management. --------- Co-authored-by: foo.bar <foo.bar@foobar.com>
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Deflake PartitionLikeDataSkippingSuite by making result comparison insensitive to nondeterministic record ordering. ## How was this patch tested? See test change. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Block metadata changes on UC-managed tables Without this, metadata mutations on a managed table succeed locally but leave the UC catalog unaware, causing metadata drift. Two kill switches in OptimisticTransaction: - updateMetadata(): blocks schema, partition, description, and configuration changes. Covers INSERT with autoMerge, MERGE with schema evolution, and DataFrame writes with mergeSchema. - commitLarge(): blocks clustering DomainMetadata changes for RESTORE TABLE, which bypasses updateMetadata(). ALTER TABLE is already blocked by UCSingleCatalog before reaching Delta. ## How was this patch tested? New tests added. ## Does this PR introduce _any_ user-facing changes? Yes. Table schema evolution with DML is blocked for UC managed tables. --------- Signed-off-by: Yi Li <yi.li@databricks.com>
…in CRC file (delta-io#6282) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Added a conf to make delta-spark write fileSizeHistogram (and not histogramOpt) as field name in CRC file. ## How was this patch tested? UTs ## Does this PR introduce _any_ user-facing changes? No
…elta-io#6281) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description This PR updates Kernel's `CRCInfo` to **read** both `fileSizeHistogram` and `histogramOpt` fields to ensure backward compatibility with Delta-Spark. While the [Delta spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#version-checksum-file-schema) officially uses `fileSizeHistogram` in the Version Checksum file, Delta-Spark has historically used histogramOpt ([code](https://github.com/delta-io/delta/blob/5ba861fa6d6999d7079706a307a7e293b59d4880/spark/src/main/scala/org/apache/spark/sql/delta/Checksum.scala#L87)). Following [a recent fix](delta-io@021755f) that allows Delta-Spark to **read** both fields, this PR mirrors that compatibility in Kernel. This ensures smooth read operations as we fully transition Delta-Spark to write the correct `fileSizeHistogram` field. ## How was this patch tested? New unit tests. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…ale (delta-io#6354) ## Summary - Fix `ExpressionUtils.convertValueToKernelLiteral` crash when a `BigDecimal` has precision less than scale - Normalize precision to `Math.max(bd.precision(), bd.scale() + 1)` before calling `Literal.ofDecimal` - Add unit test for the edge case ## Problem A query with a decimal `IN` predicate containing `0.00` crashes when using the V2 connector: ```sql SELECT * FROM delta_table WHERE dec10_2 IN (0.00, 100.00) ``` This throws an `IllegalArgumentException: Invalid precision and scale combo` from Kernel's `Literal.ofDecimal`. **Root cause:** Java's `BigDecimal("0.00")` reports `precision()=1` and `scale()=2`. This violates the invariant that `precision >= scale` required by Kernel's `Literal.ofDecimal`. The V2 code in `ExpressionUtils.convertValueToKernelLiteral` was passing these values through without normalization. ## Fix Before constructing the Kernel literal, normalize precision: ```java int precision = Math.max(bd.precision(), bd.scale() + 1); ``` For `BigDecimal("0.00")`, this yields `precision=3, scale=2` (i.e., `DECIMAL(3,2)`), which correctly represents the value. ## Test plan - [x] New unit test `testConvertValueToKernelLiteral_DecimalWithScaleExceedingPrecision` passes - [x] All 73 `ExpressionUtilsTest` tests pass
…without a supported IcebergCompat version (delta-io#6352) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description When a user enables `delta.universalFormat.enabledFormats = 'iceberg'` alongside an unrecognized IcebergCompat property (e.g. `delta.enableIcebergCompatV3`), the error message currently says: > To enable IcebergCompatV2, set the table property 'delta.enableIcebergCompatV2' = 'true'. This is misleading because: 1. Spark Delta silently ignores the unrecognized V3 property (only V1 and V2 are in `IcebergCompat.knownVersions`) 2. The error then directs the user to V2 with no indication that V3 is not supported The fix updates the message to explicitly state that the supported versions are IcebergCompatV1 and IcebergCompatV2, making clear to users that higher versions are not yet supported in this Spark Delta release. Resolves delta-io#6351 ## How was this patch tested? The change is limited to an error message string. Manual reproduction: ```sql SET spark.databricks.delta.allowArbitraryProperties.enabled=true; CREATE TABLE demo.icebergCompatV3 (i INT, s STRING) USING DELTA TBLPROPERTIES ( 'delta.columnMapping.mode' = 'name', 'delta.enableIcebergCompatV3' = 'true', 'delta.enableDeletionVectors' = 'false', 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` **Before:** `"To enable IcebergCompatV2, set the table property 'delta.enableIcebergCompatV2' = 'true'."` **After:** `"Supported versions are IcebergCompatV1 and IcebergCompatV2."` ## Does this PR introduce _any_ user-facing changes? Yes. The error message for `DELTA_UNIVERSAL_FORMAT_VIOLATION` (when UniForm Iceberg is enabled without a recognized IcebergCompat version) now explicitly lists the supported IcebergCompat versions (V1 and V2), instead of directing the user to enable V2. This helps users who set an unsupported version (e.g. V3) understand why the error occurred. --------- Signed-off-by: openinx <openinx@gmail.com>
…commits (delta-io#6338) ## Description This PR adds a regression test that exposes a **data loss bug** in Delta streaming when used with Coordinated Commits and non-trivial backfill batch sizes. Related issue: delta-io#6339 ### Observed Behavior The test writes 100 sequential single-row commits to a Delta table while a streaming query is running, then verifies all 100 values appear in the sink. - **batchSize = 1**: Test passes consistently. All 100 values present. - **batchSize = 2**: Test fails. ~4 out of 100 commits are lost on average. **Only odd-versioned commits are lost** (these are the versions that are not immediately backfilled). - **batchSize = 3**: Test fails. The loss pattern follows the backfill cycle: - `v % 3 == 0`: no loss (these versions trigger backfill) - `v % 3 == 1`: v may be lost, and if v is lost, v+1 is always lost together (they get backfilled as a pair) - `v % 3 == 2`: v may be lost independently The pattern strongly correlates with which commits are sitting unbackfilled in the coordinator at any given time, suggesting the bug is related to the interaction between backfill and commit listing. ### Hypothesized Root Cause We suspect the issue is a race condition in `CoordinatedCommitsUtils.commitFilesIterator`, which is used by `DeltaSource.getFileChanges` during both `latestOffset` and `getBatch`. This method lists commits in two lazy, sequential steps: 1. List backfilled commits from the filesystem (`listedDeltas`) 2. Query the coordinator for unbackfilled commits (`tailFromSnapshot`) These two steps are **not atomic**. A possible scenario (example with batchSize = 3): 1. **During `latestOffset`**: filesystem has `[0.json]`, coordinator has `[1, 2]`. `latestOffset` correctly computes endOffset covering through version 2. 2. **During `getBatch`**: the filesystem listing iterator runs and sees `[0.json]`. 3. **Between the two iterators**: a concurrent write creates version 3, triggering `backfillToVersion(3)`. This writes versions 1, 2, 3 to the filesystem and **removes them from the coordinator** via `registerBackfill`. 4. **The coordinator query runs**: returns empty — versions 1 and 2 have been removed. 5. **Result**: `getBatch` misses versions 1 and 2. The next batch starts from version 3, so they are never re-read. ## How was this patch tested? Added a new test `"streaming processes 100 sequential single-value commits and contains all values 0 to 99"` that: - Creates a Delta table and starts a streaming query - Appends 100 single-row commits while the stream is running - Verifies all 100 values appear in the sink The test passes for batchSize = 1 but fails for batchSize = 2 and 3 due to the data loss. ## Does this PR introduce _any_ user-facing changes? No. This PR only adds a test to demonstrate the existing bug. A fix will follow in a subsequent PR.
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de) to review incremental changes. - [stack/ignoreDeletesV2](delta-io#6245) [[Files changed](https://github.com/delta-io/delta/pull/6245/files)] - [stack/skipChangeCommitsV2](delta-io#6246) [[Files changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)] - [**stack/ignoreChangesV2**](delta-io#6249) [[Files changed](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de)] - [stack/ignoreFileDeletionV2](delta-io#6250) [[Files changed](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Support ignoreChanges read option in DSv2, which skip all remove file actions but keep the add file actions in the same commit. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Unit tests that test parity between v1 and v2 connector on both pure deletes commit (only remove) and change commit (add + remove). Integration tests: - streaming with ignoreChanges = true allows both delete and change commits <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
milastdbx
force-pushed
the
geospatial-expressions
branch
from
March 24, 2026 14:06
5329973 to
b6a1661
Compare
…elta-io#6335) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Write atomic-supported property to Iceberg compact tables. This is part of project to support write to UC managed UniForm tables ## How was this patch tested? Add UTs ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniversalFormatSuite" ```
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) ## Description As the title says <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Will be merged later <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
delta-io#5567) Co-authored-by: Coauthor One andreAmorimF Co-authored-by: Coauthor Two aryadneguardieiro Co-authored-by: Coauthor Three yurimantchev-sudo <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Allow writes to CDF tables for add-only, remove-only, and non-data-change transactions. More context: https://delta-users.slack.com/archives/C04TRPG3LHZ/p1763402127485709?thread_ts=1763400653.848629&cid=C04TRPG3LHZ <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Co-authored-by: André Fonseca <1077309+andreAmorimF@users.noreply.github.com> Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com>
…ipenv in spark_test workflow (delta-io#6390) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (CI/GitHub Actions) ## Description This PR hardens the GitHub Actions CI pipelines against supply chain attacks targeting the Python/PyPI package ecosystem. It reduces the attack surface by eliminating unnecessary `pip install` invocations from workflows where they are not required for the tests being run. ### Changes **`spark_test.yaml` — Remove redundant Python environment setup** The `spark_test` workflow runs Scala/Java tests via `build/sbt sparkGroup/test`, but previously set up a full Python environment (pyenv, pipenv, and 20+ `pipenv run pip install` commands) that was not consumed by any test step. This PR removes the `Install Job dependencies` step entirely and switches the test invocation from `pipenv run python run-tests.py` to `python3 run-tests.py`, which requires only the system Python (stdlib-only — `run-tests.py` has no third-party imports). This change is consistent with delta-io#6384, which independently identified the same redundancy. **`iceberg_test.yaml`, `spark_python_test.yaml`, `publish_docs.yaml` — Disable workflows with heavy PyPI dependency chains** These three workflows install packages from PyPI at runtime via `pip`/`pipenv`/`conda` and cannot function without those installs. Rather than deleting them, they are renamed to `disabled_*.yaml` to preserve them for future re-enablement once the dependency installation strategy is hardened (e.g., via hash-pinned lockfiles or reproducible resolution with `UV_EXCLUDE_NEWER`). | Workflow | Reason for disabling | |---|---| | `disabled_spark_python_test.yaml` | Installs 20+ PyPI packages via pyenv/pipenv including pyspark, pandas, pyarrow, grpcio | | `disabled_iceberg_test.yaml` | Installs pyenv/pipenv and runs tests through `pipenv run python` | | `disabled_publish_docs.yaml` | Sets up a full conda environment (`docs/environment.yml`) with 50+ packages for Sphinx doc generation | **Workflows left unchanged (no PyPI exposure)** The following workflows use only system Python (stdlib-only scripts) or no Python at all, and are unaffected by this change: - `build.yaml` — stdlib Python only (`test_cross_spark_publish.py`), no pip - `kernel_test.yaml` — stdlib Python test runner (`run-tests.py`), no pip - `spark_examples_test.yaml` — stdlib Python for version info, tests are pure SBT - `flink_test.yaml` — pure SBT - `kernel_unitycatalog_test.yaml` — pure SBT - `kernel_docs.yaml` — pure SBT - `unidoc.yaml` — pure SBT ### Additional context All 52 GitHub Actions `uses:` references across all workflows are already pinned to full 40-character commit SHAs with version comments. An audit confirmed that 14 of 15 unique pinned SHAs are older than 30 days; the single exception (`pnpm/action-setup@v5.0.0`, 12 days old) is in the now-disabled `publish_docs.yaml`. ## How was this patch tested? 1. **CI run on fork**: The changes were pushed to a sandboxed public fork ([seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta)) and all enabled workflows executed on GitHub Actions. The `spark_test`, `build`, `spark_examples_test`, `flink_test`, `kernel_unitycatalog_test`, `kernel_docs`, and `unidoc` workflows ran without issues related to the CI changes. 2. **SHA pin audit**: All `uses:` references were verified to be pinned to full commit SHAs with correct version comments using a custom scanner. ## Does this PR introduce _any_ user-facing changes? No. This PR only modifies CI workflow configuration. No source code, build definitions, APIs, or runtime behavior are changed. Three CI jobs (`spark_python_test`, `iceberg_test`, `publish_docs`) are temporarily disabled and can be re-enabled once a hardened dependency installation strategy is in place.
#### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description In this PR we are adding geospatial table feature as per spec: delta-io#4726 ## How was this patch tested? Tests ## Does this PR introduce _any_ user-facing changes? Adds new table feature
## Description `commitFilesIterator` discovers commit files in two phases: Phase 1 lists backfilled files from the filesystem, Phase 2 queries the coordinator for unbackfilled commits. If all unbackfilled commits get concurrently backfilled between the two phases (e.g., a concurrent writer triggers `backfillToVersion` which copies staged files to `_delta_log/` and removes them from the coordinator via `registerBackfill`), Phase 2 finds no `UnbackfilledDeltaFile` entries and the existing gap check has no anchor (`headOption` returns `None`), returning `Iterator.empty`. Versions between `maxVersionSeen + 1` and `endSnapshot.version` are silently dropped. This race manifests when a streaming reader and a concurrent writer run on different processes (different DeltaLog instances). The reader's Phase 1 lists the filesystem and finds nothing new. The writer commits and triggers backfill. The reader's Phase 2 does a fresh `deltaLog.update()`, which classifies the now-backfilled files as `BackfilledDeltaFile` and gets empty `getCommits()`. The gap check fails and data is lost. Related issue: delta-io#6339 ## How was this patch tested? Added `DeltaSourceWithCoordinatedCommitsBatch2Suite` and `DeltaSourceWithCoordinatedCommitsBatch3Suite` to exercise batch-size-2 and batch-size-3 backfill boundaries. ## Does this PR introduce _any_ user-facing changes? No. This is a bug fix for an internal race condition in commit file discovery.
## Summary Replace the static bearer token in `IcebergRESTCatalogPlanningClient` with a per-request `tokenSupplier` backed by UC's `TokenProvider`, enabling OAuth client credentials flow. ## Auth config **Static token (existing, unchanged):** ``` spark.sql.catalog.<name>.token = "dapi..." ``` **OAuth client credentials (new):** ``` spark.sql.catalog.<name>.auth.type = oauth spark.sql.catalog.<name>.auth.oauth.uri = https://<idp>/token spark.sql.catalog.<name>.auth.oauth.clientId = <id> spark.sql.catalog.<name>.auth.oauth.clientSecret = <secret> ``` Follows UC's `AuthConfigUtils` pattern. Error if both `.token` and `.auth.token` are set. ## Key changes - `ServerSidePlanningMetadata`: `authToken` -> `tokenSupplier` (per-request function) - `UnityCatalogMetadata`: Extract `auth.*` configs, create `TokenProvider`. Legacy `.token` fallback with conflict check. - `IcebergRESTCatalogPlanningClient`: `token: String` -> `tokenSupplier: () => String`. `HttpRequestInterceptor` for per-request Bearer token injection. ## Test plan - [ ] `extractAuthConfig` tests: OAuth, static, legacy, conflict error, legacy wins, empty - [ ] Existing SSP tests updated for new constructor signatures - [ ] CI on delta-io/delta This pull request was AI-assisted by Isaac.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Introduce an `ANALYSIS` phase for the fuzz tester. ## How was this patch tested? Fuzz tester ## Does this PR introduce _any_ user-facing changes? No
…rwrite partitions that contain NULL in the table (delta-io#6375) Co-authored-by: Isaac <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description When users specify this option with DPO Mode, DPO will not overwrite partitions that contain NULL in the table. ## How was this patch tested? Added UTs. ## Does this PR introduce _any_ user-facing changes? Adding a new `DeltaOption` `useNullIntolerantEqualityWithDPO`, to ignore overwriting partitions that contain NULL in Dynamic Partition Overwrite mode.
… is from Legacy (delta-io#6392) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (Sharing) ## Description When DeltaFormatSharingSource restarts from a legacy DeltaSharingSourceOffset checkpoint that is mid-version (index != -1), the stream must finish the current version before transitioning to the new DeltaSourceOffset format. - Lucky case (legacy offset at a version boundary, index == -1): transition immediately to DeltaSourceOffset, fetch file IDs with SHA256, and continue with normal multi-version batches. - Unlucky case (legacy offset mid-version): restrict the fetch to a single version using MD5 file IDs (matching the legacy hash), and convert the end offset back to legacy format until the stream reaches a version boundary. - Propagate fileIdHash through getTableFileChanges, DeltaSharingUtils.getRefresherForGetFiles, getRefresherForGetFilesWithStartingVersion, and getRefresherForGetCDFFiles so the server returns file IDs with the correct hash algorithm. ## How was this patch tested? New test cases ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…trings (delta-io#6301) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This is **PR 1 of 3** for geospatial data skipping support. It teaches the stats system (JSON parsing, schema validation, DataFileStatistics) to handle geometry/geography columns as WKT string values, and adds GeometryUtils for WKT POINT parsing/formatting. Delta spec: delta-io#4726 ### Changes - **StatsSchemaHelper**: Mark `GeometryType`/`GeographyType` as eligible for min/max stats - **JsonUtils**: Handle geometry/geography as WKT string in `parseJsonValueToLiteral()` - **SchemaUtils**: Add geometry/geography to supported types in `validateSupportedType()` - **DataFileStatistics**: Validate geometry/geography stats accept string literals - **DefaultJsonRow**: Decode geometry/geography JSON values as strings - **GeometryUtils** (new): WKT POINT parsing (`parsePointXY`) and formatting (`formatPointWKT`) ### PR dependency graph ``` master | +-- PR A (this): Stats Parsing + GeometryUtils | | | +-- PR B: Expressions + Predicate Translation | | | +-- PR C: Parquet / Data Layer ``` ## How was this patch tested? - `GeometryUtilsSuite`: 15 tests covering WKT parsing/formatting, round-trips, error cases - `DefaultJsonHandlerSuite`: pending test placeholder for geometry/geography parsing ## Does this PR introduce _any_ user-facing changes? Adds new table feature --------- Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
…elta-io#6355) ## Summary - Add support for converting Spark's `AlwaysTrue` and `AlwaysFalse` filters to Kernel `ALWAYS_TRUE` and `ALWAYS_FALSE` predicates in the V2 connector's `ExpressionUtils`. - `AlwaysFalse` pushdown allows the Kernel to skip all files entirely, while `AlwaysTrue` is a correct no-op passthrough. - These were the only two Spark filter types with direct Kernel equivalents that were missing from the V2 connector's filter conversion. ## Test plan - [x] Added `testAlwaysTrueFilter` and `testAlwaysFalseFilter` unit tests in `ExpressionUtilsTest` - [x] All 74 tests pass in `ExpressionUtilsTest` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…lta-io#6155) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6155/files) to review all changes. **Stack:** - **[[UC Commit Metrics] Add skeleton transport wiring and smoke tests](delta-io#6155 [[Files changed](https://github.com/delta-io/delta/pull/6155/files)] ⬅️ _This PR_ - [[UC Commit Metrics] Add full payload construction and schema tests](delta-io#6156) [[Files changed](https://github.com/delta-io/delta/pull/6156/files/4b5eadb28ab152d9ac180423bfd51720b6219b5b..1f39302986a7b6138cd358f9515386fc283810a8)] - [[UC Commit Metrics] Add feature flag and async dispatch](delta-io#6333) [[Files changed](https://github.com/delta-io/delta/pull/6333/files/1f39302986a7b6138cd358f9515386fc283810a8..7b6d633980419060404203551bb1577ccfe922aa)] --- ## Summary - Self-contained post-commit hook in `hooks/metrics/UpdateMetricsHook.scala` (case class + companion object with payload builder, HTTP client, and constants). - Registered only for UC-managed tables (`CatalogTableUtils.isUnityCatalogManagedTable` check at registration time). - Derives endpoint from `spark.sql.catalog.<name>.uri`, authenticates via `UCCommitCoordinatorBuilder` + `TokenProvider`. - Best-effort: failures logged as warnings, never fail the commit. ## Test plan - [x] `build/sbt "spark/testOnly *metrics.UpdateMetricsHookSuite"` -- 6 tests pass --------- Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com> Signed-off-by: foo.bar <foo.bar@foobar.com> Co-authored-by: foo.bar <foo.bar@foobar.com>
…ta-io#6377) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6377/files) to review incremental changes. - [**stack/ddl-1-foundations**](delta-io#6377) [[Files changed](https://github.com/delta-io/delta/pull/6377/files)] - [stack/ddl-2-builder](delta-io#6378) [[Files changed](https://github.com/delta-io/delta/pull/6378/files/b8c478ac19da5afb5aea0d8ce25dfa744b689e4f..a652586290caadbe79201d594ecf0e95f129b328)] - [stack/ddl-2b-builder](delta-io#6442) [[Files changed](https://github.com/delta-io/delta/pull/6442/files/a652586290caadbe79201d594ecf0e95f129b328..88c17741e4c8f17a56c4a0523c1b744c36789c95)] - [stack/ddl-3-wiring](delta-io#6379) [[Files changed](https://github.com/delta-io/delta/pull/6379/files/88c17741e4c8f17a56c4a0523c1b744c36789c95..113ebc006446bd7645be6a75c900104b0b0ed8a4)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? ## Description In this stack of PRs we are implementing the foundation for DDL operations (metadata-only CREATE, CTAS, RTAS). For context, CREATE TABLE in this DSv2 world is 3 main steps: 1. pre-register the table to the catalog (UCProxy.createTable or HMS.createTable) 2. build a transaction for table creation (Delta commit 0) 3. commit (actually commit the transaction above) **We also want to make implementing future DDL operations easier in the future**. That's why in this PR we create a unified DDL request POJO. This POJO/request will be fed into the transaction builder. **Significance**: CreateTableBuilder.prepare() does all the Spark → Kernel conversion - DDLContext is a normalized request for downstream Kernel operations. > By the time DDLContext is constructed, everything is Kernel-native. buildTransaction() just passes it straight through to TableManager or UCCatalogManagedClient with no further conversion. ``` SQL: CREATE TABLE t (id INT, name STRING) │ ▼ ┌──────────────────────────────────────────┐ │ UCSingleCatalog.createTable( │ │ ident, schema, partitions, props) │ │ │ │ Spark's UC catalog plugin. Delegates │ │ to DeltaCatalog for Delta tables. │ └─────────────────────┬────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────────────────────────────┐ │ DeltaCatalog.createTable(ident, schema, partitions, properties) │ │ │ │ 1. DeltaV2Mode routing check │ │ STRICT → always Kernel path │ │ AUTO → Kernel only if UC + catalogManaged feature flag │ │ NONE → V1 path (super.createTable) │ │ │ │ 2. If UC: preRegisterIfNeeded(ident, schema, parts, props) │ │ └→ Optional<CatalogTable> │ │ then injectStorageCredentials(ct, hadoopConf) │ │ then UCUtils.extractTableInfo(ct, spark) → Optional<UCTableInfo> │ │ │ │ 3. CreateTableBuilder.prepare(ident, schema, parts, props, hadoopConf, ucTableInfo) │ │ └→ DDLContext │ │ │ │ 4. CreateTableBuilder.buildTransaction(ddlContext) → Transaction │ │ │ │ 5. txn.commit(engine, emptyIterable()) — metadata-only │ │ │ │ 6. return new SparkTable(ident, tablePath) — V2 connector │ └──────┬─────────────────────┬──────────────────────────┬──────────────────────┬───────────┘ │ │ │ │ ▼ ▼ ▼ ▼ STEP 2 STEP 3 STEP 4 STEP 5 Pre-register Prepare Build Txn Commit UC-only. Converts Spark Routes to Kernel txn.commit( Calls delegate types to Kernel: API based on engine, .createTable() ucTableInfo: emptyIterable() via UCProxy. sparkSchema → ) kernelSchema ┌── UC present? ───┐ UC assigns: (SchemaUtils) │ │ Kernel handles • tableId │ YES │ conflict • managed Transform[] → │ UCCatalog │ resolution, location DataLayoutSpec │ ManagedClient │ protocol, • ucUri (identity or │ .buildCreate │ and log write. • authConfig cluster-by) │ TableTxn( │ │ tableId, │ For UC: commit Returns properties → │ path, │ goes through Optional filtered props │ schema, │ UC committer. <CatalogTable> (strip DSv2 keys: │ engineInfo) │ location, provider, │ + UC committer │ For path-based: Non-UC: returns comment, owner, │ + UC properties │ writes directly Optional.empty() external, path, │ │ to filesystem. option.path) │ NO │ On failure: │ TableManager │ Returns drops pre- hadoopConf → │ .buildCreate │ TransactionCommit registered Engine │ TableTxn( │ Result with: catalog entry (DefaultEngine) │ path, │ • version (0) │ schema, │ • postCommit path resolved: │ engineInfo) │ Snapshot UC → ucTableInfo └───────┬──────────┘ .tablePath non-UC → props Then applies: ["location"] • .withTableProperties(filtered) (throws if • .withDataLayoutSpec(layout) none found) • .build(engine) → Transaction Returns: DDLContext POJO (ident, tablePath, kernelSchema, filteredProps, comment, dataLayoutSpec, engine, ucTableInfo, engineInfo) ``` <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…6438) ## Description Add `DeltaV2TableManager` trait to `v2/interop/` that abstracts `DeltaLog`'s role as the entry point for Delta table operations. `DeltaLog` extends this trait. This is the foundation for migrating Delta protocol reading from `DeltaLog` to Kernel. The trait defines the minimal surface that commands and `OptimisticTransaction` need from the table manager: - Table identity: `logPath`, `dataPath`, `clock`, `options`, `tableId` - Configuration: `newDeltaHadoopConf()` - Snapshot access: `snapshot`, `unsafeVolatileSnapshot`, `tableExists` - Setup: `createLogDirectoriesIfNotExists()` With this trait in place, a future `KernelTableManager` can implement the same interface via Kernel instead of log replay, and all existing business logic (commands, `OptimisticTransaction`) works with either backend. **This PR has zero behavior change** — `DeltaLog` already implements every method defined in the trait, so `with DeltaV2TableManager` is a no-op. ## How was this patch tested? All existing `DeltaLog` tests exercise the new trait. No test changes needed. ## Does this PR introduce _any_ user-facing changes? No.
… test (delta-io#6348) ## Description Fixes a flaky test in `DeltaRetentionWithCatalogOwnedBatch1Suite` that fails intermittently with: ``` 20 did not equal 0 Delta files before the last checkpoint version should have been deleted (DeltaRetentionSuite.scala:152) ``` ### Root Cause The test `"log files being already deleted shouldn't fail log deletion job"` fails **only when executed between 00:00-01:00 UTC**. The mechanism: 1. `getStartTimeForRetentionTest()` returns today at 01:00 UTC — which is a **future** timestamp if the test runs before 01:00 UTC. 2. Delta files get `setLastModified` to this future time (01:00:xx UTC). 3. V2 checkpoint files (introduced by CatalogOwned in delta-io#5841) use UUID-based names (`*.checkpoint.<uuid>.json`) that don't match `checkpointFileSingular` format (`*.checkpoint.parquet`), so the test never sets their timestamps — they keep their real wallclock time (~00:04 UTC). 4. `BufferingLogDeletionIterator` checks `needsTimeAdjustment()` **before** `isCheckpointFile()`. When a V2 checkpoint (modTime=00:04) follows a delta file (modTime=01:01), the checkpoint's earlier real timestamp triggers time adjustment, causing it to be treated as a regular file instead of a checkpoint boundary. The buffer is never flushed, and zero files get deleted. ### Fix Replace the `checkpointFileSingular` lookup (lines 133-136) with the `getCheckpointFiles` + `checkpointVersion` pattern already used and working correctly elsewhere in the same file (lines 724-734). This finds all checkpoint file formats including V2. ### Verified Reproduced the failure by forcing `getStartTimeForRetentionTest` to return a future timestamp: - **Without fix**: `FAILED` — `20 did not equal 0` (identical to CI failures) - **With fix**: `PASSED` Confirmed all CI failures occurred in the 00:00-01:00 UTC window: 00:04:33, 00:13:10, 00:21:48, 00:04:47 UTC. ## How was this patch tested? Existing test `DeltaRetentionWithCatalogOwnedBatch1Suite."log files being already deleted shouldn't fail log deletion job"` — this PR fixes it. ## Does this PR introduce _any_ user-facing changes? No. Test-only change. This pull request was AI-assisted by Isaac.
…-io#6443) Extract the file name construction logic from DelayedCommitProtocol.getFileName into a shared companion object method DelayedCommitProtocol.buildFileName. This enables reuse of the standard Delta file naming convention (part-NNNNN-uuid.ext / cdc-NNNNN-uuid.ext) by both the V1 write path (DelayedCommitProtocol) and the DSv2 write path (DataWriter), ensuring consistent file naming across both code paths. Changes: - Move file name building logic to DelayedCommitProtocol.buildFileName companion method - Simplify getFileName to delegate to the new shared method - Preserve all existing behavior (test prefix, CDC prefix, split formatting)
… translation Add the public StIntersectsBoxes expression and its internal translation to ST_INTERSECTS_BOXES_ON_STATS for geospatial data skipping: - StIntersectsBoxes: public predicate taking column + WKT query bounds - DataSkippingUtils: translate to stats-based predicate with min/max columns - ExpressionVisitor: dispatch ST_INTERSECTS_BOXES_ON_STATS - DefaultExpressionEvaluator: transform (type validation) and eval (AABB intersection via WKT parsing) - GeometryDataSkippingSuite: end-to-end tests with 4-quadrant layout, null stats, and combined AND predicates Co-authored-by: Isaac
milastdbx
force-pushed
the
geospatial-expressions
branch
from
March 31, 2026 10:18
59bdcaa to
04dd064
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
In this PR we are adding geospatial table feature as per spec:
This is PR 2 of 3 for geospatial data skipping support. It adds the
StGeometryBoxesIntersectpublic expression and its internal translation toST_GEOMETRY_BOEXES_INTERSECT_ON_STATSfor data skipping, including expression evaluation and end-to-end data skipping tests.Depends on: PR A - Geospatial stats parsing (will rebase to master after PR A merges)
This is following delta protocol delta-io#4726
Changes
ST_INTERSECTS_BOXEStoST_INTERSECTS_BOXES_ON_STATSwith min/max stats columnsvisitStIntersectsBoxesOnStatsdispatchPR dependency graph
How was this patch tested?
GeometryDataSkippingSuite: 5 end-to-end tests covering 4-quadrant layout, null stats handling (conservative inclusion), and combined AND predicates with geometry + scalar columns. Tests bypass Parquet writer by injectingDataFileStatisticswith WKT string literals directly.Does this PR introduce any user-facing changes?
New expression