Repository navigation
[kernel] Add delta.parquet.compression.codec table property - #11
Closed
scottsand-db wants to merge 1527 commits into
Closed
scottsand-db wants to merge 1527 commits into
scottsand-db wants to merge 1527 commits into
Conversation
…olumn pruning (delta-io#5894) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Update SparkScan estimated size in bytes to take account in column pruning. This behavior change follows the interface comment in [SupportsReportStatistics](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsReportStatistics.java#L26-L28) where this stats API is being implemented, since now that DSv2 only returns the columns after pruning if column filter is pushed down. For the estimation calculation, it follows the exact same behavior as in `SizeInBytesOnlyStatsPlanVisitor.visitUnaryNode` from Spark. - I looked into delta features such as `materializePartitionColumns` that allows writing partition columns to the data files, which may in theory impact estimated size; however, this feature allows arbitrary turning on and off, which can result in a table containing some files with materialized partition columns within the data files and others without. Given that the existing estimation code based on schema size is arguably already not accurate, I decided to keep the logic as is today for complete feature parity, but this potentially could be a future improvement area. ## How was this patch tested? Unit test. Also tested this against local Spark code with corresponding changes and loggings, and verified that in `CostBasedJoinReorder`, size estimate is the exact same as in before: ``` v1 61329:[JoinReorderWithGroups] Input: Project 61330:[JoinReorderWithGroups] sizeInBytes=98028 61331:[JoinReorderWithGroups] output.size=1 61332:[JoinReorderWithGroups] canBroadcastBySize=true 61333:[JoinReorderWithGroups] Leaf: LogicalRelation, sizeInBytes=2418043, output.size=28 61334:[JoinReorderWithGroups] Input: Project 61335:[JoinReorderWithGroups] sizeInBytes=949 61336:[JoinReorderWithGroups] output.size=2 61337:[JoinReorderWithGroups] canBroadcastBySize=true 61338:[JoinReorderWithGroups] Leaf: LogicalRelation, sizeInBytes=11636, output.size=26 --- v2 29166:[JoinReorderWithGroups] Input: Project 29167:[JoinReorderWithGroups] sizeInBytes=98028 29168:[JoinReorderWithGroups] output.size=1 29169:[JoinReorderWithGroups] canBroadcastBySize=true 29170:[JoinReorderWithGroups] Leaf: DataSourceV2ScanRelation, sizeInBytes=130705, output.size=2 29171:[JoinReorderWithGroups] Input: Filter 29172:[JoinReorderWithGroups] sizeInBytes=949 29173:[JoinReorderWithGroups] output.size=2 29174:[JoinReorderWithGroups] canBroadcastBySize=true 29175:[JoinReorderWithGroups] Leaf: DataSourceV2ScanRelation, sizeInBytes=949, output.size=2 ``` ## Does this PR introduce _any_ user-facing changes? no --------- Co-authored-by: Gengliang Wang <gengliang@apache.org>
…arkTable equality check (delta-io#5903) - Also improved thread safety with filter() method for updating state mutations #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description - This is a follow up of delta-io#5827 , to add the missing snapshot info to the equality check of `SparkTable`. - Separately, it was noticed that the current `filter()` method in `SparkScan` mutates states, but such mutation was not thread safe and not reflected in equality checks. - for equality check, only the effective filter is tracked. This is aligned with the Iceberg approach in [SparkBatchQueryScan](https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L128-L162). - However, a difference is that Iceberg uses a list (i.e. order dependent) for the incoming filters for equality check [here](https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L161-L162), where we are using a set in this change. From the original PR apache/iceberg#3400 it seems that it was not discussed, so I doubt if it is intentional. ## How was this patch tested? Additional unit test ## Does this PR introduce _any_ user-facing changes? No
…directory (delta-io#5880) #### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Follow-up of delta-io#5325 Remove obsolete .gitignore file from icebergShaded directory. It was created when icebergShaded was pulling Iceberg source code and appling patches to create a custom JAR. Now it relies on Maven JAR ## How was this patch tested? N/A ## Does this PR introduce _any_ user-facing changes? No Signed-off-by: Felipe Fujiy Pessoto <fepessot@microsoft.com>
Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) ## Description Fixes delta-io#5392 ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [x] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR exposes `delta-spark` version in python as `delta.__version__`. The user will be able to get it using `from delta import __version__`. The variable will contain the package version in the `x.y.z` format. This is a common (although optional) pattern in python packages and is provided in numpy, pandas, pyspark, and many other packages. Delta version is stored in `version.sbt`. This is a single source of truth for getting the package version and is used by the `build.sbt` file to add the version to the Scala package and by setup.py when building a delta-spark whl. This PR adds a new `generatePythonVersion` sbt task. This task will read the version from `version.sbt` file and will autogenerate the `python/delta/version.py` file. The autogeneration code can be run manually `build/sbt sparkV1/generatePythonVersion` and is run automatically on each `build/sbt sparkV1/compile` or `build/sbt compile` run. PR also contains premerge tests that compare if the version in `version.sbt` and `version.py` match. This means that if the `version.sbt` version is changed and `version.py` isn't (although the latter should be updated automatically during sbt compile), CI will block the PR and tell the developer to run `build/sbt sparkV1/generatePythonVersion`. Additionally, this PR fixes some failing linter checks: ``` # false positive when checking f-strings, fixed by ignoring E231 /home/alex.khakhlyuk/oss/delta/python/delta/tests/test_pip_utils.py:64:66: E231 missing whitespace after ':' /home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:32: E231 missing whitespace after ':' /home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:48: E231 missing whitespace after ':' /home/alex.khakhlyuk/oss/delta/python/run-tests.py:45:41: E231 missing whitespace after ',' /home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:22: E231 missing whitespace after ':' /home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:63: E231 missing whitespace after ':' /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E231 missing whitespace after ';' /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E231 missing whitespace after ';' /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E231 missing whitespace after ';' # false positive detecting backticks ` in sql queries, fixed by ignoring W604 /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:360:49: W604 backticks are deprecated, use 'repr()' /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:371:49: W604 backticks are deprecated, use 'repr()' /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:382:49: W604 backticks are deprecated, use 'repr()' # removed ; at the end of the SQL queries /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E702 multiple statements on one line (semicolon) /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E702 multiple statements on one line (semicolon) /home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E702 multiple statements on one line (semicolon) ``` <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? `test_version.py` verifies if `delta.__version__` is valid and if it's up to date with `version.sbt`. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> Python users will be able to read delta version via `from delta import __version__`.
…e reconstruction (delta-io#5916) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR adds support for preserving variant stats from JSON AddFile records during checkpointing. Since stats are stored in Z85 encoding, they need to be decoded after the `from_json` expression. This PR implements the `DecodeNestedZ85EncodedVariant` expression for decoding variant stats from the Z85 representation. This PR also uses the `DecodeNestedZ85EncodedVariant` expression when writing variant stats as struct in `stats_parsed` in checkpoint files. For checkpointing, this PR ensures that variant stats are not shredded. This is because variant reconstruction from shredded files can lose information about integer width. This PR also introduces the `EncodeNestedVariantAsZ85String` expression which is used during state reconstruction to read variant stats stored in `stats_parsed` in Checkpoint files as JSON. This expression is applied to `stats_parsed` before `to_json` to replace all variants with their Z85 representation before encoding as JSON. This PR also prohibits Variant from being a permitted for clustering. These changes are compatible with both - Spark 4.0 and Spark 4.1. ## How was this patch tested? Using golden files. We copied golden tables into this project where the AddFiles contain variant stats and tested that checkpointing on this table preserves variant stats. ## Does this PR introduce _any_ user-facing changes? Yes --------- Co-authored-by: Dhruv Arya <dhruv.arya@databricks.com>
… cleanup (delta-io#5917) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Add basic support for `checkpointProtection` table feature in Kernel by limiting log cleanup after checkpointing. Only do log cleanup when 1. Snapshot was built as "latest" by intent (not time-traveled) 2. checkpointProtection feature is not enabled 3. delta.enableExpiredLogCleanup table property is set to true <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Added tests <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
## Stacked PR Stacked on delta-io#5917 <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> In delta-io#5917, we added basic support for `checkpointProtection`. This PR adds it to the allowlist in IcebergWriterCompat. `checkpointProtection` is a Delta-internal feature that prevents some writes. It doesn't affect data formats or schemas, so is invisible to Iceberg readers — safe to be allowed. ## How was this patch tested? Existing <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…o#5918) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description As titled. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UTs. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…ta-io#5964) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR updates DataSkippingReader to exclude filters that contain user-defined functions (UDFs) from data skipping, since such UDFs can be non-deterministic even when they aren’t explicitly marked as such by the user. ## How was this patch tested? UTs ## Does this PR introduce _any_ user-facing changes? Yes, UDFs will not used as a data skipping predicate for partition filters anymore since they could be non-deterministic.
…hanging commit coordinator (delta-io#5967) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description As titled. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UTs. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? Yes, now CREATE OR REPLACE is not blocked even when explicitly specifying CC props. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
#### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR fixes an issue where we accidentally print Java objects because we're printing `Option[Array[T]]`, where Array is a Java array. Call `.toSeq` to convert to Scala and prevent Java array object references (e.g., `"[J@1a2b3c4d"`) in event data. ## How was this patch tested? Existing tests pass. Manual testing since I encountered this while debugging an issue. ## Does this PR introduce _any_ user-facing changes? No.
…aged tables (delta-io#5968) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Clarify the maintenance operation requirements in the Catalog-Managed Tables RFC. ## How was this patch tested? n/a ## Does this PR introduce _any_ user-facing changes? Doc change only.
…ure (delta-io#5708) ## Description This feature is in RFC stage and will be merged into the protocol later. This also clarifies some of the wording around enablement/support of this table feature in the RFC, bringing it back from delta-io#5619. See delta-io#5555 ## How was this patch tested? N/a ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Add a `IcebergSyncConverterHook` to `IcebergConverterHooks`, which only supports synchronization Iceberg metadata conversion ## How was this patch tested? Existing tests
…a-io#5935) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Follow up for delta-io#5726 - Change the default value of `spark.databricks.delta.v2.enableMode` from `NONE` to `AUTO` - Remove explicit mode configuration from UC streaming tests (now uses default) - Clean up unused V2_ENABLE_MODE constants from test base class ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> - UCDeltaStreamingTest - CI/CD passing with AUTO mode ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>
…lta queries (delta-io#5964)" (delta-io#5982) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This reverts commit 9997fd8 because it has a correctness bug around metadata-only queries i.e. queries referencing UDFs which don't reference any columns. ## How was this patch tested? Build ## Does this PR introduce _any_ user-facing changes? No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Update the version reference for publishing API docs. Co-authored-by: Scott Haines <newfrontcreative@gmail.com>
## Summary Implement proper handling of column names containing dots (literal column names vs. nested fields) for server-side planning. This PR adds backtick escaping for literal dotted columns in **projections** while using raw field names for **filters**. ### Why do we need to escape projections but not filters? Spark's Filter API attribute is just a String. By the time we receive it, Spark has [already resolved](https://github.com/apache/spark/blob/dd2c62f790a821440baa75792a98c6ed5a37e519/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala#L44) what column it refers to. But when Spark gives us projections, they come as a hierarchical StructType which has to be flattened. ### Challenge with flattening Naive flattening can cause ambiguity - what if there is a struct `xx` with a field `yy` versus a top-level column called `xx.yy`? We solve this by: 1. Using Spark's `SchemaUtils.explodeNestedFieldNames()` which properly escapes dotted field names 2. Filtering out parent struct fields to keep only leaf fields ## What Changed ### Projection Column Name Handling - **Use Spark's built-in escaping**: Leverage `SchemaUtils.explodeNestedFieldNames()` to flatten and escape field names with proper backtick handling - **Filter parent structs**: Remove intermediate parent struct fields from projections, keeping only leaf fields that contain actual data - For schema `STRUCT<a: STRUCT<b.c: STRING>>`, return `["a.\`b.c\`"]` not `["a", "a.\`b.c\`"]` - This ensures the server only reads specific nested fields, not entire parent structs ### Test Coverage - **End-to-end escaping test**: Validates proper projection handling for dotted column names in `ServerSidePlannedTableSuite` - **Multiple scenarios**: Tests nested structs with dotted fields, top-level dotted columns, and combinations - **Iceberg integration tests**: Verifies escaping works correctly in `IcebergRESTCatalogPlanningClientSuite` --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Replace technote-space/get-diff-action with native GitHub Actions paths-ignore to prevent workflows from running on .md and .txt file changes. This saves CI resources by not allocating runners for documentation-only PRs. Also removes references to non-existent unity/** directory in kernel_unitycatalog_test.yaml (Unity Catalog code is at kernel/unitycatalog/, already covered by kernel/**). #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (fill in here) ## Description Documentation-only changes (.md, .txt files) currently trigger all CI workflows, wasting compute resources on unnecessary test runs. Using native paths-ignore prevents workflows from starting entirely, saving CI time and removing the dependency on third-party actions. ## How was this patch tested? YAML syntax was validated locally. The workflows will be fully tested when this PR is pushed—GitHub Actions will verify the workflow files parse correctly and the path filtering behaves as expected on subsequent commits. ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sebastian Baunsgaard <sebastian.baunsgaard@databrics.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>
…ctory (delta-io#5907) ## Summary Implement reflection-based auto-registration for `ServerSidePlanningClientFactory` to automatically discover and load the `IcebergRESTCatalogPlanningClientFactory` when the delta-iceberg JAR is on the classpath. This eliminates the need for manual factory registration in production code while maintaining backward compatibility for testing scenarios. ## What Changed ### Core Auto-Registration Logic - **Lazy initialization with reflection**: Added `tryAutoRegisterFactory()` method that uses `Class.forName()` to load `IcebergRESTCatalogPlanningClientFactory` on first access - **Thread-safe initialization**: Implements double-checked locking pattern to ensure one-time, thread-safe factory registration - **Smart triggering**: Auto-registration only attempts when `getFactory()` is called and no factory has been manually set - **Clear error handling**: Throws descriptive `IllegalStateException` when delta-iceberg JAR is missing, guiding users to add it to classpath ## Testing The changes maintain existing test compatibility while enabling automatic registration in production: - Test code can still use `setFactory()` to inject mock implementations - `clearFactory()` now properly resets auto-registration state for test isolation - JarSuite verifies serverSidePlanning classes are packaged correctly ## How It Works ```mermaid flowchart TD Start[getFactory called] --> Check1{Factory already set?} Check1 -->|Yes| Return[Return factory] Check1 -->|No| TryAuto[tryAutoRegisterFactory] TryAuto --> Check2{Already attempted?} Check2 -->|Yes| CheckReg{Factory registered?} Check2 -->|No| Lock[synchronized block] Lock --> DoubleCheck{Still not attempted?} DoubleCheck -->|No| CheckReg DoubleCheck -->|Yes| SetFlag[Set autoRegistrationAttempted = true] SetFlag --> Reflect[Class.forName IcebergRESTCatalogPlanningClientFactory] Reflect --> Success{Loaded?} Success -->|Yes| Register[registeredFactory = Some factory] Success -->|No| Error[Throw IllegalStateException] Register --> Return CheckReg -->|Yes| Return CheckReg -->|No| Error ``` --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…nWiderType evolution mode (delta-io#5969) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Changes the TypeEvolution mode `AllTypeWideningToCommonWiderType` to infer a wider decimal type between integers and decimals. For example, when calling `getWidenedType(IntegerType, DecimalType(2, 1))`, the result will be `DecimalType(11,1)` because `IntegerType` is narrower than `DecimalType(10, 0)` which has a common wider type with `DecimalType(2, 1)`. ## How was this patch tested? Add to existing test in `SchemaUtilsSuite` <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…5774) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5774/files) to review incremental changes. - [**stack/dv_pr2_phase1_basic_read**](delta-io#5774) [[Files changed](https://github.com/delta-io/delta/pull/5774/files)] - [stack/dv_pr3_phase2_vectorized](delta-io#5775) [[Files changed](https://github.com/delta-io/delta/pull/5775/files/6b90bdd14bd86a5bcfb109fb12d9447b769d4d24..9446011c512f4fa4d7f06417eb4629671813dbe4)] - [stack/dv_pr4_phase3_file_splitting](delta-io#5776) [[Files changed](https://github.com/delta-io/delta/pull/5776/files/9446011c512f4fa4d7f06417eb4629671813dbe4..5e798ef3a37c07b86f12de6f5465a5db3fbff153)] - [stack/dv_pr5_streaming_support](delta-io#5877) [[Files changed](https://github.com/delta-io/delta/pull/5877/files/5e798ef3a37c07b86f12de6f5465a5db3fbff153..1c4ddacd08654e2d778bfdfb992320cb6dcead9f)] - [stack/dv_pr5_test](delta-io#5975) [[Files changed](https://github.com/delta-io/delta/pull/5975/files/6b90bdd14bd86a5bcfb109fb12d9447b769d4d24..24ae4ed8d474449407582b05c0c0ab45f97a72ed)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Add basic deletion vector (DV) read support for the Spark V2 connector using row-based filtering. ### Changes: - `DvSchemaContext`: POJO to manage DV schema context (column indices, output schema) - `DeletionVectorReadFunction`: Wraps base reader to filter deleted rows and project out DV column - `PartitionUtils`: Creates DV-aware `PartitionReaderFactory` with `DeltaParquetFileFormatV2` - Add`serializeToBase64()` to Kernel's `DeletionVectorDescriptor` ### How it works: 1. Add `__delta_internal_is_row_deleted` column to read schema 2. Filter rows where DV column != 0 (deleted) 3. Project out DV column from output ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> - `DvSchemaContextTest`: Unit tests for schema manipulation - `DeletionVectorReadFunctionTest`: Unit tests for row filtering and projection - Golden table tests with DV tables pass ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
…-io#5904) ## Summary This PR fixes Unity Catalog credential detection and refactors Iceberg REST catalog endpoint construction to properly implement the Iceberg REST catalog specification. **Key Changes:** - Fix credential detection to recognize Unity Catalog's `option.fs.*` credential properties - Refactor `/v1/config` endpoint handling to follow Iceberg REST catalog spec - Move prefix discovery logic from `UnityCatalogMetadata` to `IcebergRESTCatalogPlanningClient` - Improve robustness with trailing slash handling and HTTP client reuse - More test coverage with explicit path verification - Fix module separation issues and improve code organization ## Testing Tests now explicitly verify: - Correct `/v1/config` endpoint calls with `warehouse` query parameter - Proper prefix application in `/plan` request paths - Behavior when no prefix is returned (uses baseUri directly per Iceberg REST spec) - Token handling consistency between tests and production - Trailing slash handling robustness --- ## Compatibility This is an internal refactoring with no changes to public APIs or user-facing behavior. The changes: - Improve correctness of credential detection for Unity Catalog - Fix internal endpoint construction to follow Iceberg REST spec - Maintain backward compatibility with existing Unity Catalog configurations --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…lta-io#5963) #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description - Fix flaky RowTrackingBackfillConflictsSuite Scenarios 3 and 6 by making backfill batch ordering deterministic - Add setDeterministicInsertionTimes() to explicitly set insertion times on files after table creation, ensuring partitions 0-1 are processed by batch 1 and partitions 2-3 by batch 2 - Update concurrent transactions (DELETE, UPDATE, MERGE, OPTIMIZE) to touch only partitions 1 and 2, leaving partition 3 untouched so batch 2 always has work - Also add some tracking to allow busyWait routines to fail fast if the futures that normally get checked later already failed. ## How was this patch tested? Test-only PR ## Does this PR introduce _any_ user-facing changes? No
…f literals (delta-io#5987) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Unifying the Parsing of Partition Values. The commit delta-io@7b69041 fixed incorrect conversion of timestamp values with non-UTC session timezones. Tables with data written before this change may still exist, leading to different unnormalized partition value formats which can lead to inconsitencies for READ as well. We unify the read and write path of the partition values, ensuring that we compare only the literals with the correct data type and if needed, return a normalized string format for all data types. ## How was this patch tested? Introduced the `AddFileSuite` with unit tests for parsing different partition values. ## Does this PR introduce _any_ user-facing changes? No.
…r and ucClient.commit API (delta-io#5634) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/5634/files) to review incremental changes. - [**stack/uc_api**](delta-io#5634) [[Files changed](https://github.com/delta-io/delta/pull/5634/files)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR adds support for Delta Universal Format (UniForm) metadata to be passed from Delta Kernel through the Unity Catalog commit API. 1. Extended UCClient.commit() to accept Optional<UniformMetadata> uniform parameter 2. Added UniformAdapter Utility to extract UniForm metadata from committer properties 3. Update UCCatalogManagedCommitter now extracts Iceberg metadata from CommitMetadata.committerProperties and passes it to UCClient ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> UTs ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Move `canDowngradeToSnapshotIsolation` to allow reusability. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Existing tests. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No.
#### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This is to add an official doc on delta.io for catalog-managed tables. ## How was this patch tested? Existing doc tests ## Does this PR introduce _any_ user-facing changes? no.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR provides a table property "delta.dataSkippingStringPrefixLength" that allows users to override the prefix length table-wise. This helps with data-skipping performance on columns with long prefix that exceeds the default length (32), so that regular data-skipping does not work. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> UTs ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…elta-io#6481) ## Description Extract `createDeltaParquetFileFormat` as a public static method in `PartitionUtils`, factoring it out of the inline construction inside `createDeltaParquetReaderFactory`. This makes the `DeltaParquetFileFormatV2` construction reusable by the upcoming write path without duplicating the format creation logic. **No behavioral change** — the read path calls the new extracted method instead of constructing the format inline. ### Changes - Add `createDeltaParquetFileFormat(Snapshot, String, boolean, Option<Boolean>)` public static method to `PartitionUtils` - Update `createDeltaParquetReaderFactory` to call the new method - Remove now-unused local `protocol` and `metadata` variables from `createDeltaParquetReaderFactory` ## How was this patch tested? Existing `sparkV2` tests pass (`sbt "sparkV2/test"` — 496 tests, 0 failures). The extraction is a pure refactor with no behavioral change. --------- ## Stacked PR - **[DSv2] Extract shared createDeltaParquetFileFormat in PartitionUtils** ([delta-io#6481](delta-io#6481)) ⬅️ _This PR_ - [DSv2] Add batch write implementation ([delta-io#6482](delta-io#6482)) - [DSv2] Add WriteBuilder and wire SparkTable.newWriteBuilder ([delta-io#6483](delta-io#6483))
…#6449) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6449/files) to review incremental changes. - [stack/kernel-finalize-in-commit](delta-io#6448) [[Files changed](https://github.com/delta-io/delta/pull/6448/files)] [MERGED] - [**stack/dsv2-create-with-finalize**](delta-io#6449) [[Files changed](https://github.com/delta-io/delta/pull/6449/files)] - [stack/dsv2-wiring-e2e](delta-io#6450) [[Files changed](https://github.com/delta-io/delta/pull/6450/files/8e22865f0603c3e3fe4ce502bec55c0909ecdae1..d33e42c2fd3ab9dbf25ca3dc49554aea1d16c12a)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description The major addition of this PR in the stack is the **CreateTableBuilder**, which is the Spark-To-Kernel conversion layer for DSv2 `CREATE TABLE` to build the **Kernel Transaction.** This is the middle PR in a 3-PR stack: ``` Spark DSv2 inputs CreateTableBuilder Kernel types ------------------- ------------------- ---------------- StructType (schema) ---> prepare() ---> DDLRequestContext Transform[] (layout) ├── kernelSchema Map<String,String> ├── dataLayoutSpec Identifier + UCTableInfo ├── filtered properties Configuration ├── comment (TODO delta-io#6473) ├── engine └── transactionBuilder ├─ UC path: UCCatalogManagedClient └─ local path: TableManager -------------------- AFTER PREPARE ------------------------------- DDLRequestContext ---> buildTransaction() ---> Kernel Transaction (applies properties + data layout) ``` **CreateTableBuilder** The CreateTableBuilder has two utilities: 1. `prepare()` which converts all the Spark related types to Kernel types and filters properties for Kernel. For example, Spark | Kernel -- | -- StructType | io.delta.kernel.types.StructType Transform[] (partitions/clustering) | DataLayoutSpec via toDataLayoutSpec Map<String, String> (properties) | Filtered Map<String, String> via filterProperties COMMENT property | Optional via extractComment Configuration (Hadoop) | Engine via DefaultEngine.create(hadoopConf) 2. we also have `buildTransaction()` which takes all these converted items (bundled together into DDLRequestContext) and builds the Kernel Transaction. **Key design decisions:** - Property filtering: DSv2 plumbing keys (`location`, `provider`, `owner`) are stripped because they're catalog metadata, not Delta table properties. Protocol version overrides (`minReaderVersion`, `minWriterVersion`, `ignoreProtocolDefaults`) are stripped because Kernel resolves protocol internally and rejects these via `TableConfig.validateAndNormalizeDeltaProperties`. UC table ID is stripped as an internal routing marker. - UC vs path-based routing: When `UCTableInfo` is present, wires up `UCCatalogManagedClient` + `UCCatalogManagedCommitter` so the commit writes to the staging location and finalizes (promotes) the table in UC. Otherwise uses `TableManager.buildCreateTableTransaction` for path-based tables. - Comment preservation: `COMMENT 'x'` is extracted but not yet written to the Delta log because Kernel's `CreateTableTransactionBuilder` doesn't expose `withDescription()` yet (tracked in delta-io#6473). **DDLRequestContext change** Replaced `engineInfo` (String) with `transactionBuilder` (`CreateTableTransactionBuilder`). The transaction builder is now resolved during `prepare()` — UC or path-based — so `buildTransaction()` is catalog-agnostic. This also supports future catalog clients without changing the build step. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…io#6250) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415) to review incremental changes. - [stack/ignoreDeletesV2](delta-io#6245) [[Files changed](https://github.com/delta-io/delta/pull/6245/files)] - [stack/skipChangeCommitsV2](delta-io#6246) [[Files changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)] - [stack/ignoreChangesV2](delta-io#6249) [[Files changed](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de)] - [**stack/ignoreFileDeletionV2**](delta-io#6250) [[Files changed](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Support ignoreFileDeletion read option is DSv2, which skip a all remove file actions but keep the add file actions in the same commit. And also inform user that this read option has deprecated. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Unit tests that test parity between v1 and v2 connector on both pure deletes commit (only remove) and change commit (add + remove). Integration tests: - streaming with ignoreFileDeletion = true allows both delete and change commits DV tests: - multiple deletion vectors per file - List((ignoreFileDeletion,true)) - allow to delete files after staring a streaming query when ignoreFileDeletion is true <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…delta-io#6490) ## What changes were proposed in this pull request? Extract `DataFiltersBuilder` from `DataSkippingReader` into its own standalone file. This improves code organization by separating the data filter construction logic from the data skipping reader. ## How was this patch tested? Existing tests in `DataSkippingDeltaConstructDataFiltersSuite` continue to pass with the refactored code.
…apshot (delta-io#6298) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add end-to-end tests covering all data loss scenarios during initial snapshot for Delta streaming sources, including both DSv1 and Kernel (DSv2) code paths. The new tests verify behavior when: - Commit files are missing but a checkpoint exists (should succeed) - Both checkpoint and commit files are deleted (should fail gracefully) - Checkpoint is deleted but all commit files remain (should succeed via fallback) - A stream restarts mid-initial-snapshot after log retention cleans up old versions (should fail with appropriate error) Also updates existing `failOnDataLoss` tests to assert on the correct exception types for Kernel (DSv2), which throws `KernelException/InvalidTableException` instead of `DeltaIllegalStateException`. The V2 exclusion list in DeltaV2SourceSuite is adjusted to reflect which scenarios Kernel can/cannot handle (e.g., Kernel cannot reconstruct a snapshot without a checkpoint file). <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Test-only changes <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…hot) (delta-io#6075) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6075/files) to review incremental changes. - [**stack/cdf1**](delta-io#6075) [[Files changed](https://github.com/delta-io/delta/pull/6075/files)] - [stack/cdf2](delta-io#6076) [[Files changed](https://github.com/delta-io/delta/pull/6076/files/ab609f8e2185d3ba863c00304195c24b60f7d04b..301cf056c94305a2ca5d96dc3dcdfd88d5dbc37b)] - [stack/cdf2.5](delta-io#6391) [[Files changed](https://github.com/delta-io/delta/pull/6391/files/301cf056c94305a2ca5d96dc3dcdfd88d5dbc37b..d2ce6b47593a08ed491742e024056b8f565dd33f)] - [stack/cdf3](delta-io#6336) [[Files changed](https://github.com/delta-io/delta/pull/6336/files/70544ed3a42014a51dd0d8d97e7c4f2333e6a221..fb5a4f6e478f68410739ce68e9a3fd92b2091be1)] - [stack/cdf4](delta-io#6359) [[Files changed](https://github.com/delta-io/delta/pull/6359/files/fb5a4f6e478f68410739ce68e9a3fd92b2091be1..e52ab30e7fe3923ba302e489368231b836ff4314)] - [stack/cdf5](delta-io#6362) [[Files changed](https://github.com/delta-io/delta/pull/6362/files/e52ab30e7fe3923ba302e489368231b836ff4314..45bc4f043f996ad56d6937f6f0b5d1876fe1130a)] - [stack/cdf6](delta-io#6363) [[Files changed](https://github.com/delta-io/delta/pull/6363/files/45bc4f043f996ad56d6937f6f0b5d1876fe1130a..7d8e01ec6211eba6712b4c54e08eb68f45e542b6)] - [stack/cdf-outofrange](delta-io#6388) [[Files changed](https://github.com/delta-io/delta/pull/6388/files/216a484e5f6e7b20fa26d428703ca05dcbbb6b5a..fee69f12b67cc91d1aeed01a759f64940198405e)] - [stack/cdf7](delta-io#6370) [[Files changed](https://github.com/delta-io/delta/pull/6370/files/fee69f12b67cc91d1aeed01a759f64940198405e..7f5fc84dee6dc3386a7b18db5a0d1d7533024842)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Adds initial snapshot write-time-CDC support to the DSv2 streaming read path (SparkMicroBatchStream), bringing it closer to DSv1 feature parity. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…ncher download (delta-io#6501) #### Which Delta project/connector is this regarding? - [x] Spark - [x] Standalone - [x] Flink - [x] Kernel - [x] Other (build infrastructure) ## Description Adds optional `MAVEN_PROXY_URL` environment variable support to the SBT build scripts, enabling seamless builds behind corporate proxies or in hardened CI environments without manually editing the checked-in `build/sbt-config/repositories` file. **Changes in `build/sbt`:** - When `MAVEN_PROXY_URL` is set, generates a temporary SBT repositories config that routes all Maven and Ivy resolution through the proxy URL. - When unset, falls back to the committed `build/sbt-config/repositories` file — **no behavior change** for existing users/CI. **Changes in `build/sbt-launch-lib.bash`:** - Adds `MAVEN_PROXY_URL` as a middle fallback in the SBT launcher JAR download URL chain: `DEFAULT_ARTIFACT_REPOSITORY` → `MAVEN_PROXY_URL` → hardcoded Google Mirror default. **Usage:** ```bash export MAVEN_PROXY_URL=https://your-proxy.example.com/maven2 build/sbt compile ``` **Design notes:** - The generated proxy config only includes `maven-proxy` and `maven-proxy-ivy` resolvers, so the proxy URL should point to an aggregating/virtual repository that mirrors all required upstreams (Maven Central, Typesafe, sbt plugins, etc.). - The temporary config file is written via `mktemp` and never modifies the committed `sbt-config/repositories`. - `DEFAULT_ARTIFACT_REPOSITORY` still takes precedence over `MAVEN_PROXY_URL` for the launcher JAR download, maintaining backward compatibility. ## How was this patch tested? Manually tested with `MAVEN_PROXY_URL` set to an internal aggregating mirror: - Verified `build/sbt compile` succeeds with `MAVEN_PROXY_URL` set (all resolution routed through proxy). - Verified `build/sbt compile` works unchanged when `MAVEN_PROXY_URL` is unset (falls back to `sbt-config/repositories`). - Verified SBT launcher JAR download uses the proxy URL when set. ## Does this PR introduce _any_ user-facing changes? No behavior change for existing users. This is a purely additive, opt-in feature gated behind an environment variable that was previously unused in this project.
…sharing client (delta-io#6492) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (sharing) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> allow delta sharing client to read variantShredding table feature. ## How was this patch tested? added UT. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
…6507) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (`.claude` tooling config) ## Description Remove `.claude/scheduled_tasks.lock` which was accidentally committed. This is a local IDE/tool lock file that should not be in the repository. ## How was this patch tested? File removal only — no code changes. ## Does this PR introduce _any_ user-facing changes? No.
…gacy checkpoint (delta-io#6506) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (sharing) ## Description - On streaming query restart, the Spark engine always primes the source by calling `getBatch(start, end)` with the last two committed offsets before calling `latestOffset`. When both offsets come from a legacy `DeltaSharingSourceOffset` checkpoint, the file fetch must use MD5 `fileIdHash` to match the legacy source's file ordering. - Refactored `maybeGetLatestFileChangesFromServer` to accept separate `startConvertedFromLegacy` and `endConvertedFromLegacy` parameters along with an optional `endOffset`. When `endConvertedFromLegacy` is true, files are fetched from `startingOffset.reservoirVersion` to `endOffset.reservoirVersion` using MD5. - Updated the `latestOffset` call site to pass `endOffset = None` (no end offset available yet). - Updated the `getBatch` call site to pass `Some(endOffset)` with `endConvertedFromLegacy` when the end offset was converted from legacy. - Added detailed comments documenting the streaming query execution flow (`getBatch -> latestOffset -> getBatch` on restart) for both `Trigger.Once` and `Trigger.ProcessingTime`. ## How was this patch tested? - Added new unit test fileIdHash: MD5 for priming getBatch when both start and end are legacy — creates a synthetic checkpoint with two committed legacy batches (both at version boundary), restarts the stream, and verifies the first streaming getFiles call uses MD5 fileIdHash while subsequent calls use SHA256. - Extracted writeLegacyOffsetAndCommit helper to reduce boilerplate across legacy checkpoint tests. - Updated existing test mocks to align with the new endConvertedFromLegacy behavior in maybeGetLatestFileChangesFromServer, which limits the priming getBatch fetch range to endOffset.reservoirVersion: - E2E: legacy checkpoint isStartingVersion=false: changed mock from (1, 3) to (1, 2) + (3, 3) since priming now fetches up to endOffset.reservoirVersion=2. - E2E: legacy checkpoint mid-version (unlucky case): added batch 0 offset, replaced snapshot mock with streaming mock (1, 2) for priming getBatch, added (3, 3) for post-transition. - fileIdHash: MD5 for mid-version legacy offset: added batch 0 offset, replaced snapshot mock with streaming mock (1, 2) for priming getBatch. - E2E: convertDeltaSourceOffsetToLegacyOffset: added batch 0 offset, replaced snapshot mock with streaming mock (1, 2), updated batch assertions from batch 1 to batch 2. ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Sun Cao <sun.cao+data@databricks.com> Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…io#6505) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Previously, in ClusteredTableTestUtils, we had a flaky check for clustering columns that could, under rare occasions, not see updates to the Hive catalog. This change fixes that by moving up an await function call. ## How was this patch tested? Existing tests ## Does this PR introduce _any_ user-facing changes? No Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…lta-io#6478) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This PR removes an old spark config that was [introduced](delta-io@215996f) as a potential rollback mechanism for implicit type casting. The configuration has defaulted to true for ~2 years and removing it simplifies some code by removing early returns. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> unit tests ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No --------- Signed-off-by: Brooks Walls <brooks.walls@databricks.com> Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…#6513) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The `get_spark_version_info.py` script handles `"master"` as a special alias (maps to `isMaster=true`) but doesn't handle `"default"` (should map to `isDefault=true`). This is inconsistent with `CrossSparkVersions.scala` which handles both aliases at line 314: ```scala val resolvedInput = input match { case "default" => SparkVersionSpec.DEFAULT.fullVersion case "master" => ... } ``` When `run-tests.py` is invoked without `SPARK_VERSION` set, it passes `"default"` to `get_local_package()`, which calls the script with `--get-field default packageSuffix`. The script fails because `"default"` doesn't match any `shortVersion` or `fullVersion`, causing a fallback to an empty package suffix. After delta-io#5840 (always include Spark suffix), this produces the wrong artifact name (`delta-spark_2.13` instead of `delta-spark_4.1_2.13`). **Fix:** Add `"default"` → `isDefault=true` matching, consistent with how `"master"` → `isMaster=true` already works. ## How was this patch tested? This is a 4-line fix to the version query script. The fix aligns the Python script's alias handling with the Scala code that already handles `"default"`. ## Does this PR introduce _any_ user-facing changes? No. This is an internal build infrastructure fix.
#### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (CI / Build Infrastructure) ## Description The `dev/requirements.lock` file introduced in delta-io#6440 contains an absolute local filesystem path in the `uv pip compile` header comment. This replaces it with relative paths. Before: ``` # uv pip compile ... -o /Users/vishnu.c/.../dev/requirements.lock .../dev/requirements.txt ``` After: ``` # uv pip compile ... -o dev/requirements.lock dev/requirements.txt ``` This is a follow-up to delta-io#6440. ## How was this patch tested? Visual inspection. The lock file content (package versions and hashes) is unchanged; only the auto-generated comment header is modified. ## Does this PR introduce _any_ user-facing changes? No.
…hecksum (delta-io#6476) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Follow up for delta-io#3899. Rebases delta-io#3907 over master Major changes: 1. Makes it so that histograms of file sizes are tracked in the version checksum --- these are later used for validation. These are computed incrementally to avoid state reconstruction. 2. Also, now that Version Checksum can be incrementally computed without triggering a full state reconstruction (see delta-io#3899, delta-io#3895), this PR enables writing the version checksum by default. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Added FileSizeHistogramSuite and updated some existing suites. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…tCoordinatorClient (delta-io#6486) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Add support on UCCommitCoordinatorClient for updating UniForm metadata. This is part of project to support UniForm for UC managed table ## How was this patch tested? ``` build/sbt -DsparkVersion=4.0 "spark/testOnly *UCCommitCoordinatorClientSuite ```
…ta-io#6510) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Upgrades the OSS Unity Catalog dependency from `0.4.0` to `0.4.1` in: - `build.sbt` (used by Spark and Kernel unitycatalog modules) - `examples/scala/build.sbt` (used by Scala examples) ## How was this patch tested? Dependency version bump only. The unitycatalog integration tests cover this dependency. ## Does this PR introduce _any_ user-facing changes? No. Signed-off-by: openinx <openinx@gmail.com>
…f CTAS test should check TableInfo (delta-io#6511) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Use translatesDeprecatedUcTableIdOnManagedCreate to control if CTAS test should check TableInfo. ## How was this patch tested? Changing existing integration test ## Does this PR introduce _any_ user-facing changes? No Signed-off-by: Yi Li <yi.li@databricks.com>
…a-io#6520) ## Summary - Use `SparkSession.getActiveSession` instead of `SparkSession.active` in `DeltaUDF.createUdfFromTemplateUnsafe` to prevent `IllegalStateException` when the SparkContext is stopped (e.g. during query cancellation or cluster shutdown). - When no active session is available, falls through to `orElse` which creates a fresh UDF safely. ## Test plan - Existing unit tests for DeltaUDF should continue to pass. - Verified that the change matches the fix in databricks-eng/runtime#208197.
…torage credentials in plan trees and exceptions (delta-io#6383) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description `DeltaTableV2.toString` previously interpolated the full `CatalogTable`, `SparkSession`, and raw `options` map. When a catalog (e.g. Unity Catalog) injects Hadoop filesystem config into `CatalogTable.storage.properties` at table-load time, stringifying `CatalogTable` dumps `fs.*` keys — including auth tokens, access keys, and internal URIs — into exception messages, EXPLAIN output, and logs. The primary vector: Spark's `ExtendedAnalysisException` appends the analyzed plan tree to `getMessage`. For unsupported operations like `SHOW CREATE TABLE` on Delta, `TreeNode.argString` calls `DeltaTableV2.toString` via `case other => other :: Nil` (since `DeltaTableV2` is a V2 `Table`, not a direct `CatalogTable` product field). This PR keeps the original `toString` format with all six constructor parameters, but filters `fs.*` keys from both `CatalogTable.storage.properties` and the `options` map before stringifying. It reuses the existing `HIDDEN_STORAGE_PROPERTY_PREFIXES` constant from `DeltaTableV2.properties()` (introduced in PR delta-io#6300) as the single source of truth for which key prefixes to suppress. Complements PR delta-io#6300 which filters `fs.*` from `DeltaTableV2.properties()` (the V2 DESCRIBE/SHOW TBLPROPERTIES path). That fix does not cover `toString`. ## How was this patch tested? New `DeltaTableV2Suite` with 11 tests: - **Negative (2):** Asserts `toString` omits all `fs.*` storage keys and values (6 key varieties); verifies `fs.*` keys are filtered while non-`fs` keys (e.g. `delta.*`, `io.unitycatalog.*`) are preserved. - **Positive (4):** Verifies `CatalogTable` metadata (identifier, type, provider) is preserved; all six original parameters appear; path-only and tableIdentifier-only variants work correctly. - **Options filtering (2):** Verifies `fs.*` keys in the `options` map are filtered; verifies simultaneous filtering of both `CatalogTable` storage and `options`. - **Edge case (1):** Empty storage properties. - **Integration (2):** Uses `DummySessionCatalog` (injects `fs.myKey` into storage properties). `EXPLAIN EXTENDED` test validates no plan node leaks credentials. `SHOW CREATE TABLE` test intercepts the `AnalysisException` and asserts both `getMessage` and `toString` omit storage properties — directly exercising the `ExtendedAnalysisException` → plan tree → `DeltaTableV2.toString` chain. All 11 tests pass (`./build/sbt "spark/testOnly org.apache.spark.sql.delta.catalog.DeltaTableV2Suite"`). ## Does this PR introduce _any_ user-facing changes? Yes. The string representation of `DeltaTableV2` in exception messages and plan output no longer includes `fs.*` storage properties or `fs.*` entries from the options map. The overall `toString` format is unchanged — all six constructor parameters are still present. This is a security improvement that prevents catalog-injected credentials from appearing in error messages, EXPLAIN output, and logs.
… tables (delta-io#6356) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR adds a config to block Spark 4.0 clients from being able to write to Delta tables containing the variant data type since Spark 4.0 does not support the Variant logical type annotation. This could lead to weird inconsistencies where Spark 4.0 can perform pure inserts into these tables but cannot perform any operation that requires reading existing data that would have been written by newer clients that write the variant logical type annotation. ## How was this patch tested? UTs ## Does this PR introduce _any_ user-facing changes? Yes --------- Co-authored-by: seewishnew <vishnu.c@databricks.com>
## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/6394/files) to review incremental changes. - [**stack/refactorStartVersionNotFoundExcecption**](delta-io#6394) [[Files changed](https://github.com/delta-io/delta/pull/6394/files)] - [stack/failOnDataLoss2](delta-io#6395) [[Files changed](https://github.com/delta-io/delta/pull/6395/files/0c83e18a239a99178ec59db8b2b4634737fef7a6..c01f4d435ff5c73deae282e8b7d1d6913f55a434)] --------- #### Which Delta project/connector is this regarding? Kernel ## Description Introduces a dedicated `StartVersionNotFoundException` exception class that extends `KernelException`, replacing the plain `KernelException` previously thrown by `DeltaErrors.startVersionNotFound()`. The new exception exposes structured fields: - `getTablePath()` — the table where the version was not found - `getStartVersionRequested()` — the version that was requested but missing - `getEarliestAvailableVersion()` — the earliest version still available (`Optional<Long>`) This follows the same pattern as `CommitRangeNotFoundException` and allows callers to programmatically handle missing start versions (e.g., for `failOnDataLoss=false` in DSv2 streaming) without fragile message parsing. Resolves the kernel-side changes for delta-io#6380 ## How was this patch tested? Existing tests continue to pass since `StartVersionNotFoundException extends KernelException`. Callers that previously caught `KernelException` still catch the new subclass. The structured fields are exercised by the DSv2 `failOnDataLoss` tests in the stacked PR delta-io#6395. ## Does this PR introduce _any_ user-facing changes? No
…elta-io#6324) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (protocol) ## Description Add a proposal RFC to document parquet compression. ## How was this patch tested? N/A ## Does this PR introduce _any_ user-facing changes? No --------- Co-authored-by: Scott Sandre <scott.sandre@databricks.com>
…6518) ## Summary - `UpdateMetricsHook` now reads the post-commit snapshot's file size histogram from `checksumOpt` (the CRC) — this avoids triggering state reconstruction, even for blind appends. - When the CRC histogram is unavailable (e.g. legacy tables without CRC), the remaining metrics are still sent with the `file_size_histogram` field omitted (`None`). - Removes duplicate `DEFAULT_BINS`/`emptyHistogram` from `FileSizeHistogram` companion object — these are now provided by `FileSizeHistogramUtils` (added in delta-io#6476). Supersedes delta-io#6502 (now redundant — delta-io#6476 already exported `FileSizeHistogramUtils`) and delta-io#6503 (which was stacked on delta-io#6502 and has merge conflicts). ## Test plan - [x] `build/sbt "spark/testOnly *UpdateMetricsHookSuite"` — 21/21 pass, including: - Snapshot histogram is used when available - `file_size_histogram` omitted from JSON when CRC unavailable - End-to-end: async hook does not trigger state reconstruction (asserts `!stateReconstructionTriggered` after `awaitCompletion`) - [x] `build/sbt "spark/testOnly *FileSizeHistogram*"` — 18/18 pass (no regressions from companion object cleanup) - [x] `build/sbt "spark/compile"` — clean compilation This pull request was AI-assisted by Isaac. Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…eberg conversion (delta-io#6500) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> ## Description Fix field assignment in schema for Delta UniForm in an edge case (initial conversion with nested schema) _Context_: Iceberg `CREATE_TABLE` reassigns the field id in schema, which is not consistent with Delta in edge cases (For nested schemas). As data files written in the txn will rely on it, the schema and partitionSpec in the txn need to be overwritten to be consistent with Delta again ## How was this patch tested? Add UTs ``` build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite" ```
…der (delta-io#6524) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR limits the flink tests to only run on changes to `flink/kernel` folder. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Tested on github <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. -->
… WKB with stats (delta-io#6471) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description Adds Parquet read/write support for geometry/geography columns as part of the geospatial table feature ([spec](delta-io#4726)). Geometry and geography values are stored as WKB (Well-Known Binary) in Parquet BINARY columns annotated with GEOMETRY/GEOGRAPHY logical types. Rather than introducing new basic vector types in the kernel, the Parquet reader uses `getBinary()` to fetch geospatial data -- it is opaque binary from the reader's perspective. Geospatial statistics (bounding boxes) are stored in a separate Parquet metadata object (`GeospatialStatistics`, see [parquet-format spec](https://github.com/apache/parquet-format/blob/master/Geospatial.md)), but in Delta they are regular min/max column stats. The `ParquetStatsReader` translates between the two representations: it reads the Parquet bounding box and converts it to WKT POINT min/max literals that fit into the standard Delta stats model. ### Changes - **build.sbt**: Upgrade parquet-hadoop 1.12.3 -> 1.16.0 (needed for geospatial logical types) - **DefaultBinaryVector**: Allow `GeometryType`/`GeographyType` in constructor and `getBinary()` - **ParquetColumnReaders**: Route geometry/geography to `BinaryColumnReader` - **ParquetColumnWriters**: Route geometry/geography to `BinaryWriter` - **ParquetSchemaUtils**: Map `GeometryType` -> BINARY with `geometryType(srid)` annotation; `GeographyType` -> BINARY with `geographyType(srid, algorithm)` annotation - **ParquetStatsReader**: Read `GeospatialStatistics` bounding box from Parquet metadata; convert to WKT POINT min/max literals via `GeometryUtils.formatPointWKT()` ## How was this patch tested? - `GeometryParquetReaderSuite`: WKB roundtrip with null and non-null rows for both GeometryType and GeographyType; bounding box stats verification for GeometryType - `ParquetSchemaUtilsSuite`: GeometryType/GeographyType Parquet schema conversion, algorithm mapping, and invalid algorithm rejection ## Does this PR introduce _any_ user-facing changes? Adds new table feature --------- Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
delta.parquet.compression.codecas a recognized and validated table property in Kernel Java, following the protocol spec added in [PROTOCOL] Add delta.parquet.compression.codec property to protocol delta-io/delta#6324.uncompressed,none,snappy,gzip,lz4,lz4_raw,zstd. Default:zstdper protocol.ParquetFileWriter).Test plan
TableConfigSuite: unit tests for validation (valid/invalid values, case normalization, default, key normalization)TablePropertiesSuite: integration tests for create/update table with the property, and invalid value rejectionThis pull request was AI-assisted by Isaac.