Skip to content

[kernel] Add delta.parquet.compression.codec table property - #11

Closed
scottsand-db wants to merge 1527 commits into
masterfrom
stack/kernel_java_compression_codec
Closed

scottsand-db wants to merge 1527 commits into
masterfrom
stack/kernel_java_compression_codec

Conversation

@scottsand-db

Copy link
Copy Markdown
Owner

Summary

  • Adds delta.parquet.compression.codec as a recognized and validated table property in Kernel Java, following the protocol spec added in [PROTOCOL] Add delta.parquet.compression.codec property to protocol delta-io/delta#6324.
  • Valid values (case-insensitive): uncompressed, none, snappy, gzip, lz4, lz4_raw, zstd. Default: zstd per protocol.
  • Scope: property recognition and validation during CREATE/UPDATE/REPLACE table. Writer consumption of the property is a follow-up (TODO added to ParquetFileWriter).

Test plan

  • TableConfigSuite: unit tests for validation (valid/invalid values, case normalization, default, key normalization)
  • TablePropertiesSuite: integration tests for create/update table with the property, and invalid value rejection

This pull request was AI-assisted by Isaac.

yyanyy and others added 30 commits January 28, 2026 16:54
…olumn pruning (delta-io#5894)

#### Which Delta project/connector is this regarding?


- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description
Update SparkScan estimated size in bytes to take account in column
pruning. This behavior change follows the interface comment in
[SupportsReportStatistics](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsReportStatistics.java#L26-L28)
where this stats API is being implemented, since now that DSv2 only
returns the columns after pruning if column filter is pushed down.

For the estimation calculation, it follows the exact same behavior as in
`SizeInBytesOnlyStatsPlanVisitor.visitUnaryNode` from Spark.
- I looked into delta features such as `materializePartitionColumns`
that allows writing partition columns to the data files, which may in
theory impact estimated size; however, this feature allows arbitrary
turning on and off, which can result in a table containing some files
with materialized partition columns within the data files and others
without. Given that the existing estimation code based on schema size is
arguably already not accurate, I decided to keep the logic as is today
for complete feature parity, but this potentially could be a future
improvement area.



## How was this patch tested?

Unit test. Also tested this against local Spark code with corresponding
changes and loggings, and verified that in `CostBasedJoinReorder`, size
estimate is the exact same as in before:
```
v1
61329:[JoinReorderWithGroups] Input: Project
61330:[JoinReorderWithGroups]   sizeInBytes=98028
61331:[JoinReorderWithGroups]   output.size=1
61332:[JoinReorderWithGroups]   canBroadcastBySize=true
61333:[JoinReorderWithGroups]   Leaf: LogicalRelation, sizeInBytes=2418043, output.size=28
61334:[JoinReorderWithGroups] Input: Project
61335:[JoinReorderWithGroups]   sizeInBytes=949
61336:[JoinReorderWithGroups]   output.size=2
61337:[JoinReorderWithGroups]   canBroadcastBySize=true
61338:[JoinReorderWithGroups]   Leaf: LogicalRelation, sizeInBytes=11636, output.size=26
---
v2
29166:[JoinReorderWithGroups] Input: Project
29167:[JoinReorderWithGroups]   sizeInBytes=98028
29168:[JoinReorderWithGroups]   output.size=1
29169:[JoinReorderWithGroups]   canBroadcastBySize=true
29170:[JoinReorderWithGroups]   Leaf: DataSourceV2ScanRelation, sizeInBytes=130705, output.size=2
29171:[JoinReorderWithGroups] Input: Filter
29172:[JoinReorderWithGroups]   sizeInBytes=949
29173:[JoinReorderWithGroups]   output.size=2
29174:[JoinReorderWithGroups]   canBroadcastBySize=true
29175:[JoinReorderWithGroups]   Leaf: DataSourceV2ScanRelation, sizeInBytes=949, output.size=2

```


## Does this PR introduce _any_ user-facing changes?
no

---------

Co-authored-by: Gengliang Wang <gengliang@apache.org>
…arkTable equality check (delta-io#5903)

- Also improved thread safety with filter() method for updating state
mutations



#### Which Delta project/connector is this regarding?


- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description
- This is a follow up of delta-io#5827 ,
to add the missing snapshot info to the equality check of `SparkTable`.
- Separately, it was noticed that the current `filter()` method in
`SparkScan` mutates states, but such mutation was not thread safe and
not reflected in equality checks.
- for equality check, only the effective filter is tracked. This is
aligned with the Iceberg approach in
[SparkBatchQueryScan](https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L128-L162).
- However, a difference is that Iceberg uses a list (i.e. order
dependent) for the incoming filters for equality check
[here](https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L161-L162),
where we are using a set in this change. From the original PR
apache/iceberg#3400 it seems that it was not
discussed, so I doubt if it is intentional.
 
## How was this patch tested?

Additional unit test

## Does this PR introduce _any_ user-facing changes?
No
…directory (delta-io#5880)

#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Follow-up of delta-io#5325
Remove obsolete .gitignore file from icebergShaded directory. It was
created when icebergShaded was pulling Iceberg source code and appling
patches to create a custom JAR. Now it relies on Maven JAR

## How was this patch tested?
N/A

## Does this PR introduce _any_ user-facing changes?
No

Signed-off-by: Felipe Fujiy Pessoto <fepessot@microsoft.com>
Which Delta project/connector is this regarding?
- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (fill in here)

## Description

Fixes delta-io#5392 

## Does this PR introduce _any_ user-facing changes?
No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?


<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [x] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR exposes `delta-spark` version in python as `delta.__version__`.
The user will be able to get it using `from delta import __version__`.
The variable will contain the package version in the `x.y.z` format.
This is a common (although optional) pattern in python packages and is
provided in numpy, pandas, pyspark, and many other packages.

Delta version is stored in `version.sbt`. This is a single source of
truth for getting the package version and is used by the `build.sbt`
file to add the version to the Scala package and by setup.py when
building a delta-spark whl.

This PR adds a new `generatePythonVersion` sbt task. This task will read
the version from `version.sbt` file and will autogenerate the
`python/delta/version.py` file. The autogeneration code can be run
manually `build/sbt sparkV1/generatePythonVersion` and is run
automatically on each `build/sbt sparkV1/compile` or `build/sbt compile`
run.

PR also contains premerge tests that compare if the version in
`version.sbt` and `version.py` match. This means that if the
`version.sbt` version is changed and `version.py` isn't (although the
latter should be updated automatically during sbt compile), CI will
block the PR and tell the developer to run `build/sbt
sparkV1/generatePythonVersion`.

Additionally, this PR fixes some failing linter checks:
```
# false positive when checking f-strings, fixed by ignoring E231
/home/alex.khakhlyuk/oss/delta/python/delta/tests/test_pip_utils.py:64:66: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:32: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:48: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:45:41: E231 missing whitespace after ','
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:22: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:63: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E231 missing whitespace after ';'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E231 missing whitespace after ';'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E231 missing whitespace after ';'

# false positive detecting backticks ` in sql queries, fixed by ignoring W604
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:360:49: W604 backticks are deprecated, use 'repr()'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:371:49: W604 backticks are deprecated, use 'repr()'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:382:49: W604 backticks are deprecated, use 'repr()'

# removed ; at the end of the SQL queries
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E702 multiple statements on one line (semicolon)
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E702 multiple statements on one line (semicolon)
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E702 multiple statements on one line (semicolon)
```

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
`test_version.py` verifies if `delta.__version__` is valid and if it's
up to date with `version.sbt`.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

Python users will be able to read delta version via `from delta import
__version__`.
…e reconstruction (delta-io#5916)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR adds support for preserving variant stats from JSON AddFile
records during checkpointing. Since stats are stored in Z85 encoding,
they need to be decoded after the `from_json` expression. This PR
implements the `DecodeNestedZ85EncodedVariant` expression for decoding
variant stats from the Z85 representation. This PR also uses the
`DecodeNestedZ85EncodedVariant` expression when writing variant stats as
struct in `stats_parsed` in checkpoint files.

For checkpointing, this PR ensures that variant stats are not shredded.
This is because variant reconstruction from shredded files can lose
information about integer width.

This PR also introduces the `EncodeNestedVariantAsZ85String` expression
which is used during state reconstruction to read variant stats stored
in `stats_parsed` in Checkpoint files as JSON. This expression is
applied to `stats_parsed` before `to_json` to replace all variants with
their Z85 representation before encoding as JSON.

This PR also prohibits Variant from being a permitted for clustering.

These changes are compatible with both - Spark 4.0 and Spark 4.1.

## How was this patch tested?

Using golden files. We copied golden tables into this project where the
AddFiles contain variant stats and tested that checkpointing on this
table preserves variant stats.

## Does this PR introduce _any_ user-facing changes?

Yes

---------

Co-authored-by: Dhruv Arya <dhruv.arya@databricks.com>
… cleanup (delta-io#5917)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description
Add basic support for `checkpointProtection` table feature in Kernel by
limiting log cleanup after checkpointing. Only do log cleanup when
1. Snapshot was built as "latest" by intent (not time-traveled)
2. checkpointProtection feature is not enabled
3. delta.enableExpiredLogCleanup table property is set to true
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Added tests
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
## Stacked PR
Stacked on delta-io#5917
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
In delta-io#5917, we added basic support
for `checkpointProtection`. This PR adds it to the allowlist in
IcebergWriterCompat.

`checkpointProtection` is a Delta-internal feature that prevents some
writes. It doesn't affect data formats or schemas, so is invisible to
Iceberg readers — safe to be allowed.

## How was this patch tested?
Existing
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
…o#5918)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
As titled.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UTs.
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No.
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…ta-io#5964)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR updates DataSkippingReader to exclude filters that contain
user-defined functions (UDFs) from data skipping, since such UDFs can be
non-deterministic even when they aren’t explicitly marked as such by the
user.

## How was this patch tested?

UTs

## Does this PR introduce _any_ user-facing changes?

Yes, UDFs will not used as a data skipping predicate for partition
filters anymore since they could be non-deterministic.
…hanging commit coordinator (delta-io#5967)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
As titled. 

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UTs.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
Yes, now CREATE OR REPLACE is not blocked even when explicitly
specifying CC props.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR fixes an issue where we accidentally print Java objects because
we're printing `Option[Array[T]]`, where Array is a Java array. Call
`.toSeq` to convert to Scala and prevent Java array object references
(e.g., `"[J@1a2b3c4d"`) in event data.

## How was this patch tested?
Existing tests pass. Manual testing since I encountered this while
debugging an issue.

## Does this PR introduce _any_ user-facing changes?

No.
…aged tables (delta-io#5968)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Clarify the maintenance operation requirements in the Catalog-Managed
Tables RFC.
 
## How was this patch tested?

n/a

## Does this PR introduce _any_ user-facing changes?

Doc change only.
…ure (delta-io#5708)

## Description

This feature is in RFC stage and will be merged into the protocol later.
This also clarifies some of the wording around enablement/support of
this table feature in the RFC, bringing it back from
delta-io#5619.

See delta-io#5555 

## How was this patch tested?

N/a

## Does this PR introduce _any_ user-facing changes?

No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->


## Description
Add a `IcebergSyncConverterHook` to `IcebergConverterHooks`, which only
supports synchronization Iceberg metadata conversion

## How was this patch tested?
Existing tests
…a-io#5935)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Follow up for delta-io#5726

- Change the default value of `spark.databricks.delta.v2.enableMode`
from `NONE` to `AUTO`
- Remove explicit mode configuration from UC streaming tests (now uses
default)
- Clean up unused V2_ENABLE_MODE constants from test base class

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

- UCDeltaStreamingTest
- CI/CD passing with AUTO mode

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No

Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>
…lta queries (delta-io#5964)" (delta-io#5982)

#### Which Delta project/connector is this regarding?
- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This reverts commit 9997fd8 because it
has a correctness bug around metadata-only queries i.e. queries
referencing UDFs which don't reference any columns.

## How was this patch tested?

Build

## Does this PR introduce _any_ user-facing changes?

No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Update the version reference for publishing API docs.

Co-authored-by: Scott Haines <newfrontcreative@gmail.com>
## Summary

Implement proper handling of column names containing dots (literal
column names vs. nested fields) for server-side planning. This PR adds
backtick escaping for literal dotted columns in **projections** while
using raw field names for **filters**.

### Why do we need to escape projections but not filters?

Spark's Filter API attribute is just a String. By the time we receive
it, Spark has [already
resolved](https://github.com/apache/spark/blob/dd2c62f790a821440baa75792a98c6ed5a37e519/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala#L44)
what column it refers to. But when Spark gives us projections, they come
as a hierarchical StructType which has to be flattened.

### Challenge with flattening

Naive flattening can cause ambiguity - what if there is a struct `xx`
with a field `yy` versus a top-level column called `xx.yy`? We solve
this by:
1. Using Spark's `SchemaUtils.explodeNestedFieldNames()` which properly
escapes dotted field names
2. Filtering out parent struct fields to keep only leaf fields

## What Changed

### Projection Column Name Handling
- **Use Spark's built-in escaping**: Leverage
`SchemaUtils.explodeNestedFieldNames()` to flatten and escape field
names with proper backtick handling
- **Filter parent structs**: Remove intermediate parent struct fields
from projections, keeping only leaf fields that contain actual data
- For schema `STRUCT<a: STRUCT<b.c: STRING>>`, return `["a.\`b.c\`"]`
not `["a", "a.\`b.c\`"]`
- This ensures the server only reads specific nested fields, not entire
parent structs

### Test Coverage
- **End-to-end escaping test**: Validates proper projection handling for
dotted column names in `ServerSidePlannedTableSuite`
- **Multiple scenarios**: Tests nested structs with dotted fields,
top-level dotted columns, and combinations
- **Iceberg integration tests**: Verifies escaping works correctly in
`IcebergRESTCatalogPlanningClientSuite`

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace technote-space/get-diff-action with native GitHub Actions
paths-ignore to prevent workflows from running on .md and .txt file
changes. This saves CI resources by not allocating runners for
documentation-only PRs.

Also removes references to non-existent unity/** directory in
kernel_unitycatalog_test.yaml (Unity Catalog code is at
kernel/unitycatalog/, already covered by kernel/**).


#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [X] Other (fill in here)

## Description

Documentation-only changes (.md, .txt files) currently trigger all CI
workflows, wasting compute resources on unnecessary test runs. Using
native paths-ignore prevents workflows from starting entirely, saving CI
time and removing the dependency on third-party actions.


## How was this patch tested?

YAML syntax was validated locally. The workflows will be fully tested
when this PR is pushed—GitHub Actions will verify the workflow files
parse correctly and the path filtering behaves as expected on subsequent
commits.

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sebastian Baunsgaard <sebastian.baunsgaard@databrics.com>
Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>
…ctory (delta-io#5907)

## Summary

Implement reflection-based auto-registration for
`ServerSidePlanningClientFactory` to automatically discover and load the
`IcebergRESTCatalogPlanningClientFactory` when the delta-iceberg JAR is
on the classpath. This eliminates the need for manual factory
registration in production code while maintaining backward compatibility
for testing scenarios.

## What Changed

### Core Auto-Registration Logic
- **Lazy initialization with reflection**: Added
`tryAutoRegisterFactory()` method that uses `Class.forName()` to load
`IcebergRESTCatalogPlanningClientFactory` on first access
- **Thread-safe initialization**: Implements double-checked locking
pattern to ensure one-time, thread-safe factory registration
- **Smart triggering**: Auto-registration only attempts when
`getFactory()` is called and no factory has been manually set
- **Clear error handling**: Throws descriptive `IllegalStateException`
when delta-iceberg JAR is missing, guiding users to add it to classpath


## Testing

The changes maintain existing test compatibility while enabling
automatic registration in production:
- Test code can still use `setFactory()` to inject mock implementations
- `clearFactory()` now properly resets auto-registration state for test
isolation
- JarSuite verifies serverSidePlanning classes are packaged correctly

## How It Works

```mermaid
flowchart TD
    Start[getFactory called] --> Check1{Factory already set?}
    Check1 -->|Yes| Return[Return factory]
    Check1 -->|No| TryAuto[tryAutoRegisterFactory]
    TryAuto --> Check2{Already attempted?}
    Check2 -->|Yes| CheckReg{Factory registered?}
    Check2 -->|No| Lock[synchronized block]
    Lock --> DoubleCheck{Still not attempted?}
    DoubleCheck -->|No| CheckReg
    DoubleCheck -->|Yes| SetFlag[Set autoRegistrationAttempted = true]
    SetFlag --> Reflect[Class.forName IcebergRESTCatalogPlanningClientFactory]
    Reflect --> Success{Loaded?}
    Success -->|Yes| Register[registeredFactory = Some factory]
    Success -->|No| Error[Throw IllegalStateException]
    Register --> Return
    CheckReg -->|Yes| Return
    CheckReg -->|No| Error
```

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…nWiderType evolution mode (delta-io#5969)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Changes the TypeEvolution mode `AllTypeWideningToCommonWiderType` to
infer a wider decimal type between integers and decimals. For example,
when calling `getWidenedType(IntegerType, DecimalType(2, 1))`, the
result will be `DecimalType(11,1)` because `IntegerType` is narrower
than `DecimalType(10, 0)` which has a common wider type with
`DecimalType(2, 1)`.
## How was this patch tested?
Add to existing test in `SchemaUtilsSuite`
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…5774)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/5774/files) to
review incremental changes.
-
[**stack/dv_pr2_phase1_basic_read**](delta-io#5774)
[[Files changed](https://github.com/delta-io/delta/pull/5774/files)]
-
[stack/dv_pr3_phase2_vectorized](delta-io#5775)
[[Files
changed](https://github.com/delta-io/delta/pull/5775/files/6b90bdd14bd86a5bcfb109fb12d9447b769d4d24..9446011c512f4fa4d7f06417eb4629671813dbe4)]
-
[stack/dv_pr4_phase3_file_splitting](delta-io#5776)
[[Files
changed](https://github.com/delta-io/delta/pull/5776/files/9446011c512f4fa4d7f06417eb4629671813dbe4..5e798ef3a37c07b86f12de6f5465a5db3fbff153)]
-
[stack/dv_pr5_streaming_support](delta-io#5877)
[[Files
changed](https://github.com/delta-io/delta/pull/5877/files/5e798ef3a37c07b86f12de6f5465a5db3fbff153..1c4ddacd08654e2d778bfdfb992320cb6dcead9f)]
- [stack/dv_pr5_test](delta-io#5975)
[[Files
changed](https://github.com/delta-io/delta/pull/5975/files/6b90bdd14bd86a5bcfb109fb12d9447b769d4d24..24ae4ed8d474449407582b05c0c0ab45f97a72ed)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->


Add basic deletion vector (DV) read support for the Spark V2 connector
using row-based filtering.

### Changes:
- `DvSchemaContext`: POJO to manage DV schema context (column indices,
output schema)
- `DeletionVectorReadFunction`: Wraps base reader to filter deleted rows
and project out DV column
- `PartitionUtils`: Creates DV-aware `PartitionReaderFactory` with
`DeltaParquetFileFormatV2`
- Add`serializeToBase64()` to Kernel's `DeletionVectorDescriptor`

### How it works:
1. Add `__delta_internal_is_row_deleted` column to read schema
2. Filter rows where DV column != 0 (deleted)
3. Project out DV column from output

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

- `DvSchemaContextTest`: Unit tests for schema manipulation
- `DeletionVectorReadFunctionTest`: Unit tests for row filtering and
projection
- Golden table tests with DV tables pass
## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
…-io#5904)

## Summary
This PR fixes Unity Catalog credential detection and refactors Iceberg
REST catalog endpoint construction to properly implement the Iceberg
REST catalog specification.

**Key Changes:**
- Fix credential detection to recognize Unity Catalog's `option.fs.*`
credential properties
- Refactor `/v1/config` endpoint handling to follow Iceberg REST catalog
spec
- Move prefix discovery logic from `UnityCatalogMetadata` to
`IcebergRESTCatalogPlanningClient`
- Improve robustness with trailing slash handling and HTTP client reuse
- More test coverage with explicit path verification
- Fix module separation issues and improve code organization

## Testing
Tests now explicitly verify:
- Correct `/v1/config` endpoint calls with `warehouse` query parameter
- Proper prefix application in `/plan` request paths
- Behavior when no prefix is returned (uses baseUri directly per Iceberg
REST spec)
- Token handling consistency between tests and production
- Trailing slash handling robustness

---

## Compatibility
This is an internal refactoring with no changes to public APIs or
user-facing behavior. The changes:
- Improve correctness of credential detection for Unity Catalog
- Fix internal endpoint construction to follow Iceberg REST spec
- Maintain backward compatibility with existing Unity Catalog
configurations

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…lta-io#5963)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

- Fix flaky RowTrackingBackfillConflictsSuite Scenarios 3 and 6 by
making backfill batch ordering deterministic
- Add setDeterministicInsertionTimes() to explicitly set insertion times
on files after table creation, ensuring partitions 0-1 are processed by
batch 1 and partitions 2-3 by batch 2
- Update concurrent transactions (DELETE, UPDATE, MERGE, OPTIMIZE) to
touch only partitions 1 and 2, leaving partition 3 untouched so batch 2
always has work
- Also add some tracking to allow busyWait routines to fail fast if the
futures that normally get checked later already failed.

## How was this patch tested?

Test-only PR

## Does this PR introduce _any_ user-facing changes?

No
…f literals (delta-io#5987)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Unifying the Parsing of Partition Values.
The commit
delta-io@7b69041
fixed incorrect conversion of timestamp values with non-UTC session
timezones.
Tables with data written before this change may still exist, leading to
different unnormalized partition value formats which can lead to
inconsitencies for READ as well.
We unify the read and write path of the partition values, ensuring that
we compare only the literals with the correct data type and if needed,
return a normalized string format for all data types.

## How was this patch tested?
Introduced the `AddFileSuite` with unit tests for parsing different
partition values.

## Does this PR introduce _any_ user-facing changes?
No.
…r and ucClient.commit API (delta-io#5634)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/5634/files) to
review incremental changes.
- [**stack/uc_api**](delta-io#5634)
[[Files changed](https://github.com/delta-io/delta/pull/5634/files)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
This PR adds support for Delta Universal Format (UniForm) metadata to be
passed from Delta Kernel through the Unity Catalog commit API.

1. Extended UCClient.commit() to accept Optional<UniformMetadata>
uniform parameter
2. Added UniformAdapter Utility to extract UniForm metadata from
committer properties
3. Update UCCatalogManagedCommitter now extracts Iceberg metadata from
CommitMetadata.committerProperties and passes it to UCClient
## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
UTs
## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Move `canDowngradeToSnapshotIsolation` to allow reusability.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

Existing tests.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No.
#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This is to add an official doc on delta.io for catalog-managed tables.

## How was this patch tested?

Existing doc tests

## Does this PR introduce _any_ user-facing changes?

no.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

This PR provides a table property "delta.dataSkippingStringPrefixLength"
that allows users to override the prefix length table-wise. This helps
with data-skipping performance on columns with long prefix that exceeds
the default length (32), so that regular data-skipping does not work.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

UTs

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
murali-db and others added 29 commits April 5, 2026 10:58
…elta-io#6481)

## Description

Extract `createDeltaParquetFileFormat` as a public static method in
`PartitionUtils`, factoring it out of the inline construction inside
`createDeltaParquetReaderFactory`.

This makes the `DeltaParquetFileFormatV2` construction reusable by the
upcoming write path without duplicating the format creation logic.

**No behavioral change** — the read path calls the new extracted method
instead of constructing the format inline.

### Changes

- Add `createDeltaParquetFileFormat(Snapshot, String, boolean,
Option<Boolean>)` public static method to `PartitionUtils`
- Update `createDeltaParquetReaderFactory` to call the new method
- Remove now-unused local `protocol` and `metadata` variables from
`createDeltaParquetReaderFactory`

## How was this patch tested?

Existing `sparkV2` tests pass (`sbt "sparkV2/test"` — 496 tests, 0
failures). The extraction is a pure refactor with no behavioral change.

---------

## Stacked PR

- **[DSv2] Extract shared createDeltaParquetFileFormat in
PartitionUtils** ([delta-io#6481](delta-io#6481))
⬅️ _This PR_
- [DSv2] Add batch write implementation
([delta-io#6482](delta-io#6482))
- [DSv2] Add WriteBuilder and wire SparkTable.newWriteBuilder
([delta-io#6483](delta-io#6483))
…#6449)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6449/files) to
review incremental changes.
-
[stack/kernel-finalize-in-commit](delta-io#6448)
[[Files changed](https://github.com/delta-io/delta/pull/6448/files)]
[MERGED]
-
[**stack/dsv2-create-with-finalize**](delta-io#6449)
[[Files changed](https://github.com/delta-io/delta/pull/6449/files)]
- [stack/dsv2-wiring-e2e](delta-io#6450)
[[Files
changed](https://github.com/delta-io/delta/pull/6450/files/8e22865f0603c3e3fe4ce502bec55c0909ecdae1..d33e42c2fd3ab9dbf25ca3dc49554aea1d16c12a)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description

The major addition of this PR in the stack is the
**CreateTableBuilder**, which is the Spark-To-Kernel conversion layer
for DSv2 `CREATE TABLE` to build the **Kernel Transaction.** This is the
middle PR in a 3-PR stack:

```
Spark DSv2 inputs            CreateTableBuilder            Kernel types
-------------------          -------------------           ----------------

StructType (schema)   --->   prepare()            --->     DDLRequestContext
Transform[] (layout)                                       ├── kernelSchema
Map<String,String>                                         ├── dataLayoutSpec
Identifier + UCTableInfo                                   ├── filtered properties
Configuration                                              ├── comment (TODO delta-io#6473)
                                                           ├── engine
                                                           └── transactionBuilder
                                                                 ├─ UC path: UCCatalogManagedClient
                                                                 └─ local path: TableManager

-------------------- AFTER PREPARE -------------------------------

DDLRequestContext        --->     buildTransaction()     --->     Kernel Transaction
                                      (applies properties
                                       + data layout)
```

**CreateTableBuilder**
The CreateTableBuilder has two utilities:
1. `prepare()` which converts all the Spark related types to Kernel
types and filters properties for Kernel. For example,

Spark | Kernel
-- | --
StructType | io.delta.kernel.types.StructType
Transform[] (partitions/clustering) | DataLayoutSpec via
toDataLayoutSpec
Map<String, String> (properties) | Filtered Map<String, String> via
filterProperties
COMMENT property | Optional via extractComment
Configuration (Hadoop) | Engine via DefaultEngine.create(hadoopConf)

2. we also have `buildTransaction()` which takes all these converted
items (bundled together into DDLRequestContext) and builds the Kernel
Transaction.

**Key design decisions:**
- Property filtering: DSv2 plumbing keys (`location`, `provider`,
`owner`) are stripped because they're catalog metadata, not Delta table
properties. Protocol version overrides (`minReaderVersion`,
`minWriterVersion`, `ignoreProtocolDefaults`) are stripped because
Kernel resolves protocol internally and rejects these via
`TableConfig.validateAndNormalizeDeltaProperties`. UC table ID is
stripped as an internal routing marker.
- UC vs path-based routing: When `UCTableInfo` is present, wires up
`UCCatalogManagedClient` + `UCCatalogManagedCommitter` so the commit
writes to the staging location and finalizes (promotes) the table in UC.
Otherwise uses `TableManager.buildCreateTableTransaction` for path-based
tables.
- Comment preservation: `COMMENT 'x'` is extracted but not yet written
to the Delta log because Kernel's `CreateTableTransactionBuilder`
doesn't expose `withDescription()` yet (tracked in delta-io#6473).

**DDLRequestContext change**

Replaced `engineInfo` (String) with `transactionBuilder`
(`CreateTableTransactionBuilder`). The transaction builder is now
resolved during `prepare()` — UC or path-based — so `buildTransaction()`
is catalog-agnostic. This also supports future catalog clients without
changing the build step.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…io#6250)

## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415)
to review incremental changes.
- [stack/ignoreDeletesV2](delta-io#6245)
[[Files changed](https://github.com/delta-io/delta/pull/6245/files)]
-
[stack/skipChangeCommitsV2](delta-io#6246)
[[Files
changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)]
- [stack/ignoreChangesV2](delta-io#6249)
[[Files
changed](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de)]
-
[**stack/ignoreFileDeletionV2**](delta-io#6250)
[[Files
changed](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Support ignoreFileDeletion read option is DSv2, which skip a all remove
file actions but keep the add file actions in the same commit. And also
inform user that this read option has deprecated.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Unit tests that test parity between v1 and v2 connector on both pure
deletes commit (only remove) and change commit (add + remove).

Integration tests:
- streaming with ignoreFileDeletion = true allows both delete and change
commits

DV tests:
 - multiple deletion vectors per file - List((ignoreFileDeletion,true))
- allow to delete files after staring a streaming query when
ignoreFileDeletion is true
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…delta-io#6490)

## What changes were proposed in this pull request?

Extract `DataFiltersBuilder` from `DataSkippingReader` into its own
standalone file. This improves code organization by separating the data
filter construction logic from the data skipping reader.

## How was this patch tested?

Existing tests in `DataSkippingDeltaConstructDataFiltersSuite` continue
to pass with the refactored code.
…apshot (delta-io#6298)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Add end-to-end tests covering all data loss scenarios during initial
snapshot for Delta streaming sources, including both DSv1 and Kernel
(DSv2) code paths. The new tests verify behavior when:

  - Commit files are missing but a checkpoint exists (should succeed)
- Both checkpoint and commit files are deleted (should fail gracefully)
- Checkpoint is deleted but all commit files remain (should succeed via
fallback)
- A stream restarts mid-initial-snapshot after log retention cleans up
old versions (should fail with appropriate error)

Also updates existing `failOnDataLoss` tests to assert on the correct
exception types for Kernel (DSv2), which throws
`KernelException/InvalidTableException` instead of
`DeltaIllegalStateException`.

The V2 exclusion list in DeltaV2SourceSuite is adjusted to reflect which
scenarios Kernel can/cannot handle (e.g., Kernel cannot reconstruct a
snapshot without a checkpoint file).

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Test-only changes
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…hot) (delta-io#6075)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6075/files) to
review incremental changes.
- [**stack/cdf1**](delta-io#6075) [[Files
changed](https://github.com/delta-io/delta/pull/6075/files)]
- [stack/cdf2](delta-io#6076) [[Files
changed](https://github.com/delta-io/delta/pull/6076/files/ab609f8e2185d3ba863c00304195c24b60f7d04b..301cf056c94305a2ca5d96dc3dcdfd88d5dbc37b)]
- [stack/cdf2.5](delta-io#6391) [[Files
changed](https://github.com/delta-io/delta/pull/6391/files/301cf056c94305a2ca5d96dc3dcdfd88d5dbc37b..d2ce6b47593a08ed491742e024056b8f565dd33f)]
- [stack/cdf3](delta-io#6336) [[Files
changed](https://github.com/delta-io/delta/pull/6336/files/70544ed3a42014a51dd0d8d97e7c4f2333e6a221..fb5a4f6e478f68410739ce68e9a3fd92b2091be1)]
- [stack/cdf4](delta-io#6359) [[Files
changed](https://github.com/delta-io/delta/pull/6359/files/fb5a4f6e478f68410739ce68e9a3fd92b2091be1..e52ab30e7fe3923ba302e489368231b836ff4314)]
- [stack/cdf5](delta-io#6362) [[Files
changed](https://github.com/delta-io/delta/pull/6362/files/e52ab30e7fe3923ba302e489368231b836ff4314..45bc4f043f996ad56d6937f6f0b5d1876fe1130a)]
- [stack/cdf6](delta-io#6363) [[Files
changed](https://github.com/delta-io/delta/pull/6363/files/45bc4f043f996ad56d6937f6f0b5d1876fe1130a..7d8e01ec6211eba6712b4c54e08eb68f45e542b6)]
- [stack/cdf-outofrange](delta-io#6388)
[[Files
changed](https://github.com/delta-io/delta/pull/6388/files/216a484e5f6e7b20fa26d428703ca05dcbbb6b5a..fee69f12b67cc91d1aeed01a759f64940198405e)]
- [stack/cdf7](delta-io#6370) [[Files
changed](https://github.com/delta-io/delta/pull/6370/files/fee69f12b67cc91d1aeed01a759f64940198405e..7f5fc84dee6dc3386a7b18db5a0d1d7533024842)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

Adds initial snapshot write-time-CDC support to the DSv2 streaming read
path (SparkMicroBatchStream), bringing it closer to DSv1 feature parity.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…ncher download (delta-io#6501)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [x] Standalone
- [x] Flink
- [x] Kernel
- [x] Other (build infrastructure)

## Description

Adds optional `MAVEN_PROXY_URL` environment variable support to the SBT
build scripts, enabling seamless builds behind corporate proxies or in
hardened CI environments without manually editing the checked-in
`build/sbt-config/repositories` file.

**Changes in `build/sbt`:**
- When `MAVEN_PROXY_URL` is set, generates a temporary SBT repositories
config that routes all Maven and Ivy resolution through the proxy URL.
- When unset, falls back to the committed
`build/sbt-config/repositories` file — **no behavior change** for
existing users/CI.

**Changes in `build/sbt-launch-lib.bash`:**
- Adds `MAVEN_PROXY_URL` as a middle fallback in the SBT launcher JAR
download URL chain: `DEFAULT_ARTIFACT_REPOSITORY` → `MAVEN_PROXY_URL` →
hardcoded Google Mirror default.

**Usage:**
```bash
export MAVEN_PROXY_URL=https://your-proxy.example.com/maven2
build/sbt compile
```

**Design notes:**
- The generated proxy config only includes `maven-proxy` and
`maven-proxy-ivy` resolvers, so the proxy URL should point to an
aggregating/virtual repository that mirrors all required upstreams
(Maven Central, Typesafe, sbt plugins, etc.).
- The temporary config file is written via `mktemp` and never modifies
the committed `sbt-config/repositories`.
- `DEFAULT_ARTIFACT_REPOSITORY` still takes precedence over
`MAVEN_PROXY_URL` for the launcher JAR download, maintaining backward
compatibility.

## How was this patch tested?

Manually tested with `MAVEN_PROXY_URL` set to an internal aggregating
mirror:
- Verified `build/sbt compile` succeeds with `MAVEN_PROXY_URL` set (all
resolution routed through proxy).
- Verified `build/sbt compile` works unchanged when `MAVEN_PROXY_URL` is
unset (falls back to `sbt-config/repositories`).
- Verified SBT launcher JAR download uses the proxy URL when set.

## Does this PR introduce _any_ user-facing changes?

No behavior change for existing users. This is a purely additive, opt-in
feature gated behind an environment variable that was previously unused
in this project.
…sharing client (delta-io#6492)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (sharing)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

allow delta sharing client to read variantShredding table feature.

## How was this patch tested?

added UT.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…6507)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (`.claude` tooling config)

## Description

Remove `.claude/scheduled_tasks.lock` which was accidentally committed.
This is a local IDE/tool lock file that should not be in the repository.

## How was this patch tested?

File removal only — no code changes.

## Does this PR introduce _any_ user-facing changes?

No.
…gacy checkpoint (delta-io#6506)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [X] Other (sharing)

## Description

- On streaming query restart, the Spark engine always primes the source
by calling `getBatch(start, end)` with the last two committed offsets
before calling `latestOffset`. When both offsets come from a legacy
`DeltaSharingSourceOffset` checkpoint, the file fetch must use MD5
`fileIdHash` to match the legacy source's file ordering.
- Refactored `maybeGetLatestFileChangesFromServer` to accept separate
`startConvertedFromLegacy` and `endConvertedFromLegacy` parameters along
with an optional `endOffset`. When `endConvertedFromLegacy` is true,
files are fetched from `startingOffset.reservoirVersion` to
`endOffset.reservoirVersion` using MD5.
- Updated the `latestOffset` call site to pass `endOffset = None` (no
end offset available yet).
- Updated the `getBatch` call site to pass `Some(endOffset)` with
`endConvertedFromLegacy` when the end offset was converted from legacy.
- Added detailed comments documenting the streaming query execution flow
(`getBatch -> latestOffset -> getBatch` on restart) for both
`Trigger.Once` and `Trigger.ProcessingTime`.

## How was this patch tested?

- Added new unit test fileIdHash: MD5 for priming getBatch when both
start and end are legacy — creates a synthetic checkpoint with two
committed legacy batches (both at version boundary), restarts the
stream, and verifies the first streaming getFiles call uses MD5
fileIdHash while subsequent calls use SHA256.
- Extracted writeLegacyOffsetAndCommit helper to reduce boilerplate
across legacy checkpoint tests.
- Updated existing test mocks to align with the new
endConvertedFromLegacy behavior in maybeGetLatestFileChangesFromServer,
which limits the priming getBatch fetch range to
endOffset.reservoirVersion:
- E2E: legacy checkpoint isStartingVersion=false: changed mock from (1,
3) to (1, 2) + (3, 3) since priming now fetches up to
endOffset.reservoirVersion=2.
- E2E: legacy checkpoint mid-version (unlucky case): added batch 0
offset, replaced snapshot mock with streaming mock (1, 2) for priming
getBatch, added (3, 3) for post-transition.
- fileIdHash: MD5 for mid-version legacy offset: added batch 0 offset,
replaced snapshot mock with streaming mock (1, 2) for priming getBatch.
- E2E: convertDeltaSourceOffsetToLegacyOffset: added batch 0 offset,
replaced snapshot mock with streaming mock (1, 2), updated batch
assertions from batch 1 to batch 2.

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…io#6505)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Previously, in ClusteredTableTestUtils, we had a flaky check for
clustering columns that could, under rare occasions, not see updates to
the Hive catalog. This change fixes that by moving up an await function
call.

## How was this patch tested?

Existing tests

## Does this PR introduce _any_ user-facing changes?

No

Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…lta-io#6478)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
This PR removes an old spark config that was
[introduced](delta-io@215996f)
as a potential rollback mechanism for implicit type casting. The
configuration has defaulted to true for ~2 years and removing it
simplifies some code by removing early returns.
## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
unit tests
## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Signed-off-by: Brooks Walls <brooks.walls@databricks.com>
Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…#6513)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The `get_spark_version_info.py` script handles `"master"` as a special
alias (maps to `isMaster=true`) but doesn't handle `"default"` (should
map to `isDefault=true`). This is inconsistent with
`CrossSparkVersions.scala` which handles both aliases at line 314:

```scala
val resolvedInput = input match {
  case "default" => SparkVersionSpec.DEFAULT.fullVersion
  case "master" => ...
}
```

When `run-tests.py` is invoked without `SPARK_VERSION` set, it passes
`"default"` to `get_local_package()`, which calls the script with
`--get-field default packageSuffix`. The script fails because
`"default"` doesn't match any `shortVersion` or `fullVersion`, causing a
fallback to an empty package suffix. After delta-io#5840 (always include Spark
suffix), this produces the wrong artifact name (`delta-spark_2.13`
instead of `delta-spark_4.1_2.13`).

**Fix:** Add `"default"` → `isDefault=true` matching, consistent with
how `"master"` → `isMaster=true` already works.

## How was this patch tested?

This is a 4-line fix to the version query script. The fix aligns the
Python script's alias handling with the Scala code that already handles
`"default"`.

## Does this PR introduce _any_ user-facing changes?

No. This is an internal build infrastructure fix.
#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (CI / Build Infrastructure)

## Description

The `dev/requirements.lock` file introduced in delta-io#6440 contains an
absolute local filesystem path in the `uv pip compile` header comment.
This replaces it with relative paths.

Before:
```
#    uv pip compile ... -o /Users/vishnu.c/.../dev/requirements.lock .../dev/requirements.txt
```

After:
```
#    uv pip compile ... -o dev/requirements.lock dev/requirements.txt
```

This is a follow-up to delta-io#6440.

## How was this patch tested?

Visual inspection. The lock file content (package versions and hashes)
is unchanged; only the auto-generated comment header is modified.

## Does this PR introduce _any_ user-facing changes?

No.
…hecksum (delta-io#6476)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Follow up for delta-io#3899. Rebases
delta-io#3907 over master
Major changes:
1. Makes it so that histograms of file sizes are tracked in the version
checksum --- these are later used for validation. These are computed
incrementally to avoid state reconstruction.
2. Also, now that Version Checksum can be incrementally computed without
triggering a full state reconstruction (see
delta-io#3899,
delta-io#3895), this PR enables writing
the version checksum by default.


## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
Added FileSizeHistogramSuite and updated some existing suites.


## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…tCoordinatorClient (delta-io#6486)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->


## Description
Add support on UCCommitCoordinatorClient for updating UniForm metadata.
This is part of project to support UniForm for UC managed table

## How was this patch tested?
```
build/sbt -DsparkVersion=4.0 "spark/testOnly *UCCommitCoordinatorClientSuite
```
…ta-io#6510)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

Upgrades the OSS Unity Catalog dependency from `0.4.0` to `0.4.1` in:
- `build.sbt` (used by Spark and Kernel unitycatalog modules)
- `examples/scala/build.sbt` (used by Scala examples)

## How was this patch tested?

Dependency version bump only. The unitycatalog integration tests cover
this dependency.

## Does this PR introduce _any_ user-facing changes?

No.

Signed-off-by: openinx <openinx@gmail.com>
…f CTAS test should check TableInfo (delta-io#6511)

#### Which Delta project/connector is this regarding?
- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Use translatesDeprecatedUcTableIdOnManagedCreate to control if CTAS test
should check TableInfo.

## How was this patch tested?
Changing existing integration test

## Does this PR introduce _any_ user-facing changes?
No

Signed-off-by: Yi Li <yi.li@databricks.com>
…a-io#6520)

## Summary
- Use `SparkSession.getActiveSession` instead of `SparkSession.active`
in `DeltaUDF.createUdfFromTemplateUnsafe` to prevent
`IllegalStateException` when the SparkContext is stopped (e.g. during
query cancellation or cluster shutdown).
- When no active session is available, falls through to `orElse` which
creates a fresh UDF safely.

## Test plan
- Existing unit tests for DeltaUDF should continue to pass.
- Verified that the change matches the fix in
databricks-eng/runtime#208197.
…torage credentials in plan trees and exceptions (delta-io#6383)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

`DeltaTableV2.toString` previously interpolated the full `CatalogTable`,
`SparkSession`, and raw `options` map. When a catalog (e.g. Unity
Catalog) injects Hadoop filesystem config into
`CatalogTable.storage.properties` at table-load time, stringifying
`CatalogTable` dumps `fs.*` keys — including auth tokens, access keys,
and internal URIs — into exception messages, EXPLAIN output, and logs.

The primary vector: Spark's `ExtendedAnalysisException` appends the
analyzed plan tree to `getMessage`. For unsupported operations like
`SHOW CREATE TABLE` on Delta, `TreeNode.argString` calls
`DeltaTableV2.toString` via `case other => other :: Nil` (since
`DeltaTableV2` is a V2 `Table`, not a direct `CatalogTable` product
field).

This PR keeps the original `toString` format with all six constructor
parameters, but filters `fs.*` keys from both
`CatalogTable.storage.properties` and the `options` map before
stringifying. It reuses the existing `HIDDEN_STORAGE_PROPERTY_PREFIXES`
constant from `DeltaTableV2.properties()` (introduced in PR delta-io#6300) as
the single source of truth for which key prefixes to suppress.

Complements PR delta-io#6300 which filters `fs.*` from
`DeltaTableV2.properties()` (the V2 DESCRIBE/SHOW TBLPROPERTIES path).
That fix does not cover `toString`.

## How was this patch tested?

New `DeltaTableV2Suite` with 11 tests:

- **Negative (2):** Asserts `toString` omits all `fs.*` storage keys and
values (6 key varieties); verifies `fs.*` keys are filtered while
non-`fs` keys (e.g. `delta.*`, `io.unitycatalog.*`) are preserved.
- **Positive (4):** Verifies `CatalogTable` metadata (identifier, type,
provider) is preserved; all six original parameters appear; path-only
and tableIdentifier-only variants work correctly.
- **Options filtering (2):** Verifies `fs.*` keys in the `options` map
are filtered; verifies simultaneous filtering of both `CatalogTable`
storage and `options`.
- **Edge case (1):** Empty storage properties.
- **Integration (2):** Uses `DummySessionCatalog` (injects `fs.myKey`
into storage properties). `EXPLAIN EXTENDED` test validates no plan node
leaks credentials. `SHOW CREATE TABLE` test intercepts the
`AnalysisException` and asserts both `getMessage` and `toString` omit
storage properties — directly exercising the `ExtendedAnalysisException`
→ plan tree → `DeltaTableV2.toString` chain.

All 11 tests pass (`./build/sbt "spark/testOnly
org.apache.spark.sql.delta.catalog.DeltaTableV2Suite"`).

## Does this PR introduce _any_ user-facing changes?

Yes. The string representation of `DeltaTableV2` in exception messages
and plan output no longer includes `fs.*` storage properties or `fs.*`
entries from the options map. The overall `toString` format is unchanged
— all six constructor parameters are still present. This is a security
improvement that prevents catalog-injected credentials from appearing in
error messages, EXPLAIN output, and logs.
… tables (delta-io#6356)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR adds a config to block Spark 4.0 clients from being able to
write to Delta tables containing the variant data type since Spark 4.0
does not support the Variant logical type annotation. This could lead to
weird inconsistencies where Spark 4.0 can perform pure inserts into
these tables but cannot perform any operation that requires reading
existing data that would have been written by newer clients that write
the variant logical type annotation.

## How was this patch tested?

UTs

## Does this PR introduce _any_ user-facing changes?

Yes

---------

Co-authored-by: seewishnew <vishnu.c@databricks.com>
## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6394/files) to
review incremental changes.
-
[**stack/refactorStartVersionNotFoundExcecption**](delta-io#6394)
[[Files changed](https://github.com/delta-io/delta/pull/6394/files)]
- [stack/failOnDataLoss2](delta-io#6395)
[[Files
changed](https://github.com/delta-io/delta/pull/6395/files/0c83e18a239a99178ec59db8b2b4634737fef7a6..c01f4d435ff5c73deae282e8b7d1d6913f55a434)]

---------
#### Which Delta project/connector is this regarding?

Kernel

## Description

Introduces a dedicated `StartVersionNotFoundException` exception class
that extends `KernelException`, replacing the plain `KernelException`
previously thrown by `DeltaErrors.startVersionNotFound()`.

The new exception exposes structured fields:
- `getTablePath()` — the table where the version was not found
- `getStartVersionRequested()` — the version that was requested but
missing
- `getEarliestAvailableVersion()` — the earliest version still available
(`Optional<Long>`)

This follows the same pattern as `CommitRangeNotFoundException` and
allows callers to programmatically handle missing start versions (e.g.,
for `failOnDataLoss=false` in DSv2 streaming) without fragile message
parsing.

Resolves the kernel-side changes for
delta-io#6380

## How was this patch tested?

Existing tests continue to pass since `StartVersionNotFoundException
extends KernelException`. Callers that previously caught
`KernelException` still catch the new subclass. The structured fields
are exercised by the DSv2 `failOnDataLoss` tests in the stacked PR
delta-io#6395.

## Does this PR introduce _any_ user-facing changes?

No
…elta-io#6324)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (protocol)

## Description

Add a proposal RFC to document parquet compression.

## How was this patch tested?

N/A

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Scott Sandre <scott.sandre@databricks.com>
…6518)

## Summary

- `UpdateMetricsHook` now reads the post-commit snapshot's file size
histogram from `checksumOpt` (the CRC) — this avoids triggering state
reconstruction, even for blind appends.
- When the CRC histogram is unavailable (e.g. legacy tables without
CRC), the remaining metrics are still sent with the
`file_size_histogram` field omitted (`None`).
- Removes duplicate `DEFAULT_BINS`/`emptyHistogram` from
`FileSizeHistogram` companion object — these are now provided by
`FileSizeHistogramUtils` (added in delta-io#6476).

Supersedes delta-io#6502 (now redundant — delta-io#6476 already exported
`FileSizeHistogramUtils`) and delta-io#6503 (which was stacked on delta-io#6502 and has
merge conflicts).

## Test plan

- [x] `build/sbt "spark/testOnly *UpdateMetricsHookSuite"` — 21/21 pass,
including:
  - Snapshot histogram is used when available
  - `file_size_histogram` omitted from JSON when CRC unavailable
- End-to-end: async hook does not trigger state reconstruction (asserts
`!stateReconstructionTriggered` after `awaitCompletion`)
- [x] `build/sbt "spark/testOnly *FileSizeHistogram*"` — 18/18 pass (no
regressions from companion object cleanup)
- [x] `build/sbt "spark/compile"` — clean compilation

This pull request was AI-assisted by Isaac.

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…eberg conversion (delta-io#6500)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Fix field assignment in schema for Delta UniForm in an edge case
(initial conversion with nested schema)

_Context_: Iceberg `CREATE_TABLE` reassigns the field id in schema,
which is not consistent with Delta in edge cases (For nested schemas).
As data files written in the txn will rely on it, the schema and
partitionSpec in the txn need to be overwritten to be consistent with
Delta again

## How was this patch tested?
Add UTs
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite"
```
…der (delta-io#6524)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR limits the flink tests to only run on changes to `flink/kernel`
folder.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Tested on github
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
… WKB with stats (delta-io#6471)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [X] Kernel
- [ ] Other (fill in here)

## Description

Adds Parquet read/write support for geometry/geography columns as part
of the geospatial table feature
([spec](delta-io#4726)).

Geometry and geography values are stored as WKB (Well-Known Binary) in
Parquet BINARY columns annotated with GEOMETRY/GEOGRAPHY logical types.
Rather than introducing new basic vector types in the kernel, the
Parquet reader uses `getBinary()` to fetch geospatial data -- it is
opaque binary from the reader's perspective.

Geospatial statistics (bounding boxes) are stored in a separate Parquet
metadata object (`GeospatialStatistics`, see [parquet-format
spec](https://github.com/apache/parquet-format/blob/master/Geospatial.md)),
but in Delta they are regular min/max column stats. The
`ParquetStatsReader` translates between the two representations: it
reads the Parquet bounding box and converts it to WKT POINT min/max
literals that fit into the standard Delta stats model.

### Changes
- **build.sbt**: Upgrade parquet-hadoop 1.12.3 -> 1.16.0 (needed for
geospatial logical types)
- **DefaultBinaryVector**: Allow `GeometryType`/`GeographyType` in
constructor and `getBinary()`
- **ParquetColumnReaders**: Route geometry/geography to
`BinaryColumnReader`
- **ParquetColumnWriters**: Route geometry/geography to `BinaryWriter`
- **ParquetSchemaUtils**: Map `GeometryType` -> BINARY with
`geometryType(srid)` annotation; `GeographyType` -> BINARY with
`geographyType(srid, algorithm)` annotation
- **ParquetStatsReader**: Read `GeospatialStatistics` bounding box from
Parquet metadata; convert to WKT POINT min/max literals via
`GeometryUtils.formatPointWKT()`

## How was this patch tested?

- `GeometryParquetReaderSuite`: WKB roundtrip with null and non-null
rows for both GeometryType and GeographyType; bounding box stats
verification for GeometryType
- `ParquetSchemaUtilsSuite`: GeometryType/GeographyType Parquet schema
conversion, algorithm mapping, and invalid algorithm rejection

## Does this PR introduce _any_ user-facing changes?

Adds new table feature

---------

Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.