Skip to content

[Kernel] Geospatial Parquet support: read/write geometry/geography as WKB with stats - #2

Closed
milastdbx wants to merge 100 commits into
geospatial-stats-parsingfrom
geospatial-parquet
Closed

milastdbx wants to merge 100 commits into
geospatial-stats-parsingfrom
geospatial-parquet

Conversation

@milastdbx

Copy link
Copy Markdown
Owner

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

In this PR we are adding geospatial table feature as per spec: delta-io#4726

This is PR 3 of 3 for geospatial data skipping support. It adds Parquet read/write support for geometry/geography columns (as WKB binary), including Parquet schema mapping with GEOMETRY/GEOGRAPHY logical type annotations, and reading geospatial bounding box statistics from Parquet file metadata.

Depends on: PR A - Geospatial stats parsing (will rebase to master after PR A merges)

Changes

  • build.sbt: Upgrade parquet-hadoop 1.12.3 -> 1.16.0 (needed for geospatial logical types)
  • DefaultBinaryVector: Allow GeometryType/GeographyType in constructor and getBinary()
  • ParquetColumnReaders: Route geometry/geography to BinaryColumnReader
  • ParquetColumnWriters: Route geometry/geography to BinaryWriter
  • ParquetSchemaUtils: Map GeometryType -> BINARY with geometryType(srid) annotation; GeographyType -> BINARY with geographyType(srid, algorithm) annotation
  • ParquetStatsReader: Read GeospatialStatistics bounding box from Parquet metadata; convert to WKT string literal via GeometryUtils.formatPointWKT()

PR dependency graph

master
  |
  +-- PR A: Stats Parsing + GeometryUtils
  |     |
  |     +-- PR B: Expressions + Predicate Translation
  |     |
  |     +-- PR C (this): Parquet / Data Layer

How was this patch tested?

  • GeometryParquetReaderSuite: 8 tests covering WKB roundtrip, null rows, all-null batch, getString throws for both GeometryType and GeographyType
  • ParquetSchemaUtilsSuite: 9 new tests covering GeometryType/GeographyType Parquet schema conversion, algorithm mapping, and invalid algorithm rejection

Does this PR introduce any user-facing changes?

Adds new table feature

PorridgeSwim and others added 30 commits March 10, 2026 11:47
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Before we support schema tracking log, we should disallow all
non-additive schema change in dsv2.
Column Mapping related code change is pure refactoring that for v2 to
reuse the utilities.
## How was this patch tested?
Unit test:
- New test case for drop column in a nested struct
- Modify existed tests for non-additive schema change to throws the same
error as v1

Integration test:
- drop column: should fail with non-additive schema change error
- drop column: should succeed with unsafe column mapping schema change
flag enabled
- rename column: should fail with non-additive schema change error
- rename column: should throw schema change error with unsafe flag
enabled
- type widening: should fail with non-additive schema change error when
enable schema tracking
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…ty (delta-io#6227)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6227/files) to
review incremental changes.
-
[**stack/type-nullability-e2e-test**](delta-io#6227)
[[Files changed](https://github.com/delta-io/delta/pull/6227/files)]
-
[stack/forbid-schema-evolution-with-staleDF-v2](delta-io#6228)
[[Files
changed](https://github.com/delta-io/delta/pull/6228/files/cafe746303c3e32bdcf5499c8e99fd6a59c033c4..94b77dc62d21e896e9f4fdafd3ff17fafddd606f)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Added schema evolution e2e tests for type widening and relax nullability
for both refresh dataframe and stale dataframe cases.

Refresh dataframe cases work for both v1 & v2 connector, while stale
dataframe cases only work for v1 connector.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
test-only PR
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…m 50000 to 100000 (delta-io#6240)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Double the default value of `DELTA_STREAMING_INITIAL_SNAPSHOT_MAX_FILES`
from 50,000 to 100,000 to better support larger tables for V2 streaming
initial snapshots out of the box.

## How was this patch tested?

Existing tests continue to pass since this is a default-value-only
change. Tests that explicitly set this config (e.g.,
`SparkMicroBatchStreamTest`) are unaffected.

## Does this PR introduce _any_ user-facing changes?

Yes. The default maximum number of files allowed in the initial snapshot
for V2 streaming is increased from 50,000 to 100,000. Users with tables
between 50k–100k files will no longer hit the limit without explicitly
raising the config. This change only affects the unreleased master
branch and the future release.
…ype widening and relaxing nullability (delta-io#6228)

## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)
to review incremental changes.
-
[stack/type-nullability-e2e-test](delta-io#6227)
[[Files changed](https://github.com/delta-io/delta/pull/6227/files)]
-
[**stack/forbid-schema-evolution-with-staleDF-v2**](delta-io#6228)
[[Files
changed](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Add schema validation during stream restart e2e tests for type widening
and relaxing nullability to block user from adopt these two schema
change without refresh the dataframe in v2 connector

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Test-only PR
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…a-io#5837)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Add the DataFrame basic test coverage for preventing future regression. 

## How was this patch tested?

Do the locally unit tests and remotely integration tests. 

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: openinx <openinx@gmail.com>
… repartition the table's columns (delta-io#6238)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
- Updated partition column change tests in `OptimisticTransactionSuite`
to use `checkError` instead of calling `ex.getErrorClass` directly
- Each `checkError` call now verifies the error condition
(`DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE`), SQLSTATE (`42P10`), and
named parameter values (`operation`, `oldPartitionColumns`,
`newPartitionColumns`)
- Added expected operation name to the parameterized blocked-operations
test cases to support per-case parameter verification
- Collapsed the `DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE` error
message from three lines into a single line in
`delta-error-classes.json`, `databricks-error-classes.json`, and
`delta-error-classes-data-classification.json` (the multi-line split was
not justified as the column lists render as single-line comma-separated
strings)


## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
No new tests added. Existing tests in `OptimisticTransactionSuite`
covering partition column change validation were updated to use
`checkError` as required by the error message review guidelines.

## Does this PR introduce _any_ user-facing changes?
N/A
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
delta-io#5775)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/5775/files) to
review incremental changes.
-
[**stack/dv_pr3_phase2_vectorized**](delta-io#5775)
[[Files changed](https://github.com/delta-io/delta/pull/5775/files)]
-
[stack/dv_pr4_phase3_file_splitting](delta-io#5776)
[[Files
changed](https://github.com/delta-io/delta/pull/5776/files/463011af2fae489f7f3fe8c39fe86acbba06277b..0e534ed9c0257735523532c8c6f8e15ea33ae114)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Enable vectorized (columnar batch) reading for DV tables in the Spark V2
connector.

### Changes:
- `ColumnVectorWithFilter`: Wraps `ColumnVector` with row ID mapping for
zero-copy filtering (follows Iceberg pattern)
- `DeletionVectorReadFunction`: Extended to handle `ColumnarBatch` in
addition to `InternalRow`
- Enable vectorized reader for DV tables (remove `!isTableSupportDv`
condition)

### How it works:
1. For `ColumnarBatch`: Build live row mapping, wrap columns with
`ColumnVectorWithFilter`
2. For `InternalRow`: Filter deleted rows, project using
`ProjectingInternalRow`

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

- `ColumnVectorWithFilterTest`: Unit tests for various data types and
filtering scenarios
- `DeletionVectorVectorizedReaderTest`: Integration tests verifying
vectorized path is used
- All DV golden table tests pass with vectorized reading
## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…a-io#6104)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Spark's StringStartsWith(attribute, value) filter was not converted to a
Kernel predicate, meaning WHERE col LIKE 'prefix%' queries had no filter
pushdown in the V2 connector. The Kernel's STARTS_WITH predicate maps
directly: STARTS_WITH(column, string_literal).

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…sts (delta-io#6210)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Add the DataFrame streaming integration tests to cover the spark + uc
case.

## How was this patch tested?

Run in both locally UC server and remote UC server.

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: openinx <openinx@gmail.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [X] Other (sharing)

## Description

Upgrade delta sharing client to 1.3.10 

## How was this patch tested?

CI pipeline

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR blocks writes with NullType in UDT columns, similar to how we
block NullType in arrays/maps. NullType is typically used as a temporary
placeholder, which is later evolved into some other type. However, UDTs
cannot evolve over time, hence NullType inside UDT remains as NullType
forever. Additionally, our old check missed arrays/maps inside UDT and
didn't check whether they contain NullType or not. When trying to write
such Parquet files, the Parquet writer failed. This PR fixes these
problems.

## How was this patch tested?

New unit tests.

## Does this PR introduce _any_ user-facing changes?

No.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR adds support to excludeRegex read option in deltaDSv2
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
Unit tests:
Create table partitioned by category with 4 versions of data.
```
Version 0: CREATE TABLE (empty)
Version 1: ids {1,2} in category=alpha
Version 2: ids {3,4} in category=beta
Version 3: ids {5,6} in category=alpha
Version 4: ids {7,8} in category=gamma
```
Test cases: "No excludeRegex - all versions", "Exclude alpha category",
"Exclude beta category", "Exclude gamma category", "Exclude alpha and
beta categories", "Regex matching no files", "Regex matching all files",
"Exclude alpha versions 2 to 4", "Exclude beta version 2 only", "Initial
snapshot - no excludeRegex", "Initial snapshot - exclude alpha".

Enable integration test: "excludeRegex works and doesn't mess up offsets
across restarts - parquet version"
## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6204)

#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Send table schema to UC server when creating managed tables

## How was this patch tested?
Integration test

## Does this PR introduce _any_ user-facing changes?
No.

---------

Signed-off-by: Yi Li <yi.li@databricks.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] General repository

## Description

Adds a note regarding the required Java version into the README.md.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

Inapplicable.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

No.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

---------

Co-authored-by: Leonid Lygin <leonid.lygin@databricks.com>
Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Iceberg)

## Description

We have the following NPE:

```
java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:209)
	at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:296)
	at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:144)
	at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:129)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:138)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
```

Referring to this line:


https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/core/src/main/java/org/apache/iceberg/SchemaParser.java#L296

This comes from the following line:


https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L144

The field `schema` is only set in V1 fields. I believe the reference
implementation (Java OSS) also sets this field for V2/V3 to maximize
compatibility, but this might be different for other writers.

By passing in the spec, we don't execute the branch and we avoid the
NPE:


https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L126-L130


<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6203)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

Convert CatalogStatistics to V2 connector Statistics and thread them
through SparkTable → SparkScanBuilder → SparkScan.estimateStatistics().
Gate numRows/columnStats on cboEnabled || planStatsEnabled to match V1
LogicalRelation.computeStats() behavior.

Use per-column avgLen from catalog stats in size estimation instead of
defaultSize(), mirroring EstimationUtils.getSizePerRow() with UTF8String
overhead for StringType columns.

Resolves delta-io#5952


## How was this patch tested?

Unit test

## Does this PR introduce _any_ user-facing changes?

No
…a-io#5888)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
This PR adds a new Spark flag that will make all code paths in
DataSkippingReader to collect stats.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
New tests.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No.
…iveCatalog for Delta UniForm (delta-io#6189)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Use UnityCatalog as Iceberg committing catalog instead of HiveCatalog
for Delta UniForm


## How was this patch tested?
Add UTs
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite"
```
…flictSuiteBase (delta-io#6277)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Add an overridable `sqlWithTotalRpcBound` method to
`IdentityColumnConflictSuiteBase` that defaults to calling `sql()`. This
is a no-op in OSS - it allows internal subclasses to override the method
with custom assertions on the winning transaction's SQL execution path.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

No behavioral change - the new method defaults to `sql()`, identical to
the existing behavior. All existing `IdentityColumnConflictSuite` tests
pass unchanged.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No.

Co-authored-by: Shlok Jhawar <shlok.jhawar@databricks.com>
…o#6271)

## Problem
In `buildTransactionInternal()`, when `!needsMetadataOrProtocolUpdate`,
a new `TransactionImpl` is constructed but never returned. Execution
falls through to the full metadata/protocol update pipeline, doing
unnecessary work and constructing a second `TransactionImpl`. Fixes
delta-io#4713.

## Fix
Add `return` before `new TransactionImpl(` so the early-exit path
actually returns.

The early return skips `TransactionMetadataFactory`, which also runs
`validateRowTrackingConfigsNotMissing` on existing tables. To preserve
this validation, it is moved into `validateWriteToExistingTable()` so it
runs on every write regardless of whether metadata/protocol updates are
needed.

## Validation
- Confirmed by reading lines 230-247.
- New unit test verifies the early-return path (uses a fake
`delta.feature.*` property so the factory throws if the early return is
bypassed).
- Verified `LegacyRowTrackingSuite` passes (45/45 tests), including the
row tracking config validation test.

## Regression Prevention
- `TransactionBuilderImplSuite`: tests the no-update transaction path.
- Existing `LegacyRowTrackingSuite`: covers the row tracking validation
that was moved.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…lta-io#6278)

## Problem
Spark's CheckDeletionVectorDisabled checks both newestProtocol and
prevSnapshot to guard against concurrent writers enabling DVs. The
Kernel's CHECK_HAS_NO_DELETION_VECTORS only checks newProtocol, missing
the concurrent writer guard.

## Fix
Add the previous protocol check to the Kernel's DV validation, matching
the Spark implementation's defense-in-depth approach.

## Validation
- Confirmed by comparing Spark IcebergCompat.scala with Kernel
IcebergCompatMetadataValidatorAndUpdater.java.
- New test verifies the check catches DV enablement from a previous
protocol.

## Regression Prevention
New test case explicitly validates the previous-protocol DV check.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…n in dsv2 (delta-io#6246)

## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)
to review incremental changes.
- [stack/ignoreDeletesV2](delta-io#6245)
[[Files changed](https://github.com/delta-io/delta/pull/6245/files)]
-
[**stack/skipChangeCommitsV2**](delta-io#6246)
[[Files
changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)]
- [stack/ignoreChangesV2](delta-io#6249)
[[Files
changed](https://github.com/delta-io/delta/pull/6249/files/c1c942553a20800e318a5d2d384f869a3c0c5d3c..7209f24b489535e57a8396d712f0f1da577520d2)]
-
[stack/ignoreFileDeletionV2](delta-io#6250)
[[Files
changed](https://github.com/delta-io/delta/pull/6250/files/7209f24b489535e57a8396d712f0f1da577520d2..8690f78ca1bae137706f46ed1e05cdb05702ee6b)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Support 
- skipChangeCommits read option is DSv2, which skip a commit that
contains remove file actions regardless of if there is an add file
action or not.
- ignoreDeletes read option is DSv2, which skip a commit that contains
only remove file actions but no add file action.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Unit tests that test parity between v1 and v2 connector on both pure
deletes commit (only remove) and change commit (add + remove).

Integration tests:
- streaming with skipChangeCommits = true skips both delete and change
commits
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6267)

## Problem
`generateIcebergCompatWriterV3RemoveAction()` accepts `baseRowId` and
`defaultRowCommitVersion` parameters but silently discards them,
hardcoding `Optional.empty()` instead. This breaks row tracking for V3
writer compat remove actions.

## Fix
Forward the caller-provided `baseRowId` and `defaultRowCommitVersion` to
`convertRemoveDataFileStatus()` instead of passing `Optional.empty()`.

## Validation
- Confirmed by reading the method: params accepted at lines 266-274,
discarded at lines 308-310.
- New test verifies non-empty values are preserved in the output Row.

## Regression Prevention
New test case explicitly checks that baseRowId and
defaultRowCommitVersion are forwarded through to the RemoveFile row.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

During data-skipping, we add one millisecond to timestamp max stats to
compensate for the fact that JSON serialization will truncate them to
milliseconds. This has the consequence that if the data contains a
timestamp within the range (Long.MaxValue - 1000, Long.MaxValue], then
the addition will throw an overflow exception.

This PR fixes this by adding a check for if the stats is within that
range, then we saturate the stats to Long.MaxValue instead of adding
1000 to it.

## How was this patch tested?
Unit test

## Does this PR introduce _any_ user-facing changes?

No
…io#6268)

## Problem
`IcebergRESTCatalogPlanningClient` implements `AutoCloseable` but the
`ServerSidePlanningClient` trait does not extend it, and
`ServerSidePlannedTable` never calls `close()`. This leaks HTTP client
connections.

## Fix
Add proper lifecycle management so the HTTP client is closed when no
longer needed.

## Validation
- Confirmed by reading the trait hierarchy and table construction code.
- New test verifies the client is properly closed.

## Regression Prevention
New test verifies client lifecycle management.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Deflake PartitionLikeDataSkippingSuite by making result comparison
insensitive to nondeterministic record ordering.

## How was this patch tested?
See test change.

## Does this PR introduce _any_ user-facing changes?
No.
#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Block metadata changes on UC-managed tables
Without this, metadata mutations on a managed table succeed locally but
leave the UC catalog unaware, causing metadata drift.

Two kill switches in OptimisticTransaction:
- updateMetadata(): blocks schema, partition, description, and
configuration changes. Covers INSERT with autoMerge, MERGE with schema
evolution, and DataFrame writes with mergeSchema.
- commitLarge(): blocks clustering DomainMetadata changes for RESTORE
TABLE, which bypasses updateMetadata(). ALTER TABLE is already blocked
by UCSingleCatalog before reaching Delta.

## How was this patch tested?
New tests added.

## Does this PR introduce _any_ user-facing changes?
Yes. Table schema evolution with DML is blocked for UC managed tables.

---------

Signed-off-by: Yi Li <yi.li@databricks.com>
…in CRC file (delta-io#6282)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Added a conf to make delta-spark write fileSizeHistogram (and not
histogramOpt) as field name in CRC file.


## How was this patch tested?

UTs

## Does this PR introduce _any_ user-facing changes?

No
…elta-io#6281)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

This PR updates Kernel's `CRCInfo` to **read** both `fileSizeHistogram`
and `histogramOpt` fields to ensure backward compatibility with
Delta-Spark.

While the [Delta
spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#version-checksum-file-schema)
officially uses `fileSizeHistogram` in the Version Checksum file,
Delta-Spark has historically used histogramOpt
([code](https://github.com/delta-io/delta/blob/5ba861fa6d6999d7079706a307a7e293b59d4880/spark/src/main/scala/org/apache/spark/sql/delta/Checksum.scala#L87)).
Following [a recent
fix](delta-io@021755f)
that allows Delta-Spark to **read** both fields, this PR mirrors that
compatibility in Kernel. This ensures smooth read operations as we fully
transition Delta-Spark to write the correct `fileSizeHistogram` field.


## How was this patch tested?
New unit tests.
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
murali-db and others added 29 commits March 30, 2026 20:08
…lta-io#6155)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6155/files) to
review all changes.

**Stack:**
- **[[UC Commit Metrics] Add skeleton transport wiring and smoke
tests](delta-io#6155 [[Files
changed](https://github.com/delta-io/delta/pull/6155/files)] ⬅️ _This
PR_
- [[UC Commit Metrics] Add full payload construction and schema
tests](delta-io#6156) [[Files
changed](https://github.com/delta-io/delta/pull/6156/files/4b5eadb28ab152d9ac180423bfd51720b6219b5b..1f39302986a7b6138cd358f9515386fc283810a8)]
- [[UC Commit Metrics] Add feature flag and async
dispatch](delta-io#6333) [[Files
changed](https://github.com/delta-io/delta/pull/6333/files/1f39302986a7b6138cd358f9515386fc283810a8..7b6d633980419060404203551bb1577ccfe922aa)]

---

## Summary
- Self-contained post-commit hook in
`hooks/metrics/UpdateMetricsHook.scala` (case class + companion object
with payload builder, HTTP client, and constants).
- Registered only for UC-managed tables
(`CatalogTableUtils.isUnityCatalogManagedTable` check at registration
time).
- Derives endpoint from `spark.sql.catalog.<name>.uri`, authenticates
via `UCCommitCoordinatorBuilder` + `TokenProvider`.
- Best-effort: failures logged as warnings, never fail the commit.

## Test plan
- [x] `build/sbt "spark/testOnly *metrics.UpdateMetricsHookSuite"` -- 6
tests pass

---------

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: foo.bar <foo.bar@foobar.com>
Co-authored-by: foo.bar <foo.bar@foobar.com>
…ta-io#6377)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6377/files) to
review incremental changes.
-
[**stack/ddl-1-foundations**](delta-io#6377)
[[Files changed](https://github.com/delta-io/delta/pull/6377/files)]
- [stack/ddl-2-builder](delta-io#6378)
[[Files
changed](https://github.com/delta-io/delta/pull/6378/files/b8c478ac19da5afb5aea0d8ce25dfa744b689e4f..a652586290caadbe79201d594ecf0e95f129b328)]
- [stack/ddl-2b-builder](delta-io#6442)
[[Files
changed](https://github.com/delta-io/delta/pull/6442/files/a652586290caadbe79201d594ecf0e95f129b328..88c17741e4c8f17a56c4a0523c1b744c36789c95)]
- [stack/ddl-3-wiring](delta-io#6379)
[[Files
changed](https://github.com/delta-io/delta/pull/6379/files/88c17741e4c8f17a56c4a0523c1b744c36789c95..113ebc006446bd7645be6a75c900104b0b0ed8a4)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

## Description
In this stack of PRs we are implementing the foundation for DDL
operations (metadata-only CREATE, CTAS, RTAS).

For context, CREATE TABLE in this DSv2 world is 3 main steps:
1. pre-register the table to the catalog (UCProxy.createTable or
HMS.createTable)
 2. build a transaction for table creation (Delta commit 0)
 3. commit (actually commit the transaction above)

**We also want to make implementing future DDL operations easier in the
future**. That's why in this PR we create a unified DDL request POJO.
This POJO/request will be fed into the transaction builder.

**Significance**: CreateTableBuilder.prepare() does all the Spark →
Kernel conversion - DDLContext is a normalized request for downstream
Kernel operations.
> By the time DDLContext is constructed, everything is Kernel-native.
buildTransaction() just passes it straight through to TableManager or
UCCatalogManagedClient with no further conversion.
```
                                  SQL: CREATE TABLE t (id INT, name STRING)
                                                       │
                                                       ▼
                                 ┌──────────────────────────────────────────┐
                                 │  UCSingleCatalog.createTable(            │
                                 │    ident, schema, partitions, props)     │
                                 │                                          │
                                 │  Spark's UC catalog plugin. Delegates    │
                                 │  to DeltaCatalog for Delta tables.       │
                                 └─────────────────────┬────────────────────┘
                                                       │
                                                       ▼
  ┌──────────────────────────────────────────────────────────────────────────────────────────┐
  │  DeltaCatalog.createTable(ident, schema, partitions, properties)                         │
  │                                                                                          │
  │  1. DeltaV2Mode routing check                                                            │
  │     STRICT → always Kernel path                                                          │
  │     AUTO   → Kernel only if UC + catalogManaged feature flag                             │
  │     NONE   → V1 path (super.createTable)                                                 │
  │                                                                                          │
  │  2. If UC: preRegisterIfNeeded(ident, schema, parts, props)                              │
  │     └→ Optional<CatalogTable>                                                            │
  │     then injectStorageCredentials(ct, hadoopConf)                                        │
  │     then UCUtils.extractTableInfo(ct, spark) → Optional<UCTableInfo>                     │
  │                                                                                          │
  │  3. CreateTableBuilder.prepare(ident, schema, parts, props, hadoopConf, ucTableInfo)     │
  │     └→ DDLContext                                                                        │
  │                                                                                          │
  │  4. CreateTableBuilder.buildTransaction(ddlContext) → Transaction                        │
  │                                                                                          │
  │  5. txn.commit(engine, emptyIterable()) — metadata-only                                  │
  │                                                                                          │
  │  6. return new SparkTable(ident, tablePath) — V2 connector                               │
  └──────┬─────────────────────┬──────────────────────────┬──────────────────────┬───────────┘
         │                     │                          │                      │
         ▼                     ▼                          ▼                      ▼

       STEP 2                STEP 3                     STEP 4                 STEP 5
       Pre-register          Prepare                    Build Txn              Commit

       UC-only.              Converts Spark             Routes to Kernel       txn.commit(
       Calls delegate        types to Kernel:            API based on            engine,
       .createTable()                                    ucTableInfo:            emptyIterable()
       via UCProxy.          sparkSchema →                                       )
                              kernelSchema              ┌── UC present? ───┐
       UC assigns:            (SchemaUtils)             │                  │   Kernel handles
       • tableId                                        │  YES             │   conflict
       • managed             Transform[] →              │  UCCatalog       │   resolution,
         location             DataLayoutSpec            │  ManagedClient   │   protocol,
       • ucUri                (identity or              │  .buildCreate    │    and log write.
       • authConfig           cluster-by)               │  TableTxn(       │
                                                        │    tableId,      │   For UC: commit
       Returns               properties →               │    path,         │   goes through
       Optional               filtered props            │    schema,       │   UC committer.
       <CatalogTable>         (strip DSv2 keys:         │    engineInfo)   │
                               location, provider,      │  + UC committer  │   For path-based:
       Non-UC: returns         comment, owner,          │  + UC properties │   writes directly
       Optional.empty()        external, path,          │                  │   to filesystem.
                               option.path)             │  NO              │
       On failure:                                      │  TableManager    │   Returns
       drops pre-            hadoopConf →               │  .buildCreate    │    TransactionCommit
       registered             Engine                    │  TableTxn(       │   Result with:
       catalog entry          (DefaultEngine)           │    path,         │   • version (0)
                                                        │    schema,       │   • postCommit
                             path resolved:             │    engineInfo)   │     Snapshot
                              UC → ucTableInfo          └───────┬──────────┘
                                  .tablePath
                              non-UC → props            Then applies:
                                  ["location"]          • .withTableProperties(filtered)
                                  (throws if            • .withDataLayoutSpec(layout)
                                  none found)           • .build(engine) → Transaction

                             Returns:
                             DDLContext POJO
                             (ident, tablePath,
                              kernelSchema,
                              filteredProps,
                              comment,
                              dataLayoutSpec,
                              engine,
                              ucTableInfo,
                              engineInfo)
```

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…6438)

## Description

Add `DeltaV2TableManager` trait to `v2/interop/` that abstracts
`DeltaLog`'s role as the entry point for Delta table operations.
`DeltaLog` extends this trait.

This is the foundation for migrating Delta protocol reading from
`DeltaLog` to Kernel. The trait defines the minimal surface that
commands and `OptimisticTransaction` need from the table manager:

- Table identity: `logPath`, `dataPath`, `clock`, `options`, `tableId`
- Configuration: `newDeltaHadoopConf()`
- Snapshot access: `snapshot`, `unsafeVolatileSnapshot`, `tableExists`
- Setup: `createLogDirectoriesIfNotExists()`

With this trait in place, a future `KernelTableManager` can implement
the same interface via Kernel instead of log replay, and all existing
business logic (commands, `OptimisticTransaction`) works with either
backend.

**This PR has zero behavior change** — `DeltaLog` already implements
every method defined in the trait, so `with DeltaV2TableManager` is a
no-op.

## How was this patch tested?

All existing `DeltaLog` tests exercise the new trait. No test changes
needed.

## Does this PR introduce _any_ user-facing changes?

No.
… test (delta-io#6348)

## Description

Fixes a flaky test in `DeltaRetentionWithCatalogOwnedBatch1Suite` that
fails intermittently with:

```
20 did not equal 0 Delta files before the last checkpoint version should have been deleted
(DeltaRetentionSuite.scala:152)
```

### Root Cause

The test `"log files being already deleted shouldn't fail log deletion
job"` fails **only when executed between 00:00-01:00 UTC**. The
mechanism:

1. `getStartTimeForRetentionTest()` returns today at 01:00 UTC — which
is a **future** timestamp if the test runs before 01:00 UTC.
2. Delta files get `setLastModified` to this future time (01:00:xx UTC).
3. V2 checkpoint files (introduced by CatalogOwned in delta-io#5841) use
UUID-based names (`*.checkpoint.<uuid>.json`) that don't match
`checkpointFileSingular` format (`*.checkpoint.parquet`), so the test
never sets their timestamps — they keep their real wallclock time
(~00:04 UTC).
4. `BufferingLogDeletionIterator` checks `needsTimeAdjustment()`
**before** `isCheckpointFile()`. When a V2 checkpoint (modTime=00:04)
follows a delta file (modTime=01:01), the checkpoint's earlier real
timestamp triggers time adjustment, causing it to be treated as a
regular file instead of a checkpoint boundary. The buffer is never
flushed, and zero files get deleted.

### Fix

Replace the `checkpointFileSingular` lookup (lines 133-136) with the
`getCheckpointFiles` + `checkpointVersion` pattern already used and
working correctly elsewhere in the same file (lines 724-734). This finds
all checkpoint file formats including V2.

### Verified

Reproduced the failure by forcing `getStartTimeForRetentionTest` to
return a future timestamp:
- **Without fix**: `FAILED` — `20 did not equal 0` (identical to CI
failures)
- **With fix**: `PASSED`

Confirmed all CI failures occurred in the 00:00-01:00 UTC window:
00:04:33, 00:13:10, 00:21:48, 00:04:47 UTC.

## How was this patch tested?

Existing test `DeltaRetentionWithCatalogOwnedBatch1Suite."log files
being already deleted shouldn't fail log deletion job"` — this PR fixes
it.

## Does this PR introduce _any_ user-facing changes?

No. Test-only change.

This pull request was AI-assisted by Isaac.
…-io#6443)

Extract the file name construction logic from
DelayedCommitProtocol.getFileName into a shared companion object method
DelayedCommitProtocol.buildFileName.

This enables reuse of the standard Delta file naming convention
(part-NNNNN-uuid.ext / cdc-NNNNN-uuid.ext) by both the V1 write path
(DelayedCommitProtocol) and the DSv2 write path (DataWriter), ensuring
consistent file naming across both code paths.

Changes:
- Move file name building logic to DelayedCommitProtocol.buildFileName
companion method
- Simplify getFileName to delegate to the new shared method
- Preserve all existing behavior (test prefix, CDC prefix, split
formatting)
…leanup race (delta-io#6371)

## Summary

Fixes a flaky test failure in
`V2StreamingReadTest.testStreamingQueryStopDoesNotSurfaceException` that
occurs intermittently in CI.

### Error

```
org.junit.platform.commons.JUnitException: Failed to close extension context, took 6.353s
Caused by: java.io.IOException: Failed to delete temp directory
```

### Root Cause

The static `DeltaLog` cache holds references to snapshot files in
`@TempDir` directories. When a streaming query stops, these cached
instances keep file handles open. JUnit's `@TempDir` cleanup then races
against these open handles during extension context closure.

Under CI resource pressure (**4 parallel JVMs on 2 vCPUs**), this race
is wide enough for the recursive deletion to block and exceed JUnit's
~5s extension context close timeout, producing a flaky `JUnitException`.

### Fix

Call `DeltaLog.clearCache()` in `V2TestBase` at two points:
- **`tearDown()`**: Before `spark.stop()`, release cached DeltaLog
instances so the class-level `@TempDir` can be cleaned up.
- **`processStreamingQuery()`**: After `query.stop()`, release cached
instances so the method-level `@TempDir` can be cleaned up.

This is the same approach already used in
`testStreamingQueryStopDoesNotSurfaceException` itself (which calls
`DeltaLog.clearCache()` at line 299), but applied to the shared test
infrastructure so all test methods benefit.

### Reproduction

The flakiness was **reproduced locally** by simulating CI conditions:

```bash
# Unfixed: 1 out of 4 instances FAILS
taskset -c 0-1 ./build/sbt "sparkV2/testOnly ...V2StreamingReadTest" &  # x4 parallel
# → StreamingQueryException: Thread was interrupted

# Fixed: all instances PASS across 3 rounds of 2-parallel runs
taskset -c 0-1 ./build/sbt "sparkV2/testOnly ...V2StreamingReadTest" &  # x2 parallel, 3 rounds
# → Passed: Total 8, Failed 0 (every run)
```

### CI Evidence

- **Failed run**:
https://github.com/delta-io/delta/actions/runs/23449372021 (attempt 1,
Shard 1)
- **Passed re-run**: Same run, attempt 2 — zero code changes, same SHA
`f864a12c`

## Test plan

- [x] `sparkV2/testOnly ...V2StreamingReadTest` — all 8 tests pass
- [x] Reproduction under CI-like conditions (parallel JVMs, 2 cores):
unfixed fails, fixed passes
- [x] No changes to production code — test infrastructure only

This pull request was AI-assisted by Isaac.
#### Which Delta project/connector is this regarding?
- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Change master version to 4.2.0-SNAPSHOT

## How was this patch tested?
CI.

## Does this PR introduce _any_ user-facing changes?
No

Signed-off-by: Yi Li <yi.li@databricks.com>
)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The Delta protocol specifies that `offset` is optional for UUID and
path-based Deletion Vector descriptors. Early implementations stored one
DV per file (no offset needed); later implementations combined multiple
DVs into a single file for performance, making offset necessary in
practice. The protocol only specifies how to read DVs — a missing offset
means read from offset 0.

The factory methods (`onDiskWithRelativePath`, `onDiskWithAbsolutePath`)
and downstream usage sites already treat `offset` as optional
(defaulting to `None` / `getOrElse(0)`). However, the
`serializeToBase64` helper introduced in PR #98496 asserts that
`offset.isDefined` for non-inline DVs, which contradicts the protocol
and is inconsistent with the rest of the codebase.

This PR:
- Removes the assertion in `serializeToBase64()` that required offset to
be present for non-inline DVs
- Uses uppercase storage type markers (`U`, `P`) in the base64 format to
signal that no offset bytes follow, preserving the `None` vs `Some(0)`
distinction through round-trips
- Updates `deserializeFromBase64()` to decode the new markers back to
the canonical lowercase storage type with `offset = None`

## How was this patch tested?

New base64 round-trip tests for inline DVs and on-disk DVs with both
`offset=None` and `offset=Some(25)`, plus existing UTs.

## Does this PR introduce _any_ user-facing changes?

No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR provides implementation to `DeltaTable` and `DeltaCatalog`.
We add two new classes: `UnityCatalog` and `CatalogManagedTable`
`UnityCatalog` implements `DeltaCatalog` and supports
* getTable
* listTable
* createTable
* credential vending

`CatalogManagedTable` implements `DeltaTable` and supports
* commit to a catalog support catalog-managed Delta tables

For completeness, we also add `HadoopCatalog` and `HadoopTable` that
supports path-based Delta table access with static credentials.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UT/Int
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
… transitions (delta-io#6439)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Sharing)

## Description

This PR adds the back-conversion path for the "unlucky case" where a
legacy checkpoint is mid-version:

- Add convertDeltaSourceOffsetToLegacyOffset helper that converts a
DeltaSourceOffset back to a legacy DeltaSharingSourceOffset. This is
needed when the stream restarts from a legacy checkpoint mid-version
(index != -1) and cannot finish the current version in a single batch.
- In latestOffset, after computing the end offset, check whether both
start and end offsets are still mid-version during a legacy transition.
If so, convert the end offset back to legacy format so the next
micro-batch continues using the legacy file-id hash (MD5). Once the
stream reaches a version boundary, the conversion stops and the stream
fully transitions to DeltaSourceOffset.

## How was this patch tested?

New test case

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
…a write codes (delta-io#6386)

#### Which Delta project/connector is this regarding?

  - [x] Spark
  - [ ] Standalone
  - [ ] Flink
  - [ ] Kernel
  - [ ] Other (fill in here)

  ## Description

We have this pattern of create a table relation from the current txn, in
order to create a `LogicalPlan` and execute that `LogicalPlan`.

We have an incoming PR that will add additional logic to this pattern,
so best to refactor them.

  ## How was this patch tested?

  Existing UTs, just a refactoring.

  ## Does this PR introduce _any_ user-facing changes?

  No.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Add implementation of Flink Writer and Committers
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UT
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…-io#6315)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Support converting uncommitted txn for Delta UniForm. This is part of
external Delta UniForm write support.

## How was this patch tested?
Add UTs
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite"
```
…BigDecimal (delta-io#6259)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

`Literal.ofDecimal(BigDecimal value, int precision, int scale)` creates
a `DecimalType(precision, scale)` directly from the caller-provided
arguments. However, Java's `BigDecimal.precision()` returns the count of
significant digits in the unscaled value, not the SQL precision. This
can be less than `scale()` -- for example, `BigDecimal.valueOf(0, 18)`
has `precision()=1` and `scale()=18`.

When a caller passes `bd.precision()` as the precision argument (as
Spark's `ExpressionUtils.convertValueToKernelLiteral` does), `ofDecimal`
creates `DecimalType(1, 18)` which violates the Kernel constraint `scale
<= precision` and throws `IllegalArgumentException: Invalid precision
and scale combo`.

This surfaces during data-skipping in Spark's DSv2 path, where predicate
literals may have a small unscaled value with a large scale after
Spark's type widening.

Changes:
- `ofDecimal()`: Adjust precision upward to `max(precision, scale)`
before constructing the `DecimalType`, so the invariant `precision >=
scale` always holds.

Related: delta-io#6257 (fixes a different Decimal bug in the expression
evaluator where casts between Decimal types with different precisions
were unsupported)

## How was this patch tested?

New unit tests in `ExpressionsSuite`:
- `ofDecimal: adjusts precision when scale exceeds caller-provided
precision` -- the core bug repro (`BigDecimal.valueOf(0, 18)`)
- `ofDecimal: normal case with precision >= scale is unchanged` -- no
regression for normal inputs
- `ofDecimal: rejects value that exceeds adjusted precision` --
validation still fires when the value genuinely overflows

## Does this PR introduce _any_ user-facing changes?

No.

---------

Co-authored-by: rapoth <rapoth@users.noreply.github.com>
…s and validation (delta-io#6445)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->
 - [x] Spark
 - [ ] Standalone
 - [ ] Flink
 - [ ] Kernel
 - [ ] Other (fill in here)

## Description

`replaceOn` and `replaceUsing` are new Selective Overwrite options for
Delta tables, complementing the existing `replaceWhere`:

- **`replaceOn`**: Accepts a matching condition (e.g.
`.option("replaceOn", "t.id = s.id")`) to determine which existing rows
to replace with incoming data.
- **`replaceUsing`**: Accepts a comma-separated list of columns (e.g.
`.option("replaceUsing", "id")`) to match existing rows by equality on
those columns.

Both provide row-level replacement semantics — matched rows are deleted
and replaced by the new data, while unmatched existing rows are
preserved.

This PR introduces the option definitions, feature gate confs, and
validation logic that prevents invalid option combinations. The actual
execution paths will follow in a subsequent PR.

## How was this patch tested?

- `DeltaInsertReplaceOnOrUsingDFOptionCombinationSuite` — tests all
invalid option combinations with both `replaceOn` and `replaceUsing`
- `DeltaInsertReplaceOnOrUsingDFv2CreateReplaceBlockingSuite` — tests
DFv2 `create()`/`replace()`/`createOrReplace()` blocking
- `DeltaSuite` — tests `replaceOn`/`replaceUsing` blocked when feature
confs are disabled

## Does this PR introduce any user-facing changes?

Yes. Two new DataFrame writer options `replaceOn` and `replaceUsing` are
now recognized by Delta, along with their feature gate confs. When the
confs are disabled (default: enabled), using these options will throw
`DELTA_OPERATION_NOT_ALLOWED`. When enabled, invalid combinations with
other options (e.g. `overwriteSchema`, `dataChange=false`) will throw
descriptive errors. The actual replace execution is not yet implemented
in this PR.
… eviction (delta-io#6334)

## What changes are proposed in this pull request?

CI builds on PRs may be compiling dependencies from scratch even though
`build.sbt`
rarely changes and `actions/cache` should be providing warm caches.

**Root cause:** GitHub Actions caches are scoped by git ref. A PR's
cache can only be read
by that same PR, never by sibling PRs. Every PR writes cache entries
(across 5 active
workflows with 20+ matrix jobs) that no other PR can use. This fills the
10GB cache budget,
LRU-evicts master's shared entries, and forces every PR to recompile
from scratch.

**Fix:** Split `actions/cache` into `actions/cache/restore` (runs on all
events) and
`actions/cache/save` (only on `push` to `master`). PRs still read from
master's cache
(exact key hit when dependencies match), but no longer write their own
entries. Master's
cache stays alive, and every PR benefits from it.

The save condition checks both `event_name == 'push'` and `ref ==
refs/heads/master`
to prevent accidental cache writes from non-master push events.

All cache actions are SHA-pinned to `actions/cache@0057852b...`
(v4.3.0), consistent
with the repo's supply-chain hardening convention.

Also normalizes cache paths in `kernel_test.yaml` and `flink_test.yaml`
to match
other workflows (`~/.ivy2` instead of `~/.ivy2/cache`, drops redundant
`~/.coursier/cache`).

### Workflows modified (5 active):
- `build.yaml` — 1 cache key
- `spark_test.yaml` — N cache keys (per Spark version)
- `spark_examples_test.yaml` — N cache keys (per Spark version)
- `kernel_test.yaml` — 1 cache key
- `flink_test.yaml` — 1 cache key

Disabled workflows (`disabled_iceberg_test.yaml`,
`disabled_spark_python_test.yaml`,
`disabled_publish_docs.yaml`) are not modified.

## How was this change tested?

All CI checks passed on a previous iteration of this PR (before rebase).
The
`cache/restore` step works identically to the old `cache` step for
reads.

### When will the benefit be visible?

**This PR itself does not show dramatic speed improvements.** The
optimization is about
**preventing future degradation**, not speeding up individual runs.

1. **Today**, this PR reads from master's existing cache via
`actions/cache/restore` —
same as before. The change is that it **does not write** cache entries
(by design).
2. **The real benefit comes after merging.** Currently, every PR writes
cache entries that
   evict master's caches via LRU. After this merges:
   - Master pushes write caches (and keep them warm)
   - PRs only read, never write
   - Master's caches survive LRU eviction indefinitely
- **All future PRs** get consistent cache hits instead of random misses
3. **Where savings will show:** The SBT dependency resolution +
compilation phase is where
cache hits save time. On a cold cache miss, SBT dependency resolution
alone can add
**5-15 minutes per job**. With 20+ matrix jobs per PR, this prevents
significant wasted
   aggregate CI time when cache misses occur.

### Post-merge verification plan:
1. Observe master's CI run writes caches (`"Saved cache"` in post-action
logs)
2. Open a new PR and verify it hits master's cache and does NOT write
its own
3. Compare compile times on new PRs against pre-merge baseline

Inspired by
[delta-io/delta-kernel-rs#2130](delta-io/delta-kernel-rs#2130).
…letion Vectors enabled (delta-io#6329)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

In [this PR](delta-io#6328), we blocked
preservation of variant stats until the preservation of stats during DML
commands is improved. This PR adds Z85 encoding of variant stats during
writes when deletion vectors are involved and it doing so, it re-enables
the config controlling the preservation of variant stats.

## How was this patch tested?

Test making sure variant stats are preserved after DML commands.

## Does this PR introduce _any_ user-facing changes?

No
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (CI / Build Infrastructure)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.

If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

**Supply chain hardening: replace pip/pipenv with UV, add hash-verified
lock files, and harden the Dockerfile.**

This PR migrates all Python package management in CI workflows and the
Docker build from pip/pipenv to [uv](https://github.com/astral-sh/uv),
with cryptographic integrity verification and temporal lockdown to
mitigate supply chain attacks targeting PyPI.

### Changes

**1. UV-ify disabled CI workflows (`disabled_iceberg_test.yaml`,
`disabled_spark_python_test.yaml`)**
- Replace `pyenv install` + `pipenv` with `uv python install` + `uv
venv` + `uv pip install`
- Install uv via inline SHA256-verified tarball download (no GitHub
Action dependency)
- Install all Python packages from hash-verified lock files using
`--require-hashes`
- Add `UV_EXCLUDE_NEWER="2026-03-10T00:00:00Z"` workflow-level env as a
temporal firewall — uv refuses any package version published after this
date
- pyspark installed separately with `--no-deps` (its only transitive
dep, py4j, is included in the lock file)
- Workflows remain disabled (`workflow_dispatch` + `if: false`); the UV
changes are staged so that re-enabling is just removing the disable
guards
- Add SHA256 checksum verification for the `buf` binary download

**2. Add hash-verified lock files for CI (`spark4.0.lock`,
`spark4.1.lock`)**
- `.github/ci-requirements/spark-python/spark4.0.in` and `spark4.1.in` —
direct dependency declarations (extracted from the original per-package
`pipenv run pip install` lines)
- `.github/ci-requirements/spark-python/spark4.0.lock` and
`spark4.1.lock` — full transitive closure with SHA256 hashes, generated
by `uv pip compile --generate-hashes` with `UV_EXCLUDE_NEWER`
- Spark 4.0 and 4.1 have separate lock files due to differing
grpcio/protobuf version requirements
- These lock files were validated end-to-end in CI on the
[seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta)
staging repo before being ported here
- To add a new dependency to the spark_python_test set of python
dependencies, update the corresponding cross-spark version's `.in` file
and regenerate the lock file using the command:
```
UV_EXCLUDE_NEWER="2026-03-10T00:00:00Z" uv pip compile \
     --python-version 3.10 \
     --python-platform linux \
     --generate-hashes \
     -o sparkx.x.lock \
     sparkx.x.in
```

**3. Lock developer tooling (`dev/requirements.lock`)**
- Compiled from `dev/requirements.txt` (mypy, flake8, black, grpcio,
mypy-protobuf) using `uv pip compile --generate-hashes` with
`UV_EXCLUDE_NEWER`
- Pins the previously-unpinned grpcio/googleapis-common-protos ranges to
specific versions with hashes
- 19 packages, 249 lines

**4. Harden `disabled_publish_docs.yaml`**
- `pnpm install` → `pnpm install --frozen-lockfile` (enforces lock file
in CI, prevents drift)
- Pin `netlify-cli` to `23.15.1` (published 2026-02-04, pre-attack
baseline; was previously unpinned)

**5. Harden Dockerfile**
- Pin base image by SHA256 digest:
`ubuntu:focal-20221019@sha256:450e06...`
- Replace `python3-pip` + 14 bare `pip3 install` commands with uv
(SHA256-verified musl binary)
- All packages installed from `docker-build-requirements.lock` with
`--require-hashes` (43 packages, full transitive closure)
- Add `UV_EXCLUDE_NEWER="2026-02-10T00:00:00Z"` as temporal firewall
- Isolated `/opt/venv` instead of system-wide pip install
- Collapse 8 `RUN apt-get` layers into single layer with
`--no-install-recommends` and cache cleanup
- Fix legacy `ENV key value` → `ENV key=value` format
- `ARG UV_INDEX_URL` build arg for optional PyPI mirror/proxy

### Files changed (11 files, +2684/−112)

| File | Change |
|------|--------|
| `disabled_iceberg_test.yaml` | UV-ified (pyenv/pipenv → uv), still
disabled |
| `disabled_spark_python_test.yaml` | UV-ified (pyenv/pipenv → uv),
still disabled |
| `spark4.0.in` / `spark4.0.lock` | New — CI lock file for Spark 4.0
test variant |
| `spark4.1.in` / `spark4.1.lock` | New — CI lock file for Spark 4.1
test variant |
| `dev/requirements.lock` | New — hash-verified dev tooling lock |
| `disabled_publish_docs.yaml` | `--frozen-lockfile`, pin netlify-cli |
| `Dockerfile` | Full rewrite: digest-pinned base, uv, lock file, venv |
| `docker-build-requirements.in` / `.lock` | New — Dockerfile dependency
lock |

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

1. **CI workflow changes validated end-to-end in sandbox repo**: The
UV-ified `iceberg_test.yaml` and `spark_python_test.yaml` were pushed to
[seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta)
with the disabled guards removed, and all CI checks passed (uv install,
venv creation, hash-verified package installation, sbt test execution).

2. **Lock file generation verified**: All lock files were generated with
`UV_EXCLUDE_NEWER` set to the pre-attack baseline date. Every package
version in the lock files pre-dates the cutoff.

3. **Byte-identical verification**: The workflow files in this PR were
diffed against the CI-validated sandboxed-delta versions — the only
differences are the disable guards (`workflow_dispatch`, `if: false`,
`[DISABLED]` name prefix).

4. **Dockerfile build tested locally**: `docker build . -t
test_delta_spark --platform=linux/amd64 --build-arg
UV_INDEX_URL=<proxy>` — uv SHA verification passes, venv creation
succeeds, hash-verified package installation succeeds.

5. **netlify-cli version audit**: `23.15.1` confirmed via npm registry
API as published 2026-02-04, before the Feb 10 cutoff.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No. All changes are to CI infrastructure, the Docker build image, and
developer tooling. The published `delta_spark` Python package, its
`install_requires`, and all Scala/Java code are unchanged.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR adds support to Sink SQL API
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UT/Int
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Provide README and Docker Compose for easy testing
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
No code change
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…lta-io#6263)

## Summary

- Adds a **non-blocking** CI workflow (`spark_test_uc_master.yaml`) that
runs UC integration tests (`sparkUnityCatalog/test` and
`kernelUnityCatalog/test`) against Unity Catalog built from source (main
branch)
- Makes the UC version configurable at build time via
`-DunityCatalogVersion` system property (defaults to the current pinned
release `0.4.0`); the new workflow derives the version dynamically from
UC's `version.sbt` after building from source
- Includes a `setup_unitycatalog_main.sh` script that clones UC, builds
it, and publishes jars to the local Maven cache

<img width="993" height="145" alt="Screenshot 2026-03-31 at 6 39 07 PM"
src="https://github.com/user-attachments/assets/976718b0-ce0a-4adf-b682-364b5053db39"
/>


## Motivation

Today, Delta CI pins to the last UC release. This means:
1. **Incompatibilities are discovered in bulk** at version-bump time
near release — costly to fix under pressure
2. **No way to distinguish transient UC bugs from real Delta issues** —
when UC has a bug already fixed on main but not yet released, Delta CI
is red on every PR with no signal that it's transient

This non-blocking job gives a second signal on every PR:

| UC Release | UC Main | Interpretation |
|---|---|---|
| Pass | Pass | All good |
| Pass | Fail | Heads-up: upcoming UC change will require a Delta patch
|
| Fail | Pass | UC bug already fixed on main — transient, safe to land |
| Fail | Fail | Real issue in Delta code |

The job is **not a required status check**, so it cannot block PR
merges. Reviewers see the signal and can choose to investigate or not.

## CI cost impact

**Zero impact on critical path.** The existing CI takes ~81 min
end-to-end (dominated by the Delta Spark sharded tests). The new UC
master job completes in ~13 min — well before the existing jobs finish —
so total wall-clock time remains ~81 min.

Total compute cost increase is ~1.3% (13 runner-minutes on top of ~1040
existing runner-minutes).

## Design choices

- **Targeted tests only**: Runs only the UC integration test suites
(`sparkUnityCatalog/test`, `kernelUnityCatalog/test`) rather than the
full sharded test suite. This keeps the workflow to a **single CI job**
while still catching UC API/behavior breakages.
- **Default Spark version**: Lets SBT use its own default Spark version
rather than testing all versions, since we're testing UC compatibility,
not Spark version compatibility.
- **No hardcoded UC version**: The setup script extracts the version
from UC's `version.sbt` after cloning, so the workflow automatically
adapts when UC bumps versions.

## Changes

| File | What |
|------|------|
| `build.sbt` | `unityCatalogVersion` now reads from
`-DunityCatalogVersion` system property, defaulting to `0.4.0` |
| `project/scripts/setup_unitycatalog_main.sh` | New script: clones UC
repo, builds from source, publishes to `~/.m2`, outputs the built
version |
| `.github/workflows/spark_test_uc_master.yaml` | New single-job
workflow: builds UC from main, runs UC integration tests |

## Test plan

- [ ] Verify the existing `spark_test.yaml` workflow is unchanged and
still passes
- [ ] Verify the new `spark_test_uc_master.yaml` workflow runs and
reports status without blocking merge
- [ ] Verify `setup_unitycatalog_main.sh` successfully builds UC from
source
- [ ] Confirm the new workflow is NOT listed as a required status check
in branch protection

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…elta-io#6319)

## Description
Support for schema evolution in INSERT landed in Spark master:
apache/spark#54488

A few changes are required to maintain compatibility in Delta:
- V2Write commands (AppendData, OverwriteByExpression, ..) now have an
extra parameter `withSchemaEvolution` and introduce a few methods to
implement for schema evolution for V2Write commands (e.p.
`writePrivileges`)
- Spark doesn't set delta's writer option "mergeSchema" anymore, which
was used as workaround until a `withSchemaEvolution` was introduced on
write plan nodes. Instead, Spark now sets `withSchemaEvolution`
directly, and a pre-resolution rule
`PropagateSchemaEvolutionWriteOption` is added in Delta to set the
writer option `mergeSchema` when `withSchemaEvolution=true` if it's not
already explicitly set by the user.

## How was this patch tested?
Existing tests

## Does this PR introduce _any_ user-facing changes?
No
… ran it (delta-io#6270)

## Problem
`performIcebergCompatUpgradesAndValidation()` runs IcebergCompatV3
validation twice when WriterCompatV3 is enabled: once nested as a
post-processor, and once standalone. This causes redundant schema
traversals and feature checks.

## Fix
Track whether V3 was already validated by WriterCompatV3 and skip the
standalone call if so. Same optimization applied for V2/V1.

## Validation
- Confirmed by reading TransactionMetadataFactory.java lines 480-531.
- Tests verify both paths: WriterCompatV3+CompatV3 (runs once) and
CompatV3-only (still runs).

## Regression Prevention
Test cases cover both the optimized and standalone validation paths.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…in IcebergConverterSuite (delta-io#6430)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Support verifying read result on Iceberg metadata location inside
Iceberg test

## How was this patch tested?
Test-only change
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite"
```
…6468)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (test infrastructure)

## Description

Pass the `EXTRA_MAVEN_REPO` environment variable into the Docker
container when running tests via the Docker path in `run-tests.py`, so
that Python tests inside Docker can resolve artifacts from a customized
Maven repository (e.g., a staging repo for release verification).

Currently, when running Python tests inside Docker (`USE_DOCKER=1
./run-tests.py --group spark-python`), the `EXTRA_MAVEN_REPO` env var is
not forwarded into the container. This means the PyPi packaging test
(`using_with_pip.py`) inside Docker cannot pick up the custom Maven
repo.

### Where is `EXTRA_MAVEN_REPO` consumed?

`EXTRA_MAVEN_REPO` is read by several downstream scripts to configure
`spark.jars.repositories`:

| File | How it's used |
|------|--------------|
| `examples/python/using_with_pip.py` |
`builder.config("spark.jars.repositories",
os.getenv('EXTRA_MAVEN_REPO'))` |
| `examples/python/image_storage.py` | Same as above |
| `examples/scala/build.sbt` | `sys.env.get("EXTRA_MAVEN_REPO")` added
as a resolver |
| `run-integration-tests.py` | Sets `env["EXTRA_MAVEN_REPO"]` for
integration test subprocesses |

### Call stack: Python tests in Docker

```
[Host] USE_DOCKER=1 ./run-tests.py --group spark-python
│
├─ pull_or_build_docker_image()                     # run-tests.py:288
│
└─ run_tests_in_docker(image_tag, "spark-python")   # run-tests.py:289
     Forwards env vars into Docker container:
       JENKINS_URL, SBT_1_5_5_MIRROR_JAR_URL, SCALA_VERSION,
       TEST_PARALLELISM_COUNT, DISABLE_UNIDOC,
       EXTRA_MAVEN_REPO  ← this PR
     │
     └─ docker run ... ./run-tests.py --group spark-python
          │
          [Inside Docker — USE_DOCKER is NOT forwarded, prevents recursion]
          │
          └─ run_python_tests(root_dir)                # run-tests.py:291→101
               │
               └─ python3 python/run-tests.py          # run-tests.py:105
                    │
                    ├─ prepare()                        # python/run-tests.py:65
                    │    └─ sbt clean publishM2
                    │         (builds Delta jars → ~/.m2/repository)
                    │
                    ├─ run_python_style_checks()        # lint
                    ├─ run_mypy_tests()                 # type checking
                    │
                    ├─ run_pypi_packaging_tests()       # python/run-tests.py:165
                    │    ├─ pip install delta-spark wheel
                    │    └─ python3 examples/python/using_with_pip.py
                    │         └─ reads EXTRA_MAVEN_REPO ← consumed here
                    │            builder.config("spark.jars.repositories", ...)
                    │            SparkSession starts with custom Maven repo
                    │
                    └─ test(root_dir, "delta", ...)     # python/run-tests.py:26
                         └─ spark-submit \
                              --repositories file://~/.m2/repo,<public repos> \
                              --packages io.delta:delta-spark_2.13:X.Y.Z \
                              <test_file.py>
```

## How was this patch tested?

Manually verified the env var is correctly forwarded into the `docker
run` command.

## Does this PR introduce _any_ user-facing changes?

No.
#### Which Delta project/connector is this regarding?

- [x] Spark
- [x] Standalone
- [x] Flink
- [x] Kernel
- [x] Other (CI / GitHub Actions)

## Description

Gate all active CI workflows to only trigger on pushes to `master` and
pull requests targeting `master`. This is a follow-up to delta-io#6390 (supply
chain attack mitigation) and delta-io#6372 (SHA-pinning GitHub Actions).

**What changed**: Added `branches: [master]` to both `push` and
`pull_request` triggers in all 7 active workflow files:

| Workflow | Trigger style |
|---|---|
| `build.yaml` | Added `branches: [master]` to existing
`push`/`pull_request` |
| `flink_test.yaml` | Expanded `on: [push, pull_request]` → long form
with `branches: [master]` |
| `kernel_test.yaml` | Added `branches: [master]` to existing
`push`/`pull_request` |
| `kernel_unitycatalog_test.yaml` | Added `branches: [master]` to
existing `push`/`pull_request` |
| `spark_examples_test.yaml` | Added `branches: [master]` to existing
`push`/`pull_request` |
| `spark_test.yaml` | Added `branches: [master]` to existing
`push`/`pull_request` |
| `unidoc.yaml` | Expanded `on: [push, pull_request]` → long form with
`branches: [master]` |

The 3 disabled workflows (`disabled_iceberg_test.yaml`,
`disabled_publish_docs.yaml`, `disabled_spark_python_test.yaml`) already
use `workflow_dispatch` only and require no changes.

**Why**: Without branch gating, every push to any branch and every PR
targeting any branch triggers the full CI suite. This wastes GitHub
Actions minutes on feature branches, forks, and experimental work that
will eventually be validated when merged to `master` anyway. More
importantly, in the context of the ongoing supply chain attack targeting
OSS package ecosystems, restricting CI execution surface reduces the
window for potential abuse via workflow triggers on non-default
branches.

**Effect**: Only pushes to `master` and PRs targeting `master` will
trigger CI. All other branch activity is silenced. Existing
`paths`/`paths-ignore` filters are preserved and compose with the branch
filter (both conditions must match).

## How was this patch tested?

These changes were validated in a mirror fork
([seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta))
before being ported here:

1. **Positive test**: Pushed a commit to `master` — all 7 workflows
triggered as expected.
2. **Negative test**: Opened a PR from a non-master feature branch
targeting a non-master base — zero workflows triggered (confirmed via
the PR's Checks tab showing no status checks).
3. **PR-to-master test**: Opened a PR targeting `master` — all
applicable workflows triggered correctly.
4. **Byte-level parity**: `diff -r` of `.github/workflows/` between this
branch and the validated sandbox confirms zero differences.

## Does this PR introduce _any_ user-facing changes?

No. This change only affects when CI workflows run (branch-scoped
gating). It does not change any build outputs, library behavior, or
APIs. Contributors opening PRs against `master` (the standard workflow)
will see no change in CI behavior.
…o#6456)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Change `build.sbt` to add `releaseSettings` to the flink project.
Add a Flink runner to `run-integration-tests.py`
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Build change and test only change. Manually tested
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
… data skipping (delta-io#6444)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [X] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

In this PR we are adding geospatial table feature as per spec:

This is **PR 2 of 3** for geospatial data skipping support. It adds the
`StGeometryBoxesIntersect` public expression and its internal
translation to `ST_GEOMETRY_BOEXES_INTERSECT_ON_STATS` for data
skipping, including expression evaluation and end-to-end data skipping
tests.

To avoid introducing complex geospatial calculations into kernel and
default kernel, we don't allow evaluating ST_GEOMETRY_BOXES_INTERSECT
predicate onto rows, we only use it to to translate it to stats
predicate which is trivial to implement and does not require complex
geometric algorithms.

Geography boxes intersection is more invovled and requires accounting
for antimeridian and the fact that min can bigger than max, so we leave
that for other PRs.

This is following delta protocol
delta-io#4726

### Changes
- **StIntersectsBoxes** (new): Public predicate taking column + WKT
query bounds (queryMin, queryMax)
- **DataSkippingUtils**: Translate `ST_INTERSECTS_BOXES` to
`ST_INTERSECTS_BOXES_ON_STATS` with min/max stats columns
- **ExpressionVisitor**: Add `visitStIntersectsBoxesOnStats` dispatch
- **DefaultExpressionEvaluator**: Transform (type validation) and eval
(AABB intersection via WKT parsing with GeometryUtils)


## How was this patch tested?

- `GeometryDataSkippingSuite`: 5 end-to-end tests covering 4-quadrant
layout, null stats handling (conservative inclusion), and combined AND
predicates with geometry + scalar columns. Tests bypass Parquet writer
by injecting `DataFileStatistics` with WKT string literals directly.

## Does this PR introduce _any_ user-facing changes?

New expression
@milastdbx milastdbx closed this Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.