Skip to content

[Kernel] Geometry expressions: StGeometryBoxesIntersect predicate and data skipping - #1

Closed
milastdbx wants to merge 80 commits into
geospatial-stats-parsingfrom
geospatial-expressions
Closed

milastdbx wants to merge 80 commits into
geospatial-stats-parsingfrom
geospatial-expressions

Conversation

@milastdbx

@milastdbx milastdbx commented Mar 17, 2026 •

Copy link
Copy Markdown
Owner

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

In this PR we are adding geospatial table feature as per spec:

This is PR 2 of 3 for geospatial data skipping support. It adds the StGeometryBoxesIntersect public expression and its internal translation to ST_GEOMETRY_BOEXES_INTERSECT_ON_STATS for data skipping, including expression evaluation and end-to-end data skipping tests.

Depends on: PR A - Geospatial stats parsing (will rebase to master after PR A merges)

This is following delta protocol delta-io#4726

Changes

  • StIntersectsBoxes (new): Public predicate taking column + WKT query bounds (queryMin, queryMax)
  • DataSkippingUtils: Translate ST_INTERSECTS_BOXES to ST_INTERSECTS_BOXES_ON_STATS with min/max stats columns
  • ExpressionVisitor: Add visitStIntersectsBoxesOnStats dispatch
  • DefaultExpressionEvaluator: Transform (type validation) and eval (AABB intersection via WKT parsing with GeometryUtils)

PR dependency graph

master
  |
  +-- PR A: Stats Parsing + GeometryUtils
  |     |
  |     +-- PR B (this): Expressions + Predicate Translation
  |     |
  |     +-- PR C: Parquet / Data Layer

How was this patch tested?

  • GeometryDataSkippingSuite: 5 end-to-end tests covering 4-quadrant layout, null stats handling (conservative inclusion), and combined AND predicates with geometry + scalar columns. Tests bypass Parquet writer by injecting DataFileStatistics with WKT string literals directly.

Does this PR introduce any user-facing changes?

New expression

PorridgeSwim and others added 30 commits March 10, 2026 11:47
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
Before we support schema tracking log, we should disallow all
non-additive schema change in dsv2.
Column Mapping related code change is pure refactoring that for v2 to
reuse the utilities.
## How was this patch tested?
Unit test:
- New test case for drop column in a nested struct
- Modify existed tests for non-additive schema change to throws the same
error as v1

Integration test:
- drop column: should fail with non-additive schema change error
- drop column: should succeed with unsafe column mapping schema change
flag enabled
- rename column: should fail with non-additive schema change error
- rename column: should throw schema change error with unsafe flag
enabled
- type widening: should fail with non-additive schema change error when
enable schema tracking
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…ty (delta-io#6227)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6227/files) to
review incremental changes.
-
[**stack/type-nullability-e2e-test**](delta-io#6227)
[[Files changed](https://github.com/delta-io/delta/pull/6227/files)]
-
[stack/forbid-schema-evolution-with-staleDF-v2](delta-io#6228)
[[Files
changed](https://github.com/delta-io/delta/pull/6228/files/cafe746303c3e32bdcf5499c8e99fd6a59c033c4..94b77dc62d21e896e9f4fdafd3ff17fafddd606f)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Added schema evolution e2e tests for type widening and relax nullability
for both refresh dataframe and stale dataframe cases.

Refresh dataframe cases work for both v1 & v2 connector, while stale
dataframe cases only work for v1 connector.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
test-only PR
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…m 50000 to 100000 (delta-io#6240)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Double the default value of `DELTA_STREAMING_INITIAL_SNAPSHOT_MAX_FILES`
from 50,000 to 100,000 to better support larger tables for V2 streaming
initial snapshots out of the box.

## How was this patch tested?

Existing tests continue to pass since this is a default-value-only
change. Tests that explicitly set this config (e.g.,
`SparkMicroBatchStreamTest`) are unaffected.

## Does this PR introduce _any_ user-facing changes?

Yes. The default maximum number of files allowed in the initial snapshot
for V2 streaming is increased from 50,000 to 100,000. Users with tables
between 50k–100k files will no longer hit the limit without explicitly
raising the config. This change only affects the unreleased master
branch and the future release.
…ype widening and relaxing nullability (delta-io#6228)

## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)
to review incremental changes.
-
[stack/type-nullability-e2e-test](delta-io#6227)
[[Files changed](https://github.com/delta-io/delta/pull/6227/files)]
-
[**stack/forbid-schema-evolution-with-staleDF-v2**](delta-io#6228)
[[Files
changed](https://github.com/delta-io/delta/pull/6228/files/58309c897e41296ee84b4f0714867c4a229b130b..781fa7a841d9f7dc1e6551426f12bf5fe90f8055)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Add schema validation during stream restart e2e tests for type widening
and relaxing nullability to block user from adopt these two schema
change without refresh the dataframe in v2 connector

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Test-only PR
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…a-io#5837)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Add the DataFrame basic test coverage for preventing future regression. 

## How was this patch tested?

Do the locally unit tests and remotely integration tests. 

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: openinx <openinx@gmail.com>
… repartition the table's columns (delta-io#6238)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
- Updated partition column change tests in `OptimisticTransactionSuite`
to use `checkError` instead of calling `ex.getErrorClass` directly
- Each `checkError` call now verifies the error condition
(`DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE`), SQLSTATE (`42P10`), and
named parameter values (`operation`, `oldPartitionColumns`,
`newPartitionColumns`)
- Added expected operation name to the parameterized blocked-operations
test cases to support per-case parameter verification
- Collapsed the `DELTA_UNSUPPORTED_PARTITION_COLUMN_CHANGE` error
message from three lines into a single line in
`delta-error-classes.json`, `databricks-error-classes.json`, and
`delta-error-classes-data-classification.json` (the multi-line split was
not justified as the column lists render as single-line comma-separated
strings)


## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
No new tests added. Existing tests in `OptimisticTransactionSuite`
covering partition column change validation were updated to use
`checkError` as required by the error message review guidelines.

## Does this PR introduce _any_ user-facing changes?
N/A
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
delta-io#5775)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/5775/files) to
review incremental changes.
-
[**stack/dv_pr3_phase2_vectorized**](delta-io#5775)
[[Files changed](https://github.com/delta-io/delta/pull/5775/files)]
-
[stack/dv_pr4_phase3_file_splitting](delta-io#5776)
[[Files
changed](https://github.com/delta-io/delta/pull/5776/files/463011af2fae489f7f3fe8c39fe86acbba06277b..0e534ed9c0257735523532c8c6f8e15ea33ae114)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Enable vectorized (columnar batch) reading for DV tables in the Spark V2
connector.

### Changes:
- `ColumnVectorWithFilter`: Wraps `ColumnVector` with row ID mapping for
zero-copy filtering (follows Iceberg pattern)
- `DeletionVectorReadFunction`: Extended to handle `ColumnarBatch` in
addition to `InternalRow`
- Enable vectorized reader for DV tables (remove `!isTableSupportDv`
condition)

### How it works:
1. For `ColumnarBatch`: Build live row mapping, wrap columns with
`ColumnVectorWithFilter`
2. For `InternalRow`: Filter deleted rows, project using
`ProjectingInternalRow`

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

- `ColumnVectorWithFilterTest`: Unit tests for various data types and
filtering scenarios
- `DeletionVectorVectorizedReaderTest`: Integration tests verifying
vectorized path is used
- All DV golden table tests pass with vectorized reading
## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…a-io#6104)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Spark's StringStartsWith(attribute, value) filter was not converted to a
Kernel predicate, meaning WHERE col LIKE 'prefix%' queries had no filter
pushdown in the V2 connector. The Kernel's STARTS_WITH predicate maps
directly: STARTS_WITH(column, string_literal).

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…sts (delta-io#6210)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Add the DataFrame streaming integration tests to cover the spark + uc
case.

## How was this patch tested?

Run in both locally UC server and remote UC server.

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: openinx <openinx@gmail.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [X] Other (sharing)

## Description

Upgrade delta sharing client to 1.3.10 

## How was this patch tested?

CI pipeline

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR blocks writes with NullType in UDT columns, similar to how we
block NullType in arrays/maps. NullType is typically used as a temporary
placeholder, which is later evolved into some other type. However, UDTs
cannot evolve over time, hence NullType inside UDT remains as NullType
forever. Additionally, our old check missed arrays/maps inside UDT and
didn't check whether they contain NullType or not. When trying to write
such Parquet files, the Parquet writer failed. This PR fixes these
problems.

## How was this patch tested?

New unit tests.

## Does this PR introduce _any_ user-facing changes?

No.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
This PR adds support to excludeRegex read option in deltaDSv2
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
Unit tests:
Create table partitioned by category with 4 versions of data.
```
Version 0: CREATE TABLE (empty)
Version 1: ids {1,2} in category=alpha
Version 2: ids {3,4} in category=beta
Version 3: ids {5,6} in category=alpha
Version 4: ids {7,8} in category=gamma
```
Test cases: "No excludeRegex - all versions", "Exclude alpha category",
"Exclude beta category", "Exclude gamma category", "Exclude alpha and
beta categories", "Regex matching no files", "Regex matching all files",
"Exclude alpha versions 2 to 4", "Exclude beta version 2 only", "Initial
snapshot - no excludeRegex", "Initial snapshot - exclude alpha".

Enable integration test: "excludeRegex works and doesn't mess up offsets
across restarts - parquet version"
## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6204)

#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Send table schema to UC server when creating managed tables

## How was this patch tested?
Integration test

## Does this PR introduce _any_ user-facing changes?
No.

---------

Signed-off-by: Yi Li <yi.li@databricks.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] General repository

## Description

Adds a note regarding the required Java version into the README.md.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

Inapplicable.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

No.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

---------

Co-authored-by: Leonid Lygin <leonid.lygin@databricks.com>
Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Iceberg)

## Description

We have the following NPE:

```
java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:209)
	at org.apache.iceberg.SchemaParser.fromJson(SchemaParser.java:296)
	at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:144)
	at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:129)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:138)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
```

Referring to this line:


https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/core/src/main/java/org/apache/iceberg/SchemaParser.java#L296

This comes from the following line:


https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L144

The field `schema` is only set in V1 fields. I believe the reference
implementation (Java OSS) also sets this field for V2/V3 to maximize
compatibility, but this might be different for other writers.

By passing in the spec, we don't execute the branch and we avoid the
NPE:


https://github.com/apache/iceberg/blob/7dbafb438ee1e68d0047bebcb587265d7d87d8a1/core/src/main/java/org/apache/iceberg/ManifestReader.java#L126-L130


<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6203)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

Convert CatalogStatistics to V2 connector Statistics and thread them
through SparkTable → SparkScanBuilder → SparkScan.estimateStatistics().
Gate numRows/columnStats on cboEnabled || planStatsEnabled to match V1
LogicalRelation.computeStats() behavior.

Use per-column avgLen from catalog stats in size estimation instead of
defaultSize(), mirroring EstimationUtils.getSizePerRow() with UTF8String
overhead for StringType columns.

Resolves delta-io#5952


## How was this patch tested?

Unit test

## Does this PR introduce _any_ user-facing changes?

No
…a-io#5888)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->
This PR adds a new Spark flag that will make all code paths in
DataSkippingReader to collect stats.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->
New tests.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No.
…iveCatalog for Delta UniForm (delta-io#6189)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Use UnityCatalog as Iceberg committing catalog instead of HiveCatalog
for Delta UniForm


## How was this patch tested?
Add UTs
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniFormConverterSuite"
```
…flictSuiteBase (delta-io#6277)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Add an overridable `sqlWithTotalRpcBound` method to
`IdentityColumnConflictSuiteBase` that defaults to calling `sql()`. This
is a no-op in OSS - it allows internal subclasses to override the method
with custom assertions on the winning transaction's SQL execution path.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

No behavioral change - the new method defaults to `sql()`, identical to
the existing behavior. All existing `IdentityColumnConflictSuite` tests
pass unchanged.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No.

Co-authored-by: Shlok Jhawar <shlok.jhawar@databricks.com>
…o#6271)

## Problem
In `buildTransactionInternal()`, when `!needsMetadataOrProtocolUpdate`,
a new `TransactionImpl` is constructed but never returned. Execution
falls through to the full metadata/protocol update pipeline, doing
unnecessary work and constructing a second `TransactionImpl`. Fixes
delta-io#4713.

## Fix
Add `return` before `new TransactionImpl(` so the early-exit path
actually returns.

The early return skips `TransactionMetadataFactory`, which also runs
`validateRowTrackingConfigsNotMissing` on existing tables. To preserve
this validation, it is moved into `validateWriteToExistingTable()` so it
runs on every write regardless of whether metadata/protocol updates are
needed.

## Validation
- Confirmed by reading lines 230-247.
- New unit test verifies the early-return path (uses a fake
`delta.feature.*` property so the factory throws if the early return is
bypassed).
- Verified `LegacyRowTrackingSuite` passes (45/45 tests), including the
row tracking config validation test.

## Regression Prevention
- `TransactionBuilderImplSuite`: tests the no-update transaction path.
- Existing `LegacyRowTrackingSuite`: covers the row tracking validation
that was moved.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…lta-io#6278)

## Problem
Spark's CheckDeletionVectorDisabled checks both newestProtocol and
prevSnapshot to guard against concurrent writers enabling DVs. The
Kernel's CHECK_HAS_NO_DELETION_VECTORS only checks newProtocol, missing
the concurrent writer guard.

## Fix
Add the previous protocol check to the Kernel's DV validation, matching
the Spark implementation's defense-in-depth approach.

## Validation
- Confirmed by comparing Spark IcebergCompat.scala with Kernel
IcebergCompatMetadataValidatorAndUpdater.java.
- New test verifies the check catches DV enablement from a previous
protocol.

## Regression Prevention
New test case explicitly validates the previous-protocol DV check.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
…n in dsv2 (delta-io#6246)

## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)
to review incremental changes.
- [stack/ignoreDeletesV2](delta-io#6245)
[[Files changed](https://github.com/delta-io/delta/pull/6245/files)]
-
[**stack/skipChangeCommitsV2**](delta-io#6246)
[[Files
changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)]
- [stack/ignoreChangesV2](delta-io#6249)
[[Files
changed](https://github.com/delta-io/delta/pull/6249/files/c1c942553a20800e318a5d2d384f869a3c0c5d3c..7209f24b489535e57a8396d712f0f1da577520d2)]
-
[stack/ignoreFileDeletionV2](delta-io#6250)
[[Files
changed](https://github.com/delta-io/delta/pull/6250/files/7209f24b489535e57a8396d712f0f1da577520d2..8690f78ca1bae137706f46ed1e05cdb05702ee6b)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Support 
- skipChangeCommits read option is DSv2, which skip a commit that
contains remove file actions regardless of if there is an add file
action or not.
- ignoreDeletes read option is DSv2, which skip a commit that contains
only remove file actions but no add file action.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Unit tests that test parity between v1 and v2 connector on both pure
deletes commit (only remove) and change commit (add + remove).

Integration tests:
- streaming with skipChangeCommits = true skips both delete and change
commits
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…elta-io#6267)

## Problem
`generateIcebergCompatWriterV3RemoveAction()` accepts `baseRowId` and
`defaultRowCommitVersion` parameters but silently discards them,
hardcoding `Optional.empty()` instead. This breaks row tracking for V3
writer compat remove actions.

## Fix
Forward the caller-provided `baseRowId` and `defaultRowCommitVersion` to
`convertRemoveDataFileStatus()` instead of passing `Optional.empty()`.

## Validation
- Confirmed by reading the method: params accepted at lines 266-274,
discarded at lines 308-310.
- New test verifies non-empty values are preserved in the output Row.

## Regression Prevention
New test case explicitly checks that baseRowId and
defaultRowCommitVersion are forwarded through to the RemoveFile row.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

During data-skipping, we add one millisecond to timestamp max stats to
compensate for the fact that JSON serialization will truncate them to
milliseconds. This has the consequence that if the data contains a
timestamp within the range (Long.MaxValue - 1000, Long.MaxValue], then
the addition will throw an overflow exception.

This PR fixes this by adding a check for if the stats is within that
range, then we saturate the stats to Long.MaxValue instead of adding
1000 to it.

## How was this patch tested?
Unit test

## Does this PR introduce _any_ user-facing changes?

No
…io#6268)

## Problem
`IcebergRESTCatalogPlanningClient` implements `AutoCloseable` but the
`ServerSidePlanningClient` trait does not extend it, and
`ServerSidePlannedTable` never calls `close()`. This leaks HTTP client
connections.

## Fix
Add proper lifecycle management so the HTTP client is closed when no
longer needed.

## Validation
- Confirmed by reading the trait hierarchy and table construction code.
- New test verifies the client is properly closed.

## Regression Prevention
New test verifies client lifecycle management.

---------

Co-authored-by: foo.bar <foo.bar@foobar.com>
#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Deflake PartitionLikeDataSkippingSuite by making result comparison
insensitive to nondeterministic record ordering.

## How was this patch tested?
See test change.

## Does this PR introduce _any_ user-facing changes?
No.
#### Which Delta project/connector is this regarding?
- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Block metadata changes on UC-managed tables
Without this, metadata mutations on a managed table succeed locally but
leave the UC catalog unaware, causing metadata drift.

Two kill switches in OptimisticTransaction:
- updateMetadata(): blocks schema, partition, description, and
configuration changes. Covers INSERT with autoMerge, MERGE with schema
evolution, and DataFrame writes with mergeSchema.
- commitLarge(): blocks clustering DomainMetadata changes for RESTORE
TABLE, which bypasses updateMetadata(). ALTER TABLE is already blocked
by UCSingleCatalog before reaching Delta.

## How was this patch tested?
New tests added.

## Does this PR introduce _any_ user-facing changes?
Yes. Table schema evolution with DML is blocked for UC managed tables.

---------

Signed-off-by: Yi Li <yi.li@databricks.com>
…in CRC file (delta-io#6282)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Added a conf to make delta-spark write fileSizeHistogram (and not
histogramOpt) as field name in CRC file.


## How was this patch tested?

UTs

## Does this PR introduce _any_ user-facing changes?

No
…elta-io#6281)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

This PR updates Kernel's `CRCInfo` to **read** both `fileSizeHistogram`
and `histogramOpt` fields to ensure backward compatibility with
Delta-Spark.

While the [Delta
spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#version-checksum-file-schema)
officially uses `fileSizeHistogram` in the Version Checksum file,
Delta-Spark has historically used histogramOpt
([code](https://github.com/delta-io/delta/blob/5ba861fa6d6999d7079706a307a7e293b59d4880/spark/src/main/scala/org/apache/spark/sql/delta/Checksum.scala#L87)).
Following [a recent
fix](delta-io@021755f)
that allows Delta-Spark to **read** both fields, this PR mirrors that
compatibility in Kernel. This ensures smooth read operations as we fully
transition Delta-Spark to write the correct `fileSizeHistogram` field.


## How was this patch tested?
New unit tests.
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No
gengliangwang and others added 4 commits March 23, 2026 14:47
…ale (delta-io#6354)

## Summary
- Fix `ExpressionUtils.convertValueToKernelLiteral` crash when a
`BigDecimal` has precision less than scale
- Normalize precision to `Math.max(bd.precision(), bd.scale() + 1)`
before calling `Literal.ofDecimal`
- Add unit test for the edge case

## Problem

A query with a decimal `IN` predicate containing `0.00` crashes when
using the V2 connector:

```sql
SELECT * FROM delta_table WHERE dec10_2 IN (0.00, 100.00)
```

This throws an `IllegalArgumentException: Invalid precision and scale
combo` from Kernel's `Literal.ofDecimal`.

**Root cause:** Java's `BigDecimal("0.00")` reports `precision()=1` and
`scale()=2`. This violates the invariant that `precision >= scale`
required by Kernel's `Literal.ofDecimal`. The V2 code in
`ExpressionUtils.convertValueToKernelLiteral` was passing these values
through without normalization.

## Fix

Before constructing the Kernel literal, normalize precision:

```java
int precision = Math.max(bd.precision(), bd.scale() + 1);
```

For `BigDecimal("0.00")`, this yields `precision=3, scale=2` (i.e.,
`DECIMAL(3,2)`), which correctly represents the value.

## Test plan
- [x] New unit test
`testConvertValueToKernelLiteral_DecimalWithScaleExceedingPrecision`
passes
- [x] All 73 `ExpressionUtilsTest` tests pass
…without a supported IcebergCompat version (delta-io#6352)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

When a user enables `delta.universalFormat.enabledFormats = 'iceberg'`
alongside an unrecognized IcebergCompat property (e.g.
`delta.enableIcebergCompatV3`), the error message currently says:

> To enable IcebergCompatV2, set the table property
'delta.enableIcebergCompatV2' = 'true'.

This is misleading because:
1. Spark Delta silently ignores the unrecognized V3 property (only V1
and V2 are in `IcebergCompat.knownVersions`)
2. The error then directs the user to V2 with no indication that V3 is
not supported

The fix updates the message to explicitly state that the supported
versions are IcebergCompatV1 and IcebergCompatV2, making clear to users
that higher versions are not yet supported in this Spark Delta release.

Resolves delta-io#6351

## How was this patch tested?

The change is limited to an error message string. Manual reproduction:

```sql
SET spark.databricks.delta.allowArbitraryProperties.enabled=true;

CREATE TABLE demo.icebergCompatV3 (i INT, s STRING) USING DELTA
TBLPROPERTIES (
  'delta.columnMapping.mode'             = 'name',
  'delta.enableIcebergCompatV3'          = 'true',
  'delta.enableDeletionVectors'          = 'false',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);
```

**Before:** `"To enable IcebergCompatV2, set the table property
'delta.enableIcebergCompatV2' = 'true'."`

**After:** `"Supported versions are IcebergCompatV1 and
IcebergCompatV2."`

## Does this PR introduce _any_ user-facing changes?

Yes. The error message for `DELTA_UNIVERSAL_FORMAT_VIOLATION` (when
UniForm Iceberg is enabled without a recognized IcebergCompat version)
now explicitly lists the supported IcebergCompat versions (V1 and V2),
instead of directing the user to enable V2. This helps users who set an
unsupported version (e.g. V3) understand why the error occurred.

---------

Signed-off-by: openinx <openinx@gmail.com>
…commits (delta-io#6338)

## Description

This PR adds a regression test that exposes a **data loss bug** in Delta
streaming when used with Coordinated Commits and non-trivial backfill
batch sizes.

Related issue: delta-io#6339

### Observed Behavior

The test writes 100 sequential single-row commits to a Delta table while
a streaming query is running, then verifies all 100 values appear in the
sink.

- **batchSize = 1**: Test passes consistently. All 100 values present.
- **batchSize = 2**: Test fails. ~4 out of 100 commits are lost on
average. **Only odd-versioned commits are lost** (these are the versions
that are not immediately backfilled).
- **batchSize = 3**: Test fails. The loss pattern follows the backfill
cycle:
  - `v % 3 == 0`: no loss (these versions trigger backfill)
- `v % 3 == 1`: v may be lost, and if v is lost, v+1 is always lost
together (they get backfilled as a pair)
  - `v % 3 == 2`: v may be lost independently

The pattern strongly correlates with which commits are sitting
unbackfilled in the coordinator at any given time, suggesting the bug is
related to the interaction between backfill and commit listing.

### Hypothesized Root Cause

We suspect the issue is a race condition in
`CoordinatedCommitsUtils.commitFilesIterator`, which is used by
`DeltaSource.getFileChanges` during both `latestOffset` and `getBatch`.
This method lists commits in two lazy, sequential steps:

1. List backfilled commits from the filesystem (`listedDeltas`)
2. Query the coordinator for unbackfilled commits (`tailFromSnapshot`)

These two steps are **not atomic**. A possible scenario (example with
batchSize = 3):

1. **During `latestOffset`**: filesystem has `[0.json]`, coordinator has
`[1, 2]`. `latestOffset` correctly computes endOffset covering through
version 2.
2. **During `getBatch`**: the filesystem listing iterator runs and sees
`[0.json]`.
3. **Between the two iterators**: a concurrent write creates version 3,
triggering `backfillToVersion(3)`. This writes versions 1, 2, 3 to the
filesystem and **removes them from the coordinator** via
`registerBackfill`.
4. **The coordinator query runs**: returns empty — versions 1 and 2 have
been removed.
5. **Result**: `getBatch` misses versions 1 and 2. The next batch starts
from version 3, so they are never re-read.

## How was this patch tested?

Added a new test `"streaming processes 100 sequential single-value
commits and contains all values 0 to 99"` that:
- Creates a Delta table and starts a streaming query
- Appends 100 single-row commits while the stream is running
- Verifies all 100 values appear in the sink

The test passes for batchSize = 1 but fails for batchSize = 2 and 3 due
to the data loss.

## Does this PR introduce _any_ user-facing changes?

No. This PR only adds a test to demonstrate the existing bug. A fix will
follow in a subsequent PR.
## 🥞 Stacked PR
Use this
[link](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de)
to review incremental changes.
- [stack/ignoreDeletesV2](delta-io#6245)
[[Files changed](https://github.com/delta-io/delta/pull/6245/files)]
-
[stack/skipChangeCommitsV2](delta-io#6246)
[[Files
changed](https://github.com/delta-io/delta/pull/6246/files/6e9962d4a30a63ed14786830bae2b668004a76c9..2e111cf6ac9d1e5f84d83d94412b05486f543613)]
-
[**stack/ignoreChangesV2**](delta-io#6249)
[[Files
changed](https://github.com/delta-io/delta/pull/6249/files/0d54da51e7eade47b8115d92aaf7be1e8e4c011f..7e400f87189bef892d3b0022c0aede238a0c84de)]
-
[stack/ignoreFileDeletionV2](delta-io#6250)
[[Files
changed](https://github.com/delta-io/delta/pull/6250/files/7e400f87189bef892d3b0022c0aede238a0c84de..af230590d0d9c1a5c37cc6a7b6af404025a6d415)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Support ignoreChanges read option in DSv2, which skip all remove file
actions but keep the add file actions in the same commit.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Unit tests that test parity between v1 and v2 connector on both pure
deletes commit (only remove) and change commit (add + remove).

Integration tests:
- streaming with ignoreChanges = true allows both delete and change
commits
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
@milastdbx
milastdbx force-pushed the geospatial-expressions branch from 5329973 to b6a1661 Compare March 24, 2026 14:06
@milastdbx milastdbx changed the title [Kernel] Geospatial expressions: StIntersectsBoxes predicate and data skipping [Kernel] Geometry expressions: StGeometryBoxesIntersect predicate and data skipping Mar 24, 2026
ChengJiX and others added 22 commits March 24, 2026 21:27
…elta-io#6335)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

## Description
Write atomic-supported property to Iceberg compact tables. This is part
of project to support write to UC managed UniForm tables

## How was this patch tested?
Add UTs
```
build/sbt -DsparkVersion=4.0 "iceberg/testOnly org.apache.spark.sql.delta.uniform.UniversalFormatSuite"
```
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (fill in here)

## Description

As the title says
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

Will be merged later
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
delta-io#5567)

Co-authored-by: Coauthor One andreAmorimF
Co-authored-by: Coauthor Two aryadneguardieiro
Co-authored-by: Coauthor Three yurimantchev-sudo

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description
Allow writes to CDF tables for add-only, remove-only, and
non-data-change transactions.
More context:
https://delta-users.slack.com/archives/C04TRPG3LHZ/p1763402127485709?thread_ts=1763400653.848629&cid=C04TRPG3LHZ
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

---------

Co-authored-by: André Fonseca <1077309+andreAmorimF@users.noreply.github.com>
Co-authored-by: OussamaSaoudi <45303303+OussamaSaoudi@users.noreply.github.com>
…ipenv in spark_test workflow (delta-io#6390)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (CI/GitHub Actions)

## Description

This PR hardens the GitHub Actions CI pipelines against supply chain
attacks
targeting the Python/PyPI package ecosystem. It reduces the attack
surface by
eliminating unnecessary `pip install` invocations from workflows where
they are
not required for the tests being run.

### Changes

**`spark_test.yaml` — Remove redundant Python environment setup**

The `spark_test` workflow runs Scala/Java tests via `build/sbt
sparkGroup/test`,
but previously set up a full Python environment (pyenv, pipenv, and 20+
`pipenv run pip install` commands) that was not consumed by any test
step. This
PR removes the `Install Job dependencies` step entirely and switches the
test
invocation from `pipenv run python run-tests.py` to `python3
run-tests.py`,
which requires only the system Python (stdlib-only — `run-tests.py` has
no
third-party imports).

This change is consistent with delta-io#6384, which independently identified the
same
redundancy.

**`iceberg_test.yaml`, `spark_python_test.yaml`, `publish_docs.yaml` —
Disable
workflows with heavy PyPI dependency chains**

These three workflows install packages from PyPI at runtime via
`pip`/`pipenv`/`conda` and cannot function without those installs.
Rather than
deleting them, they are renamed to `disabled_*.yaml` to preserve them
for
future re-enablement once the dependency installation strategy is
hardened
(e.g., via hash-pinned lockfiles or reproducible resolution with
`UV_EXCLUDE_NEWER`).

| Workflow | Reason for disabling |
|---|---|
| `disabled_spark_python_test.yaml` | Installs 20+ PyPI packages via
pyenv/pipenv including pyspark, pandas, pyarrow, grpcio |
| `disabled_iceberg_test.yaml` | Installs pyenv/pipenv and runs tests
through `pipenv run python` |
| `disabled_publish_docs.yaml` | Sets up a full conda environment
(`docs/environment.yml`) with 50+ packages for Sphinx doc generation |

**Workflows left unchanged (no PyPI exposure)**

The following workflows use only system Python (stdlib-only scripts) or
no
Python at all, and are unaffected by this change:

- `build.yaml` — stdlib Python only (`test_cross_spark_publish.py`), no
pip
- `kernel_test.yaml` — stdlib Python test runner (`run-tests.py`), no
pip
- `spark_examples_test.yaml` — stdlib Python for version info, tests are
pure SBT
- `flink_test.yaml` — pure SBT
- `kernel_unitycatalog_test.yaml` — pure SBT
- `kernel_docs.yaml` — pure SBT
- `unidoc.yaml` — pure SBT

### Additional context

All 52 GitHub Actions `uses:` references across all workflows are
already pinned
to full 40-character commit SHAs with version comments. An audit
confirmed that
14 of 15 unique pinned SHAs are older than 30 days; the single exception
(`pnpm/action-setup@v5.0.0`, 12 days old) is in the now-disabled
`publish_docs.yaml`.

## How was this patch tested?

1. **CI run on fork**: The changes were pushed to a sandboxed public
fork

([seewishnew/sandboxed-delta](https://github.com/seewishnew/sandboxed-delta))
and all enabled workflows executed on GitHub Actions. The `spark_test`,
`build`, `spark_examples_test`, `flink_test`,
`kernel_unitycatalog_test`,
`kernel_docs`, and `unidoc` workflows ran without issues related to the
CI
   changes.

2. **SHA pin audit**: All `uses:` references were verified to be pinned
to full
   commit SHAs with correct version comments using a custom scanner.

## Does this PR introduce _any_ user-facing changes?

No. This PR only modifies CI workflow configuration. No source code,
build
definitions, APIs, or runtime behavior are changed. Three CI jobs
(`spark_python_test`, `iceberg_test`, `publish_docs`) are temporarily
disabled
and can be re-enabled once a hardened dependency installation strategy
is in
place.
#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [X] Kernel
- [ ] Other (fill in here)

## Description

In this PR we are adding geospatial table feature as per spec:
delta-io#4726

## How was this patch tested?

Tests

## Does this PR introduce _any_ user-facing changes?

Adds new table feature
## Description

`commitFilesIterator` discovers commit files in two phases: Phase 1
lists backfilled files from the filesystem, Phase 2 queries the
coordinator for unbackfilled commits. If all unbackfilled commits get
concurrently backfilled between the two phases (e.g., a concurrent
writer triggers `backfillToVersion` which copies staged files to
`_delta_log/` and removes them from the coordinator via
`registerBackfill`), Phase 2 finds no `UnbackfilledDeltaFile` entries
and the existing gap check has no anchor (`headOption` returns `None`),
returning `Iterator.empty`. Versions between `maxVersionSeen + 1` and
`endSnapshot.version` are silently dropped.

This race manifests when a streaming reader and a concurrent writer run
on different processes (different DeltaLog instances). The reader's
Phase 1 lists the filesystem and finds nothing new. The writer commits
and triggers backfill. The reader's Phase 2 does a fresh
`deltaLog.update()`, which classifies the now-backfilled files as
`BackfilledDeltaFile` and gets empty `getCommits()`. The gap check fails
and data is lost.

Related issue: delta-io#6339

## How was this patch tested?

Added `DeltaSourceWithCoordinatedCommitsBatch2Suite` and
`DeltaSourceWithCoordinatedCommitsBatch3Suite` to exercise batch-size-2
and batch-size-3 backfill boundaries.

## Does this PR introduce _any_ user-facing changes?

No. This is a bug fix for an internal race condition in commit file
discovery.
## Summary
Replace the static bearer token in `IcebergRESTCatalogPlanningClient`
with a per-request `tokenSupplier` backed by UC's `TokenProvider`,
enabling OAuth client credentials flow.

## Auth config

**Static token (existing, unchanged):**
```
spark.sql.catalog.<name>.token = "dapi..."
```

**OAuth client credentials (new):**
```
spark.sql.catalog.<name>.auth.type = oauth
spark.sql.catalog.<name>.auth.oauth.uri = https://<idp>/token
spark.sql.catalog.<name>.auth.oauth.clientId = <id>
spark.sql.catalog.<name>.auth.oauth.clientSecret = <secret>
```

Follows UC's `AuthConfigUtils` pattern. Error if both `.token` and
`.auth.token` are set.

## Key changes
- `ServerSidePlanningMetadata`: `authToken` -> `tokenSupplier`
(per-request function)
- `UnityCatalogMetadata`: Extract `auth.*` configs, create
`TokenProvider`. Legacy `.token` fallback with conflict check.
- `IcebergRESTCatalogPlanningClient`: `token: String` -> `tokenSupplier:
() => String`. `HttpRequestInterceptor` for per-request Bearer token
injection.

## Test plan
- [ ] `extractAuthConfig` tests: OAuth, static, legacy, conflict error,
legacy wins, empty
- [ ] Existing SSP tests updated for new constructor signatures
- [ ] CI on delta-io/delta

This pull request was AI-assisted by Isaac.
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Introduce an `ANALYSIS` phase for the fuzz tester.

## How was this patch tested?
Fuzz tester

## Does this PR introduce _any_ user-facing changes?
No
…rwrite partitions that contain NULL in the table (delta-io#6375)

Co-authored-by: Isaac

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

  - [x] Spark
  - [ ] Standalone
  - [ ] Flink
  - [ ] Kernel
  - [ ] Other (fill in here)

  ## Description

When users specify this option with DPO Mode, DPO will not overwrite
partitions that contain NULL in the table.


  ## How was this patch tested?

  Added UTs.

  ## Does this PR introduce _any_ user-facing changes?

Adding a new `DeltaOption` `useNullIntolerantEqualityWithDPO`, to ignore
overwriting partitions that contain NULL in Dynamic Partition Overwrite
mode.
… is from Legacy (delta-io#6392)

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [X] Other (Sharing)

## Description

When DeltaFormatSharingSource restarts from a legacy
DeltaSharingSourceOffset checkpoint that is mid-version (index != -1),
the stream must finish the current version before transitioning to the
new DeltaSourceOffset format.

- Lucky case (legacy offset at a version boundary, index == -1):
transition immediately to DeltaSourceOffset, fetch file IDs with SHA256,
and continue with normal multi-version batches.
- Unlucky case (legacy offset mid-version): restrict the fetch to a
single version using MD5 file IDs (matching the legacy hash), and
convert the end offset back to legacy format until the stream reaches a
version boundary.
- Propagate fileIdHash through getTableFileChanges,
DeltaSharingUtils.getRefresherForGetFiles,
getRefresherForGetFilesWithStartingVersion, and
getRefresherForGetCDFFiles so the server returns file IDs with the
correct hash algorithm.

## How was this patch tested?

New test cases

## Does this PR introduce _any_ user-facing changes?

No

---------

Co-authored-by: Sun Cao <sun.cao+data@databricks.com>
Co-authored-by: Rakesh Veeramacheneni <173086727+raveeram-db@users.noreply.github.com>
…trings (delta-io#6301)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [X] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

This is **PR 1 of 3** for geospatial data skipping support. It teaches
the stats system (JSON parsing, schema validation, DataFileStatistics)
to handle geometry/geography columns as WKT string values, and adds
GeometryUtils for WKT POINT parsing/formatting.

Delta spec: delta-io#4726

### Changes
- **StatsSchemaHelper**: Mark `GeometryType`/`GeographyType` as eligible
for min/max stats
- **JsonUtils**: Handle geometry/geography as WKT string in
`parseJsonValueToLiteral()`
- **SchemaUtils**: Add geometry/geography to supported types in
`validateSupportedType()`
- **DataFileStatistics**: Validate geometry/geography stats accept
string literals
- **DefaultJsonRow**: Decode geometry/geography JSON values as strings
- **GeometryUtils** (new): WKT POINT parsing (`parsePointXY`) and
formatting (`formatPointWKT`)

### PR dependency graph
```
master
  |
  +-- PR A (this): Stats Parsing + GeometryUtils
  |     |
  |     +-- PR B: Expressions + Predicate Translation
  |     |
  |     +-- PR C: Parquet / Data Layer
```

## How was this patch tested?

- `GeometryUtilsSuite`: 15 tests covering WKT parsing/formatting,
round-trips, error cases
- `DefaultJsonHandlerSuite`: pending test placeholder for
geometry/geography parsing

## Does this PR introduce _any_ user-facing changes?

Adds new table feature

---------

Co-authored-by: Uros Bojanic <uros.bojanic@databricks.com>
…elta-io#6355)

## Summary
- Add support for converting Spark's `AlwaysTrue` and `AlwaysFalse`
filters to Kernel `ALWAYS_TRUE` and `ALWAYS_FALSE` predicates in the V2
connector's `ExpressionUtils`.
- `AlwaysFalse` pushdown allows the Kernel to skip all files entirely,
while `AlwaysTrue` is a correct no-op passthrough.
- These were the only two Spark filter types with direct Kernel
equivalents that were missing from the V2 connector's filter conversion.

## Test plan
- [x] Added `testAlwaysTrueFilter` and `testAlwaysFalseFilter` unit
tests in `ExpressionUtilsTest`
- [x] All 74 tests pass in `ExpressionUtilsTest`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…lta-io#6155)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6155/files) to
review all changes.

**Stack:**
- **[[UC Commit Metrics] Add skeleton transport wiring and smoke
tests](delta-io#6155 [[Files
changed](https://github.com/delta-io/delta/pull/6155/files)] ⬅️ _This
PR_
- [[UC Commit Metrics] Add full payload construction and schema
tests](delta-io#6156) [[Files
changed](https://github.com/delta-io/delta/pull/6156/files/4b5eadb28ab152d9ac180423bfd51720b6219b5b..1f39302986a7b6138cd358f9515386fc283810a8)]
- [[UC Commit Metrics] Add feature flag and async
dispatch](delta-io#6333) [[Files
changed](https://github.com/delta-io/delta/pull/6333/files/1f39302986a7b6138cd358f9515386fc283810a8..7b6d633980419060404203551bb1577ccfe922aa)]

---

## Summary
- Self-contained post-commit hook in
`hooks/metrics/UpdateMetricsHook.scala` (case class + companion object
with payload builder, HTTP client, and constants).
- Registered only for UC-managed tables
(`CatalogTableUtils.isUnityCatalogManagedTable` check at registration
time).
- Derives endpoint from `spark.sql.catalog.<name>.uri`, authenticates
via `UCCommitCoordinatorBuilder` + `TokenProvider`.
- Best-effort: failures logged as warnings, never fail the commit.

## Test plan
- [x] `build/sbt "spark/testOnly *metrics.UpdateMetricsHookSuite"` -- 6
tests pass

---------

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
Signed-off-by: foo.bar <foo.bar@foobar.com>
Co-authored-by: foo.bar <foo.bar@foobar.com>
…ta-io#6377)

## 🥞 Stacked PR
Use this [link](https://github.com/delta-io/delta/pull/6377/files) to
review incremental changes.
-
[**stack/ddl-1-foundations**](delta-io#6377)
[[Files changed](https://github.com/delta-io/delta/pull/6377/files)]
- [stack/ddl-2-builder](delta-io#6378)
[[Files
changed](https://github.com/delta-io/delta/pull/6378/files/b8c478ac19da5afb5aea0d8ce25dfa744b689e4f..a652586290caadbe79201d594ecf0e95f129b328)]
- [stack/ddl-2b-builder](delta-io#6442)
[[Files
changed](https://github.com/delta-io/delta/pull/6442/files/a652586290caadbe79201d594ecf0e95f129b328..88c17741e4c8f17a56c4a0523c1b744c36789c95)]
- [stack/ddl-3-wiring](delta-io#6379)
[[Files
changed](https://github.com/delta-io/delta/pull/6379/files/88c17741e4c8f17a56c4a0523c1b744c36789c95..113ebc006446bd7645be6a75c900104b0b0ed8a4)]

---------
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?

## Description
In this stack of PRs we are implementing the foundation for DDL
operations (metadata-only CREATE, CTAS, RTAS).

For context, CREATE TABLE in this DSv2 world is 3 main steps:
1. pre-register the table to the catalog (UCProxy.createTable or
HMS.createTable)
 2. build a transaction for table creation (Delta commit 0)
 3. commit (actually commit the transaction above)

**We also want to make implementing future DDL operations easier in the
future**. That's why in this PR we create a unified DDL request POJO.
This POJO/request will be fed into the transaction builder.

**Significance**: CreateTableBuilder.prepare() does all the Spark →
Kernel conversion - DDLContext is a normalized request for downstream
Kernel operations.
> By the time DDLContext is constructed, everything is Kernel-native.
buildTransaction() just passes it straight through to TableManager or
UCCatalogManagedClient with no further conversion.
```
                                  SQL: CREATE TABLE t (id INT, name STRING)
                                                       │
                                                       ▼
                                 ┌──────────────────────────────────────────┐
                                 │  UCSingleCatalog.createTable(            │
                                 │    ident, schema, partitions, props)     │
                                 │                                          │
                                 │  Spark's UC catalog plugin. Delegates    │
                                 │  to DeltaCatalog for Delta tables.       │
                                 └─────────────────────┬────────────────────┘
                                                       │
                                                       ▼
  ┌──────────────────────────────────────────────────────────────────────────────────────────┐
  │  DeltaCatalog.createTable(ident, schema, partitions, properties)                         │
  │                                                                                          │
  │  1. DeltaV2Mode routing check                                                            │
  │     STRICT → always Kernel path                                                          │
  │     AUTO   → Kernel only if UC + catalogManaged feature flag                             │
  │     NONE   → V1 path (super.createTable)                                                 │
  │                                                                                          │
  │  2. If UC: preRegisterIfNeeded(ident, schema, parts, props)                              │
  │     └→ Optional<CatalogTable>                                                            │
  │     then injectStorageCredentials(ct, hadoopConf)                                        │
  │     then UCUtils.extractTableInfo(ct, spark) → Optional<UCTableInfo>                     │
  │                                                                                          │
  │  3. CreateTableBuilder.prepare(ident, schema, parts, props, hadoopConf, ucTableInfo)     │
  │     └→ DDLContext                                                                        │
  │                                                                                          │
  │  4. CreateTableBuilder.buildTransaction(ddlContext) → Transaction                        │
  │                                                                                          │
  │  5. txn.commit(engine, emptyIterable()) — metadata-only                                  │
  │                                                                                          │
  │  6. return new SparkTable(ident, tablePath) — V2 connector                               │
  └──────┬─────────────────────┬──────────────────────────┬──────────────────────┬───────────┘
         │                     │                          │                      │
         ▼                     ▼                          ▼                      ▼

       STEP 2                STEP 3                     STEP 4                 STEP 5
       Pre-register          Prepare                    Build Txn              Commit

       UC-only.              Converts Spark             Routes to Kernel       txn.commit(
       Calls delegate        types to Kernel:            API based on            engine,
       .createTable()                                    ucTableInfo:            emptyIterable()
       via UCProxy.          sparkSchema →                                       )
                              kernelSchema              ┌── UC present? ───┐
       UC assigns:            (SchemaUtils)             │                  │   Kernel handles
       • tableId                                        │  YES             │   conflict
       • managed             Transform[] →              │  UCCatalog       │   resolution,
         location             DataLayoutSpec            │  ManagedClient   │   protocol,
       • ucUri                (identity or              │  .buildCreate    │    and log write.
       • authConfig           cluster-by)               │  TableTxn(       │
                                                        │    tableId,      │   For UC: commit
       Returns               properties →               │    path,         │   goes through
       Optional               filtered props            │    schema,       │   UC committer.
       <CatalogTable>         (strip DSv2 keys:         │    engineInfo)   │
                               location, provider,      │  + UC committer  │   For path-based:
       Non-UC: returns         comment, owner,          │  + UC properties │   writes directly
       Optional.empty()        external, path,          │                  │   to filesystem.
                               option.path)             │  NO              │
       On failure:                                      │  TableManager    │   Returns
       drops pre-            hadoopConf →               │  .buildCreate    │    TransactionCommit
       registered             Engine                    │  TableTxn(       │   Result with:
       catalog entry          (DefaultEngine)           │    path,         │   • version (0)
                                                        │    schema,       │   • postCommit
                             path resolved:             │    engineInfo)   │     Snapshot
                              UC → ucTableInfo          └───────┬──────────┘
                                  .tablePath
                              non-UC → props            Then applies:
                                  ["location"]          • .withTableProperties(filtered)
                                  (throws if            • .withDataLayoutSpec(layout)
                                  none found)           • .build(engine) → Transaction

                             Returns:
                             DDLContext POJO
                             (ident, tablePath,
                              kernelSchema,
                              filteredProps,
                              comment,
                              dataLayoutSpec,
                              engine,
                              ucTableInfo,
                              engineInfo)
```

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
…6438)

## Description

Add `DeltaV2TableManager` trait to `v2/interop/` that abstracts
`DeltaLog`'s role as the entry point for Delta table operations.
`DeltaLog` extends this trait.

This is the foundation for migrating Delta protocol reading from
`DeltaLog` to Kernel. The trait defines the minimal surface that
commands and `OptimisticTransaction` need from the table manager:

- Table identity: `logPath`, `dataPath`, `clock`, `options`, `tableId`
- Configuration: `newDeltaHadoopConf()`
- Snapshot access: `snapshot`, `unsafeVolatileSnapshot`, `tableExists`
- Setup: `createLogDirectoriesIfNotExists()`

With this trait in place, a future `KernelTableManager` can implement
the same interface via Kernel instead of log replay, and all existing
business logic (commands, `OptimisticTransaction`) works with either
backend.

**This PR has zero behavior change** — `DeltaLog` already implements
every method defined in the trait, so `with DeltaV2TableManager` is a
no-op.

## How was this patch tested?

All existing `DeltaLog` tests exercise the new trait. No test changes
needed.

## Does this PR introduce _any_ user-facing changes?

No.
… test (delta-io#6348)

## Description

Fixes a flaky test in `DeltaRetentionWithCatalogOwnedBatch1Suite` that
fails intermittently with:

```
20 did not equal 0 Delta files before the last checkpoint version should have been deleted
(DeltaRetentionSuite.scala:152)
```

### Root Cause

The test `"log files being already deleted shouldn't fail log deletion
job"` fails **only when executed between 00:00-01:00 UTC**. The
mechanism:

1. `getStartTimeForRetentionTest()` returns today at 01:00 UTC — which
is a **future** timestamp if the test runs before 01:00 UTC.
2. Delta files get `setLastModified` to this future time (01:00:xx UTC).
3. V2 checkpoint files (introduced by CatalogOwned in delta-io#5841) use
UUID-based names (`*.checkpoint.<uuid>.json`) that don't match
`checkpointFileSingular` format (`*.checkpoint.parquet`), so the test
never sets their timestamps — they keep their real wallclock time
(~00:04 UTC).
4. `BufferingLogDeletionIterator` checks `needsTimeAdjustment()`
**before** `isCheckpointFile()`. When a V2 checkpoint (modTime=00:04)
follows a delta file (modTime=01:01), the checkpoint's earlier real
timestamp triggers time adjustment, causing it to be treated as a
regular file instead of a checkpoint boundary. The buffer is never
flushed, and zero files get deleted.

### Fix

Replace the `checkpointFileSingular` lookup (lines 133-136) with the
`getCheckpointFiles` + `checkpointVersion` pattern already used and
working correctly elsewhere in the same file (lines 724-734). This finds
all checkpoint file formats including V2.

### Verified

Reproduced the failure by forcing `getStartTimeForRetentionTest` to
return a future timestamp:
- **Without fix**: `FAILED` — `20 did not equal 0` (identical to CI
failures)
- **With fix**: `PASSED`

Confirmed all CI failures occurred in the 00:00-01:00 UTC window:
00:04:33, 00:13:10, 00:21:48, 00:04:47 UTC.

## How was this patch tested?

Existing test `DeltaRetentionWithCatalogOwnedBatch1Suite."log files
being already deleted shouldn't fail log deletion job"` — this PR fixes
it.

## Does this PR introduce _any_ user-facing changes?

No. Test-only change.

This pull request was AI-assisted by Isaac.
…-io#6443)

Extract the file name construction logic from
DelayedCommitProtocol.getFileName into a shared companion object method
DelayedCommitProtocol.buildFileName.

This enables reuse of the standard Delta file naming convention
(part-NNNNN-uuid.ext / cdc-NNNNN-uuid.ext) by both the V1 write path
(DelayedCommitProtocol) and the DSv2 write path (DataWriter), ensuring
consistent file naming across both code paths.

Changes:
- Move file name building logic to DelayedCommitProtocol.buildFileName
companion method
- Simplify getFileName to delegate to the new shared method
- Preserve all existing behavior (test prefix, CDC prefix, split
formatting)
… translation

Add the public StIntersectsBoxes expression and its internal translation
to ST_INTERSECTS_BOXES_ON_STATS for geospatial data skipping:
- StIntersectsBoxes: public predicate taking column + WKT query bounds
- DataSkippingUtils: translate to stats-based predicate with min/max columns
- ExpressionVisitor: dispatch ST_INTERSECTS_BOXES_ON_STATS
- DefaultExpressionEvaluator: transform (type validation) and eval (AABB
  intersection via WKT parsing)
- GeometryDataSkippingSuite: end-to-end tests with 4-quadrant layout,
  null stats, and combined AND predicates

Co-authored-by: Isaac
@milastdbx
milastdbx force-pushed the geospatial-expressions branch from 59bdcaa to 04dd064 Compare March 31, 2026 10:18
@milastdbx milastdbx closed this Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.