Skip to content

[Spark] Update changelog shims for Spark 4.2 API - #7222

Merged
tdas merged 5 commits into
delta-io:masterfrom
murali-db:stack/delta-changeloginfo-shim-spark42
Jul 20, 2026
Merged

tdas merged 5 commits into
delta-io:masterfrom
murali-db:stack/delta-changeloginfo-shim-spark42

Conversation

@murali-db

@murali-db murali-db commented Jul 15, 2026 •

Copy link
Copy Markdown
Collaborator

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

The Spark 4.2 source API used by Delta revised its changelog connector contract:

  • ChangelogInfoUtils became ChangelogContextUtils.
  • ChangelogInfo became ChangelogContext.
  • loadChangelog(Identifier, ChangelogInfo) became loadChangelog(Identifier, ChangelogContext, CaseInsensitiveStringMap).

This updates the Spark 4.2 shims to that API and advances the centrally configured Spark source ref to the matching commit. It intentionally retains the existing source-built Spark cache/test workflow. Moving the 4.2 lane to released 4.2.0, together with any compatibility changes exposed by that move, is being handled separately.

The two changelog callers retain their existing semantics:

  • The native Spark DataFrame changes() path keeps standard processing options such as update pre/post images, inclusive or exclusive bounds, and netChanges deduplication.
  • The Delta table_changes and readChangeFeed bridge keeps its fixed carryover/update behavior and rejects caller-selected processing modes before Spark parses them.
  • With the feature gate disabled, the catalog shim follows the Spark default TableCatalog contract and preserves the expected unsupported-feature error.

The focused tests use the unified session extension so they exercise the Spark 4.2 DSv2 routing rule. Coverage includes native processing modes, Delta bridge option rejection, the disabled feature gate, and update image computation.

How was this patch tested?

  • Built and locally published the configured Spark 4.2 source target: 39 modules.

  • Ran spark/testOnly io.delta.spark.internal.v2.read.changelog.*: 36 total, 35 passed, 1 pre-existing test skipped.

  • Ran the source-build metadata helper tests, including synthetic resolver metadata: all passed.

  • Ran:

    python3.12 -m py_compile project/scripts/get_spark_version_info.py project/tests/test_cross_spark_publish.py
    bash -n build/sbt project/scripts/build_spark.sh
    git diff --check

Does this PR introduce any user-facing changes?

No. This is a Spark 4.2 API compatibility update that preserves the existing changelog behavior.

Comment thread project/CrossSparkVersions.scala Resolved
@murali-db
murali-db force-pushed the stack/delta-changeloginfo-shim-spark42 branch from 45f36ed to 2fed12b Compare July 17, 2026 02:20

@timothyw553 timothyw553 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with some nits

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
- Keep Delta CDF processing-option validation scoped to the legacy bridge
- Preserve native Spark processing options and disabled feature-gate behavior
- Make source-build metadata tests independent of the current Spark pin

Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
@murali-db
murali-db force-pushed the stack/delta-changeloginfo-shim-spark42 branch from e7f4612 to bae9283 Compare July 17, 2026 19:00
@murali-db
murali-db requested review from timothyw553 and yili-db and removed request for timothyw553 and yili-db July 17, 2026 19:10
- Cover carryover removal with a multi-row base file
- Use literal source-build artifact expectations
@tdas
tdas merged commit b417490 into delta-io:master Jul 20, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants