Repository navigation
[Spark] Update changelog shims for Spark 4.2 API - #7222
Merged
tdas merged 5 commits intoJul 20, 2026
Merged
Conversation
murali-db
requested review from
huan233usc,
raveeram-db,
tdas and
timothyw553
as code owners
July 15, 2026 23:15
timothyw553
reviewed
Jul 16, 2026
timothyw553
reviewed
Jul 16, 2026
5 of 8 tasks
murali-db
force-pushed
the
stack/delta-changeloginfo-shim-spark42
branch
from
July 17, 2026 02:20
45f36ed to
2fed12b
Compare
timothyw553
approved these changes
Jul 17, 2026
timothyw553
left a comment
Collaborator
There was a problem hiding this comment.
approved with some nits
yili-db
approved these changes
Jul 17, 2026
Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
- Keep Delta CDF processing-option validation scoped to the legacy bridge - Preserve native Spark processing options and disabled feature-gate behavior - Make source-build metadata tests independent of the current Spark pin Signed-off-by: Murali Ramanujam <murali.ramanujam@databricks.com>
murali-db
force-pushed
the
stack/delta-changeloginfo-shim-spark42
branch
from
July 17, 2026 19:00
e7f4612 to
bae9283
Compare
murali-db
requested review from
timothyw553 and
yili-db
and removed request for
timothyw553 and
yili-db
July 17, 2026 19:10
7 of 12 tasks
tdas
requested changes
Jul 20, 2026
- Cover carryover removal with a multi-row base file - Use literal source-build artifact expectations
tdas
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
The Spark 4.2 source API used by Delta revised its changelog connector contract:
ChangelogInfoUtilsbecameChangelogContextUtils.ChangelogInfobecameChangelogContext.loadChangelog(Identifier, ChangelogInfo)becameloadChangelog(Identifier, ChangelogContext, CaseInsensitiveStringMap).This updates the Spark 4.2 shims to that API and advances the centrally configured Spark source ref to the matching commit. It intentionally retains the existing source-built Spark cache/test workflow. Moving the 4.2 lane to released
4.2.0, together with any compatibility changes exposed by that move, is being handled separately.The two changelog callers retain their existing semantics:
changes()path keeps standard processing options such as update pre/post images, inclusive or exclusive bounds, andnetChangesdeduplication.table_changesandreadChangeFeedbridge keeps its fixed carryover/update behavior and rejects caller-selected processing modes before Spark parses them.TableCatalogcontract and preserves the expected unsupported-feature error.The focused tests use the unified session extension so they exercise the Spark 4.2 DSv2 routing rule. Coverage includes native processing modes, Delta bridge option rejection, the disabled feature gate, and update image computation.
How was this patch tested?
Built and locally published the configured Spark 4.2 source target: 39 modules.
Ran
spark/testOnly io.delta.spark.internal.v2.read.changelog.*: 36 total, 35 passed, 1 pre-existing test skipped.Ran the source-build metadata helper tests, including synthetic resolver metadata: all passed.
Ran:
Does this PR introduce any user-facing changes?
No. This is a Spark 4.2 API compatibility update that preserves the existing changelog behavior.