Repository navigation
Write dataChange on delta-format cdc actions - #187
Closed
littlegrasscao wants to merge 1 commit into
Closed
littlegrasscao wants to merge 1 commit into
littlegrasscao wants to merge 1 commit into
Conversation
littlegrasscao
requested review from
chakankardb,
hueyhuilonghan and
linzhou-db
as code owners
November 12, 2026 22:37
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). |
Change Data Feed commits were read with Kernel's cdc schema, which has no dataChange, so delta-format responses wrote cdc actions without it even when the commit had it. Delta readers require the field: the Python connector's delta-kernel-rs reader rejects the response with "Encountered unmasked nulls in non-nullable StructArray child: Field dataChange". Read dataChange with the cdc action and write it as false, which the Delta protocol fixes for cdc actions, also when a commit omits it or writes it as true. The override lives with the response writer, next to the path presigning override, so the reader only carries the slot. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
littlegrasscao
force-pushed
the
fix/cdc-data-change
branch
from
November 13, 2026 01:21
2c010ce to
8c18e77
Compare
This was referenced Nov 13, 2026
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Delta-format Change Data Feed responses wrote
cdcactions withoutdataChange. Commit files are read with Kernel'sDeltaAction.CDC.schema, which in Kernel 4.4.1 has onlypath,partitionValues,size, andtags, so the field was dropped even when the commit had it.Delta readers require the field. The Delta Sharing Python connector reads delta-format CDF through delta-kernel-rs, which rejects the response:
Changes
DeltaTableQueryReader: readscdcactions with Kernel's schema plusdataChange, and writesdataChange: falseon everycdcaction, the value the Delta protocol fixes for them, also when a commit omits it.DeltaTableQueryReaderTest.changeDataFeedPrefersCdcFilesAndSkipsNoOpMerges: asserts the delta-formatcdcline has"dataChange":false.Parquet-format responses are unchanged.
Test plan
DeltaTableQueryReaderTestfails before the fix (thecdcline is{"path":…,"partitionValues":…,"size":1}) and passes after itopensharing-serverunit tests: 170 run, 0 failures. The 7 errors are in S3-backed integration classes and occur on untouchedorigin/mainin this environment too, because the cloud cases run whenAWS_ACCESS_KEY_IDis set but no region is configured.Signed-off-by: Sun Cao sun.cao+data@databricks.com