Repository navigation
Write dataChange on delta-format cdc actions - #192
Merged
Merged
Conversation
littlegrasscao
requested review from
chakankardb,
hueyhuilonghan and
linzhou-db
as code owners
November 13, 2026 02:11
This was referenced Nov 13, 2026
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). |
Change Data Feed commits were read with Kernel's cdc schema, which has no dataChange, so delta-format responses wrote cdc actions without it even when the commit had it. Delta readers require the field: the Python connector's delta-kernel-rs reader rejects the response with "Encountered unmasked nulls in non-nullable StructArray child: Field dataChange". Read dataChange with the cdc action and write it as false, which the Delta protocol fixes for cdc actions, also when a commit omits it or writes it as true. The override lives with the response writer, next to the path presigning override, so the reader only carries the slot. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
littlegrasscao
force-pushed
the
fix/cdc-data-change
branch
from
November 13, 2026 07:49
8c18e77 to
1a65097
Compare
zhu-tom
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #187 (moved to an OpenSharing-IO branch for the stack). Bottom of the stack; #193 builds on it.
Summary
Delta-format Change Data Feed responses wrote
cdcactions withoutdataChange. Commit files are read with Kernel'sDeltaAction.CDC.schema, which in Kernel 4.4.1 has onlypath,partitionValues,size, andtags, so the field was dropped even when the commit had it.Delta readers require the field. The Delta Sharing Python connector reads delta-format CDF through delta-kernel-rs, which rejects the response:
Changes
DeltaTableQueryReader: readscdcactions with Kernel's schema plus adataChangeslot.TableActions: writesdataChange: falseon every delta-formatcdcaction, the value the Delta protocol fixes for them, also when a commit omits it or writes it as true. The override sits next to the path-presigning override, so the reader only carries the slot.DeltaTableQueryReaderTest: asserts the parsedcdcdataChangeis boolean false, that a commit writingtruestill answersfalse, that parquet-format cdc lines carry nodataChange, and that add and remove keep the log's value.Parquet-format responses are unchanged.
Test plan
DeltaTableQueryReaderTestfails before the fix and passes after itmvn -pl opensharing-server-core,opensharing-server clean test(Java 17): all tests pass; the S3-backed integration classes need cloud credentials and region and were not run hereThis pull request and its description were written by Isaac.