Skip to content

Write dataChange on delta-format cdc actions - #192

Merged
littlegrasscao merged 1 commit into
mainfrom
fix/cdc-data-change
Nov 13, 2026
Merged

littlegrasscao merged 1 commit into
mainfrom
fix/cdc-data-change

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Replaces #187 (moved to an OpenSharing-IO branch for the stack). Bottom of the stack; #193 builds on it.

Summary

Delta-format Change Data Feed responses wrote cdc actions without dataChange. Commit files are read with Kernel's DeltaAction.CDC.schema, which in Kernel 4.4.1 has only path, partitionValues, size, and tags, so the field was dropped even when the commit had it.

Delta readers require the field. The Delta Sharing Python connector reads delta-format CDF through delta-kernel-rs, which rejects the response:

pyarrow.lib.ArrowInvalid: External error: Json error: whilst decoding field 'cdc':
Encountered unmasked nulls in non-nullable StructArray child: Field { "dataChange": Boolean ... }

Changes

  • DeltaTableQueryReader: reads cdc actions with Kernel's schema plus a dataChange slot.
  • TableActions: writes dataChange: false on every delta-format cdc action, the value the Delta protocol fixes for them, also when a commit omits it or writes it as true. The override sits next to the path-presigning override, so the reader only carries the slot.
  • DeltaTableQueryReaderTest: asserts the parsed cdc dataChange is boolean false, that a commit writing true still answers false, that parquet-format cdc lines carry no dataChange, and that add and remove keep the log's value.

Parquet-format responses are unchanged.

Test plan

  • DeltaTableQueryReaderTest fails before the fix and passes after it
  • mvn -pl opensharing-server-core,opensharing-server clean test (Java 17): all tests pass; the S3-backed integration classes need cloud credentials and region and were not run here
  • Cloud integration tests on S3 (not run here)

This pull request and its description were written by Isaac.

@github-actions

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

Change Data Feed commits were read with Kernel's cdc schema, which has
no dataChange, so delta-format responses wrote cdc actions without it
even when the commit had it. Delta readers require the field: the
Python connector's delta-kernel-rs reader rejects the response with
"Encountered unmasked nulls in non-nullable StructArray child: Field
dataChange".

Read dataChange with the cdc action and write it as false, which the
Delta protocol fixes for cdc actions, also when a commit omits it or
writes it as true. The override lives with the response writer, next to
the path presigning override, so the reader only carries the slot.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao
littlegrasscao merged commit d13248f into main Nov 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants