Skip to content

Write dataChange on delta-format cdc actions - #187

Closed
littlegrasscao wants to merge 1 commit into
OpenSharing-IO:mainfrom
littlegrasscao:fix/cdc-data-change
Closed

littlegrasscao wants to merge 1 commit into
OpenSharing-IO:mainfrom
littlegrasscao:fix/cdc-data-change

Conversation

@littlegrasscao

Copy link
Copy Markdown
Collaborator

Summary

Delta-format Change Data Feed responses wrote cdc actions without dataChange. Commit files are read with Kernel's DeltaAction.CDC.schema, which in Kernel 4.4.1 has only path, partitionValues, size, and tags, so the field was dropped even when the commit had it.

Delta readers require the field. The Delta Sharing Python connector reads delta-format CDF through delta-kernel-rs, which rejects the response:

pyarrow.lib.ArrowInvalid: External error: Json error: whilst decoding field 'cdc':
Encountered unmasked nulls in non-nullable StructArray child: Field { "dataChange": Boolean ... }

Changes

  • DeltaTableQueryReader: reads cdc actions with Kernel's schema plus dataChange, and writes dataChange: false on every cdc action, the value the Delta protocol fixes for them, also when a commit omits it.
  • DeltaTableQueryReaderTest.changeDataFeedPrefersCdcFilesAndSkipsNoOpMerges: asserts the delta-format cdc line has "dataChange":false.

Parquet-format responses are unchanged.

Test plan

  • DeltaTableQueryReaderTest fails before the fix (the cdc line is {"path":…,"partitionValues":…,"size":1}) and passes after it
  • Full opensharing-server unit tests: 170 run, 0 failures. The 7 errors are in S3-backed integration classes and occur on untouched origin/main in this environment too, because the cloud cases run when AWS_ACCESS_KEY_ID is set but no region is configured.
  • Cloud integration tests on S3 (not run here)

Signed-off-by: Sun Cao sun.cao+data@databricks.com

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

Change Data Feed commits were read with Kernel's cdc schema, which has
no dataChange, so delta-format responses wrote cdc actions without it
even when the commit had it. Delta readers require the field: the
Python connector's delta-kernel-rs reader rejects the response with
"Encountered unmasked nulls in non-nullable StructArray child: Field
dataChange".

Read dataChange with the cdc action and write it as false, which the
Delta protocol fixes for cdc actions, also when a commit omits it or
writes it as true. The override lives with the response writer, next to
the path presigning override, so the reader only carries the slot.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao

Copy link
Copy Markdown
Collaborator Author

Superseded by #192, which carries the same commit from an OpenSharing-IO branch so it can be the base of a stacked PR (#192 → #193).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant