Skip to content

[Prototype] Add CDC (Change Data Feed) streaming support to DSv2 connector - #8

Closed
zikangh wants to merge 1 commit into
masterfrom
stack/cdc-streaming
Closed

zikangh wants to merge 1 commit into
masterfrom
stack/cdc-streaming

Conversation

@zikangh

@zikangh zikangh commented Feb 9, 2026

Copy link
Copy Markdown
Owner

Add end-to-end CDC streaming support for the Kernel-based V2 Delta connector, enabling readChangeFeed=true for streaming queries. Key changes:

  • IndexedFile: Track CDC metadata (changeType, commitTimestamp, CDCFileInfo)
  • SparkMicroBatchStream: CDC file change iteration (explicit CDC files + inferred from AddFile/RemoveFile), CDF validation, ensureCDCColumnsInSchema
  • SparkPartitionReader: In-schema CDC column override via CDCOverrideRow (row path) and constant OnHeapColumnVectors (columnar path), replacing JoinedRow approach
  • PartitionUtils: CDC PartitionedFile builders with constant metadata columns, vectorized reading enabled for CDC
  • SparkScan/SparkScanBuilder: CDC column handling in readSchema and pruneColumns
  • StreamingHelper: Extract AddCDCFile and RemoveFile actions from kernel batches
  • ApplyV2Streaming: Include CDC columns in schema for V2 streaming
  • DeltaV2CDCStreamSuite: Integration tests for initial snapshot, deletes, and startingVersion=latest

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

@zikangh
zikangh force-pushed the stack/cdc-streaming branch from 373103b to e56413e Compare February 12, 2026 22:46
@zikangh
zikangh force-pushed the stack/cdc-streaming branch from e56413e to d1e275e Compare April 16, 2026 01:07
@zikangh zikangh closed this Apr 16, 2026
@zikangh
zikangh deleted the stack/cdc-streaming branch April 16, 2026 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant