Repository navigation
Send metaData.version only for historical CDF metadata - #188
littlegrasscao wants to merge 2 commits into
Conversation
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s). |
3ac4c27 to
986c374
Compare
986c374 to
2d25985
Compare
Change Data Feed commits were read with Kernel's cdc schema, which has no dataChange, so delta-format responses wrote cdc actions without it even when the commit had it. Delta readers require the field: the Python connector's delta-kernel-rs reader rejects the response with "Encountered unmasked nulls in non-nullable StructArray child: Field dataChange". Read dataChange with the cdc action and write it as false, which the Delta protocol fixes for cdc actions, also when a commit omits it or writes it as true. The override lives with the response writer, next to the path presigning override, so the reader only carries the slot. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
2d25985 to
48c3abb
Compare
|
Review follow-up for the run that routed one finding here: the format reshaping at TableActions.reshaped now keeps only Delta Sharing's format fields - This comment was written by Isaac. |
9dc6897 to
2e3f0fa
Compare
|
Follow-up: the field reshaping introduced for review has been removed. JSON key order is not significant to any client, so responses now write action fields in their natural (Kernel) order, and the delta-sharing suite's exact-substring checks are being made order-insensitive on the suite side. The response content alignment - metaData.version rules, checksum size/numFiles, historical file version and timestamp, and per-kind tag redaction - is unchanged and still tested (TableActionsTest, DeltaTableQueryReaderTest, ProtocolTableMetadataTest, ProtocolTableQueryTest). Current head: 2e3f0fa. This comment was written by Isaac. |
Query Table and Query Table Metadata responses echoed the queried version into metaData. Delta Sharing sends metaData.version only on a Change Data Feed response with includeHistoricalMetadata: the selected version of a version or timestamp query travels only in the Delta-Table-Version header, and a Change Data Feed without includeHistoricalMetadata carries no version at all. OpenSharing echoed the version in metaData, which clients compare as whole metadata objects. Send metaData.version only for the Change Data Feed's historical metadata, keeping size and numFiles as the snapshot's checksum provides them, and a historical snapshot file's version and timestamp as the commit records them. Delta-format responses also align each action's content with Delta Sharing: an add's and a cdc action's tags are redacted to an empty object, a remove carries no tags, a metaData's format writes its options as an empty object when the log omits them, and a cdc action's dataChange is false as the protocol requires. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
2e3f0fa to
72f9fa4
Compare
Depends on #187
Problem
OpenSharing's Query Table and Query Table Metadata responses echoed the queried version into metaData on version and timestamp queries. Delta Sharing sends
metaData.versiononly on a Change Data Feed response withincludeHistoricalMetadata=true; the selected version of a version or timestamp query travels only in theDelta-Table-Versionheader. Clients that compare whole metadata objects fail on the extra field.Aligned response content
metaData.versionis null for latest and for version/timestamp metadata and query responses; the selected version is only in theDelta-Table-Versionheader. A Change Data Feed response withincludeHistoricalMetadata=truecarries the historical metaData's version, and the per-commit metaData of astartingVersionquery keeps its commit's version. A Change Data Feed withoutincludeHistoricalMetadatacarries no version.metaData.sizeandnumFilesare returned whenever the snapshot's checksum carries the table's size and file count - latest, version, timestamp, andstartingVersionresponses alike - and are omitted when no checksum exists for exactly the snapshot's version. Change Data Feed metaData carries neither.tagsare redacted to an empty object, a remove carries no tags, a metaData's format writesoptionsas an empty object when the log omits them, and a cdc action'sdataChangeis false.Change
DeltaTableMetadataReaderandDeltaTableQueryReaderpassversion = nullinto metaData for metadata and query responses, and no checksum stats for Change Data Feed responses; the version stays in theDelta-Table-Versionheader.TableActionsredacts tags and defaults a format'soptionson the Kernel rows it already writes; no reordering of action fields (key order is not significant to clients).Testing
Java 17,
mvn -pl opensharing-server-core,opensharing-server clean test: 646 tests, 0 failures. Tests cover metaData.version absence for historical queries, size/numFiles from a checksum fixture (present for latest and version queries, absent without a checksum), file version/timestamp on historical queries and neither on latest snapshots in both formats, and per-kind tag redaction through commit fixtures.This pull request and its description were written by Isaac.