Skip to content

Send metaData.version only for historical CDF metadata - #188

Closed
littlegrasscao wants to merge 2 commits into
OpenSharing-IO:mainfrom
littlegrasscao:fix/response-shape
Closed

littlegrasscao wants to merge 2 commits into
OpenSharing-IO:mainfrom
littlegrasscao:fix/response-shape

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #187

Problem

OpenSharing's Query Table and Query Table Metadata responses echoed the queried version into metaData on version and timestamp queries. Delta Sharing sends metaData.version only on a Change Data Feed response with includeHistoricalMetadata=true; the selected version of a version or timestamp query travels only in the Delta-Table-Version header. Clients that compare whole metadata objects fail on the extra field.

Aligned response content

  • metaData.version is null for latest and for version/timestamp metadata and query responses; the selected version is only in the Delta-Table-Version header. A Change Data Feed response with includeHistoricalMetadata=true carries the historical metaData's version, and the per-commit metaData of a startingVersion query keeps its commit's version. A Change Data Feed without includeHistoricalMetadata carries no version.
  • metaData.size and numFiles are returned whenever the snapshot's checksum carries the table's size and file count - latest, version, timestamp, and startingVersion responses alike - and are omitted when no checksum exists for exactly the snapshot's version. Change Data Feed metaData carries neither.
  • Snapshot file actions of a version or timestamp query carry both the version and that version's commit timestamp; latest-snapshot files carry neither, in both response formats. Change Data Feed files always carry their commit's version and timestamp.
  • Delta-format actions: an add's and a cdc action's tags are redacted to an empty object, a remove carries no tags, a metaData's format writes options as an empty object when the log omits them, and a cdc action's dataChange is false.

Change

  • DeltaTableMetadataReader and DeltaTableQueryReader pass version = null into metaData for metadata and query responses, and no checksum stats for Change Data Feed responses; the version stays in the Delta-Table-Version header.
  • TableActions redacts tags and defaults a format's options on the Kernel rows it already writes; no reordering of action fields (key order is not significant to clients).

Testing

Java 17, mvn -pl opensharing-server-core,opensharing-server clean test: 646 tests, 0 failures. Tests cover metaData.version absence for historical queries, size/numFiles from a checksum fixture (present for latest and version queries, absent without a checksum), file version/timestamp on historical queries and neither on latest snapshots in both formats, and per-kind tag redaction through commit fixtures.

This pull request and its description were written by Isaac.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

@littlegrasscao littlegrasscao changed the title Send metaData without version, size, or numFiles Send metaData.version only for historical CDF metadata Nov 13, 2026
Change Data Feed commits were read with Kernel's cdc schema, which has
no dataChange, so delta-format responses wrote cdc actions without it
even when the commit had it. Delta readers require the field: the
Python connector's delta-kernel-rs reader rejects the response with
"Encountered unmasked nulls in non-nullable StructArray child: Field
dataChange".

Read dataChange with the cdc action and write it as false, which the
Delta protocol fixes for cdc actions, also when a commit omits it or
writes it as true. The override lives with the response writer, next to
the path presigning override, so the reader only carries the slot.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao

littlegrasscao commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review follow-up for the run that routed one finding here: the format reshaping at TableActions.reshaped now keeps only Delta Sharing's format fields - provider then options, with options defaulting to an empty object - and drops any others. The reshaped node is also built without mutating the node being iterated, so unknown fields cannot trigger a ConcurrentModificationException. Both are covered by TableActionsTest.writesDeltaMetadataFieldsInDeltaSharingOrder and reshapesARemoveWithoutADeletionTimestampByItsTypeNotItsKeys. Fixed in 48c3abb.

This comment was written by Isaac.

@littlegrasscao
littlegrasscao force-pushed the fix/response-shape branch 2 times, most recently from 9dc6897 to 2e3f0fa Compare November 13, 2026 02:04
@littlegrasscao

Copy link
Copy Markdown
Collaborator Author

Follow-up: the field reshaping introduced for review has been removed. JSON key order is not significant to any client, so responses now write action fields in their natural (Kernel) order, and the delta-sharing suite's exact-substring checks are being made order-insensitive on the suite side. The response content alignment - metaData.version rules, checksum size/numFiles, historical file version and timestamp, and per-kind tag redaction - is unchanged and still tested (TableActionsTest, DeltaTableQueryReaderTest, ProtocolTableMetadataTest, ProtocolTableQueryTest). Current head: 2e3f0fa.

This comment was written by Isaac.

Query Table and Query Table Metadata responses echoed the queried
version into metaData. Delta Sharing sends metaData.version only on a
Change Data Feed response with includeHistoricalMetadata: the selected
version of a version or timestamp query travels only in the
Delta-Table-Version header, and a Change Data Feed without
includeHistoricalMetadata carries no version at all.

OpenSharing echoed the version in metaData, which clients compare as
whole metadata objects. Send metaData.version only for the Change Data
Feed's historical metadata, keeping size and numFiles as the snapshot's
checksum provides them, and a historical snapshot file's version and
timestamp as the commit records them.

Delta-format responses also align each action's content with Delta
Sharing: an add's and a cdc action's tags are redacted to an empty
object, a remove carries no tags, a metaData's format writes its
options as an empty object when the log omits them, and a cdc action's
dataChange is false as the protocol requires.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@littlegrasscao

Copy link
Copy Markdown
Collaborator Author

Superseded by #193, which carries this change stacked on #192 from an OpenSharing-IO branch. Earlier review threads here were addressed before the move.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant