Repository navigation
Refuse tables whose Delta properties the client did not advertise - #189
littlegrasscao wants to merge 5 commits into
Conversation
|
🔍 OpenCodeReview found 5 issue(s) in this PR.
📄
|
4ad8fa3 to
9773bdb
Compare
…ame the data provider A shared table whose Delta log is missing, starts after version 0, has a gap, or is otherwise invalid, or whose version Kernel cannot read, returned 500 INTERNAL_ERROR or an error code and message of this server's own. The recipient cannot fix any of these, so the answer now says what is wrong and, when the shared table itself is at fault, whom to contact. DeltaKernel classifies every Kernel failure once, by Kernel's exception type rather than its message text: - A missing log (TableNotFoundException), or a file that is missing or denied (FileNotFoundException, AccessDeniedException), is the shared table's own state: 400 BAD_REQUEST "Unable to access table <share>.<schema>.<table>: something is wrong with the table. Please contact your data provider." The table is named as the share names it, and its storage path stays in the log. - An InvalidTableException, such as a gap in the log, is reported in Kernel's own words, with every spelling of the table's path and any other location removed, and the recipient is asked to contact the data provider: 400 BAD_REQUEST. - Any other Kernel report, such as a version or timestamp the table does not have, is the request's mistake in Kernel's words, without the path: 400 BAD_REQUEST. - A failure of the storage layer for a reason of its own stays a server error. - A reader feature or version Kernel cannot read is named in a fixed order: 400 INVALID_PARAMETER_VALUE "Table features <features> are found in table[ version: <n>]. This server cannot read them. Please contact your data provider." The version endpoint no longer pre-checks a timestamp against the latest commit and leaves that to Kernel, and a snapshot is opened straight at the version or timestamp asked for. Adds ErrorCodes.BAD_REQUEST, ApiException.badRequest and ApiException.unsupportedFeatures, and fixtures for a log with in-commit timestamps but no CommitInfo (in a directory with a space in its name) and a table that is readable at its latest version but not at version 0. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
9773bdb to
255e713
Compare
A request with a version or timestamp the table does not have, a malformed or contradictory range, or a Change Data Feed the source table does not record got errors of this server's own wording, and some leaked the table's storage path. Each now has one specific answer: - A non-numeric version, startingVersion or endingVersion is 400 INVALID_PARAMETER_VALUE "<name> is not a valid number.", and a negative one "<name> cannot be negative." - More than one of version, timestamp and startingVersion, or more than one starting or ending bound of a Change Data Feed, names the parameters to choose from. A reversed range names both versions. - A Change Data Feed start or end after the table's latest version, or an end timestamp after its latest commit, is rejected with the latest version or commit time to use, rather than capped. - A table that did not record change data for a version in the range is the source table's own state: 403 PERMISSION_DENIED "Unable to access change data feed for <share>.<schema>.<table>. CDF is not enabled on the original delta table for version <n>. Please contact your data provider." - A version or timestamp Kernel rejects keeps Kernel's own message, in 400 BAD_REQUEST, without the table's storage path. A Change Data Feed range is read whole for its protocols and metaData before the response begins, so a version that stops recording change data, or needs reader features the client did not advertise, answers with its status instead of resetting a stream that has begun. The checks and their messages live in TableRequestChecks, and run once, in ProtocolTables, before the table is read. The snapshot and metadata readers no longer check for a version together with a timestamp. Stacked on the change that classifies Kernel's failures. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
A table whose Delta properties enable a reader feature that the recipient's client did not advertise cannot be served to that client: column mapping by name (delta.columnMapping.mode) and deletion vectors (delta.enableDeletionVectors) are only readable by a client that declares them in delta-sharing-capabilities. Such a request now answers 400 INVALID_PARAMETER_VALUE "Table features <properties> are found in table version: <n>. Query with responseFormat=delta and advertise the features in delta-sharing-capabilities, or ask your data provider to disable them." on every snapshot the server serves: the metadata, query and change answers, for the latest and for a historical version, and for the metadata of each commit of a change feed, before the response is written; a change feed's commits are read whole first. A property set to its off value never blocks a request, and a property no feature depends on is ignored. The same answer, with the same wording, now reports reader features the client did not advertise, naming every one of them in a fixed order with the table version, where it named the first. Deletion vectors stay in a protocol after they are turned off, so only their property decides. The advertised reader features are parsed once per request, and DeltaSharingCapabilities keeps its properties in an ordered list. Adds column-mapping, deletion-vector, properties-off and later-column- mapping tables to the local fixtures. Stacked on the change that answers version, range and change-data mistakes. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
255e713 to
519ffbd
Compare
Delta reads delta.columnMapping.mode and delta.enableDeletionVectors without regard to case, so "NONE" and "False" turn their features off. The check no longer refuses a table that sets them so. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
A denied read of the table's files can be this server's own expired or misconfigured credential, so it is logged at error level for operators, while the recipient is still told to contact the data provider. The table path is also removed when Kernel's message spells it fully percent-encoded, and tests cover s3, s3a, abfss, gs and file locations in each spelling. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
Depends on #190 (which depends on #186).
A table whose Delta properties enable a reader feature that the recipient's client did not advertise cannot be served to that client: column mapping by name (
delta.columnMapping.mode) and deletion vectors (delta.enableDeletionVectors) are only readable by a client that declares them indelta-sharing-capabilities. Such a request now answers 400INVALID_PARAMETER_VALUE:It applies to every snapshot the server serves: metadata, query and changes, for the latest and for a historical version, and for the metadata of each commit of a change feed, which is read whole before the response begins. A property at its off value never blocks a request, and an unrelated property is ignored.
The same answer now reports reader features the client did not advertise, naming all of them in a fixed order with the table version. Deletion vectors stay in a protocol after they are turned off, so only their property decides. A parquet-only client on a table that needs reader features gets the same answer.
The advertised features are parsed once per request, and the checked properties are kept in an ordered list.
Test plan
opensharing-server-coreandopensharing-servertest runs pass locally (Java 17, Maven). S3-backed integration classes skip without cloud credentials.This pull request and its description were written by Isaac.