Repository navigation
Read WASB tables through ABFS at the blob host - #191
Closed
littlegrasscao wants to merge 1 commit into
Closed
littlegrasscao wants to merge 1 commit into
littlegrasscao wants to merge 1 commit into
Conversation
A wasb:// or wasbs:// table read failed with a 501 before the catalog was even asked for credentials, so the delta-sharing suites' share_azure tables (table_wasb and friends, at wasbs:// blob hosts) could not be read at all. Read them through ABFS at the same blob host: the location's container, host, and path carry over to an abfss:// path, the account is declared to have no hierarchical namespace as ABFS requires for a blob host, and the file URLs are signed on the blob host they already name. This restores the scheme set, kernel path, and SAS settings superseded by OpenSharing-IO#148, whose final state rejected WASB reads because no file URL could be signed for them; the blob host a wasb location names is exactly the host the signed URL needs, as OpenSharing-IO#148's first commit had it. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
littlegrasscao
requested review from
chakankardb,
hueyhuilonghan and
linzhou-db
as code owners
November 12, 2026 23:57
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). |
Collaborator
Author
|
Closing. OpenSharing serves Azure tables only through abfs/abfss and rejects wasb/wasbs locations; main already answers wasb reads with 501 NOT_IMPLEMENTED, so no change is needed. The branch stays available for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
wasb://orwasbs://table read fails with a 501 before the catalog is even asked for credentials (HadoopStorageConfiguration.requireSupported), so the delta-sharing suites'share_azuretables -table_wasb,table_wasb_change_query_id, and friends, all atwasbs://container@account.blob.core.windows.net/...locations - cannot be read by OpenSharing at all. The unmodified client, Spark, and Python suites exercise these tables, so every one of those tests fails.Change
Read WASB locations through ABFS at the same blob host, restoring the approach from the first commit of #148 before its final state replaced it with the 501 rejection:
StorageLocations.AZURE_SCHEMESincludeswasbandwasbsagain, so both the URL signer and table reads pick them up through the shared scheme set.HadoopStorageConfiguration.kernelPathtranslates awasb(s)://container@account.blob.core.windows.net/keylocation toabfss://at the same host, where Hadoop's ABFS filesystem reads it.configureAzuredeclares a.blob.host's account as having no hierarchical namespace (fs.azure.account.hns.enabled.<host>=false), which ABFS requires to read a blob host.account.blob.core.windows.nethost the signed URL uses, andAzureSasUrlSignerbuilds exactly that URL.This supersedes part of #148, whose final commit rejected WASB reads "as no file URL could be signed for it". A wasb location's host is precisely the blob host the signed URL needs, as #148's own first commit (and its
readsWasbThroughAbfsAtTheBlobHosttest, restored here) relied on. No legacyazure-storagedependency returns; the ABFS stack reads the blob host.Testing
Java 17,
mvn -pl opensharing-server-core,opensharing-server test:opensharing-server-core: 170/170 pass, including the restoredreadsWasbThroughAbfsAtTheBlobHost(asserts Kernel opensSecureAzureBlobFileSystemfor a wasbs location), the extendedreadsOnlySupportedStorage, andsignsAzurePathsAtTheBlobHost.opensharing-server:ProtocolUnsupportedStorageTestnow covers a genuinely unsupported scheme (hdfs) and passes. The remaining 2 failures are the pre-existing ones onmain(missing-log / unreadable-protocol error mapping, owned by separate fixes), plusqueriesGrantedTableMetadata, which this branch inherits frommain's metaData.version assertion and which PR Send metaData.version only for historical CDF metadata #188 fixes.This should fix the delta-sharing tests that read
table_wasb*inshare_azure(client, Spark, and Python suites). Merge after #188, or rebase onto it, to avoid the transientqueriesGrantedTableMetadatafailure.This pull request and its description were written by Isaac.