Skip to content

Read WASB tables through ABFS at the blob host - #191

Closed
littlegrasscao wants to merge 1 commit into
OpenSharing-IO:mainfrom
littlegrasscao:fix/wasb-tables
Closed

littlegrasscao wants to merge 1 commit into
OpenSharing-IO:mainfrom
littlegrasscao:fix/wasb-tables

Conversation

@littlegrasscao

Copy link
Copy Markdown
Collaborator

Problem

A wasb:// or wasbs:// table read fails with a 501 before the catalog is even asked for credentials (HadoopStorageConfiguration.requireSupported), so the delta-sharing suites' share_azure tables - table_wasb, table_wasb_change_query_id, and friends, all at wasbs://container@account.blob.core.windows.net/... locations - cannot be read by OpenSharing at all. The unmodified client, Spark, and Python suites exercise these tables, so every one of those tests fails.

Change

Read WASB locations through ABFS at the same blob host, restoring the approach from the first commit of #148 before its final state replaced it with the 501 rejection:

  • StorageLocations.AZURE_SCHEMES includes wasb and wasbs again, so both the URL signer and table reads pick them up through the shared scheme set.
  • HadoopStorageConfiguration.kernelPath translates a wasb(s)://container@account.blob.core.windows.net/key location to abfss:// at the same host, where Hadoop's ABFS filesystem reads it.
  • configureAzure declares a .blob. host's account as having no hierarchical namespace (fs.azure.account.hns.enabled.<host>=false), which ABFS requires to read a blob host.
  • File URLs need no new signing logic: a wasb location already names the account.blob.core.windows.net host the signed URL uses, and AzureSasUrlSigner builds exactly that URL.

This supersedes part of #148, whose final commit rejected WASB reads "as no file URL could be signed for it". A wasb location's host is precisely the blob host the signed URL needs, as #148's own first commit (and its readsWasbThroughAbfsAtTheBlobHost test, restored here) relied on. No legacy azure-storage dependency returns; the ABFS stack reads the blob host.

Testing

Java 17, mvn -pl opensharing-server-core,opensharing-server test:

  • opensharing-server-core: 170/170 pass, including the restored readsWasbThroughAbfsAtTheBlobHost (asserts Kernel opens SecureAzureBlobFileSystem for a wasbs location), the extended readsOnlySupportedStorage, and signsAzurePathsAtTheBlobHost.
  • opensharing-server: ProtocolUnsupportedStorageTest now covers a genuinely unsupported scheme (hdfs) and passes. The remaining 2 failures are the pre-existing ones on main (missing-log / unreadable-protocol error mapping, owned by separate fixes), plus queriesGrantedTableMetadata, which this branch inherits from main's metaData.version assertion and which PR Send metaData.version only for historical CDF metadata #188 fixes.

This should fix the delta-sharing tests that read table_wasb* in share_azure (client, Spark, and Python suites). Merge after #188, or rebase onto it, to avoid the transient queriesGrantedTableMetadata failure.

This pull request and its description were written by Isaac.

A wasb:// or wasbs:// table read failed with a 501 before the catalog
was even asked for credentials, so the delta-sharing suites' share_azure
tables (table_wasb and friends, at wasbs:// blob hosts) could not be
read at all.

Read them through ABFS at the same blob host: the location's container,
host, and path carry over to an abfss:// path, the account is declared
to have no hierarchical namespace as ABFS requires for a blob host, and
the file URLs are signed on the blob host they already name. This
restores the scheme set, kernel path, and SAS settings superseded by
OpenSharing-IO#148, whose final state rejected WASB reads because no file URL could
be signed for them; the blob host a wasb location names is exactly the
host the signed URL needs, as OpenSharing-IO#148's first commit had it.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Sun Cao <sun.cao+data@databricks.com>
@github-actions

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

@littlegrasscao

littlegrasscao commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Closing. OpenSharing serves Azure tables only through abfs/abfss and rejects wasb/wasbs locations; main already answers wasb reads with 501 NOT_IMPLEMENTED, so no change is needed. The branch stays available for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant