Repository navigation
GH-55683: [C++][FS][Azure] Use the DFS URL for the Data Lake client with SAS credentials - #55684
Open
LigaData-ltsentsiper wants to merge 1 commit into
Open
LigaData-ltsentsiper wants to merge 1 commit into
LigaData-ltsentsiper wants to merge 1 commit into
Conversation
…ient with SAS credentials With a SAS token, MakeDataLakeServiceClient built the DataLakeServiceClient from the blob URL. The Azure SDK sends Data Lake REST calls to the URL it is given, so the hierarchical namespace check went to the blob endpoint, got a 400 and AzureFileSystem treated a hierarchical namespace account as flat (Move() not implemented, DeleteFile() failing). Use AccountDfsUrl like every other credential kind. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
With a SAS token,
MakeDataLakeServiceClientbuilds theDataLakeServiceClientfrom the blob URL. The Azure SDK sendsData Lake REST calls to the URL it is given, so the hierarchical namespace check goes to the blob endpoint, gets a 400, and
AzureFileSystemtreats a hierarchical namespace account as flat:Move()is "not implemented" andDeleteFile()fails.What changes are included in this PR?
MakeDataLakeServiceClientusesAccountDfsUrl(account_name) + sas_token_for the SAS credential kind, like every othercredential kind.
Are these changes tested?
Manually, against a real storage account with hierarchical namespace: a small program calling
AzureFileSystemdirectly(
CreateDir, write,Move(),GetFileInfo,DeleteFile, results in the issue) with stock and patched 26.0.1 and anaccount SAS, and an application that writes Parquet files and renames them (2,000,000 rows) with an account SAS and a
container SAS. No automated test: the existing Azure tests run against Azurite, which has no hierarchical namespace.
A unit test in
TestAzureOptionscould buildAzureOptionswith a SAS token and check thatMakeDataLakeServiceClient()->GetUrl()uses thedfshost; I have not written or run it, and can add it if wanted.Are there any user-facing changes?
SAS token users on hierarchical namespace accounts get the hierarchical namespace code paths (
Move(),DeleteFile(),CreateDir) instead of the flat namespace ones.🤖 Generated with Claude Code