HADOOP-19778. [ABFS] Removing Deprecated Wasb Code by anujmodi2021 · Pull Request #8207 · apache/hadoop

anujmodi2021 · 2026-01-23T13:27:55Z

JIRA: https://issues.apache.org/jira/browse/HADOOP-19778

Description of PR

WASB Driver has been deprecated.
Correspondinf support for FNS accounts using Blob Endpoint APIs is added to ABFS Driver itself.

ABFS driver for FNS account is more robust, performant and TPS Optimized as compared to WASB.
Moving ahead ABFS Driver will be the only supported hadoop connector for ADLS Gen 2 accounts (HNS and FNS).

To reduce maintainance overhead, we propose to remove WASB Driver code completely.

How was this patch tested

New tests added to make sure any attempt to use WASB fails with relevant message.

cnauroth

I'm in favor of removal for 3.5.0. I'd like at least one other committer to chime in though, as I'm no longer actively using any of the Azure FileSystems.

Will the strategy here be to remove the module entirely, or leave behind an empty placeholder module like in hadoop-openstack/pom.xml? Either way, I think the PR likely needs some additional pom.xml changes.

Pre-submit infrastructure appears to be very unhealthy at the moment. The automatic run failed, and when I retried, it failed for the same reason:

hudson.remoting.ProxyException: java.nio.file.FileSystemException: /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-8207: Read-only file system

cnauroth

Two more thoughts:

You probably want to remove the documentation too (and clean up links to that documentation).
Can you please edit the release note field in HADOOP-19778 to mention the removal, as it will be a backward-incompatible change?

anujmodi2021 · 2026-01-28T18:43:53Z

Hi @cnauroth
Thanks for your valuable review and suggestions.

Regarding the strategy here, we are going to keep a placeholder implementation of WASB File System for any user using "wasb(s)://" schema. There is no separate pom for wasb alone, it is shared with ABFS So pom will continue to live but with settings only relevant to ABFS.

As per your suggestion I have updated the documentation and sites to refelct this change.
Also updated the release note in JIRA.

anujmodi2021 · 2026-01-28T18:44:40Z

Hi @steveloughran
I would request your kind review on this patch.

Thanks for all the efforts

bhattmanish98 · 2026-01-29T05:54:41Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

-              createPermissionStatus(FsPermission.getDefault()));
-        }
+  public static final String ERROR_MESSAGE =
+      "WASB Driver using wasb(s) schema is No longer Supported. "


Upper Case fix: is no longer supported

bhattmanish98 · 2026-01-29T05:55:58Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

-      }
-    }
+  public NativeAzureFileSystem() {
+    // set store in initialize()


I think this comment is redundant, we can remove this.

bhattmanish98 · 2026-01-29T06:47:31Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

-    workingDir = makeAbsolute(newDir);
+  public void initialize(URI uri, Configuration conf)
+      throws IOException, IllegalArgumentException {
+    throw new IllegalArgumentException(ERROR_MESSAGE);


Instead of IllegalArgumentException, should we use UnsupportedOperationException?

bhattmanish98 · 2026-01-29T06:49:59Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

-    } else {
-      store.changePermissionStatus(key, newPermissionStatus);
-    }
+    return null;


Should we throw an error instead of just returning null at all places? With null we are suppressing the exception.

anmolanmol1234 · 2026-01-29T06:56:39Z

...-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier


-org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenIdentifier
-org.apache.hadoop.fs.azure.security.WasbDelegationTokenIdentifier
+org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenIdentifier


anmolanmol1234 · 2026-01-29T06:56:49Z

...oop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer


-org.apache.hadoop.fs.azurebfs.security.AbfsTokenRenewer
-org.apache.hadoop.fs.azure.security.WasbTokenRenewer
+org.apache.hadoop.fs.azurebfs.security.AbfsTokenRenewer


anmolanmol1234 · 2026-01-29T07:33:06Z

...op-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRandomRead.java

@@ -107,69 +107,6 @@ public void testBasicRead() throws Exception {
    }
  }

-  /**


Why would we remove this test ?

This was using WASB Client. Without wasb client there is nothing left in this test to assert

anmolanmol1234 · 2026-01-29T07:37:42Z

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AbfsTestUtils.java

+            + FS_AZURE_SCALE_TEST_ENABLED,
+        enabled);
+  }
+}


anmolanmol1234 · 2026-01-29T07:41:24Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java

-  public Path getWorkingDirectory() {
-    return workingDir;
+  public String getScheme() {
+    return "wasb";


can come from a constant

anmolanmol1234 · 2026-01-29T07:44:13Z

hadoop-tools/hadoop-azure/src/site/markdown/deprecated_wasb.md


 # Azure Blob Storage Support by Deprecated WASB Driver

+### WASB Driver has been deprecated and removed from official hadoop releases starting from hadoop-3.5.0.


I think we can remove this file fully.

manika137 · 2026-01-29T07:45:13Z

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AbfsTestUtils.java

+            + FS_AZURE_SCALE_TEST_ENABLED,
+        enabled);
+  }
+}


manika137 · 2026-01-29T07:45:31Z

hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md

+big data analytics, built on top of Azure Blob Storage. The ABFS and ABFSS
+schemes target the ADLS Gen 2 REST API now having support for both HNS and FNS Accounts.
+ADLS Gen 2 with HNS Enabled using DFS Endpoint offers better performance and
+scalability.  ADLS Gen 2 also offers authentication and authorization compatible


nit: double space

anujmodi2021 · 2026-01-29T11:32:05Z

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 859, Failures: 0, Errors: 0, Skipped: 173
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 862, Failures: 0, Errors: 0, Skipped: 119
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 859, Failures: 0, Errors: 0, Skipped: 184
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 709, Failures: 0, Errors: 0, Skipped: 143
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 706, Failures: 0, Errors: 0, Skipped: 155
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 701, Failures: 0, Errors: 0, Skipped: 176
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 228, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 733, Failures: 0, Errors: 0, Skipped: 182
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23

cnauroth · 2026-01-29T21:07:21Z

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbInitFailure.java

+      FileSystem.newInstance(uri, conf).close();
+    });
+    Assertions.assertThat(ex.getMessage())
+        .contains("WASB Driver using wasb(s) schema is No longer Supported.");


Do these assertions pass now? There was earlier code review feedback to fix capitalization ("is no longer supported"), but it wasn't changed here.

My bad. Thanks for thorough review
Our test suite do not run WASB tests hence it was not caught.

cnauroth

+1 pending pre-commit. Thank you!

Unfortunately, it looks like we continue to have some infrastructure problems with pre-commit:

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8207/8/consoleText

java.nio.file.FileSystemException: /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-8207: Read-only file system

We might need to chase this down with ASF infra.

anujmodi2021 · 2026-01-31T08:51:50Z

+1 pending pre-commit. Thank you!

Unfortunately, it looks like we continue to have some infrastructure problems with pre-commit:

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8207/8/consoleText
java.nio.file.FileSystemException: /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-8207: Read-only file system
We might need to chase this down with ASF infra.

Thanks @cnauroth

We need this to be merged before 3.5.0 release. It would be really helpful if you can guide me on how to get this resolved?
Whom and how should I contact in ASF regarding this?

anujmodi2021 · 2026-01-31T08:54:48Z

@steveloughran
Gentle reminder to review this.
Your thoughts and guidance will be really helpful

WASB Code Removal

99aa98e

github-actions bot added trunk TOOLS ABFS labels Jan 23, 2026

Code Cleanup

69fde5a

anujmodi2021 marked this pull request as ready for review January 23, 2026 15:30

anujmodi2021 requested review from cnauroth, slfan1989 and steveloughran January 23, 2026 15:30

cnauroth reviewed Jan 23, 2026

View reviewed changes

pom cleanup

426a817

github-actions bot added the build label Jan 27, 2026

Fixing Doc

72b36b1

Merge branch 'trunk' into HADOOP-19778

a378134

bhattmanish98 reviewed Jan 29, 2026

View reviewed changes

anmolanmol1234 reviewed Jan 29, 2026

View reviewed changes

manika137 reviewed Jan 29, 2026

View reviewed changes

Comments

90a6107

anmolanmol1234 approved these changes Jan 29, 2026

View reviewed changes

manika137 approved these changes Jan 29, 2026

View reviewed changes

cnauroth reviewed Jan 29, 2026

View reviewed changes

Fixing WASB test

0649429

cnauroth approved these changes Jan 30, 2026

View reviewed changes

anmolanmol1234 mentioned this pull request Feb 5, 2026

HADOOP-19802. [ABFS] Removing azure storage dependency from hadoop-azure code. #8229

Open

@@ @@ -107,69 +107,6 @@ public void testBasicRead() throws Exception { @@
                   }
                 }
-                /**


		# Azure Blob Storage Support by Deprecated WASB Driver

		### WASB Driver has been deprecated and removed from official hadoop releases starting from hadoop-3.5.0.

Conversation

anujmodi2021 commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 commented Jan 28, 2026

Uh oh!

anujmodi2021 commented Jan 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 commented Jan 29, 2026

============================================================ HNS-OAuth-DFS

============================================================ HNS-SharedKey-DFS

============================================================ AppendBlob-HNS-OAuth-DFS

============================================================ NonHNS-SharedKey-Blob

============================================================ NonHNS-OAuth-Blob

============================================================ AppendBlob-NonHNS-OAuth-Blob

============================================================ HNS-Oauth-DFS-IngressBlob

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cnauroth left a comment

anujmodi2021 commented Jan 23, 2026 •

edited

Loading

============================================================
HNS-OAuth-DFS

============================================================
HNS-SharedKey-DFS

============================================================
AppendBlob-HNS-OAuth-DFS

============================================================
NonHNS-SharedKey-Blob

============================================================
NonHNS-OAuth-Blob

============================================================
AppendBlob-NonHNS-OAuth-Blob

============================================================
HNS-Oauth-DFS-IngressBlob