Support ADLS with Pyarrow file IO #2111

NikitaMatskevich · 2025-06-17T13:41:30Z

Rationale for this change

Starting from version 20, PyArrow supports ADLS filesystem. This PR adds Pyarrow Azure support to Pyiceberg.

PyArrow is the default IO for Pyiceberg catalogs. In Azure environment it handles wider spectrum of auth strategies then Fsspec, including, for instance, Managed Identities. Also, prior to this PR #1663 (that is not merged yet) there was no support for wasb(s) with Fsspec.

See the corresponding issue for more details: #2112

Are these changes tested?

Tests are added under tests/io/test_pyarrow.py.

Are there any user-facing changes?

There are no API breaking changes. Direct impact of the PR: Pyarrow FileIO in Pyiceberg supports Azure cloud environment. Examples of impact for final users:

Pyiceberg is usable in services with Managed Identities auth strategy.
Pyiceberg is usable with wasb(s) schemes in Azure.

pyproject.toml

Fokko · 2025-06-17T17:35:20Z

@NikitaMatskevich Thanks for working on this, I know a lot of users are waiting for this. It looks like some tests are failing (you can run the linters locally using make lint), could you look into those?

NikitaMatskevich · 2025-06-18T13:14:15Z

@Fokko thank you for looking into this! Sorry, indeed, missed some formatting issues. Now it should be fine.

@NikitaMatskevich Thanks for working on this, I know a lot of users are waiting for this. It looks like some tests are failing (you can run the linters locally using make lint), could you look into those?

djouallah · 2025-06-18T23:02:45Z

I am one of those users, does this support authentication using auth token ? (not sas token)

NikitaMatskevich · 2025-06-19T12:25:35Z

I am one of those users, does this support authentication using auth token ? (not sas token)

From the docs:

If neither account_key or sas_token is specified a DefaultAzureCredential is used for authentication. This means it will try several types of authentication and go with the first one that works. If any authentication parameters are provided when initialising the FileSystem, they will be used instead of the default credential.

Here is the diagram of a DefaultAzureCredential flow.

kevinjqliu

Thanks for working on this! Generally LGTM

I have a few comments. We can either address them here or in a follow up PR

kevinjqliu · 2025-06-17T15:17:37Z

pyiceberg/io/__init__.py

@@ -82,6 +82,10 @@
 ADLS_CLIENT_ID = "adls.client-id"
 ADLS_CLIENT_SECRET = "adls.client-secret"
 ADLS_ACCOUNT_HOST = "adls.account-host"
+ADLS_BLOB_STORAGE_AUTHORITY = "adls.blob-storage-authority"


we should add these to the docs https://py.iceberg.apache.org/configuration/#azure-data-lake

pyiceberg/io/__init__.py

kevinjqliu · 2025-06-19T15:17:41Z

pyiceberg/io/pyarrow.py

@@ -197,6 +204,7 @@
 MAP_VALUE_NAME = "value"
 DOC = "doc"
 UTC_ALIASES = {"UTC", "+00:00", "Etc/UTC", "Z"}
+MIN_PYARROW_VERSION_SUPPORTING_AZURE_FS = "20.0.0"


nit: inline this at the function level

Then I will have to keep in sync the version used in tests and this one... I can do it if it's ok for you

i think thats fine, we can just use

if version.parse(pyarrow.__version__) < version.parse("20.0.0"):

This is technically a "public" variable and I dont want users to be able to import it.

kevinjqliu · 2025-06-19T15:18:23Z

pyiceberg/io/pyarrow.py

@@ -394,6 +402,9 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste
        elif scheme in {"gs", "gcs"}:
            return self._initialize_gcs_fs()

+        elif scheme in {"abfs", "abfss", "wasb", "wasbs"}:


I cant find any pyarrow docs that indicates wasb and wasbs is supported.

pyiceberg/io/pyarrow.py

tests/conftest.py

kevinjqliu · 2025-06-19T15:31:40Z

tests/io/test_pyarrow.py

@@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None:


 @pytest.mark.gcs
-@pytest.mark.skip(reason="Open issue on Arrow: https://github.com/apache/arrow/issues/36993")


i see that apache/arrow#36993 is still open. is the issue resolved and we can run this test?

running make test-gcs locally fails

weird, do we not run these integration tests in CI??

Sorry, I wrongly assumed they will be run with make test...

no worries! I was surprised too. we gotta fix it :)

Tracking issue #2124

Actually on my machine only 1 test was failing. I restored the annotation on it. Does make test-gcs run normally for you now?

it still fails for me, even after recreating the poetry env.

we can just take out all these changes and address them in a separate PR

kevinjqliu

Revert the change to not skip gcs tests

@pytest.mark.skip(reason="Open issue on Arrow: https://github.com/apache/arrow/issues/36993")

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

kevinjqliu

LGTM, lets just remove the changes to gcs integration test and address them in a separate PR.

#2125 would help make sure the file io integration changes are safe to make

kevinjqliu · 2025-06-20T02:38:58Z

tests/io/test_pyarrow.py

@@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None:


 @pytest.mark.gcs
-@pytest.mark.skip(reason="Open issue on Arrow: https://github.com/apache/arrow/issues/36993")


it still fails for me, even after recreating the poetry env.

we can just take out all these changes and address them in a separate PR

kevinjqliu · 2025-06-20T02:40:20Z

pyiceberg/io/pyarrow.py

@@ -197,6 +204,7 @@
 MAP_VALUE_NAME = "value"
 DOC = "doc"
 UTC_ALIASES = {"UTC", "+00:00", "Etc/UTC", "Z"}
+MIN_PYARROW_VERSION_SUPPORTING_AZURE_FS = "20.0.0"


i think thats fine, we can just use

if version.parse(pyarrow.__version__) < version.parse("20.0.0"):

This is technically a "public" variable and I dont want users to be able to import it.

kevinjqliu

LGTM! Thanks for adding this feature :)

kevinjqliu · 2025-06-20T14:39:45Z

@NikitaMatskevich looks like the linter errored, could you run make lint?

kevinjqliu · 2025-06-20T14:49:23Z

actually i just push the make lint change :)

kevinjqliu · 2025-06-20T15:11:29Z

Thanks for working on this @NikitaMatskevich and thanks @Fokko for the review

# Rationale for this change Starting from version 20, PyArrow supports ADLS filesystem. This PR adds Pyarrow Azure support to Pyiceberg. PyArrow is the [default IO](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L366-L369) for Pyiceberg catalogs. In Azure environment it handles wider spectrum of auth strategies then Fsspec, including, for instance, [Managed Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). Also, prior to this PR apache#1663 (that is not merged yet) there was no support for wasb(s) with Fsspec. See the corresponding issue for more details: apache#2112 # Are these changes tested? Tests are added under tests/io/test_pyarrow.py. # Are there any user-facing changes? There are no API breaking changes. Direct impact of the PR: Pyarrow FileIO in Pyiceberg supports Azure cloud environment. Examples of impact for final users: - Pyiceberg is usable in services with Managed Identities auth strategy. - Pyiceberg is usable with wasb(s) schemes in Azure.  --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>

NikitaMatskevich mentioned this pull request Jun 17, 2025

Support Azure FileSystem with Pyarrow FileIO #2112

Closed

NikitaMatskevich added 3 commits June 17, 2025 17:48

bump pyarrow version deps

494f2fe

add support for pyarrow adls file io

5efd4ee

modify poetry lock

076b68b

NikitaMatskevich force-pushed the nmatckevich/support-adls-pyarrow-file-io branch from e4e7260 to 076b68b Compare June 17, 2025 15:48

revert

9a8ce25

Fokko reviewed Jun 17, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

make lint

f609160

NikitaMatskevich added 2 commits June 18, 2025 16:41

check for pyarrow version dynamically

9667b31

revert poetry lock

3ab3e99

kevinjqliu self-requested a review June 18, 2025 15:48

Fokko approved these changes Jun 18, 2025

View reviewed changes

kevinjqliu approved these changes Jun 19, 2025

View reviewed changes

kevinjqliu requested changes Jun 19, 2025

View reviewed changes

Update tests/conftest.py

018125f

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

kevinjqliu mentioned this pull request Jun 19, 2025

CI: run FileIO integration tests (s3/adls/gcs) #2124

Closed

3 tasks

revert skip

d96f858

kevinjqliu approved these changes Jun 20, 2025

View reviewed changes

NikitaMatskevich added 3 commits June 20, 2025 11:59

add docs for new properties of adls

7e513eb

revert skip tests

3e36526

inline pyarrow version def

5a6dd72

kevinjqliu approved these changes Jun 20, 2025

View reviewed changes

make lint

dae20c8

kevinjqliu merged commit 84c91f0 into apache:main Jun 20, 2025
11 checks passed

Fokko mentioned this pull request Jun 25, 2025

Add ALDS support for PyArrow #1532

Closed

		@@ -1670,9 +1678,8 @@ def test_new_output_file_gcs(pyarrow_fileio_gcs: PyArrowFileIO) -> None:


		@pytest.mark.gcs
		@pytest.mark.skip(reason="Open issue on Arrow: https://github.com/apache/arrow/issues/36993")

Support ADLS with Pyarrow file IO #2111

Support ADLS with Pyarrow file IO #2111

Uh oh!

Conversation

NikitaMatskevich commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Fokko commented Jun 17, 2025

Uh oh!

NikitaMatskevich commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

djouallah commented Jun 18, 2025

Uh oh!

NikitaMatskevich commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NikitaMatskevich Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Jun 20, 2025

Uh oh!

kevinjqliu commented Jun 20, 2025

Uh oh!

Uh oh!

kevinjqliu commented Jun 20, 2025

Uh oh!

Uh oh!

NikitaMatskevich commented Jun 17, 2025 •

edited

Loading

NikitaMatskevich commented Jun 18, 2025 •

edited

Loading

NikitaMatskevich commented Jun 19, 2025 •

edited

Loading

NikitaMatskevich Jun 19, 2025 •

edited

Loading