Skip to content

Add support for custom credential loader for S3 FileIO #1528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 21, 2025

Conversation

phillipleblanc
Copy link
Contributor

Which issue does this PR close?

What changes are included in this PR?

Adds the ability to provide custom extensions to the FileIOBuilder. Currently the only supported extension is CustomAwsCredentialLoader which is a newtype around AwsCredentialLoad, which is what OpenDAL expects.

I've added extensions to the RestCatalog as well, and when its constructing FileIO for table operations, passes along any defined extensions into the FileIOBuilder, which then get passed into the underlying OpenDAL constructors.

Are these changes tested?

Yes, tests added in crates/iceberg/tests/file_io_s3_test.rs that verify the extension is working.

Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @phillipleblanc for working on this. Ideally, we should have this support built into reqsign or opendal, but I'm happy to merge this PR to give our users access.

@Xuanwo Xuanwo merged commit 1d217d4 into apache:main Jul 21, 2025
18 checks passed
@phillipleblanc phillipleblanc deleted the phillip/250719-override-credential-loader branch July 21, 2025 05:51
@liurenjie1024
Copy link
Contributor

This is another motivating case of #1314

@phillipleblanc
Copy link
Contributor Author

This is another motivating case of #1314

Ah yeah, I wasn't aware of that discussion - thanks for the link! I represent another case where I'm already heavily invested in the object_store ecosystem and would like to be able to use that to power my Iceberg reads/writes in DataFusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ability to specify a custom AWS credential loader for S3 FileIO
4 participants