Upgrade to AWS Java SDK v2

[jorgee]

This PR contains the changes to port the Amazon plugin to AWS SDK version 2. Find below the most relevant changes:

• S3 Global Region flag: In v1, it was activated with S3Client.withForceGlobalBucketAcessEnabled(flag). In v2, it set the following flags S3Client.Builder.crossRegionAccessEnabled(flag) and S3Configuration.multiRegionAccessEnabled(flag).
• Two S3Clients are created: the async client is used for operations performed through the S3TransferManager, and the sync client is used for other actions.

- AmazonS3Client.getS3AccountOwner() is not available in SDK v2. It was providing an ID used for checking the file access. In V2, the only way to retrieve the same ID is from a bucket owned by the user. To do it we need to list the buckets and get the owner field in the GetBucketACLResponse. If it is not possible to retrieve the ID because the user does not own any bucket, we perform the following fallback. In the case of READ access, it tries to retrieve the head of the object, It will fail if there isn't read access. In the case of writting, a warning is printed. It is the same as AWS NIO is doing to check the file access.

• The setEndpoint and setRegion methods in the S3Client wrapper are removed as it is not available in the v2 clients. They were only used in tests.

• CannedAccessControlList is split in two classes one for objects and another for buckets. In most of the code it has been substituted by ObjectCannedACL.

• ContentType and ContentLength are part of the request instead of the ObjectMetadata, and they can be obtained invoking the S3client.headObject method in the SDK v2

• S3ClientConfiguration doesn't exist in SDK v2. Two new classes have been created to emulate the same behaviour. They convert the properties to the SDK v2 sync and async client configurations.

• SsoCredentialsProviderV1 class is not needed anymore as SDK v2 already manages the SSO credentials. The custom provider chain created in the S3FileSystemProvider.getCredetialsProvider0 to include the SsoCredentialsProviderV1 ihas been replace by the DefaultCredentialProvider in v2.

• Credentials and config are automatically merged by SDK v2. No option for NXF_DISABLE_AWS_CONFIG_MERGE.

• In V2, clients and requests are immutable and must be generated with a builder class. Some helper methods have been modified to pass builder classes instead of requests, such as makeJobDefRequest, configJobRefRequest, addVolumeMountsToContainer, etc.

• S3 Parallel Download was deprecated and S3CopyStream was not used. They have been removed.

• In v1, the upload directory was performed by walking through the different directory files and uploading them one by one. In v2, it has been substituted by the uploadDirectory method in the SDK.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	90a1457
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6865787296f9f6000850e1b3
😎 Deploy Preview	https://deploy-preview-6165--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[jorgee]

It is ready for review

[pditommaso]

Looks awesome. Made a few minor comments

[jorgee]

I have found an issue with multi-part uploads when uploading large files. I move to draft until I fix it.

[pditommaso]

@jorgee can you make a quick summary about this effort? any blocking issues?

[jorgee]

@pditommaso, this is the summary that I breifly explained to @bentsherman, yesterday.

I have been working on these two issues:

• Custom Signer: Java SDK v2 only support sAWS V4 signer either in netty and CRT. This is also supported by Minio and Ceph. I think a custom signer could be requires if a user uses a non up-to-date version of these tools. In Netty, there is the possibility to pass a custom signer, but there aren't other implementations in the SDK than V4. So the user should provide the implementation. In CRT, there is no possibility to override any HTTP advanced options, so neither Signer or user agent can be specified.

The solution that I have pushed in the latest commits is the following. In the case that a custom signer or user agent is specified, Nextflow will configure a Netty async client of the S3 Transfer Manager. A custom signer can be used including the implementation in the classpath and adding the FQDN of the custom signer class in the signerOverride option.
In this issue, we have an old implementation that could be used in case we need to support V2 Signer.

• The multipart upload problems: They were mainly due to using the old default uploadPartSize of 100MB with async clients. The transfer manager threads are not limited by the executor threads like in the past, they are managed by the async client maxConcurrency that is much more larger than threads. So, more parts are managed in parallel creating OOM problems when using small instances such as 1-2GB. With the default async multipart block size (8MB), it is not happening. So, I have kept the uploadPartSize option and the other upload* options for the case that is still using the old programmatic multipart upload (FileSystemProvider.newOutputStream case). For the transfers managed by the new S3 transfer manager, I have added new options to configure the async client.

Now, I am running the benchmarks to see if there are differences with the changes, but the code is again ready for review.

[pditommaso]

What's CRT? 😄

[jorgee]

CRT is the AWS Common Runtime. It is a C implementations of the AWS Client.
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html

[pditommaso]

OK, let's wait the result of the benchmark. However my advice is to focus on merging this PR with the core migration ignoring corner cases such Custom signer and user agent problem, and addressing them if needed in a separate PRs.

Regarding the multipart upload problem, is it related with a specific config or a general one?

[bentsherman]

Note that with the custom signer, if we don't support it in this PR then we have to deprecate it as it would no longer do anything

[jorgee]

I have pushed the changes to use only CRT version deprecating signerOverride and userAgent options.

[bentsherman]

See my comments above.

I'm mainly confused about the use of the S3 transfer manager vs the old aws.client.upload* config options. The fact that the old options are still used by newOutputStream() , to a user this essentially means "sometimes we use the old options, sometimes we use the new options". It would be nice to deprecate these old options altogether.

It's also not clear to me how to select the async client vs sync client, as I don't see any explicit config option for this

[pditommaso]

As a nextflow user I have no idea what's the asynchronous S3. That's not something should be exposed in the documentation. If I have understand correctly this is related to the CRT vs Netty runtime. Is it correct?

[pditommaso]

We talked. Better to document these are using used by the data transfer component.

[bentsherman]

I don't mind mentioning the async S3 client if there is also an option for selecting the async / sync client. But if we can make it so that Nextflow only uses one then we don't need to mention it in the docs

[jorgee]

I changed the text to refer to S3 transfer manager client. I have also added this section in the AWS part, to document when sync and async client is used.

[jorgee]

It's also not clear to me how to select the async client vs sync client, as I don't see any explicit config option for this

The sync client is used in all api call except the S3 transfer manager actions ( S3 copies, uploads and downloads). For transfer manager there is no option to use the sync client. I didn't changed all calls to async because it implied a lot of changes in the code

[jorgee]

I have added a subsection in the aws part of the documentation mentioning the most important changes in the v1 to v2 migration