Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3ng include md5 checksum on put #4100

Merged
merged 4 commits into from
Aug 8, 2023
Merged

S3ng include md5 checksum on put #4100

merged 4 commits into from
Aug 8, 2023

Conversation

wkloucek
Copy link
Contributor

@wkloucek wkloucek commented Aug 4, 2023

Bugfix: S3ng include md5 checksum on put

We've fixed the S3 put operation of the S3ng storage to include a md5 checksum.

This md5 checksum is needed when a bucket has a retention period configured (see https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html).

@wkloucek wkloucek requested a review from aduffeck August 4, 2023 07:51
@wkloucek wkloucek marked this pull request as ready for review August 4, 2023 07:51
@@ -71,9 +71,9 @@ func (bs *Blobstore) Upload(node *node.Node, source string) error {
}
defer reader.Close()

_, err1 := bs.client.PutObject(context.Background(), bs.bucket, bs.path(node), reader, node.Blobsize, minio.PutObjectOptions{ContentType: "application/octet-stream"})
_, err = bs.client.PutObject(context.Background(), bs.bucket, bs.path(node), reader, node.Blobsize, minio.PutObjectOptions{ContentType: "application/octet-stream", SendContentMd5: true})
Copy link
Contributor

@butonic butonic Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a performance problem. SendContentMd5: true causes minio-go to reread the whole file before making the actual putobject request as it needs to send a Content-MD5 header. Maybe a more recent version of minio-go supports sending that as a trailer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah ... we can't send our MD5 with this lib ... the SendContentMd5 flag will always calculate the hash itself and there is no way to sneak in a Content-MD5 header.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's try to get a fix into upstream: minio/minio-go#1867

@aduffeck aduffeck mentioned this pull request Aug 8, 2023
@butonic
Copy link
Contributor

butonic commented Aug 8, 2023

we cannot use the precalculated md5 hash when large files are sent using multipart upload. For now stick to the lib calculating the hash. We can work on minio/minio-go#1867 to make it use a user provided hash when it does not use multipart uploads.

@butonic butonic merged commit a944f8f into cs3org:edge Aug 8, 2023
7 checks passed
@wkloucek wkloucek deleted the s3-put-md5 branch August 27, 2024 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants