Support sending precalculated Content-MD5 header

[butonic]

We already have the MD5 of the content and want to send along the Content-MD5 header without minio-go having to recalculate the hash. With this PR we can set SendContentMd5: false and send our hash as UserMetadata.

[harshavardhana]

Please add a functional_tests.go @butonic

[butonic]

@harshavardhana I tried to copy and adapt an existing test. Let me know if this works for you.

[butonic]

wow ... trying to run go test ./functional_tests.go gives me

# command-line-arguments
./functional_tests.go:2169:130: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./functional_tests.go:2169:154: conversion from int64 to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
./functional_tests.go:2844:129: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
...

[klauspost]

This requires multipart uploads to be disabled. I am a bit hesitant, since that isn't really obvious.

Also errors:

Error: ./functional_tests.go:11409:4: syntax error: unexpected newline in composite literal; possibly missing comma or }
Error: make: *** [Makefile:32: functional-test] Error 1
Error: Process completed with exit code 2.

[butonic]

hm, looking at the makefile make test uses go test ./... but does not seem ro run the functional tests. If I introduce a bug it won't be picked up by go test ./...:

➜  minio-go git:(support-sending-custom-content-md5) ✗ git diff
diff --git a/functional_tests.go b/functional_tests.go
index ed15946..0ff9653 100644
--- a/functional_tests.go
+++ b/functional_tests.go
@@ -11407,7 +11407,7 @@ func testPutObjectContentMD5() {
                "bucketName": "",
                "objectName": "",
                "size":       3,
-               "opts":       opts,
+               "opts":       opts      // introduce a bug
        }
 
        // Seed random based on current time.
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test ./functional_tests.go 
# command-line-arguments
./functional_tests.go:11410:21: syntax error: unexpected newline in composite literal; possibly missing comma or }
FAIL
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test ./...
ok      github.com/minio/minio-go/v7    (cached)
?       github.com/minio/minio-go/v7/pkg/encrypt        [no test files]
ok      github.com/minio/minio-go/v7/pkg/credentials    (cached)
ok      github.com/minio/minio-go/v7/pkg/lifecycle      (cached)
ok      github.com/minio/minio-go/v7/pkg/notification   (cached)
ok      github.com/minio/minio-go/v7/pkg/policy (cached)
ok      github.com/minio/minio-go/v7/pkg/replication    (cached)
ok      github.com/minio/minio-go/v7/pkg/s3utils        (cached)
ok      github.com/minio/minio-go/v7/pkg/set    (cached)
?       github.com/minio/minio-go/v7/pkg/sse    [no test files]
ok      github.com/minio/minio-go/v7/pkg/signer (cached)
ok      github.com/minio/minio-go/v7/pkg/tags   (cached)
➜  minio-go git:(support-sending-custom-content-md5) ✗ go test 
PASS
ok      github.com/minio/minio-go/v7    3.033s

that seems fishy ...

[harshavardhana]

diff --git a/functional_tests.go b/functional_tests.go
index ed15946..0ff9653 100644
--- a/functional_tests.go
+++ b/functional_tests.go
@@ -11407,7 +11407,7 @@ func testPutObjectContentMD5() {
"bucketName": "",
"objectName": "",
"size": 3,

•           "opts":       opts,
  

•           "opts":       opts      // introduce a bug
    }
  

you are missing ,

[butonic]

yes, that was on purpose to show that a sytax error in functional_tests.go will not stop make test, which uses go run ./... from succeeding. I don't understand why ✗ go test ./functional_tests.go returns errors like

./functional_tests.go:2169:130: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
...

while go test ./... is happy at the same time.

[butonic]

I see ... the functional_tests.go has a main() and is meant to be executed ... sorry for the confusion.

@klauspost so what do we do about the multipart uploads? we could disable them ... and add an explicit ContentMD5 property to the minio.PutObjectOptions?

[butonic]

Multipart uploads are automatically used for files > 16MB (minPartSize) or the opts.PartSize. We have to calculate other hashes and can easily precalculate multiple md5hashes for every part.

@klauspost what about a ContentMD5Hashes []string in the PutObjectOptions struct. When the SendContentMd5 flag is enabled the different put implementations can skip calculating the md5 when the array index for that part is non empty (if opts.ContentMD5Hashes [partNumber-1]).

[harshavardhana]

There is no way to do this in the current API style cleanly, you can use minio.Core() functionality to pass your own md5sum and also construct your own API wrappers.

That should serve the purpose.