storage/minio: Add options to disable signature and multipart for Minio Client

[stevefan1999-personal]

Fixes #21778, hopefully

This may also help rclone/rclone#6461 in the future, I don't think gofakes3 would have those two features implemented well either.

The Docker package is available at https://github.com/users/stevefan1999-personal/packages/container/package/gitea, feel free to checkout and test.

[KN4CK3R]

Are you able to test it? From looking at the minio code it may be enough to set DisableMultipart. But then all storage.Save calls need to provide the size of the content. Otherwise the upload just fails:
https://github.com/minio/minio-go/blob/7aa4b0e0d1a9fdb4a99f50df715c21ec21d91753/api-put-object.go#L236
Currently we do not provide a correct size everywhere:
https://github.com/go-gitea/gitea/pull/15264/files

[stevefan1999-personal]

@KN4CK3R that's exactly the reason I put this on draft

Indeed, I tracked some storage.Save call, it seems like some of them passes -1 which indicates it is a stream or some sort. This would be quite tricky

[stevefan1999-personal]

I can confirm that by disabling signature in Minio client, Docker registry works (I have both pushed an image, pulled it back, and ran the image). I'm not sure about others though.

[stevefan1999-personal]

I have ran this internally in my company for the last few month and it pretty much works. I'm looking to get it merged ASAP

[yardenshoham]

Missing documentation changes, see #23166 for example

[techknowlogick]

Thanks for this PR! Could you resolve conflicts?

[yardenshoham]

Still missing documentation

[wxiaoguang]

I agree with KN4CK3R here. After reading the code again, I think the callers to Save should provide the size/length.

Especially the doArchive in archiver.go, if the Save reads all the stream content to memory, it would take GB memory?

[lunny]

I agree with KN4CK3R here. After reading the code again, I think the callers to Save should provide the size/length.

Especially the doArchive in archiver.go, if the Save reads all the stream content to memory, it would take GB memory?

I can confirm the size parameter was added to fix the memory problem.

[stevefan1999-personal]

Gimme a moment, I was on a trip and didn't catch up so soon :)

[stevefan1999-personal]

I have another idea about multi-part upload, and that is to use mmap. If we can lock the file beforehand, using a memory map will not only solve the potential DoS problem, but that it will also drastically speed it up. I will see if I can implement it soon

[lunny]

Please resolve the conflicts

[yardenshoham]

Still missing documentation

[lunny]

So how to handle when size == 0?

[StarAurryon]

Hi there, do you have any news.

I am having the same issue. What were the remaining points to get this merged ?

I can do another PR if needed.