Very large number of TLS handshakes

[gabrielrussoc]

Hi all,

We've been using a service called Bazel Remote and it happens to use the minio go library to communicate with S3. I realised this service was spending a lot of CPU time doing TLS handshakes and was able to pin this down to minio S3 calls. The relevant code is:

func get(ctx context.Context, minioCore *minio.Core, bucket string, key string, wg *sync.WaitGroup) {
    defer wg.Done()
    _, _, _, err := minioCore.GetObject(
        ctx,
        bucket,
        key,
        minio.GetObjectOptions{},
    )
    
}

func TestHandshake(t *testing.T) {
    opts := &minio.Options{
        Creds:  credentials.NewEnvAWS(),
        Region: "us-west-2",
        Secure: true,
    }
    minioCore, err := minio.NewCore("s3.us-west-2.amazonaws.com", opts)
    ...
    calls := 100
    ...
    for i := 0; i < calls; i++ {
        go get(ctx, minioCore, bucket, key, &wg)
    }
    ...
}

The full repro is here:
https://github.com/gabrielrussoc/repro-minio-handshakes

When I look at the flame graph, I see a lot of time spent on http.(*persistConn).addTLS.func2 and can also see 602 entries of the same function on the trace profile. Naively, I'd expect at most 100 handshakes, and much less in practice since we have a connection pool.

I suspect this might be golang/go#50984, but I couldn't figure out any workarounds

Versions

go version

go version go1.20.3 darwin/arm64

go env

GO111MODULE="" GOARCH="arm64" GOBIN="" GOCACHE="/Users/gabriel.russo/Library/Caches/go-build" GOENV="/Users/gabriel.russo/Library/Application Support/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="arm64" GOHOSTOS="darwin" GOINSECURE="" GOMODCACHE="/Users/gabriel.russo/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/gabriel.russo/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64" GOVCS="" GOVERSION="go1.20.3" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/dev/null" GOWORK="" CGO_CFLAGS="-O2 -g" CGO_CPPFLAGS="" CGO_CXXFLAGS="-O2 -g" CGO_FFLAGS="-O2 -g" CGO_LDFLAGS="-O2 -g" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/6s/97qg1zl11sq91zj5k2v6_jgh0000gp/T/go-build4268782293=/tmp/go-build -gno-record-gcc-switches -fno-common"

minio version

v7.0.60

Screenshots

[harshavardhana]

I will check. Application would need to configure relevant transport and optimizations they want here by passing a custom transport.

[gabrielrussoc]

@harshavardhana How would a custom transport help here?

[harshavardhana]

@harshavardhana How would a custom transport help here?

You have to try to enable TLS session cookies to try to further optimize this such as

        tlsClientConfig := tls.Config{
                ClientSessionCache: tls.NewLRUClientSessionCache(tlsClientSessionCacheSize),
        }

you can increase the

                MaxIdleConnsPerHost:   16,

to 1024 - default is 16. You should play around with these values, default settings may not be suitable for the level of concurrency that you may have.

[harshavardhana]

yeah looks like none of these help - the bug exists in the TLS stack in Go @gabrielrussoc

[harshavardhana]

so I don't know if we can do anything here to fix it.