can not download file from console

[fwt11]

I have a 3-nodes MinIO cluster. After configuring an Nginx proxy for MinIO, I cannot download any files by clicking the Download button in the MinIO console. The server responds with:

Failed to get Stat() response from server for test.zip (version 7f560dfb-970f-4248-a26c-d6f901d95a54):
Last-Modified time format is invalid, failed with "unable to parse" in any of the input formats:
[Mon, 2 Jan 2006 15:04:05 GMT Mon, *2 Jan 2006 15:04:05 GMT Mon, *2 Jan 06 15:04:05 GMT]

However, I can download files by clicking the Share button and visiting the shared link. Additionally, accessing files through the Java API works fine.

Context

Regression

Your Environment

• Version used (minio --version): RELEASE.2024-07-26T20-48-21Z
• Server setup and configuration:
  minio configs:

     MINIO_VOLUMES="http://min{1...3}:9000/data"
     MINIO_OPTS="--console-address :9001"
     MINIO_SERVER_URL="http://minio.test.com"
     MINIO_BROWSER_REDIRECT_URL="http://minio-console.test.com/"

nginx configs:

server {
       listen 80;
       server_name minio.test.com;

       # ssl_certificate /etc/nginx/conf/secret/cert.pem;
       # ssl_certificate_key /etc/nginx/conf/secret/key.pem;
       # ssl_session_cache shared:SSL:10m;
       # ssl_session_timeout 10m;
       # ssl_protocols TLSv1.2;
       # ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE;
       # ssl_prefer_server_ciphers on;
       access_log /var/log/nginx/storage.log combined;
       error_log  /var/log/nginx/storage_error.log debug;

       # Allow special characters in headers
       ignore_invalid_headers off;
       # Allow any size file to be uploaded.
       # Set to a value such as 1000m; to restrict file size to a specific value
       client_max_body_size 0;
       # Disable buffering
       proxy_buffering off;
       proxy_request_buffering off;

       location / {
           proxy_set_header Host $http_host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           #proxy_redirect http:// $scheme://;
           proxy_redirect off;
           proxy_set_header X-Forwarded-Proto $scheme;
           #proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;

           proxy_connect_timeout 600s;
           # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
           proxy_http_version 1.1;
           proxy_set_header Connection "";
           chunked_transfer_encoding off;

           # add_header Last-Modified "";
           # if_modified_since exact;
           # expires 30d;

           proxy_pass http://storage/;
       }
  }

    server {
      listen       80;
      server_name minio-console.test.com 
      # Allow special characters in headers
      ignore_invalid_headers off;
      # Allow any size file to be uploaded.
      # Set to a value such as 1000m; to restrict file size to a specific value
      client_max_body_size 0;
      # Disable buffering
      proxy_buffering off;
      proxy_request_buffering off;

       location / {
           proxy_set_header Host $http_host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
           proxy_set_header X-NginX-Proxy true;
           proxy_redirect http:// $scheme://;
           # This is necessary to pass the correct IP to be hashed
           real_ip_header X-Real-IP;

           proxy_connect_timeout 300;

           # To support websockets in MinIO versions released after January 2023
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection "upgrade";
           proxy_read_timeout 600s;
          
           chunked_transfer_encoding off;

           proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance

       }
   }

• Operating System and version (uname -a):

[prakashsvmx]

Have you checked downloading without nginx ? i.e directly accessing minio console ?

[fwt11]

Thank you for your reply. Downloading without nginx works fine. This nginx configuration was copied from the Minio website. I also suspect that nginx is introducing the issue, but I don't know where the problem lies."

[ramondeklein]

Is MinIO (the S3 api) still accessible via http://minio.test.com (as configured in MINIO_SERVER_URL)?

[fwt11]

Yes, but this address is resolved within our local area network.

[ramondeklein]

You may want to check your NGINX logging to see if the actual Stat request is actually received. The URL should be the http://minio.test.com/<bucket>/<object-key> and it should use either a HEAD or GET request.

[fwt11]

This is the NGINX logs when I click the Download button.

minio-console.test.com:

192.168.100.213 - - [19/Sep/2024:09:19:09 +0800] "GET /api/v1/buckets/git/objects/download?prefix=gitea-dump.zip&version_id=90c0d6e6-4310-4c1d-9e8d-13fbea1da7cc HTTP/1.1" 500 294 "http://minio-console.test.com/browser/git/gitea-dump.zip" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"

minio.test.com:

192.168.100.31 - - [19/Sep/2024:09:19:09 +0800] "GET /git/?location= HTTP/1.1" 200 128 "-" "MinIO (linux; amd64) minio-go/v7.0.73 MinIO Console/(dev)"
192.168.100.31 - - [19/Sep/2024:09:19:09 +0800] "HEAD /git/gitea-dump.zip?versionId=90c0d6e6-4310-4c1d-9e8d-13fbea1da7cc HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.73 MinIO Console/(dev)"

where 192.168.100.31 is one of the MinIO node.

And I can not copy MinIO files to local with mc

mc: Unable to prepare URL for copying. Unable to guess the type of copy operation.
(2) cp-main.go:297 cmd.printCopyURLsError(..)
(1) cp-url.go:339 cmd.prepareCopyURLs.func1(..) Tags: [old/git/gitea-dump.zip]
(0) typed-errors.go:46 cmd.init.func18(..)
Release-Tag:RELEASE.2024-09-16T17-43-14Z | Commit:11ebe952ea30 | Host:fwt-PC | OS:linux | Arch:amd64 | Lang:go1.22.6 | Mem:5.2 MiB/22 MiB | Heap:5.2 MiB/15 MiB

and by using an old version of mc. the message

mc.RELEASE.2023-09-29T16-41-22Z: Unable to validate source minio/git/gitea-dump.zip: Last-Modified time format is invalid, failed with unable to parse in any of the input formats: [Mon, 2 Jan 2006 15:04:05 GMT Mon, _2 Jan 2006 15:04:05 GMT Mon, _2 Jan 06 15:04:05 GMT]

copy from local to MinIO is fine.

It's seems this is a problem related to Go language or Go SDK ?
Using Java or Python SDK to get files is fine. With go SDK it generated the same error message

Last-Modified time format is invalid, failed with unable to parse in any of the input formats: [Mon, 2 Jan 2006 15:04:05 GMT Mon, _2 Jan 2006 15:04:05 GMT Mon, _2 Jan 06 15:04:05 GMT]

This is the simple Go program I have tested.

package main

import (
	"context"
	"fmt"
	"log"
	"github.com/minio/minio-go/v7"
	"github.com/minio/minio-go/v7/pkg/credentials"
)

func main() {

	endpoint := "minio.test.com"
	accessKeyID := "admin"
	secretAccessKey := "abcd" 
	useSSL := false

	minioClient, err := minio.New(endpoint, &minio.Options{
		Creds:  credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
		Secure: useSSL,
	})
	if err != nil {
		log.Fatalln(err)
	}

	log.Println("Successfully created a MinIO client")
	err = minioClient.FGetObject(context.Background(), "git", "gitea-dump.tar", "myobject", minio.GetObjectOptions{})
	if err != nil {
		fmt.Println(err)
		return
	}
}

This message is reported by the function parseTime in utils.go in the Go Minio SDK.The response headers from Nginx don't include 'Last-Modified', which leads to parsing failure. However, the Python and Java code may have considered the case where it's empty?

I've tried various methods, but I can't manage to add the Last-Modified header to Nginx's response.

[ramondeklein]

Can you please run mc stat --debug <alias>/<bucket>/<key> and post the output? The response headers should look something like this:

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 305
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Thu, 19 Sep 2024 12:24:21 GMT
Etag: "92ea36ab1f951739d0dd21b251b0f3ce"
Last-Modified: Thu, 19 Sep 2024 12:24:08 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 17F6A4C25231465A
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 4073
X-Ratelimit-Remaining: 4073
X-Xss-Protection: 1; mode=block

The mc stat should include the Last-Modified response header. If you have the response header when using a direct connection, but not when going throught NGINX, then something is wrong with your NGINX configuration...