Media upload fails with ValueError: seek of closed file

[jonaharagon]

Description

Media uploads are failing.

Steps to reproduce

Uploading any file in Riot (any platform) results in the upload failing.

2019-10-10 14:57:47,521 - synapse.http.server - 109 - ERROR - POST-329537- Failed handle request via 'UploadResource': <XForwardedForRequest at 0x7f1fb281c240 method='POST' uri='/_matrix/media/v1/upload?filename=ima_424b122.png' clientproto='HTTP/1.0' site=8008>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 77, in wrapped_request_handler
    await h(self, request)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/media/v1/upload_resource.py", line 82, in _async_render_POST
    media_type, upload_name, request.content, content_length, requester.user
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/media/v1/media_repository.py", line 160, in create_content
    fname = yield self.media_storage.store_file(content, file_info)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/media/v1/media_storage.py", line 68, in store_file
    self.hs.get_reactor(), _write_file_synchronously, source, f
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/logging/context.py", line 716, in g
    return f(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/media/v1/media_storage.py", line 244, in _write_file_synchronously
    source.seek(0)  # Ensure we read from the start of the file
ValueError: seek of closed file
2019-10-10 14:57:47,578 - synapse.http.server - 458 - WARNING - POST-329537- Not sending response to request <XForwardedForRequest at 0x7f1fb281c240 method='POST' uri='/_matrix/media/v1/upload?filename=ima_424b122.png' clientproto='HTTP/1.0' site=8008>, already disconnected.

I also see similar errors ending in ValueError: I/O operation on closed file. on some uploads. I can upload the homeserver.log if needed.

Version information

• Homeserver: chat.privacytools.io

If not matrix.org:

• Version: 1.4.0

• Install method: apt

• Platform: Ubuntu 18.04.3 LTS, running in an LXC container. The Matrix HS is behind an nginx reverse proxy on another container/

[Jukong]

I'm also running into this problem.

Also running Ubuntu 18.04 and behind an nginx reverse proxy inside a VM (Hyper-V).

Has anyone been able to get more information on this?

[Jukong]

I found the fix.

Uncomment the max upload line in the homeserver.yaml settings and change the limit to something like 25M or whatever you'd like.

Go to the reverse proxy and open the config file for your site (/etc/nginx/sites-available) add client_max_body_size 25M; parameter somewhere in the server block. Can make the size whatever you'd like.

Then lastly, add the same parameter for the nginx config on the server that is hosting matrix-synapse.

[richvdh]

it's expected that an upload will fail if you try to upload a file larger than the limit. But it sounds like the error message could be clearer, and synapse shouldn't spit exceptions into its logs.

[richvdh]

keywords: max_upload_size

Related: #1610

[jonaharagon]

This doesn't appear to be the issue in my configuration.

[aaronraimist]

@jonaharagon well something is different because I get a nice error message.

What is your max_upload_size set to and how big is the file?

[retrotails]

I'm having the same issue, with very similar errors to the OP, both in the server logs and the client. It happens to me for files at around 400MB+, but uploading the same exact file sometimes succeeds, sometimes fails. I'm not using a reverse proxy, files of any type fail, and I get the same "Upload Failed" error as OP.

[squahtx]

The error seen appears to happen as a result of a client (or proxy) closing an upload connection after sending all the data but before Synapse has responded. It looks like Twisted does some buffering, so turning off request buffering in the reverse proxy doesn't change the timings.

Steps to reproduce the error

1. Put a breakpoint in UploadResource._async_render_POST
2. curl -v -X POST -d some_test_data https://your.homeserver.org/_matrix/media/r0/upload -H "Authorization: Bearer syt_your_access_token_here"
3. Wait for the breakpoint to be hit
4. Abort curl with Ctrl+C
5. Continue execution from the breakpoint
6. See ValueError: I/O operation on closed file.

Perhaps one cause is Synapse taking too long to process the upload and clients timing out?

[squahtx]

Synapse may take a long time to process the upload if all threads in the default thread pool are busy (see #11049).
Giving the media repo its own thread pool might help with some of these failures.

Can people confirm whether they were running Synapse in single process or worker mode when they saw these failures?

[retrotails]

@squahtx single process for me. I didn't know workers were a thing, my homeserver has 3 users. if CPU usage is of any value, I'll note I've never seen above 2%. my load averages are all 0.00.
I'm not experiencing the same errors as above anymore. I still can't upload large files, but synapse doesn't log any errors. element android just tries uploading over and over again, and gomuks prints the following:

Failed to upload media: request error: Post "https://(...)/_matrix/media/r0/upload": context deadline exceeded (Client Timeout exceeded while awaiting headers)

I don't know whether my problem is still related to this issue.

[squahtx]

@retrotails Do you have a reverse proxy in front of Synapse, such as nginx? If you do, could you test the proxy_request_buffering off setting, or the equivalent for your reverse proxy? By default, nginx buffers the entire upload to disk before forwarding it on to Synapse, which buffers the entire upload to disk again. During the latter part, it would look like nothing is happening and clients might time out the upload if Synapse's I/O is slow enough.

[squahtx]

1.46.0rc1 (to be released within the next day) may reduce the frequency of upload failures thanks to the fix for #11049.

The race condition still needs to be fixed.

[retrotails]

@squahtx when I made my comment 20 months ago and had nearly an identical error to the OP, I was not using a reverse proxy. I am now using nginx, so I set proxy_request_buffering off, and I still have the same problem but synapse does log one thing when it fails now.
synapse.http.site - 323 - INFO - sentinel - Connection from IPv6Address(type='TCP', host='::1', port=40604, flowInfo=0, scopeID=0) lost before request headers were read
also I discovered the upload times out after exactly 10 minutes.

[squahtx]

@retrotails The timeout could be originating from anything in between your client and Synapse. From inspecting the code, Synapse ought to not time anything out while it's continually receiving data and I think the same applies to nginx.

Could you clarify which client(s) you're seeing the issue on, plus their versions? Are there any proxies in between your client and Synapse, apart from nginx?

[retrotails]

@squahtx no proxy aside from nginx, I have the issue with both element android (the latest on fdroid, reports as 1.3.3 [40103030] (F-651e3478)) and the latest version of gomuks. I also tried from a different client internet connection, same issue. the only thing I haven't ruled out is the internet connection the server is on, which I can't easily change atm.

[KeenMaron]

Any updates? I can send jpegs but a larger file (>10mb) fails with the same error message of op.

[squahtx]

I'm wholly unable to reproduce the issue. The error in the OP means that the client, reverse proxy, or anything in between aborted the request before Synapse received all the data and saved it. If you're having issues with a self-hosted instance of Synapse, it'd be extremely helpful to provide:

• The size of the file being uploaded
• Whether uploading the same file always fails or sometimes succeeds
• The client version you're using
• The speed of the internet connection being used to upload the file
• Whether you're using mobile data or Wi-Fi
• Whether there's a reverse proxy in front of Synapse
  • The values of proxy_request_buffering and client_max_body_size in the config if you're using nginx
• The Synapse version you're using

[KeenMaron]

Hey sorry. I fixed the problem with the solution of Jukong (Go to the reverse proxy and open the config file for your site (/etc/nginx/ ...) add client_max_body_size 25M;)

[patrickulrich]

I'm running into what I imagine is the same issue as well.

• The size of the file being uploaded
  16.63KB
• Whether uploading the same file always fails or sometimes succeeds
  Always fails regardless of media type
• The client version you're using
  Element 1.9.4
• The speed of the internet connection being used to upload the file
  93Mbps
• Whether you're using mobile data or Wi-Fi
  Wi-Fi
• Whether there's a reverse proxy in front of Synapse
  Reverse Proxy
• The values of proxy_request_buffering and client_max_body_size in the config if you're using nginx
  proxy_request_buffering false
  client_max_body_size 25M
• The Synapse version you're using
  1.47

[richvdh]

(have hidden comments unrelated to this issue. Please note we're unable to provide general user support via the issue tracker.)

[00011100]

Any updates on this issue?

[squahtx]

For large files (>100MiB), we suspect #13009 to be the cause.
We've not yet been able to track down the cause for small files.

[Hiers]

I was having this problem too. No file, no matter how big or small, could be uploaded.
I eventually found the problem was in the homeserver.yaml "media_store_path" config. It was set to /var/lib/synapse/media_store, and I was running synapse as another user (not synapse) who didn't have write access to that directory.
Either changing the directory to somewhere you have write permission or having synapse be run as the synapse user will fix it if this is your case.