Gzip Middleware content-length is incorrect

[abersheeran]

The following exception is thrown when I use uvicorn to drive my starlette project. After control variates, I am sure this is caused by Gzip Middleware.

  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\h11\_writers.py", line 102, in send_eom
    raise LocalProtocolError("Too little data for declared Content-Length") 
h11._util.LocalProtocolError: Too little data for declared Content-Length

[tomchristie]

I've taken a quick look over the GZipMiddleware class, which shows that we are:

• Removing the Content-Length header for streaming responses.
• Setting the Content-Length header to the GZipped content length of the gzipped body.

All of that looks correct to me. (Although I'm certainly prepared to believe that there could be an issue here, it's just that it's not obvious at first sight?)

Can you narrow this down to a simple reproducible case?

[abersheeran]

Thank you for your reply. But I'm in China, it's already late at night. Tomorrow i will give you a minimal implementation.

[abersheeran]

from starlette.applications import Starlette
from starlette.middleware.gzip import GZipMiddleware
from starlette.middleware.wsgi import WSGIMiddleware
from starlette.routing import Route

from wsgi_example.wsgi import application


routes = [
    Route("/", endpoint=WSGIMiddleware(application)),
]

app = Starlette(routes=routes)
app.add_middleware(GZipMiddleware)

wsgi_example is that I use the latest version of django-admin startproject to automatically generate the project without any modification.

running uvicorn star:app will output:

INFO:     Started server process [13088]
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:5247 - "GET / HTTP/1.1" 200
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\applications.py", line 102, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\errors.py", line 178, in __call__
    raise exc from None
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\errors.py", line 156, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\gzip.py", line 18, in __call__
    await responder(scope, receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\gzip.py", line 35, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc from None
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\routing.py", line 550, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\wsgi.py", line 62, in __call__
    await responder(receive, send)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\wsgi.py", line 91, in __call__
    await asyncio.wait_for(sender, None)
  File "C:\Users\AberS\AppData\Local\Programs\Python\Python37\Lib\asyncio\tasks.py", line 414, in wait_for
    return await fut
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\wsgi.py", line 106, in sender
    await send(message)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\exceptions.py", line 68, in sender
    await send(message)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\gzip.py", line 93, in send_with_gzip
    await self.send(message)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\starlette\middleware\errors.py", line 153, in _send
    await send(message)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 486, in send
    output = self.conn.send(event)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\h11\_connection.py", line 502, in send_with_data_passthrough
    writer(event, data_list.append)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\h11\_writers.py", line 79, in __call__
    self.send_eom(event.headers, write)
  File "C:\Users\AberS\Documents\Github\index.py\.venv\lib\site-packages\h11\_writers.py", line 102, in send_eom
    raise LocalProtocolError("Too little data for declared Content-Length")
h11._util.LocalProtocolError: Too little data for declared Content-Length
INFO:     127.0.0.1:5248 - "GET /static/admin/css/fonts.css HTTP/1.1" 404
INFO:     127.0.0.1:5248 - "GET /favicon.ico HTTP/1.1" 404

The three library versions involved are Django: 3.0.2, starlette: 0.13.0, uvicorn: 0.11.1.

[tomchristie]

Okay, it'll almost certainly be an issue with attempting to apply GZip compression to a response that's already GZipped compressed. You probably have a Gzip middleware installed in the Django settings, as well as on the containing Starlette application.

If you remove the GZip middleware from one of those two places then the issue should resolve. (Although we should still fix this case in Starlette.)

[abersheeran]

I checked and found that Django does not have the default Gzip behavior, and I haven't found anything about Gzip in Middleware. I don't quite understand what you mean, I only used Gzip once in starlette.

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
]

[tomchristie]

Thanks for raising this. Turns out the WSGI middleware was not lowercasing the ASGI headers, as mandated by the ASGI spec, and as a result a del headers["Content-Length"] was not working as expected in the GZipMiddleware.

[Kludex]

Hmmm... Actually, this issue was solved on #813. It was my mistake for reopening it.

The issue we're trying to solve on #1609 is not really related.