aiohttp 2.3 + python 3.5 broken because of multidict 4.4

[labbati]

Long story short

Using python 3.5 with aiohttp 2.3 (or less) results in a TypeError: Cannot subscript an existing Union. Use Union[u, t] instead. error.

Expected behaviour

Given that aiohttp declares python 3.4.2 as the minimum requirement, it should be possible to use aiohttp 2.x with python 3.5.

Steps to reproduce

Given the sample.py file containing the example from the main README:

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, 'http://python.org')
        print(html)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Here is the actual behavior:

$> python --version
Python 3.5.2

$> pip install "aiohttp>=2.3,<2.4"

$> pip freeze
pip freeze
aiohttp==2.3.10
async-timeout==3.0.0
chardet==3.0.4
idna==2.7
idna-ssl==1.1.0
multidict==4.4.1
yarl==1.2.6

$> python sample.py
Traceback (most recent call last):
  File "sample.py", line 1, in <module>
    import aiohttp
  File "/Users/user_name/projects/dd-trace-py/aaa/lib/python3.5/site-packages/aiohttp/__init__.py", line 5, in <module>
    from . import hdrs  # noqa
  File "/Users/user_name/projects/dd-trace-py/aaa/lib/python3.5/site-packages/aiohttp/hdrs.py", line 2, in <module>
    from multidict import istr
  File "/Users/user_name/projects/dd-trace-py/aaa/lib/python3.5/site-packages/multidict/__init__.py", line 18, in <module>
    from ._abc import MultiMapping, MutableMultiMapping
  File "/Users/user_name/projects/dd-trace-py/aaa/lib/python3.5/site-packages/multidict/_abc.py", line 37, in <module>
    class MutableMultiMapping(MultiMapping[_T], MutableMapping[str, _T]):
  File "/Users/user_name/projects/dd-trace-py/aaa/lib/python3.5/site-packages/multidict/_abc.py", line 44, in MutableMultiMapping
    def extend(self, *args: _Arg[_T], **kwargs: _T) -> None:
  File "/Users/user_name/.pyenv/versions/3.5.2/lib/python3.5/typing.py", line 546, in __getitem__
    "Cannot subscript an existing Union. Use Union[u, t] instead.")
TypeError: Cannot subscript an existing Union. Use Union[u, t] instead.

That is fixed downgrading the multidict version to 4.3.x.

$> pip install "multidict>=4.3,<4.4"

$> pip freeze
aiohttp==2.3.10
async-timeout==3.0.0
chardet==3.0.4
idna==2.7
idna-ssl==1.1.0
multidict==4.3.1
yarl==1.2.6

$> python sample.py
<some-beautiful-html />

Your environment

I can tell that the issue applies to the client, but it may be also apply to the server.

Environment, both:

• Docker: Linux linuxkit-025000000001 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• MAC OS: Darwin COMP10588.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:59:54 PDT 2018; root:xnu-4570.71.2~2/RELEASE_X86_64 x86_64

Installed packages (when error occurs)

aiohttp==2.3.10
async-timeout==3.0.0
chardet==3.0.4
idna==2.7
idna-ssl==1.1.0
multidict==4.4.1    <<<<<<<<<<<<<<<<<<<<<<
yarl==1.2.6

Installed packages (when error does not occur):

aiohttp==2.3.10
async-timeout==3.0.0
chardet==3.0.4
idna==2.7
idna-ssl==1.1.0
multidict==4.3.1    <<<<<<<<<<<<<<<<<<<<<<
yarl==1.2.6

The same issue seems to exist for aiohttp 1.x

Possible fix

It seems to me that pinning the dependency from current 'multidict>=4.0.0' to 'multidict>=4.0.0,<4.4' should be enough. But there may be more appropriate solutions.

[asvetlov]

GitMate.io thinks possibly related issues are #2057 (aiohttp-2.x.x wheels for Python 3.5 include .so files linked against Python 3.4), #1763 (Odd Python 3.5/3.6 error when using aiohttp 2.0.4), #1764 (Odd Python 3.5/3.6 error when using aiohttp 2.0.4), #1839 (aiohttp webserver with python 3.7 broken), and #676 (Python 3.4.0 support).

[Harmon758]

Relevant multidict issue: aio-libs/multidict#286

[asvetlov]

Fixed by multidict 4.4.2 release.
Note: aiohttp 3.x worth to be upgraded on :)

[labbati]

Hi @asvetlov, thanks for the quick response and action.

Fixed by multidict 4.4.2 release.

It works perfectly, thanks!

Note: aiohttp 3.x worth to be upgraded on :)

I do certainly agree with you. The point for us (https://github.com/DataDog/dd-trace-py) is that we have to integrate with and support a diverse set of dependency versions and we still need to provide support for aiohttp < 3

Have a good day!

[asvetlov]

Welcome!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new