Host header position causes certain sites to not respond

[Hemable1]

Outbound requests add the Host header last instead of first which causes an issue fetching certain sites. Normally this shouldn't matter, however I'm coming across servers that won't respond unless it's defined first as it is in a browser.

Example:
This is through a browser and successfully responds:

GET / HTTP/1.1
Host: www.accuweather.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: */*
Accept-Encoding: gzip, deflate
Connection: close

This is through aiohttp and does not respond (notice the Host header position):

GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: */*
Accept-Encoding: gzip, deflate
Host: www.accuweather.com
Connection: close

To replicate this behavior use the the following code and notice it will time out.

import asyncio
import aiohttp

async def req(url):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0'
    }
    timeout = aiohttp.ClientTimeout(total=10)

    async with aiohttp.ClientSession(timeout=timeout, headers=headers) as session:
        async with session.get(url, ssl=False) as resp:
            print(resp.status)
            print(await resp.text())

loop = asyncio.get_event_loop()
loop.run_until_complete(req("https://www.accuweather.com"))

Tested on:
Windows 7x64
Python 3.7.0
aiohttp 3.3.2

[asvetlov]

I'm sorry but the server is not correct.

We can change an order of request headers construction to put Host first but in general there is no guarantee to satisfy other weird servers requirements.

[Hemable1]

Yes the server is incorrect, however this should be developed to replicate a browser as closely as possible. There are a few of sites that fall into this category, https://www.nasdaq.com is another one. No browsers I've tried had any issues visiting these sites.

[webknjaz]

😮

[asvetlov]

The patch is pretty easy.

1. Update Host before processing default headers in client_reqrep.ClientRequest.update_auto_headers.
2. Add a functional test to make sure that the order is saved. The test should check headers order on server side
3. I doubt if we need support Host header overriding. Raise ValueError is Host key is passed to client session default headers.

The PR is very appreciated.

[GenericStudent]

@asvetlov, I've created a PR with initial attempt at this. I have moved the Host header adding under update_auto_headers to be before the default headers. This works and I have added a test for it. But I don't feel the test is optimal.

How do I check / test the view / state of the headers as seen from the server perspective?

Also I'm not sure what you meant by the third point.

[asvetlov]

Regarding tests: please take a look on https://github.com/aio-libs/aiohttp/blob/master/tests/test_client_functional.py
On server side request.headers has request's headers in the same order as they are sent.

The third point is about client = aiohttp.ClientSession({'host': 'overridden.host.com'}). It is weird, please don't care about

[asvetlov]

Fixed by #3342

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new