Resolution for "Unclosed connection" warning

[rijulg]

Background

Lately in our production system we have been receiving numerous "Unclosed connection" errors (warnings), due to our logging stack's peculiarities this resulted in a large number of false positives that has caused serious alert fatigue.

Since the previous discussion thread on this issue (#5277) has been closed and I found a solution that worked for us I felt this is the best place to share our findings.

The problem

The following two snippets of code raise the "Unclosed connection" warning because when the python object corresponding to the requested is garbage cleaned (__del__ted) the request has not been closed. (code that throws the warning)

import asyncio
import aiohttp

async def req():
    async with aiohttp.ClientSession() as session:
        await session.get("http://localhost:4000")

async def main(i=10):
    for _ in range(i):
        await req()

if __name__ == "__main__":
    asyncio.run(main())

import asyncio
import aiohttp

async def req():
    async with aiohttp.ClientSession() as session:
        for _ in range(10):
            # simulate multiple requests in same session
            await session.get("http://localhost:4000")

async def main(i=10):
    for _ in range(i):
        await req()

if __name__ == "__main__":
    asyncio.run(main())

Both of the codes throw the following error on stderr

Unclosed connection
client_connection: Connection<ConnectionKey(host='localhost', port=4000, is_ssl=False, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>

Solution

Since we identified that the root cause of the error is the unclosed connection we know that we need to close the connection. This was indicated by @kk here (#5277 (comment)), however we still need to handle the case for multiple requests in a session. To address multiple requests we empirically verified that closing the last request object was enough.

The following two snippets throw no warnings/errors

import asyncio
import aiohttp

async def req():
    async with aiohttp.ClientSession() as session:
        r = await session.get("http://localhost:4000")
        r.close()

async def main(i=10):
    for _ in range(i):
        await req()

if __name__ == "__main__":
    asyncio.run(main())

import asyncio
import aiohttp

async def req():
    async with aiohttp.ClientSession() as session:
        for _ in range(10):
            # simulate multiple requests in same session
            r = await session.get("http://localhost:4000")
        r.close()

async def main(i=10):
    for _ in range(i):
        await req()

if __name__ == "__main__":
    asyncio.run(main())

[webknjaz]

Use async with session.get() as http_response: ... so that you don't forget to close resources.