Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Proxy rotation is not possible when using a session #294

Closed
ZentixUA opened this issue Apr 13, 2024 · 3 comments
Closed

[BUG] Proxy rotation is not possible when using a session #294

ZentixUA opened this issue Apr 13, 2024 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@ZentixUA
Copy link

ZentixUA commented Apr 13, 2024

Describe the bug
When using curl-cffi, using a proxy through Proxy Gateway for rotation on each request becomes impossible (the IP becomes static for some reason, despite the fact that static sessions are disabled in the proxy provider's settings).

To Reproduce

  1. Install aiohttp using pip install aiohttp
  2. Install httpx using pip install httpx
  3. Install curl-cffi using pip install curl-cffi
  4. Paste this example:
import asyncio
import sys

from aiohttp import ClientSession
from curl_cffi.requests import AsyncSession
from httpx import AsyncClient

PROXY_URL = 'YOUR_PROXY_GATEWAY_URL'
REPETITION_COUNT = 3


async def test_aiohttp():
    print('----- aiohttp -----')
    async with ClientSession() as session:
        for i in range(REPETITION_COUNT):
            async with session.get('http://httpbin.org/get', proxy=PROXY_URL) as response:
                response_json = await response.json()
            origin = response_json['origin']
            print(f'Request #{i}, origin: {origin}')


async def test_httpx():
    print('----- httpx -----')
    async with AsyncClient(proxy=PROXY_URL) as session:
        for i in range(REPETITION_COUNT):
            response = await session.get('http://httpbin.org/get')
            origin = response.json()['origin']
            print(f'Request #{i}, origin: {origin}')


async def test_curl_cffi():
    print('----- curl-cffi -----')
    async with AsyncSession(proxy=PROXY_URL) as session:
        for i in range(REPETITION_COUNT):
            response = await session.get('http://httpbin.org/get')
            origin = response.json()['origin']
            print(f'Request #{i}, origin: {origin}')


if __name__ == '__main__':
    if sys.platform == 'win32':
        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(test_aiohttp())
    asyncio.run(test_httpx())
    asyncio.run(test_curl_cffi())
  1. Replace PROXY_URL with the URL of your proxy gateway
  2. Run this code and look at the output in the console :)

Expected behavior
The IP changes with each request, similar to how it happens in aiohttp or httpx.

Versions

  • OS: Windows 11
  • curl_cffi version: 0.6.2
  • pip freeze dump:
aiohttp==3.9.4
aiosignal==1.3.1
anyio==4.3.0
async-timeout==4.0.3
attrs==23.2.0
certifi==2024.2.2
cffi==1.16.0
curl_cffi==0.6.2
exceptiongroup==1.2.0
frozenlist==1.4.1
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
idna==3.7
multidict==6.0.5
pycparser==2.22
sniffio==1.3.1
typing_extensions==4.11.0
yarl==1.9.4

Additional context

  • Which session are you using? Async.
  • If using async session, which loop implementation are you using? Asyncio.
@ZentixUA ZentixUA added the bug Something isn't working label Apr 13, 2024
@perklet
Copy link
Collaborator

perklet commented Apr 14, 2024

Things are complicated when you use so-called tunnel proxies where the exit node changes but proxy server url does not change. When the proxy url stays the same, it's reasonable that the client reuses the old connections, actually I think the way libcurl handles it is the correct way.

A few things you can try to see if it helps:

  • Turn off http/2, requests.get(url, http_version=CurlHttpVersion.V1_1)
  • Use the proxy when requesting, session.get(url, proxy=...)

@ZentixUA
Copy link
Author

The solutions you provided did not work (at least in my case). However, I found two solutions myself.

According to this documentation, the Connection header can be set to close to ensure that the connection will close after processing the request. HTTP 1.0 has close as the default, so here are two solutions:

  1. headers={'Connection': 'close'}
  2. http_version=CurlHttpVersion.V1_0

@perklet
Copy link
Collaborator

perklet commented Apr 14, 2024

Thanks for posting your solution here. I would not recommend http/1.0 as it has been deprecated for a very long time. Adding connection: close is the right way to go if it solves your problem.

@perklet perklet closed this as completed Apr 14, 2024
@perklet perklet pinned this issue Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants