Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client sometimes fails with httpx.LocalProtocolError #7442

Closed
4 tasks done
jpuris opened this issue Nov 5, 2022 · 56 comments
Closed
4 tasks done

Client sometimes fails with httpx.LocalProtocolError #7442

jpuris opened this issue Nov 5, 2022 · 56 comments
Labels
bug Something isn't working cloud Related to Prefect Cloud upstream dependency An upstream issue caused by a bug in one of our dependencies

Comments

@jpuris
Copy link

jpuris commented Nov 5, 2022

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Occasionally it seems agent is not able to retrieve the jobs to run from Prefect Cloud. It may be hours until the issue appears.

The problem had been discovered, when we received a "Late" scheduled flow notification and consequently by checking the agent logs.

The error appeared the same day we upgraded from 2.4.0 to 2.6.5 via pip.

Prefect slack thread for reference

Reproduction

1. Prefect agent 2.6.5 (or 2.6.6) running on AWS EC2 with 22.04.1 LTS (Jammy Jellyfish) x86_64
2. `pip install prefect==2.6.6`
3. `prefect agent start -q 'my_queue'`
4. tail the logs until one sees the failed scheduled flows

Error

22:24:31.719 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
22:24:31.800 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

Versions

Version:             2.6.6
API version:         0.8.3
Python version:      3.10.6
Git commit:          87767cda
Built:               Thu, Nov 3, 2022 1:15 PM
OS/Arch:             linux/x86_64
Profile:             production
Server type:         cloud

Additional context

output of pip install prefect==2.6.6

$ pip install prefect==2.6.6
Requirement already satisfied: prefect==2.6.6 in ./venv/lib/python3.10/site-packages (2.6.6)
Requirement already satisfied: click<8.2,>=8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (8.1.3)
Requirement already satisfied: pendulum>=2.1.2 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.1.2)
Requirement already satisfied: aiofiles>=0.7.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (22.1.0)
Requirement already satisfied: pytz>=2021.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.2.1)
Requirement already satisfied: apprise>=1.0.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.0)
Requirement already satisfied: kubernetes>=24.2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (24.2.0)
Requirement already satisfied: pydantic>=1.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.10.2)
Requirement already satisfied: rich>=11.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (12.5.1)
Requirement already satisfied: anyio>=3.4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.6.1)
Requirement already satisfied: asgi-lifespan>=1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.1)
Requirement already satisfied: cloudpickle>=2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.2.0)
Requirement already satisfied: asyncpg>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.26.0)
Requirement already satisfied: aiohttp>=3.8.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.3)
Requirement already satisfied: sqlalchemy[asyncio]!=1.4.33,>=1.4.20 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.4.41)
Requirement already satisfied: cryptography>=36.0.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (38.0.1)
Requirement already satisfied: pathspec>=0.8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.1)
Requirement already satisfied: toml>=0.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.2)
Requirement already satisfied: coolname>=1.0.4 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.1.0)
Requirement already satisfied: uvicorn>=0.14.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.18.3)
Requirement already satisfied: python-slugify>=5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.1.2)
Requirement already satisfied: readchar>=3.0.6 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.0.3)
Requirement already satisfied: orjson>=3.7 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.0)
Requirement already satisfied: httpx[http2]>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.23.0)
Requirement already satisfied: docker>=4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0.0)
Requirement already satisfied: fsspec>=2022.5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.8.2)
Requirement already satisfied: typer>=0.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.6.1)
Requirement already satisfied: alembic>=1.7.5 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.8.1)
Requirement already satisfied: aiosqlite>=0.17.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.17.0)
Requirement already satisfied: pyyaml>=5.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0)
Requirement already satisfied: croniter>=1.0.12 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.3.7)
Requirement already satisfied: packaging>=21.3 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (21.3)
Requirement already satisfied: jsonpatch>=1.32 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.32)
Requirement already satisfied: griffe>=0.20.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.22.2)
Requirement already satisfied: fastapi>=0.70 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.85.0)
Requirement already satisfied: typing-extensions>=4.1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.3.0)
Requirement already satisfied: aiosignal>=1.1.2 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.3.1)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (2.1.1)
Requirement already satisfied: multidict<7.0,>=4.5 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (6.0.2)
Requirement already satisfied: attrs>=17.3.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (22.1.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.8.1)
Requirement already satisfied: Mako in ./venv/lib/python3.10/site-packages (from alembic>=1.7.5->prefect==2.6.6) (1.2.3)
Requirement already satisfied: idna>=2.8 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (3.4)
Requirement already satisfied: sniffio>=1.1 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (1.3.0)
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.16.0)
Requirement already satisfied: requests-oauthlib in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.3.1)
Requirement already satisfied: requests in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (2.28.1)
Requirement already satisfied: markdown in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (3.4.1)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.10/site-packages (from croniter>=1.0.12->prefect==2.6.6) (2.8.2)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.10/site-packages (from cryptography>=36.0.1->prefect==2.6.6) (1.15.1)
Requirement already satisfied: websocket-client>=0.32.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.4.1)
Requirement already satisfied: urllib3>=1.26.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.26.12)
Requirement already satisfied: starlette==0.20.4 in ./venv/lib/python3.10/site-packages (from fastapi>=0.70->prefect==2.6.6) (0.20.4)
Requirement already satisfied: certifi in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (2022.9.24)
Requirement already satisfied: httpcore<0.16.0,>=0.15.0 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (0.15.0)
Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (1.5.0)
Requirement already satisfied: h2<5,>=3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (4.1.0)
Requirement already satisfied: jsonpointer>=1.9 in ./venv/lib/python3.10/site-packages (from jsonpatch>=1.32->prefect==2.6.6) (2.3)
Requirement already satisfied: google-auth>=1.0.1 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (2.11.1)
Requirement already satisfied: setuptools>=21.0.0 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (59.6.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.10/site-packages (from packaging>=21.3->prefect==2.6.6) (3.0.9)
Requirement already satisfied: pytzdata>=2020.1 in ./venv/lib/python3.10/site-packages (from pendulum>=2.1.2->prefect==2.6.6) (2020.1)
Requirement already satisfied: text-unidecode>=1.3 in ./venv/lib/python3.10/site-packages (from python-slugify>=5.0->prefect==2.6.6) (1.3)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (2.13.0)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (0.9.1)
Requirement already satisfied: greenlet!=0.4.17 in ./venv/lib/python3.10/site-packages (from sqlalchemy[asyncio]!=1.4.33,>=1.4.20->prefect==2.6.6) (1.1.3)
Requirement already satisfied: h11>=0.8 in ./venv/lib/python3.10/site-packages (from uvicorn>=0.14.0->prefect==2.6.6) (0.12.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.1->prefect==2.6.6) (2.21)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (4.9)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (5.2.0)
Requirement already satisfied: hpack<5,>=4.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (6.0.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./venv/lib/python3.10/site-packages (from Mako->alembic>=1.7.5->prefect==2.6.6) (2.1.1)
Requirement already satisfied: oauthlib>=3.0.0 in ./venv/lib/python3.10/site-packages (from requests-oauthlib->apprise>=1.0.0->prefect==2.6.6) (3.2.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.4.8)
@jpuris jpuris added bug Something isn't working status:triage labels Nov 5, 2022
@zanieb zanieb added needs:research Blocked by investigation into feasibility and cause cloud Related to Prefect Cloud upstream dependency An upstream issue caused by a bug in one of our dependencies and removed status:triage labels Nov 5, 2022
@zanieb
Copy link
Contributor

zanieb commented Nov 5, 2022

Thanks for the well written bug report! We'll see what we can find out about the cause.

@zanieb
Copy link
Contributor

zanieb commented Nov 6, 2022

Possibly related: python-hyper/h2#1199

@zanieb zanieb mentioned this issue Nov 6, 2022
Open
@tjgalvin
Copy link

tjgalvin commented Nov 8, 2022

I think I am seeing this when an agent is finishing processing a flow that it has pulled down from a work queue.

11:05:16.247 | INFO    | Flow run 'quixotic-sponge' - Finished in state Completed('All states completed.')
11:05:20.310 | INFO    | Flow run 'sceptical-axolotl' - Finished in state Completed('All states completed.')
Successful readonly open of default-locked table /askapbuffer/payne/tgalvin/holography/prefect2_44641/44641/2022-10-07_184525_0.ms/FIELD: 9 columns, 1 rows
11:05:22.837 | INFO    | prefect.infrastructure.process - Process 'sceptical-axolotl' exited cleanly.
11:15:04.001 | ERROR   | prefect.agent -
Traceback (most recent call last):
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/asyncio/selector_events.py", line 854, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1274, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
    raise exc
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 84, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 148, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 177, in _receive_event
    data = await self._network_stream.read(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
    return b""
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
    response = await self._client.post(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/group/askap/miniconda3/envs/acesprefect2/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

@zanieb
Copy link
Contributor

zanieb commented Nov 8, 2022

Hi @tjgalvin, this issue is specifically for the LocalProtocolError. You are encountering a ConnectionResetError which is different, it'd be better if you opened a separate issue.

@tjgalvin
Copy link

tjgalvin commented Nov 8, 2022

Sigh -- of course it was me that found a new problem :P Sorry for my confusion.

I can definitely open up a new issue, but I am afraid I would not be able to provide a MWE to replicate the issue at the moment.

@zanieb zanieb mentioned this issue Nov 8, 2022
Closed
4 tasks
@boggdan95
Copy link

Hi ,I'm running a parent flow that maps the parameters into 8 deployments runs. All deployments runs correctly but when it finish fails and the subsequent flow cannot be triggered.

Encountered exception during execution:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/engine.py", line 1239, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/tmp/tmpepdgkb1rprefect/flows/xtra/xtra-data-ingestion.py", line 104, in trigger_EL_flow
    flow_run_model = run_deployment(name=f"XtraIngestion/{deployment_name}")
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 197, in coroutine_wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 148, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/deployments.py", line 131, in run_deployment
    flow_run = await client.read_flow_run(flow_run_id)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/orion.py", line 1439, in read_flow_run
    response = await self._client.get(f"/flow_runs/{flow_run_id}")
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
    await super().send(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
08:00:03 AM
trigger_EL_flow-5fd80932-9
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

@zanieb zanieb mentioned this issue Nov 21, 2022
Closed
3 tasks
@biancaines biancaines mentioned this issue Nov 22, 2022
Closed
4 tasks
@discdiver
Copy link
Contributor

Got it today. Running K8s through Docker Desktop. I was cleaning stuff up and then looked back in my terminal and saw the traceback. The job wouldn’t have been running more than 3 minutes.

conda list results for relevant packages
# Name                    Version                   Build  Channel
h2                        4.1.0                    pypi_0    pypi
aiohttp                   3.8.1                    pypi_0    pypi
google-auth-httplib2      0.1.0                    pypi_0    pypi
httpcore                  0.15.0                   pypi_0    pypi
httplib2                  0.20.4                   pypi_0    pypi
httpx                     0.23.0                   pypi_0    pypi

Version:             2.6.8
API version:         0.8.3
Python version:      3.10.5
Git commit:          68044e28
Built:               Thu, Nov 17, 2022 3:19 PM
OS/Arch:             darwin/arm64
Profile:             jeffmshale
Server type:         cloud

12:31:25.171 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/client/orion.py", line 758, in get_runs_in_work_queue
    response = await self._client.post(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/client/base.py", line 160, in send
    await super().send(*args, **kwargs)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/jeffhale/miniforge3/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
12:31:30.194 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
    queue_runs = await self.client.get_runs_in_work_queue(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/client/orion.py", line 758, in get_runs_in_work_queue
    response = await self._client.post(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/prefect/client/base.py", line 160, in send
    await super().send(*args, **kwargs)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/jeffhale/miniforge3/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/jeffhale/miniforge3/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

@jnoynaert
Copy link

This is impacting us as well

This was referenced Nov 30, 2022
Open
Merged
@zanieb
Copy link
Contributor

zanieb commented Dec 1, 2022

Note we believe this is addressed in 2.7.0 with #7593

@zanieb zanieb mentioned this issue Dec 8, 2022
Closed
4 tasks
@zanieb
Copy link
Contributor

zanieb commented Dec 8, 2022

We're adding #7802 to allow HTTP2 to be disabled when this issue persists.

@jpuris
Copy link
Author

jpuris commented Dec 8, 2022 •
edited
Loading

Can confirm that the issue has been fixed with 2.7.0.

We have been running said version in production since it was released and have not been able to reproduce the mentioned problem.

Thank you!

@jpuris jpuris closed this as completed Dec 8, 2022
@zanieb zanieb mentioned this issue Dec 28, 2022
Closed
4 tasks
@zanieb zanieb removed the needs:research Blocked by investigation into feasibility and cause label Dec 28, 2022
@LouisBB
Copy link

LouisBB commented Jan 13, 2023 •
edited
Loading

Hi @madkinsz , I'm running prefect 2.7.7 and had the same problem several times today and yesterday. The Prefect agent runs on an EC2 machine and runs an ECS Fargate Task. The error I got was : "httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED". Do you know how I could fix this please ? Any help would be greatly appreciated :)

@zanieb
Copy link
Contributor

zanieb commented Jan 13, 2023

Hi @LouisBB! You can disable HTTP/2 to avoid this bug in an upstream library. We're still working with the maintainers there to try to improve their HTTP/2 implementation. See #7802

@LouisBB
Copy link

LouisBB commented Jan 13, 2023

Ok great thank you very much !

@MattDelac
Copy link

FYI I was having the same issue with Prefect 2.10.6 earlier today

@zanieb
Copy link
Contributor

zanieb commented May 1, 2023

@MattDelac please include the debug logs as described in #7442 (comment)

@MohammedSiddiqui
Copy link

Also been facing this issue for the past couple of days, even on v2.10.6.

Longer-running flows just get stuck and their respective containers seem to get killed on the same ECS cluster.
It seems to be happening multiple times per day.

I will attach the agent debug logs after enabling it for a while.

akdienes added a commit to akdienes/prefect-helm that referenced this issue May 2, 2023
@akdienes
set http2 to default false
b7c2e8c 
PrefectHQ/prefect#7442
encode/httpx#2112

considering that the above two issues mean that using `http2: false` leads consistently to irrecoverable worker failures, I think the default should be false
@akdienes akdienes mentioned this issue May 2, 2023
Closed
@MohammedSiddiqui
Copy link

@madkinsz the issue popped up again crashing the agent. But thankfully we have better logs now. Attaching them here for your reference.

In the meanwhile, is there anything we can do to temporarily fix this issue? My flows keep hanging whenever the agent restarts. Would downgrading to some other version potentially fix this?

log-events-viewer-result.csv

@zanieb
Copy link
Contributor

zanieb commented May 2, 2023

@MohammedSiddiqui you can disable HTTP2 as described a few times above.

@zanieb
Copy link
Contributor

zanieb commented May 2, 2023

Unfortunately I'm not seeing much valuable context from the httpx logger. We may need to enable httpcore logs as well?

PREFECT_LOGGING_EXTRA_LOGGERS=httpx,httpcore

@MattDelac
Copy link

MattDelac commented May 3, 2023 •
edited
Loading

@MattDelac please include the debug logs as described in #7442 (comment)

Yes I will add the debugging. But FYI it failed again for no reason ... 😢 This is very frustrating

[EDIT] Just added PREFECT_LOGGING_EXTRA_LOGGERS=httpx,httpcore too. Will share the logs once available

@MattDelac
Copy link

And even if my agent started properly, it shows up as "unhealthy" in prefect cloud 🤷
image
image

@MattDelac
Copy link

I don't know if the Error getting config status, workload certificates may not be configured: HTTP 404 is an issue or not as the server does communicate through https with Prefect cloud but there is no domain name assigned to the server.

@MattDelac
Copy link

MattDelac commented May 3, 2023 •
edited
Loading

@MattDelac please include the debug logs as described in #7442 (comment)

Yes I will add the debugging. But FYI it failed again for no reason ... 😢 This is very frustrating

[EDIT] Just added PREFECT_LOGGING_EXTRA_LOGGERS=httpx,httpcore too. Will share the logs once available

And @madkinsz there is no log since the agent started even when using the following command PREFECT_LOGGING_EXTRA_LOGGERS=httpx,httpcore PREFECT_API_ENABLE_HTTP2=false prefect agent start --pool default-agent-pool

@zanieb
Copy link
Contributor

zanieb commented May 3, 2023

If you disable HTTP2 there is no reason to collect httpx/httpcore logs. We already know this issue is resolved by disabling HTTP2.

For others, I presume you'd also want to set PREFECT_LOGGING_LEVEL=DEBUG to get meaningful information from the HTTP loggers when HTTP2 is enabled.

@MattDelac
Copy link

If you disable HTTP2 there is no reason to collect httpx/httpcore logs. We already know this issue is resolved by disabling HTTP2.

For others, I presume you'd also want to set PREFECT_LOGGING_LEVEL=DEBUG to get meaningful information from the HTTP loggers when HTTP2 is enabled.

The debug just shows that my agent thinks it's connected but Prefect Cloud thinks my agent is not ...

image
image

@trahloff trahloff mentioned this issue May 4, 2023
Closed
3 tasks
@MattDelac

This comment was marked as off-topic.

Sign in to view
@espenthaem
Copy link

I'm seeing similar httpx errors while using 2.10.6. I'm using Docker on EC2 for my agent queues.

I'll try to set export PREFECT_API_ENABLE_HTTP2=false and report back in case that solves it.

Failed the last 3 attempts. Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/http2.py", line 213, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpcore/_async/http2.py", line 152, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/utilities/services.py", line 46, in critical_service_loop
    await workload()
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/agent.py", line 261, in check_for_cancelled_flow_runs
    async for work_queue in self.get_work_queues():
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/agent.py", line 144, in get_work_queues
    work_queue = await self.client.read_work_queue_by_name(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/client/orchestration.py", line 850, in read_work_queue_by_name
    response = await self._client.get(f"/work_queues/name/{name}")
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1754, in get
    return await self.request(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/client/base.py", line 251, in send
    response = await self._send_with_retry(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/client/base.py", line 194, in _send_with_retry
    response = await request()
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 260, in coroutine_wrapper
    return call()
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 245, in __call__
    return self.result()
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async
    result = await coro
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/cli/agent.py", line 201, in start
    tg.start_soon(
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/home/ec2-user/venvs/prefect_2/lib/python3.9/site-packages/prefect/utilities/services.py", line 104, in critical_service_loop
    raise RuntimeError("Service exceeded error threshold.")
RuntimeError: Service exceeded error threshold.
An exception occurred.

@zanieb
Copy link
Contributor

zanieb commented May 5, 2023

@MattDelac

RuntimeError: Timed out after 602.8259875774384s while waiting for Cloud Run Job execution to complete. Your job may still be running on GCP.
An error occured while monitoring flow run 'cdbc0be6-c964-45b5-ba1c-fce2d4e36f17'. The flow run will not be marked as failed, but an issue may have occurred.

This is a separate issue, please open a question in the prefect-gcp repository if you want to discuss that further. It looks like your flow is running longer than the default timeout.

@espenthaem

Disabling HTTP2 will definitely resolve those errors. Please consider collecting httpcore/httpx debug logs for us first!

@MattDelac

This comment was marked as off-topic.

Sign in to view
@MattDelac MattDelac mentioned this issue Apr 26, 2024
Open
1 task
@ethanve
Copy link

ethanve commented May 5, 2023

I'm also experiencing this issue running my agent on ECS Fargate. Will try to get some logs too

@zanieb
Copy link
Contributor

zanieb commented May 5, 2023

httpcore logs

14:29:53.989 | DEBUG   | httpcore - http2.send_request_headers.started request=<Request [b'GET']> stream_id=2001
14:29:53.989 | DEBUG   | httpcore - http2.send_request_headers.failed exception=ProtocolError('Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED')
14:29:53.990 | DEBUG   | httpcore - http2.response_closed.started stream_id=2001
14:29:53.990 | DEBUG   | httpcore - http2.response_closed.complete
14:29:58.128 | DEBUG   | httpcore - http2.send_request_headers.started request=<Request [b'POST']> stream_id=2001
14:29:58.128 | DEBUG   | httpcore - http2.send_request_headers.failed exception=ProtocolError('Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED')
14:29:58.129 | DEBUG   | httpcore - http2.response_closed.started stream_id=2001
14:29:58.129 | DEBUG   | httpcore - http2.response_closed.complete
14:29:58.130 | DEBUG   | prefect.client - Encountered retryable exception during request. Another attempt will be made in 7.161775481468718s. This is attempt 3/6.
Traceback (most recent call last):
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/http2.py", line 116, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/http2.py", line 213, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/dev/contrib/httpcore/httpcore/_async/http2.py", line 152, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mz/dev/prefect/src/prefect/client/base.py", line 194, in _send_with_retry
    response = await request()
               ^^^^^^^^^^^^^^^
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/mz/.pyenv/versions/3.11.2/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/mz/.pyenv/versions/3.11.2/envs/prefect-311/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc

@zanieb
Copy link
Contributor

zanieb commented May 5, 2023 •
edited
Loading

Here's a new one for

KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.RECV_WINDOW_UPDATE: 13>)

https://basedbin.fly.dev/p/ThNogZ.txt

And additional logs that include displays of the H2 Events processed

https://basedbin.fly.dev/p/21ZzfS.txt

@zanieb zanieb mentioned this issue May 5, 2023
Merged
@zanieb
Copy link
Contributor

zanieb commented May 5, 2023

Drafting a solution at encode/httpcore#679

@zachloertscher
Copy link

zachloertscher commented May 17, 2023 •
edited
Loading

Got this issue today on Prefect 2.8.7:
Running on an EC2

main error:
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

Full trace:

May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 224, in process_input
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: func, target_state = self._transitions[(self.state, input_)]
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: During handling of the above exception, another exception occurred:
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: await self._send_request_headers(request=request, stream_id=stream_id)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 213, in _send_request_headers
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 766, in send_headers
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 228, in process_input
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: raise ProtocolError(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: During handling of the above exception, another exception occurred:
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: yield
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: resp = await self._pool.handle_async_request(req)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: raise exc
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await connection.handle_async_request(request)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: return await self._connection.handle_async_request(request)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 152, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: raise LocalProtocolError(exc)  # pragma: nocover
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: The above exception was the direct cause of the following exception:
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/agent.py", line 535, in _propose_pending_state
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: state = await propose_state(self.client, Pending(), flow_run_id=flow_run.id)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/engine.py", line 1925, in propose_state
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await set_state_and_handle_waits(set_state)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/engine.py", line 1909, in set_state_and_handle_waits
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await set_state_func()
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/orchestration.py", line 1713, in set_flow_run_state
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._client.post(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1845, in post
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: return await self.request(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1530, in request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: return await self.send(request, auth=auth, follow_redirects=follow_redirects)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/base.py", line 243, in send
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_with_retry(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/base.py", line 189, in _send_with_retry
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await request()
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1617, in send
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_handling_auth(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_handling_redirects(
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_single_request(request)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1719, in _send_single_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: response = await transport.handle_async_request(request)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: resp = await self._pool.handle_async_request(req)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: self.gen.throw(type, value, traceback)
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: raise mapped_exc(message) from exc
May 17 07:00:58 ip-10-75-13-171.ec2.internal prefect[8931]: httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: 07:01:07.820 | ERROR   | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 224, in process_input
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: func, target_state = self._transitions[(self.state, input_)]
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: During handling of the above exception, another exception occurred:
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 116, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: await self._send_request_headers(request=request, stream_id=stream_id)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 213, in _send_request_headers
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 766, in send_headers
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/h2/connection.py", line 228, in process_input
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: raise ProtocolError(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: During handling of the above exception, another exception occurred:
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: yield
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: resp = await self._pool.handle_async_request(req)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: raise exc
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await connection.handle_async_request(request)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: return await self._connection.handle_async_request(request)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpcore/_async/http2.py", line 152, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: raise LocalProtocolError(exc)  # pragma: nocover
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: The above exception was the direct cause of the following exception:
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: Traceback (most recent call last):
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/agent.py", line 212, in get_and_submit_flow_runs
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: queue_runs = await self.client.get_runs_in_work_queue(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/orchestration.py", line 907, in get_runs_in_work_queue
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._client.post(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1845, in post
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: return await self.request(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1530, in request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: return await self.send(request, auth=auth, follow_redirects=follow_redirects)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/base.py", line 243, in send
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_with_retry(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/prefect/client/base.py", line 189, in _send_with_retry
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await request()
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1617, in send
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_handling_auth(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_handling_redirects(
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await self._send_single_request(request)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_client.py", line 1719, in _send_single_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: response = await transport.handle_async_request(request)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: resp = await self._pool.handle_async_request(req)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: self.gen.throw(type, value, traceback)
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: File "/home/airbyte/.local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: raise mapped_exc(message) from exc
May 17 07:01:07 ip-10-75-13-171.ec2.internal prefect[8931]: httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

I'll try setting the PREFECT_API_ENABLE_HTTP2=False and see if this helps

@zanieb
Copy link
Contributor

zanieb commented May 18, 2023 •
edited
Loading

With the latest httpcore release my fix is available upstream.

pip install -U httpcore

to resolve.

@zanieb zanieb closed this as completed May 18, 2023
@tomchristie tomchristie mentioned this issue Jun 16, 2023
Closed
@j-tr j-tr mentioned this issue Jan 17, 2024
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working cloud Related to Prefect Cloud upstream dependency An upstream issue caused by a bug in one of our dependencies
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests