Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ValueError: Can't patch loop of type <class 'uvloop.Loop'> #1270

Open
gustavo-has-stone opened this issue Sep 10, 2024 · 5 comments
Open

ValueError: Can't patch loop of type <class 'uvloop.Loop'> #1270

gustavo-has-stone opened this issue Sep 10, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@gustavo-has-stone
Copy link

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
I am trying to integrate RAGAS with an API based on FastAPI, and I'm getting an error: ValueError: Can't patch loop of type <class 'uvloop.Loop'>

Ragas version: 0.1.15
Python version: 3.10.12

Code to Reproduce

from datasets import Dataset
from fastapi import FastAPI
from ragas import evaluate
from ragas.metrics import faithfulness

app = FastAPI()

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
}

async def async_evaluation():
    dataset = Dataset.from_dict(data_samples)
    score = evaluate(dataset,metrics=[faithfulness])
    return {"score": score['faithfulness']}

@app.get("/evaluate")
async def evaluate_data():
    return await async_evaluation()

Error trace

Traceback (most recent call last):
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 39, in evaluate_data
    return await async_evaluation()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/main.py", line 33, in async_evaluation
    score = evaluate(dataset,metrics=[faithfulness])
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/_analytics.py", line 129, in wrapper
    result = func(*args, **kwargs)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 280, in evaluate
    raise e
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 260, in evaluate
    results = executor.results()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/ragas/executor.py", line 93, in results
    nest_asyncio.apply()
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 19, in apply
    _patch_loop(loop)
  File "/home/gustavo.dossantos/Desktop/tests/ragas-haystack/.venv/lib/python3.10/site-packages/nest_asyncio.py", line 193, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>

Expected behavior
I expected to be able to integrate RAGAS with our API based on FastAPI

Additional context

@gustavo-has-stone gustavo-has-stone added the bug Something isn't working label Sep 10, 2024
Copy link

dosubot bot commented Sep 10, 2024

Hey there, @gustavo-has-stone! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

To resolve the ValueError: Can't patch loop of type <class 'uvloop.Loop'> when integrating RAGAS with a FastAPI-based API, you need to ensure that nest_asyncio is not applied when using uvloop. The uvloop event loop is not compatible with nest_asyncio.

You can modify the results method in the Executor class to check if uvloop is being used and avoid applying nest_asyncio in that case. Here is the updated code:

def results(self) -> t.List[t.Any]:
    if is_event_loop_running():
        # an event loop is running so call nested_asyncio to fix this
        try:
            import uvloop
            if isinstance(asyncio.get_event_loop(), uvloop.Loop):
                raise RuntimeError("Cannot use nest_asyncio with uvloop")
        except ImportError:
            pass

        try:
            import nest_asyncio
        except ImportError:
            raise ImportError(
                "It seems like you're running this in a jupyter-like environment. Please install nest_asyncio with `pip install nest_asyncio` to make it work."
            )
        if not self._nest_asyncio_applied:
            nest_asyncio.apply()
            self._nest_asyncio_applied = True

    # create a generator for which returns tasks as they finish
    futures_as_they_finish = as_completed(
        coros=[afunc(*args, **kwargs) for afunc, args, kwargs, _ in self.jobs],
        max_workers=(self.run_config or RunConfig()).max_workers,
    )

    async def _aresults() -> t.List[t.Any]:
        results = []
        for future in tqdm(
            futures_as_they_finish,
            desc=self.desc,
            total=len(self.jobs),
            # whether you want to keep the progress bar after completion
            leave=self.keep_progress_bar,
        ):
            r = await future
            results.append(r)

        return results

    results = asyncio.run(_aresults())
    sorted_results = sorted(results, key=lambda x: x[0])
    return [r[1] for r in sorted_results]

This modification checks if uvloop is being used and raises a RuntimeError if an attempt is made to apply nest_asyncio with uvloop. This should resolve the ValueError you are encountering [1].

To continue talking to Dosu, mention @dosu.

@tradicio
Copy link

Hi, I also encountered the same error using ragas versions earlier than 0.1.15 (I would say I tried 0.1.1 to 0.1.6).

Do you know what is a way to solve the problem other than editing the results method in the source code? I don't find the solution proposed by the bot very practical.

Thank you very much!

@richness
Copy link

I'm having the same issue,Can't patch loop of type <class 'uvloop.Loop'>。Did you solve it?

@gustavo-has-stone
Copy link
Author

I didn't. The only workaround I know would involve using asyncio when running uvicorn

@richness
Copy link

I'm really introducing Ragas in the Sanic frame,
app = Sanic("CharAI")
app.config.USE_UVLOOP = False
can be solved, but it will reduce Saanich's performance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants