Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc_server: fix tokenizer group usage #4

Merged
merged 1 commit into from
Apr 24, 2024

Conversation

dtrifiro
Copy link

@dtrifiro dtrifiro commented Apr 24, 2024

fixes the following crash on startup

ERROR:    Traceback (most recent call last):
  File "/opt/vllm/lib/python3.11/site-packages/starlette/routing.py", line 732, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/opt/vllm/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/vllm/entrypoints/openai/api_server.py", line 50, in lifespan
    grpc_server = await start_grpc_server(async_llm_engine, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/vllm/entrypoints/grpc/grpc_server.py", line 611, in start_grpc_server
    await service._post_init()
  File "/workspace/vllm/entrypoints/grpc/grpc_server.py", line 106, in _post_init
    self.tokenizer_group = await self.engine.get_tokenizer_group()
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'AsyncLLMEngine' object has no attribute 'get_tokenizer_group'

ERROR:    Application startup failed. Exiting.

@dtrifiro dtrifiro requested a review from z103cb April 24, 2024 13:08
@z103cb
Copy link

z103cb commented Apr 24, 2024

/lgtm

@z103cb
Copy link

z103cb commented Apr 24, 2024

/approve

Copy link

openshift-ci bot commented Apr 24, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dtrifiro, z103cb

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@dtrifiro dtrifiro merged commit 1e44f5e into opendatahub-io:ibm_main Apr 24, 2024
1 check was pending
@dtrifiro dtrifiro deleted the fix-tokenizer-group branch April 24, 2024 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants