Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vLLM 5.3+ support #60

Merged
merged 2 commits into from
Jul 24, 2024
Merged

vLLM 5.3+ support #60

merged 2 commits into from
Jul 24, 2024

Conversation

joerunde
Copy link
Contributor

Description

This PR adds the necessary changes (that I found, at least) to run the adapter with vllm 0.5.3.

The main change was the removal of TextTokensPrompt, it was removed with this PR: vllm-project/vllm@739b61a
I'm not 100% sure if simply replacing with LLMInputs is the correct fix here, but it does seem to work. (I think I see some more complex processing happening on the upstream openai serving engine)

The initializers for all of the OpenAIServing* classes changed, I copied over the new defs from upstream.

Also I noticed that there was no initialization for the tokenizer, so I added it.

How Has This Been Tested?

Booting up a pod with a mig gpu slice, installing vllm@5.3 and vllm-tgis-adapter@llm-inputs, and running the default facebook/opt-125m model with python3 -m vllm_tgis_adapter.

Requests sent using the swagger page at /docs and grpcui (though grpc should be unaffected)

Merge criteria:

  • The commits are squashed in a cohesive manner and have meaningful messages.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work

@codecov-commenter
Copy link

codecov-commenter commented Jul 23, 2024

Codecov Report

Attention: Patch coverage is 52.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 62.73%. Comparing base (0f7df61) to head (60f1ad0).
Report is 1 commits behind head on main.

Files Patch % Lines
src/vllm_tgis_adapter/__main__.py 42.85% 11 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #60      +/-   ##
==========================================
- Coverage   62.96%   62.73%   -0.24%     
==========================================
  Files          18       18              
  Lines        1288     1280       -8     
  Branches      229      227       -2     
==========================================
- Hits          811      803       -8     
  Misses        399      399              
  Partials       78       78              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dtrifiro dtrifiro force-pushed the llm-inputs branch 2 times, most recently from d94573a to 60f1ad0 Compare July 24, 2024 12:34
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
transformers is one of `vllm`'s requirement, hence pinning it here
can cause dependency issues
@joerunde joerunde added this pull request to the merge queue Jul 24, 2024
Merged via the queue into main with commit df76b22 Jul 24, 2024
3 checks passed
@joerunde joerunde deleted the llm-inputs branch July 24, 2024 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants