Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate vllm with example Lora and Mistral #3077

Merged
merged 17 commits into from
May 3, 2024
Merged

Integrate vllm with example Lora and Mistral #3077

merged 17 commits into from
May 3, 2024

Conversation

lxning
Copy link
Collaborator

@lxning lxning commented Apr 7, 2024

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test Lora
cd serve/examples/large_models/vllm/lora
cat prompt.json
{
  "prompt": "A robot may not injure a human being",
  "max_new_tokens": 50,
  "temperature": 0.8,
  "logprobs": 1,
  "prompt_logprobs": 1,
  "max_tokens": 128,
  "adapter": "adapter_1"
}
python ../../utils/test_llm_streaming_response.py -m lora -o 50 -t 1 -n 1 --prompt-text "@prompt.json" --prompt-json
Tasks are completed
payload={'prompt': 'A robot may not injure a human being', 'max_new_tokens': 50, 'temperature': 0.8, 'logprobs': 1, 'prompt_logprobs': 1, 'max_tokens': 128, 'adapter': 'adapter_1'}
, output= or, through inaction, allow a human being to come to harm
o
  • Test Mistral
cd serve/examples/large_models/vllm/mistral
cat prompt.json
{
  "prompt": "A robot may not injure a human being",
  "max_new_tokens": 50,
  "temperature": 0.8,
  "logprobs": 1,
  "prompt_logprobs": 1,
  "max_tokens": 128
}
python ../../utils/test_llm_streaming_response.py -m mistral -o 50 -t 1 -n 1 --prompt-text "@prompt.json" --prompt-json
Tasks are completed
payload={'prompt': 'A robot may not injure a human being', 'max_new_tokens': 50, 'temperature': 0.8, 'logprobs': 1, 'prompt_logprobs': 1, 'max_tokens': 128}
, output= or, through inaction, allow a human being to come to harm.
  • Test pytest
pytest test_example_vllm.py::test_vllm_lora_mar
======================= test session starts ========================
platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/ubuntu/serve
plugins: cov-4.1.0, mock-3.12.0
collected 1 item

test_example_vllm.py .                                       [100%]

================== 1 passed in 141.14s (0:02:21) ===================

Checklist:

  • Did you have fun?
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@lxning lxning self-assigned this Apr 7, 2024
@lxning lxning changed the title [WIP] Lora example integrate vllm with example Lora and Mistral Apr 30, 2024
@lxning lxning changed the title integrate vllm with example Lora and Mistral Integrate vllm with example Lora and Mistral Apr 30, 2024
@lxning lxning requested review from agunapal and mreso April 30, 2024 19:45
@lxning lxning added documentation Improvements or additions to documentation enhancement New feature or request labels Apr 30, 2024
@lxning lxning added this to the v0.10.1 milestone Apr 30, 2024
Copy link
Collaborator

@mreso mreso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

"--prompt-json",
action=argparse.BooleanOptionalAction,
default=False,
help="Flag the imput prompt is a json format with prompt parameters",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

enable_lora: true
max_loras: 4
max_cpu_loras: 4
max_num_seqs: 16
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vllm uses paged attention which typically allows for a larger batch sizes. We need to figure out a way to saturate the engine as setting batchSize == max_num_seqs will lead to under utilization.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use a similar strategy like for the micro-batching to always have enough requests for the engine available. Preferred would be an async mode which will just route all requests to the backend and gets replies asynchronously (as discussed earlier)

)

model_archiver.generate_model_archive(config)
shutil.move(LORA_SRC_PATH / "model", mar_file_path)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we move the files and delete them we can run the test only once before we need to put them back manually. Can we use symbolic links instead?

@mreso mreso enabled auto-merge May 3, 2024 05:59
@mreso mreso added this pull request to the merge queue May 3, 2024
Merged via the queue into master with commit f2c26f3 May 3, 2024
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants