Opensource examples - Runpod w/Text-Generation-WebUI API Endpoint #247
Conversation
…nstructor into opensource_examples
…nstructor into opensource_examples
…nstructor into opensource_examples
WalkthroughThe recent updates introduce integration with the Openrouter and Runpod APIs, enhancing the environment with new variables and endpoints. A notable addition is the Changes
TipsChat with CodeRabbit Bot (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (2)
- examples/open_source_examples/README.md (1 hunks)
- examples/open_source_examples/runpod.py (1 hunks)
Files not summarized due to errors (1)
- examples/open_source_examples/runpod.py: Error: Disallowed special token found: <|im_start|>
Files not reviewed due to errors (1)
- examples/open_source_examples/runpod.py (Error: Disallowed special token found: <|im_start|>)
Additional comments: 2
examples/open_source_examples/README.md (2)
11-26: The summary mentions the addition of an Openrouter API endpoint, but the hunk shows the addition of a Perplexity API endpoint. Please verify the correct service and update the documentation accordingly.
11-26: The instructions for setting up Perplexity and Runpod accounts are clear and detailed, providing users with step-by-step guidance.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 7
Configuration used: CodeRabbit UI
Files selected for processing (2)
- examples/open_source_examples/README.md (1 hunks)
- examples/open_source_examples/runpod.py (1 hunks)
Additional comments: 1
examples/open_source_examples/README.md (1)
- 25-26: The instructions for setting up the Runpod environment are clear and detailed, including the addition of the Runpod API endpoint and key to the environment variables. Ensure that the placeholder
your-runpod-linkis replaced with the actual link provided by Runpod after deployment.
| 4. Add Openrouter API endpoint to environment - `export PERPLEXITY_BASE_URL=https://api.perplexity.ai` [See https://docs.perplexity.ai/reference/post_chat_completions for potential updates] |
There was a problem hiding this comment.
The environment variable PERPLEXITY_BASE_URL is incorrectly referred to as the Openrouter API endpoint in the instructions. This should be corrected to reflect the Perplexity API endpoint.
- 4. Add Openrouter API endpoint to environment - `export PERPLEXITY_BASE_URL=https://api.perplexity.ai`
+ 4. Add Perplexity API endpoint to environment - `export PERPLEXITY_BASE_URL=https://api.perplexity.ai`Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| 4. Add Openrouter API endpoint to environment - `export PERPLEXITY_BASE_URL=https://api.perplexity.ai` [See https://docs.perplexity.ai/reference/post_chat_completions for potential updates] | |
| 4. Add Perplexity API endpoint to environment - `export PERPLEXITY_BASE_URL=https://api.perplexity.ai` [See https://docs.perplexity.ai/reference/post_chat_completions for potential updates] |
| models = [ | ||
| "TheBloke_OpenHermes-2.5-Mistral-7B-GPTQ", | ||
| ] |
There was a problem hiding this comment.
The models list is defined but not used in the code. If it's intended for future use, consider adding a comment to clarify its purpose, or remove it if it's not needed.
it's good to go! |
There was a problem hiding this comment.
Review Status
Actionable comments generated: 5
Configuration used: CodeRabbit UI
Files selected for processing (2)
- examples/open_source_examples/openrouter.py (1 hunks)
- examples/open_source_examples/runpod.py (1 hunks)
Files skipped from review due to trivial changes (1)
- examples/open_source_examples/openrouter.py
| runpod_api_key = os.environ.get("RUNPOD_API_KEY") | ||
| assert runpod_api_key, "RUNPOD_API_KEY is not set in environment variables" |
There was a problem hiding this comment.
The use of assert for checking environment variables can cause the application to crash if the variables are not set. Consider handling this with proper error messages and exit the application gracefully.
- assert runpod_api_key, "RUNPOD_API_KEY is not set in environment variables"
+ if not runpod_api_key:
+ raise ValueError("RUNPOD_API_KEY is not set in environment variables")Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| runpod_api_key = os.environ.get("RUNPOD_API_KEY") | |
| assert runpod_api_key, "RUNPOD_API_KEY is not set in environment variables" | |
| runpod_api_key = os.environ.get("RUNPOD_API_KEY") | |
| if not runpod_api_key: | |
| raise ValueError("RUNPOD_API_KEY is not set in environment variables") |
| runpod_base_url = os.environ.get("RUNPOD_BASE_URL") | ||
| assert runpod_base_url, "RUNPOD_BASE_URL is not set in environment variables" |
There was a problem hiding this comment.
Similar issue as above with RUNPOD_BASE_URL. Replace the assert with a more graceful error handling mechanism.
- assert runpod_base_url, "RUNPOD_BASE_URL is not set in environment variables"
+ if not runpod_base_url:
+ raise ValueError("RUNPOD_BASE_URL is not set in environment variables")Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| runpod_base_url = os.environ.get("RUNPOD_BASE_URL") | |
| assert runpod_base_url, "RUNPOD_BASE_URL is not set in environment variables" | |
| runpod_base_url = os.environ.get("RUNPOD_BASE_URL") | |
| if not runpod_base_url: | |
| raise ValueError("RUNPOD_BASE_URL is not set in environment variables") |
| class UserDetail(BaseModel): | ||
| name: str = Field(description="Name extracted from the text") | ||
| age: int = Field(description="Age extracted from the text") | ||
| occupation: Optional[str] = Field( | ||
| default=None, description="Occupation extracted from the text" | ||
| ) |
There was a problem hiding this comment.
The UserDetail class is defined within the if __name__ == "__main__": block, which limits its scope. If this class is intended for use outside of this script, consider moving it to the module level.
| try: | ||
| user = client.chat.completions.create( | ||
| response_model=UserDetail, | ||
| model="TheBloke_OpenHermes-2.5-Mistral-7B-GPTQ", |
There was a problem hiding this comment.
The model name is hardcoded in the client.chat.completions.create call. Consider using the models list or an environment variable to make it configurable.
- model="TheBloke_OpenHermes-2.5-Mistral-7B-GPTQ",
+ model=models[0], # Assuming models list is intended to be used hereCommittable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| model="TheBloke_OpenHermes-2.5-Mistral-7B-GPTQ", | |
| model=models[0], # Assuming models list is intended to be used here |
| except Exception as e: | ||
| print(f"Error: {e}") |
There was a problem hiding this comment.
The error handling in the loop is basic and only prints the error message. Consider implementing a more robust error handling strategy, such as logging the errors to a file or an error tracking system.
Adding Runpod example that utlizes a premade Ubuntu LLM Container on Runpod, allowing the ability to quickly create an endpoint for the variety of GPTQ models available via TheBloke[https://huggingface.co/TheBloke], with Openai compatible endpoint.
Example utilzes Openhermes 2.5 Mistral 7B - GPTQ model.
Removed Maybe BaseModel, response model = UserDetail.
Summary by CodeRabbit
Documentation
New Features
UserDetailclass to enhance data handling within the application.