Added LLama-2-13b-Chat plugin and modified README.md #5
Conversation
Cfn nag fixes
README.md
Outdated
| @@ -10,6 +10,7 @@ The directions below explain how to build and deploy the plugins. For more infor | |||
| 1. AI21 LLM: Uses AI21's Jurassic model API - requires an AI21 account with an API Key | |||
| 2. Anthropic LLM: Uses Anthropic's Claude model API - requires an Anthropic account with an API Key | |||
| 3. Amazon Bedrock Embeddings and LLM: Uses Amazon Bedrock service API (preview) - requires access to Amazon Bedrock service (currently in private preview) | |||
| 4. Llama 2 13b Chat LLM: Uses Llama 2 13b Chat model - requires Llama-2-chat model to be deployed via SageMaker JumpStart. | |||
There was a problem hiding this comment.
Add a reference link to point users to guide on how to install the model in SageMaker JumpStart
| import io | ||
| from typing import Dict | ||
|
|
||
| # TEMPERATURE = os.environ.get("TEMPERATURE", 1e-10) |
There was a problem hiding this comment.
Remove commented out code throughout, if not used, for cleanliness.
Presumably these settings are supplied via model parameters in the event, not local function environment vars.
| # MAX_NEW_TOKENS = os.environ.get("MAX_NEW_TOKENS", 1024) # max number of tokens to generate in the output | ||
|
|
||
| # grab environment variables | ||
| ENDPOINT_NAME = os.environ['ENDPOINT_NAME'] |
There was a problem hiding this comment.
Can you rename to SAGEMAKER_ENDPOINT_NAME so it doesn't get confused with ENDPOINT_URL used in the other functions.
| Description: QnABot on AWS LLM Plugin for Llama-2-13b-chat | ||
|
|
||
| Parameters: | ||
|
|
There was a problem hiding this comment.
I think we need parameters to identify the (existing)
- SageMakerEndpointName, and
- SageMakerEndpointArn
And add more helpful text to the parameter descriptions to make it very clear how they should provision the endpoint (eg Jumpstart docs) and obtain the Name and Arn.
| Action: | ||
| - "sagemaker:InvokeEndpoint" | ||
| Resource: | ||
| - "*" |
There was a problem hiding this comment.
Scope this to be least privilege.. Lambda only needs to invoke the specific endpoint specified as a parameter eg SageMakerEndpointArn
|
|
||
| Parameters: | ||
|
|
||
| LLMModel: |
There was a problem hiding this comment.
Is this meant to me the SageMaker endpoint name? If se, let's rename it to SageMakerEndpointName and provide more complete description.
And also add one for Arn, so you can scope the invoke policy properly.
| Runtime: python3.10 | ||
| Environment: | ||
| Variables: | ||
| ENDPOINT_NAME: !Ref LLMModel |
There was a problem hiding this comment.
As noted above, param name is confusing, since you want the endpoint name, not the model. E.g. Call it SageMakerEndpointName as suggested earlier.
| @@ -0,0 +1,112 @@ | |||
| AWSTemplateFormatVersion: "2010-09-09" | |||
| Description: QnABot on AWS LLM Plugin for Llama-2-13b-chat | |||
There was a problem hiding this comment.
deployed using SageMaker JumpStart (add link)
… construct the ARN using the SageMakerEndpointName parameter
aaron-sim
force-pushed
the
llama2_plugin
branch
from
cc70a8e
to
d5f0ee8
Compare
Hi,
I have created a plugin which uses Llama-2-13b-chat model (requires the model to be deployed via SageMaker Jumpstart)
Added an additional folder for llama-2-13b-chat-llm plugin and modified the README.md to include the information related to Llama.