docs: example on Vision API (image to ad copy) #293
Conversation
WalkthroughThe updates involve enhancing a project with functionalities for processing images and generating advertising copy using AI. A Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on X ? TipsChat with CodeRabbit Bot (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 2
Configuration used: CodeRabbit UI
Files selected for processing (3)
- .gitignore (2 hunks)
- docs/examples/image_to_ad_copy.md (1 hunks)
- examples/vision/image_to_ad_copy.py (1 hunks)
Files skipped from review due to trivial changes (1)
- .gitignore
Additional comments: 19
docs/examples/image_to_ad_copy.md (9)
1-3: The introduction clearly explains the purpose of the document and its relevance to the target audience.
5-5: Ensure the GitHub link points to the correct repository and directory where the full code is available.
9-11: The
Productclass is well-documented, and the description provides a clear understanding of its purpose.25-35: The
Productclass fields are well-defined with descriptions and examples. However, ensure that thekey_featuresfield, which is optional, is handled correctly in the code where it's used, as it can beNone.39-48: The
generate_promptmethod is a good example of how to create a string representation of theProductobject for use in prompts. It's clear and concise.53-53: The
IdentifiedProductclass includes an error flag and message, which is a good practice for error handling in data processing.79-79: The
AdCopyclass is well-documented, and the fields are descriptive, providing clear guidance on what each field represents.107-107: The
read_imagesfunction description is clear, but ensure that theinstructorlibrary and theclient_imageobject are properly documented elsewhere, as they are critical to understanding how the API is called.146-146: Clarify whether the two different clients (
client_imageandclient_copy) are necessary and if they could be combined for simplicity, or if their separation is due to API constraints.examples/vision/image_to_ad_copy.py (10)
1-11: The imports and setup of the environment variables are correct. Ensure that the
.envfile is properly secured and not checked into version control.15-18: The logger is set up correctly with an appropriate log level. This is good for monitoring the application's behavior.
21-55: The
Productclass is defined correctly, mirroring the documentation. Ensure that thekey_featuresfield is handled properly in the code, as it can beNone.58-79: The
IdentifiedProductclass includes an error flag and message, which is good for error handling. The__bool__method is a nice touch for checking if products were identified.82-106: The
AdCopyclass is well-defined with examples for each field. Ensure that thenamefield is used consistently with theProductclass.109-115: The instantiation of the API clients is done using environment variables, which is secure. However, ensure that the
instructorlibrary is properly documented and that its usage is secure and efficient.118-146: The
read_imagesfunction correctly uses the Vision API client. Ensure that themodelspecified is the correct one for the task and that theresponse_modelis properly implemented to handle the API response.149-167: The
generate_ad_copyfunction is well-documented and uses the Chat API client. Verify that themodelandtemperaturesettings are appropriate for the desired output.170-190: The
runfunction orchestrates the process correctly. Ensure that error handling is robust and that the function gracefully handles cases where no products are identified or ad copy generation fails.193-229: The main execution block is set up correctly with error handling for command-line arguments. Ensure that the results are written to a secure location and that sensitive data is not exposed.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 4
Configuration used: CodeRabbit UI
Files selected for processing (1)
- docs/examples/image_to_ad_copy.md (1 hunks)
Additional comments: 4
docs/examples/image_to_ad_copy.md (4)
56-74: The
IdentifiedProductclass is well-structured. It's good to see error handling incorporated with theerrorflag andmessage. Ensure that the error handling is consistent and integrated throughout the codebase where this class is used.110-137: The
read_imagesfunction is clear and seems to align with the PR's objective of using the Vision API. However, ensure that theclient_imageis properly initialized and authenticated before this function is called.149-167: The
generate_ad_copyfunction is well-documented and uses a separate client for the Chat API, which is appropriate given the different response formats. Ensure that theclient_copyis also properly initialized and authenticated.170-172: The section on putting it all together is mentioned but lacks a code example or detailed explanation. Providing a complete example or walkthrough, as previously suggested, would be beneficial for users to understand the end-to-end process.
docs/examples/image_to_ad_copy.md
Outdated
| ```python | ||
| class Product(BaseModel): | ||
| """ | ||
| Represents a product extracted from an image using AI. | ||
|
|
||
| The product attributes are dynamically determined based on the content | ||
| of the image and the AI's interpretation. This class serves as a structured | ||
| representation of the identified product characteristics. | ||
| """ | ||
|
|
||
| name: str = Field( | ||
| description="A generic name for the product.", example="Headphones" | ||
| ) | ||
| key_features: Optional[list[str]] = Field( | ||
| description="A list of key features of the product that stand out.", | ||
| default=None, | ||
| ) | ||
|
|
||
| description: Optional[str] = Field( | ||
| description="A description of the product.", | ||
| default=None, | ||
| ) | ||
|
|
||
| # Can be customized and automatically generated | ||
| def generate_prompt(self): | ||
| prompt = f"Product: {self.name}\n" | ||
| if self.description: | ||
| prompt += f"Description: {self.description}\n" | ||
| if self.key_features: | ||
| prompt += f"Key Features: {', '.join(self.key_features)}\n" | ||
| return prompt | ||
|
|
||
| def __repr__(self): | ||
| return self.generate_prompt() |
There was a problem hiding this comment.
The Product class is well-documented and includes examples for fields. Consider adding type hints for key_features to specify the type of elements in the list, enhancing type safety and clarity.
- key_features: Optional[list[str]] = Field(
+ key_features: Optional[List[str]] = Field(Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| ```python | |
| class Product(BaseModel): | |
| """ | |
| Represents a product extracted from an image using AI. | |
| The product attributes are dynamically determined based on the content | |
| of the image and the AI's interpretation. This class serves as a structured | |
| representation of the identified product characteristics. | |
| """ | |
| name: str = Field( | |
| description="A generic name for the product.", example="Headphones" | |
| ) | |
| key_features: Optional[list[str]] = Field( | |
| description="A list of key features of the product that stand out.", | |
| default=None, | |
| ) | |
| description: Optional[str] = Field( | |
| description="A description of the product.", | |
| default=None, | |
| ) | |
| # Can be customized and automatically generated | |
| def generate_prompt(self): | |
| prompt = f"Product: {self.name}\n" | |
| if self.description: | |
| prompt += f"Description: {self.description}\n" | |
| if self.key_features: | |
| prompt += f"Key Features: {', '.join(self.key_features)}\n" | |
| return prompt | |
| def __repr__(self): | |
| return self.generate_prompt() | |
| class Product(BaseModel): | |
| """ | |
| Represents a product extracted from an image using AI. | |
| The product attributes are dynamically determined based on the content | |
| of the image and the AI's interpretation. This class serves as a structured | |
| representation of the identified product characteristics. | |
| """ | |
| name: str = Field( | |
| description="A generic name for the product.", example="Headphones" | |
| ) | |
| key_features: Optional[List[str]] = Field( | |
| description="A list of key features of the product that stand out.", | |
| default=None, | |
| ) | |
| description: Optional[str] = Field( | |
| description="A description of the product.", | |
| default=None, | |
| ) | |
| # Can be customized and automatically generated | |
| def generate_prompt(self): | |
| prompt = f"Product: {self.name}\n" | |
| if self.description: | |
| prompt += f"Description: {self.description}\n" | |
| if self.key_features: | |
| prompt += f"Key Features: {', '.join(self.key_features)}\n" | |
| return prompt | |
| def __repr__(self): | |
| return self.generate_prompt() |
docs/examples/image_to_ad_copy.md
Outdated
| class AdCopy(BaseModel): | ||
| """ | ||
| Represents a generated ad copy. | ||
| """ | ||
|
|
||
| headline: str = Field( | ||
| description="A short, catchy, and memorable headline for the given product. The headline should invoke curiosity and interest in the product.", | ||
| ) | ||
| ad_copy: str = Field( | ||
| description="A long-form advertisement copy for the given product. This will be used in campaigns to promote the product with a persuasive message and a call-to-action with the objective of driving sales.", | ||
| ) | ||
| name: str = Field( | ||
| description="The name of the product being advertised.", ) | ||
|
|
||
| def __str__(self): | ||
| return f"{self.name}: \n" + "-" * 100 + f"{self.headline}\n{self.ad_copy}" | ||
|
|
||
| def __repr__(self): | ||
| return str(self) |
There was a problem hiding this comment.
The AdCopy class includes a name field, which might be redundant if it's meant to represent the same product name as in the Product class. If the intention is to reference the product, consider using a reference or a foreign key instead of duplicating the name.
lakshyaag
changed the title
|
Can you include the example output the same way I'd did in the vision post? |
docs/examples/image_to_ad_copy.md
Outdated
| name: str = Field( | ||
| description="The name of the product being advertised.", ) | ||
|
|
||
| def __str__(self): |
There was a problem hiding this comment.
no need for str
usually model_dumps() is enough
|
|
||
|
|
||
| # Define clients | ||
| client_image = instructor.patch( |
There was a problem hiding this comment.
i see, we shuold work on being able to overload the default mode in the create call. in the future
| logger.info("Writing results to file...") | ||
|
|
||
| with open("results.json", "w") as f: | ||
| json.dump( |
There was a problem hiding this comment.
does a single products.model_dump() not work?
Co-authored-by: Jason Liu <jxnl@users.noreply.github.com>
There was a problem hiding this comment.
Review Status
Actionable comments generated: 2
Configuration used: CodeRabbit UI
Files selected for processing (1)
- docs/examples/image_to_ad_copy.md (1 hunks)
Additional comments: 5
docs/examples/image_to_ad_copy.md (5)
56-75: The
IdentifiedProductclass is well-structured to hold identified products and error information. It's good practice to include error handling within data models for robust processing.82-101:
110-137: The
read_imagesfunction is correctly documented and uses the OpenAI vision model to identify products in images. The use of theinstructorlibrary to patch the OpenAI client is a good approach for extending functionality.149-167: The
generate_ad_copyfunction is well-documented and correctly uses a separate OpenAI client to generate advertising copy, acknowledging the incompatibility between different models.174-181:
docs/examples/image_to_ad_copy.md
Outdated
| def __repr__(self): | ||
| return self.generate_prompt() |
There was a problem hiding this comment.
The __repr__ method in the Product class is unnecessary if it only calls another method (generate_prompt) that is not intended for debugging purposes. Consider removing it to follow Python's convention of using __repr__ for unambiguous object representations.
- def __repr__(self):
- return self.generate_prompt()Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def __repr__(self): | |
| return self.generate_prompt() |
There was a problem hiding this comment.
Review Status
Actionable comments generated: 6
Configuration used: CodeRabbit UI
Files ignored due to filter (1)
- mkdocs.yml
Files selected for processing (2)
- docs/examples/image_to_ad_copy.md (1 hunks)
- examples/vision/image_to_ad_copy.py (1 hunks)
Additional comments: 12
docs/examples/image_to_ad_copy.md (6)
1-3: The introduction provides a clear overview of the purpose of the document.
7-45: The
Productclass is well-documented with a clear description and examples for each field.48-71: The
IdentifiedProductclass includes an error flag and message, which is good for error handling.76-92: The
AdCopyclass is well-documented and includes examples, which will be helpful for users.166-168: The input file section is brief and has been previously commented on for improvement. Since the previous comment is still valid, skip generating a similar comment.
175-227: The example output is a good addition, providing a clear idea of what the user can expect from using the code.
examples/vision/image_to_ad_copy.py (6)
21-52: The
Productclass is well-defined with clear documentation and examples.55-76: The
IdentifiedProductclass includes an error flag and message, which is good for error handling. The__bool__method is a nice touch for easy truthiness checks.79-97: The
AdCopyclass is well-documented with examples, which will be helpful for users.100-106: The global variables
client_imageandclient_copyare introduced for managing API client instances. Ensure that these instances are used securely and that the API key is not exposed.Verification successful
The verification process did not find any instances of the
OPENAI_API_KEYbeing exposed through print or log statements in the Python files of the codebase. The use of environment variables to handle the API key is secure, and the absence of the key in any print or log statements suggests that it is not being inadvertently exposed.* 184-222: The main execution block is well-structured and includes error handling. Ensure that the `results.json` file is written to a secure location and that sensitive data is not exposed.Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that the API key is not exposed in the codebase rg --type py "OPENAI_API_KEY"Length of output: 609
Script:
#!/bin/bash # Check for logging or print statements that might inadvertently expose the API key rg --type py -e "print.*OPENAI_API_KEY" -e "log.*OPENAI_API_KEY"Length of output: 66
- 223-271: The example output provided in the docstring is a good addition, providing a clear idea of what the user can expect from using the code.
| The `read_images` function uses OpenAI's vision model to process a list of image URLs and identify products in each of them. We utilize the `instructor` library to patch the OpenAI client for this purpose. | ||
|
|
||
| ```python | ||
| def read_images(image_urls: List[str]) -> IdentifiedProduct: | ||
| """ | ||
| Given a list of image URLs, identify the products in the images. | ||
| """ | ||
|
|
||
| logger.info(f"Identifying products in images... {len(image_urls)} images") | ||
|
|
||
| return client_image.chat.completions.create( | ||
| model="gpt-4-vision-preview", | ||
| response_model=IdentifiedProduct, | ||
| max_tokens=1024, # can be changed | ||
| temperature=0, | ||
| messages=[ | ||
| { | ||
| "role": "user", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "Identify products using the given images and generate key features for each product.", | ||
| }, | ||
| *[ | ||
| {"type": "image_url", "image_url": {"url": url}} | ||
| for url in image_urls | ||
| ], | ||
| ], | ||
| } | ||
| ], | ||
| ) |
There was a problem hiding this comment.
The read_images function documentation should specify the expected return type for clarity.
- def read_images(image_urls: List[str]) -> IdentifiedProduct:
+ def read_images(image_urls: List[str]) -> List[IdentifiedProduct]:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| The `read_images` function uses OpenAI's vision model to process a list of image URLs and identify products in each of them. We utilize the `instructor` library to patch the OpenAI client for this purpose. | |
| ```python | |
| def read_images(image_urls: List[str]) -> IdentifiedProduct: | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, # can be changed | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) | |
| def read_images(image_urls: List[str]) -> List[IdentifiedProduct]: | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, # can be changed | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) |
docs/examples/image_to_ad_copy.md
Outdated
| Two clients are defined for the two different models. This is because the `gpt-4-vision-preview` model is not compatible with the `gpt-4-1106-preview` model in terms of their response format. | ||
|
|
||
| ```python | ||
| def generate_ad_copy(product: Product): | ||
| """ | ||
| Given a product, generate an ad copy for the product. | ||
| """ | ||
|
|
||
| logger.info(f"Generating ad copy for product: {product.name}") | ||
|
|
||
| return client_copy.chat.completions.create( | ||
| model="gpt-4-1106-preview", | ||
| response_model=AdCopy, | ||
| temperature=0.3, | ||
| messages=[ | ||
| { | ||
| "role": "system", | ||
| "content": "You are an expert marketing assistant for all products. Your task is to generate an advertisement copy for a product using the name, description, and key features.", | ||
| }, | ||
| {"role": "user", "content": product.generate_prompt()}, | ||
| ], |
There was a problem hiding this comment.
The generate_ad_copy function is well-documented, but it would be beneficial to include the expected return type in the documentation.
- def generate_ad_copy(product: Product):
+ def generate_ad_copy(product: Product) -> AdCopy:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| Two clients are defined for the two different models. This is because the `gpt-4-vision-preview` model is not compatible with the `gpt-4-1106-preview` model in terms of their response format. | |
| ```python | |
| def generate_ad_copy(product: Product): | |
| """ | |
| Given a product, generate an ad copy for the product. | |
| """ | |
| logger.info(f"Generating ad copy for product: {product.name}") | |
| return client_copy.chat.completions.create( | |
| model="gpt-4-1106-preview", | |
| response_model=AdCopy, | |
| temperature=0.3, | |
| messages=[ | |
| { | |
| "role": "system", | |
| "content": "You are an expert marketing assistant for all products. Your task is to generate an advertisement copy for a product using the name, description, and key features.", | |
| }, | |
| {"role": "user", "content": product.generate_prompt()}, | |
| ], | |
| def generate_ad_copy(product: Product) -> AdCopy: | |
| """ | |
| Given a product, generate an ad copy for the product. | |
| """ | |
| logger.info(f"Generating ad copy for product: {product.name}") | |
| return client_copy.chat.completions.create( | |
| model="gpt-4-1106-preview", | |
| response_model=AdCopy, | |
| temperature=0.3, | |
| messages=[ | |
| { | |
| "role": "system", | |
| "content": "You are an expert marketing assistant for all products. Your task is to generate an advertisement copy for a product using the name, description, and key features.", | |
| }, | |
| {"role": "user", "content": product.generate_prompt()}, | |
| ], |
|
|
||
| ### Putting it all together | ||
|
|
||
| Finally, we can put it all together in a single function that takes a list of image URLs and generates advertising copy for the products identified in the images. Please refer to the [full code](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py) for the complete implementation. |
There was a problem hiding this comment.
The summary of the run function is clear, but it would be helpful to include the return type in the documentation for consistency.
- def run(images: list[str]):
+ def run(images: list[str]) -> Tuple[List[Product], List[AdCopy]]:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| Finally, we can put it all together in a single function that takes a list of image URLs and generates advertising copy for the products identified in the images. Please refer to the [full code](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py) for the complete implementation. | |
| Finally, we can put it all together in a single function that takes a list of image URLs and generates advertising copy for the products identified in the images. Please refer to the [full code](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py) for the complete implementation. |
examples/vision/image_to_ad_copy.py
Outdated
| def read_images(image_urls: List[str]): | ||
| """ | ||
| Given a list of image URLs, identify the products in the images. | ||
| """ | ||
|
|
||
| logger.info(f"Identifying products in images... {len(image_urls)} images") | ||
|
|
||
| return client_image.chat.completions.create( | ||
| model="gpt-4-vision-preview", | ||
| response_model=IdentifiedProduct, | ||
| max_tokens=1024, | ||
| temperature=0, | ||
| messages=[ | ||
| { | ||
| "role": "user", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "Identify products using the given images and generate key features for each product.", | ||
| }, | ||
| *[ | ||
| {"type": "image_url", "image_url": {"url": url}} | ||
| for url in image_urls | ||
| ], | ||
| ], | ||
| } | ||
| ], | ||
| ) |
There was a problem hiding this comment.
The read_images function should specify the expected return type for clarity.
- def read_images(image_urls: List[str]):
+ def read_images(image_urls: List[str]) -> IdentifiedProduct:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def read_images(image_urls: List[str]): | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) | |
| def read_images(image_urls: List[str]) -> IdentifiedProduct: | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) |
examples/vision/image_to_ad_copy.py
Outdated
| def run(images: list[str]): | ||
| """ | ||
| Given a list of images, identify the products in the images and generate ad copy for each product. | ||
| """ | ||
|
|
||
| identified_products: IdentifiedProduct = read_images(images) | ||
| ad_copies = [] | ||
|
|
||
| if identified_products.error: | ||
| rprint(f"[red]Error: {identified_products.message}[/red]") | ||
| return [] | ||
|
|
||
| if not identified_products: | ||
| rprint("[yellow]No products identified.[/yellow]") | ||
| return [] | ||
|
|
||
| for product in identified_products.products: | ||
| ad_copy: AdCopy = generate_ad_copy(product) | ||
| ad_copies.append(ad_copy) | ||
|
|
||
| return identified_products.products, ad_copies |
There was a problem hiding this comment.
The run function is well-documented, but it would be helpful to include the return type in the documentation for consistency.
- def run(images: list[str]):
+ def run(images: list[str]) -> Tuple[List[Product], List[AdCopy]]:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def run(images: list[str]): | |
| """ | |
| Given a list of images, identify the products in the images and generate ad copy for each product. | |
| """ | |
| identified_products: IdentifiedProduct = read_images(images) | |
| ad_copies = [] | |
| if identified_products.error: | |
| rprint(f"[red]Error: {identified_products.message}[/red]") | |
| return [] | |
| if not identified_products: | |
| rprint("[yellow]No products identified.[/yellow]") | |
| return [] | |
| for product in identified_products.products: | |
| ad_copy: AdCopy = generate_ad_copy(product) | |
| ad_copies.append(ad_copy) | |
| return identified_products.products, ad_copies | |
| def run(images: list[str]) -> Tuple[List[Product], List[AdCopy]]: | |
| """ | |
| Given a list of images, identify the products in the images and generate ad copy for each product. | |
| """ | |
| identified_products: IdentifiedProduct = read_images(images) | |
| ad_copies = [] | |
| if identified_products.error: | |
| rprint(f"[red]Error: {identified_products.message}[/red]") | |
| return [] | |
| if not identified_products: | |
| rprint("[yellow]No products identified.[/yellow]") | |
| return [] | |
| for product in identified_products.products: | |
| ad_copy: AdCopy = generate_ad_copy(product) | |
| ad_copies.append(ad_copy) | |
| return identified_products.products, ad_copies |
There was a problem hiding this comment.
Review Status
Actionable comments generated: 2
Configuration used: CodeRabbit UI
Files selected for processing (2)
- docs/examples/image_to_ad_copy.md (1 hunks)
- examples/vision/image_to_ad_copy.py (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- docs/examples/image_to_ad_copy.md
Additional comments: 8
examples/vision/image_to_ad_copy.py (8)
22-53: The
Productclass is well-defined with appropriate type annotations and default values. Thegenerate_promptmethod concatenates product information into a prompt string. Ensure that the method is used in the context where a prompt is required.56-77: The
IdentifiedProductclass includes anerrorfield and amessagefield for error handling. The__bool__method is implemented to quickly check if any products are identified. Verify that the error handling mechanism is consistent with the rest of the codebase.80-98: The
AdCopyclass is defined with fields for the headline, ad copy, and product name. The descriptions and examples provided in the Field declarations are helpful for understanding the intended use of each field.110-138: The
read_imagesfunction uses theclient_imageAPI client to identify products in images. Ensure that themodelandresponse_modelparameters are correctly configured and that the function properly handles the response from the API.141-159: The
generate_ad_copyfunction uses theclient_copyAPI client to generate ad copy for a product. The function should handle the response from the API and return anAdCopyinstance. Verify that themodelparameter is correctly set and that the temperature is appropriate for the desired creativity in the ad copy.162-182: The
runfunction orchestrates the process of reading images and generating ad copy. It includes error handling and uses theIdentifiedProductandAdCopyclasses. Ensure that the error messages are clear and that the function returns the expected tuple of products and ad copies.185-222: The main execution block includes command-line argument handling, file operations, and function calls to process images and generate ad copy. Ensure that the file operations are robust and that exceptions are handled appropriately. Also, verify that the
results.jsonfile is written to the correct location and contains the expected output format.223-273: The example output JSON at the end of the file provides a clear example of the expected output structure. This is helpful for understanding the output format and for testing purposes. Ensure that the actual output matches this structure.
| import json | ||
| import logging | ||
| import os | ||
| import sys | ||
| from typing import List, Optional, Tuple | ||
|
|
||
| from dotenv import find_dotenv, load_dotenv | ||
| from openai import OpenAI | ||
| from pydantic import BaseModel, Field | ||
| from rich import print as rprint | ||
|
|
||
| import instructor | ||
|
|
||
| load_dotenv(find_dotenv()) | ||
|
|
||
| # Add logger | ||
| logging.basicConfig() | ||
| logger = logging.getLogger("app") | ||
| logger.setLevel("INFO") |
There was a problem hiding this comment.
Consider grouping standard library imports (json, logging, os, sys) and separating them from third-party library imports (dotenv, openai, pydantic, rich) with a newline for better readability.
| # Define clients | ||
| client_image = instructor.patch( | ||
| OpenAI(api_key=os.getenv("OPENAI_API_KEY")), mode=instructor.Mode.MD_JSON | ||
| ) | ||
| client_copy = instructor.patch( | ||
| OpenAI(api_key=os.getenv("OPENAI_API_KEY")), mode=instructor.Mode.FUNCTIONS | ||
| ) |
There was a problem hiding this comment.
Global variables client_image and client_copy are initialized with API clients. It's generally not recommended to use global variables for state management. Consider encapsulating these within a class or a function to avoid potential side effects and improve testability.
|
the images don't look like i see a desert and water, not the products |
Oh yes, I didn't upload the actual image links because I thought there would be some sort of licensing restrictions. EDIT: I've updated the image links in the example. It should correspond with the results now. |
There was a problem hiding this comment.
Review Status
Actionable comments generated: 4
Configuration used: CodeRabbit UI
Files selected for processing (1)
- docs/examples/image_to_ad_copy.md (1 hunks)
Additional comments: 5
docs/examples/image_to_ad_copy.md (5)
1-3: The introduction provides a clear overview of the document's purpose.
16-45: The
Productclass is well-defined with clear documentation and a method to generate a prompt for the Chat API.79-92: The
AdCopyclass is clearly documented and structured, providing a good format for the generated advertising copy.141-159: The
generate_ad_copyfunction is well-documented and seems to correctly utilize the Chat API to generate advertising copy.177-229: The output section provides a clear example of what the user can expect when running the code, which is helpful for understanding the functionality.
|
|
||
| This post demonstrates how to use GPT-4 Vision API and the Chat API to automatically generate advertising copy from product images. This method can be useful for marketing and advertising teams, as well as for e-commerce platforms. | ||
|
|
||
| The full code is available on [GitHub](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py). |
There was a problem hiding this comment.
The GitHub link is still incorrect and should be updated to point to the official repository path.
- [GitHub](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py)
+ [GitHub](https://github.com/<official-repo>/examples/vision/image_to_ad_copy.py)Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| The full code is available on [GitHub](https://github.com/jxnl/instructor/examples/vision/image_to_ad_copy.py). | |
| The full code is available on [GitHub](https://github.com/<official-repo>/examples/vision/image_to_ad_copy.py). |
| class IdentifiedProduct(BaseModel): | ||
| """ | ||
| Represents a list of products identified in the images. | ||
| """ | ||
|
|
||
| products: Optional[List[Product]] = Field( | ||
| description="A list of products identified by the AI.", | ||
| example=[ | ||
| Product( | ||
| name="Headphones", | ||
| description="Wireless headphones with noise cancellation.", | ||
| key_features=["Wireless", "Noise Cancellation"], | ||
| ) | ||
| ], | ||
| default=None, | ||
| ) | ||
|
|
||
| error: bool = Field(default=False) | ||
| message: Optional[str] = Field(default=None) |
There was a problem hiding this comment.
The IdentifiedProduct class is well-structured, but it's not clear why it has an error flag and message. Ensure that the error handling is consistent with the rest of the application.
| def read_images(image_urls: List[str]) -> IdentifiedProduct: | ||
| """ | ||
| Given a list of image URLs, identify the products in the images. | ||
| """ | ||
|
|
||
| logger.info(f"Identifying products in images... {len(image_urls)} images") | ||
|
|
||
| return client_image.chat.completions.create( | ||
| model="gpt-4-vision-preview", | ||
| response_model=IdentifiedProduct, | ||
| max_tokens=1024, # can be changed | ||
| temperature=0, | ||
| messages=[ | ||
| { | ||
| "role": "user", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "Identify products using the given images and generate key features for each product.", | ||
| }, | ||
| *[ | ||
| {"type": "image_url", "image_url": {"url": url}} | ||
| for url in image_urls | ||
| ], | ||
| ], | ||
| } | ||
| ], | ||
| ) |
There was a problem hiding this comment.
The return type of the read_images function should be a list of IdentifiedProduct objects, as one function call may process multiple images.
- def read_images(image_urls: List[str]) -> IdentifiedProduct:
+ def read_images(image_urls: List[str]) -> List[IdentifiedProduct]:Committable suggestion
❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def read_images(image_urls: List[str]) -> IdentifiedProduct: | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, # can be changed | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) | |
| def read_images(image_urls: List[str]) -> List[IdentifiedProduct]: | |
| """ | |
| Given a list of image URLs, identify the products in the images. | |
| """ | |
| logger.info(f"Identifying products in images... {len(image_urls)} images") | |
| return client_image.chat.completions.create( | |
| model="gpt-4-vision-preview", | |
| response_model=IdentifiedProduct, | |
| max_tokens=1024, # can be changed | |
| temperature=0, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "Identify products using the given images and generate key features for each product.", | |
| }, | |
| *[ | |
| {"type": "image_url", "image_url": {"url": url}} | |
| for url in image_urls | |
| ], | |
| ], | |
| } | |
| ], | |
| ) |
| ## Input file | ||
|
|
||
| The input file is currently a list of image URLs, but this trivial to change to any required format. |
There was a problem hiding this comment.
The input file section remains brief. Providing a more detailed explanation or an example of the expected format would be beneficial for users.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (1)
- docs/examples/image_to_ad_copy.md (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- docs/examples/image_to_ad_copy.md
This pull request adds the .DS_Store file to the .gitignore file to prevent it from being tracked in the repository. It also includes an example on how to use the GPT-4 Vision API and the Chat API to automatically generate advertising copy from product images.
Summary by CodeRabbit
New Features
Documentation
image_to_ad_copy.mdto showcase usage and capabilities of the new feature.Chores
.gitignoreto exclude macOS system files.