Added new techniques #810
Added new techniques #810
Conversation
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to dc377ad in 30 seconds
More details
- Looked at
539lines of code in4files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:253
- Draft comment:
The new markdown filesself_calibration.md,self_refine.md, andself_verification.mdare not added to the navigation inmkdocs.yml. Please update the navigation section to include these new documents to ensure they are accessible in the generated site navigation. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_ZoYkdu2hx4MdvAda
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Deploying instructor with
|
| Latest commit: |
07497d3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://99ccf1ef.instructor.pages.dev |
| Branch Preview URL: | https://self-verification.instructor.pages.dev |
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 72e8ec6 in 45 seconds
More details
- Looked at
290lines of code in2files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:255
- Draft comment:
The navigation titles for the newly added techniques inmkdocs.ymlshould match the document names for clarity. Consider updating the titles to match the document names exactly as they appear in the PR description.
- Self-Verification: 'prompting/self_criticism/self_verification.md'
- Self-Calibration: 'prompting/self_criticism/self_calibration.md'
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_2GpsgPIpoVkIyTg9
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on c6d7465 in 38 seconds
More details
- Looked at
252lines of code in2files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. docs/prompting/self_criticism/self_verification.md:2
- Draft comment:
The description contains a grammatical error. Please correct "it's" to "its" to use the correct possessive form:
description: "Self Verification involves getting language models to generate a candidate response before evaluating each individual intermediate reasoning step to verify if its logical entailment holds"
- Reason this comment was not posted:
Confidence of 80% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_HpBzdpeBnAMkPtQU
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 1ae0daa in 34 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:254
- Draft comment:
The PR description indicates that new documents were added (Self-VerificationandSelf-Calibration), but these are not included in the navigation structure inmkdocs.yml. Please update the navigation to include these new documents to ensure they are accessible to users. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_TaStpnGb5vwToWrI
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 3ad7c73 in 35 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:254
- Draft comment:
The PR description indicates the addition ofSelf-VerificationandSelf-Calibrationtechniques, but themkdocs.ymldoes not reflect these additions in the navigation structure. Please update the navigation to include these new documents to ensure they are accessible.
- Self-Verification: 'prompting/self_criticism/self_verification.md'
- Self-Calibration: 'prompting/self_criticism/self_calibration.md'
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_FBsYHRxwkkHLnIqh
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on d7cad25 in 59 seconds
More details
- Looked at
160lines of code in1files - Skipped
0files when reviewing. - Skipped posting
2drafted comments based on config settings.
1. docs/prompting/decomposition/decomp.md:81
- Draft comment:
Consider making the model identifier configurable instead of hardcoding it togpt-4o. This can be done by passing it as a parameter or setting it through an environment variable to enhance flexibility and maintainability.
model=configurable_model_identifier,
- Reason this comment was not posted:
Confidence of 30% on close inspection, compared to threshold of 85%.
2. docs/prompting/decomposition/decomp.md:69
- Draft comment:
Consider adding error handling around the API call inderive_action_planto manage exceptions gracefully. This could involve using try-except blocks to catch and handle specific exceptions that might occur during the API request.
try:
return client.chat.completions.create(
messages=[
{
"role": "system",
"content": "Generate an action plan to help you complete
the task outlined by the user",
},
{"role": "user", "content": task_description},
],
response_model=ActionPlan,
max_retries=3,
model="gpt-4o",
)
except Exception as e:
print(f"An error occurred: {e}")
return None
- Reason this comment was not posted:
Confidence of 30% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_bsjh57rWMOoDkTtQ
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 3205b28 in 52 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. docs/prompting/decomposition/decomp.md:15
- Draft comment:
The PR description mentions updates to several documents andmkdocs.yml, but the diff provided only shows a minor change in this document. Please ensure that all intended changes are included in the PR. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_GI4vp2TZ17DvYm4F
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 200bae4 in 43 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:256
- Draft comment:
The PR description indicates that new documentsself_verification.mdandself_calibration.mdwere added, but they are not included in themkdocs.ymlnavigation structure. Please update the navigation to include these new documents to ensure they are accessible from the website.
- Self-Verification: 'prompting/self_criticism/self_verification.md'
- Self-Calibration: 'prompting/self_criticism/self_calibration.md'
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_bC7y9HLDcSqNotei
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 11c7a23 in 25 seconds
More details
- Looked at
121lines of code in5files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_4JHkP6NcqyQWDBpy
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
| Review A: {review_a} | ||
| Review B: {review_b}. | ||
|
|
||
| Pick your answer from ['Review A', 'Review B', 'both', |
There was a problem hiding this comment.
should this part go into the docstring of PairwiseEvaluation
There was a problem hiding this comment.
Hmm I think it would be better in the prompt personally so that it's a bit clearer. Seems a bit abrupt to have a stand alone pick your answer in the docstring? Or are you referring to something like
class PairwiseEvaluation(BaseModel):
"""
This is a class that represents a comparison of two reviews. Make sure to identify
which one is more aligned with the target sentiment. Pick your answer from
the given alignment_results - Rview A, Review B or both
""""
feedback: str
alignment_result: Literal[
"Review A",
"Review B",
"Both",
]
ivanleomk
force-pushed
the
self-verification
branch
from
b6a148a
to
1074e0a
Compare
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 1074e0a in 1 minute and 3 seconds
More details
- Looked at
328lines of code in3files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_84DoPjq34RJ5okcT
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on b6a148a in 1 minute and 31 seconds
More details
- Looked at
328lines of code in3files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_12vuXnUH0wq9cf1I
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on b4cef66 in 1 minute and 4 seconds
More details
- Looked at
331lines of code in2files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. docs/prompting/self_criticism/self_refine.md:1
- Draft comment:
The content of this file has been replaced with placeholders and marked as[wip], which contradicts the PR description stating that new techniques and documentation updates were added. Please ensure the content is complete before merging.
This comment applies to self_verification.md as well.
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 85%.
Workflow ID: wflow_aOMZ5GykQE7CekxT
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Added more prompting techniques
Added some new techniques for validating and verifying LLM outputs
Summary:
Added new techniques for validating and verifying LLM outputs, including Self-Verification, Self-Calibration, Cumulative Reasoning, Reverse Chain of Thought, and Decomposition, with updated documentation and navigation.
Key points:
Self-Verificationtechnique indocs/prompting/self_criticism/self_verification.mdSelf-Calibrationtechnique indocs/prompting/self_criticism/self_calibration.mdCumulative Reasoningtechnique indocs/prompting/self_criticism/cumulative_reason.mdReverse Chain of Thoughttechnique indocs/prompting/self_criticism/reversecot.mddocs/prompting/self_criticism/self_refine.mdwith implementation detailsmkdocs.ymlto include new techniques in the navigation structureGenerated with ❤️ by ellipsis.dev