feat(blog): Add new post on llama-cpp-python and instructor library usage
#434
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ellipsis-dev
bot
changed the title
llama-cpp-python and instructor library usage
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good to me! Reviewed entire PR up to commit bf826c3
Reviewed 121 lines of code across 1 files in 1 minute(s) and 3 second(s).
See details
- Skipped files: 0 (please contact us to request support for these files)
- Confidence threshold:
85% - Drafted
0additional comments. - Workflow ID:
wflow_b0ZkUnb5GvB5ybuj
Something look wrong? You can customize Ellipsis by editing the ellipsis.yaml for this repository.
Generated with ❤️ by ellipsis.dev
abetlen
suggested changes
Feb 12, 2024
docs/blog/posts/llama-cpp-python.md
Outdated
|
|
||
| Recently llama-cpp-python has made support structured outputs via JSON schema available. This is a time-saving alternative to extensive prompt engineering and can be used to obtain structured outputs. | ||
|
|
||
| In this example we'll cover a more advanced use case of by using `JSON_SCHEMA` mode to stream out partial models. To learn more partial streaming check out [partial streaming](../../concepts/partial.md). |
There was a problem hiding this comment.
In this example we'll cover a more advanced use case of JSON_SCHEMA mode to stream out partial models. To learn more partial streaming check out partial streaming.
docs/blog/posts/llama-cpp-python.md
Outdated
| console.print(obj) | ||
| ``` | ||
|
|
||
| 1. We use `LlamaPromptLookupDecoding` to obtain structured outputs using JSON schema via a mixture of constrained sampling and speculative decoding. 10 is good for GPU, 2 is good for CPU. |
There was a problem hiding this comment.
We use LlamaPromptLookupDecoding to speed up structured output generation using speculative decoding. The draft model generates candidate tokens during generation 10 is good for GPU, 2 is good for CPU.
jxnl
commented
Feb 12, 2024
jxnl
commented
Feb 12, 2024
jxnl
commented
Feb 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR adds a new blog post discussing the use of
llama-cpp-pythonfor structured outputs and the enhancement ofcreatecalls with theinstructorlibrary, including a Python code example.Key points:
/docs/blog/posts/llama-cpp-python.mdllama-cpp-pythonfor structured outputscreatecalls withinstructorlibraryGenerated with ❤️ by ellipsis.dev