Deprecate InferenceParams sampler configuration properties
#871
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…` as obsolete, pushing users towards the newer `SamplingPipeline` system. - Removed old sampling code from `LLamaContext`, instead if no `SamplingPipeline` is supplied one is created (existing one is re-used, as much as possible). - Updated all examples to use new system, uncovered a bug in `TalkToYourself` which tried to use `Temperature` _and_ mirostate - not compatible! This is exactly the kind of bug this is trying to fix. - Added `AsSpan` to `FixedSizeQueue` to avoid allocations of temporary arrays for every token!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IInferenceParamsas obsolete, pushing to the newerSamplingPipelinesystem.LLamaContext, instead if noSamplingPipelineis supplied one is created (existing one is re-used, as much as possible).AsSpantoFixedSizeQueueto avoid allocations of temporary arrays for every token!The new sampling pipeline system was added a long time ago to address the issues with
IInferenceParams. Configuring everything with one config object doesn't allow for re-ordering of sampling steps, doesn't allow custom samplers, and allows property combinations which are meaningless. For example using Mirostat sampling ignores almost every other property!See the modified example files to see how the new system is used. In most cases it will simply involve passing a
DefaultSamplingPipelineobject.