compress/snappy: configurable compression level #1014
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are three snappy compression routines provided by https://github.com/klauspost/compress. See https://github.com/klauspost/compress/tree/master/s2#readme for more information.
At the moment kafka-go is hard-coded to use
snappy.Encode
aka.s2.EncodeSnappyBetter
.This PR makes the compression level configurable, allowing you to choose either a faster compression routine (
s2.EncodeSnappy
) or a slower one that compresses better (s2.EncodeSnappyBest
).There appears to be a global set of codecs shared by all producers. To set the compression level for all producers (configured to use snappy) you'd need to do something like this:
This is the same approach taken by the gzip codec and zstd codec, although in those cases they accept an integer level. Ideally we'd be able to set the compression level per producer, but it looks like a much larger refactor would be required.