Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compress/snappy: configurable compression level #1014

Merged
merged 1 commit into from
Oct 26, 2022

Conversation

chriso
Copy link
Contributor

@chriso chriso commented Oct 22, 2022

There are three snappy compression routines provided by https://github.com/klauspost/compress. See https://github.com/klauspost/compress/tree/master/s2#readme for more information.

At the moment kafka-go is hard-coded to use snappy.Encode aka. s2.EncodeSnappyBetter.

This PR makes the compression level configurable, allowing you to choose either a faster compression routine (s2.EncodeSnappy) or a slower one that compresses better (s2.EncodeSnappyBest).

There appears to be a global set of codecs shared by all producers. To set the compression level for all producers (configured to use snappy) you'd need to do something like this:

compress.SnappyCodec.Compression = snappy.FasterCompression

This is the same approach taken by the gzip codec and zstd codec, although in those cases they accept an integer level. Ideally we'd be able to set the compression level per producer, but it looks like a much larger refactor would be required.

Copy link
Contributor

@achille-roussel achille-roussel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice addition, thanks for the contribution!

@chriso chriso merged commit c1240f0 into main Oct 26, 2022
@chriso chriso deleted the snappy-configurable-level branch October 26, 2022 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants