Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RawProduce API #1233

Merged
merged 32 commits into from
Dec 5, 2023
Merged

Add RawProduce API #1233

merged 32 commits into from
Dec 5, 2023

Conversation

apacker
Copy link
Contributor

@apacker apacker commented Nov 16, 2023

This PR adds a new RawProduce API to the client. The background/context for this API is: we have an internal system that does buffering of Kafka messages on disk, and needs to flush that data to Kafka without having to decode/re-encode the data and incur a 2x cost of compression/encoding. In this context, it would be optimal to be able to pass a raw/pre-encoded RecordSet to the Produce API, hence the introduction of a new RawProduce interface.

Notes:

  • The existing ProduceResponse type is reused as there's no reason to introduce a new response type.
  • Since request/response types and their resulting encodings are linked directly to an API key, I've introduced a mechanism to be able to override types via a OverrideTypeMessage interface.

@apacker apacker marked this pull request as ready for review November 16, 2023 17:57
Copy link
Contributor

@petedannemann petedannemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 great work, two small comments for now. I'll think this over a bit more and re-review later this week

protocol/protocol.go Outdated Show resolved Hide resolved
protocol/protocol.go Outdated Show resolved Hide resolved
apacker and others added 2 commits November 30, 2023 14:11
Co-authored-by: Peter Dannemann <28637185+petedannemann@users.noreply.github.com>
@apacker apacker merged commit edf45f6 into main Dec 5, 2023
14 checks passed
@apacker apacker deleted the raw-produce-api branch December 5, 2023 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants