-
Notifications
You must be signed in to change notification settings - Fork 506
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Merged by Bors] - feat: Add Partitioner trait with SiphashRoundRobinPartitioner impl #965
[Merged by Bors] - feat: Add Partitioner trait with SiphashRoundRobinPartitioner impl #965
Conversation
Can you link this to issue? |
src/client/src/producer.rs
Outdated
.push(DefaultBatch::new(records)); | ||
topic_request.name = self.topic.clone(); | ||
topic_request.partitions.push(partition_request); | ||
request.acks = 1; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should not be hard coded
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is how the producer request was already being built (see the removed diff in send_records_raw
. If we want to change this we should do it in a new issue
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. Let's create issue to allow developers to configure producer and profile. We probably need producer builder or something like that.
0c564dc
to
81a2321
Compare
Ok, newest changes:
trait Partitioner {
fn partition(&mut self, keys: &[Option<&[u8]>]) -> Vec<i32>;
fn update_config(&mut self, config: PartitionerConfig);
}
|
3233bb7
to
3c4a7c7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good progress. See other comments regarding moving keyless api as separate issue
Also, test with replication factor = 2 |
ee6bebc
to
3b2356e
Compare
Can you add this to changelog? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making good progress. Couple of minor changes needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Nice work!
bors r+ |
) Closes #838 - Introduces `Partitioner` trait to enable new implementations in the future - Adds `SiphashRoundRobinPartitioner` as a first implementor of `Partitioner` - Uses Siphash-2-4 on records that have keys - Uses round-robin to assign partitions to records with no keys - Adds a `StoreContext<TopicSpec>` to `MetdataStores` for syncing topic metadata - Implements watch requests on SPU for Topic and SPG metadata
Pull request successfully merged into master. Build succeeded: |
Closes #838
Partitioner
trait to enable new implementations in the futureSiphashRoundRobinPartitioner
as a first implementor ofPartitioner
StoreContext<TopicSpec>
toMetdataStores
for syncing topic metadata