Reassign topics uniformly after partition is added #9622
Conversation
mmaslankaprv
force-pushed
the
uniform-assignments
branch
4 times, most recently
from
c604a7c
to
969a8be
Compare
mmaslankaprv
changed the title
mmaslankaprv
requested review from
ztlpn,
ZeDRoman,
ballard26,
bharathv and
rystsov
|
@vshtokman : Added this to v23.1.4 (technically speaking the backport PR should be in that milestone rather than this PR thats targeted to dev, but just so we don't lose track that this is needed for v23.1.4). |
| domain, | ||
| target_replicas_per_node, | ||
| diff); | ||
| err += std::abs(diff); |
There was a problem hiding this comment.
side note: squaring the diff here is probably better because if we have, say, node partition counts 0, 2 (with the target count of 3), this is worse than 1, 1 (and the least_allocated constraint will prefer the latter), even though they have the same error in the L1 metric.
Signed-off-by: Michal Maslanka <michal@redpanda.com>
Signed-off-by: Michal Maslanka <michal@redpanda.com>
Introduced a reallation strategy type to make reallocation code testable and customizable outside of the members backend. Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
uniform-assignments
branch
from
969a8be
to
a9b3e0e
Compare
mmaslankaprv
force-pushed
the
uniform-assignments
branch
from
a9b3e0e
to
ba958a9
Compare
|
hmm, |
Added simple test validating correctness of backend reallcation strategy. Signed-off-by: Michal Maslanka <michal@redpanda.com>
Using reservoir sampling to chose the replicas that are going to be reallocated to the new node provides better uniformity of topic replicas distribution across the cluster. Additionally reservoir sampling algorithm isn't prone to not selecting enough reallocations. Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
uniform-assignments
branch
from
ba958a9
to
d2456cc
Compare
|
/backport v23.1.x |
|
Failed to run cherry-pick command. I executed the below command: |
|
/backport v23.1.x |
|
Failed to run cherry-pick command. I executed the below command: |
|
Backport only with fix #9870 |
| }; | ||
|
|
||
| using node_replicas_map_t = absl::node_hash_map<model::node_id, node_replicas>; | ||
| static absl::node_hash_map<model::node_id, node_replicas> |
There was a problem hiding this comment.
since you are inside an anonymous namespace you don't need to mark the function as file scope to prevent it from becoming an exported symbol
|
@mmaslankaprv , I think this still needs a v23.1.x backport. |
|
This is already backported to v23.1.x |
Using reservoir sampling to chose the replicas that are going to be
reallocated to the new node provides better uniformity of topic replicas
distribution across the cluster. Additionally reservoir sampling
algorithm isn't prone to not selecting enough reallocations.
Fixes #7756
Backports Required
Release Notes
Improvements