CI Failure (partitions_rebalanced times out) in ScalingUpTest.test_on_demand_rebalancing

[VadimPlh]

https://buildkite.com/redpanda/redpanda/builds/19738#01850f7e-257c-4950-be48-96aeee2ae651

TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/scaling_up_test.py", line 195, in test_on_demand_rebalancing
    self.wait_for_partitions_rebalanced(total_replicas=total_replicas,
  File "/root/tests/rptest/tests/scaling_up_test.py", line 70, in wait_for_partitions_rebalanced
    wait_until(partitions_rebalanced,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

[piyushredpanda]

@rystsov : could you please help out here?

[ZeDRoman]

Failure happend because redpana with node_id: 4 couldn't join cluster.
This happened because data on docker-rp-4 was not cleaned before the tests, so node already had data in controller log and it's controller log term was 5 on startup. So it couldn't connect to cluster.

This problem is already fixed by Michal in #7789 by adding node clean up in commit: 1f8c381

[michael-redpanda]

Possibly another occurrence occurred: https://buildkite.com/redpanda/redpanda/builds/25427#0186f66b-05ed-40b6-bffd-e4cffa205526

[VladLazar]

https://buildkite.com/redpanda/redpanda/builds/25457#0186fdb6-f8ee-4bc5-9434-85f0874f32fc

[VladLazar]

https://buildkite.com/redpanda/redpanda/builds/25686#01870dbf-acec-45f4-afca-c75174125096

[Lazin]

https://buildkite.com/redpanda/redpanda/builds/25852#01872264-70c4-4838-9cf9-f59ed9a474f2/6-1999

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/25828#01871c88-1e53-4862-9fe9-aa926e17dd79

Module: rptest.tests.scaling_up_test
Class:  ScalingUpTest
Method: test_on_demand_rebalancing
Arguments:
{
  "partition_count": 20
}

test_id:    rptest.tests.scaling_up_test.ScalingUpTest.test_on_demand_rebalancing.partition_count=20
status:     FAIL
run time:   3 minutes 43.785 seconds

    TimeoutError('')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/scaling_up_test.py", line 211, in test_on_demand_rebalancing
    self.wait_for_partitions_rebalanced(total_replicas=total_replicas,
  File "/root/tests/rptest/tests/scaling_up_test.py", line 84, in wait_for_partitions_rebalanced
    wait_until(partitions_rebalanced,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

[rystsov]

another instance: https://buildkite.com/redpanda/redpanda/builds/25846#018721b0-ed95-4a92-828a-4abd40993ba1

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/25916#018727e7-b82c-4de8-9400-e8c0034d6dfa

[VladLazar]

https://buildkite.com/redpanda/redpanda/builds/26082#0187323c-240a-4606-ad3e-d21643e9709a

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/26094#018732e3-c45a-485e-82f4-950bef0e027c

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/26047#01872f89-3c44-4977-80ac-825ced5789f0

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/26130#018734cf-ad03-4e62-85d7-a245858a95a8 (partition count 1
https://buildkite.com/redpanda/redpanda/builds/26146#0187364c-c0e7-4c31-8f55-1d8251fdeef7 (partition count 20)