Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.1.x] Reproduce the OOM on fetching from many partitions #10652

Merged
merged 4 commits into from
May 10, 2023
Merged

[v23.1.x] Reproduce the OOM on fetching from many partitions #10652

merged 4 commits into from
May 10, 2023

Conversation

vbotbuildovich
Copy link
Collaborator

Backport of PR #9905

dlex added 4 commits May 10, 2023 01:36
@dlex @vbotbuildovich
tests/kaf_consumer: support num_records in KafConsumer
b055e8c 
KafConsumer has `num_records` parameter that leads to nowhere. This commit
makes it pass the specified value as `--limit-messages` argument of
`kaf consume`. Change of the default value of `num_records` does not
affect any existing code

(cherry picked from commit fdb0e58)
@dlex @vbotbuildovich
tests/rpk_producer: Add max_message_bytes to RpkProducer
d1b2560 
`max_message_bytes` is passed as the corresponding `rpk topic produce`
argument to control maximum batch size in Produce API.

(cherry picked from commit cf964b3)
@dlex @vbotbuildovich
tests/kafka_cli: Add max.message.bytes topic config
c6265d0 
Add `max.message.bytes` topic config to KCL wrapper's `create_topic`
so that it's possible to specify that parameter while using the regular
RedpandaTest way to create topics

(cherry picked from commit f84688d)
@dlex @vbotbuildovich
tests: MemoryStressTest.test_fetch_with_many_partitions
a49eb85 
Add a test to reproduce the case when redpanda OOMs on reading at least
a single batch from each of many partitions in a fetch request, so that
the memory required to store `(partition leaders in a shard)*(batch size)`
is too much to allocate.

(cherry picked from commit 7f18d94)
@vbotbuildovich vbotbuildovich added this to the v23.1.x-next milestone May 10, 2023
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label May 10, 2023
@dlex dlex mentioned this pull request May 10, 2023
Closed
7 tasks
@dlex dlex marked this pull request as ready for review May 10, 2023 01:42
dlex
dlex approved these changes May 10, 2023
View reviewed changes
Copy link
Contributor

@dlex dlex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Identical to the original

@dlex dlex modified the milestones: v23.1.x-next, v23.1.9 May 10, 2023
@dlex
Copy link
Contributor

dlex commented May 10, 2023

CI failure in https://buildkite.com/redpanda/redpanda/builds/28861#01880370-6349-41bf-9b7d-e4d412055511:

@dlex dlex merged commit 4059e26 into redpanda-data:v23.1.x May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dlex dlex dlex approved these changes

@graphcareful graphcareful Awaiting requested review from graphcareful

Assignees
No one assigned
Labels
kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v23.1.9
Development

Successfully merging this pull request may close these issues.
2 participants
@vbotbuildovich @dlex