Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.1.x] Reproduce the OOM on fetching from many partitions #10652

Merged
merged 4 commits into from
May 10, 2023
Merged

[v23.1.x] Reproduce the OOM on fetching from many partitions #10652

merged 4 commits into from
May 10, 2023

Commits on May 10, 2023

  1. tests/kaf_consumer: support num_records in KafConsumer 

    KafConsumer has `num_records` parameter that leads to nowhere. This commit
makes it pass the specified value as `--limit-messages` argument of
`kaf consume`. Change of the default value of `num_records` does not
affect any existing code

(cherry picked from commit fdb0e58)
    @dlex @vbotbuildovich
    dlex authored and vbotbuildovich committed May 10, 2023
    Configuration menu
    Copy the full SHA
    b055e8c View commit details
    Browse the repository at this point in the history

  2. tests/rpk_producer: Add max_message_bytes to RpkProducer 

    `max_message_bytes` is passed as the corresponding `rpk topic produce`
argument to control maximum batch size in Produce API.

(cherry picked from commit cf964b3)
    @dlex @vbotbuildovich
    dlex authored and vbotbuildovich committed May 10, 2023
    Configuration menu
    Copy the full SHA
    d1b2560 View commit details
    Browse the repository at this point in the history

  3. tests/kafka_cli: Add max.message.bytes topic config 

    Add `max.message.bytes` topic config to KCL wrapper's `create_topic`
so that it's possible to specify that parameter while using the regular
RedpandaTest way to create topics

(cherry picked from commit f84688d)
    @dlex @vbotbuildovich
    dlex authored and vbotbuildovich committed May 10, 2023
    Configuration menu
    Copy the full SHA
    c6265d0 View commit details
    Browse the repository at this point in the history

  4. tests: MemoryStressTest.test_fetch_with_many_partitions 

    Add a test to reproduce the case when redpanda OOMs on reading at least
a single batch from each of many partitions in a fetch request, so that
the memory required to store `(partition leaders in a shard)*(batch size)`
is too much to allocate.

(cherry picked from commit 7f18d94)
    @dlex @vbotbuildovich
    dlex authored and vbotbuildovich committed May 10, 2023
    Configuration menu
    Copy the full SHA
    a49eb85 View commit details
    Browse the repository at this point in the history