[Bug]: Cannot read from Kafka due to short poll timeout of consumer in KafkaIO #30870
Open
1 of 16 tasks
Comments
|
.take-issue |
|
This also affects the Python SDK. |
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 9, 2024
…ansion service and Python
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 10, 2024
apache#30870: support consumer polling timeout in KafkaIO expansion service
xianhualiu
added a commit
to xianhualiu/beam
that referenced
this issue
Apr 10, 2024
damccorm
pushed a commit
that referenced
this issue
Apr 16, 2024
…ce (#30915) * [#30870]: support consumer polling timeout in KafkaIO expansion service * fixed spotless complains * fixed python format complains * Update sdks/python/apache_beam/io/kafka.py Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * Update sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * Update sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com> * fixed formating issue * fixed pylint and pydoc issues * shorten the variable name * fixed format and upgrade test * fixed test * fixed test --------- Co-authored-by: Jonathan Sabbagh <108473809+jbsabbagh@users.noreply.github.com>
Abacn
added a commit
to Abacn/beam
that referenced
this issue
Apr 16, 2024
…xpansion service (apache#30915)" This reverts commit a44c4f1.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
The default Kafka consumer poll timeout is set to 1 second. It works fine when the the response can get messages from the kafka broker server within this 1 second, such as when client accesses broker within the same region. But if the responding time is more than 1 second, the consumer will not retrieve any messages. One customer reported that throughput of processed messages was extremely low in cross-region read since most of the time the responding time takes more than 1 second.
As a solution, the Kafka consumer polling timeout needs to be configurable, so customer can adjust it according to their needs.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
The text was updated successfully, but these errors were encountered: