Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Regex based subscriptions to KafkaIo Dynamic Read. #21338

Closed
damccorm opened this issue Jun 4, 2022 · 1 comment
Closed

Adding Regex based subscriptions to KafkaIo Dynamic Read. #21338

damccorm opened this issue Jun 4, 2022 · 1 comment

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

In https://issues.apache.org/jira/browse/BEAM-11325 the ability for Kafka to read from topics dynamically was added.

Along with this change, the ability to use regex to subscribe to topics in a dynamic way was discussed in the design for this change. 

https://docs.google.com/document/d/1FU3GxVRetHPLVizP3Mdv6mP5tpjZ3fd99qNjUI5DT5k/edit?disco=AAAALGbMoak

Pointing out the idea that subscribing to all topics in a cluster isn't particularly useful for most kafka users, where pattern based subscription is a very common pattern of use.

https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe-java.util.regex.Pattern-

https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/WatchKafkaTopicPartitionDoFn.java#L113

The current implementation does not utilize a pattern in anyway to subscribe to topics. The comments on the document mention piggy backing off the existing functionality in the KafkaConsumers subscribe method.

However, piggy backing on the existing consumer method is made difficult by the per partition subscription method used by beam.

But I believe a simple solution exists, 

public Read<K, V> withDynamicRead(Duration duration) {

As apart of the with dynamic read method, allow the option to pass a Pattern.

As the watchkafkatopicpartitiondofn does now, call listTopics, and then match against the list of topics using the supplied pattern. 

Imported from Jira BEAM-13987. Original Jira may contain additional context.
Reported by: njohnson223.

@Abacn
Copy link
Contributor

Abacn commented Jul 27, 2023

fixed by #26948

@Abacn Abacn closed this as completed Jul 27, 2023
@github-actions github-actions bot added this to the 2.50.0 Release milestone Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants