Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python sdk's PeriodicImpulse generates a bounded PCollection #21606

Closed
damccorm opened this issue Jun 5, 2022 · 2 comments · Fixed by #21815
Closed

Python sdk's PeriodicImpulse generates a bounded PCollection #21606

damccorm opened this issue Jun 5, 2022 · 2 comments · Fixed by #21815
Labels

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 5, 2022

See the dev mail list thread for details: https://lists.apache.org/thread/ps3m0jc0ngqp1y2s0mv2n6hxhvgkr3vw

PeriodicImpluse transform in Java sdk generates unbounded
PCollection; while in Python sdk it generates bounded PCollection. The
latter case may cause issues in streaming.

Per Cham: Note that the primary use-case of PeriodicImpulse (according to the design doc) was to
generate a fixed/bounded input that can slowly change over time but
changing over time dimension would make it unbounded.

Seems like we need to make python PeriodicImpulse generates an unbounded pcoll, in alignment with Java implementation, and also make sure that the change does not break the current implementation of its original use case (stream enrichment problem).

Imported from Jira BEAM-14498. Original Jira may contain additional context.
Reported by: yihu.

@Abacn
Copy link
Contributor

Abacn commented Jun 11, 2022

@lostluck lostluck added this to the 2.40.0 Release milestone Jun 13, 2022
@kennknowles kennknowles removed this from the 2.40.0 Release milestone Jun 14, 2022
@kennknowles
Copy link
Member

It is a clear bug, but I don't think we need to block the release on this do we?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants