Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability for "aggregate-init" to read the last record in a stream at connector init #3120

Closed
ajhunyady opened this issue Apr 5, 2023 · 1 comment
Labels
enhancement New feature or request features/smartmodule good first issue Good for newcomers help wanted Good issue for community involvement no-issue-activity

Comments

@ajhunyady
Copy link
Contributor

ajhunyady commented Apr 5, 2023

When using an aggregate with producers, the aggregate needs and initial value. However, connectors don't currently have the ability to keep state, tough it has access to the data stream.

We should implement an aggregate-init function that gives the user access to the last record in the stream to initialize the aggregate value. As a references, there was a "custom" implementation of this in one of the legacy connectors:

https://github.com/infinyon/fluvio-connectors/blob/main/examples/github-stars/http-connector.yaml

This function should be available to aggregate smartmodules template in place of the parameter init.

Without this function, I end-up with this:
{"forks":134,"stars":1723}
{"forks":134,"stars":1723}
{"forks":134,"stars":1723,"old_forks":0,"old_stars":0}
{"forks":134,"stars":1723}
{"forks":134,"stars":1723} (edited)

every time the producer connector restarts.

@ajhunyady ajhunyady added bug Something isn't working enhancement New feature or request help wanted Good issue for community involvement good first issue Good for newcomers features/smartmodule and removed bug Something isn't working labels Apr 5, 2023
@github-actions
Copy link

github-actions bot commented Jun 5, 2023

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request features/smartmodule good first issue Good for newcomers help wanted Good issue for community involvement no-issue-activity
Projects
None yet
Development

No branches or pull requests

1 participant