Add example of real time Anomaly Detection using RunInference #23497

shub-kris · 2022-10-05T11:16:04Z

This PR aims to illustrates an example of setting up a real time text anomaly detection pipeline using PubSub and RunInference.

The entire implementation is divided into two different pipelines:

write_data_to_pubsub_pipeline pushes data to PubSub
anomaly_detection_pipeline reads data from PubSub, uses a trained HDBSCAN model and does anomaly detection.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

shub-kris · 2022-10-05T11:16:56Z

@damccorm @andyxiexu please have a look and give your comments.

codecov · 2022-10-05T11:56:27Z

Codecov Report

Merging #23497 (046227b) into master (b9aa159) will decrease coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #23497      +/-   ##
==========================================
- Coverage   73.45%   73.33%   -0.12%     
==========================================
  Files         718      719       +1     
  Lines       95884    95798      -86     
==========================================
- Hits        70427    70249     -178     
- Misses      24146    24238      +92     
  Partials     1311     1311

Flag	Coverage Δ
python	`83.04% <ø> (-0.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/gcp/bigquery_tools.py	`73.34% <0.00%> (-12.39%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery.py	`71.31% <0.00%> (-2.94%)`	⬇️
sdks/python/apache_beam/internal/gcp/json_value.py	`85.50% <0.00%> (-2.90%)`	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	`80.80% <0.00%> (-2.12%)`	⬇️
sdks/python/apache_beam/runners/runner.py	`72.39% <0.00%> (-1.85%)`	⬇️
...apache_beam/typehints/native_type_compatibility.py	`85.52% <0.00%> (-1.06%)`	⬇️
...thon/apache_beam/runners/worker/sdk_worker_main.py	`77.71% <0.00%> (-0.78%)`	⬇️
sdks/python/apache_beam/pipeline.py	`92.25% <0.00%> (-0.60%)`	⬇️
sdks/python/apache_beam/io/iobase.py	`85.92% <0.00%> (-0.50%)`	⬇️
sdks/python/apache_beam/transforms/combiners.py	`93.05% <0.00%> (-0.39%)`	⬇️
... and 18 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

damccorm

Overall looks good to me, had a number of smaller comments (mostly wording)

website/www/site/content/en/documentation/ml/anomaly-detection.md

...che_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/pipeline/options.py

.../python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/setup.py

damccorm

Just a few lingering comments, should be good to go after those though. Thanks!

website/www/site/content/en/documentation/ml/anomaly-detection.md

sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/main.py

...che_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/pipeline/options.py

.../examples/inference/anomaly_detection/anomaly_detection_pipeline/pipeline/transformations.py

...ython/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/main.py

...he_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/pipeline/utils.py

github-actions · 2022-10-06T15:41:45Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @pabloem for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

rezarokni · 2022-10-11T14:23:34Z

website/www/site/content/en/documentation/ml/overview.md

@@ -59,3 +59,4 @@ In order to automate and track the AI/ML workflows throughout your project, you

 You can find examples of end-to-end AI/ML pipelines for several use cases:


Please add more descriptions to this list, for each item a small paragraph of what this does with high level example of the use case shown .

@Juta please have a look and let's discuss this

I agree it would be good to add a little bit of detail on these, we can also probably defer that to a future PR though where we clean up that page more generally. @shub-kris @Juta thoughts?

@damccorm I will also prefer to do in a different PR. @Juta what do you think?

SGTM, I'll go ahead and merge and we can circle back to this

Yes ok let's pick this up in a next PR

.../examples/inference/anomaly_detection/anomaly_detection_pipeline/pipeline/transformations.py

...hon/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/config.py

...ython/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/main.py

shub-kris · 2022-10-12T12:12:04Z

retest this please

...ython/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/main.py

damccorm · 2022-10-14T15:01:25Z

website/www/site/content/en/documentation/ml/overview.md

@@ -59,3 +59,4 @@ In order to automate and track the AI/ML workflows throughout your project, you

 You can find examples of end-to-end AI/ML pipelines for several use cases:


I agree it would be good to add a little bit of detail on these, we can also probably defer that to a future PR though where we clean up that page more generally. @shub-kris @Juta thoughts?

damccorm

added one small comment, and it looks like there are 2 other lingering comments from Reza (the PROJECT_ID one and the overview.md one)

damccorm · 2022-10-19T18:47:59Z

Thanks! I'll merge once tests complete

Add example of Anomaly Detection and Documentation

9347b81

github-actions bot added examples python website labels Oct 5, 2022

Fix formatting and linting error

fa33069

damccorm requested changes Oct 5, 2022

View reviewed changes

Fix code links, edit documentation, minor changes in code

4ffcb64

damccorm reviewed Oct 6, 2022

View reviewed changes

website/www/site/content/en/documentation/ml/anomaly-detection.md Outdated Show resolved Hide resolved

website/www/site/content/en/documentation/ml/anomaly-detection.md Outdated Show resolved Hide resolved

damccorm mentioned this pull request Oct 6, 2022

Grant actions using GITHUB_TOKEN the appropriate permission set #23521

Merged

4 tasks

yeandy reviewed Oct 6, 2022

View reviewed changes

github-actions bot added the Next Action: Reviewers label Oct 6, 2022

Shubham Krishna added 3 commits October 10, 2022 10:13

Improve documentation and docstrings

b6de3ef

Add new lines to improve documentation

6868e18

Add new line to fix python formatting

adfa2ef

rezarokni reviewed Oct 11, 2022

View reviewed changes

Shubham Krishna added 2 commits October 12, 2022 08:22

Improve documentation and change custom DOFn class names

5abc762

Fix import pylint error

538203f

shub-kris requested review from damccorm and yeandy and removed request for damccorm October 14, 2022 06:07

damccorm reviewed Oct 14, 2022

View reviewed changes

damccorm added Next Action: Author and removed Next Action: Reviewers labels Oct 14, 2022

Unset PROJECT_ID in config.py and fix docstring

046227b

github-actions bot added Next Action: Reviewers and removed Next Action: Author labels Oct 19, 2022

damccorm approved these changes Oct 19, 2022

View reviewed changes

damccorm merged commit 8e8e89e into apache:master Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add example of real time Anomaly Detection using RunInference #23497

Add example of real time Anomaly Detection using RunInference #23497

shub-kris commented Oct 5, 2022

shub-kris commented Oct 5, 2022

codecov bot commented Oct 5, 2022 •

edited

Loading

damccorm left a comment

damccorm left a comment

github-actions bot commented Oct 6, 2022

rezarokni Oct 11, 2022

shub-kris Oct 11, 2022

damccorm Oct 14, 2022

shub-kris Oct 19, 2022

damccorm Oct 19, 2022

Juta Oct 20, 2022

shub-kris commented Oct 12, 2022

damccorm Oct 14, 2022

damccorm left a comment

damccorm commented Oct 19, 2022 •

edited

Loading

		@@ -59,3 +59,4 @@ In order to automate and track the AI/ML workflows throughout your project, you

		You can find examples of end-to-end AI/ML pipelines for several use cases:

Add example of real time Anomaly Detection using RunInference #23497

Add example of real time Anomaly Detection using RunInference #23497

Conversation

shub-kris commented Oct 5, 2022

GitHub Actions Tests Status (on master branch)

shub-kris commented Oct 5, 2022

codecov bot commented Oct 5, 2022 • edited Loading

Codecov Report

damccorm left a comment

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 6, 2022

rezarokni Oct 11, 2022

Choose a reason for hiding this comment

shub-kris Oct 11, 2022

Choose a reason for hiding this comment

damccorm Oct 14, 2022

Choose a reason for hiding this comment

shub-kris Oct 19, 2022

Choose a reason for hiding this comment

damccorm Oct 19, 2022

Choose a reason for hiding this comment

Juta Oct 20, 2022

Choose a reason for hiding this comment

shub-kris commented Oct 12, 2022

damccorm Oct 14, 2022

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

damccorm commented Oct 19, 2022 • edited Loading

codecov bot commented Oct 5, 2022 •

edited

Loading

damccorm commented Oct 19, 2022 •

edited

Loading