Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-14474] Suppress 'Mean of empty slice' Runtime Warning in dataframe unit test #17682

Merged
merged 2 commits into from
May 18, 2022

Conversation

Abacn
Copy link
Contributor

@Abacn Abacn commented May 16, 2022

Currently the log of python unit test results contains thousands of lines of runtime warning / future warning / deprecation warnings, making trouble shooting during development inconvenient. The most popped warning is hundreds of lines of 'RuntimeWarning: Mean of empty slice
return np.nanmean(a, axis, out=out, keepdims=keepdims)'
raised in dataframe tests, which costs > 400 lines of logs. This warning could be suppressed.

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@asf-ci
Copy link

asf-ci commented May 16, 2022

Can one of the admins verify this patch?

1 similar comment
@asf-ci
Copy link

asf-ci commented May 16, 2022

Can one of the admins verify this patch?

@codecov
Copy link

codecov bot commented May 16, 2022

Codecov Report

Merging #17682 (0a93326) into master (ee5888d) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #17682      +/-   ##
==========================================
- Coverage   73.99%   73.99%   -0.01%     
==========================================
  Files         695      695              
  Lines       91798    91798              
==========================================
- Hits        67927    67922       -5     
- Misses      22623    22628       +5     
  Partials     1248     1248              
Flag Coverage Δ
python 83.73% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../python/apache_beam/testing/test_stream_service.py 88.09% <0.00%> (-4.77%) ⬇️
...che_beam/runners/interactive/interactive_runner.py 89.20% <0.00%> (-1.44%) ⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py 93.28% <0.00%> (-0.75%) ⬇️
sdks/python/apache_beam/runners/common.py 87.94% <0.00%> (-0.13%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.55% <0.00%> (ø)
sdks/python/apache_beam/utils/interactive_utils.py 97.56% <0.00%> (+2.43%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee5888d...0a93326. Read the comment docs.

@Abacn Abacn marked this pull request as ready for review May 16, 2022 20:19
@Abacn
Copy link
Contributor Author

Abacn commented May 16, 2022

R: @TheNeuralBit

Copy link
Member

@TheNeuralBit TheNeuralBit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

I think we should consider filtering these warnings in the actual implementation so they don't show up when the user is building a pipeline and we're generating a proxy. Would you mind filing a jira for that?


def median_sum_fn(x):
return (x.foo + x.bar).median()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would actually keep these where they are and just duplicate the filterwarnings calls, so that the tests are more self-contained and easy to inspect (but I'm also fine with the move if you prefer it that way)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I agree with the idea. Given the amount of code added I considered pick up them into a separate helper function. Nevertheless it is still in the GroupByTest test class.

@staticmethod
def median_sum_fn(x):
warnings.filterwarnings("ignore", message="Mean of empty slice")
return (x.foo + x.bar).median()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use a context here to make sure the filters get reset?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Applied.

@Abacn
Copy link
Contributor Author

Abacn commented May 16, 2022

Entered BEAM-14477 for construction time warnings.

@TheNeuralBit TheNeuralBit merged commit 857f8d3 into apache:master May 18, 2022
@Abacn Abacn deleted the suprressnanmean branch May 18, 2022 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants