Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make handler tests callable with pytest #2173

Merged
merged 4 commits into from
Mar 13, 2023
Merged

Conversation

mreso
Copy link
Collaborator

@mreso mreso commented Mar 11, 2023

Description

This PR refactors the torch_handler unit tests to make them callable with pytest and integrates them into torchserve_sanity.py

Fixes #(issue)
1667 (partially)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test A
    Logs for Test A

  • Test B
    Logs for Test B

Checklist:

  • Did you have fun?
  • Have you added tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@codecov
Copy link

codecov bot commented Mar 11, 2023

Codecov Report

Merging #2173 (710ead3) into master (fd8f1b3) will increase coverage by 18.07%.
The diff coverage is 99.52%.

❗ Current head 710ead3 differs from pull request most recent head 3f5d4a5. Consider uploading reports for the commit 3f5d4a5 to get more accurate results

@@             Coverage Diff             @@
##           master    #2173       +/-   ##
===========================================
+ Coverage   53.37%   71.45%   +18.07%     
===========================================
  Files          71       73        +2     
  Lines        3226     3296       +70     
  Branches       57       57               
===========================================
+ Hits         1722     2355      +633     
+ Misses       1504      941      -563     
Impacted Files Coverage Δ
ts/torch_handler/unit_tests/models/base_model.py 80.00% <75.00%> (+80.00%) ⬆️
ts/torch_handler/unit_tests/conftest.py 100.00% <100.00%> (ø)
ts/torch_handler/unit_tests/test_base_handler.py 100.00% <100.00%> (+100.00%) ⬆️
ts/torch_handler/unit_tests/test_envelopes.py 100.00% <100.00%> (+100.00%) ⬆️
.../torch_handler/unit_tests/test_image_classifier.py 100.00% <100.00%> (+100.00%) ⬆️
...s/torch_handler/unit_tests/test_image_segmenter.py 100.00% <100.00%> (+100.00%) ⬆️
ts/torch_handler/unit_tests/test_mnist_kf.py 100.00% <100.00%> (+100.00%) ⬆️
...s/torch_handler/unit_tests/test_object_detector.py 100.00% <100.00%> (+100.00%) ⬆️
...s/torch_handler/unit_tests/test_utils/model_dir.py 100.00% <100.00%> (ø)

... and 12 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@msaroufim msaroufim self-requested a review March 11, 2023 21:48
Copy link
Member

@msaroufim msaroufim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will increase coverage by 18.03%.

:O

@mreso mreso changed the title [WIP]Make handler tests callable with pytest Make handler tests callable with pytest Mar 13, 2023
Copy link
Collaborator

@agunapal agunapal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mreso Could you please point to the code for integration with TS sanity .

Also, could you please attach the sanity logs to the PR

@mreso
Copy link
Collaborator Author

mreso commented Mar 13, 2023

@mreso Could you please point to the code for integration with TS sanity .

Also, could you please attach the sanity logs to the PR

@ankithagunapal Sure, integration point is here:

ts_test_cmd = f"python -m pytest --cov-report xml:{report_output_dir} --cov={coverage_dir} {test_dir} {handler_test_dir}"

This is output from

python -c "import ts_scripts.backend_utils; ts_scripts.backend_utils.test_torchserve()"
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/ubuntu/serve
plugins: mock-3.10.0, cov-4.0.0
collected 90 items

ts/tests/unit_tests/test_backend_metric.py .                             [  1%]
ts/tests/unit_tests/test_model_loader.py ........                        [ 10%]
ts/tests/unit_tests/test_model_service_worker.py .............           [ 24%]
ts/tests/unit_tests/test_otf_codec_protocol.py .........                 [ 34%]
ts/tests/unit_tests/test_version.py .                                    [ 35%]
ts/tests/unit_tests/test_worker_service.py ....                          [ 40%]
ts/tests/unit_tests/metrics_yaml_testing/metric_cache_unit_test.py ..... [ 45%]
...................................                                      [ 84%]
ts/torch_handler/unit_tests/test_base_handler.py ..                      [ 86%]
ts/torch_handler/unit_tests/test_envelopes.py .....                      [ 92%]
ts/torch_handler/unit_tests/test_image_classifier.py ..                  [ 94%]
ts/torch_handler/unit_tests/test_image_segmenter.py .                    [ 95%]
ts/torch_handler/unit_tests/test_mnist_kf.py ...                         [ 98%]
ts/torch_handler/unit_tests/test_object_detector.py .                    [100%]

=============================== warnings summary ===============================
../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

ts/torch_handler/unit_tests/test_image_segmenter.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
    warnings.warn(

ts/torch_handler/unit_tests/test_image_segmenter.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet101_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet101_Weights.DEFAULT` to get the most up-to-date weights.
    warnings.warn(msg)

ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle
ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle_kf
ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle_kfv2
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
    warnings.warn(warn_msg)

ts/torch_handler/unit_tests/test_object_detector.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:135: UserWarning: Using 'backbone_name' and 'weights' as positional parameter(s) is deprecated since 0.13 and may be removed in the future. Please use keyword parameter(s) instead.
    warnings.warn(

ts/torch_handler/unit_tests/test_object_detector.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights.
    warnings.warn(msg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.9.16-final-0 -----------
Coverage XML written to file ts/tests/unit_tests/coverage.xml

======================= 90 passed, 11 warnings in 5.95s ========================
## Started torchserve linting
## Started torchserve pytests
## In directory: /home/ubuntu/serve | Executing command: python -m pytest --cov-report xml:ts/tests/unit_tests/coverage.xml --cov=ts ts/tests/unit_tests ts/torch_handler/unit_tests

Let me know if you need full log.

Copy link
Collaborator

@agunapal agunapal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mreso mreso merged commit 9412c0a into master Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants