test: change way pytest is run in CI

[terriko]

fixes: #1159

This switches the way we're running tests so that it does a full pytest gather for the async tests (minus a few files known to cause rate limiting issues). This will run a few tests that were not previously being run, and will help us so that new test files no longer need to be added manually to CI before they will run.

[codecov-commenter]

Codecov Report

Merging #1251 (f71fc33) into main (792280b) will increase coverage by 2.19%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1251      +/-   ##
==========================================
+ Coverage   76.60%   78.80%   +2.19%     
==========================================
  Files         240      240              
  Lines        4501     4501              
  Branches      559      559              
==========================================
+ Hits         3448     3547      +99     
+ Misses        906      804     -102     
- Partials      147      150       +3     

Flag	Coverage Δ
longtests	78.80% <ø> (+2.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cve_bin_tool/async_utils.py	91.66% <0.00%> (+1.38%)	⬆️
cve_bin_tool/helper_script.py	41.83% <0.00%> (+41.83%)	⬆️
test/test_async_utils.py	100.00% <0.00%> (+100.00%)	⬆️
test/test_helper_script.py	100.00% <0.00%> (+100.00%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 792280b...f71fc33. Read the comment docs.

[terriko]

Failing in test_helper_script on windows:

    def test_parse_filename(self, filename, product_name, version_name):
        hs = HelperScript(filename)
>       assert (product_name, version_name) == hs.parse_filename(filename)
E       AssertionError: assert ('bash', '4.2.46') == ('./Packages/bash', '4.2.46')
E         At index 0 diff: 'bash' != './Packages/bash'
E         Full diff:
E         - ('./Packages/bash', '4.2.46')
E         ?   -----------
E         + ('bash', '4.2.46')

test\test_helper_script.py:53: AssertionError
------------------------------ Captured log call ------------------------------
WARNING  cve_bin_tool.HelperScript:helper_script.py:247 
=================================================================
No match was found for "./Packages/bash" in database. 
Please check your file or try specifying the "product_name" also.
=================================================================
=========================== short test summary info ===========================
FAILED test/test_helper_script.py::TestHelperScript::test_parse_filename[./Packages/bash-4.2.46-34.el7.x86_64.rpm-bash-4.2.46]
================= 1 failed, 234 passed, 223 skipped in 26.68s =================

[terriko]

I've disabled test_helper_script.py in windows for now; I'll open up an issue to track it if I merge this before we have a fix there.

[terriko]

Saving a potential commit message:

• fixes: Missing tests in CI #1159
• test: change way pytest is run in CI

Switch to using a full pytest gather for the async tests. Several test files are singled out to be run synchronously because they touch NVD and can cause us to get rate limited if they run in parallel.

This will ensure that no one needs to add new test files explicitly to CI unless they need to be run synchronously.

Several tests are (temporarily) disabled because they are not passing; we'll enable those as they are fixed.