Automatically register the schema tester pytest plugin

[drdavella]

Previously packages were required to explicitly list the schema tester plugin in their conftest.py files. Now, the plugin is registered automatically with pytest.

The schema tests are no longer enabled by default. Instead, packages can do one of two things:

1. Add asdf_schema_tests_enabled = true to the [tool:pytest] section of their setup.cfg file to enable tests by default, or
2. Add --asdf-tests to the pytest command line to enable tests on a per-run basis.

The new plugin is in a different namespace than the previous plugin, so it will not cause conflicts with packages that still include the plugin in conftest.py. However, the old conftest.py configuration should be removed unless packages want to retain backwards compatibility with older versions of ASDF.

This closes #657.

cc @astrofrog

[stsci-bot]

Hi there @drdavella 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog. I help make sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

[stsci-bot]

Hi there @drdavella 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog. I help make sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

[codecov]

Codecov Report

Merging #676 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #676   +/-   ##
=======================================
  Coverage   93.34%   93.34%           
=======================================
  Files          39       39           
  Lines        4295     4295           
=======================================
  Hits         4009     4009           
  Misses        286      286

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8d8b3b...5ee090d. Read the comment docs.

[drdavella]

Packaging the plugin in this way is proving to be unnecessarily burdensome. The right way to do it would be to move the plugin to another package entirely, but I don't think that's the right way to go.

[drdavella]

Closing since this isn't really worth any more time/effort at the moment.

[astrofrog]

Could you clarify what the issue is here with registering the pytest plugin?

[astrofrog]

Packaging the plugin in this way is proving to be unnecessarily burdensome. The right way to do it would be to move the plugin to another package entirely, but I don't think that's the right way to go.

What would be wrong with moving the plugin to another package? In a way you already have a separate package for a sphinx plugin, so why not for a pytest plugin?

[drdavella]

@astrofrog the issues have all been related to using the plugin to test the asdf package itself. I'm not sure I understand the root cause, but it seems to be related to the fact that the asdf package gets imported by pytest when the plugin is loaded, but also when test collection occurs. I think I found workarounds for most of these issues, but it's still causing failures on Windows for unknown reasons. I don't really want to abandon testing on Windows just because of this, but I'm not sure how much more time/effort I can afford at the moment. I'll try one more idea here now, but then I might have to leave this effort to someone else.

As for moving the plugin to another package, I think the future maintenance burden would be lower if it remained part of the same repository. However, with the way I have set things up here, the plugin is technically installed as a separate package.

[drdavella]

Looks like eliminating potentially circular references in the plugin got things working. Thanks @astrofrog for encouraging me to continue!

[jdavies-st]

Nice. This is potentially useful elsewhere to get the full-bells-and-whistles resolver so that we don't have to instantiate a dummy AsdfFile object just to get to the resolver.

[jdavies-st]

So is the following necessary in the top-level conftest.py for a package that would like to use asdf_schema_tester in order to work with both the old schema tester and new one via entry points?

import pkg_resources

entry_points = []
for entry_point in pkg_resources.iter_entry_points('pytest11'): 
    entry_points.append(entry_point.name)

if "asdf_schema_tester" not in entry_points:
    pytest_plugins = ['asdf.tests.schema_tester']

Plus enabling it in setup.cfg. Or is there a simpler way?

[drdavella]

@jdavies-st since the new plugin is in a different namespace, I think there's actually no action required for packages that want to retain backwards compatibility. You will need to add asdf_schema_tests_enabled to your setup.cfg in order to enable the tests by default with the new plugin, but other than that there should be no issues.

[jdavies-st]

But asdf.tests.schema_tester no longer exists in asdf namespace. So if we have

pytest_plugins = ['asdf.tests.schema_tester']

in our conftest.py, then we get a plugin import error. Our 2 test runs yesterday against asdf/master failed because of this:

https://travis-ci.org/spacetelescope/jwst/builds/518489243

https://travis-ci.org/spacetelescope/jwst/jobs/518489251#L685

[drdavella]

Ah yeah good point. Yes I guess you do need logic like what you posted above, then. I wonder if it would be better to use the version of asdf as the condition though?

[jdavies-st]

Agree. Once a version of asdf is released with the new plugin, then version would be better. Thanks for the suggestion. =)