Add (or document) a way to skip integration tests during local development

[bhrutledge]

Currently, if I just run pytest or tox -e py38, it runs all the tests, including test_integration.py, with a total runtime of nearly a minute. It'd be nice to get faster feedback.

I know pytest has lots of facilities for selecting/excluding tests, but I don't know which one(s) makes sense here. I'm guessing there's plenty of prior art for this sort of thing.

Off the top of my head, maybe they could be skipped by default, unless there's an environment variable set?

[bhrutledge]

Also, @jaraco, because I don't think I've said it yet: thanks for adding the integration tests. 😉

[jaraco]

I’ve been thinking the same thing. I’ve seen a pattern where slow tests are marked as slow and only run on demand.

I should point out that although the test take more than a minute on first run, because they reuse tox environments, they run faster on subsequent runs, as fast as 20 sec in my environment. Still, that’s 10x slower than runs without integration tests so probably worth making them opt-in.

[jaraco]

For a short term workaround, try -k 'not integration'.

[jaraco]

(to pytest, so tox -- -k 'not integration' for tox)

[jaraco]

You could implement a command-line option per these docs and enable it in CI with a PYTEST_ADDOPTS environment variable.

[deveshks]

Can I take a stab at this?

I ran tox -e py with and without skipping integration and there is a marked difference in the time taken (13 s vs 200 s) . Just running the integration test took 40 s

$ time tox -e py -- -k 'not integration'
real	0m13.827s
user	0m6.929s
sys	0m1.740s

$ time tox -e py 
real	2m10.033s
user	1m50.862s
sys	0m20.968s

$ time tox -e py -- tests/test_integration.py 
real	0m41.993s
user	0m44.723s
sys	0m5.849s

Perhaps adding instructions on skipping a test can be added at Contributing docs , something along the lines of how Getting started in pip does it?

I used python 3.8.2 with twine installed from the master branch, on an OSX Catalina 10.15.4

$ python --version
Python 3.8.2

$ twine --version
twine version 3.1.2.dev40+g653e82f (pkginfo: 1.5.0.1, requests: 2.23.0, setuptools: 41.2.0, requests-toolbelt: 0.9.1, tqdm: 4.44.1)

[sigmavirus24]

Given how widely used tox and pytest are in Python, do we really need to document their well-documented feature-set for contributors? Can we instead just link to their documentation?

With respect to skipping the integration tests, I wonder if we can instead use pytest's parallelism so the tests aren't run serially to improve the test performance. I'm wary to split out separate environments because that can lead to confusion for contributors (the tests passed locally but failed in CI because of these other things I didn't know I needed to run). Further, if our CI is the only thing running integration tests that will only make CI times longer as we add to them and don't experience their full pain ourselves. I'd rather feel that pain so as to not rely too heavily on integration tests than allow our CI runs to take 30 minutes or more eventually

[bhrutledge]

I think adding a sentence like this would be sufficient:

To skip the slower integration tests, use tox -- -k 'not integration'.

FWIW, I've been making this easier by setting these environment variables:

PYTEST_ADDOPTS=-k "not integration"
TOX_TESTENV_PASSENV=PYTEST_ADDOPTS

Which will skip integration tests if I just run tox, but still lets me run tox -- -k integration.

https://docs.pytest.org/en/latest/customize.html#adding-default-options
https://tox.readthedocs.io/en/latest/config.html#conf-passenv

Side note: the contributing docs don't mention pytest at all. So maybe another sentence like:

We use pytest to write and run tests.

[deveshks]

To skip the slower integration tests, use tox -- -k 'not integration'.

Specifically tagging the integration test as slower might not be the way to go IMO, we can generalize this by saying that we can skip test(s) by providing the -k, flag, and one example to skip a generally observed slower integration test is this.

FWIW, I've been making this easier by setting these environment variables:

PYTEST_ADDOPTS=-k "not integration"
TOX_TESTENV_PASSENV=PYTEST_ADDOPTS

Is this something done by twine in general, or specifically in your development env

Side note: the contributing docs don't mention pytest at all. So maybe another sentence like:

We use pytest to write and run tests.

I also observed that we are using pretend to create test stubs. Is that worth a mention too?

[bhrutledge]

Specifically tagging the integration test as slower might not be the way to go IMO, we can generalize this by saying that we can skip test(s) by providing the -k, flag, and one example to skip a generally observed slower integration test is this.

Are you game to submit a PR that includes your proposed language? We can workshop it there. 😉

Is this something done by twine in general, or specifically in your development env

Just in my dev env, via direnv.

I also observed that we are using pretend to create test stubs. Is that worth a mention too?

I think this would be better suited for a style guide (which doesn't exist). But I'd merge a brief mention of it.

[bhrutledge]

Skipping via PYTEST_ADDOPTS is now documented in the contributing guide: https://twine.readthedocs.io/en/latest/contributing.html#testing.

I'm going to close this for now; happy to revisit in the future, if needed.