How to disable plugin based on environment?

[jaraco]

In python/typed_ast#144, I confirmed that typed_ast, and by extension pytest-black and pytest-mypy, are not supported on PyPy (and cause crashes on installation). I'd like to fashion a technique that reflects this limitation and bypasses the installation and enabling of these plugins when testing on PyPy.

I've found I can selectively include or exclude the plugin modules by adding environment markers to the requirement:

	pytest-black >= 0.3.7; platform_python_implementation != "PyPy"
	pytest-cov
	pytest-mypy; platform_python_implementation != "PyPy"

However, these plugins require a two-step installation process:

1. Install the dependency.
2. Enable the plugin with --black or --mypy during pytest invocation.

Leaving the --<plugin> in pytest.ini (example) while deselecting the plugin for installation results in an InvocationError:

keyring master $ tox -r -e pypy3
pypy3 recreate: /Users/jaraco/code/main/keyring/.tox/pypy3
pypy3 develop-inst: /Users/jaraco/code/main/keyring
pypy3 installed: attrs==20.1.0,cffi==1.14.0,coverage==5.2.1,docutils==0.16,flake8==3.8.3,greenlet==0.4.13,importlib-metadata==1.7.0,iniconfig==1.0.1,-e git+gh://jaraco/keyring@a6980e4b0a0c25143ab027e437a1b5b81f9c0bf5#egg=keyring,mccabe==0.6.1,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.0.1,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-flake8==1.0.6,readline==6.2.4.1,six==1.15.0,toml==0.10.1,zipp==3.1.0
pypy3 run-test-pre: PYTHONHASHSEED='4203199892'
pypy3 run-test: commands[0] | pytest
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --black --mypy
  inifile: /Users/jaraco/code/main/keyring/pytest.ini
  rootdir: /Users/jaraco/code/main/keyring

ERROR: InvocationError for command /Users/jaraco/code/main/keyring/.tox/pypy3/bin/pytest (exited with code 4)
_____________________________________________________________ summary ______________________________________________________________
ERROR:   pypy3: commands failed

Best I can tell, there's no easy way to programmatically disable the plugin (or only enable it if present).

In pytest-dev/pytest-cov#418, I encountered a similar problem where it was suitable to install coverage but its presence needed to be disabled at runtime. The workaround for that approach was specifically tuned to implementation details of that plugin.

What I'd really like is a uniform way for plugins to declare there presence, for them to be enabled by default, and layers of configuration (global, system, user, environment, invocation) where the enabling of that feature can be toggled on or off based on certain factors (maybe environment markers or maybe arbitrary Python logic).

But thinking short-term, are there any options available to selectively disable the enabling of the black and mypy plugins, such that they're enabled by default, but disabled when the environment is built on PyPy?

[bluetech]

If I understand correctly, you want to load the plugins, so that pytest accepts the flags, but you don't want them to actually run. This is not a normal scenario so there isn't any particular support for it. I think the best solution would be to just have the plugins behave appropriately (i.e. not crash) on PyPy.

However if this is really not possible, some workarounds I can think of are:

• unregister the plugins at runtime, after cmd line parsing but before they do anything. Not sure how well that works.

• Add a local plugin which parses and ignores the --black and --mypy flags, just so things work when the real plugins are not loaded.

[jaraco]

you want to load the plugins, so that pytest accepts the flags

Actually, that doesn't help in this particular situation, because the plugins can't even be installed.

What I imagine is a framework that offers one or more of these features:

• installed (present) plugins are enabled by default, so that only one action is required to add a plugin (declare/install the plugin) or disable it (uninstall the plugin).
• plugins can be enabled or disabled through environment variables, config files, and command-line overrides.
• unrecognized command line options will raise warnings but proceed, or can be configured to exist (e.g. pytest.allow_missing_params=--black,--mypy).

Actually, just the first feature would be sufficient, but I suspect that without the second feature, many users would find the enabled-by-default functionality annoying or even problemmatic.

Features one and two could be achieved by decoupling a plugin's config from the pytest API. If each plugin was responsible to load its own config (or had its own section in config), the absence of a plugin would be unaffected by the presence of config for that plugin. Pytest could provide a generic interface, like --enable-plugin=black or --disable-plugin=mypy that would allow overriding of any plugin by name at the command-line.

The fact that almost all configuration is buried in a single setting "addopts" makes it non-obvious what the behavior is when the value is set in the environment and the config and the command-line and puts the burden on plugins to support settings that override settings at a broader scope.

The first workaround won't work in this case because the plugins aren't installed. The second option doesn't exactly work because it needs to work across every project that might be affected (any project that may want to use pytest-black or pytest-mypy on pypy). I'll try the second option and create a generic plugin that does nothing but implement support for select command-line options.

[jaraco]

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml. For example, the following toml will work to expect --black and --mypy parameters when those plugins aren't present:

[tool.jaraco.pytest.opts.--black]
action = "store_true"

[tool.jaraco.pytest.opts.--mypy]
action = "store_true"

[nicoddemus]

Hi @jaraco,

Actually, just the first feature would be sufficient, but I suspect that without the second feature, many users would find the enabled-by-default functionality annoying or even problemmatic.

This is actually the plugin's implementation choice, that they require an extra flag to be active. They could also optionally be enabled via config, which does not produce an error for unknown configs.

About disabling the plugins, that's possible through all the different ways you listed:

• environment variables: PYTEST_ADDOPTS, with -p no:<plugin>.
• config files: using addopts: -p no:<plugin>
• command-line overrides: again -p to the rescue.

Features one and two could be achieved by decoupling a plugin's config from the pytest API. If each plugin was responsible to load its own config (or had its own section in config), the absence of a plugin would be unaffected by the presence of config for that plugin. Pytest could provide a generic interface, like --enable-plugin=black or --disable-plugin=mypy that would allow overriding of any plugin by name at the command-line.

Not sure, there's nothing that prevents a plugin to load its configuration from some other place, it's their choice. Of course, almost all plugins use the pytest config API as that's what plugins are supposed to do. Given that this is not a normal situation, I don't think it makes much sense for pytest to implement an alternative interface.

The fact that almost all configuration is buried in a single setting "addopts" makes it non-obvious what the behavior is when the value is set in the environment and the config and the command-line and puts the burden on plugins to support settings that override settings at a broader scope.

Not sure I follow, the precedence of each is well defined: command-line first, config second, env vars last.

I'll try the second option and create a generic plugin that does nothing but implement support for select command-line options.

Yeah I think this is the way to go. While I can see we adding a switch to pytest to transform unknown options into warnings instead of errors, I think this is not common enough to add a new switch to the already large list of configuration options we have.

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml.

Nice! This allows a escape hatch for others in the same situation as you (that I believe are a bit rare however).

[graingert]

I've released jaraco.test 3.0, which has a pytest plugin that allows defining meaningless options in pyproject.toml.

I made pytest-forwards-compatible for addopts options, but it's currently only useful for .ini/.cfg projects

[jaraco]

The technique in jaraco.test doesn't work (jaraco/jaraco.test#1), so I'm back to the drawing board.

I'll check out pytest-forwards-compatible.

[jaraco]

Not sure, there's nothing that prevents a plugin to load its configuration from some other place, it's their choice. Of course, almost all plugins use the pytest config API as that's what plugins are supposed to do.

Right, so plugins are advised to follow an approach that runs afoul of this issue. Since pytest advises plugins to solicit their config from command-line parameters, it becomes difficult to selectively provide that config. The "addopts" technique also dosen't allow overriding a value at a closer scope that was defined at a broader scope. That is, you can't define "PYTEST_ADDOPTS=--black" at the environment level then "pytest_addopts=--no-black" at the config level then "--black" at the command-line. More importantly, the absence of the plugin causes the whole test suite to break due to any of these settings.

I think my next approach is going to be to provide a translator. To solicit settings for various plugins in separate sections of a config file and to only enable those settings when the plugin is available.

I apologize for being unclear about the challenges I'm facing. I've long found the "addopts" approach to be clumsy but it's only now that its clumsiness is really affecting my ability to use pytest. I'm conflating several separate concerns:

• need for configuration to be decoupled from plugin presence
• desire for plugin presence to imply plugin functionality
• ability to disable a plugin based on environment
• ability to configure plugins (with overrides) at various scopes

[nicoddemus]

Since pytest advises plugins to solicit their config from command-line parameters, it becomes difficult to selectively provide that config.

Would be possible for pytest-black and pytest-mypy to add configuration options (in pytest.ini)? Then you can drop --black and --mypy from the command-line, and add them to your pytest.ini.

EDIT: hmm but I see now that probably doesn't apply to pytest-black and pytest-mypy, as they are more like linters that you want to run in separate pytest invocations, correct?

I apologize for being unclear about the challenges I'm facing. I've long found the "addopts" approach to be clumsy but it's only now that its clumsiness is really affecting my ability to use pytest

No worries, I agree addopts is a bit clumsy sometimes.

need for configuration to be decoupled from plugin presence

You mean command-line arguments here, correct? To be clear, ini options today don't emit an error when an unknown option is defined in pytest.ini (which arguably is a problem). You can for example add a key named black=1 in pytest.ini and pytest will only show a warning (unless --strict-config is given, in which case this becomes an error).

desire for plugin presence to imply plugin functionality

This is really up to each plugin to implement as they see fit, and not something pytest can really enforce, I think. For example, installing pytest-sugar will enable it right away, without the need for a configuration flag.

ability to disable a plugin based on environment
ability to configure plugins (with overrides) at various scopes

That's interesting, but we probably would need a formal proposal to be able to discuss all the details. We should also continue to try to figure out if this can be implemented as a separate plugin, instead of putting directly into the core.

[jaraco]

they are more like linters that you want to run in separate pytest invocations, correct?

I prefer to think of linters as just another class of test that are run as a matter of course when testing the code, enabled by default and disabled selectively if needed (such as through -k "not black" or --plugin="no:black").

You mean command-line arguments here, correct?

I mean configuration generally, but since the recommendation is for plugin authors to solicit configuration from the command line and that's what plugin authors are doing, that's where it's problematic.

[jaraco]

In jaraco.test 3.1.1, I've created a plugin that appears to be working and I suspect isn't subject to the race conditions because it uses the pytest_load_initial_conftests hook. The implementation ultimately was pretty straightforward.

This approach has the added advantage that supplying -p no:black now works instead of causing the invocation to fail with "no option --black". I plan to utilize this for --cov and --flake8 also. I'm pretty happy with this behavior. I'll test it out for some time, but do feel like a similar design would be good for pytest to adopt as a first-class feature and recommended usage.

[jaraco]

I tried using this technique for --cov, but it doesn't work. At the point where pytest_load_initial_configuration is called, early_config.pluginmanager.has_plugin('_cov') is False (same for 'cov'). It seems pytest-cov already uses pytest_load_initial_configuration with pytest.mark.tryfirst, so it seems it may not be possible to enable coverage based on the presence of the the plugin :(.

[jaraco]

Is there a hook that can run before the pytest.mark.tryfirst(pytest_load_initial_configuration)?

[jaraco]

I figured out an (ugly) hack (jaraco/jaraco.test@f92cf8f).

[nicoddemus]

Indeed I suspect pytest-cov does anything it can to get coverage started as early as possible, which makes sense.

Let me comment back to some of your previous bullet points:

1. need for configuration to be decoupled from plugin presence

If I understand what you mean, I'm not sure how that would work. How would pytest know about a plugin configuration (and here I understand command-line flags) without the plugin being installed (that's what I get from "plugin presence", correct me if I'm wrong)? Do you have some idea on how that would work in practice?

2. desire for plugin presence to imply plugin functionality

I mentioned this a few times, but I don't seem to be getting my point across, sorry.

When pytest loads a plugin, all it can do after that point is call hooks on the plugins. What plugins do in their hooks is decided by the authors, so some authors decide to enable their functionality somehow: a command-line flag (black, flake8), the presence of a utility (xvfb), always on by default (sugar, cpp), and any other way they can think of. All of those are really implementation-dependent, and vary from plugin to plugin.

To illustrate this, a plugin might decide to only do its functionality if a certain command-line flag is passed. A reasonable implementation is to skip installing its hooks entirely during pytest_configure if the flag is not present in the command-line:

def pytest_configure(config):
    if config.getoption("myoption"):
         config.pluginmanager.register(MyActualHooks())

This makes it clear that pytest cannot enforce the plugin to do anything by default: the author chose to only provide its functionality when myoption is given.

Or do you mean by this that pytest should encourage plugins to always be on (in the docs for example), without relying on flags/environment/tools/etc?

If that's the case I don't see that being practical or enforceable, there are just too many plugin variants for that to work.

• 3. Ability to disable a plugin based on environment

That's possible already but we could extend even further. For example, a hook called very early that allows arbitrary code to disable other plugins, but we hit problem 1 again: unrecognized options will be a problem.

• 4. ability to configure plugins (with overrides) at various scopes

Can you detail this a bit more please?

---

Sorry for the long post, but lets backtrack a bit.

AFAIU the problem you are facing is:

1. You have a single source of configuration for the pytest command-line, such as your .travis.yml file or tox.ini configuration, and there you call pytest with pytest --black (using just pytest-black for the sake of discussion for now).
2. pytest-black crashes on installation on PyPy, so you skip its installation on that platform.
3. The problem: pytest-black does not get installed on PyPy, but now pytest --black raises an error about the unrecognized --black option.

Perhaps a solution would be to include a hook on pytest to let plugins decide how to deal with unknown command-line options (and possibly ini-options as well), instead of raising an error. The default implementation would continue to raise an error, but a user might override that to print a warning instead. That should be really simple to implement and flexible enough to solve the immediate problem you are having.

Thoughts?

[The-Compiler]

I wonder if it'd help to have something like a addopts_optional in pytest.ini? Those options would then be added if they are recognized/registered, and ignored if they are unknown.

However, I guess all those would need to be --flags without any additional arguments (--flag foo wouldn't work, how would pytest know what to consume without a plugin registering it?).

Still, it might be a solution for the issues mentioned in this thread? I've certainly seen similar issues when I e.g. want to addopts = --instafail to immediately see failure output, but if Linux distributions want to run the tests in their environment, that means they'll have to package pytest-instafail (or patch my pytest.ini) just so they can run the tests.