Specify how to deal with requirements for Python packages

[jvdzwaan]

Requirements can be put in setup.py and/or requirements.txt. We could have some more advice on what to put where.

See for example:
https://packaging.python.org/discussions/install-requires-vs-requirements/
https://caremad.io/posts/2013/07/setup-vs-requirement/

[jspaaks]

I tried having a single source of truth by reading the contents of requirements.txt and requirements-dev.txt from setup.py

https://github.com/citation-file-format/cff-converter-python/blob/44a82dfd36c9120bf2fcf3e56e74a483a8f40262/setup.py#L8-L17

Generally speaking, I found this matter confusing and not at all Pythonic in the sense that there should be only one way to do things right (and multiple somewhat convoluted ways to work around problems).

[LourensVeen]

Hmm. Well, I started off using requirements.txt, now I mostly use setup.py with a separate dev section for things like pytest, but I still don't have a clue what I'm doing. Reading those references makes things a bit clearer, but not completely.

One thing I've recently taken up and which I really like is to have the dependencies open-ended on the develop branch, and nailing them down to specific versions in releases. Then I have requires.io check everything for being outdated. That has the advantage that any incompatible library updates will show up as failed Travis tests on the develop branch, while they won't affect released versions. If a library has a security update I will get a warning from requires.io, and then I can test it, update the dependency to the newer version, and make a new release.

I do that now by changing the setup.py in a release branch (I use git flow, not Github flow), but perhaps the trick would be to have open-ended settings in setup.py, and then have a requirements.txt with fixed dependency versions only in releases. Then have some Travis trickery to use requirements.txt if it exists (for release branches), and fall back to setup.py if it doesn't (for the develop branch).

[sverhoeven]

You need to make a distinction between libraries, which you install with pip install ..., and applications or a library dev environment, which you normally install with a git clone + pip install -r requirements.txt.

I am switching from requirements.txt+virtualenv to pipenv see https://docs.pipenv.org/advanced/#pipfile-vs-setuppy

[LourensVeen]

We were using pipenv in the MDStudio software for the protein-binding project, and our lives are a lot easier now that we've removed it again. It's still pretty new, and apparently not able to deal with complex dependencies well. Maybe @felipeZ would like to comment?

[bouweandela]

I feel that the first link provided by @jvdzwaan already explains the best practice, so I don't we why we should start inventing our own. Summarizing what it says there, roughly this is the best practice:

• use setup.py to specify the names of the direct dependencies, pin as little versions as possible
• use requirements.txt to specify a full environment in which your software is known to work, so with all versions pinned and dependencies of dependencies also listed.

[jspaaks]

Note that "pin as little versions as possible" assumes that semver is a system that works. I can point you to a few examples where people who are professional software developers, who have a broad support network of very knowledgable people, and who really tried to get their semantic versioning right, still MESS IT UP.

[LourensVeen]

Yep, I've had things fail on API changes in minor version number updates as well. But that's why we have a requirements.txt with fixed dependencies as well; if you're just using our software rather than developing it, that's what you'd use, and then it's guaranteed to work.

[bouweandela]

Poetry looks like a nice tool too, we could probably mention it. Anyone experience using this?

[egpbos]

I came across this issue again in my project just now. Reading up on current sources, e.g.:

https://packaging.python.org/guides/tool-recommendations/
https://packaging.python.org/tutorials/managing-dependencies/
http://andrewsforge.com/article/python-new-package-landscape/
https://snarky.ca/a-tutorial-on-python-package-building/

and generally Googling around, it seems like the packaging field is still in disarray under development. Even the "official" recommendations offer many options. The Managing Dependencies tutorial even has a big caveat section saying Pipenv may be in its death throes. Nobody seems to be able to predict what comes next. Maybe Poetry, maybe Flit, maybe Pipenv will recover and regain the backing of the "official" crowd, who knows.

Given this situation, I would not recommend any modern tool, except the really well established options of setup.py and requirements.txt.

I think we can sum up our advice on using those as follows (combining what was said above):

• The primary place to specify dependencies should be setup.py's install_requires. Keep version constraints to a minimum; use, in order of descending preference: lower bounds, lower + upper bounds, exact versions.
• If (e.g. for release/production installation) you want to specify a full list of exact dependency versions, possibly even including a specific package repository URL, use requirements.txt. Otherwise omit it.

To be honest, I'm not really sure we should even include the second point. It really depends on the project what kind of exact/concrete dependencies you need. Probably many people nowadays would rather use Docker for instance, to "really" fix all dependencies, or maybe conda, which can also pin some of the non-Python dependencies. It feels to me like just one possible way to solve this problem and in that case I don't know why we should give it a prominent mention as "the" best practice. To make things worse, at the other end of the scale, there will be many packages where no distinction between abstract and concrete is required at all. Personally, I rarely use requirements.txt at all nowadays, except maybe for readthedocs or MyBinder, i.e. automated external services that require such a file specifically, for some reason. Otherwise, I just go with setup.py.

Ok, so to sum up: I would vote for recommending setup.py as the primary place to list dependencies. We should still mention requirements.txt, but I think we should actually maybe be a bit rebellious and discourage its use, unless [blabla see above story].

I could make a PR for this if people agree.

[bouweandela]

I agree with recommending setup.py unless you have a good reason not to. Another reason to use requirements.txt would be if you're not building a package, but e.g. just a bunch of jupyter notebooks.

Probably many people nowadays would rather use Docker for instance, to "really" fix all dependencies, or maybe conda, which can also pin some of the non-Python dependencies.

Both of these solutions have problems 'really' fixing dependencies. It's easy and not uncommon for packages to be removed from conda and docker has this thing where it will automatically use newer versions of container layers that you're building upon.

[egpbos]

You're right, those aren't perfect solutions either. So the only real solid option if you really want to permanently fix your dependencies is just to package everything together, I guess :)

[LourensVeen]

Of course, that will make your package more difficult to use, because now it's incompatible with many other packages which require a slightly different version of a dependency. I think in general fixing dependencies really only works if it's an application and you're the only user of the resulting system. That's what you do with Docker, you build an image, deploy it, test it, and if it works put it into production. When it's time to upgrade, you find a new set of compatible dependencies, build a new image, test it, and put it into production. If you want to share it, you push the image to DockerHub, and then other people can download and deploy it and get something that works. But that doesn't work for a library package.

Since I mostly make libraries, I use a mix. I use setup.py, leave open what seems to work, add version constraints to work around issues with certain versions (I recently automatically tested a whole range of versions of a dependency to see what did and didn't work, then used that to write my constraints), and then test in a bunch of different environments (all new-enough supported versions of Ubuntu is a good start, or particular Python versions that users have reported problems with in the past) and fix or work around anything that comes up. The main remaining issue is pip's terrible dependency conflict resolution...

[egpbos]

In accordance with these new guidelines (rewriting the chapter right now), we should also remove requirements.txt from the python-template.

[sverhoeven]

Yep, the dev requirements can be installed with pip install -e .[dev], though it is undocumented.

[egpbos]

I think [dev] etc will soon be deprecated in setuptools, they plan on simplifying the tool to make it more maintainable.