Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pip's 2020 dependency resolver #1539

Merged
merged 6 commits into from
Jun 29, 2022

Conversation

atugushev
Copy link
Member

@atugushev atugushev commented Nov 22, 2021

What's new?

Added new option --resolver [backtracking|legacy] to pip-compile (default is legacy).

How to use?

To enable 2020 dependency resolver run pip-compile --resolver=backtracking.

Backtracking resolver example

$ echo "oslo.utils==1.4.0" | pip-compile - --resolver=backtracking --allow-unsafe --annotation-style=line -qo-
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --allow-unsafe --annotation-style=line --output-file=- --resolver=backtracking -
#
babel==2.10.3             # via oslo-i18n, oslo-utils
iso8601==1.0.2            # via oslo-utils
netaddr==0.8.0            # via oslo-utils
netifaces==0.11.0         # via oslo-utils
oslo-i18n==2.1.0          # via oslo-utils
oslo-utils==1.4.0         # via -r -
pbr==0.11.1               # via oslo-i18n, oslo-utils
pytz==2022.1              # via babel
six==1.16.0               # via oslo-i18n, oslo-utils

# The following packages are considered to be unsafe in a requirements file:
pip==22.1.2               # via pbr

Legacy resolver example

$ echo "oslo.utils==1.4.0" | pip-compile - --resolver=legacy --allow-unsafe -qo-
Could not find a version that matches pbr!=0.7,!=2.1.0,<1.0,>=0.6,>=2.0.0 (from oslo.utils==1.4.0->-r -)
Tried: 0.5.2.5.g5b3e942, 0.5.0, 0.5.1, 0.5.2, 0.5.4, 0.5.5, 0.5.6, 0.5.7, 0.5.8, 0.5.10, 0.5.11, 0.5.12, 0.5.13, 0.5.14, 0.5.15, 0.5.16, 0.5.17, 0.5.18, 0.5.19, 0.5.20, 0.5.21, 0.5.22, 0.5.23, 0.6, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.0, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.10.2, 0.10.2, 0.10.3, 0.10.3, 0.10.4, 0.10.4, 0.10.5, 0.10.5, 0.10.6, 0.10.6, 0.10.7, 0.10.7, 0.10.8, 0.10.8, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.9.1, 1.9.1, 1.10.0, 1.10.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.0.2, 4.0.2, 4.0.3, 4.0.3, 4.0.4, 4.0.4, 4.1.0, 4.1.0, 4.1.1, 4.1.1, 4.2.0, 4.2.0, 4.3.0, 4.3.0, 5.0.0, 5.0.0, 5.1.0, 5.1.0, 5.1.1, 5.1.1, 5.1.2, 5.1.2, 5.1.3, 5.1.3, 5.2.0, 5.2.0, 5.2.1, 5.2.1, 5.3.0, 5.3.0, 5.3.1, 5.3.1, 5.4.0, 5.4.0, 5.4.1, 5.4.1, 5.4.2, 5.4.2, 5.4.3, 5.4.3, 5.4.4, 5.4.4, 5.4.5, 5.4.5, 5.5.0, 5.5.0, 5.5.1, 5.5.1, 5.6.0, 5.6.0, 5.7.0, 5.7.0, 5.8.0, 5.8.0
There are incompatible versions in the resolved dependencies:
  pbr!=0.7,<1.0,>=0.6 (from oslo.utils==1.4.0->-r -)
  pbr!=2.1.0,>=2.0.0 (from oslo.i18n==5.1.0->oslo.utils==1.4.0->-r -)
Contributor checklist
  • Provided the tests for the changes.
  • Assure PR title is short, clear, and good to be included in the user-oriented changelog
Maintainer checklist
  • Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
  • Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

@astrojuanlu
Copy link
Contributor

I confirm that this fixes my original use case in #1277 🚀

(.venv) juanlu at arion2 in /tmp/reqs
$ echo 'jupyterlab==3.0.0rc13' > requirements.in
(.venv) juanlu at arion2 in /tmp/reqs
$ pip-compile
Could not find a version that matches jupyter-server~=1.0.1,~=1.1,~=1.5 (from jupyterlab==3.0.0rc13->-r requirements.in (line 1))
Tried: 0.0.1, 0.0.1, 0.0.2, 0.0.2, 0.0.3, 0.0.3, 0.0.4, 0.0.4, 0.0.5, 0.0.5, 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.2.0, 0.2.0, 0.2.1, 0.2.1, 0.3.0, 0.3.0, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.0.5, 1.0.5, 1.0.6, 1.0.6, 1.0.7, 1.0.7, 1.0.8, 1.0.8, 1.0.9, 1.0.9, 1.0.10, 1.0.10, 1.0.11, 1.0.11, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.1.3, 1.1.3, 1.1.4, 1.1.4, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.2.2, 1.2.2, 1.2.3, 1.2.3, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.11.0, 1.11.0, 1.11.1, 1.11.1, 1.11.2, 1.11.2
Skipped pre-versions: 1.0.0rc0, 1.0.0rc0, 1.0.0rc1, 1.0.0rc1, 1.0.0rc2, 1.0.0rc2, 1.0.0rc3, 1.0.0rc3, 1.0.0rc4, 1.0.0rc4, 1.0.0rc5, 1.0.0rc5, 1.0.0rc6, 1.0.0rc6, 1.0.0rc7, 1.0.0rc7, 1.0.0rc8, 1.0.0rc8, 1.0.0rc9, 1.0.0rc9, 1.0.0rc10, 1.0.0rc10, 1.0.0rc11, 1.0.0rc11, 1.0.0rc12, 1.0.0rc12, 1.0.0rc13, 1.0.0rc13, 1.0.0rc14, 1.0.0rc14, 1.0.0rc15, 1.0.0rc15, 1.0.0rc16, 1.0.0rc16, 1.7.0a1, 1.7.0a1, 1.7.0a2, 1.7.0a2
There are incompatible versions in the resolved dependencies:
  jupyter-server~=1.0.1 (from jupyterlab==3.0.0rc13->-r requirements.in (line 1))
  jupyter-server~=1.1 (from jupyterlab-server==2.0.0->jupyterlab==3.0.0rc13->-r requirements.in (line 1))
  jupyter-server~=1.5 (from nbclassic==0.2.8->jupyterlab==3.0.0rc13->-r requirements.in (line 1))
(.venv) juanlu at arion2 in /tmp/reqs
$ pip-compile --resolver=2020
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile --resolver=2020
#
argon2-cffi==21.1.0
    # via notebook
attrs==21.2.0
    # via jsonschema
babel==2.9.1
    # via jupyterlab-server
backcall==0.2.0
    # via ipython
bleach==4.1.0
    # via nbconvert
certifi==2021.10.8
    # via requests
cffi==1.15.0
    # via argon2-cffi
charset-normalizer==2.0.7
    # via requests
debugpy==1.5.1
    # via ipykernel
decorator==5.1.0
    # via ipython
defusedxml==0.7.1
    # via nbconvert
entrypoints==0.3
    # via
    #   jupyter-client
    #   nbconvert
idna==3.3
    # via requests
ipykernel==6.5.1
    # via notebook
ipython==7.29.0
    # via
    #   ipykernel
    #   jupyterlab
ipython-genutils==0.2.0
    # via
    #   jupyter-server
    #   nbformat
    #   notebook
jedi==0.18.1
    # via ipython
jinja2==3.0.3
    # via
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
    #   notebook
json5==0.9.6
    # via jupyterlab-server
jsonschema==4.2.1
    # via
    #   jupyterlab-server
    #   nbformat
jupyter-client==7.0.6
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
    #   notebook
jupyter-core==4.9.1
    # via
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   nbconvert
    #   nbformat
    #   notebook
jupyter-server==1.0.11
    # via
    #   jupyterlab
    #   jupyterlab-server
    #   nbclassic
jupyterlab==3.0.0rc13
    # via -r requirements.in
jupyterlab-pygments==0.1.2
    # via nbconvert
jupyterlab-server==2.0.0rc7
    # via jupyterlab
markupsafe==2.0.1
    # via jinja2
matplotlib-inline==0.1.3
    # via
    #   ipykernel
    #   ipython
mistune==0.8.4
    # via nbconvert
nbclassic==0.2.4
    # via jupyterlab
nbclient==0.5.9
    # via nbconvert
nbconvert==6.3.0
    # via
    #   jupyter-server
    #   notebook
nbformat==5.1.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
    #   notebook
nest-asyncio==1.5.1
    # via
    #   jupyter-client
    #   nbclient
    #   notebook
notebook==6.4.6
    # via nbclassic
packaging==21.3
    # via
    #   bleach
    #   jupyterlab
    #   jupyterlab-server
pandocfilters==1.5.0
    # via nbconvert
parso==0.8.2
    # via jedi
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
prometheus-client==0.12.0
    # via
    #   jupyter-server
    #   notebook
prompt-toolkit==3.0.22
    # via ipython
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pycparser==2.21
    # via cffi
pygments==2.10.0
    # via
    #   ipython
    #   jupyterlab-pygments
    #   nbconvert
pyparsing==3.0.6
    # via packaging
pyrsistent==0.18.0
    # via jsonschema
python-dateutil==2.8.2
    # via jupyter-client
pytz==2021.3
    # via babel
pyzmq==22.3.0
    # via
    #   jupyter-client
    #   jupyter-server
    #   notebook
requests==2.26.0
    # via jupyterlab-server
send2trash==1.8.0
    # via
    #   jupyter-server
    #   notebook
six==1.16.0
    # via
    #   bleach
    #   python-dateutil
terminado==0.12.1
    # via
    #   jupyter-server
    #   notebook
testpath==0.5.0
    # via nbconvert
tornado==6.1
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   notebook
    #   terminado
traitlets==5.1.1
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-server
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
    #   notebook
urllib3==1.26.7
    # via requests
wcwidth==0.2.5
    # via prompt-toolkit
webencodings==0.5.1
    # via bleach

# The following packages are considered to be unsafe in a requirements file:
# setuptools

@FlorentJeannot
Copy link
Contributor

Hello @atugushev

Thank you for pinging me to test your branch, and thank you for taking the time to work on this feature!

I tried it and I noted some remarks:

  1. ℹ️ From now on, -c requirements.txt does not work if the requirements.txt contains for example a direct reference. It requires a package name with a constraint. This is not related to your PR, but to how pip works. I thought it was interesting to mention it.

  2. ❓ The listing of dependencies in the generated .txt file is a bit weird, for example:

certifi==2021.10.8
    # via
    #   requests
    #   requests[socks]

I don't know if we want to output the same package name twice?

  1. ⚠️ The generated .txt file does not output the same dependencies depending on the order in the .in file.

For example:

crochet==2.0.0
Scrapy==2.5.1

Will output:

[...]
twisted==21.7.0
    # via
    #   crochet
    #   twisted[http2]
[...]

While:

Scrapy==2.5.1
crochet==2.0.0

Will output:

[...]
 twisted[http2]==21.7.0
    # via
    #   crochet
    #   scrapy
    #   twisted[http2]
[...]

@atugushev
Copy link
Member Author

@FlorentJeannot thanks for the feedback and good catch! Looks like we need more tests for packages with extras. I'll address the bugs shortly.

@FlorentJeannot
Copy link
Contributor

FlorentJeannot commented Nov 22, 2021

Thank you @atugushev. I found another one, not sure if it's related to your changes though.

When I declare a local file which contains an egg (for example file:./vendored/package.zip[requests]), the output of the .txt is the absolute path of the file on my machine.

Let me know if you need an exemple and thank you.
Also let me know when you have made some fixes, so that I can test it again and let you know if I spot other things.

@webknjaz
Copy link
Member

@FlorentJeannot if you could send a PR with a failing test, it'd be easier to come up with a fix separately, once it's committed in Git. Per https://pganssle-talks.github.io/xfail-lightning/.

@AndydeCleyre
Copy link
Contributor

When I declare a local file which contains an egg (for example file:./vendored/package.zip[requests]), the output of the .txt is the absolute path of the file on my machine.

@FlorentJeannot I don't know if @atugushev wants to merge this before or after #1329

piptools/resolver.py Outdated Show resolved Hide resolved
piptools/resolver.py Outdated Show resolved Hide resolved
piptools/resolver.py Outdated Show resolved Hide resolved
piptools/resolver.py Outdated Show resolved Hide resolved
piptools/resolver.py Outdated Show resolved Hide resolved
PIP_EXISTS_ACTION="i"
):
# Mark direct/primary/user_supplied packages
for ireq in self.constraints:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why not apply this conversion to the constraints list during the initialization. Also, I don't really understand why this needs to be mutable. Is that expectation that it'll be modified by external callers post-initialization?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why not apply this conversion to the constraints list during the initialization.

Do you mean inside Resolver.__init__()?

piptools/utils.py Outdated Show resolved Hide resolved
piptools/writer.py Outdated Show resolved Hide resolved
@AndydeCleyre
Copy link
Contributor

I wonder if the varying extras behavior noted by @FlorentJeannot is resolved by @richafrank's sitting PRs.

@atugushev Do you think we'll be finalizing this one before any other PRs, or are there some you plan to review and merge first?

tests/test_cli_compile.py Outdated Show resolved Hide resolved
atugushev and others added 6 commits June 29, 2022 23:29
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
It shouldn't raise an error when a preexisting output file's
pins conflict with the input files' constraints.

jazzband#1539 (comment)
Co-authored-by: Richard Frank <rich@quantopian.com>
@atugushev atugushev merged commit bf0a669 into jazzband:master Jun 29, 2022
@atugushev
Copy link
Member Author

I'm so happy this finally merged. Many thanks to my heroes @AndydeCleyre, @FlorentJeannot, @richafrank, @ssbarnea, @thomas-riccardi, @webknjaz, and all others involved for your support.

@gschaffner

This comment was marked as resolved.

@atugushev

This comment was marked as resolved.

@deifactor
Copy link

deifactor commented Aug 4, 2022

One thing I noticed is that this is significantly slower than the legacy resolver, presumably because it's not using the pip-compile cache. What's worse is that for things that don't have pypi-built wheels for the target host, it has to rebuild the wheels every time, which is especially painful when doing things involving cython.

@ssbarnea
Copy link
Member

ssbarnea commented Aug 4, 2022

@deifactor It is not a bug, just an opportunity for you to make a pull-request and address this performance regression! ;)

@astrojuanlu
Copy link
Contributor

Worth opening an issue for wider visibility?

JiachenSmith added a commit to JiachenSmith/pip-tools that referenced this pull request Mar 21, 2023
It shouldn't raise an error when a preexisting output file's
pins conflict with the input files' constraints.

jazzband/pip-tools#1539 (comment)
@taeefnajib
Copy link

Hi @atugushev I have a question! I need both flytekit and bentoml installed in the same project. protobuf = ">=4.21.1,<5.0.0" is required by flytekit and protobuf<4.0 is required by bentoml. When I try pip-compile requirements.in I get an error. But after using your solution, i.e. pip-compile requirements.in --resolver=backtracking pip-compile generates requirements.txt file. But the version of protobuf is 3.20.3 How can I have both versions of protobuf in my requirements.txt file?

@ryanhiebert
Copy link
Contributor

ryanhiebert commented Apr 27, 2023

@taeefnajib Python doesn't allow you to install two different versions of the same dependency in the same environment, so we can't support two versions of a package in a requirements.txt file. You'll need to find a set of dependencies that all work with the same version of protobuf correctly. This isn't always easy, but it is necessary.

@taeefnajib
Copy link

@ryanhiebert thanks. let me see why bentoml locked the version of protobuf to <4.0

@taeefnajib
Copy link

@ryanhiebert Just a quick question! When I put bentoml and flytekit in my requirements.txt file manually (without mentioning versions), and use pip install -r requirements.txt it installs multiple versions of protobuf It's pip-compile that cannot generate a requirements.txt file with multiple versions. Do you know why is that happening and if I can find a workaround?

@AndydeCleyre
Copy link
Contributor

@taeefnajib it's probably installing one version, then uninstalling that to install a different version. Run pip list to see the final state.

@taeefnajib
Copy link

taeefnajib commented Apr 27, 2023

@AndydeCleyre you are right. I can only see 3.20.3 Thanks for the clarification

@luhn
Copy link

luhn commented Apr 27, 2023

It looks like pip will install flytekit 1.2.11 (latest release is 1.5.0), which requires protobuf>=3.6.1,<4, in order to satisfy all dependencies.

>> echo "bentoml" >> req.txt
>> echo "flytekit" >> req.txt
>> pip install -qr req.txt
>> pip list | grep protobuf
protobuf                                     3.20.3
>> pip list | grep flytekit
flytekit                                     1.2.11

@taeefnajib
Copy link

@luhn thanks. Yes, it's installing 1.2.11 even though I was hoping to install 1.5.0

@atugushev atugushev deleted the new-resolver branch July 29, 2023 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvements to functionality resolver Related to dependency resolver
Projects
None yet
Development

Successfully merging this pull request may close these issues.