pip-compile absolute links when given relative path

[ColtonProvias]

requirements.in:

-e .
-e ./sqlalchemy_jsonapi

compiles to:

-e file:///Users/coltonprovias/MyProject
-e file:///Users/coltonprovias/MyProject/sqlalchemy_jsonapi

It should compile to

-e .
-e ./sqlalchemy_jsonapi

It's not really useful for me to provide a requirements.txt file then to anybody unless they use a Mac and have the same name as me. Although if that person does exist, I would like to meet them.

[cdwilson]

I ran into the same issue today. Why are relative paths are expanded?

[michaeljones]

This has affected me too as I'm trying to install packages which are submodules of my project. It seems the issues arises here: https://github.com/pypa/pip/blob/develop/pip/req/req_install.py#L1131

I suspect it is hard to avoid as pip-tools uses the pip library and the pip library has no reason to keep a relative path around. It wants to deal with absolute paths.

Which means that it is unlikely that we'd get the pip library changed to help us so which suggests that we'd have to get the output from the pip library and go back and make any paths relative again which is an option but not incredibly desirable as it might be hard to know exactly when to do it and when not to.

Personally, I'm going to try switching from using submodules and relative paths to having git+git:// paths in my requirements.in file but at the same time it seems very reasonable to try to fix this as it makes little sense to have absolute paths in a requirements.txt file when there is little chance that the paths in different development environments or the production environment is going to be the same. That is certainly what affects me. Of course, the requirements.txt should also be in git so it should be possible to pip-compile and then edit the requirements.txt before committing it.

[HelloGrayson]

Looks like #325 is related

[jredwards]

I've been planning to move all of the requirements handling across all of our repos to using pip-tools and just realized it had this issue, which may completely shelve that plan. This is a pretty significant problem.

[orf]

So the core of this comes down to this function here, inside pip. I don't think it would be easy to fix in pip itself, so I will see if I can manipulate the result while outputting it to the file by making the result relative.

[ztane]

the relative paths should actually use file:foo/bar so that the #egg=spam fragment could be appended. Unfortunately this breaks in pip-compile, as pip-compile tries reading foo/bar/setup.py after setting foo/bar as the current directory!

[ztane]

as for why #egg=spam would be needed, is for pip dependency resolver to find out what abstract requirement is provided by the said file link! Otherwise it is going to attempt to download spam from Warehouse!

[rollcat]

If your parrot is a git submodule, this works well: -e git+path/to/dead/parrot#egg=parrot.

[ztane]

@rollcat almost works... except that we merged 2 repos so it is a slug and hardly speaks.

[ssanderson]

I took another stab at trying to fix this in #702.

[tillahoffmann]

I've been using the following post-processing step to get around the problem: make_paths_relative.py < requirements.txt (script below)

#!/usr/bin/env python
import os
import re
import sys

pattern = re.compile(r"\s*-e\s+file://(?P<path>.*?)\n")
inputs = sys.stdin.readlines()
for line in inputs:
    match = pattern.match(line)
    if match:
        path = match.group("path")
        path = os.path.relpath(path)
        line = f"-e {path}\n"
    sys.stdout.write(line)

[leontrolski]

A(nother) hacky solution, in requirements.in:

-e file:relative/path/to/my_package

then to compile:

pip-compile -q -o - | sed s:///::g > requirements.txt

[mmerickel]

Ran into this issue as well. If I modify all my -e src/foo links to -e file:src/foo#egg=foo then the compiled requirements file drops out the same -e file:src/foo#egg=foo and this works as expected. Quite a lot more verbose than necessary though. The egg on the end is required, or the sed script above in #204 (comment) otherwise the later pip install -r will not properly find the "foo" project to satisfy the dependency lists of other projects depending on "foo".

[AndydeCleyre]

I'm sorry say that #1650 has been rejected, and I encourage anyone interested in the problem to think about alternative approaches.

[henridwyer]

The workaround -e file:local/path/to/lib#egg=lib mentioned by @AndydeCleyre here works when using requirements.in (and will get copied to the final requirements.txt), but is there a working syntax if you want to use setup.cfg or pyproject.toml to define the requirements?

[sfdye]

The -e file:src/foo#egg=foo work-around does not seem to work with extra requires, e.g. -e file:src/foo[extra]#egg=foo

pip._internal.exceptions.InstallationError: File "setup.py" not found for legacy project foo from file:///src/foo[extra]#egg=foo

[AndydeCleyre]

I'll note here that, without much guarantee, as long as it's pretty simple to do, I'll try to keep my relpaths branch updated. You can use pipx to invoke it in a one-off fashion, like

$ pipx run --spec 'git+https://github.com/AndydeCleyre/pip-tools@feature/relpaths-post-6.8.0' pip-compile

Doing so, you should be able to use the simple syntax:

-e ./src/foo[extra]

[FeryET]

The workaround -e file:local/path/to/lib#egg=lib mentioned by @AndydeCleyre here works when using requirements.in (and will get copied to the final requirements.txt), but is there a working syntax if you want to use setup.cfg or pyproject.toml to define the requirements?

Hi! Thanks for this great tool. I'm stuck on using pip-tools with pyproject.toml since I have extras that include other extras, wanted to see if there are any updates on this.

[AndydeCleyre]

Do any pyproject.toml build backends support relative paths for dependencies? I don't think Poetry or Flit do.

If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.

[FeryET]

Do any pyproject.toml build backends support relative paths for dependencies? I don't think Poetry or Flit do.

If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.

The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.

Example:

# package_name is the name of the package itself
[project.optional-dependencies]
dev = ["package_name[test,tools]"]
tools = ["pip-tools"]
test = ["pytest""]

The generated pip-compile for dev extra will be something like this:

package_name @ file:///path/to/package/dir
pip-tools
pytest

Which is nonsensical to be honset.

[AndydeCleyre]

It looks like that's as intended based on discussion at #1685, and maybe @q0w or @atugushev can speak to the rationale.

I personally don't like specifying a package as depending on itself, because that seems like an impossible infinite loop, but I now know it's commonly done. Instead I maintain dependency groups as *-requirements.in files, and script injection of that data into the pyproject.toml.

So that example would be managed like:

test-requirements.in:

pytest

tools-requirements.in:

pip-tools

dev-requirements.in:

-r test-requirements.in
-c test-requirements.txt

-r tools-requirements.in
-c tools-requirements.txt

$ pypc  # inject deps according to above files into pyproject.toml, part of zpy, my wrapper of pip-tools
$ grep -A 3 optional-dependencies pyproject.toml

[project.optional-dependencies]
dev = ["pip-tools", "pytest"]
tools = ["pip-tools"]
test = ["pytest"]

[FeryET]

It looks like that's as intended based on discussion at #1685, and maybe @q0w or @atugushev can speak to the rationale.

I personally don't like specifying a package as depending on itself, because that seems like an impossible infinite loop, but I now know it's commonly done. Instead I maintain dependency groups as *-requirements.in files, and script injection of that data into the pyproject.toml.

So that example would be managed like:

test-requirements.in:

pytest

tools-requirements.in:

pip-tools

dev-requirements.in:

-r test-requirements.in
-c test-requirements.txt

-r tools-requirements.in
-c tools-requirements.txt

$ pypc  # inject deps according to above files into pyproject.toml, part of zpy, my wrapper of pip-tools
$ grep -A 3 optional-dependencies pyproject.toml

[project.optional-dependencies]
dev = ["pip-tools", "pytest"]
tools = ["pip-tools"]
test = ["pytest"]

I was using a mix of requirements.in files and pyproject.toml + dynamic dependencies of setuptools to aleviate this issue, but setuptools does not support -c or -r flags in the dependency declaration files. I don't know what can be the best option here, the pypc package seems ancient and I would probably use a sed or awk based bash script to rewrite these lines, but that seems infinitely hacky anyway.

I was thinking of migrating to setup.cfg and using the %option% syntax.

[AndydeCleyre]

To be clear, the pypc here has nothing to do with the package of that name on PyPI, but is part of my project zpy (here on GitHub). You can check that implementation, but basically you can use tomlkit to inject the data without being hacky.

[tboddyspargo]

I just want to reiterate for all the watchers that while we're still waiting on path syntaxes to settle upstream, much of the time, for now, you'll get what you want by using file:./relpath, such as -e file:..

@AndydeCleyre - I'm super grateful for all the time, energy, and patience you've invested into this issue. My team would certainly still love to see this get supported at some point, but we also understand the challenges involved and the resistance to the changes at this point in time.

QUESTION: My question is related to the work-around. I was delighted to see that using file:./package-name in my requirements.in file worked (path remained relative in requirements.txt), but unfortunately, using file:../package-name did not (path became absolute, presumably due to the parent directory reference: ..). Confusingly, using -e file:../package-name did result in a relative in the lockfile. Are you aware of a work-around that avoids --editable but still allows the resulting lockfile to use relative paths when it contains ..?

FWIW, my goal is to have a lockfile that I can use locally during development within a monorepo, using relative paths to express dependencies on packages elsewhere in the monorepo. I also want the same lockfile to be used in a multi-stage Dockerfile where we copy a virtual environment between stages, but nothing else. Since --editable installs don't actually keep the source code in the virtual environment, I don't think I can use them and still have a "portable" venv...

Any input or advice would be greatly appreciated!

[AndydeCleyre]

@tboddyspargo

I understand if this is not a suitable solution for your needs, but my feature/relpaths-post-6.8.0 branch should indeed work if you specify e.g. ../package-name or file:../package-name.

It's been a while since I was testing out all the combinations of input and output observed with the main branch and don't know if there's a certain syntax that works there, but I wonder if a workaround for you might instead be found in using relpath symlinks to the packages you need.

$ ln -s ../package-name
$ echo 'file:./package-name' >>requirements.in

[tboddyspargo]

I wonder if a workaround for you might instead be found in using relpath symlinks to the packages you need.

$ ln -s ../package-name
$ echo 'file:./package-name' >>requirements.in

Clever! That does seem to work! We'll consider using that as a short-term option. Many thanks!

[q0w]

Do any pyproject.toml build backends support relative paths for dependencies? I don't think Poetry or Flit do.
If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.

The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.

Example:

# package_name is the name of the package itself
[project.optional-dependencies]
dev = ["package_name[test,tools]"]
tools = ["pip-tools"]
test = ["pytest""]

The generated pip-compile for dev extra will be something like this:

package_name @ file:///path/to/package/dir
pip-tools
pytest

Which is nonsensical to be honset.

It's because how pip resolves recursive extras. When you do pip install .[dev] it works only because . converts to LinkCandidate containing your local path. So without @ filepath it would be package from pypi

[FeryET]

Do any pyproject.toml build backends support relative paths for dependencies? I don't think Poetry or Flit do.
If my branch isn't working for certain extras configurations when using requirements.in please let me know, I'll try to reproduce and fix.

The problem is not relative paths for me as it is that extra dependency groups that include other optional dependencies will be compiled to something with absotule paths.
Example:

# package_name is the name of the package itself
[project.optional-dependencies]
dev = ["package_name[test,tools]"]
tools = ["pip-tools"]
test = ["pytest""]

The generated pip-compile for dev extra will be something like this:

package_name @ file:///path/to/package/dir
pip-tools
pytest

Which is nonsensical to be honset.

It's because how pip resolves recursive extras. When you do pip install .[dev] it works only because . converts to LinkCandidate containing your local path. So without @ filepath it would be package from pypi

Using setup.cfg and %(extra)s directives in declaring optional dependencies allievated the issue. Do not know why pyproject.toml does not support that btw.

[options.extras_require]
test = 
   pytest

dev = 
   pip-tools

all =
   %(dev)s
   %(test)s

[q0w]

Did you provide setuptools as build backend in pyproject.toml? Does not work

[allanlewis]

Do any pyproject.toml build backends support relative paths for dependencies? I don't think Poetry or Flit do.

Just to answer this question directly: Poetry does support relative paths for dependencies, I've tried it with a real example in one of my repos.

[stdedos]

I'm coming here from the #325 tl;dr.

In pylint-dev/pylint-pytest@a410e15 (#33) I am seeing that pylint (a dependency of pylint-pytest) is disappearing (and hence my action fails).

I'm assuming that Dependabot does not pick up the two arguments to the pip-compile for some reason.

Trying to use the #325 (comment) as a workaround (pylint-dev/pylint-pytest@b51c82f), then I get https://github.com/pylint-dev/pylint-pytest/actions/runs/7688457033/job/20949696021?pr=33#step:8:23

ERROR: The editable requirement pylint-pytest from file:///.#egg=pylint-pytest (from -r /tmp/tmp985jkgq3 (line 96)) cannot be installed when requiring hashes, because there is no single file to hash.

I want to require hashes (ofc), but it might be hard to get hashes from -e file:.#egg=foobar. The suggestion Consider using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip might be helpful for CI (eventually), but I'd like to solve the local environment problem too.

Any ideas? 😕

[astrojuanlu]

omg so happy to see this fixed ❤️