Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is uv pip sync with --find-links broken? #2688

Closed
gwdekker opened this issue Mar 27, 2024 · 8 comments · Fixed by #2692
Closed

is uv pip sync with --find-links broken? #2688

gwdekker opened this issue Mar 27, 2024 · 8 comments · Fixed by #2692
Assignees
Labels
bug Something isn't working

Comments

@gwdekker
Copy link

uv version: 0.1.24, running on MacOS.

In my use case, I want to install dependencies, some of them living as .whl files in a local directory. When running `uv pip sync ... -find-links LOCAL_PATH the whls were not found. I made a small example that reproduces the case for me, with disabled network lookup to simplify the case

Case 1: uv pip install seems to work correctly
uv -v pip install MY_PACKAGE --find-links /Users/OTHER_STUFF/libraries/.wheels --no-index
result: based on the logs, it clearly finds the package (and only starts failing later, as this package has external dependencies which it is now blocked to fetch due to no-index flag). Snippet from the log:

DEBUG Found 4 packages in `--find-links` entry: /Users/OTHER_STUFF/libraries/.wheels
DEBUG Solving with target Python version 3.11.4
DEBUG Adding direct dependency: MY_PACKAGE*
DEBUG Searching for a compatible version of MY_PACKAGE (*)
DEBUG Selecting: MY_PACKAGE==0.0.1 (MY_PACKAGE-0.0.1-py3-none-any.whl)
DEBUG Adding transitive dependency: pytest*
...

Case 2: uv pip sync seems to fail
uv -v pip sync requirements-dev2.txt --find-links /Users/OTHER_STUFF/libraries/.wheels --no-index

content of the requirements file:
MY_PACKAGE==0.0.1

Snippet from the log:

INFO Found a virtualenv through VIRTUAL_ENV at: /Users/SOME_PATH
DEBUG Cached interpreter info for Python 3.11.4, skipping probing: .venv/bin/python
DEBUG Using Python 3.11.4 environment at .venv/bin/python
DEBUG Using registry request timeout of 300s
DEBUG Found 4 packages in `--find-links` entry: /Users/OTHER_STUFF/libraries/.wheels
DEBUG Identified uncached requirement: MY_PACKAGE ==0.0.1
DEBUG Unnecessary package: ruff==0.1.13
error: MY_PACKAGE isn't available locally, but making network requests to registries was banned.

Note: there actually is a whitespace in DEBUG Identified uncached requirement: MY_PACKAGE ==0.0.1. Could this be a pointer to what is wrong?

I also tried pip-sync for case 2 instead of uv pip sync, which also seems to work.

@charliermarsh charliermarsh added the needs-mre Needs more information for reproduction label Mar 27, 2024
@charliermarsh
Copy link
Member

I feel like the space probably is indicative of what's going wrong, but I don't see how that could be happening. Can you upload the exact requirements file?

@gwdekker
Copy link
Author

mre.zip

I made you a small MRE that replicates the issue on my side, see the README.md for instructions.

@charliermarsh charliermarsh self-assigned this Mar 27, 2024
@charliermarsh
Copy link
Member

Thank you!

@charliermarsh
Copy link
Member

Okay, running on main, I do see error: lib1 isn't available locally, but making network requests to registries was banned, but I no longer see the space in lib1==0.0.1, so just gonna ignore that for now.

@charliermarsh
Copy link
Member

charliermarsh commented Mar 27, 2024

By the way, removing --no-index does succeed (but it should still succeed without it).

@charliermarsh charliermarsh added bug Something isn't working and removed needs-mre Needs more information for reproduction labels Mar 27, 2024
charliermarsh added a commit that referenced this issue Mar 27, 2024
## Summary

In `pip sync`, we weren't properly handling cases in which a package
_only_ existed in `--find-links` (e.g., the user passed `--offline` or
`--no-index`).

I plan to explore removing `Finder` entirely to avoid these mismatch
bugs between `pip sync` and other commands, but this is fine for now.

Closes #2688.

## Test Plan

`cargo test`
@charliermarsh
Copy link
Member

Thanks for the great MRE! Fixed in the next release.

@gwdekker
Copy link
Author

Thanks a lot! Just a quick check: does it work without —no-index because it finds the local library, or is there a lib1 on PyPI mirroring the local library and is the problem therefore masked?

@charliermarsh
Copy link
Member

It finds both (like pip), but it correctly selects the local library since you pinned the version (in my testing).

charliermarsh added a commit that referenced this issue Mar 27, 2024
## Summary

This PR removes the custom `DistFinder` that we use in `pip sync`. This
originally existed because `VersionMap` wasn't lazy, and so we saved a
lot of time in `DistFinder` by reading distribution data lazily. But
now, AFAICT, there's really no benefit. Maintaining `DistFinder` means
we effectively have to maintain two resolvers, and end up fixing bugs in
`DistFinder` that don't exist in the `Resolver` (like #2688.

Closes #2694.

Closes #2443.

## Test Plan

I ran this benchmark a bunch. It's basically a wash. Sometimes one is
faster than the other.

```
❯ python -m scripts.bench \
        --uv-path ./target/release/main \
        --uv-path ./target/release/uv \
        scripts/requirements/compiled/trio.txt --min-runs 50 --benchmark install-warm --warmup 25
Benchmark 1: ./target/release/main (install-warm)
  Time (mean ± σ):      54.0 ms ±  10.6 ms    [User: 8.7 ms, System: 98.1 ms]
  Range (min … max):    45.5 ms …  94.3 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./target/release/uv (install-warm)
  Time (mean ± σ):      50.7 ms ±   9.2 ms    [User: 8.7 ms, System: 98.6 ms]
  Range (min … max):    44.0 ms …  98.6 ms    50 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (77.6 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Summary
  './target/release/uv (install-warm)' ran
    1.06 ± 0.29 times faster than './target/release/main (install-warm)'
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants