Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow customizing the project environment path with UV_PROJECT_ENVIRONMENT #6834

Merged
merged 1 commit into from
Sep 3, 2024

Conversation

zanieb
Copy link
Member

@zanieb zanieb commented Aug 29, 2024

Allows configuration of the (currently hard-coded) path to the virtual environment in projects using the UV_PROJECT_ENVIRONMENT environment variable.

If empty, we'll ignore it. If a relative path, it will be resolved relative to the workspace root. If an absolute path, we'll use that.

This feature targets use in Docker images and CI. The variable is intended to be set once in an isolated system and used for all uv operations.

We do not expose a CLI option or configuration file setting — we may pursue those later but I see them as lower priority. I think a system-level environment variable addresses the most pressing use-cases here.

This doesn't special-case the system environment. Which means that you can use this to write to the system Python environment. I would generally strongly recommend against doing so. The insightful comment from @edmorley at #5229 (comment) provides some context on why. More generally, uv sync will remove packages from the environment by default. This means that if the system environment contains any packages relevant to the operation of the system (that are not dependencies of your project), uv sync will break it. I'd only use this in Docker or CI, if anywhere. Virtual environments have lots of benefits, and it's only one line to "activate" them.

If you are considering using this feature to use Docker bind mounts for developing in containers, I would highly recommend reading our Docker container development documentation first. If the solutions there do not work for you, please open an issue describing your use-case and why.

We do not read VIRTUAL_ENV and do not have plans to at this time. Reading VIRTUAL_ENV is high-risk, because users can easily leave an environment active and use the uv project interface today. Reading VIRTUAL_ENV would be a breaking change. Additionally, uv is intentionally moving away from the concept of "active environments" and I don't think syncing to an "active" environment is the right behavior while managing projects. I plan to add a warning if VIRTUAL_ENV is set, to avoid confusion in this area (see #6864).

This does not directly enable centrally managed virtual environments. If you set UV_PROJECT_ENVIRONMENT to an absolute path and use it across multiple projects, they will clobber each other's environments. However, you could use this with something like direnv to achieve "centrally managed" environments. I intend to build a prototype of this eventually. See #1495 for more details on this use-case.

Lots of discussion about this feature in:

Follow-ups:

Closes #6669
Closes #5229
Closes #6612

@zanieb zanieb added the configuration Settings and such label Aug 29, 2024
Comment on lines +79 to +81
// Even if there's a `pyproject.toml`
uv_snapshot!(context.filters(), context.venv().env("UV_PROJECT_ENVIRONMENT", "foo"), @r###"
success: true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will change in #6835

@zanieb
Copy link
Member Author

zanieb commented Aug 29, 2024

I tested installation into a system environment with:

FROM python:3.11.9-slim-bookworm

# Add my locally built binary
ADD uv /bin/uv
RUN chmod +x /bin/uv

# Create a project
RUN uv init example
WORKDIR example

# Use the system Python environment
ENV UV_PROJECT_ENVIRONMENT="/usr/local/"

# Add httpx to the project
RUN uv add httpx

# Import it using the system Python
RUN python -c "import httpx"

@zanieb
Copy link
Member Author

zanieb commented Aug 29, 2024

@edmorley please let me know if this will work for CNBs — I find that use-case particularly compelling.

I also plan to validate this approach works well for multi-stage Docker builds.

@zanieb zanieb marked this pull request as ready for review August 30, 2024 03:24
@hynek
Copy link
Contributor

hynek commented Aug 30, 2024

To clarify, since my Rust is not quite up there: given that uv pip install is not part of the greater "project" topic, it will ignore the variable? To me it's not too bad to run it as uv pip install --python=$UV_PROJECT_ENVIRONMENT but it will 100% confuse people so it maybe should be called out?

@zanieb
Copy link
Member Author

zanieb commented Aug 30, 2024

Yes uv pip install will ignore this. It's not a part of the "project" interface. Maybe it should support it in a project root (as I did for venv), but I'm hesitant.

@inoa-jboliveira
Copy link

inoa-jboliveira commented Aug 30, 2024

Hello! Since it is called UV_PROJECT_ENVIRONMENT (emphasis on project), would it be too much of a stretch to have it be also set in pyproject.toml or uv.toml?

I understand you are focusing on docker where it is standard to set environment variables, but there are much more wide use cases for this.

Thank you for providing this. It is better than the VIRTUAL_ENV variable, but the reason that is an env variable is because of the activation script. Otherwise a config file might be extremely useful

@hynek
Copy link
Contributor

hynek commented Aug 30, 2024

Since it is called UV_PROJECT_ENVIRONMENT (emphasis on project), would it be too much of a stretch to have it be also set in pyproject.toml or uv.toml?

I understand you are focusing on docker where it is standard to set environment variables, but there are much more wide use cases for this.

I'm not sure this makes semantically: if you're cooperating with others on a project, you're dictating on them where to put their venv. If you're working on it yourself, it would be very repetitive and a better solution would be a global config that allows to set some template or something.

Env variables aren't just a Docker thing; people use them locally for exactly these kinds customizations using .env files or, even better, Direnv.

Yes uv pip install will ignore this. It's not a part of the "project" interface.

Yeah I figured, and I'm not arguing for the contrary. I just think it's a potential footgun to keep in mind.

@zanieb
Copy link
Member Author

zanieb commented Aug 30, 2024

I think it would make sense to respect UV_PROJECT_ENVIRONMNET when using uv pip install from a project root as in #6835 — but there's a bunch places we'd need to add inference (e.g., in every pip command) and we'd also need --no-project support. I can't do it right now — but I think it's the right behavior. cc @charliermarsh

@inoa-jboliveira
Copy link

inoa-jboliveira commented Aug 30, 2024

if you're cooperating with others on a project, you're dictating on them where to put their venv.

Unfortunately this is a reality in some projects (think about python has been around for 25 years). This prevents uv add/sync from being the norm by not providing a way to migrating while being compatible.

Then it might be finally possible for projects just forget about this need when uv is the norm.

Being in a transition stage we need features that might not make sense if you are just trying to be purist about what is right and wrong.

@ramarnat
Copy link

This solution works for me. Thanks for the responsive turnaround and the docs on the docker dev environment too.

@zanieb
Copy link
Member Author

zanieb commented Aug 30, 2024

if you're cooperating with others on a project, you're dictating on them where to put their venv.

Unfortunately this is a reality in some projects (think about python has been around for 25 years). This prevents uv add/sync from being the norm by not providing a way to migrating while being compatible.

I think the suggestion here is to use .env or similar if this is important to your project. We'll consider config-level options, but not with the urgency in which we pursued this change.

@ramarnat
Copy link

This won't change uv tool behavior of using its own place for deployment, will it?

@charliermarsh
Copy link
Member

This won't change uv tool behavior of using its own place for deployment, will it?

No, it won't have any effect on uv tool.

@inoa-jboliveira
Copy link

inoa-jboliveira commented Aug 30, 2024

if you're cooperating with others on a project, you're dictating on them where to put their venv.

I've been thinking more and more about this statement. UV does dictate where the venv is located. As if it prevents a "project maintainer" from deciding it because it was already decided and enforced for all users. Seems contradictory.

We'll consider config-level options, but not with the urgency in which we pursued this change.

Sure, thank you! I'm in favor of any command line option or env variable being configurable in .toml files because it allows me to give users a project ready to use where they just need to learn the minimal commands and not care about extra steps.

I used to work only in mac/linux environments, but for the past 2 years I moved to windows and there are way more challenges than in a unix based system to make things simply work

@FBen3
Copy link

FBen3 commented Aug 31, 2024

First off, thank you for putting so much effort into this project, and keeping up with all the requests & issues. uv is awesome.

Couple of stupid questions:

This does not directly enable centrally managed virtual environments. If you set UV_PROJECT_ENVIRONMENT to an absolute path and use it across multiple projects, they will clobber each other's environments.

So does this mean I can't do something like UV_PROJECT_ENVIRONMENT=VIRTUAL_ENV in my .zshrc file, and whenever I activate a venv it would resolve to my active virtual environment (and, e.g. uv add requests, would not re-create .venv in the project)?

I guess I can do something like alias activate='source venv/bin/activate && export UV_PROJECT_ENVIRONMENT=$VIRTUAL_ENV' to my .zshrc file so every time I activate / switch to a new virtual environment UV_PROJECT_ENVIRONMENT is updated.

We do not expose a CLI option or configuration file setting — we may pursue those later but I see them as lower priority.

Would it be easy / How does the dev team feel about having a --sync CLI option (that's like the opposite of --no-sync), which updates the active virtual environment? So users who run into issues like #6849 could just do uv add ruff --sync)

Additionally, uv is intentionally moving away from the concept of "active environments" and I don't think syncing to an "active" environment is the right behavior while managing projects

What is the reason behind this? Isn't it advantageous to explicitly define which environment a user is using? Otherwise how can I feel safe I'm not modifying my global environment?

@zanieb
Copy link
Member Author

zanieb commented Sep 3, 2024

Responding to the above

  1. Yes, you could do that. I wouldn't recommend it, but it seems okay to use this to match your active environment.
  2. I don't think we'll add a CLI option to sync the active environment. Yes, it'd be easy to implement but it comes with a lot of complexity.
  3. The problem of "global environments" is something we're trying to avoid — uv's project interface doesn't ever modify a global environment so you just don't need to worry about that. You easily could mutate an environment outside your current project if we just synced the active environment — this is the kind of problem we're trying to avoid. Generally speaking, activating environments shouldn't be necessary when using uv. Instead, use uvx and uv run and you'll always have the correct environment for your current context.

@zanieb zanieb merged commit 1234b6d into main Sep 3, 2024
57 checks passed
@zanieb zanieb deleted the zb/venv-path branch September 3, 2024 17:52
@edmorley
Copy link
Contributor

edmorley commented Sep 3, 2024

@zanieb Sorry for the delayed reply. This new env var will work well for our use-case (#5229 (comment)) - thank you! We'll likely also set VIRTUAL_ENV too (to the same location) for parity with how we set up pip/Poetry venvs, in case some other tooling checks for it (rather than checking for sys.prefix != sys.base_prefix), but this will be no different to pip, where we set both PIP_PYTHON and VIRTUAL_ENV already.

Instead, use uvx and uv run and you'll always have the correct environment for your current context.

So this probably something you've already considered/are aware of (and were instead talking mainly about the host-machine development use-case), but in a container context, it will be reasonably common to exclude uv from the final app image (in the case of Dockerfile via multi-stage builds, or in the case of CNBs via build-only layers), in which case the final app image can't wrap commands with uv run in eg CMD / ENTRYPOINT, and instead containers that are using venvs (for the reasons in #5229 (comment)) will need to have "activated" the venv by adding it to PATH etc instead.

@zanieb
Copy link
Member Author

zanieb commented Sep 3, 2024

Yeah totally agree — but that use-case doesn't require uv to read and mutate an "active" virtual environment.

charliermarsh pushed a commit that referenced this pull request Sep 3, 2024
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Sep 4, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.0` -> `0.4.4` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.4`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#044)

[Compare Source](astral-sh/uv@0.4.3...0.4.4)

##### Enhancements

-   Allow customizing the project environment path with `UV_PROJECT_ENVIRONMENT` ([#&#8203;6834](astral-sh/uv#6834))
-   Warn when `VIRTUAL_ENV` is set but will not be respected in project commands ([#&#8203;6864](astral-sh/uv#6864))
-   Add `--no-hashes` to `uv export` ([#&#8203;6954](astral-sh/uv#6954))
-   Make HTTP headers title case for backward compatibility ([#&#8203;6887](astral-sh/uv#6887))
-   Pin `.python-version` in `uv init` ([#&#8203;6869](astral-sh/uv#6869))
-   Support `file://` URLs for `UV_PYTHON_INSTALL_MIRROR` ([#&#8203;6950](astral-sh/uv#6950))
-   Introduce more docker tags for uv ([#&#8203;6053](astral-sh/uv#6053))

##### Bug fixes

-   Avoid canonicalizing the cache directory ([#&#8203;6949](astral-sh/uv#6949))
-   Show all PyPy versions in `uv python list --all-versions` ([#&#8203;6917](astral-sh/uv#6917))
-   Avoid incorrect `requires-python` marker simplifications ([#&#8203;6268](astral-sh/uv#6268))

##### Documentation

-   Add documentation for `UV_PROJECT_ENVIRONMENT` ([#&#8203;6987](astral-sh/uv#6987))
-   Add optional dependencies section to the lockfile document ([#&#8203;6982](astral-sh/uv#6982))
-   Document use of the `file://` scheme in Python installation mirrors ([#&#8203;6984](astral-sh/uv#6984))
-   Fix outdated references to the help menu documentation in the first steps page ([#&#8203;6980](astral-sh/uv#6980))
-   Show env option in CLI reference documentation ([#&#8203;6863](astral-sh/uv#6863))
-   Add bind mount example to `docker.md` ([#&#8203;6921](astral-sh/uv#6921))

### [`v0.4.3`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#043)

[Compare Source](astral-sh/uv@0.4.2...0.4.3)

##### Enhancements

-   Show build backend output when `--verbose` is provided ([#&#8203;6903](astral-sh/uv#6903))
-   Allow `uv sync --frozen --package` without copying member `pyproject.toml` ([#&#8203;6943](astral-sh/uv#6943))

##### Bug fixes

-   Avoid panic with missing temporary directory ([#&#8203;6929](astral-sh/uv#6929))
-   Avoid updating incorrect dependencies for sorted `uv add` ([#&#8203;6939](astral-sh/uv#6939))
-   Use lower-bound semantics for all Python compatibility comparisons ([#&#8203;6882](astral-sh/uv#6882))

### [`v0.4.2`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#042)

[Compare Source](astral-sh/uv@0.4.1...0.4.2)

##### Enhancements

-   Adding support for `.pyc`  files in `uv run` ([#&#8203;6886](astral-sh/uv#6886))
-   Treat missing `top_level.txt` as non-fatal ([#&#8203;6881](astral-sh/uv#6881))

##### Bug fixes

-   Fix `is_disjoint` check for supported environments ([#&#8203;6902](astral-sh/uv#6902))
-   Remove dangling archives in `uv cache clean ${package}` ([#&#8203;6915](astral-sh/uv#6915))
-   Error when discovered Python is incompatible with `--isolated` workspace ([#&#8203;6885](astral-sh/uv#6885))
-   Warn when discovered Python is incompatible with PEP 723 script ([#&#8203;6884](astral-sh/uv#6884))

### [`v0.4.1`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#041)

[Compare Source](astral-sh/uv@0.4.0...0.4.1)

##### Enhancements

-   Add `uv export --format requirements-txt` ([#&#8203;6778](astral-sh/uv#6778))
-   Allow `@` references in `uv tool install --from` ([#&#8203;6842](astral-sh/uv#6842))
-   Normalize version specifiers by sorting ([#&#8203;6333](astral-sh/uv#6333))
-   Respect the user's upper-bound in `requires-python` ([#&#8203;6824](astral-sh/uv#6824))
-   Use Windows registry to discover Python on Windows directly ([#&#8203;6761](astral-sh/uv#6761))
-   Hint at `--no-workspace` in `uv init` failures ([#&#8203;6815](astral-sh/uv#6815))
-   Update to last PyPy releases ([#&#8203;6784](astral-sh/uv#6784))

##### Bug fixes

-   Avoid deadlocks when multiple uv processes lock resources ([#&#8203;6790](astral-sh/uv#6790))
-   Expand tildes when matching against `PATH` ([#&#8203;6829](astral-sh/uv#6829))
-   Fix `uv init --no-project` alias ([#&#8203;6837](astral-sh/uv#6837))
-   Ignore pre-release segments when discovering via `requires-python` ([#&#8203;6813](astral-sh/uv#6813))
-   Support inline optional tables in `uv add` and `uv remove` ([#&#8203;6787](astral-sh/uv#6787))
-   Update default `hello.py` to pass `ruff format` ([#&#8203;6811](astral-sh/uv#6811))
-   Avoid stripping root for user path display ([#&#8203;6865](astral-sh/uv#6865))
-   Error when user-provided environments are disjoint with Python ([#&#8203;6841](astral-sh/uv#6841))
-   Retain alphabetical sorting for `pyproject.toml` in `uv add` operations ([#&#8203;6388](astral-sh/uv#6388))))

##### Documentation

-   Add a link to the multiple index docs in the alternative index guide ([#&#8203;6826](astral-sh/uv#6826))
-   Add docs for inline exclude newer in PEP 723 scripts ([#&#8203;6831](astral-sh/uv#6831))
-   Enumerate available Docker tags ([#&#8203;6768](astral-sh/uv#6768))
-   Omit `[pip]` section from configuration file docs ([#&#8203;6814](astral-sh/uv#6814))
-   Update `project.urls` in `pyproject.toml`  ([#&#8203;6844](astral-sh/uv#6844))
-   Add docs for AWS CodeArtifact usage ([#&#8203;6816](astral-sh/uv#6816))

##### Other changes

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
configuration Settings and such
Projects
None yet
7 participants