Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing callable venv_backend for creation of session virtualenv #753

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

wpk-nist-gov
Copy link

@wpk-nist-gov wpk-nist-gov commented Dec 11, 2023

This addresses #751

This PR allows for callable venv_backend to nox.session. This functionality can be used to, for example,

  • Create conda environments from environment.yaml files during creation
  • Easily override the virtual environment location

This is a general solution that should be applicable to a number of open issues.

Main changes:

  • Update nox.sessions.SessionRunner._create_venv to look for a callable.
  • Included a section in Customized virtual environment creation in docs/tutorial.rst and example usage
    in docs/cookbook.rst.
  • Added (simple) tests of this functionality to tests_session.py. Nothing too deep, but there is coverage.
  • Changed the conda_tests session of noxfile.py to use this functionality. This now creates conda environments using
    conda env create -f .... with environment files under environment/py{py_version}-conda-test.yaml. These files are created from requirements-conda-test.txt. There is also a session to use conda-lock.
    This is primarily for demonstration purposes.

Open issues:

  • The conda examples refer to private variables (SessionRunner._reused and SessionRunner._clean_location). Does it make sense to make these public? I think that this is "advanced" functionality, so I'm happy to go with it.
  • The callable has the signature
def venv_backend(
    location: str,
    interpreter: str | None,
    reuse_existing: bool,
    venv_params: Any,
    runner: SessionRunner,
) -> CondaEnv/VirtualEnv/..."
...

The parameters where copied from the previous SessionRunner._create_env calls to VirtualEnv, etc, with the addition of runner. I added this if someone wants to access the calling runner instance. This might be seen as repetitive, as all these parameters are derived from runner.

Please let me know how I can improve the PR. Thank you!

wpk added 7 commits December 8, 2023 16:31
This branch allows passing a callable for `venv_backend` in `nox.session`.
This allows the user to apply any special commands during the virtualenv
creation.
As an example, `noxfile.py` includes some demo session using this functionality.

* conda-env-backend: creates an environment using `conda env create`
* conda-lock-backend: creates an environment using `conda-lock install`
* dev-example: Creates a development.  This creates an environment in
`.nox/.venv` for demo purposes

Other potential uses:

* Alter the python interpreter search path
* Add smart caching of environment.  i.e., add check in callback to
  reinstall/recreate only if dependencies have changed

The whole point is that the approach is general.

Theres a degree of buyer beware with this functionality.  In the examples
in `noxfile.py`, the correct python version must be specified in the
`environment.yaml`
files.  To me, the added functionality is worth it.
Pretty basic testing, but now have coverage
@wpk-nist-gov
Copy link
Author

Any chance someone could take a look at this?

@jayqi
Copy link

jayqi commented Feb 10, 2024

It was mentioned in #751 that this could be a solution for supporting micromamba (#680). It would be great if there were an example of that in the cookbook.

* Added micromamba_test sessions to `noxfile.py`
* Updated conda env create section of cookbook to also include a micromamba example
@wpk-nist-gov
Copy link
Author

Thank you for the suggestion @jayqi! I added sessions micromamba_tests to noxfile.py and updated the cookbook example to use both conda env create and micromamba create.

@cjolowicz
Copy link
Collaborator

I think this merits broader discussion because you can't use the callable without accessing undocumented Nox internals.

@henryiii
Copy link
Collaborator

henryiii commented Feb 24, 2024

(I was thinking that, was getting worried about how much of nox's internals would be expected to be touched by users with this. #762 changed the internals a bit, and we might not have been able to do that if this was public)

@wpk-nist-gov
Copy link
Author

Thank you for taking a look at this PR! I agree that using private/undocumented internals is less than ideal. But I still think functionality like this would be awesome. Before #762, I was myself playing with using uv via the callable backend. I'd like to take a look at #762 to get a feel of what internals have changed.

Would it make sense to propose making a minimal set of the private internals touched here public? I'd be happy to take a crack at that.

Thanks again!

@henryiii
Copy link
Collaborator

Let's wait till after the next release, then it might be interesting to discuss. A few more environment related changes are going in.

@wpk-nist-gov
Copy link
Author

wpk-nist-gov commented Feb 26, 2024

I think I have a better solution, that will require less interaction with nox internals. There are two things I'm trying to override with the callback.

  1. venv location
  2. venv creation command.

For (1), the obvious use case is using nox to manage development environments. Something like the following:

@nox.session(venv_location=".venv", ....)
def dev(session: nox.Session) -> None:
    ....

For (2), the use case is customizing the creation of conda environments. Instead of trying to hook into the virtualenv creation, you could instead just ask for the venv class to do everything (check for env reuse, delete previous venv, etc) but skip actually creating the virtualenv. In this case, it would be on the user to create the virtualenv in the session function. Something like:

@nox.session(venv_skip_create=True)
def test(session: nox.Session) -> None:

    if not session.virtualenv._reused:  # I'd like to make the "reused property public"
        session.run_always(
            "conda",
            "env",
            "create",
            "-f",
            f"py{session.python}-environment.yaml",
            "--prefix",
            session.virtualenv.location,  # might add session.location, session.location_name?
        )

    ...

where as noted, I'd like to make the reused property public, and perhaps expose some properties in session.virtualenv to session level.

I'm happy to work on this. Should I make a new PR with the above functionality? If so, should I split into 2 PR's or just one?

@henryiii
Copy link
Collaborator

IMO, venv location makes sense. Being able to set it to something other than .nox/{name} would be useful. Maybe that would be a good thing to start with?

@wpk-nist-gov
Copy link
Author

Excellent! I'll put a new PR together with that functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants