pip_import support for Piplock files

[samiashi]

Is it possible to maintain a Piplock file and import it using pip_import instead of having to maintain a requirements.txt?

If the above is not currently possible, what would you say the best approach is for tackling this issue?

[thundergolfer]

The best approach seems to be just doing pipenv lock -r > requirements.txt whenever your Pipfile.lock gets updated.

We do basically this @ Canva

[samiashi]

Yea this seems like the only solution at the moment. I was wondering if piplock files can be officially supported so we can lose this extra step.

[thundergolfer]

It's possible that rules_python could side-step Pipfile.lock support like how rules_jvm_external sidesteps a Pom.xml.

The missing key feature is transitive dependency resolving. If rules_python can support that without needing Pipfile.lock, that'd be great.

Supporting pipenv's lock file may lead to having to support poetry's lock file, as the latter is getting increasingly popular partly because of the problems with pipenv.

[dgrnbrg]

I've actually written a program that takes a declarative dependency graph, uses poetry to resolve the transitive compatible dependency versions, and then write out bazel targets which include the sha256 & https location of the wheels. It also binary compatibility mapping, for system & architecture matching. I'm not sure how to write the bazel rules that could consume this, but I'd be happy to collaborate.

I think that we could end up with a solution like http://github.com/johnynek/bazel-deps, which is very pleasant.

[thundergolfer]

I also like bazel-deps and went and wrote a similar tool for our team and called it py-bazel-deps 😄.

Is yours open-sourced anywhere?

[dgrnbrg]

It’s unfortunately not complete, because I didn’t have time to figure out what to do with the downloaded wheels to incorporate them into rules. I would be glad to share that code to combine it with the additional rules needed to make it a usable solution

[jondo2010]

It’s unfortunately not complete, because I didn’t have time to figure out what to do with the downloaded wheels to incorporate them into rules. I would be glad to share that code to combine it with the additional rules needed to make it a usable solution

I'm in need of exactly this type of tool, and am in the planning stages of writing it myself as OSS. I'd be extremely happy to collaborate if you'd like to use your codebase as a starting point.

[dgrnbrg]

@jondo2010 I am working on getting this together to share :)

[thundergolfer]

At our company we are using https://github.com/dillon-giacoppo/rules_python_external which can do the transitive dependency resolving. I'm not too familiar with it's locking behaviour.

cc @dillon-giacoppo

[dillon-giacoppo]

https://github.com/dillon-giacoppo/rules_python_external doesn't do anything special with locking. It uses pip to automatically download all the requirements and then parses the wheel metadata to recalculate dependencies. To lock the dependencies, you would need to make requirements.txt a transitive closure.

A custom resolver was the only piece missing for allowing very efficient pinning like in rules_jvm_external. If @dgrnbrg has a resolver we could potentially add a lock solution like https://github.com/bazelbuild/rules_jvm_external#pinning-artifacts-and-integration-with-bazels-downloader. It has a lock file format like this:

{
    "coord": "com.google.code.findbugs:jsr305:3.0.2",
    "dependencies": [],
    "directDependencies": [],
    "file": "v1/https/jcenter.bintray.com/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar",
    "mirror_urls": [
        "https://jcenter.bintray.com/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar"
    ],
    "sha256": "766ad2a0783f2687962c8ad74ceecc38a28b9f72a2d085ee438b7813e928d0c7",
    "url": "https://jcenter.bintray.com/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar"
},

This would also allow us to throw everything directly into the Bazel HTTP cache.

[thundergolfer]

To lock the dependencies, you would need to make requirements.txt a transitive closure.

Ah yep. We're doing pipenv lock > requirements.txt to get the locked transitive closure of our Pipfile's top-level deps.

[thundergolfer]

I'll close this as wonfix. Given the state of the Pipenv project and a developing opinion that these Bazel rules shouldn't support consuming the lock files of alternative packaging system but instead focus on developing and maintaining it's own description of a locked transitive dependency list, the requested functionality is not worth adding.