Is hatchling really needed? #120
Comments
|
Hello again!
Hatch is now more like Poetry & Flit which have their own build backends,
Yeah based on https://www.python.org/dev/peps/pep-0517/#in-tree-build-backends they all do that:
Maybe this helps? https://src.fedoraproject.org/rpms/python-poetry-core/blob/rawhide/f/python-poetry-core.spec
They are mixed in with Hatch https://github.com/ofek/hatch/tree/master/tests/backend. However, https://github.com/ofek/hatch/tree/master/tests/backend/downstream is standalone and I can move it tonight!
No, one repo is easier to develop on. |
|
Hi from Debian, I'm here for the same reason (building userpath). Currently hatchling's version seems unrelated to hatch, is it expected to stay that way? Normally we'd build both of these modules from the same source tree (they share docs, tests, etc.), but that gets messy if they have separate versions. |
|
Hey! Yes they are unrelated, and I'd be reluctant to change that. There is precedence: Poetry: PDM: It's best to conceptualize them as separate projects. Are tests & docs required when packaging for distros like Debian & Fedora? |
Not strictly required in Fedora but it's very good to have them. Especially when we package a new Python version sooner than others. Also, tests can uncover incompatibilities in dependencies faster than dependant packages. Let me mention a few points, why the current situation is not ideal from our point of view:
Poetry and poetry-core are similar but because they are separated packages, we can package them as they are and treat them individually. |
|
I do like the idea of having the pep517 backend being a minimal library, that doesn't bring in all the baggage that the main CLI does, 👍 for following that trend. In The difficulty we have here is that if we like to be able to run the upstream test suites, when possible:
The tests aren't in the source tarball on PyPI, only in the github repo. The github repo contains both hatch and hatchling, as well as their tests. So either we do:
2 seems like the best option, but it involves annoying duplication. |
The problem with this option also is that we'd like to run tests of hatch with hatchling from the RPM package and vice versa so we can check that the hatchling is correctly packaged during the build of hatch. That would be harder when both projects share the same repo because they see each other and probably ignore system site-packages. |
|
Good point. That can be worked around by deleting one of the libraries, of course. But if they're intertwined, that gets messy. |
No path manipulation is done, but to make it extra explicit I switched each to a
I think you can just do:
Hatchling now ships the downstream tests https://pypi.org/project/hatchling/0.11.0/ |
|
I’m currently investigating packaging It’s too early to tell if I will encounter any serious issues, but I very much appreciate the switch to |
|
Please let me know if I can be of further assistance to all of you. I greatly appreciate your efforts on this 🙏 |
|
It looks like the tests for …and this assumes the original Then, they download (via What I’m seeing in the test that builds a copy of Is it true that projects using If dependencies must always be downloaded, it probably won’t be possible to package |
No. All that hacky test logic is to force How would you recommend I change the script to meet your needs? |
|
That makes sense. It occurred to me after I commented that I’ll have to look more closely at what the script is doing. I see two obvious ways it could go:
|
|
I’ve decided to do without I’ve been trying to make a working package for
|
Hmm, what determines that?
That's a |
Virtualenv in Fedora does not use the embedded wheels, we have our own provided by RPM package python-pip-wheel so the pip in virtualenv is the same version as the system pip provided by python3-pip. But this is just a note. I believe that this should make no difference for hatch. |
|
The error basically shows that virtualenv fails to load the pip wheel to install it. Can someone create a Docker image repro of the problem for deep dive into the issue? |
|
I’ve determined that this virtualenv --pip embedin Fedora, and that virtualenv --pip bundleworks fine. So it’s easy to reproduce and not at all specific to Unfortunately, that means the I found a couple of issues filed upstream on virtualenv by Fedora packagers with similar issues (pypa/virtualenv#2283 pypa/virtualenv#2247). In both cases, upstream’s response was more or less “this is due to a Fedora patch, so you’re on your own.” I haven’t been able to find any documentation about what the difference between I’ve opened a Fedora issue to solicit discussion on whether this is expected behavior, and what, if anything, can be done about it. |
|
You might want to read https://virtualenv.pypa.io/en/latest/user_guide.html#wheels |
Interesting. Thanks for the pointer. It seems like Fedora’s diff --git a/src/virtualenv/seed/wheels/embed/__init__.py b/src/virtualenv/seed/wheels/embed/__init__.py
index d38ccf8..4b80d98 100644
--- a/src/virtualenv/seed/wheels/embed/__init__.py
+++ b/src/virtualenv/seed/wheels/embed/__init__.py
@@ -48,8 +48,11 @@ BUNDLE_SUPPORT = {
}
MAX = "3.11"
+# Redefined here because bundled wheels are removed in RPM build
+BUNDLE_SUPPORT = None
def get_embed_wheel(distribution, for_py_version):
+ return None # BUNDLE_SUPPORT == None anyway
path = BUNDLE_FOLDER / (BUNDLE_SUPPORT.get(for_py_version, {}) or BUNDLE_SUPPORT[MAX]).get(distribution)
return Wheel.from_path(path)
Anyway, I’m curious what the core maintainers of the Python stack in Fedora have to say about the matter. I haven’t spent much time in the internals of these tools. |
I came to the same decision, those tests all depend on pulling code from the Internet and executing it, which isn't acceptable in our build and test infrastructure. If there were unit tests, or a fully off-line test suite, those would be better suited to our use-case. RE Fedora's FWIW, Debian's virtualenv patch-set basically just removes the bundled wheels: https://salsa.debian.org/python-team/packages/python-virtualenv/-/tree/master/debian/patches The |
I mentioned this in the I use Windows, so maybe it's just a Windows-only bug. I'm fine with removing that flag to help here, though can Debian/Fedora guarantee virtual environments are populated with at least version 21.3 of |
I studied this a little, and my understanding is as follows: It’s controlled by |
There's been many fixes to that part in the last two months, so that might be now fixed 👌 |
In Fedora, the answer is basically “yes”:
Alternatively, if the I’d be happy to patch out |
Wow. python/mypy#8802 I had no idea that was just a legacy
I'll remove that flag now as well. |
|
As a test, I applied diff -Naur hatch-hatch-v1.0.0rc12-original/src/hatch/venv/core.py hatch-hatch-v1.0.0rc12/src/hatch/venv/core.py
--- hatch-hatch-v1.0.0rc12-original/src/hatch/venv/core.py 2022-02-05 00:58:01.000000000 -0500
+++ hatch-hatch-v1.0.0rc12/src/hatch/venv/core.py 2022-02-13 11:41:04.927660495 -0500
@@ -47,7 +47,7 @@
self.directory.ensure_parent_dir_exists()
- command = [str(self.directory), '--no-download', '--no-periodic-update', '--pip', 'embed', '--python', python]
+ command = [str(self.directory), '--no-download', '--no-periodic-update', '--python', python]
if allow_system_packages:
command.append('--system-site-packages')The I can confirm that this change fixes the Some of those tests now produce an |
|
I don't have time to dive into it today but I promise to take a look tomorrow. I'm the author of the Fedora virtualenv patch so if we find a better way to handle our own wheels, I should be able to implement it. |
Thanks! I really appreciate it. (It doesn’t have to be tomorrow.) |
Thanks a lot for the explanation. I think something like this might be useful to have in the documentation.
Let's discuss this in the bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2053948 . This discussion is important for us but not that much for upstream projects. |
|
@musicinmybrain Were you able to successfully add Hatchling to Fedora? I couldn't find a link. |
|
It is in the package review process. Once that is done, it will be here. @frenzymadness is handing over primary maintenance of the I have been using COPR to test these packages, and have successfully built As I noted, it will take some time to work through the remaining failing tests in |
|
Great news, thank you all very much! I'll keep this open until both
I'm planning to release and announce the non-RC v1.0.0 of the CLI the week of PyCon which begins on Wednesday, April 27. If you'd prefer, I can do the release earlier so that the system packages are already available by then. |
Version 1.0 of There is plenty of time to get version 1.0 of Fedora 36 is currently targeted for 2022-04-26. Getting
Personally, I think that’s all pushing it—more risk than necessary, and more intensive effort in the short term than I want to sign up for—and it’s better to keep |
Do either or both of these minimum versions apply only to Currently, looking at the sources in this repository:
It seems like any minimum versions should be explicitly encoded in the wheel metadata; or, failing that, if I understand exactly what they are, I can at least add them at the RPM level to remind me not to do any unwise backports. The python-hatchling package is now in the development version of Fedora (Rawhide) as well as in the upcoming Fedora 36 release, by the way. |
Only to
Great news, thanks!!! |
Fixed: |
|
I just started looking at preparing a hatch 1.0 package for Fedora again, using 1.0.0rc16. I see that there is a clever pytest mark called which, to me, seems to imply that these tests need to install wheels from PyPI. Should the (Note that, for now, Fedora Rawhide has |
|
Yeah I forgot after a while, please make a PR 🙂 Any ideas on how to emulate no internet in CI? |
Will do. Thanks.
If all you need to do is break export PIP_PROXY='https://localhost:1'should do the trick. |
|
To what extent should Generally, in Fedora Linux, only |
|
Never as of latest RC, default should now be |
Okay, I like that answer. I’ll start a PR for any exceptions that I find. |
Okay, it turns out that this seems to be fine now. Awesome! I’ve helped get With that work done, I’m down to a single failing test for Should TOML output tests be made insensitive to blank lines? |
|
Nice!!! Hmm, looks like new |
|
Fixed ^, + 3 other tests that started failing b/c of some different output from the new version of Hatchling |
|
Okay, with #171 and #174 all the tests pass, except that I forgot I still have BuildRequires: python-unversioned-commandin my spec file as a workaround. When I remove that, such that the build environment has only What do you think is the correct fix for this? |
|
#176 ? |
|
FYI I'm about to release Hatchling 0.23.0 then finally Hatch 1.0.0, just in time for PyCon 😄 I'll close this issue after that |
|
Thanks for helping me understand Hatch and Hatchling and working with me and others to get them properly packaged in Fedora Linux. The number of packages built with Hatchling is already increasing rapidly. Meanwhile, I announced Hatch 1.0.0 on the |
|
Thank you all very much! Note that Hatch will likely be adopted by PyPA in the next week so the repo location will change: |
Hello.
I wanted to update hatch and userpath in Fedora rawhide but I see that they use a new build backend hatchling. What is the motivation for using your own build backend? It's making my situation much harder because if I want to update the mentioned package, I need to create a new one. And the new one seems to use itself for the build. Also, where are the tests for hatchling? Is there any plan to make it a separate project?
The text was updated successfully, but these errors were encountered: