[BUG] test_editable_install fails on Fedora

[jaraco]

setuptools version

main

Python version

Python 3.12

OS

Fedora (latest as found on Docker)

Additional environment information

No response

Description

When running tests on Fedora, the test_editable_install tests are failing with "No module named sample" and other errors.

Expected behavior

Tests should pass (or be marked as xfail if they can't pass). This test is already xfailing for macos due to #4328.

How to Reproduce

Run this Dockerfile:

FROM fedora
RUN dnf install -y python3-devel python3-pip make gcc patch zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel xz-devel libuuid-devel gdbm-libs libnsl2 git
RUN pip install --user tox
RUN git clone https://github.com/pypa/setuptools --depth 100
WORKDIR setuptools
CMD python3 -m tox -- -p no:cov -p no:dist -k test_editable_install

Output

[root@51b54ad691be setuptools]# tox -- -p no:cov -k test_editable_install
.pkg: _optional_hooks> python /root/.local/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_editable> python /root/.local/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_editable> python /root/.local/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py: install_package> python -I -m pip install --force-reinstall --no-deps /setuptools/.tox/.tmp/package/5/setuptools-69.5.1.post20240520-0.editable-py3-none-any.whl
py: commands[0]> pytest -p no:cov -k test_editable_install
============================================================= test session starts ==============================================================
platform linux -- Python 3.12.3, pytest-8.2.1, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /setuptools
configfile: pytest.ini
plugins: typeguard-4.2.1, xdist-3.6.1, home-0.5.1, ruff-0.3.2, subprocess-1.5.0, timeout-2.3.1, mypy-0.10.3, enabler-3.1.1, checkdocs-2.13.0, perf-0.14.0
12 workers [50 items]     
............F...............F...............F.F...                                                                                       [100%]
=================================================================== FAILURES ===================================================================
______________________________________________________ test_editable_with_prefix[lenient] ______________________________________________________
[gw9] linux -- Python 3.12.3 /setuptools/.tox/py/bin/python

tmp_path = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0')
sample_project = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/sampleproject'), editable_opts = []

    @pytest.mark.xfail(
        platform.python_implementation() == 'PyPy',
        reason="Workaround fails on PyPy (why?)",
    )
    def test_editable_with_prefix(tmp_path, sample_project, editable_opts):
        """
        Editable install to a prefix should be discoverable.
        """
        prefix = tmp_path / 'prefix'
    
        # figure out where pip will likely install the package
        site_packages = prefix / next(
            Path(path).relative_to(sys.prefix)
            for path in sys.path
            if 'site-packages' in path and path.startswith(sys.prefix)
        )
        site_packages.mkdir(parents=True)
    
        # install workaround
        _addsitedir(site_packages)
    
        env = dict(os.environ, PYTHONPATH=str(site_packages))
        cmd = [
            sys.executable,
            '-m',
            'pip',
            'install',
            '--editable',
            str(sample_project),
            '--prefix',
            str(prefix),
            '--no-build-isolation',
            *editable_opts,
        ]
        subprocess.check_call(cmd, env=env)
    
        # now run 'sample' with the prefix on the PYTHONPATH
        bin = 'Scripts' if platform.system() == 'Windows' else 'bin'
        exe = prefix / bin / 'sample'
>       subprocess.check_call([exe], env=env)

setuptools/tests/test_editable_install.py:439: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = ([PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/prefix/bin/sample')],)
kwargs = {'env': {'COLUMNS': '144', 'HOME': '/root', 'LANG': 'C.UTF-8', 'LINES': '45', ...}}, retcode = 1
cmd = [PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/prefix/bin/sample')]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '[PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/prefix/bin/sample')]' returned non-zero exit status 1.

/usr/lib64/python3.12/subprocess.py:413: CalledProcessError
------------------------------------------------------------ Captured stderr setup -------------------------------------------------------------
Cloning into 'sampleproject'...
------------------------------------------------------------- Captured stdout call -------------------------------------------------------------
Obtaining file:///tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/sampleproject
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Collecting peppercorn (from sampleproject==3.0.0)
  Using cached peppercorn-0.6-py3-none-any.whl.metadata (3.4 kB)
Using cached peppercorn-0.6-py3-none-any.whl (4.8 kB)
Building wheels for collected packages: sampleproject
  Building editable for sampleproject (pyproject.toml): started
  Building editable for sampleproject (pyproject.toml): finished with status 'done'
  Created wheel for sampleproject: filename=sampleproject-3.0.0-0.editable-py3-none-any.whl size=4315 sha256=9df19ac1bb30d3fb1a3d8aa842878ba78b5192fcca468e75080437a525fe56ad
  Stored in directory: /tmp/pip-ephem-wheel-cache-7m0rf11o/wheels/88/94/2f/b5fed8b2a95fe76d2023215387b213369f311f942bffac2b8e
Successfully built sampleproject
Installing collected packages: peppercorn, sampleproject
Successfully installed peppercorn-0.6 sampleproject-3.0.0
------------------------------------------------------------- Captured stderr call -------------------------------------------------------------
/setuptools/.tox/py/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
Traceback (most recent call last):
  File "/tmp/pytest-of-root/pytest-4/popen-gw9/test_editable_with_prefix_leni0/prefix/bin/sample", line 5, in <module>
    from sample import main
ModuleNotFoundError: No module named 'sample'
______________________________________________________ test_editable_with_prefix[strict] _______________________________________________________
[gw10] linux -- Python 3.12.3 /setuptools/.tox/py/bin/python

tmp_path = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0')
sample_project = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/sampleproject')
editable_opts = ['--config-settings', 'editable-mode=strict']

    @pytest.mark.xfail(
        platform.python_implementation() == 'PyPy',
        reason="Workaround fails on PyPy (why?)",
    )
    def test_editable_with_prefix(tmp_path, sample_project, editable_opts):
        """
        Editable install to a prefix should be discoverable.
        """
        prefix = tmp_path / 'prefix'
    
        # figure out where pip will likely install the package
        site_packages = prefix / next(
            Path(path).relative_to(sys.prefix)
            for path in sys.path
            if 'site-packages' in path and path.startswith(sys.prefix)
        )
        site_packages.mkdir(parents=True)
    
        # install workaround
        _addsitedir(site_packages)
    
        env = dict(os.environ, PYTHONPATH=str(site_packages))
        cmd = [
            sys.executable,
            '-m',
            'pip',
            'install',
            '--editable',
            str(sample_project),
            '--prefix',
            str(prefix),
            '--no-build-isolation',
            *editable_opts,
        ]
        subprocess.check_call(cmd, env=env)
    
        # now run 'sample' with the prefix on the PYTHONPATH
        bin = 'Scripts' if platform.system() == 'Windows' else 'bin'
        exe = prefix / bin / 'sample'
>       subprocess.check_call([exe], env=env)

setuptools/tests/test_editable_install.py:439: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = ([PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/prefix/bin/sample')],)
kwargs = {'env': {'COLUMNS': '144', 'HOME': '/root', 'LANG': 'C.UTF-8', 'LINES': '45', ...}}, retcode = 1
cmd = [PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/prefix/bin/sample')]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '[PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/prefix/bin/sample')]' returned non-zero exit status 1.

/usr/lib64/python3.12/subprocess.py:413: CalledProcessError
------------------------------------------------------------ Captured stderr setup -------------------------------------------------------------
Cloning into 'sampleproject'...
------------------------------------------------------------- Captured stdout call -------------------------------------------------------------
Obtaining file:///tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/sampleproject
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Collecting peppercorn (from sampleproject==3.0.0)
  Using cached peppercorn-0.6-py3-none-any.whl.metadata (3.4 kB)
Using cached peppercorn-0.6-py3-none-any.whl (4.8 kB)
Building wheels for collected packages: sampleproject
  Building editable for sampleproject (pyproject.toml): started
  Building editable for sampleproject (pyproject.toml): finished with status 'done'
  Created wheel for sampleproject: filename=sampleproject-3.0.0-0.editable-py3-none-any.whl size=4339 sha256=9aee59a1470df09b2c535967f524989da1553bbde3123abed4e6dc6daf6f82a3
  Stored in directory: /tmp/pip-ephem-wheel-cache-h8_c74dv/wheels/68/8b/73/ae0ff8ceee705f4282f93bf7df4228a148912bc518db33f388
Successfully built sampleproject
Installing collected packages: peppercorn, sampleproject
Successfully installed peppercorn-0.6 sampleproject-3.0.0
------------------------------------------------------------- Captured stderr call -------------------------------------------------------------
/setuptools/.tox/py/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py:141: EncodingWarning: 'encoding' argument not specified.
  proc = subprocess.Popen(
Traceback (most recent call last):
  File "/tmp/pytest-of-root/pytest-4/popen-gw10/test_editable_with_prefix_stri0/prefix/bin/sample", line 5, in <module>
    from sample import main
ModuleNotFoundError: No module named 'sample'
______________________________________ TestOverallBehaviour.test_editable_install[lenient-custom-layout] _______________________________________
[gw4] linux -- Python 3.12.3 /setuptools/.tox/py/bin/python

self = <setuptools.tests.test_editable_install.TestOverallBehaviour object at 0xffffb0dc9250>
tmp_path = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw4/test_editable_install_lenient_0')
venv = <setuptools.tests.environment.VirtualEnv object at 0xffffb0e3cd40>, layout = 'custom-layout', editable_opts = []

    @pytest.mark.xfail(sys.platform == "darwin", reason="pypa/setuptools#4328")
    @pytest.mark.parametrize("layout", EXAMPLES.keys())
    def test_editable_install(self, tmp_path, venv, layout, editable_opts):
        project, _ = install_project(
            "mypkg", venv, tmp_path, self.EXAMPLES[layout], *editable_opts
        )
    
        # Ensure stray files are not importable
        cmd_import_error = """\
        try:
            import otherfile
        except ImportError as ex:
            print(ex)
        """
        out = venv.run(["python", "-c", dedent(cmd_import_error)])
        assert "No module named 'otherfile'" in out
    
        # Ensure the modules are importable
        cmd_get_vars = """\
        import mypkg, mypkg.mod1, mypkg.subpackage.mod2
        print(mypkg.mod1.var, mypkg.subpackage.mod2.var)
        """
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" in out
    
        # Ensure resources are reachable
        cmd_get_resource = """\
        import mypkg.subpackage
        from setuptools._importlib import resources as importlib_resources
        text = importlib_resources.files(mypkg.subpackage) / "resource_file.txt"
        print(text.read_text(encoding="utf-8"))
        """
        out = venv.run(["python", "-c", dedent(cmd_get_resource)])
        assert "resource 39" in out
    
        # Ensure files are editable
        mod1 = next(project.glob("**/mod1.py"))
        mod2 = next(project.glob("**/mod2.py"))
        resource_file = next(project.glob("**/resource_file.txt"))
    
        mod1.write_text("var = 17", encoding="utf-8")
        mod2.write_text("var = 781", encoding="utf-8")
        resource_file.write_text("resource 374", encoding="utf-8")
    
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" not in out
>       assert "17 781" in out
E       AssertionError

setuptools/tests/test_editable_install.py:947: AssertionError
------------------------------------------------------------ Captured stdout setup -------------------------------------------------------------
created virtual environment CPython3.12.3.final.0-64 in 81ms
  creator CPython3Posix(dest=/tmp/pytest-of-root/pytest-4/popen-gw4/test_editable_install_lenient_0/venv/.env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==24.0, wheel==0.43.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Processing /tmp/pytest-of-root/pytest-4/dist_build/setuptools-69.5.1.post20240520-py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-69.5.1.post20240520
________________________________________ TestOverallBehaviour.test_editable_install[lenient-namespace] _________________________________________
[gw3] linux -- Python 3.12.3 /setuptools/.tox/py/bin/python

self = <setuptools.tests.test_editable_install.TestOverallBehaviour object at 0xffff91c69310>
tmp_path = PosixPath('/tmp/pytest-of-root/pytest-4/popen-gw3/test_editable_install_lenient_0')
venv = <setuptools.tests.environment.VirtualEnv object at 0xffff91d003e0>, layout = 'namespace', editable_opts = []

    @pytest.mark.xfail(sys.platform == "darwin", reason="pypa/setuptools#4328")
    @pytest.mark.parametrize("layout", EXAMPLES.keys())
    def test_editable_install(self, tmp_path, venv, layout, editable_opts):
        project, _ = install_project(
            "mypkg", venv, tmp_path, self.EXAMPLES[layout], *editable_opts
        )
    
        # Ensure stray files are not importable
        cmd_import_error = """\
        try:
            import otherfile
        except ImportError as ex:
            print(ex)
        """
        out = venv.run(["python", "-c", dedent(cmd_import_error)])
        assert "No module named 'otherfile'" in out
    
        # Ensure the modules are importable
        cmd_get_vars = """\
        import mypkg, mypkg.mod1, mypkg.subpackage.mod2
        print(mypkg.mod1.var, mypkg.subpackage.mod2.var)
        """
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" in out
    
        # Ensure resources are reachable
        cmd_get_resource = """\
        import mypkg.subpackage
        from setuptools._importlib import resources as importlib_resources
        text = importlib_resources.files(mypkg.subpackage) / "resource_file.txt"
        print(text.read_text(encoding="utf-8"))
        """
        out = venv.run(["python", "-c", dedent(cmd_get_resource)])
        assert "resource 39" in out
    
        # Ensure files are editable
        mod1 = next(project.glob("**/mod1.py"))
        mod2 = next(project.glob("**/mod2.py"))
        resource_file = next(project.glob("**/resource_file.txt"))
    
        mod1.write_text("var = 17", encoding="utf-8")
        mod2.write_text("var = 781", encoding="utf-8")
        resource_file.write_text("resource 374", encoding="utf-8")
    
        out = venv.run(["python", "-c", dedent(cmd_get_vars)])
        assert "42 13" not in out
>       assert "17 781" in out
E       AssertionError

setuptools/tests/test_editable_install.py:947: AssertionError
------------------------------------------------------------ Captured stdout setup -------------------------------------------------------------
created virtual environment CPython3.12.3.final.0-64 in 102ms
  creator CPython3Posix(dest=/tmp/pytest-of-root/pytest-4/popen-gw3/test_editable_install_lenient_0/venv/.env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==24.0, wheel==0.43.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Processing /tmp/pytest-of-root/pytest-4/dist_build/setuptools-69.5.1.post20240520-py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-69.5.1.post20240520
===================================================================== mypy =====================================================================
Success: no issues found in 1 source file
======================================================== 4 failed, 46 passed in 10.57s =========================================================
py: exit 1 (10.75 seconds) /setuptools> pytest -p no:cov -k test_editable_install pid=9012
  py: FAIL code 1 (11.35=setup[0.60]+cmd[10.75] seconds)
  evaluation failed :( (11.39 seconds)

[jaraco]

Other tests in test_editable_install.py also fail under certain conditions, such as test_editable_with_prefix.

[abravalheri]

I wonder if there is any chance this happens because in those systems Python don't try to import sitecustomize?

https://github.com/pypa/setuptools/blob/6b7f7a18afc90007544092c446dc0cd856d86b17/setuptools/tests/test_editable_install.py#L1253C1-L1260C70

---

We can see that the .pth file in the container:

cat /tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/prefix/lib/python3.12/site-packages/__editable__.sampleproject-3.0.0.pth
/tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/build/__editable__.sampleproject-3.0.0-py3-none-any

We can also see that the directory contain a link farm:

dnf install -y tree
tree -a /tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/build/__editable__.sampleproject-3.0.0-py3-none-any
/tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/build/__editable__.sampleproject-3.0.0-py3-none-any
└── sample
    ├── __init__.py -> /tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/src/sample/__init__.py
    ├── package_data.dat -> /tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/src/sample/package_data.dat
    └── simple.py -> /tmp/pytest-of-root/pytest-0/popen-gw9/test_editable_with_prefix_stri0/sampleproject/src/sample/simple.py

2 directories, 3 files

[abravalheri]

This might be a problem with <prefix>/lib/pythonX.X/site-packages vs <prefix>/lib64/pythonX.X/site-packages assumptions.

[jaraco]

Yes, that's also what @webknjaz and @agronholm both found. Fix incoming.