Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider update pyinstaller for python fuzzing to newer version #12324

Open
arthurscchan opened this issue Aug 9, 2024 · 1 comment
Open

Comments

@arthurscchan
Copy link
Contributor

arthurscchan commented Aug 9, 2024

Pyinstaller is used to compile Python fuzzers into executables, pyinstaller 5.0.1 is installed in https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/install_python.sh#L22 for separate projects to use it to compile python fuzzers.
It is found that this version (and some older versions) of pyinstaller have some compatibility issues with the inspect package. The inspect package has some functions, like getSource that require the real py source file and seems that pyinstaller failed to pick them up during the compile process. This results in a random OSError when the target project calls the functions in the inspect package. This problem was discovered when initialising the fuzzers for the huggingface-evaluate project in #12323 when the huggingface-datasets library depended on the evaluate project use inspect.getSource in its code.

It is suggested to update the version of pyinstaller in install_python.sh to a newer version. In #12323, it is tested pyinstaller version 6.9.0 has no problems on the above situation and could allow the fuzzers to compile and fuzz normally. Thus updating pyinstaller to 6.9.0 should help to solve the compatibility issue with the inspect package. Alternatively, updates to the pyinstaller could be done project by project to avoid affecting all the current Python projects.

For reference, the fuzzers in #12323 can be run directly without the pyinstaller, it only fails when it is compiled to an executable by pyinstaller version 5.0.1.

@DaveLak
Copy link
Contributor

DaveLak commented Aug 30, 2024

I bumped Pyinstaller to v6.10 (and some other outdated Python tools that are frequently the cause of broken builds, e.g. pip and setuptools) in #12027. If that PR lands, it would address this issue, but according to the most recent trail builds, bumping Pyinstaller seems to be low risk of causing a widespread breakage so it could probably be done independently of a Python version bump too.

I agree it would be a good idea. Recent versions of Pyinstaller have a lot of nice improvements related to dependency collection for bundled packages.

If it is done separately though, it would be nice if setuptools was also bumped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants