[Python][Packaging] Support Python 3.12 and upload wheels to PyPI

[EwoutH]

Describe the enhancement requested

Currently there are no pyarrow wheels for Python 3.12 on PyPI. Python 3.12 releases in a week (Monday 2023-10-02), so it would be great if wheels could be build and uploaded to PyPI, especially since many packages depend on it.

Component(s)

Python

[AlenkaF]

@raulcd

[raulcd]

Hi @EwoutH , thanks for opening the issue. Due to the pyarrow wheels being currently an artifact of the release for Apache Arrow we can't release them independently and we have to do an Arrow release. We have a scheduled release (14.0.0) in the next weeks. Code freeze is expected for the 10th of October. Due to the current complexity on the release process and the Apache guidelines unfortunately they won't be available just yet but I'll work on creating the pyarrow wheels for Python 3.12 and will add it to the 14.0.0 release.
As soon as they are available if necessary you'll be able to test with our nightly development wheels, those are not official releases but can help development and testing: https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages
cc ~ @jorisvandenbossche

[jsherman256]

FYI: this impacts docker images based on python:3 if they try to install pyarrow or packages dependent on pyarrow. This is because the python:3 image now refers to python 3.12, but pyarrow cannot be built inside that image by default, since it's missing cmake:

#7 381.6   Building wheel for pyarrow (pyproject.toml): started
#7 382.1   Building wheel for pyarrow (pyproject.toml): finished with status 'error'
#7 382.1   error: subprocess-exited-with-error
#7 382.1   
#7 382.1   × Building wheel for pyarrow (pyproject.toml) did not run successfully.
...
#7 382.1       error: command 'cmake' failed: No such file or directory
#7 382.1       [end of output]
#7 382.1   
#7 382.1   note: This error originates from a subprocess, and is likely not a problem with pip.
#7 382.1   ERROR: Failed building wheel for pyarrow

Just leaving this here in case it helps someone else debugging why their python container suddenly wont build anymore. I worked around this by moving from python:3 to python:3.11.6

[raulcd]

I have merged the wheels support for 3.12.
Those will be available as nightly development in our next daily builds.
The PyPI availability will be ready as soon as we release Arrow 14.0.0 which we started today with preparations but takes some time (~2 weeks) while we stabilize some nightly failures + verify and vote the release.

[arjonv]

@raulcd any indication on when PyPi availability will be ready

[raulcd]

Arrow 14.0.0 has just been published on PyPI with wheels for Python 3.12.

[RubTalha]

https://stackoverflow.com/questions/77318492/building-wheel-for-pyarrow-pyproject-toml-did-not-run-successfully/77318636#77318636

[SeaOtocinclus]

Any update on this?

[jorisvandenbossche]

Quoting the comment just above:

Arrow 14.0.0 has just been published on PyPI with wheels for Python 3.12.