[Bug]: Overly restrictive Protobuf version in PyPI repository

[jhoareau]

What happened?

When installing Apache Beam with Poetry, if any other package requires some version of protobuf, for example 3.20.0, then the dependency resolution fails.

This is because Poetry uses the PyPI definition for its dependency resolution, this definition can be found at https://pypi.org/pypi/apache-beam/2.44.0/json for the current version.

The issue appeared in Beam 2.43 (it worked well with 2.42), and in particular this block of code:

beam/sdks/python/setup.py

Lines 183 to 189 in aee2c84

	if sys.platform == 'darwin' and (
	sys.version_info.major == 3 and sys.version_info.minor == 10):
	# TODO (https://github.com/apache/beam/issues/23585): Protobuf wheels
	# for version 3.19.5, 3.19.6 and 3.20.x on Python 3.10 and MacOS are
	# rolled back due to some errors on MacOS. So, for Python 3.10 on MacOS
	# restrict the protobuf with tight upper bound(3.19.4)
	protobuf_dependency = ['protobuf>3.12.2,<3.19.5']

We can see, in this JSON, that the requirement is the tight bound for the quite particular case of Python 3.10 + macOS, rather than the wide general requirement <4. Since I don't think the PyPI repository definition supports dependency setting per environment, it would make more sense to have the wide requirement than the narrow requirement in PyPI, and let the setup script take care of failing installation if the proper protobuf is not installed.

See discussion in #23982 @tvalentyn @AnandInguva

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[AnandInguva]

Hi, I am working on updating protobuf. I will tackle this issue as part of that effort(early next week).