-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Protobuf and GCP dependencies in Beam Python SDK #24599
Conversation
Local tested error in Python 3.8 MacOS env:
|
b84821d
to
e7a7858
Compare
e7a7858
to
04824b7
Compare
The Can be reproduced with
Update: Looked more into it. We generate the sidecar urn files to support mypy type checker. Some of the types of Protobuf classes has been changed and they were not getting matched with the current code, hence the URNs are not getting generated. |
04824b7
to
ee1da36
Compare
Run Python 3.7 PostCommit |
Run Python 3.9 PostCommit |
Run Python 3.7 PostCommit |
blocker: tensorflow_hub 0.12 (latest) is not compatible with tensorflow 2.12, which makes our TF integration tests fail. We need to wait until tensorflow_hub releases a newer version.
|
Run Python 3.9 PostCommit |
Run Python 3.7 PostCommit |
Run Python_PVR_Flink PreCommit |
merging |
Thanks, @AnandInguva ! |
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
@@ -32,14 +32,11 @@ cython<1 | |||
# some versions of libraries that launch Beam pipelines, like tensorflow-transform. | |||
# Leaving 'future' in our containers for now prevent breaking tft users. | |||
future | |||
# TODO: Remove the upper bound once Tensorflow 2.11 is released. | |||
# https://github.com/apache/beam/issues/23355 | |||
google-cloud-profiler<4.0.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason removing google-cloud-profiler dep in base image? This has broken the google-cloud-profiler support on Beam v2.47.0
Update: confirmed that support on v2.47.0 is not affected. It only affects dev version for now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was not intentional. Looks like Dataflow containers still include it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we intentionally removed google-cloud-debugger from Dataflow containers, perhaps there was confusion between these two deps.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dataflow container now does not include it. Can be seen by https://github.com/apache/beam/blob/master/sdks/python/container/py38/base_image_requirements.txt where google-cloud-profiler is no longer listed in. I now get error using profiler: #26698
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahh...this was not intentional.
@Abacn these are beam external images. Dataflow images are internal: https://source.corp.google.com/piper///depot/google3/third_party/cloud/dataflow/worker/images/base_python/Dockerfile.base-python;rcl=522192081;l=117
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this is less severe, so it only affects dev version
* Update build dependencies and mypy-protobuf * Update import * Add os.environ['PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION'] = 'python' * Update gen_protos.py * Manually generate proto file. * update proto file * Remove redundant code * Add encoding * Update protobuf * Remove tensorflow ad google cloud profiler * update dependencies * update facets-overview * Update goolge-cloud-langugage code to support breaking changes from v2.0 * Update video intelligence breaking changes * Update lower bounds on some gcp deps * Update deps * Debug * Update gen_protos.py to supoort google._upb * Change checking condition to cpp to upb * Update google-cloud-datastore ByteSize * Update Dockerfile * raise Error * Update dependencies bound * Add optional field * Add license headers * Add more optional * Changes types to make them compatible with mypy-protobuf generated stubs * Add license for grpcio-status * update mypy to 0.790 * add more optional fields to satisfy mypy type checker * Identify and solve the mypy type checks * fix lint * Revert "add more optional fields to satisfy mypy type checker" This reverts commit 1ebc815. * Revert "Add more optional" This reverts commit 3801c16. * Revert "Add optional field" This reverts commit 35631f5. * add relax_strict_optional_primitives to mypy_out * Fix up formatting * Fix up lint * Fix up formatting * Fix up lint * fix lint * Fix up docs * Add gh issue to update code and doc strings * Update lower bound of gcp dependencies * Add tensorflow rc * Add tf rc version and comment onnx tests * Update container dependencies * comment onnx task * Change bound on google-cloud-recommendations * Modify FakeMutation to support mocking proto messages * Update comment * Add todo for uncommenting onnx changes * update comment * Update dependencies * revert doc change * Refactor code/remove helper function from util * Update comment * Uncomment changes * Update tensorflow * Comment tfx-bsl tests * Update tox.ini syntax * refactor * fix requirements * Skip tensorflow tests on Python 3.7
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
Fixes: #23355
Fixes: #24569
Fixes: #25581
Fixes: #25328
Fixes: #20991
Fixes: #21019
Fixes: #24432
Fixes: #23585
Fixes: #22742
Also look at #22319 for proto2_coder_test_messages_pb2.py generation
Tensorflow Protobuf update GH issue: tensorflow/tensorflow#59221
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.