-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Embeddings in mltransform #29564
Support Embeddings in mltransform #29564
Conversation
Fix tox.ini Fix pydoc Fix indent in pydoc
R: @damccorm This is ready for review. |
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Chunking the review a bit. Just reviewed base.py thus far, though I imagine that's the place I'll have the most comments
sdks/python/apache_beam/ml/transforms/embeddings/tensorflow_hub_test.py
Outdated
Show resolved
Hide resolved
sdks/python/apache_beam/ml/transforms/embeddings/sentence_transformer.py
Outdated
Show resolved
Hide resolved
sdks/python/apache_beam/ml/transforms/embeddings/tensorflow_hub.py
Outdated
Show resolved
Hide resolved
sdks/python/apache_beam/ml/transforms/embeddings/sentence_transformer.py
Outdated
Show resolved
Hide resolved
sdks/python/apache_beam/ml/transforms/embeddings/sentence_transformer_test.py
Outdated
Show resolved
Hide resolved
… on different machines
This reverts commit cfb1883.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mostly looks good, had a few more comments though
raise FileExistsError( | ||
"The artifact location %s already exists and contains %s. Please " | ||
"specify a different location." % | ||
(artifact_location, _ATTRIBUTE_FILE_NAME)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call - one possible future enhancement would be to support an overwrite argument that allows users to do this
sdks/python/apache_beam/ml/transforms/embeddings/vertex_ai_test.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One last nit, otherwise this LGTM. Feel free to make the change and merge once checks pass
Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Make base.py framework agnostic and add helper transforms * Add tests for base.py * Add sentence-transformers * Add tensorflow hub * Add vertex_ai * Make TFTProcessHandler a PTransform * raise RuntimeError in ArtifactsFetcher when it is used for embeddings * Add JsonPickle to requirements * Add tox tests * Mock frameworks in pydocs Fix tox.ini Fix pydoc Fix indent in pydoc * Add Row type check * Remove requires_chaining * change name of PTransformProvider to MLTransformProvider * remove batch_len in utility fun * Change type annotation and redundant comments * Remove get_transforms method * remove requires_chaining from tft * add tests to sentence-transformers * Pass inference_args to RunInference * Add TODO GH issue * refactor variables in vertex_ai embeddings * remove try/catch and throw error if options is empty for GCS artifact location * Refactor NotImplementedError message * remove tensorflow hub from this PR * Add _validate_transform method * add more tests * fix test * Fix test * Add more tests in sentence-transformer * use np.max instead of max * round to 2 decimals * Remove gradle command action * Refactor throwing dataflow client exception * skip the test if gcp is not installed * remove toxTests for hub * remove toxTests for hub * Fix values in assert for sentence_transformer_test * rename sentence_transformers to huggingface * fix pydocs * Change the model name for tests since it is getting different results on different machines * Fix pydoc in vertexai * add suffix to artifact_location * Revert "add suffix to artifact_location" This reverts commit cfb1883. * add no_xdist * Try fixing pydoc for vertexai * change tox.ini to use pytest directly * raise FileExistError if Attribute file is already present * modify build.gradle to match tox task names * Add note to CHANGES.md * change gcs bucket to gs://temp-storage-for-perf-tests * Add TODO GH links * Update CHANGES.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Make base.py framework agnostic and add helper transforms * Add tests for base.py * Add sentence-transformers * Add tensorflow hub * Add vertex_ai * Make TFTProcessHandler a PTransform * raise RuntimeError in ArtifactsFetcher when it is used for embeddings * Add JsonPickle to requirements * Add tox tests * Mock frameworks in pydocs Fix tox.ini Fix pydoc Fix indent in pydoc * Add Row type check * Remove requires_chaining * change name of PTransformProvider to MLTransformProvider * remove batch_len in utility fun * Change type annotation and redundant comments * Remove get_transforms method * remove requires_chaining from tft * add tests to sentence-transformers * Pass inference_args to RunInference * Add TODO GH issue * refactor variables in vertex_ai embeddings * remove try/catch and throw error if options is empty for GCS artifact location * Refactor NotImplementedError message * remove tensorflow hub from this PR * Add _validate_transform method * add more tests * fix test * Fix test * Add more tests in sentence-transformer * use np.max instead of max * round to 2 decimals * Remove gradle command action * Refactor throwing dataflow client exception * skip the test if gcp is not installed * remove toxTests for hub * remove toxTests for hub * Fix values in assert for sentence_transformer_test * rename sentence_transformers to huggingface * fix pydocs * Change the model name for tests since it is getting different results on different machines * Fix pydoc in vertexai * add suffix to artifact_location * Revert "add suffix to artifact_location" This reverts commit cfb1883. * add no_xdist * Try fixing pydoc for vertexai * change tox.ini to use pytest directly * raise FileExistError if Attribute file is already present * modify build.gradle to match tox task names * Add note to CHANGES.md * change gcs bucket to gs://temp-storage-for-perf-tests * Add TODO GH links * Update CHANGES.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
Each config will define its own model handler so that model handler can be compatible with the inputs passed to the MLTransform.
This PR supports
vertex_ai
,hugging face
(sentence-transformers).Changes made:
ProcessHandler
a PTransform._TextEmbeddingHandler
which takes aModelHandler
but it is responsible to work onDict[str, Any]
inputs.MLTransform
would be the container that holds the list of transforms and these list of transforms are passed to the_MLTransformToPTransformMapper
which maps each data processing transforms to its PTransformjsonpickle
to store and load the ptransforms instances. This is done to seal the gap between training and inference.jsonpickle
is cross compatible across python versions and has backward compatibility with older versions of its versions.This PR will throw if input is a
Dict[str, List[str]]
. Works forDict[str, str]
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.