Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Embeddings in mltransform #29564

Merged
merged 58 commits into from
Dec 11, 2023
Merged

Support Embeddings in mltransform #29564

merged 58 commits into from
Dec 11, 2023

Commits on Nov 29, 2023

  1. Make base.py framework agnostic and add helper transforms

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    80e5c4a View commit details
    Browse the repository at this point in the history

  2. Add tests for base.py

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    0d34847 View commit details
    Browse the repository at this point in the history

  3. Add sentence-transformers

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    58b24f6 View commit details
    Browse the repository at this point in the history

  4. Add tensorflow hub

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    88f9ceb View commit details
    Browse the repository at this point in the history

  5. Add vertex_ai

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    23f7027 View commit details
    Browse the repository at this point in the history

  6. Make TFTProcessHandler a PTransform

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    04ebdb0 View commit details
    Browse the repository at this point in the history

  7. raise RuntimeError in ArtifactsFetcher when it is used for embeddings

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    f86c259 View commit details
    Browse the repository at this point in the history

  8. Add JsonPickle to requirements

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    fc4ec00 View commit details
    Browse the repository at this point in the history

  9. Add tox tests

    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    3da5ce8 View commit details
    Browse the repository at this point in the history

  10. Mock frameworks in pydocs 

    Fix tox.ini

Fix pydoc

Fix indent in pydoc
    @AnandInguva
    AnandInguva committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    4b4ee58 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Add Row type check

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    01ba217 View commit details
    Browse the repository at this point in the history

  2. Remove requires_chaining

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    f080c25 View commit details
    Browse the repository at this point in the history

  3. change name of PTransformProvider to MLTransformProvider

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    6111c31 View commit details
    Browse the repository at this point in the history

  4. remove batch_len in utility fun

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    ba24e81 View commit details
    Browse the repository at this point in the history

  5. Change type annotation and redundant comments

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    d690aec View commit details
    Browse the repository at this point in the history

  6. Remove get_transforms method

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    af7496b View commit details
    Browse the repository at this point in the history

  7. remove requires_chaining from tft

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    d713555 View commit details
    Browse the repository at this point in the history

  8. add tests to sentence-transformers

    @AnandInguva
    AnandInguva committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    50450f3 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. Merge remote-tracking branch 'origin/master' into embeddings_mltransform

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    c2b691f View commit details
    Browse the repository at this point in the history

  2. Pass inference_args to RunInference

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8823a75 View commit details
    Browse the repository at this point in the history

  3. Add TODO GH issue

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a7e2bd3 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu… 

    …va/beam into embeddings_mltransform
    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    519b3ed View commit details
    Browse the repository at this point in the history

  5. refactor variables in vertex_ai embeddings

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f77ae60 View commit details
    Browse the repository at this point in the history

  6. remove try/catch and throw error if options is empty for GCS artifact… 

    … location
    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    95ed3c5 View commit details
    Browse the repository at this point in the history

  7. Refactor NotImplementedError message

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    c235499 View commit details
    Browse the repository at this point in the history

  8. remove tensorflow hub from this PR

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    6eebfa4 View commit details
    Browse the repository at this point in the history

  9. Add _validate_transform method

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    c27aabb View commit details
    Browse the repository at this point in the history

  10. add more tests

    @AnandInguva
    AnandInguva committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    422a86a View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. fix test

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    08b3665 View commit details
    Browse the repository at this point in the history

  2. Fix test

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    91255ad View commit details
    Browse the repository at this point in the history

  3. Add more tests in sentence-transformer

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    c7237c3 View commit details
    Browse the repository at this point in the history

  4. use np.max instead of max

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    a942885 View commit details
    Browse the repository at this point in the history

  5. round to 2 decimals

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    89c19fb View commit details
    Browse the repository at this point in the history

  6. Remove gradle command action

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    2db4a20 View commit details
    Browse the repository at this point in the history

  7. Refactor throwing dataflow client exception

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    b7a48d5 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu… 

    …va/beam into embeddings_mltransform
    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    eb46e08 View commit details
    Browse the repository at this point in the history

  9. skip the test if gcp is not installed

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    bad1b3b View commit details
    Browse the repository at this point in the history

  10. remove toxTests for hub

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    b850cee View commit details
    Browse the repository at this point in the history

  11. remove toxTests for hub

    @AnandInguva
    AnandInguva committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    ffff21a View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2023

  1. Fix values in assert for sentence_transformer_test

    @AnandInguva
    AnandInguva committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    88412ea View commit details
    Browse the repository at this point in the history

  2. rename sentence_transformers to huggingface

    @AnandInguva
    AnandInguva committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    617f9d6 View commit details
    Browse the repository at this point in the history

  3. fix pydocs

    @AnandInguva
    AnandInguva committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    5cae04b View commit details
    Browse the repository at this point in the history

  4. Change the model name for tests since it is getting different results… 

    … on different machines
    @AnandInguva
    AnandInguva committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    489200f View commit details
    Browse the repository at this point in the history

  5. Fix pydoc in vertexai

    @AnandInguva
    AnandInguva committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    816174a View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2023

  1. add suffix to artifact_location

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    cfb1883 View commit details
    Browse the repository at this point in the history

  2. Revert "add suffix to artifact_location" 

    This reverts commit cfb1883.
    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    2cb6f03 View commit details
    Browse the repository at this point in the history

  3. add no_xdist

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    cd7050e View commit details
    Browse the repository at this point in the history

  4. Try fixing pydoc for vertexai

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    98cd949 View commit details
    Browse the repository at this point in the history

  5. change tox.ini to use pytest directly

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    8ea0906 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/master' into embeddings_mltransform

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    5187b0e View commit details
    Browse the repository at this point in the history

  7. raise FileExistError if Attribute file is already present

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    6f83d3c View commit details
    Browse the repository at this point in the history

  8. Merge branch 'embeddings_mltransform' of https://github.com/AnandIngu… 

    …va/beam into embeddings_mltransform
    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    c9ddb25 View commit details
    Browse the repository at this point in the history

  9. modify build.gradle to match tox task names

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    9dce3cf View commit details
    Browse the repository at this point in the history

  10. Add note to CHANGES.md

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    539c9ad View commit details
    Browse the repository at this point in the history

  11. change gcs bucket to gs://temp-storage-for-perf-tests

    @AnandInguva
    AnandInguva committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    b967cd8 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. Add TODO GH links

    @AnandInguva
    AnandInguva committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    f1bb42c View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into embeddings_mltransform

    @AnandInguva
    AnandInguva committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    8d0b47d View commit details
    Browse the repository at this point in the history

  3. Update CHANGES.md 

    Co-authored-by: Danny McCormick <dannymccormick@google.com>
    @AnandInguva @damccorm
    AnandInguva and damccorm authored Dec 11, 2023
    Configuration menu
    Copy the full SHA
    c173d6a View commit details
    Browse the repository at this point in the history