Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relax transform coder equality #4780

Merged
merged 4 commits into from
Apr 19, 2023
Merged

Relax transform coder equality #4780

merged 4 commits into from
Apr 19, 2023

Conversation

RustedBones
Copy link
Contributor

Streaming pipeline using coders created with xmap fail to update with:

Workflow failed. Causes: The new job is not compatible with xxx. The original job (if it existed) has not been aborted., The Coder or type for step xxx has changed.

TransformCoder was defined the same way as DelegateCoder from beam, making sure underlying functions were equal.

This is very unlikely that user define several coders with same type signature with xmap (and will likely result in implicit conflict).

Relax the serialization/equality so pipeline upgrade pass the validation.

@codecov
Copy link

codecov bot commented Apr 13, 2023

Codecov Report

Merging #4780 (a68777a) into main (01ca7c8) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

❗ Current head a68777a differs from pull request most recent head 475b7b2. Consider uploading reports for the commit 475b7b2 to get more accurate results

@@            Coverage Diff             @@
##             main    #4780      +/-   ##
==========================================
- Coverage   61.21%   61.18%   -0.04%     
==========================================
  Files         286      286              
  Lines       10562    10553       -9     
  Branches      773      769       -4     
==========================================
- Hits         6466     6457       -9     
  Misses       4096     4096              
Impacted Files Coverage Δ
...in/scala/com/spotify/scio/values/SCollection.scala 94.02% <ø> (ø)
...src/main/scala/com/spotify/scio/coders/Coder.scala 85.26% <100.00%> (-0.39%) ⬇️
...a/com/spotify/scio/util/random/RandomSampler.scala 96.42% <100.00%> (ø)
...y/scio/values/PairSkewedSCollectionFunctions.scala 96.85% <100.00%> (-0.08%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@RustedBones RustedBones requested a review from a team April 14, 2023 11:49
@RustedBones RustedBones merged commit 5945d90 into main Apr 19, 2023
@RustedBones RustedBones deleted the relax-transform-coder branch April 19, 2023 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants