-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make ZSTD default compression for Parquet writes #4726
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🙌
Codecov Report
@@ Coverage Diff @@
## main #4726 +/- ##
==========================================
+ Coverage 61.26% 61.78% +0.51%
==========================================
Files 286 288 +2
Lines 10572 10598 +26
Branches 776 758 -18
==========================================
+ Hits 6477 6548 +71
+ Misses 4095 4050 -45
|
@clairemcginty what the state of this one ? |
@RustedBones ready to merge for Scio 0.13 👍 |
(fix #4698)
The Parquet Java library supports ZSTD with a default level of 3 (doc) based off the zstd-jni library; level can be customized using the Configuration
parquet.compression.codec.zstd.level
.ZSTD has been shown to have moderate performance improvements over GZIP or Snappy, and is supported for BigQuery loads.