Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for compressions and make LZ4 default for backup 2.0 #833

Open
wants to merge 5 commits into
base: 3.x
Choose a base branch
from

Conversation

arunagrawal84
Copy link
Contributor

@arunagrawal84 arunagrawal84 commented Sep 20, 2019

We have supported Snappy as the only compression type while uploading the files. We wanted to support customizable compression types and thus support for LZ4 and no compression.
In this PR we make LZ4 as the default compression type for backup 2.0
Future work: Do not compress Data.db which are already compressed.

In this PR we have encapsulated the directives on how to upload or download a file to class UploadDownloadDirectives. This includes directives like compression, encryption, retries to use etc.

@codecov
Copy link

codecov bot commented Sep 20, 2019

Codecov Report

Merging #833 into 3.x will increase coverage by 0.45%.
The diff coverage is 65.43%.

Impacted file tree graph

@@             Coverage Diff              @@
##                3.x     #833      +/-   ##
============================================
+ Coverage     46.56%   47.01%   +0.45%     
- Complexity     1052     1063      +11     
============================================
  Files           166      169       +3     
  Lines          7280     7393     +113     
  Branches        746      760      +14     
============================================
+ Hits           3390     3476      +86     
- Misses         3638     3660      +22     
- Partials        252      257       +5
Impacted Files Coverage Δ Complexity Δ
.../java/com/netflix/priam/backup/SnapshotBackup.java 38.55% <ø> (ø) 13 <0> (ø) ⬇️
...com/netflix/priam/tuner/CassandraTunerService.java 100% <ø> (ø) 4 <0> (ø) ⬇️
...va/com/netflix/priam/backup/IBackupFileSystem.java 0% <ø> (ø) 0 <0> (ø) ⬇️
...va/com/netflix/priam/backup/IncrementalBackup.java 77.77% <ø> (ø) 13 <0> (ø) ⬇️
...n/java/com/netflix/priam/aws/S3FileSystemBase.java 64.16% <ø> (-0.3%) 20 <0> (ø)
...java/com/netflix/priam/backup/CommitLogBackup.java 0% <0%> (ø) 0 <0> (ø) ⬇️
...va/com/netflix/priam/resources/CassandraAdmin.java 0% <0%> (ø) 0 <0> (ø) ⬇️
.../netflix/priam/backupv2/MetaFileWriterBuilder.java 70.49% <0%> (-2.39%) 3 <0> (ø)
...a/com/netflix/priam/aws/S3EncryptedFileSystem.java 0% <0%> (ø) 0 <0> (ø) ⬇️
...om/netflix/priam/restore/EncryptedRestoreBase.java 0% <0%> (ø) 0 <0> (ø) ⬇️
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7f216c...645c058. Read the comment docs.

* this to sometimes C* creating files which are zero bytes, and giving that in
* snapshot for some unknown reason.
*/
if (chunk.length > 0) rateLimiter.acquire(chunk.length);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we still want to proceed with the upload of this 0 byte file to keep our s3 snapshots consistent with C* snapshot view, and hence we do not skip the rest of the upload code path even if the file has 0 byte - is this correct?


// Cache hit. Return the value.
if (cacheResult != null) return cacheResult;
if (lastUpdatedTimestamp != null) return true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we do not really check the value of the timestamp, what difference does this make relative to having a Boolean? Maybe I will discover its importance in other fails, if I do, will delete this comment.

import java.time.temporal.ChronoUnit;
import javax.crypto.KeyGenerator;

public class s3test {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has a lot of commented code. Any reason for keeping the commented code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants