Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature(copy): add copy option commit_paths. #11551

Open
youngsofun opened this issue May 23, 2023 · 1 comment
Open

Feature(copy): add copy option commit_paths. #11551

youngsofun opened this issue May 23, 2023 · 1 comment
Labels
C-feature Category: feature

Comments

@youngsofun
Copy link
Member

youngsofun commented May 23, 2023

Summary

currently we support 2 options:

  1. force, false by default. if true:1. not check meta when list files, 2. not return an error if found dup when committing paths.
  2. purge, false by default. if true, try delete copied files after committing. note that deleting can only be a best effort.

since purge is best effort. we rely on meta (force=false) to make sure we will not commit the same file twice.

some times, we may need to reload some files ( or reuse paths?). so we need force=true.
checking and committing paths may be slow when there are many small files.
so we used to introduce a feature: #10855, [purge] and [force] are both enabled, will not update copied-files set.
this is obscure to user, for completeness and orthogonality, I think it is better to add another copy option for this. Like force, it is an Advanced option.

after copy support return copied files lists,#7730. users may be able to dedup the files by themselves.

@youngsofun youngsofun added the C-feature Category: feature label May 23, 2023
@youngsofun
Copy link
Member Author

cc @dantengsky

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

1 participant