-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
V2.2.0 convenient downloadscripts #446
base: main
Are you sure you want to change the base?
V2.2.0 convenient downloadscripts #446
Conversation
…d missing exit statement
…ttle more informative
…directory' errors
…irectory - allowing for faster decompression
possible to use unpigz fixed error, due to pushing into the root dir and then trying to 'mv /<file> ...
…error when pushing to and trying to gunzip /
…tion to parallel uncompress (at least a bit)
In the last two commits, I had to notice by means of a user report, that only the executing user had read permissions. This, of course, needed to be fixed for a multi-user system. Essentially, I did Opinions? |
Hi, I hoped to at least spark a bit of a discussion, as the mentioned issues still persists for multiuser systems. Whether the work with |
Hi,
I tried to add a few features to the download scripts as a remedy to some potentially annoying issues causing tickets and to speed up the download-uncompress processes.
Specifically:
yes
, the scripts will remove theROOT_DIR
first. Else, the scripts will cowardly refuse to proceed. Why? Because triggering a specific download script in error, will else lead to operate again. (might yield some less-annoyed users)pigz
in PATH, uncompressing withpigz
is attempted. The parallelism is NOT in the decompression, however, as the file handling is separated from the decompressing step a minor speed-up can be achieved.find "${RAW_DIR}/" -type f -iname "*.gz" -exec gunzip {} +
, which takes ages to complete. Here, switching tofind "${RAW_DIR}/" -type f -iname "*.gz" -print0 | xargs -0 -P2 "${uncompress_cmd}"
yields a speed-up of about factor 2. The hardcoded-P2
is a bit unfortunate, yet I do not know whether it makes sense to figure out, what parallelism is allowed for the user (e.g. reading number of processors, reading the c-group, taking the minimum value), because much will depend on the file system and the current status (strain) it is in.Your comments are most appreciated. I hope, that you find my contribution worth considering.
Best regards
Christian Meesters