Skip to content
/ file_backup Public

Command line application to copy folder content from one location to another.

License

GPL-2.0 license
0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

Denis-Source/file_backup

BranchesTags

Repository files navigation

File Backup

About

Command line application to copy the folder content from one location to another.

Location is not restricted to the local file system as there is an ability to choose both input and output handlers.

All the handlers can be used in an arbitrary order.

Application has the following handlers:

  • "vanilla" file system;
  • remote SFTP server;
  • Google Drive;
  • DropBox.

SFTP handler requires server configurations (IP address, key file location), GDrive handler requires credentials for the V3 API and Dropbox requires authorization token.

Used libraries

  • googleapiclient
  • pysftp
  • os
  • dropbox

LocalHandler mainly uses built-in os library as it provides tools for the folder creation, file system inromation retriving. SFTPHandler uses pysftp library that is very similar to the os library GDriveHandler uses GoogleDrive V3 API. DropBoxHandler uses DropBox Python SDK for API v2

Installation

git clone https://github.com/Denis-Source/file_backup.git
python3 -m venv env/
source env/bin/activate
python main.py -i {input_folder} -o {output_folder}

Showcase

The console application has the following signature:

usage: main.py [-h] [-ih INPUT_HANDLER] [-oh OUTPUT_HANDLER] [-i INPUT] -o OUTPUT [-v]

File Backup command line App
optional arguments:
  -h, --help                 show this help message and exit
  -ih, --input_handler       type of a handler as an input
  -oh, --output_handler				type of a handler as an output
  -i, --input                input location
  -o, --output               output location
  -v, --validation           use validators defined in the configs

All the examples below were provided using WSL to avoid mixing of UNIX and Windows file system path formats.

Local Handler

Considering we have the following file tree structure:

~/samples$ tree
.
├── BuzzBuzz_edited.wav
├── SX_Door_origin.wav
├── SX_Phone_origin.wav
└── edited
    └── Wow_Edited.wav

1 directory, 4 files

By Default if input or output handlers are not specified, the local handler will be used

We can copy the folder content from one location to another, using -i and -o keys:

python3 main.py -i ~/samples/ -o /mnt/c/file_backup/samples

The folders content were copied to the specified output folder:

tree /mnt/c/file_backup/samples/
/mnt/c/file_backup/samples/
├── BuzzBuzz_edited.wav
├── SX_Door_origin.wav
├── SX_Phone_origin.wav
└── edited
    └── Wow_Edited.wav

1 directory, 4 files

The logging output:

2022-08-27 17:13:04,560 INFO    app             Backuping files from /home/den/samples/ using local handler to /mnt/c/file_backup/samples using local handler not using validators
2022-08-27 17:13:04,560 INFO    local           Got information about a folder at /home/den/samples/
2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:13:04,561 INFO    local           Got information about a folder at /home/den/samples//edited
2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:13:04,561 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:13:04,561 INFO    local           Creating folder at /mnt/c/file_backup/samples
2022-08-27 17:13:04,566 INFO    local           Got information about a folder at /mnt/c/file_backup/samples
2022-08-27 17:13:04,567 INFO    local           Сopying File at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:13:04,569 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read
2022-08-27 17:13:04,571 INFO    local           Created new file at /mnt/c/file_backup/samples/SX_Door_origin.wav
2022-08-27 17:13:04,572 INFO    local           Сopying Folder at /home/den/samples//edited
2022-08-27 17:13:04,572 INFO    local           Creating folder at /mnt/c/file_backup/samples/edited
2022-08-27 17:13:04,579 INFO    local           Got information about a folder at /mnt/c/file_backup/samples/edited
2022-08-27 17:13:04,580 INFO    local           Сopying File at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:13:04,581 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read
2022-08-27 17:13:04,584 INFO    local           Created new file at /mnt/c/file_backup/samples/edited/Wow_Edited.wav
2022-08-27 17:13:04,586 INFO    local           Сopying File at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:13:04,588 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read
2022-08-27 17:13:04,590 INFO    local           Created new file at /mnt/c/file_backup/samples/SX_Phone_origin.wav
2022-08-27 17:13:04,591 INFO    local           Сopying File at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:13:04,594 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read
2022-08-27 17:13:04,596 INFO    local           Created new file at /mnt/c/file_backup/samples/BuzzBuzz_edited.wav
2022-08-27 17:13:04,598 INFO    app             Done in 0:00:00!

SFTP Handler

We can specify sftp both as the input or output handlers using -ih and -oh:

To use SFTP it is required to configure the following constants in the config.py file: HOST, USERNAME, KEY LOCATION.

To upload the previous local samples/ folder to the SFTP server, we should use the following command:

python main.py -i ~/samples/ -o /backups/samples -oh sftp

Program logging:

2022-08-27 17:34:14,438 INFO    app             Backuping files from /home/den/samples/ using local handler to backups/samples using local handler not using validators
2022-08-27 17:34:14,439 INFO    local           Got information about a folder at /home/den/samples/
2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:34:14,439 INFO    local           Got information about a folder at /home/den/samples//edited
2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:34:14,439 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:34:14,440 INFO    sftp            Creating folder at backups/samples
2022-08-27 17:34:14,771 INFO    sftp            Got information about a folder at backups/samples
2022-08-27 17:34:14,771 INFO    sftp            Сopying File at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:34:14,790 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read
2022-08-27 17:34:14,901 INFO    sftp            Created new file at backups/samples/SX_Door_origin.wav
2022-08-27 17:34:14,901 INFO    sftp            Сopying Folder at /home/den/samples//edited
2022-08-27 17:34:14,901 INFO    sftp            Creating folder at backups/samples/edited
2022-08-27 17:34:15,036 INFO    sftp            Got information about a folder at backups/samples/edited
2022-08-27 17:34:15,037 INFO    sftp            Сopying File at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:34:15,055 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read
2022-08-27 17:34:15,176 INFO    sftp            Created new file at backups/samples/edited/Wow_Edited.wav
2022-08-27 17:34:15,177 INFO    sftp            Сopying File at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:34:15,197 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read
2022-08-27 17:34:15,259 INFO    sftp            Created new file at backups/samples/SX_Phone_origin.wav
2022-08-27 17:34:15,259 INFO    sftp            Сopying File at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:34:15,279 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read
2022-08-27 17:34:15,459 INFO    sftp            Created new file at backups/samples/BuzzBuzz_edited.wav
2022-08-27 17:34:15,460 INFO    app             Done in 0:00:01!

The remote server has the following file structure:

root@vaua0073687:~/backups# tree ~/backups/
/root/backups/
└── samples
    ├── BuzzBuzz_edited.wav
    ├── edited
    │   └── Wow_Edited.wav
    ├── SX_Door_origin.wav
    └── SX_Phone_origin.wav

2 directories, 4 files

We can also swap the handlers:

python main.py -o ~/samples/ -i backups/samples -ih sftp -oh local

If -ih or -oh keys are not specified, the local handler will be used

Google Drive Handler

We can use Google Drive Handler to upload or download folders.

To use GDrive handler, we should provide secrets.json file.

To upload samples/ folder we can use the following command:

python main.py -i ~/samples/ -o backups/samples -oh gdrive

Program output:

2022-08-27 17:43:17,764 INFO    app             Backuping files from /home/den/samples/ using local handler to backups/samples using local handler not using validators
2022-08-27 17:43:17,764 INFO    local           Got information about a folder at /home/den/samples/
2022-08-27 17:43:17,764 INFO    local           Got information about a file at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:43:17,765 INFO    local           Got information about a folder at /home/den/samples//edited
2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:43:17,765 INFO    local           Got information about a file at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:43:17,765 INFO    gdrive          Creating folder at backups/samples
2022-08-27 17:43:18,461 INFO    gdrive          Found records with name backups in folder 0AH3CXGph3HmOUk9PVA: 0
2022-08-27 17:43:19,363 INFO    gdrive          Created folder backups at
2022-08-27 17:43:19,640 INFO    gdrive          Found records with name samples in folder 1N9LMu8J4rQrLNwTYWcWh67FAF6a4Y3gs: 0
2022-08-27 17:43:20,254 INFO    gdrive          Created folder samples at /backups
2022-08-27 17:43:20,254 INFO    gdrive          Сopying File at /home/den/samples//SX_Door_origin.wav
2022-08-27 17:43:20,595 INFO    gdrive          Found records with name SX_Door_origin.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0
2022-08-27 17:43:20,595 INFO    local           File at /home/den/samples//SX_Door_origin.wav is read
2022-08-27 17:43:21,692 INFO    gdrive          Сopying Folder at /home/den/samples//edited
2022-08-27 17:43:21,692 INFO    gdrive          Creating folder at backups/samples/edited
2022-08-27 17:43:22,304 INFO    gdrive          Found records with name backups in folder 0AH3CXGph3HmOUk9PVA: 1
2022-08-27 17:43:22,304 INFO    gdrive          Folder backups is already exists at
2022-08-27 17:43:22,569 INFO    gdrive          Found records with name samples in folder 1N9LMu8J4rQrLNwTYWcWh67FAF6a4Y3gs: 1
2022-08-27 17:43:22,569 INFO    gdrive          Folder samples is already exists at /backups
2022-08-27 17:43:22,839 INFO    gdrive          Found records with name edited in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0
2022-08-27 17:43:23,370 INFO    gdrive          Created folder edited at /backups/samples
2022-08-27 17:43:23,370 INFO    gdrive          Сopying File at /home/den/samples//edited/Wow_Edited.wav
2022-08-27 17:43:23,628 INFO    gdrive          Found records with name Wow_Edited.wav in folder 174fISaVT9lqcSKnHXLMTEoaTHifMiBXt: 0
2022-08-27 17:43:23,629 INFO    local           File at /home/den/samples//edited/Wow_Edited.wav is read
2022-08-27 17:43:24,820 INFO    gdrive          Сopying File at /home/den/samples//SX_Phone_origin.wav
2022-08-27 17:43:25,085 INFO    gdrive          Found records with name SX_Phone_origin.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0
2022-08-27 17:43:25,086 INFO    local           File at /home/den/samples//SX_Phone_origin.wav is read
2022-08-27 17:43:26,264 INFO    gdrive          Сopying File at /home/den/samples//BuzzBuzz_edited.wav
2022-08-27 17:43:26,498 INFO    gdrive          Found records with name BuzzBuzz_edited.wav in folder 1SgfUNnq8JVNNvJL5c1c9rMseiZ0oVpd2: 0
2022-08-27 17:43:26,498 INFO    local           File at /home/den/samples//BuzzBuzz_edited.wav is read
2022-08-27 17:43:27,634 INFO    app             Done in 0:00:10!

Google Drive folder contents: google drive folder

DropBox

We can also use DropBox Handler to copy the previously mentioned samples from the Google Drive to DropBox.

In order to function, the API token should be generated from application console with the needed permissions. The token itself should be specified in the configs.

To copy samples/ folder we can use the following command:

python main.py  -i backups -ih gdrive -o backups -oh dropbox

Program output:

2022-11-19 06:47:12,604 INFO    app             Backuping files from backups using gdrive handler to backups using gdrive handler not using validators
2022-11-19 06:47:12,928 INFO    gdrive          Got id about the root folder: 0AH3CXGph3HmOUk9PVA
2022-11-19 06:47:14,925 INFO    dropbox         deleted already created folder
2022-11-19 06:47:16,355 INFO    gdrive          Request to read file backups/samples/BuzzBuzz_edited.wav is successful
2022-11-19 06:47:16,355 INFO    gdrive          Downloading file in memory
2022-11-19 06:47:17,130 INFO    gdrive          Download is successful
2022-11-19 06:47:18,584 INFO    dropbox         Uploaded file backups/samples/BuzzBuzz_edited.wav
2022-11-19 06:47:18,584 INFO    gdrive          Request to read file backups/samples/SX_Phone_origin.wav is successful
2022-11-19 06:47:18,584 INFO    gdrive          Downloading file in memory
2022-11-19 06:47:19,142 INFO    gdrive          Download is successful
2022-11-19 06:47:20,109 INFO    dropbox         Uploaded file backups/samples/SX_Phone_origin.wav
2022-11-19 06:47:20,972 INFO    gdrive          Request to read file backups/samples/edited/Wow_Edited.wav is successful
2022-11-19 06:47:20,972 INFO    gdrive          Downloading file in memory
2022-11-19 06:47:21,600 INFO    gdrive          Download is successful
2022-11-19 06:47:23,160 INFO    dropbox         Uploaded file backups/samples/edited/Wow_Edited.wav
2022-11-19 06:47:23,168 INFO    gdrive          Request to read file backups/samples/SX_Door_origin.wav is successful
2022-11-19 06:47:23,168 INFO    gdrive          Downloading file in memory
2022-11-19 06:47:23,817 INFO    gdrive          Download is successful
2022-11-19 06:47:24,866 INFO    dropbox         Uploaded file backups/samples/SX_Door_origin.wav
2022-11-19 06:47:24,866 INFO    app             Done in 0:00:12!

DropBox folder contents:

dropbox_example

Validators

If we try to copy the contents of the source folder:

python main.py -o /mnt/c/file_backup/source_code

We will see that the folder has ~14k files:

tree /mnt/c/file_backup/source_code/ | wc -l
14037

If -i key is not specified, it will copy the current folder.

We can filter out unnecessary files (.pyc binaries, .env files, etc) by setting file\folder filtering using config.py fle using regular expressions:

EXCLUDED_NAMES = [
    r"^__",
    r"^env",
    r"^venv",
    r"^\.",
    "^lib$",
]
EXCLUDED_FORMATS = [
    "pyc",
    "pyd",
]

To use them, we simply provide -v key:

python main.py -o /mnt/c/file_backup/source_code -v

Now the output folder contains only 22 files:

tree /mnt/c/file_backup/source_code/ | wc -l
22

About

Command line application to copy folder content from one location to another.

Topics

python backup command-line filesystem sftp google-drive-api-v3

Resources

Readme

License

GPL-2.0 license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Languages