Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added multipart upload, added optional encryption/decryption. #24

Closed
wants to merge 85 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
7ba14fa
.gitignore added.
Mar 1, 2013
68af3c1
Multipart upload added.
Mar 1, 2013
6a90f7f
Importing default part size from boto.
Mar 1, 2013
4be17d7
Added option to specify number of threads in multipart upload.
Mar 1, 2013
ec13d93
Using ConcurrentUploader directly to upload file to glacier.
Mar 12, 2013
6f7174c
First shot at GPG encryption/decryption of transferred messages.
Mar 13, 2013
389a1b8
Fixed decryption when retrieving file.
Mar 13, 2013
85cc6b4
Fixed missing encryptor.
Mar 13, 2013
88e499f
Add some docstrings.
Mar 15, 2013
724f65f
Added option to disable encryption/decryption.
Mar 15, 2013
806c31e
Using command line option to decide whether to encrypt/decrypt or not.
Mar 15, 2013
7f30654
Updated readme.
Mar 15, 2013
d5af2ed
Added setup.py
Mar 15, 2013
720bdbd
Fix package names in setup file.
Mar 15, 2013
e8a25bb
Fixed setup.py one more time.
Mar 15, 2013
dbae589
Merge public master with counsyl encryption additions.
sakoht Nov 22, 2013
2af4c6f
remove comments from merge
sakoht Nov 22, 2013
724d692
fix handling of file name vs obj
sakoht Nov 22, 2013
b51724a
fix handling of file name vs obj
sakoht Nov 22, 2013
bfcdaa4
set DEFAULT_NUM_THREADS to 10 as boto does internally
sakoht Nov 23, 2013
d4088e6
Ensure archive multipart_upload passes the multipart flag.
sakoht Nov 23, 2013
635fbb4
Cache encrypted file for retries until upload completes.
sakoht Nov 23, 2013
30a86c0
cleanup of new code around encryption temp files
sakoht Nov 24, 2013
a72a504
fix temporarily disabled check around --encryption-dir
sakoht Nov 24, 2013
20dd737
remove extra argument checking: not operating cleanly
sakoht Nov 24, 2013
f11cf78
return to default gnupg key handling (use homedir, override with env)
sakoht Nov 25, 2013
d3bb117
Remove --encryption-directory.
sakoht Dec 2, 2013
b578431
add whitespace around operator
sakoht Dec 2, 2013
e5db4d8
whitespace around operator
sakoht Dec 2, 2013
1b99931
Merge pull request #6 from dev/101_setup_aws_glacier_for_uploading_ba…
sakoht Dec 4, 2013
0643e9a
Add koality
Dec 10, 2013
a6c3aed
add mock
Dec 10, 2013
0deb231
add MIT license
sakoht Dec 11, 2013
8bd36ad
make mutipart_upload a parameter to upload instead of a separate command
sakoht Dec 11, 2013
5bba053
Merge pull request #9 from dev/add_koality
Dec 11, 2013
3319d20
set type for archive upload file parameter
sakoht Dec 11, 2013
857d450
restore support for stdin streaming for "-"
sakoht Dec 11, 2013
8f248c2
renamed callback to "sortable_date"
sakoht Dec 12, 2013
5ae273a
Merge branch 'master' into 7_sync_with_public_project
sakoht Dec 12, 2013
d14d4da
Remove codequality from koality
Dec 12, 2013
e0cc3ad
test archive upload w/ encrypt, multi-part
sakoht Dec 12, 2013
de61cf4
Merge branch 'master' into 7_sync_with_public_project
sakoht Dec 12, 2013
dc688a2
Get glacier_test.py working. (Failing on master.)
sakoht Dec 12, 2013
e9f1736
add multipart upload unit test
sakoht Dec 12, 2013
2627cc1
test cleanup
sakoht Dec 12, 2013
3839287
remove koality.yml
sakoht Dec 12, 2013
863a74b
warn the user that multipart uploads do not support streaming from stdin
sakoht Dec 12, 2013
1f29e41
gnupg test keys
sakoht Dec 12, 2013
1958366
rename --multi-part to --multipart
sakoht Dec 12, 2013
07a6348
restore koality.yml
sakoht Dec 12, 2013
c3b1564
Updated setup.py
chrisgilmerproj Dec 13, 2013
c8c58fb
Fix missing comma
chrisgilmerproj Dec 13, 2013
426e6b2
Merge branch 'master' into 7_sync_with_public_project
sakoht Dec 21, 2013
f10aa31
leave stubs for additional tests
sakoht Dec 21, 2013
d7bc036
Add ConcurrentDownloader, rename --multipart to --concurrent.
sakoht Dec 21, 2013
d7c5c5e
Merge branch '7_sync_with_public_project' of github.counsyl.com:dev/g…
sakoht Dec 21, 2013
81b6d89
fix docstring
sakoht Dec 21, 2013
232e5ec
add additional spaces for pep8
sakoht Dec 21, 2013
e68a102
clarify code around temp file naming
sakoht Dec 21, 2013
c765a16
make multipart downloads use the concurrent downloader by default
sakoht Dec 21, 2013
f866664
rename --multi-part to --concurrent in test case
sakoht Dec 31, 2013
ce133df
remove code to explictly delete tmp dir
sakoht Jan 2, 2014
e1a9114
test combinations of encryption and concurrency with uploading
sakoht Jan 2, 2014
8232f81
test combinations of --encrypt and --concurrent when uploading
sakoht Jan 2, 2014
06f5f38
update setup.py to wrap duplicate requirements{,-dev}.txt
sakoht Jan 2, 2014
96b5fa9
update koality.yml to run the test case correctly
sakoht Jan 2, 2014
6c2ead1
update tmpdir check to work on Mac, not expect /tmp
sakoht Jan 2, 2014
efeee9f
Configure the test case to use pre-generated testing GPNUPG keys.
sakoht Jan 3, 2014
287fdab
updated random seet for gnupg_test_home
sakoht Jan 3, 2014
e21681f
fix indentation bug preventing final output after decrypt
sakoht Jan 7, 2014
24b741a
remove commented-out code
sakoht Jan 7, 2014
9c81466
verify that incorrect combinations of new args raise exceptions
sakoht Jan 7, 2014
59f31bd
removed extra line
sakoht Jan 14, 2014
929abb1
shorten new lines that are too long
sakoht Jan 14, 2014
2afd841
shorten new lines that are too long
sakoht Jan 14, 2014
bffdeb6
Merge pull request #10 from dev/7_sync_with_public_project
sakoht Jan 14, 2014
9211c64
rename --multipart-size to --part-size
sakoht Jan 14, 2014
d8bd041
update README.md to reflect new args
sakoht Jan 14, 2014
e13c63f
fix typo in README.md update
sakoht Jan 14, 2014
12b2c37
update [] around optional --concurrent arg
sakoht Jan 14, 2014
62ef6cb
Merge pull request #12 from dev/11_rename_mulitpart_size_to_part_size
sakoht Jan 14, 2014
a3b68ef
allow glacier-cli to use newer sqlalchemy, but not require it
sakoht Jan 21, 2014
7dd297b
Update requirements-dev.txt
Feb 5, 2014
9a70cf2
Update requirements-dev.txt
Feb 7, 2014
58721b4
Update version to 0.1.1.alpha
Feb 14, 2014
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.pyc
*~
*#
4 changes: 4 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
include COPYING
include MANIFEST.in
include README.md
include *.py
49 changes: 49 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#! /usr/bin/make

default:
python setup.py check build

.PHONY: register dist inspect upload clean docs test

register:
if [ ! -f ~/.pypirc ]; then \
echo "Missing ~/.pypirc file"; \
exit 1; \
fi; \
python setup.py register

dist:
python setup.py sdist

inspect:
python setup.py clean
rm -rf build/
rm -rf dist/
rm -rf *.egg/
rm -rf *.egg-info/
rm -f MANIFEST
python setup.py sdist
cd dist/ && tar xzvf *.tar.gz

upload:
if [ ! -f ~/.pypirc ]; then \
echo "Missing ~/.pypirc file"; \
exit 1; \
fi; \
python setup.py sdist upload

clean:
python setup.py clean
rm -rf build/
rm -rf dist/
rm -rf *.egg/
rm -rf *.egg-info/
rm -rf __pycache__/
rm -f MANIFEST
rm -rf docs/_*/

docs:
cd docs && make html

test:
nosetests -c tests/nose.cfg
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,9 +211,8 @@ Commands
* <code>glacier vault create <em>vault-name</em></code>
* <code>glacier vault sync [--wait] [--fix] [--max-age <em>hours</em>] <em>vault-name</em></code>
* <code>glacier archive list <em>vault-name</em></code>
* <code>glacier archive upload [--name <em>archive-name</em>] <em>vault-name</em> <em>filename</em></code>
* <code>glacier archive retrieve [--wait] [-o <em>filename</em>] [--multipart-size <em>bytes</em>] <em>vault-name</em> <em>archive-name</em></code>
* <code>glacier archive retrieve [--wait] [--multipart-size <em>bytes</em>] <em>vault-name</em> <em>archive-name</em> [<em>archive-name</em>...]</code>
* <code>glacier archive upload [--encrypt] [--concurrent [--part-size <em>size</em>] [--num-threads <em>count</em>]] [--name <em>archive-name</em>] <em>vault-name</em> <em>filename</em></code>
* <code>glacier archive retrieve [--wait] [--decrypt] [-o <em>filename</em>] [--part-size <em>bytes</em>] <em>vault-name</em> <em>archive-name</em> [<em>archive-name</em>...]</code>
* <code>glacier archive delete <em>vault-name</em> <em>archive-name</em></code>
* <code>glacier job list</code>

Expand Down
Loading