Skip to content
This repository has been archived by the owner on Jan 23, 2020. It is now read-only.

clari/clari_dynamo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

88 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

*Still in early alpha and limited to puts and gets. Goal is to achieve extremely robust functionality before adding more features.

ClariDynamo - DynamoDB with a cherry on top

Wrapper service providing several high level features on top of DynamoDB, S3, KMS and other Amazon services, served with CherryPy.

Features

  • Autoscales per request and in background based on QPS via dynamic-dynamodb
  • Automatic schema migrations
  • Multi-tenant
  • Multi-environment
  • S3 fields for large objects (encrypted with KMS)
  • Installs and runs dynamo local automatically
  • Easy, fast test environments with in-memory DB
  • Each operation tracked, timed, and logged with app-level purpose
  • Timestamped created_at and updated_at for all entities
  • Deploy right now on Heroku - TODO: Add Heroku button

Setup

  • Get anaconda python 2.7 http://continuum.io/downloads
  • Java 6+ for DynamoDB local
  • pip install -r requirements.txt
  • cp conf/secrets.example.py conf/secrets.py and setup your secrets

Migrations

  • Add new with python new_migration.py
  • Runs at server start (TODO: in background)
  • Retries failed migrations on restart

Python 3

  • Everything is Python3 ready except for dynamic-dynamodb which is on a separate worker container

TODO

[ ] Run migrations in background [ ] S3 backups [ ] Use IAM credentials in non-dev environments instead of basic auth [ ] Queries, scans, and underlying paging ability. [ ] Add hash/range key hash to encrypted_tenant_id to prevent duplicates Right now encrypted_tenant_id's are all unique but this is not enforced on read. Most efficient way to do this would be to include hash/range key in encrypted_tenant_id.

Possibilities

  • Instant transformations - map, filter, join, split, move, etc...
  • Spark accelerated migrations and transformations
  • Web UI
  • Replicate to Redshift/Aurora for analytics querying
  • Topic subscriptions
  • Copy on write QA environment

About

Customizable service layer around DynamoDB

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages