change-streams

Change stream

This is an early release feature supporting change streams. Change streams are streams that contain CDC data for materializing views in a database. Below is the e2e pipeline.

flowchart TD;
    dg{{DataGen}}-->raw_stream-->pl[[pipeline_aggregation]]-->change_stream-->
    pg[(Postgres_Materialized_View)];

Loading

Getting Started

This repo requires the decodable cli. Installation instructions can be found here.

Environment

Create a .env file locally and provide values to the variables.

HOST={{ POSTGRES_HOST_NAME }}
DB={{ POSTGRES_DATABASE }}
USER={{ POSTGRES_USER }}
PWD={{ POSTGRES_PASSWORD }}

ACCOUNT={{ DECODABLE_ACCOUNT_NAME }}

Postgres

Log into your Postgres and execute this file.

psql -h $(HOST) -d $(DB) -U $(USER) -W -f sql/postgres.sql

Startup

From the command line, execute this make command:

$ make pipeline
$ make active

Clean up

$ make clean