Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add whitelisting #22

Merged
merged 5 commits into from
Apr 15, 2019
Merged

add whitelisting #22

merged 5 commits into from
Apr 15, 2019

Conversation

nick-mccoy
Copy link
Contributor

@nick-mccoy nick-mccoy commented Apr 11, 2019

Since the Stripe events stream has a ton of nested fields, it can cause the number of unique fields of a batch of records to exceed 1600. To mitigate this, this PR introduces a new, optional config field called whitelist_map that takes the following form:

{
   <stream_name>: [
      [<field_1_breadcrumb>],
      [<field_2_breadcrumb>],
      ...
      [<field_n_breadcrumb>]
   ]
}

for example:

{
    "events": [
        ["data", "object", "customer"],
        ["data", "object", "id"],
        ["data", "object", "plan", "id"],
        ["data", "object", "plan", "amount"],
        ["data", "object", "plan", "created"],
        ["data", "object", "plan", "interval"],
        ["data", "object", "plan", "name"],
        ["data", "object", "plan", "object"],
        ["data", "object", "created"],
        ["data", "object", "start"],
        ["data", "object", "quantity"],
        ["data", "object", "status"],
        ["data", "object", "subscription"],
        ["data", "object", "amount"],
        ["data", "object", "canceled_at"],
        ["data", "object", "payment"],
        ["data", "object", "metadata", "email"],
        ["data", "object", "metadata", "orgid"],
        ["data", "previous_attributes", "plan", "created"],
        ["data", "previous_attributes", "plan", "id"],
        ["data", "previous_attributes", "plan", "name"],
        ["data", "previous_attributes", "status"],
        ["data", "previous_attributes", "plan", "object"],
        ["data", "previous_attributes", "quantity"],
        ["data", "previous_attributes", "plan", "amount"],
    ]
}

Note that all breadcrumbs should have more than one entry since top level fields can already be deselected by the user. The tap asserts that all breadcrumbs have a length > 1 and will therefore error if it receives a breadcrumb of length = 1.

Copy link
Contributor

@dmosorast dmosorast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@nick-mccoy nick-mccoy merged commit 3aa9531 into master Apr 15, 2019
@nick-mccoy nick-mccoy deleted the add_whitelisting branch April 15, 2019 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants