Feature/bq incremental strategy insert_overwrite

[jtcohen6]

This is a small feature that builds on top of the tremendous work from #2140. It shouldn't have any breaking changes, so I think we could ship it in 0.16.1. I'm opening this now so that I can link to it in a forthcoming post about dbt + BigQuery + incremental models.

A common request from the community is an incremental materialization on BigQuery to just drop and replace an entire day of data. By setting incremental_strategy = "insert_overwrite" in the config, any partition with new data will be completely dropped and recreated.

Example usage:

{{ config(
    materialized='incremental',
    partition_by="ts",
    cluster_by="id",
    incremental_strategy = "insert_overwrite"
) }}

with data as (
    select 1 as id, cast('2019-01-01' as date) as ts union all
    select 2 as id, cast('2019-01-02' as date) as ts union all
    select 3 as id, cast('2019-01-02' as date) as ts union all
    select 4 as id, cast('2019-01-03' as date) as ts union all
    select 5 as id, cast('2019-01-04' as date)
)

select *
from data

{% if is_incremental() %}
where ts > _dbt_max_partition
{% endif %}

[drewbanin]

@jtcohen6 this is great! And self-contained! I think I just convinced myself that we should try to ship this for 0.16.0.... i can't think of any reason at all why we should not do that. Can you?

[drewbanin]

i could live 1,000 more years and i would still not understand this DML...

[drewbanin]

I have lived 2 days and I have read the documentation, and I now understand this DML.

The key thing I was missing: when not matched by source is always true here because we're using a constant-false predicate. I was errantly thinking that we were still merging on a unique_key, which we are not.

From the docs:

If the merge_condition is FALSE, the query optimizer avoids using a JOIN. This optimization is referred to as a constant false predicate. A constant false predicate is useful when you perform an atomic DELETE on the target plus an INSERT from a source (DELETE with INSERT is also known as a REPLACE operation).

Cool!

[jtcohen6]

Neat! As part of finalizing my Discourse post about new BQ partitioning + incremental modeling in 0.16.0, I'm going to test this strategy on a (public) dataset of some size.

[drewbanin]

The test failures here appear to be intermittent weirdness on Snowflake's end... Looks like it was returning Arrow data instead of JSON data? Merging this one for 0.16.0!

[hui-zheng]

@jtcohen6 @drewbanin
It's exciting to see this feature out. We implemented something similar in-house and had long waited for this dbt native feature!