run hooks outside of a transaction

[drewbanin]

This is required to get vacuum working in post- hooks. Pre/Post hooks can now be:

• a string
• a json string containing a "sql" string key and a "transaction" key

All of the following are valid:

{{ config(
    'post-hook': [
        'select 1 -- inside transaction',
        {"sql": "select 2 -- inside transaction", "transaction": true},
        {"sql": "vacuum {{ this }}", "transaction": false},
        '{{ after_commit("vacuum " ~ adapter.quote_schema_and_table(this.schema, this.name)) }}',
        '{{ vacuum(this) }} -- this is a macro which expands to a post-transaction json string'
    ]
) }}

[jthandy]

neat, i hadn't realized we were actually working actively on this.

is it possible to do this?

post-hook:
  - {sql: "grant select on {{ this }} to db_reader", transaction: true}
  - "vacuum {{ this.schema }}.{{ this.name }}"

essentially, the first argument is always the sql and an optional second argument specifies the transaction status. most times it would be nice to just declare the sql syntax and not think about transactions at all, and i'm also nervous about adding additional configuration parameters given that we have a bit of a configuration spaghetti bowl at the moment.

[drewbanin]

yeah, i'm with you @jthandy -- doesn't really make sense to add another config arg for this. I like that idea!

[cmcarthur]

I like the idea of defaulting to transaction = true. One recommendation: name this thing after_commit, it's more clear. Like:

post-hook:
  - "grant select on {{ this }} to db_reader"
  - { after_commit: true, sql: "vacuum {{ this.schema }}.{{ this.name }}" }

[drewbanin]

@cmcarthur that makes sense, but we need this to work for pre-hooks too ;)

[cmcarthur]

before_begin?

[drewbanin]

@cmcarthur @jthandy just pushed some more code... all of these are valid ways to define pre- or post-hooks:

    'post-hook': [
        'select 1 -- inside transaction',
        {"sql": "select 2 -- inside transaction", "transaction": true},
        {"sql": "vacuum \"dbt_dbanin\".\"dbt_users\"", "transaction": false},
        '{{ after_commit("vacuum " ~ adapter.quote_schema_and_table(this.schema, this.name)) }}',
        '{{ vacuum(this) }}'
    ]

Some of these macros don't belong here, but I'm using it as a proof-of-concept: https://github.com/fishtown-analytics/dbt/blob/0e6044508def2e9901a1d964861c0b678be28617/dbt/include/global_project/macros/materializations/helpers.sql

[cmcarthur]

does this automatically roll back open transactions? would want to be sure we don't create a deadlock because of an open tx

[drewbanin]

yeah, release_connection calls rollback under the hood if the transaction is open. This change is to fix an issue where we'd try to rollback transactions that weren't open

[cmcarthur]

this is slick

[cmcarthur]

this is the only part of this branch that doesn't feel right to me. it seems like we should do the opposite: parse all the hooks to their dictionary version and return those. then you don't need to decode each hook individually in a loop before you can perform filter logic.

that would let you write, for example:

{% macro run_hooks(hooks, inside_transaction=True) %}
  {% for hook in hooks | selectattr('transaction', 'equalto', inside_transaction)  %}
    {% call statement(auto_begin=inside_transaction) %}
      {{ hook.get('sql') }}
    {% endcall %}
  {% endfor %}
{% endmacro %}

(and, in general, lets you do more powerful comprehensions on hooks)

[cmcarthur]

this looks great. approved if tests pass

[cmcarthur]

nice

[cmcarthur]

i like this

[drewbanin]

i intend to rewrite dbt-core in sql, and this is step 1 of 10 billion

i think this could equivalently be:

tags = tags or set()

but i always find that confusing, and I think codeclimate would still complain about cyclomatic complexity. 👍

[drewbanin]

Summary, because this branch kind of morphed from its initial purpose:

This branch formalizes how hooks should work. Now, a hook is either:

• a sql string
• a dict in the format {"sql": sql_string, "transaction": True/False}

If a sql string is provided, the hook will run inside of the model transaction. If a dict is provided, and transaction is False, then the hook will either run before the begin or after the commit for the model transaction, depending on if the hook is a pre-hook or post-hook.

This means that commands like analyze and vacuum can now be run in post-hooks.

The dictionary syntax is a little unwieldy. Three new macros, before_begin, in_transaction, and after_commit have been added to make adding hooks easier.

With this syntax, you can define a hook that looks like:

{{ config({
    "post-hook": after_transaction('vacuum ` ~ adapter.quote_schema_and_table(this.schema, this.name))
  })
}}

Finally, on-run-start and on-run-end hooks have been refactored to permit the same hook definition provided above. This means that you can define hook macros and use them either in run hooks or model hooks. Note that, for now, all on-run-* run outside of a transaction.

[drewbanin]

Fixes #486

[ryantuck]

I think the code block above should read (as of v0.10.1):

{{ config({
    "post-hook": after_commit('vacuum ' ~ adapter.quote_schema_and_table(this.schema, this.name))
  })
}}

[drewbanin]

Thanks @ryantuck - I think you're right :)