Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake query tag #2555

Merged
merged 12 commits into from
Jun 25, 2020
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## Next
- Added support for Snowflake query tags at the connection and model level ([#1030](https://github.com/fishtown-analytics/dbt/issues/1030))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please move this into the dbt 0.18.0 section, put it under a ### Features heading, and make it link to both the issue and this PR?


## dbt 0.18.0 (Release TBD)

## dbt 0.18.0b1 (June 08, 2020)
Expand All @@ -13,6 +16,7 @@
- Added support for BigQuery authorized views ([#1718](https://github.com/fishtown-analytics/dbt/issues/1718), [#2517](https://github.com/fishtown-analytics/dbt/pull/2517))
- Added support for altering BigQuery column types ([#2546](https://github.com/fishtown-analytics/dbt/issues/2546), [#2547](https://github.com/fishtown-analytics/dbt/pull/2547))


### Fixes
- Fixed an error in create_adapter_plugins.py script when -dependency arg not passed ([#2507](https://github.com/fishtown-analytics/dbt/issues/2507), [#2508](https://github.com/fishtown-analytics/dbt/pull/2508))
- Remove misleading "Opening a new connection" log message in set_connection_name. ([#2511](https://github.com/fishtown-analytics/dbt/issues/2511))
Expand All @@ -22,8 +26,8 @@ Contributors:
- [@alf-mindshift](https://github.com/alf-mindshift) ([#2431](https://github.com/fishtown-analytics/dbt/pull/2431))
- [@scarrucciu](https://github.com/scarrucciu) ([#2508](https://github.com/fishtown-analytics/dbt/pull/2508))
- [@southpolemonkey](https://github.com/southpolemonkey) ([#2511](https://github.com/fishtown-analytics/dbt/issues/2511))
- [@azhard](https://github.com/azhard) ([#2517](https://github.com/fishtown-analytics/dbt/pull/2517), ([#2521](https://github.com/fishtown-analytics/dbt/pull/2521)), [#2547](https://github.com/fishtown-analytics/dbt/pull/2547))

- [@azhard](https://github.com/azhard) ([#2517](https://github.com/fishtown-analytics/dbt/issues/2517), ([#2521](https://github.com/fishtown-analytics/dbt/pull/2521)))
- [@DrMcTaco](https://github.com/DrMcTaco) ([#1030](https://github.com/fishtown-analytics/dbt/issues/1030)))

## dbt 0.17.1 (Release TBD)

Expand All @@ -34,6 +38,7 @@ Contributors:
- dbt again respects config aliases in config() calls ([#2557](https://github.com/fishtown-analytics/dbt/issues/2557), [#2559](https://github.com/fishtown-analytics/dbt/pull/2559))



## dbt 0.17.0 (June 08, 2020)

### Fixes
Expand Down
6 changes: 6 additions & 0 deletions plugins/snowflake/dbt/adapters/snowflake/connections.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ class SnowflakeCredentials(Credentials):
token: Optional[str]
oauth_client_id: Optional[str]
oauth_client_secret: Optional[str]
query_tag: Optional[str]
client_session_keep_alive: bool = False

def __post_init__(self):
Expand Down Expand Up @@ -211,6 +212,11 @@ def open(cls, connection):
**creds.auth_args()
)

if creds.query_tag:
handle.cursor().execute(
("alter session set query_tag = '{}'")
.format(creds.query_tag))

connection.handle = handle
connection.state = 'open'
except snowflake.connector.errors.Error as e:
Expand Down
1 change: 1 addition & 0 deletions plugins/snowflake/dbt/adapters/snowflake/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ class SnowflakeConfig(AdapterConfig):
secure: Optional[bool] = None
copy_grants: Optional[bool] = None
snowflake_warehouse: Optional[str] = None
query_tag: Optional[str] = None


class SnowflakeAdapter(SQLAdapter):
Expand Down
30 changes: 30 additions & 0 deletions plugins/snowflake/dbt/include/snowflake/macros/adapters.sql
Original file line number Diff line number Diff line change
Expand Up @@ -160,3 +160,33 @@
{{ column_name }} COMMENT $${{ column_dict[column_name]['description'] | replace('$', '[$]') }}$$ {{ ',' if not loop.last else ';' }}
{% endfor %}
{% endmacro %}


{% macro get_current_query_tag() -%}
{{ return(run_query("show parameters like 'query_tag' in session").rows[0]['value']) }}
{% endmacro %}


{% macro set_query_tag() -%}
{% set new_query_tag = config.get('query_tag') %}
{% if new_query_tag %}
{% set original_query_tag = get_current_query_tag() %}
{{ log("Setting query_tag to '" ~ new_query_tag ~ "'. Will reset to '" ~ original_query_tag ~ "' after materialization.") }}
{% do run_query("alter session set query_tag = '{}'".format(new_query_tag)) %}
{{ return(original_query_tag)}}
{% endif %}
{{ return(none)}}
{% endmacro %}

{% macro unset_query_tag(original_query_tag) -%}
{% set new_query_tag = config.get('query_tag') %}
{% if new_query_tag %}
{% if original_query_tag %}
{{ log("Resetting query_tag to '" ~ original_query_tag ~ "'.") }}
{% do run_query("alter session set query_tag = '{}'".format(original_query_tag)) %}
{% else %}
{{ log("No original query_tag, unsetting parameter.") }}
{% do run_query("alter session unset query_tag") %}
{% endif %}
{% endif %}
{% endmacro %}
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@

{% materialization incremental, adapter='snowflake' -%}

{% set original_query_tag = set_query_tag() %}

{%- set unique_key = config.get('unique_key') -%}
{%- set full_refresh_mode = (should_full_refresh()) -%}

Expand Down Expand Up @@ -74,6 +76,8 @@
{% set target_relation = target_relation.incorporate(type='table') %}
{% do persist_docs(target_relation, model) %}

{% do unset_query_tag(original_query_tag) %}

{{ return({'relations': [target_relation]}) }}

{%- endmaterialization %}
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
{% materialization table, adapter='snowflake' %}

{% set original_query_tag = set_query_tag() %}

{%- set identifier = model['alias'] -%}

{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
Expand Down Expand Up @@ -32,6 +35,8 @@

{% do persist_docs(target_relation, model) %}

{% do unset_query_tag(original_query_tag) %}

{{ return({'relations': [target_relation]}) }}

{% endmaterialization %}
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
{% materialization view, adapter='snowflake' -%}

{% set original_query_tag = set_query_tag() %}
{% set to_return = create_or_replace_view() %}

{% set target_relation = this.incorporate(type='view') %}
{% do persist_docs(target_relation, model, for_columns=false) %}

{% do return(to_return) %}

{% do unset_query_tag(original_query_tag) %}

{%- endmaterialization %}