wrap models using macros #356

drewbanin · 2017-03-29T05:31:45Z

this removes the Model.compile call (and the associated, duplicate context that we need to create).

This branch is currently a proof-of concept, but in it's current state:

macros work in hooks
already_exists works (kind of)
we no longer "compile twice" (we do in practice, but we won't have to with this setup)

* code cleanup + fix full-refresh opt * make integration tests actually test flags

…ntext

…tless-macros

…t into contextless-macros

cmcarthur

this is cool. what's the next step?

cmcarthur · 2017-03-29T13:48:16Z

dbt/adapters/redshift.py

@@ -30,7 +31,7 @@ def sort_qualifier(cls, sort_type, sort):
                .format(sort_type, valid_sort_types)
            )

-        if type(sort) == str:


great catch

* always release connections at the end of execute_node * pep8 * changelog

drewbanin · 2017-03-30T17:40:29Z

dbt/main.py

@@ -31,6 +31,7 @@ def main(args=None):
        handle(args)

    except RuntimeError as e:
+        raise


@cmcarthur we should remove this -- i just threw it in there for debugging

drewbanin

this is looking really good @cmcarthur. Couple of notes, but i think once we get these PRs merged we'll be well suited to do a final pass + cleanup

drewbanin · 2017-03-31T22:54:53Z

dbt/adapters/redshift.py

+
+            connection = cls.get_connection(profile, model_name)
+
+            cls.commit(connection)


@cmcarthur is there any problem with committing if there is no open transaction?

drewbanin · 2017-03-31T22:56:39Z

dbt/compilation.py

@@ -291,34 +286,7 @@ def get_compiler_context(self, linker, model, flat_graph):

        return context

-    def get_context(self, linker, model, models):


drewbanin · 2017-03-31T23:04:41Z

dbt/include/global_project/macros/materializations/view.sql

+  {%- set already_exists = funcs.already_exists(schema, identifier) -%}
+  {%- set non_destructive_mode = flags.NON_DESTRUCTIVE == True -%}
+
+  {%- if non_destructive_mode -%}


is this the right logic for views in non-destructive mode?

drewbanin · 2017-03-31T23:06:42Z

dbt/parser.py

@@ -142,7 +188,7 @@ def parse_node(node, node_path, root_project_config, package_project_config,

    context = {}

-    context['ref'] = lambda *args: ''
+    context['ref'] = __ref(node)


drewbanin · 2017-03-31T23:06:57Z

dbt/runner.py

@@ -239,6 +239,9 @@ def execute_model(profile, model, existing):

    result = None

+    if get_materialization(model) == 'ephemeral':
+        return 'SKIP'


we're just going to silently skip these, right?

drewbanin · 2017-03-31T23:12:47Z

dbt/wrapper.py

+    sort_keys = model_config.get('sort')
+    sort_type = model_config.get('sort_type', 'compound')
+
+    if type(sort_type) != str:


drewbanin

looking good! will give this a spin today and update w/ any bugs i find

drewbanin · 2017-04-04T09:00:09Z

dbt/parser.py

@@ -399,7 +396,7 @@ def parse_schema_test(test_base, model_name, test_config, test_type,

    raw_sql = "{{{{ {macro}(model=ref('{model}'), {kwargs}) }}}}".format(**{
        'model': model_name,
-        'macro': "test_{}".format(test_type),
+        'macro': "dbt.test_{}".format(test_type),


this is ok for now, but i think ideally we wouldn't namespace this. Idea being that end users could 1) define their own tests (not in the dbt namespace) or 2) override existing tests

good point, i will switch this back.

Include global project

drewbanin and others added 22 commits March 25, 2017 22:04

code cleanup + fix full-refresh opt

c591815

make integration tests actually test flags

3d4d72d

accidently create operations

f651568

Handle concurrent DROP ... CASCADEs in Redshift (#349)

e080f73

working with a few rough edges

fe79e07

Fix/full refresh (#350)

da6c467

* code cleanup + fix full-refresh opt * make integration tests actually test flags

Merge branch 'development' into feature/already-exists-compilation-co…

f94f455

…ntext

test that should be green if macro parsing works the right way

7f54175

wip

23d42d2

Merge branch 'feature/already-exists-compilation-context' into contex…

cacf5c9

…tless-macros

validate macros don't use context vars

b43be9e

fix tests

ace4b49

working with a few rough edges

bc0e1c2

test that should be green if macro parsing works the right way

694d07b

inject into jinja with custom extension

c55ff34

wip

8537229

validate macros don't use context vars

4846515

fix tests

c06ae6d

Merge branch 'contextless-macros' of github.com:fishtown-analytics/db…

e03f804

…t into contextless-macros

add test for target in macros, disallow target

2b3f46e

add license to RTD

e2c279b

wip - materialize via macros

ce96825

cmcarthur reviewed Mar 29, 2017

View reviewed changes

cmcarthur and others added 4 commits March 29, 2017 09:51

always release connections at the end of execute_node (#354)

5ffe4a5

* always release connections at the end of execute_node * pep8 * changelog

merged development

d907fca

tests passing

ab4d5a3

integration tests passing, again

a266308

drewbanin commented Mar 31, 2017

View reviewed changes

unit tests & pep8 passing

e533b36

merge contextless macros branch

d3d2da5

cmcarthur mentioned this pull request Apr 3, 2017

contextless macros #352

Closed

Connor McArthur added 3 commits April 3, 2017 20:45

functional test tweaks

cb2bba4

remove extra raise

f1f39c1

fix integration tests

0b810c6

drewbanin commented Apr 4, 2017

View reviewed changes

Connor McArthur and others added 8 commits April 4, 2017 10:44

allow global project macros to be referenced without a namespace

4f663a0

handle compilation errors

837bff4

add manifest file

0574958

include global project files in dbt dist

1063500

Merge pull request #361 from fishtown-analytics/include-global-project

161d70c

Include global project

rename funcs to adapter

8b59748

remove compile task

b7044d9

don't throw compilr error for this/target in macro

2921a00

cmcarthur merged commit e8d70a6 into feature/schema-tests-defined-by-macros Apr 5, 2017

cmcarthur deleted the kill-model-py branch April 5, 2017 12:05

cmcarthur restored the kill-model-py branch April 5, 2017 12:41

cmcarthur deleted the kill-model-py branch April 5, 2017 12:42

drewbanin restored the kill-model-py branch August 5, 2017 19:07

drewbanin deleted the kill-model-py branch August 5, 2017 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wrap models using macros #356

wrap models using macros #356

drewbanin commented Mar 29, 2017

cmcarthur left a comment

cmcarthur Mar 29, 2017

drewbanin Mar 30, 2017

drewbanin left a comment

drewbanin Mar 31, 2017

drewbanin Mar 31, 2017

drewbanin Mar 31, 2017

drewbanin Mar 31, 2017

drewbanin Mar 31, 2017

drewbanin Mar 31, 2017

drewbanin left a comment

drewbanin Apr 4, 2017

cmcarthur Apr 4, 2017


		connection = cls.get_connection(profile, model_name)

		cls.commit(connection)

		@@ -291,34 +286,7 @@ def get_compiler_context(self, linker, model, flat_graph):

		return context

		def get_context(self, linker, model, models):

wrap models using macros #356

wrap models using macros #356

Conversation

drewbanin commented Mar 29, 2017

cmcarthur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewbanin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewbanin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment