Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move --full-refresh into a model config #1009

Closed
drewbanin opened this issue Sep 18, 2018 · 1 comment · Fixed by #2438
Closed

Move --full-refresh into a model config #1009

drewbanin opened this issue Sep 18, 2018 · 1 comment · Fixed by #2438
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors!

Comments

@drewbanin
Copy link
Contributor

Feature

Feature description

Presently, --full-refresh and --non-destructive can only be set on the CLI and apply to all models. Instead, these flags should be pushed down as model configs. The --non-destructive and --full-refresh flags can still exist, but they should set individual model configurations under the hood.

Since these configs will be specified with model-level granularity, complex behavior can be coded using variables, macros, etc.

Who will this benefit?

Users with sophisticated dbt deployments who need to trigger complex dbt invocations

Via @elexisvenator in Slack:

I want to run dbt as frequently as possible, which means ideally I only want to run models with data that has changed, or where a model has changed.
I already have the means to know which source tables have changed, and which model files have been modified since last run.

See also: #320

@drewbanin drewbanin added enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors! labels Sep 18, 2018
@drewbanin
Copy link
Contributor Author

Presently, flags are set at a global level in dbt. In main.py, dbt plucks configuration values out of the parsed CLI arguments and sticks them into the global fields defined in flags.py.

In the jinja layer, materializations use these flags to determine exactly what SQL should be run. There's an example of that over here.

A better approach is to add new configuration options (like non_destructive or full_refresh) for dbt resources, then use the global flags to set these options on the resources during parsing. Finally, the materialization code can be updated to reference an attribute of a model, like model.full_refresh, rather than pulling this value from the flags directly.

As a result of this change, the behavior of the --full-refresh flag will not change, but users can optionally supply a full_refresh configuration option to configure specific models based on the value of a variable, for instance.

This might be out of scope for this issue, but I think we should deprecate (and eventually remove) the --non-destructive flag/concept entirely from dbt. That flag existed as a workaround for Redshift's bound views, which the incidence of late-bound views wholly obviates!

@drewbanin drewbanin changed the title Move --non-destructive and --full-refresh into model configs Move --full-refresh into a model config May 1, 2019
@drewbanin drewbanin added this to the Octavius Catto milestone Mar 3, 2020
@drewbanin drewbanin modified the milestones: Octavius Catto, dbt-next Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors!
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant