[CT-110] [Feature] Ability to modify which variables are checked when partial parsing

[AntonAiven]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Partial parsing could be configured to not check if environment variables changes. We compile dbt in Github actions and we run it on a schedule in kubernetes. We change the authentication for dbt based on the environment, and do this through environment variables. This worked well in v0.21 because in that version dbt didin't check for changed environment variables, but since v1.0 it does, so now we need to compile each time we run.

Describe alternatives you've considered

No response

Who will this benefit?

Anyone who runs dbt in multiple environments

Are you interested in contributing this feature?

No response

Anything else?

No response

[gshank]

Thanks for reporting this! I assume that you have env_vars in your profile? If that's the case, then the intention was that env_vars starting with DBT_ENV_SECRET would not be used for setting off partial parsing, but we inadvertently included them. We'll be fixing this bug in the next point release.

Please let me know if using secret env_vars (when working correctly) would not meet your needs.

[gshank]

The place that needs to be fixed is in core/dbt/context/secret.py, in the env_var method. Where it saved the return_value in self.env_vars, we want to skip saving if the env_var if it starts with the SECRET_ENV_PREFIX.

[jtcohen6]

Going to open two issues as follow-ons to this one:

• Fix the SecretContext to avoid saving secrets (for v1.0.2): [CT-114] Avoid saving secrets in SecretContext #4650
• Update docs to mention that changes to secrets used in profiles.yml do not trigger full re-parses, unlike other env vars used in the active profile: Document intersection of "secret" env vars + partial parsing docs.getdbt.com#1066

[AntonAiven]

The place that needs to be fixed is in core/dbt/context/secret.py, in the env_var method. Where it saved the return_value in self.env_vars, we want to skip saving if the env_var if it starts with the SECRET_ENV_PREFIX.

Thanks for reporting this! I assume that you have env_vars in your profile? If that's the case, then the intention was that env_vars starting with DBT_ENV_SECRET would not be used for setting off partial parsing, but we inadvertently included them. We'll be fixing this bug in the next point release.

Please let me know if using secret env_vars (when working correctly) would not meet your needs.

Something like this would definitely fix it for us!

[karunpoudel]

I second this feature.

We have separate snowflake account for each environment. It would be nice if we could parse and build the image once, and run it in different environment without having to reparse. This would mean some sort of config options to ignore change in variables used in profiles.yml.

We are using dynamic secrets so the username and password can change between each dbt run. If variables with DBT_ENV_SECRET is ignored for partial parse then it should fix for us.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.