[CT-2649] [Feature] Move --target and --profile command line flags to global config

[rexledesma]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

When attempting to dynamically configure a user's profile and target, We should prefer dbt --target TARGET_NAME over dbt run --target TARGET_NAME, and similarly for --profile.

--profiles-dir is essentially already a global config with its exposure as an environment variable, so I believe other associated configurations with profiles (e.g. --target and --profile) should also be a global config.

Describe alternatives you've considered

To achieve dynamic behavior, environment variables (e.g. DBT_TARGET and DBT_PROFILE) can be used to configure --target and --profile by templating these values into profiles.yml.

However:

• These environment variables are less explicit than configuring CLI flags, and
• This requires the user to use env_var(...) for their profile.yml. Ideally, this feature should come out of the box.

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[dbeatty10]

Thanks for reaching out @rexledesma !

Can you help me better understand the things you're trying to unlock that aren't possible with using --target and --profile CLI arguments like this?

dbt run --target TARGET_NAME --profile PROFILE_NAME

Would these things you're thinking of be primarily aimed at users invoking dbt manually or systems invoking dynamic configurations on behalf of a user?

[rexledesma]

Thanks for reaching out @rexledesma !

Can you help me better understand the things you're trying to unlock that aren't possible with using --target and --profile CLI arguments like this?

dbt run --target TARGET_NAME --profile PROFILE_NAME

Would these things you're thinking of be primarily aimed at users invoking dbt manually or systems invoking dynamic configurations on behalf of a user?

This is more of just ergonomics managing configuration manually, as well as on behalf of a user. --target and --profile are both already configured on every single dbt command, so might as well be elevated to global configuration.

There are cases where I am developing my dbt models and the default profile and target are different from the profile + target that I want to use for local development. Having the ability to customize this via env var would be nice, so I can set this one time and not have to think about it.

[jtcohen6]

I buy it!

The --target and --profile params are supported by all(?) commands — although they don't actually do anything for deps, clean, init (until #7450), docs serve, ... (possibly a few others) ...

I think the change here might be as simple as:

• adding @p.profile and @p.target to the list of "global" CLI flags defined here
• adding env vars (DBT_PROFILE, DBT_TARGET) to the params here and here
• Adding a unit test here to prove these flags are supported in the "global" position, in addition to the "subcommand" position. (We already have a test to verify that dbt should raise an error if the same flag is passed in both positions.)

Going to mark this one a GFI :)

[dbeatty10]

The --target and --profile params are supported by all(?) commands

👍 Yep, looks like it is already included in all commands! See below.

Pivot table of dbt commands and supported flags

option	# dbt build	# dbt clean	# dbt compile	# dbt debug	# dbt deps	# dbt docs generate	# dbt docs serve	# dbt init	# dbt list	# dbt parse	# dbt retry	# dbt run	# dbt run operation	# dbt seed	# dbt show	# dbt snapshot	# dbt source freshness	# dbt test
@p.args													✅
@p.browser							✅
@p.compile_docs						✅
@p.config_dir				✅
@p.debug_connection				✅
@p.defer	✅		✅			✅						✅			✅	✅		✅
@p.defer_state	✅		✅			✅			✅			✅		✅	✅	✅	✅	✅
@p.deprecated_defer	✅		✅			✅						✅			✅	✅		✅
@p.deprecated_favor_state	✅		✅			✅						✅			✅	✅		✅
@p.deprecated_state	✅		✅			✅			✅			✅		✅	✅	✅	✅	✅
@p.empty_catalog						✅
@p.exclude	✅		✅			✅			✅			✅		✅	✅	✅	✅	✅
@p.fail_fast	✅										✅	✅						✅
@p.favor_state	✅		✅			✅						✅			✅	✅		✅
@p.full_refresh	✅		✅									✅		✅	✅
@p.indirect_selection	✅		✅						✅						✅			✅
@p.inline			✅												✅
@p.introspect			✅												✅
@p.models									✅
@p.output									✅
@p.output_keys									✅
@p.output_path																	✅
@p.port							✅
@p.profile	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.profiles_dir	✅	✅	✅			✅	✅		✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.profiles_dir_exists_false				✅	✅			✅
@p.project_dir	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.raw_select									✅
@p.resource_type	✅								✅
@p.select	✅		✅			✅						✅		✅	✅	✅	✅	✅
@p.selector	✅		✅			✅			✅			✅		✅	✅	✅	✅	✅
@p.show	✅													✅
@p.show_limit															✅
@p.show_output_format			✅												✅
@p.skip_profile_setup								✅
@p.state	✅		✅			✅			✅		✅	✅		✅	✅	✅	✅	✅
@p.store_failures	✅																	✅
@p.target	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.target_path	✅	✅	✅			✅	✅		✅	✅		✅	✅	✅	✅	✅	✅	✅
@p.threads	✅		✅			✅				✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.vars	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
@p.version_check	✅		✅	✅		✅				✅		✅		✅	✅			✅

Source code for pivot table

dbt_subcommand_flags.py

import re
import pandas as pd
import numpy as np


def find_instances(text):
    # Matches any string of format '# dbt subcommand'
    subcommand_pattern = r"# dbt .+"

    # Matches any string of format '@p.cli_option'
    option_pattern = r"@p\.\w+"

    # Store results in a list of dictionaries
    result_list = []
    last_subcommand_match = None

    for line in text.splitlines():
        subcommand_match = re.search(subcommand_pattern, line)
        if subcommand_match:
            last_subcommand_match = subcommand_match.group()

        option_match = re.search(option_pattern, line)
        if option_match and last_subcommand_match:
            result_list.append(
                {"subcommand": last_subcommand_match, "option": option_match.group()}
            )

    return pd.DataFrame(result_list)


def read_file(file_path):
    with open(file_path, "r") as file:
        text = file.read()
    return text


def main():
    file_path = "main.py"
    text = read_file(file_path)
    result_df = find_instances(text)

    # Represents the presence of a pair with the value 1
    result_df["count"] = 1

    # Create a pivot table
    table = pd.pivot_table(
        result_df,
        values="count",
        index=["option"],
        columns=["subcommand"],
        aggfunc=np.sum,
        fill_value=0,
    )

    # Convert 1's and 0's to applicable icons
    table.replace(to_replace=1, value="✅", inplace=True)
    table.replace(to_replace=0, value="", inplace=True)

    # Export to Markdown
    print(table.to_markdown(stralign="center"))


if __name__ == "__main__":
    main()

1. Download latest version of core/dbt/cli/main.py to your current working directory
2. pip install pandas and numpy if needed
3. python dbt_subcommand_flags.py

[runleonarun]

I moved the "user docs" label to the PR so the docs issue gets opened in docs.getdbt.com