[CT-2589] [Regression] run-operation does not use DBT_TARGET_PATH env variable

[azorej]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

run-operation command does not use DBT_TARGET_PATH env variable and doesn't have target-path argument

Expected/Previous Behavior

run-operation command has target-path argument and uses DBT_TARGET_PATH env variable

Steps To Reproduce

1. Create new project with dbt init (with any adapter):

>> dbt init test

2. Create default target dir and make it read-only:

>> mkdir target && chmod -R 0444 target

3. Make some macro:

>> cat > cat test/macros/hello_world.sql
{% macro hello_world() %}
  {{ log("Hello world!") }}
{% endmacro %}

4. Run example model with custom target-path:

>> DBT_TARGET_PATH="$(realpath custom_target)" dbt --debug run --select "example.my_first_dbt_model" --project-dir test"

5. Run macro with custom target-path (it will fail with PermissionError: [Errno 13] Permission denied: 'target/manifest.json'):

>> DBT_TARGET_PATH="$(realpath custom_target)" dbt --debug run-operation 'hello_world' --project-dir "$(realpath test)"

Relevant log output

21:00:23  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0c095fed40>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0c07efdc30>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0c07efdc00>]}
21:00:23  Running with dbt=1.5.0
21:00:23  running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/home/lkozhinov/projects/opensource/dbt_run_operation_regression', 'fail_fast': 'False', 'warn_error': 'None', 'log_path': '/home/lkozhinov/projects/opensource/dbt_run_operation_regression/test/logs', 'version_check': 'True', 'debug': 'True', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'introspect': 'True', 'log_format': 'default', 'target_path': 'None', 'send_anonymous_usage_stats': 'True'}
21:00:24  Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '8d8d2e0d-a8ab-40b4-8a3d-98d1fc4e0262', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0c07efdd20>]}
21:00:24  Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '8d8d2e0d-a8ab-40b4-8a3d-98d1fc4e0262', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0be1dfc8e0>]}
21:00:24  checksum: 4568eb639a77b8fcb3a1f4a07856f42b1ff63f1376652889143968e1dbdafbda, vars: {}, profile: , target: , version: 1.5.0
21:00:24  Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
21:00:24  Partial parsing enabled, no changes found, skipping parsing
21:00:24  Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '8d8d2e0d-a8ab-40b4-8a3d-98d1fc4e0262', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0be1bf62c0>]}
21:00:24  Encountered an error:
[Errno 13] Permission denied: 'target/manifest.json'
21:00:24  Traceback (most recent call last):
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 86, in wrapper
    result, success = func(*args, **kwargs)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 71, in wrapper
    return func(*args, **kwargs)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 142, in wrapper
    return func(*args, **kwargs)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 168, in wrapper
    return func(*args, **kwargs)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 215, in wrapper
    return func(*args, **kwargs)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/cli/requires.py", line 248, in wrapper
    write_manifest(manifest, ctx.obj["runtime_config"].target_path)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/parser/manifest.py", line 1497, in write_manifest
    manifest.write(path)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/contracts/graph/manifest.py", line 879, in write
    self.writable_manifest().write(path)
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/contracts/util.py", line 65, in write
    write_json(path, self.to_dict(omit_none=False))  # type: ignore
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/clients/system.py", line 200, in write_json
    return write_file(path, json.dumps(data, cls=dbt.utils.JSONEncoder))
  File "/home/lkozhinov/.cache/pypoetry/virtualenvs/dbt-run-operation-regression-AC0esz7o-py3.10/lib/python3.10/site-packages/dbt/clients/system.py", line 169, in write_file
    with open(path, "w", encoding="utf-8") as f:
PermissionError: [Errno 13] Permission denied: 'target/manifest.json'

21:00:24  Command `dbt run-operation` failed at 23:00:24.181352 after 0.93 seconds
21:00:24  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0c095fed40>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0be1db8e80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f0be1db8cd0>]}
21:00:24  Flushing usage events

Environment

- OS: Ubuntu 22.04
- Python: 3.10.6
- dbt (working version): 1.4.1
- dbt (regression version): 1.5.0

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

Adapter: dbt-databricks

[dbeatty10]

Thanks for reporting this @azorej !

This looks the same as #7646 to me, so I'm going to close it as a duplicate.

Please don't hesitate to reach out if you think I was too hasty and this one should be re-opened 🙏