[CT-2127] [Bug] Running dbt commands before deps results in dbt_packages folder creation at invocation location

[racheldaniel]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

On a dbt run issued through the dbtRunner.invoke, dbt is attempting to create an empty default dbt_packages directory at the location from which the command is being run, instead of at the project root.

Expected Behavior

Any creation of a dbt_packages dir should default to the project root

Steps To Reproduce

This behavior was observed through use of the dbtRunner class in 1.5.0+, though it likely existed prior.

1. In a python shell anywhere with access to dbt other than your project, do a run using the dbtRunner:

from dbt.cli.main import dbtRunner
dbt = dbtRunner()
dbt.invoke(["run", "--project-dir", "../jaffle_shop"])

2. Observe a dbt_packages dir at the root of wherever this was invoked (if you have packages you needed to install, the error will be thrown but this dir will still be created)
3. Delete this dir and repeat, but run a deps first and then a run
4. Observer dbt_packages creation at the project root, no duplicate dir at the location of invocation

Relevant log output

No response

Environment

- OS: Ventura 13.1
- Python: 3.8.13
- dbt: 1.4.0b1 branch `cl/server-on-merged`, soon to be 1.5.0a1

Which database adapter are you using with dbt?

postgres, snowflake

Additional Context

Everywhere that packages_installation_path is referenced in core, the value of that path is just dbt_packages. I checked that value on parse, run, deps — that value remains the same.

The absolute value of that path is what changes. Initially, and with any actions taken before doing a deps, the absolute value points to where I’m running it-- ~/dbt-core/dbt_packages. After running a deps and in all subsequent runs, the abspath points to the project-- ~/jaffle_shop/dbt_packages

This is the function call responsible for that change. That function calls an os.chdir, and we never change back (understandably). This function is not called for a run, so if deps is not called, this line on the Compiler object is creating an empty dir at ~/dbt-server/dbt_packages .

[stu-k]

This Friday, the execution team will be talking about the future of core as a library and how we want to abstract certain functionality moving forward. The concept of changing directories and file systems will likely come up, so we will revisit this issue after we have discussed this in more detail!

[jtcohen6]

@stu-k "Paths are tricky" is definitely a fair topic for larger discussion.

I do see this bug as being narrower in scope, and something that we could resolve without requiring us to first have full answer to the much broader question of, "How should dbt-core interact with a file system?" It is a really valuable conversation; a few days ago, I created a file_system label, and I'll add it to this issue as well.

It sounds like the Execution team will revisit this issue in a future BLG. If we decide that we shouldn't resolve this specific bug, except in the context of that much larger initiative, then we should close it.

[classicus-eth]

Glad this is already an open issue 🥇 We use the dbt Runner extensively and are currently forced to do some unstable infra gymnastics to work around this bug.

cc @tanghyd

[jtcohen6]

@classicus-eth Glad you caught the bug report. This isn't high priority for us, so if you're motivated to fix it, we'd be happy to review an external contribution!

[seub]

Also running into this bug: when running dbt compile, dbt ls, etc from another location than the project dir, then an empty dbt_packages folder is created in the current working dir.

I'm experimenting this with dbt 1.7.3. I was about to create a new issue before I found this one.

I'd like to insist on a more general point : when running dbt from another location than the project dir, it seems silly to me that dbt paths are evaluated relative to the current working dir, instead of the project dir.

In other words, when running dbt <command> --project-dir path/to/project, all paths specified in dbt_project.yml as well as externally (profiles_dir, packages_install_path, etc) should be relative to project_dir and not the cwd, imho. Fixing this would be a quality of life improvement for my team.