[CT-748] feat: resolve packages with same git repo and unique subdirectory

[philippeboyd]

resolves #5374

Problem

It is currently impossible to refer multiple git packages that are contained within the same mono repo such as:

  - git: "git@github.com:acme/git-repo.git"
    subdirectory: "dbt-source-pkg-1"
    revision: "dbt-source-pkg-1-v0.1.12"
  - git: "git@github.com:acme/git-repo.git" # This will not work...
    subdirectory: "dbt-source-pkg-2" # This will not work...
    revision: "dbt-source-pkg-2-v0.1.2" # This will not work...

Solution

Use the git and subdirectory values as the unique combination of values when managing dependencies. This will enable dbt to clone multiple git repositories but with different subdirectories; treating them as differents packages.

packages:
  - package: "calogica/dbt_date"
    version: [ ">=0.5.4", "<0.8.0" ]
  - package: "dbt-labs/dbt_utils"
    version: [ ">=1.0.0", "<2.0.0" ]
  - git: "git@github.com:acme/git-repo.git"
    subdirectory: "dbt-source-pkg-1"
    revision: "dbt-source-pkg-1-v0.1.12"
  - git: "git@github.com:acme/git-repo.git"
    subdirectory: "dbt-source-pkg-2"
    revision: "dbt-source-pkg-2-v0.1.2"

The following will import correctly

❯ dbt deps
14:49:31  Running with dbt=1.5.1
14:49:41  Installing calogica/dbt_date
14:49:41  Installed from version 0.7.2
14:49:41  Up to date!
14:49:41  Installing dbt-labs/dbt_utils
14:49:42  Installed from version 1.1.1
14:49:42  Up to date!
14:49:42  Installing git@github.com:acme/git-repo.git/dbt-source-pkg-1
14:49:43  Installed from revision dbt-source-pkg-1-v0.1.12
14:49:43  and subdirectory dbt-source-pkg-1
14:49:43  Installing git@github.com:acme/git-repo.git/dbt-source-pkg-2
14:49:44  Installed from revision dbt-source-pkg-2-v0.1.2
14:49:44  and subdirectory dbt-source-pkg-2

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[cla-bot]

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @philippeboyd

[philippeboyd]

CLA has just been signed.

[philippeboyd]

@jtcohen6 @leahwicz let me know if this PR makes sense to handle multiple same git repo and unique subdirectory

we're already using this solution (forked dbt core) at our company and it's working really well.

[philippeboyd]

@jtcohen6 @leahwicz, is there anything missing for this PR to be reviewed/merged?

[jaklan]

@jtcohen6 @leahwicz Also bumping that MR - it's absolutely crucial for monorepo users and we are waiting for the fix for more than year already.

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[philippeboyd]

changelog entry just added

[codecov]

Codecov Report

Patch coverage: 83.33% and project coverage change: -0.06% ⚠️

Comparison is base (3f5ebe8) 86.60% compared to head (0648827) 86.55%.
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8322      +/-   ##
==========================================
- Coverage   86.60%   86.55%   -0.06%     
==========================================
  Files         175      175              
  Lines       25638    25643       +5     
==========================================
- Hits        22204    22195       -9     
- Misses       3434     3448      +14     

Flag	Coverage Δ
integration	83.34% <75.00%> (-0.13%)	⬇️
unit	65.09% <58.33%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
core/dbt/deps/git.py	93.25% <83.33%> (-1.98%)	⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[graciegoheen]

Thanks @philippeboyd!

One case I wanted to check was whether or not this PR changes the package name, for the purposes of something like setting the dispatch config.

Because the package name is coming from the dbt_project.yml in the package, this shouldn't break the package naming convention!

Queuing this up for review with one of our engineers :)

[peterallenwebb]

@peterallenwebb @graciegoheen and @dbeatty10 think this looks good!