cross-project ref + model access + dependencies.yml
#3577
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for docs-getdbt-com ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
github-actions
bot
added
content
jtcohen6
changed the title
dependencies.yml, cross-project ref, and model access across projectsref + model access + dependencies.yml
MichelleArk
reviewed
Jul 6, 2023
MichelleArk
reviewed
Jul 6, 2023
MichelleArk
reviewed
Jul 6, 2023
|
|
||
| It is possible to `ref` a model from another project in two ways: | ||
| 1. As a "project" dependency, via "cross-project `ref`" (a feature of dbt Cloud Enterprise) | ||
| 2. As a "package" dependency, whereby all source code from that project is installed into your own |
There was a problem hiding this comment.
Looks like we just need to finish this thought - own project?
There was a problem hiding this comment.
I definitely meant to write your own project here — although we're using project to mean so many different things. It's almost more like:
- your own working directory
- your own environment
- your own dbt runtime
- ...
|
|
||
| dbt Core v1.6 introduces a notion of `dependencies` between dbt projects. You might already be familiar with installing other projects as [packages](/docs/build/packages), whereby you pull down another project's source code and treat it as your own. | ||
|
|
||
| There is a new kind of `project` dependency. Both dependencies can be defined in `dependencies.yml`: |
There was a problem hiding this comment.
It feels like there is gap between the opening sentences and this one. Is there some more information we want to share to help users connect the dots here?
There was a problem hiding this comment.
writing more here!
|
@MichelleArk @matthewshaver Thanks for the reviews! I'm going to give this another draft before merging, specifically for how we're positioning XP ref vis-à-vis packages |
jtcohen6
commented
Jul 7, 2023
|
|
||
| Models referenced from a `project`-type dependency must use [two-argument `ref`](/reference/dbt-jinja-functions/ref#two-argument-variant), including the project name. Only public models can be accessed in this way. | ||
|
|
||
| It is equally possible to install the `jaffle_finance` project as a `package` dependency. This will pull down its full source code and require dbt to parse all its contents. dbt will expect you to configure and run those models as your own. This can be a useful pattern to achieve certain types of unified deployments in production or to make a coordinated change across multiple projects in development. However, it can add significant complexity and latency when working within the narrower scope of a single project. |
There was a problem hiding this comment.
However, it can add significant complexity and latency when working within the narrower scope of a single project.
and cost!
mirnawong1
reviewed
Jul 13, 2023
mirnawong1
reviewed
Jul 13, 2023
| If I were to instead install the `jaffle_finance` project as a `package` dependency, this would pull down its full source code. Meaning: | ||
| - dbt needs to parse and resolve more inputs (which is slower) | ||
| - dbt expects you to configure these models as if they were your own (with `vars`, env vars, etc) | ||
| - dbt will run these models as my own, unless I explicitly `--exclude` them |
There was a problem hiding this comment.
Suggested change
| - dbt will run these models as my own, unless I explicitly `--exclude` them | |
| - dbt will run these models as your own unless you explicitly `--exclude` them |
mirnawong1
reviewed
Jul 13, 2023
| - dbt needs to parse and resolve more inputs (which is slower) | ||
| - dbt expects you to configure these models as if they were your own (with `vars`, env vars, etc) | ||
| - dbt will run these models as my own, unless I explicitly `--exclude` them | ||
| - I could be using the project's models in a way that their maintainer (the Finance team) hasn't intended |
There was a problem hiding this comment.
Suggested change
| - I could be using the project's models in a way that their maintainer (the Finance team) hasn't intended | |
| - You could be using the project's models in a way that their maintainer (the Finance team) hasn't intended |
mirnawong1
reviewed
Jul 13, 2023
| ``` | ||
|
|
||
| </File> | ||
|
|
There was a problem hiding this comment.
Suggested change
| ### Package use case | |
mirnawong1
reviewed
Jul 13, 2023
| The first is familiar: I want to use some of the macros that are already defined in `dbt_utils`. I pull down its full contents (100+ macros) as source code, and add them to my environment. I can then call any macro from the package, just as I can call macros that I define in my own project. | ||
|
|
||
| The second is new. Unlike installing a package, the models in the `jaffle_finance` project will not be pulled down as source code, and parsed into my project. Instead, dbt Cloud provides a metadata service that resolves references to [**public models**](/docs/collaborate/govern/model-access) defined in the `jaffle_finance` project. | ||
|
|
There was a problem hiding this comment.
Suggested change
| ### Projects use case |
mirnawong1
reviewed
Jul 13, 2023
| - I could be using the project's models in a way that their maintainer (the Finance team) hasn't intended | ||
|
|
||
| Installing another internal project as a package can be a useful pattern for: | ||
| - "Unified deployments" in production environments. If the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, using dbt's selection syntax, they could create a new "passthrough" project that installed both projects as packages. |
There was a problem hiding this comment.
Suggested change
| - "Unified deployments" in production environments. If the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, using dbt's selection syntax, they could create a new "passthrough" project that installed both projects as packages. | |
| - Unified deployments — In a production environment, if the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, they could use dbt's [selection syntax](/reference/node-selection/syntax) to create a new "passthrough" project that installed both projects as packages. |
mirnawong1
reviewed
Jul 13, 2023
|
|
||
| Installing another internal project as a package can be a useful pattern for: | ||
| - "Unified deployments" in production environments. If the central data platform team of Jaffle Shop wanted to schedule the deployment of models across both `jaffle_finance` and `jaffle_marketing`, using dbt's selection syntax, they could create a new "passthrough" project that installed both projects as packages. | ||
| - Making a coordinated change across multiple projects in development. If I wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`), and see how it impacts a downstream model (`jaffle_marketing.roi_by_channel`), before pushing the changes into a staging or production environment, I could install `jaffle_finance` as a package inside `jaffle_marketing`, pointing to a specific git branch. (If I find I regularly need to do end-to-end testing of changes across both projects, I should reexamine if this really represents a stable interface boundary.) |
There was a problem hiding this comment.
Suggested change
| - Making a coordinated change across multiple projects in development. If I wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`), and see how it impacts a downstream model (`jaffle_marketing.roi_by_channel`), before pushing the changes into a staging or production environment, I could install `jaffle_finance` as a package inside `jaffle_marketing`, pointing to a specific git branch. (If I find I regularly need to do end-to-end testing of changes across both projects, I should reexamine if this really represents a stable interface boundary.) | |
| - Coordinated changes — In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) _before_ introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary. |
mirnawong1
reviewed
Jul 13, 2023
| @@ -36,6 +36,8 @@ dbt Labs is committed to providing backward compatibility for all versions 1.x, | |||
|
|
|||
| [**Namespacing:**](/faqs/Models/unique-model-names) Model names can be duplicated across different namespaces (packages/projects), so long as they are unique within each package/project. We strongly encourage using [two-argument `ref`](/reference/dbt-jinja-functions/ref#two-argument-variant) when referencing a model from a different package/project. | |||
|
|
|||
| [**Project dependencies**](/docs/collaborate/govern/project-dependencies): Introduce `dependencies.yml`. Allow enforcing model access (public vs. protected/private) across project/package boundaries. Enable cross-project `ref` of public models, without requiring the installation of upstream source code, as a feature of dbt Cloud Enterprise. | |||
There was a problem hiding this comment.
Suggested change
| [**Project dependencies**](/docs/collaborate/govern/project-dependencies): Introduce `dependencies.yml`. Allow enforcing model access (public vs. protected/private) across project/package boundaries. Enable cross-project `ref` of public models, without requiring the installation of upstream source code, as a feature of dbt Cloud Enterprise. | |
| [**Project dependencies**](/docs/collaborate/govern/project-dependencies): Introduces `dependencies.yml` as a feature of dbt Cloud Enterprise. Allows enforcing model access (public vs. protected/private) across project/package boundaries. Enables cross-project `ref` of public models, without requiring the installation of upstream source code. |
mirnawong1
approved these changes
Jul 13, 2023
There was a problem hiding this comment.
some suggested changes here @jtcohen6 to tighten it up for the reader, let me know if you have any questions at all!
|
@mirnawong1 fantastic edits & suggestions!! thank you thank you thank you |
|
it was my absolute pleasure @jtcohen6 , thank you for the opportunity! this is all very exciting for users so i can't wait! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #3550
resolves #3632
resolves #3574
What are you changing in this pull request and why?
Create a new page for "cross-project
refundercollaborate > govern. I've decided to call the page "Project dependencies," and use it as an opportunity to highlight the differences between project + package dependencies.I started tackling two closely related issues, since we should be thematically consistent across all of them:
enforce_accessfor packages <> model accesspackagescan be configured in a file nameddependencies.ymlPreviews
Checklist
Adding new pages (delete if not applicable):
website/sidebars.js