Ability to write to files with Jinja

[randypitcherii]

Describe the feature

As a user, I would like a Jinja context built-in function similar to print that allows me to write to files.

Describe alternatives you've considered

Today I largely print generated code to stdout.

Who will this benefit?

The use case I have in mind is to allow the results of the codegen package to actually persist generated code to dbt yaml files directly.

Are you interested in contributing this feature?

Yes!

[jtcohen6]

Hmm. Hmm! I'm not sure if or how we could realistically support this. I so appreciate you opening the issue and starting the conversation, @randypitcherii.

Today, dbt does write to files, with Jinja, sort of: It compiles Jinja templates, from files, and writes them to other files (in the target/ directory). All the code that says how to create files in a given path, and how to write to those files, is decidedly in python. And the python for the first bit in particular leverages the os module, which we cannot expose to the Jinja context: (a) for security reasons, and (b) I can imagine lots of unpleasant surprises if macros / dbt invocations make changes to the project file system while running (target/ files excepted).

All that said...

The use case I have in mind is to allow the results of the codegen package to actually persist generated code to dbt yaml files directly.

This is compelling, I buy it, and I know there are limitations to the two ways of working around this today.

The bash-y way

Leveraging the fact that dbt/Jinja can write anything its heart desires to stdout, pipe the stdout contents somewhere else.

$ dbt run-operation generate_source --args 'schema_name: jaffle_shop' > models/staging/jaffle_shop/src_jaffle_shop.yml

Limitations:

• The simple pipe includes log messages printed by dbt, e.g. Running with dbt=0.20.0-rc1
• Totally reliant on CLI tooling. Not possible in dbt Cloud.

The file-based way

Stick each macro in a SQL file in the models/ (or even analysis/) folder, compile the project, copy-paste the contents from target/ (or just... move the files from target/ into models/).

Limitations:

• Requires the creation of one "dummy" file for every "final" file desired
• This isn't so bad when it's a SQL file, containing a macro, that compiles SQL, to end up as a SQL file. But it's fairly unintuitive when it wants to end up as a YAML file—it still needs to start as a SQL file, compile, move, change the file extension.

At some point, I'm so on board with the request—I'm just not sure if dbt Core can be the right tool for the job, or it's a better fit for a helper tool written in python.