Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend DbtTask with additional supported fields and document them #1537

Merged
merged 5 commits into from
Aug 30, 2022
Merged

Extend DbtTask with additional supported fields and document them #1537

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
20139a2
Add ProfilesDirectory and WarehouseId to DbtTask
gaborratky-db Aug 15, 2022
8f63b1d
Allow DbtTask to be specified as a single task
gaborratky-db Aug 15, 2022
83d09f4
Add dbt_task to docs/resources/job.md
gaborratky-db Aug 15, 2022
b77d677
Call out `git_source` in argument summaries
gaborratky-db Aug 16, 2022
ba91934
Merge branch 'master' into extend-jobs-dbt-task
gaborratky-db Aug 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/resources/job.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,16 @@ You can invoke Spark submit tasks only on new clusters. **In the `new_cluster` s
* `parameters` - (Optional) Parameters for the task
* `named_parameters` - (Optional) Named parameters for the task

### dbt_task Configuration Block

* `commands` - (Required) (Array) Series of dbt commands to execute in sequence. Every command must start with "dbt".
* `project_directory` - (Optional) The relative path to the directory in the repository where dbt should look in for the `dbt_project.yml` file. If not specified, defaults to the repository's root directory. Equivalent to passing `--project-dir` to a dbt command.
gaborratky-db marked this conversation as resolved.
Show resolved Hide resolved
* `profiles_directory` - (Optional) The relative path to the directory in the repository where dbt should look in for the `profiles.yml` file. If not specified, defaults to the repository's root directory. Equivalent to passing `--profile-dir` to a dbt command.
gaborratky-db marked this conversation as resolved.
Show resolved Hide resolved
* `schema` - (Optional) The name of the schema dbt should run in. Defaults to `default`.
* `warehouse_id` - (Optional) The ID of the SQL warehouse that dbt should execute against.

You also need to include a `git_source` block to configure the repository that contains the dbt project.

### email_notifications Configuration Block

* `on_failure` - (Optional) (List) list of emails to notify on failure
Expand Down
9 changes: 6 additions & 3 deletions jobs/resource_job.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,11 @@ type SqlTask struct {
// DbtTask contains information about DBT task
// TODO: add validation for non-empty commands
type DbtTask struct {
ProjectDirectory string `json:"project_directory,omitempty"`
Commands []string `json:"commands"`
Schema string `json:"schema,omitempty" tf:"default:default"`
Commands []string `json:"commands"`
ProfilesDirectory string `json:"profiles_directory,omitempty"`
ProjectDirectory string `json:"project_directory,omitempty"`
Schema string `json:"schema,omitempty" tf:"default:default"`
WarehouseId string `json:"warehouse_id,omitempty"`
}

// EmailNotifications contains the information for email notifications after job completion
Expand Down Expand Up @@ -159,6 +161,7 @@ type JobSettings struct {
SparkSubmitTask *SparkSubmitTask `json:"spark_submit_task,omitempty" tf:"group:task_type"`
PipelineTask *PipelineTask `json:"pipeline_task,omitempty" tf:"group:task_type"`
PythonWheelTask *PythonWheelTask `json:"python_wheel_task,omitempty" tf:"group:task_type"`
DbtTask *DbtTask `json:"dbt_task,omitempty" tf:"group:task_type"`
Libraries []libraries.Library `json:"libraries,omitempty" tf:"slice_set,alias:library"`
TimeoutSeconds int32 `json:"timeout_seconds,omitempty"`
MaxRetries int32 `json:"max_retries,omitempty"`
Expand Down