Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add dbt test and fixtures #2580

Merged
merged 13 commits into from
Feb 7, 2024
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,8 @@ bench_data/

# Logs
*.log

# dbt compiled models
tests/fixtures/dbt_project/target/
tests/fixtures/dbt_project/.user.yml
tests/fixtures/dbt_project/logs/
36 changes: 36 additions & 0 deletions tests/fixtures/dbt_project/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'glaredb_dbt_test'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'glaredb_dbt_test'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
test-paths: ["tests"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.

models:
dbt_test:
# Config indicated by + and applies to all files under models/example/
glaredb_data:
+materialized: view
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
select *
from {{ source('public', 'dbt_test')}}
7 changes: 7 additions & 0 deletions tests/fixtures/dbt_project/models/glaredb_data/sources.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
version: 2

sources:
- name: public
database: glaredb
tables:
- name: dbt_test
13 changes: 13 additions & 0 deletions tests/fixtures/dbt_project/profiles.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
glaredb_dbt_test:
outputs:
test_target:
dbname: glaredb
host: 127.0.0.1
pass: test
port: 5432
schema: public
threads: 1
type: postgres
user: test
sslmode: require
target: test_target
45 changes: 45 additions & 0 deletions tests/tests/test_dbt.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import psycopg2.extensions
import pytest

from tests.fixtures.glaredb import glaredb_connection

from dbt.cli.main import dbtRunner, dbtRunnerResult


def test_dbt_glaredb(
glaredb_connection: psycopg2.extensions.connection,
):
dbt: dbtRunner = dbtRunner()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if you can just add this as an argument to the test to see if the magic pytest fixture thing works.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you say more here? Are you suggesting I create a separate dbtRunner fixture?


model_name: str = "glaredb_model" # TODO
project_directory: str = "../fixtures/dbt_project/" # TODO
dbt_profiles_directory: str = "../fixtures/dbt_project/" # TODO

curr = glaredb_connection.cursor()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

context manager for cursors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. I followed the pattern here. Should we update that?


curr.execute("create table dbt_test (amount int)")
curr.execute(
"""INSERT INTO dbt_test (amount) VALUES (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)"""
talagluck marked this conversation as resolved.
Show resolved Hide resolved
)

cli_args: list = [
"run",
"--project-dir",
project_directory,
"--profiles-dir",
dbt_profiles_directory,
"-m",
model_name
]
#
res: dbtRunnerResult = dbt.invoke(cli_args)

# The below will currently fail. res contains the error message, but that message can be in different places based
talagluck marked this conversation as resolved.
Show resolved Hide resolved
# on where the failure is. Currently, it is under the top level `res.exception` key.

assert res.success is True

curr = glaredb_connection.cursor()
talagluck marked this conversation as resolved.
Show resolved Hide resolved
query_result: list = curr.execute(f"""select * from {model_name}""").fetchall()
talagluck marked this conversation as resolved.
Show resolved Hide resolved

assert len(query_result) == 10
Loading