Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3078] Improve diff displayed in stdout when a test fails #8558

Closed
Tracked by #8283
MichelleArk opened this issue Sep 5, 2023 · 5 comments
Closed
Tracked by #8283

[CT-3078] Improve diff displayed in stdout when a test fails #8558

MichelleArk opened this issue Sep 5, 2023 · 5 comments
Assignees
Labels
user docs [docs.getdbt.com] Needs better documentation

Comments

@MichelleArk
Copy link
Contributor

MichelleArk commented Sep 5, 2023

Description

Currently, a failing unit test simply logs both the actual and expected results to stdout:

❯ dbt unit-test
20:34:18  Running with dbt=1.7.0-b1
20:34:18  target not specified in profile 'postgres', using 'default'
20:34:18  Registered adapter: postgres=1.7.0-b1
20:34:19  Found 10 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 360 macros, 0 groups, 0 semantic models, 5 unit tests
20:34:19  Found 1 unit test, 2 models, 0 sources, 0 exposures, 0 metrics, 360 macros, 0 groups, 0 semantic models
20:34:19  
20:34:19  Concurrency: 1 threads (target='default')
20:34:19  
20:34:19  1 of 1 START unit_test my_model__test_my_model ................................. [RUN]
20:34:19  1 of 1 FAIL 1 my_model__test_my_model .......................................... [FAIL 1 in 0.11s]
20:34:19  
20:34:19  Finished running 1 unit_test in 0 hours 0 minutes and 0.53 seconds (0.53s).
20:34:19  
20:34:19  Completed with 1 error and 0 warnings:
20:34:19  
20:34:19  Failure in unit_test my_model__test_my_model (models/unit_testing/schema.yml)
20:34:19    

Actual:
| c |
| - |
| 3 |

Expected:
| c |
| - |
| 4 |

20:34:19  
20:34:19    compiled code at target/compiled/jaffle_shop/models/unit_testing/schema.yml/models/unit_testing/my_model__test_my_model.sql
20:34:19  
20:34:19  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

This may work fine for very small expected results (few rows & columns), but would be difficult to debug for larger results as the diff is not visualized. For example:

❯ dbt unit-test
...
Found 1 differing row between actual and expected outputs for my_model:
Actual
- {'c': 3}
          ^
Expected
+ {'c': 4}

Acceptance Criteria

Test failure message displays diff in a readable format

Impact to other teams

None

Will backports be required?

No

Context

@github-actions github-actions bot changed the title Improve diff displayed in stdout when a test fails [CT-3078] Improve diff displayed in stdout when a test fails Sep 5, 2023
@dbeatty10
Copy link
Contributor

Of the formats I've seen thus far, the daff format is the one I prefer since it makes it easier to see exactly where the differences are. The tabular diff specification for daff is described in more detail here.

image

@graciegoheen
Copy link
Contributor

graciegoheen commented Sep 22, 2023

Since we're allowing folks to define fixtures in separate csv files, I think it would be good to include the input mock data in the logs as well so they don't have to go splunking through a ton of files.

@MichelleArk
Copy link
Contributor Author

@dbeatty10 - I had a chance to spike how this might work with daff here: #8984!

@dbeatty10
Copy link
Contributor

@dbeatty10 - I had a chance to spike how this might work with daff here: #8984!

Ooooh... 🤩 beautiful output and elegant implementation!

@graciegoheen graciegoheen added the user docs [docs.getdbt.com] Needs better documentation label Nov 7, 2023
@graciegoheen
Copy link
Contributor

graciegoheen commented Nov 7, 2023

From @dbeatty10 for docs

We should let consumers know that the output format is an experimental and unstable interface designed for human eyes (rather than machines) and that we reserve the right to change the diff format that is printed to the terminal and in the logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
user docs [docs.getdbt.com] Needs better documentation
Projects
None yet
Development

No branches or pull requests

3 participants