Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-1271] Test Query Comment conversion #5971

Merged
merged 15 commits into from
Oct 11, 2022
Merged

[CT-1271] Test Query Comment conversion #5971

merged 15 commits into from
Oct 11, 2022

Conversation

McKnight-42
Copy link
Contributor

@McKnight-42 McKnight-42 commented Sep 28, 2022

resolves #5962
issue: #6272 (comment)
Description:
converting integration test 051 to a new functional test
todo:

  1. make sure new test works on dbt-core and all required adapters including:
  • Bigquery
  • Snowflake
  • Redshift
  1. check to see if we need to add to dbt-spark

Checklist

@McKnight-42 McKnight-42 added the Skip Changelog Skips GHA to check for changelog file label Sep 28, 2022
@McKnight-42 McKnight-42 self-assigned this Sep 28, 2022
@cla-bot cla-bot bot added the cla:yes label Sep 28, 2022
@McKnight-42 McKnight-42 marked this pull request as ready for review October 3, 2022 14:41
@McKnight-42 McKnight-42 requested review from a team as code owners October 3, 2022 14:41
@McKnight-42
Copy link
Contributor Author

McKnight-42 commented Oct 3, 2022

Few open questions for discussion:

  1. This was the most clear path for least amount of duplication across core and adapters how do we feel about it? (know we would prefer to not have adapter specific stuff in the model logic of core.).

  2. Would we like to try and add a test to dbt-spark?

@McKnight-42 McKnight-42 requested a review from Fleid October 3, 2022 14:50
Copy link
Contributor

@colin-rogers-dbt colin-rogers-dbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@gshank gshank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a few changes.

{% set required = ['name', 'schema', 'type', 'threads'] %}

{# Require what we docuement at https://docs.getdbt.com/docs/target #}
{% if target.type == 'postgres' or target.type == 'redshift' %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with doing it this way is that it doesn't support external adapter maintainers very well, and we don't actually provide an example of how to get this working with an external adapter except by replacing this entire model. In addition, putting the test in jinja model code is kind of a bad pattern.

It would be better to put the required fields in some kind of config that can be retrieved. Also, it would probably be better to move the checks into the actual test code, rather than embedding it in the model. I'd make the "required" fields a fixture that can be overridden in the adapter specific tests, and then pull that into the test method and test for the existence of those fields in the target in the test. I'm not exactly sure how you can get the target dictionary. Possibly from the adapter? I can help look for how to get that if you need.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not even sure why we're doing this in this test. What does this have to do with query comments?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as we talked about i'm a little hazy on why this is here other than validation logic possibly against if we did import it to a different database to make sure its not actually relying off say postgres.

would love to find a better way to work on this will play around some of these other options if we do think this chucnk is needed at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @gerda had a few questions/ideas are we able to set up some global stuff for checks in a fixture that can't be overwritten by chance? like the
{% set required = ['name', 'schema', 'type', 'threads'] %} that might not change between the adapters?

trying to get ahold of the related information that would be same as target in the fixture looks like from our TestProjInfo something like profiles_dir would contain the information we would want just not sure if theres a good way to tap into that possibly by some os read stuff?

if seen:
"Never saw a matching log message! Logs:\n{}".format("\n".join(log["msg"]))

def test_comments(self, project):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason for just forwarding to another method here. Just

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just? @gshank did something get cutoff here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe? Probably was saying "just call the method directly"

Copy link
Contributor

@gshank gshank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@McKnight-42 McKnight-42 merged commit 2c68424 into main Oct 11, 2022
@McKnight-42 McKnight-42 deleted the mcknight/CT-1271 branch October 11, 2022 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla:yes Skip Changelog Skips GHA to check for changelog file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-1271] 051_query_comments_tests
3 participants