Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of metadata-based freshness #796

Merged
merged 27 commits into from
Oct 12, 2023
Merged

Implementation of metadata-based freshness #796

merged 27 commits into from
Oct 12, 2023

Conversation

peterallenwebb
Copy link
Contributor

@peterallenwebb peterallenwebb commented Oct 9, 2023 •
edited
Loading

resolves #785

Problem

Solution

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@cla-bot cla-bot bot added the cla:yes label Oct 9, 2023
@peterallenwebb peterallenwebb marked this pull request as ready for review October 10, 2023 20:48
@peterallenwebb peterallenwebb requested a review from a team as a code owner October 10, 2023 20:48
@peterallenwebb peterallenwebb changed the title Draft implementation of metadata-based freshness Implementation of metadata-based freshness Oct 10, 2023
McKnight-42
McKnight-42 reviewed Oct 10, 2023
View reviewed changes
dev-requirements.txt Outdated Show resolved Hide resolved
mikealfare
mikealfare reviewed Oct 10, 2023
View reviewed changes
dbt/adapters/snowflake/impl.py Outdated Show resolved Hide resolved
dbt/adapters/snowflake/impl.py Outdated Show resolved Hide resolved
dbt/include/snowflake/macros/metadata.sql Show resolved Hide resolved
dbt/include/snowflake/macros/metadata.sql Show resolved Hide resolved
tests/functional/adapter/test_get_last_relation_modified.py Show resolved Hide resolved
dbt/adapters/snowflake/impl.py Outdated Show resolved Hide resolved
peterallenwebb and others added 15 commits October 10, 2023 22:25
@peterallenwebb
Simplifications per review
8e3fd10
@colin-rogers-dbt
Merge branch 'main' into paw/metadata-freshness
d909e56
@peterallenwebb
Merge remote-tracking branch 'origin/main' into paw/metadata-freshness
fbd0858
@peterallenwebb
Update for core code review changes
d04431a
@peterallenwebb
Merge branch 'main' into paw/metadata-freshness
3f1ee07
@peterallenwebb
Repoint to main core branch before merge
231fe49
@peterallenwebb
Merge remote-tracking branch 'origin/main' into paw/metadata-freshness
ffedcc3
@peterallenwebb
Temporarily skip test
cd7e3c8
@colin-rogers-dbt
test using custom schema for test
4d8647e
@colin-rogers-dbt
Merge remote-tracking branch 'origin/paw/metadata-freshness' into paw…
cb2570f 
…/metadata-freshness

# Conflicts:
#	tests/functional/adapter/test_get_last_relation_modified.py
@mikealfare
Merge branch 'main' into paw/metadata-freshness
f131751
@colin-rogers-dbt
use specific schema env var
f12d00e
@colin-rogers-dbt
use specific schema env var
61929da
@colin-rogers-dbt
cleanup env var
bff9f6c
@colin-rogers-dbt
skip test_get_last_relation_modified
fc47ba3
@peterallenwebb peterallenwebb merged commit 8970aaa into main Oct 12, 2023
25 checks passed
@peterallenwebb peterallenwebb deleted the paw/metadata-freshness branch October 12, 2023 02:09
philippe-boyd-maxa pushed a commit to maxa-ai/dbt-snowflake that referenced this pull request Nov 27, 2023
@peterallenwebb @colin-rogers-dbt
@mikealfare @philippe-boyd-maxa
Implementation of metadata-based freshness (dbt-labs#796)
29a01dc 
* Draft implementation of metadata-based freshness

* Add test of metadata-based freshness mechanism

* Strengthen test case.

* Fix some overzealous escaping.

* Simplifications per review

* Update for core code review changes

* Repoint to main core branch before merge

* Temporarily skip test

* test using custom schema for test

* use specific schema env var

* use specific schema env var

* cleanup env var

* skip test_get_last_relation_modified

* remove extra whitespace

* remove Capability.TableLastModifiedMetadata: CapabilitySupport(support=Support.Full) for CI troubleshooting

* remove metadata.sql for CI troubleshooting

* remove test_get_last_relation_modified.py for CI troubleshooting

* add non-test files back for CI troubleshooting

* add test files back for CI troubleshooting

* remove run order dependency from test cases within their test class

---------

Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
Co-authored-by: Colin <colin.rogers@dbtlabs.com>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@McKnight-42 McKnight-42 McKnight-42 left review comments

@mikealfare mikealfare mikealfare approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[ADAP-910] Support Metadata Freshness
4 participants
@peterallenwebb @mikealfare @McKnight-42 @colin-rogers-dbt