[DANGER] Update to Fideslang 2.0 #3933

ThomasLaPiana · 2023-08-17T06:48:25Z

❗ Customer config.json files should be updated as they are migrated to the accompanying release.

Description Of Changes

This PR is focused on bumping the Fideslang version and ensuring that the application is working as expected when set u as a new instance. There is a separate PR for handling migrations for current users, as they are both sufficiently complex pieces to warrant handling separately.

Much of the inspiration/work here is inspired by this PR that handled an earlier, similar change

The Data Migration work is happening here: #4030

WARNING: Merging this change as-is will break any existing instance due to default taxonomy model conflicts

Code Changes

bump requirements.txt
add a migration for the new version fields
Update DataCategories
- datasets (including collections/fields)
- policy rule (tested previously, code is copy/pasta)
- privacy declaration (tested previously, code is copy/pasta)
- data flows (egress and ingress on System models)
Update DataUses
- privacy declaration (tested previously, code is copy/pasta)
- policy rule (tested previously, code is copy/pasta)
do find/replace for the old categories/uses

Steps to Confirm

Because this PR is focused on getting new instances working instead of migrating data, the Steps to Confirm here are very simple. Tests passing is what we care about here! This PR is where the complex data migration will take place

after running nox -s teardown -- volumes and nox -s dev, everything works as intended

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

cypress · 2023-08-17T07:02:24Z

Passing run #4110 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `4a271f9` into `7f53a75`...
Project: fides	Commit: `97e809de64 ℹ️`
Status: Passed	Duration: 01:18 💡
Started: Sep 13, 2023 6:48 PM	Ended: Sep 13, 2023 6:49 PM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

ThomasLaPiana · 2023-08-17T17:43:01Z

@adamsachs update for you before I sign off!

In the description I've linked the fideslang PR where I'm making changes as needed

I'm currently working through a local dev env and shell to run specific tests and try to diagnose why some of the breaks are happening. Focused primarily on the ctl tests as they were easier to run and faster

src/fides/api/api/v1/endpoints/user_endpoints.py

…edentials_valid function

src/fides/api/db/seed.py

adamsachs · 2023-08-17T21:30:05Z

@ThomasLaPiana stepping away for a bit now, made a few changes that i think should help get things greener...tried to break it up reasonably by commit so that my changes are easy to follow -- everything is pretty straightforward cleanup so far.

will see if i can circle back a bit later this evening to knock out some more, but leaving this update here in case i don't 👍

ThomasLaPiana · 2023-08-18T03:37:19Z

@adamsachs Thank you! picking it up now

ThomasLaPiana · 2023-08-18T10:48:11Z

there's a final system update test failing here that I suspect @pattisdr might understand, other than that the ctl tests are looking good!

adamsachs · 2023-08-18T10:59:01Z

there's a final system update test failing here that I suspect @pattisdr might understand, other than that the ctl tests are looking good!

🙌 nice @ThomasLaPiana! i think i wrote those system update tests related to privacy declarations so i can take a look, i'll loop in @pattisdr if needed.

@ThomasLaPiana are you working on the ops tests or should i also take a look at that? i've got some bandwidth so i'm happy to, but if you're already on it then i'll let you continue

ThomasLaPiana · 2023-08-18T11:02:59Z

@adamsachs Go for it! I'm signing off for now

ThomasLaPiana · 2023-08-18T11:08:24Z

@adamsachs once we get tests passing, we'll merge and ship fideslang 2.0.1 and pin this to a real version

codecov · 2023-08-18T14:38:08Z

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.04% ⚠️

Comparison is base (7f53a75) 87.39% compared to head (e7ecbb4) 87.36%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3933      +/-   ##
==========================================
- Coverage   87.39%   87.36%   -0.04%     
==========================================
  Files         320      320              
  Lines       19602    19617      +15     
  Branches     2512     2512              
==========================================
+ Hits        17132    17139       +7     
- Misses       2033     2040       +7     
- Partials      437      438       +1

Files Changed	Coverage Δ
.../api/api/v1/endpoints/consent_request_endpoints.py	`85.12% <ø> (ø)`
.../api/api/v1/endpoints/privacy_request_endpoints.py	`90.34% <ø> (ø)`
src/fides/api/db/seed.py	`89.20% <ø> (ø)`
src/fides/api/models/privacy_request.py	`96.30% <ø> (ø)`
src/fides/api/schemas/privacy_request.py	`100.00% <ø> (ø)`
.../api/service/connectors/consent_email_connector.py	`97.14% <ø> (ø)`
...vice/saas_request/saas_request_override_factory.py	`100.00% <ø> (ø)`
src/fides/api/util/data_category.py	`100.00% <ø> (ø)`
.../fides/api/api/v1/endpoints/messaging_endpoints.py	`92.94% <100.00%> (+0.12%)`	⬆️
src/fides/api/models/sql_models.py	`98.12% <100.00%> (+0.07%)`	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

adamsachs · 2023-08-18T15:21:49Z

OK! i think we've got all non-external tests passing now!

@pattisdr would you mind taking a quick peak at the ops tests i fixed in 8ec6a87, since some of them are touching consent-related tests and i want to be extra sure i'm not misunderstanding the tests and invalidating them 🙏

if that looks good, i think we've got our non-external/unsafe tests good to go! the remaining items to get this ready to merge would be:

look into external test failures to determine whether these are expected/in main already or caused by some code changes here
holistic review of this PR (i've focused on getting CI green up to this point)
cut fideslang 2.0.1 per @ThomasLaPiana's note above, and update our dependency here

i should be able to make some progress on the first two items shortly...

ThomasLaPiana · 2023-09-02T11:57:45Z

spoke a bit with @pattisdr and one theory that could explain this exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup type of error from cropping up in all sorts of (seemingly unrelated) places all of a sudden is that some recent change (likely in a dependency of ours?) is causing underlying errors to get swallowed and instead be thrown up with this generic exception type.

@ThomasLaPiana do you have any further leads? i haven't dug into this much, just wanted to put this theory down on paper here. i think @pattisdr is also continuing further investigation on her end 👍

I moved this to a new issue: #4020

ThomasLaPiana · 2023-09-05T04:43:33Z

I'm moving the migration-specific work to a new PR, to de-clutter this PR and focus it specifically on getting fides running with 2.0

src/fides/api/alembic/migrations/versions/708a780b01ba_add_version_fields_to_default_types.py

pattisdr · 2023-09-12T19:07:00Z

I think we need to update the config.json files we ship with, PR here #4076 although I'm going to need some FE assistance

EDIT: done, thank you Allison!

pattisdr

Did another review here, although I didn't go through every single open comment.

Giving this a tentative 👍 to not hold up the overnight crew, assuming the corresponding migration will be merged into this branch first, this branch will be updated with main, comments will be resolved, and that tests are passing.

❗ Note that main has had a new migration merged in, you'll need to bump the downrev of this migration to match that one.

Separately, I think we need the config.json files we ship with to be updated, but FE help is needed #4076. We could also merge this in separately tomorrow.

Co-authored-by: Allison King <allisonjuliaking@gmail.com>

pattisdr · 2023-09-13T02:22:26Z

OK @ThomasLaPiana, Allison helped me with the FE, so these config.json changes are merged here -

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

ThomasLaPiana · 2023-09-13T09:59:49Z

it looks like fideslang 2.0 was missing a change that got patched into 1.4.6, I cut another release of Fideslang (2.0.3) that should contain the change and get things working here

adamsachs

🎊 let's do this!

src/fides/api/alembic/migrations/versions/708a780b01ba_add_version_fields_to_default_types.py

Update Fideslang to version 2.0.0

fc08df8

ThomasLaPiana self-assigned this Aug 17, 2023

ThomasLaPiana added 5 commits August 17, 2023 18:17

feat: get cod in place for the find/replace

c180cd5

feat: checkpoint

674769f

feat: pin a new fideslang version to get migrations working

0e28ef1

feat: find/replace new categories and uses

fd5dcbe

feat: bump fideslang version again for more fixes

78ab451

fix: ctl api tests

655fcbf

adamsachs reviewed Aug 17, 2023

View reviewed changes

src/fides/api/api/v1/endpoints/user_endpoints.py Outdated Show resolved Hide resolved

fix: revert mistaken user.authorization.credentials_valid --> user.cr…

18c493c

…edentials_valid function

adamsachs reviewed Aug 17, 2023

View reviewed changes

src/fides/api/db/seed.py Outdated Show resolved Hide resolved

adamsachs added 3 commits August 17, 2023 17:13

remove unused exception var

9c4fe11

feat: add new taxonomy model columns to db dataset annotation

a887ec2

feat: update to use new data_use in systems.yml

b26a26f

ThomasLaPiana added 3 commits August 18, 2023 13:09

fix: more ctl tests

e240949

feat: add number labels for tests with param

4ac028b

Merge branch 'main' into ThomasLaPiana-update-fideslang-200

f37ad0f

ThomasLaPiana added the run unsafe ci checks Runs fideops-related CI checks that require sensitive credentials label Aug 18, 2023

adamsachs added 2 commits August 18, 2023 08:42

fix: ctl test using outdated 'improve' data_use

8992d13

fix: clean up ops tests based on updates

8ec6a87

ThomasLaPiana changed the title ~~Update Fideslang to version 2~~ Update to Fideslang 2.0 Sep 5, 2023

Merge branch 'main' into ThomasLaPiana-update-fideslang-200

286d082

ThomasLaPiana and others added 2 commits September 6, 2023 11:16

Merge branch 'main' into ThomasLaPiana-update-fideslang-200

00fc4ef

feat: pull out the data migration and get the pure upgrade working

08cb5c6

ThomasLaPiana mentioned this pull request Sep 6, 2023

Data Migration for Fideslang 2.0 #4030

Merged

22 tasks

fix: loguru-related test error

289c16c

ThomasLaPiana requested a review from pattisdr September 6, 2023 10:16

ThomasLaPiana added the high-risk This issue suggests changes that have a high-probability of breaking existing code label Sep 6, 2023

ThomasLaPiana changed the title ~~Update to Fideslang 2.0~~ [DANGER] Update to Fideslang 2.0 Sep 6, 2023

fix: drop default uses and categories as part of the 2.0 upgrade

325dd6e

adamsachs reviewed Sep 8, 2023

View reviewed changes

src/fides/api/alembic/migrations/versions/708a780b01ba_add_version_fields_to_default_types.py Show resolved Hide resolved

fix: drop default data_qualifiers and data_subjects as part of migration

87bc209

pattisdr approved these changes Sep 12, 2023

View reviewed changes

Update config.json Improve -> Functional (#4076)

b6aecaa

Co-authored-by: Allison King <allisonjuliaking@gmail.com>

ThomasLaPiana and others added 3 commits September 13, 2023 15:45

Merge branch 'main' into ThomasLaPiana-update-fideslang-200

3ada9cc

fix: migration order

702f349

Data Migration for Fideslang 2.0 (#4030)

3d3a67e

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

ThomasLaPiana added 2 commits September 13, 2023 18:14

feat: bump fideslang version

699074f

fix: seed.py updated to use user.authorization

e7ecbb4

ThomasLaPiana requested a review from adamsachs September 13, 2023 16:37

adamsachs approved these changes Sep 13, 2023

View reviewed changes

adamsachs reviewed Sep 13, 2023

View reviewed changes

src/fides/api/alembic/migrations/versions/708a780b01ba_add_version_fields_to_default_types.py Outdated Show resolved Hide resolved

reference correct down_rev in migration comment

4a271f9

adamsachs merged commit e8f656f into main Sep 13, 2023
42 of 49 checks passed

adamsachs deleted the ThomasLaPiana-update-fideslang-200 branch September 13, 2023 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DANGER] Update to Fideslang 2.0 #3933

[DANGER] Update to Fideslang 2.0 #3933

ThomasLaPiana commented Aug 17, 2023 •

edited by pattisdr

Loading

cypress bot commented Aug 17, 2023 •

edited

Loading

ThomasLaPiana commented Aug 17, 2023 •

edited

Loading

adamsachs commented Aug 17, 2023

ThomasLaPiana commented Aug 18, 2023

ThomasLaPiana commented Aug 18, 2023

adamsachs commented Aug 18, 2023 •

edited

Loading

ThomasLaPiana commented Aug 18, 2023

ThomasLaPiana commented Aug 18, 2023

codecov bot commented Aug 18, 2023 •

edited

Loading

adamsachs commented Aug 18, 2023 •

edited by ThomasLaPiana

Loading

ThomasLaPiana commented Sep 2, 2023

ThomasLaPiana commented Sep 5, 2023

pattisdr commented Sep 12, 2023 •

edited

Loading

pattisdr left a comment •

edited

Loading

pattisdr commented Sep 13, 2023

ThomasLaPiana commented Sep 13, 2023

adamsachs left a comment

[DANGER] Update to Fideslang 2.0 #3933

[DANGER] Update to Fideslang 2.0 #3933

Conversation

ThomasLaPiana commented Aug 17, 2023 • edited by pattisdr Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Aug 17, 2023 • edited Loading

Passing run #4110 ↗︎

ThomasLaPiana commented Aug 17, 2023 • edited Loading

adamsachs commented Aug 17, 2023

ThomasLaPiana commented Aug 18, 2023

ThomasLaPiana commented Aug 18, 2023

adamsachs commented Aug 18, 2023 • edited Loading

ThomasLaPiana commented Aug 18, 2023

ThomasLaPiana commented Aug 18, 2023

codecov bot commented Aug 18, 2023 • edited Loading

Codecov Report

adamsachs commented Aug 18, 2023 • edited by ThomasLaPiana Loading

ThomasLaPiana commented Sep 2, 2023

ThomasLaPiana commented Sep 5, 2023

pattisdr commented Sep 12, 2023 • edited Loading

pattisdr left a comment • edited Loading

Choose a reason for hiding this comment

pattisdr commented Sep 13, 2023

ThomasLaPiana commented Sep 13, 2023

adamsachs left a comment

Choose a reason for hiding this comment

ThomasLaPiana commented Aug 17, 2023 •

edited by pattisdr

Loading

cypress bot commented Aug 17, 2023 •

edited

Loading

ThomasLaPiana commented Aug 17, 2023 •

edited

Loading

adamsachs commented Aug 18, 2023 •

edited

Loading

codecov bot commented Aug 18, 2023 •

edited

Loading

adamsachs commented Aug 18, 2023 •

edited by ThomasLaPiana

Loading

pattisdr commented Sep 12, 2023 •

edited

Loading

pattisdr left a comment •

edited

Loading