Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DANGER] Update to Fideslang 2.0 #3933

Merged
merged 54 commits into from
Sep 13, 2023
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
fc08df8
Update Fideslang to version 2.0.0
ThomasLaPiana Aug 17, 2023
c180cd5
feat: get cod in place for the find/replace
ThomasLaPiana Aug 17, 2023
674769f
feat: checkpoint
ThomasLaPiana Aug 17, 2023
0e28ef1
feat: pin a new fideslang version to get migrations working
ThomasLaPiana Aug 17, 2023
fd5dcbe
feat: find/replace new categories and uses
ThomasLaPiana Aug 17, 2023
78ab451
feat: bump fideslang version again for more fixes
ThomasLaPiana Aug 17, 2023
655fcbf
fix: ctl api tests
ThomasLaPiana Aug 17, 2023
18c493c
fix: revert mistaken user.authorization.credentials_valid --> user.cr…
adamsachs Aug 17, 2023
9c4fe11
remove unused exception var
adamsachs Aug 17, 2023
a887ec2
feat: add new taxonomy model columns to db dataset annotation
adamsachs Aug 17, 2023
b26a26f
feat: update to use new data_use in systems.yml
adamsachs Aug 17, 2023
e240949
fix: more ctl tests
ThomasLaPiana Aug 18, 2023
4ac028b
feat: add number labels for tests with param
ThomasLaPiana Aug 18, 2023
f37ad0f
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Aug 18, 2023
8992d13
fix: ctl test using outdated 'improve' data_use
adamsachs Aug 18, 2023
8ec6a87
fix: clean up ops tests based on updates
adamsachs Aug 18, 2023
b4e266d
feat: start adding migrations for data categories
ThomasLaPiana Aug 21, 2023
96cc067
feat: add a data migration for the datasets
ThomasLaPiana Aug 21, 2023
6c97d88
feat: update to the new pinned fideslang version
ThomasLaPiana Aug 21, 2023
c0aabac
feat: add system updates
ThomasLaPiana Aug 22, 2023
80276e5
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Aug 22, 2023
526b023
feat: small naming nit
ThomasLaPiana Aug 22, 2023
7b22f0e
fix: migrations
ThomasLaPiana Aug 22, 2023
72c8797
checkin
ThomasLaPiana Aug 23, 2023
2f2eb0f
fix: migrations for egress/ingress on systems
ThomasLaPiana Aug 23, 2023
24c6c17
fix: incorrect function calls
ThomasLaPiana Aug 25, 2023
c79cbe8
Apply suggestions from code review
ThomasLaPiana Aug 26, 2023
166ac04
fix: misname
ThomasLaPiana Aug 26, 2023
b160e17
Merge branch 'ThomasLaPiana-update-fideslang-200' of https://github.c…
ThomasLaPiana Aug 26, 2023
887e768
feat: update the data migration to do broader replacement in order to…
ThomasLaPiana Aug 28, 2023
791f67f
fix: use 'startswith' instead of 'in' to avoid overwriting changes
ThomasLaPiana Aug 28, 2023
bb67fa1
fix: don't filter out categories that aren't updated
ThomasLaPiana Aug 29, 2023
26e1203
bump to new fideslang release to keep up to date
adamsachs Aug 29, 2023
1d0ef04
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
adamsachs Aug 29, 2023
4249d8b
update down rev on migration
adamsachs Aug 29, 2023
d1d112e
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Aug 30, 2023
b4fd43e
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Aug 31, 2023
3792c71
fix: review comments
ThomasLaPiana Aug 31, 2023
321d4b2
Merge branch 'ThomasLaPiana-update-fideslang-200' of https://github.c…
ThomasLaPiana Aug 31, 2023
ccf118f
feat: add a migration step that removes all "old" default categories …
ThomasLaPiana Aug 31, 2023
9308569
Fix some bugs, consolidate a bit:
pattisdr Aug 31, 2023
286d082
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Sep 5, 2023
00fc4ef
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Sep 6, 2023
08cb5c6
feat: pull out the data migration and get the pure upgrade working
ThomasLaPiana Sep 6, 2023
289c16c
fix: loguru-related test error
ThomasLaPiana Sep 6, 2023
325dd6e
fix: drop default uses and categories as part of the 2.0 upgrade
ThomasLaPiana Sep 6, 2023
87bc209
fix: drop default data_qualifiers and data_subjects as part of migration
adamsachs Sep 8, 2023
b6aecaa
Update config.json Improve -> Functional (#4076)
pattisdr Sep 13, 2023
3ada9cc
Merge branch 'main' into ThomasLaPiana-update-fideslang-200
ThomasLaPiana Sep 13, 2023
702f349
fix: migration order
ThomasLaPiana Sep 13, 2023
3d3a67e
Data Migration for Fideslang 2.0 (#4030)
ThomasLaPiana Sep 13, 2023
699074f
feat: bump fideslang version
ThomasLaPiana Sep 13, 2023
e7ecbb4
fix: seed.py updated to use user.authorization
ThomasLaPiana Sep 13, 2023
4a271f9
reference correct down_rev in migration comment
adamsachs Sep 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .fides/db_dataset.yml
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,18 @@ dataset:
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_added
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_deprecated
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: replaced_by
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: ctl_data_qualifiers
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fields:
Expand Down Expand Up @@ -449,6 +461,18 @@ dataset:
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_added
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_deprecated
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: replaced_by
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: ctl_data_subjects
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fields:
Expand Down Expand Up @@ -504,6 +528,18 @@ dataset:
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_added
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_deprecated
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: replaced_by
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: ctl_data_uses
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fields:
Expand Down Expand Up @@ -574,6 +610,18 @@ dataset:
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_added
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: version_deprecated
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: replaced_by
data_categories:
- system.operations
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
- name: ctl_datasets
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
fields:
Expand Down
2 changes: 1 addition & 1 deletion .fides/systems.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ system:
data_categories:
- system.operations
- user.contact
data_use: improve.system
data_use: functional.service.improve
data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
data_subjects:
- anonymous_user
Expand Down
37 changes: 23 additions & 14 deletions clients/admin-ui/cypress/e2e/systems.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -341,8 +341,8 @@ describe("System management page", () => {
});

// edit the existing declaration
cy.getByTestId("accordion-header-improve.system").click();
cy.getByTestId("improve.system-form").within(() => {
cy.getByTestId("accordion-header-functional.service.improve").click();
cy.getByTestId("functional.service.improve-form").within(() => {
cy.getByTestId("input-data_subjects").type(`customer{enter}`);
cy.getByTestId("save-btn").click();
});
Expand Down Expand Up @@ -459,12 +459,14 @@ describe("System management page", () => {
cy.getByTestId("more-btn").click();
cy.getByTestId("edit-btn").click();
});
// "improve.system" and "Store system data." are already being used
// "functional.service.improve" and "Store system data." are already being used
cy.getByTestId("tab-Data uses").click();
cy.getByTestId("add-btn").click();
cy.wait(["@getDataCategories", "@getDataSubjects", "@getDataUses"]);
cy.getByTestId("new-declaration-form").within(() => {
cy.getByTestId("input-data_use").type(`improve.system{enter}`);
cy.getByTestId("input-data_use").type(
`functional.service.improve{enter}`
);
cy.getByTestId("input-name").type(`Store system data.{enter}`);
cy.getByTestId("input-data_categories").type(`user.biometric{enter}`);
cy.getByTestId("input-data_subjects").type(`anonymous{enter}`);
Expand Down Expand Up @@ -494,12 +496,14 @@ describe("System management page", () => {
cy.getByTestId("add-btn").click();
cy.wait(["@getDataCategories", "@getDataSubjects", "@getDataUses"]);

cy.getByTestId(`accordion-header-improve.system`);
cy.getByTestId(`accordion-header-functional.service.improve`);
cy.getByTestId(`accordion-header-advertising`).click();

// try to change 'advertising' to 'improve.system' and make their names match
// try to change 'advertising' to 'functional.service.improve' and make their names match
cy.getByTestId("advertising-form").within(() => {
cy.getByTestId("input-data_use").type(`improve.system{enter}`);
cy.getByTestId("input-data_use").type(
`functional.service.improve{enter}`
);
cy.getByTestId("input-name").clear().type(`Store system data.{enter}`);
cy.getByTestId("save-btn").click();
});
Expand All @@ -512,13 +516,15 @@ describe("System management page", () => {
cy.getByTestId("more-btn").click();
cy.getByTestId("edit-btn").click();
});
// "improve.system" and "Store system data." are already being used
// use "improve.system" again but a different name
// "functional.service.improve" and "Store system data." are already being used
// use "functional.service.improve" again but a different name
cy.getByTestId("tab-Data uses").click();
cy.getByTestId("add-btn").click();
cy.wait(["@getDataCategories", "@getDataSubjects", "@getDataUses"]);
cy.getByTestId("new-declaration-form").within(() => {
cy.getByTestId("input-data_use").type(`improve.system{enter}`);
cy.getByTestId("input-data_use").type(
`functional.service.improve{enter}`
);
cy.getByTestId("input-name").type(`A different description.{enter}`);
cy.getByTestId("input-data_categories").type(`user.biometric{enter}`);
cy.getByTestId("input-data_subjects").type(`anonymous{enter}`);
Expand Down Expand Up @@ -583,7 +589,7 @@ describe("System management page", () => {

// Edit the existing data use
cy.getByTestId("privacy-declaration-accordion").within(() => {
cy.getByTestId("accordion-header-improve.system").click();
cy.getByTestId("accordion-header-functional.service.improve").click();
// Add a data subject
cy.getByTestId("input-data_subjects").type(`citizen{enter}`);
cy.getByTestId("save-btn").click();
Expand Down Expand Up @@ -634,15 +640,18 @@ describe("System management page", () => {
});

it.skip("deletes an accordion privacy declaration", () => {
cy.getByTestId("accordion-header-improve.system").click();
cy.getByTestId("improve.system-form").within(() => {
cy.getByTestId("accordion-header-functional.service.improve").click();
cy.getByTestId("functional.service.improve-form").within(() => {
cy.getByTestId("delete-btn").click();
});
cy.getByTestId("continue-btn").click();
cy.wait("@putFidesSystem").then((interception) => {
const { body } = interception.request;
expect(body.privacy_declarations.length).to.eql(1);
expect(body.privacy_declarations[0].data_use !== "improve.system");
expect(
body.privacy_declarations[0].data_use !==
"functional.service.improve"
);
});
cy.getByTestId("toast-success-msg").contains("Data use deleted");
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -495,7 +495,7 @@
{
"name": "ccn",
"description": null,
"data_categories": ["user.financial.account_number"],
"data_categories": ["user.financial.bank_account"],
"data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
"retention": null,
"fides_meta": null,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
"origin": null,
"regions": ["fr", "ie"],
"consent_mechanism": "opt_in",
"data_uses": ["improve.system"],
"data_uses": ["functional.service.improve"],
"enforcement_level": "system_wide",
"disabled": false,
"has_gpc_flag": false,
Expand Down
2 changes: 1 addition & 1 deletion clients/admin-ui/cypress/fixtures/systems/system.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
{
"name": "Analyze customer behaviour for improvements.",
"data_categories": ["user.contact", "user.device.cookie_id"],
"data_use": "improve.system",
"data_use": "functional.service.improve",
"data_subjects": ["customer"],
"dataset_references": ["demo_users_dataset"],
"cookies": [],
Expand Down
4 changes: 2 additions & 2 deletions clients/admin-ui/cypress/fixtures/systems/systems.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
{
"name": "Store system data.",
"data_categories": ["system.operations", "user.contact"],
"data_use": "improve.system",
"data_use": "functional.service.improve",
"data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
"data_subjects": ["anonymous_user"],
"dataset_references": ["public"],
Expand Down Expand Up @@ -56,7 +56,7 @@
{
"name": "Analyze customer behaviour for improvements.",
"data_categories": ["user.contact", "user.device.cookie_id"],
"data_use": "improve.system",
"data_use": "functional.service.improve",
"data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
"data_subjects": ["customer"],
"dataset_references": ["demo_users_dataset"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
{
"name": "Store system data.",
"data_categories": ["system.operations", "user.contact"],
"data_use": "improve.system",
"data_use": "functional.service.improve",
"data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
"data_subjects": ["anonymous_user"],
"dataset_references": ["public"],
Expand Down Expand Up @@ -68,7 +68,7 @@
{
"name": "Analyze customer behaviour for improvements.",
"data_categories": ["user.contact", "user.device.cookie_id"],
"data_use": "improve.system",
"data_use": "functional.service.improve",
"data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
"data_subjects": ["customer"],
"dataset_references": ["demo_users_dataset"],
Expand Down
30 changes: 15 additions & 15 deletions clients/admin-ui/cypress/fixtures/taxonomy/data_categories.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@
"active": true
},
{
"fides_key": "user.browsing_history",
"fides_key": "user.behavior.browsing_history",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Browsing History",
Expand Down Expand Up @@ -160,7 +160,7 @@
"active": true
},
{
"fides_key": "user.gender",
"fides_key": "user.demographic.gender",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Gender",
Expand All @@ -180,7 +180,7 @@
"active": true
},
{
"fides_key": "user.media_consumption",
"fides_key": "user.behavior.media_consumption",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Media Consumption Data",
Expand All @@ -190,7 +190,7 @@
"active": true
},
{
"fides_key": "user.non_specific_age",
"fides_key": "user.demographic.age_range",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Non-Specific Age",
Expand All @@ -200,7 +200,7 @@
"active": true
},
{
"fides_key": "user.observed",
"fides_key": "user.behavior",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Observed Data",
Expand All @@ -210,7 +210,7 @@
"active": true
},
{
"fides_key": "user.profiling",
"fides_key": "user.demographic.profile",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Profiling Data",
Expand All @@ -220,7 +220,7 @@
"active": true
},
{
"fides_key": "user.race",
"fides_key": "user.demographic.race_ethnicity",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Race",
Expand All @@ -230,7 +230,7 @@
"active": true
},
{
"fides_key": "user.religious_belief",
"fides_key": "user.demographic.religious_belief",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Religious Belief",
Expand All @@ -240,7 +240,7 @@
"active": true
},
{
"fides_key": "user.search_history",
"fides_key": "user.behavior.search_history",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Search History",
Expand All @@ -250,7 +250,7 @@
"active": true
},
{
"fides_key": "user.sexual_orientation",
"fides_key": "user.demographic.sexual_orientation",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Sexual Orientation",
Expand Down Expand Up @@ -300,7 +300,7 @@
"active": true
},
{
"fides_key": "user.organization",
"fides_key": "user.contact.organization",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Organization Identifiable Data",
Expand Down Expand Up @@ -440,7 +440,7 @@
"active": true
},
{
"fides_key": "user.date_of_birth",
"fides_key": "user.demographic.date_of_birth",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Date of Birth",
Expand All @@ -460,7 +460,7 @@
"active": true
},
{
"fides_key": "user.financial.account_number",
"fides_key": "user.financial.bank_account",
"organization_fides_key": "default_organization",
"tags": null,
"name": "User Financial Account Number",
Expand All @@ -470,7 +470,7 @@
"active": true
},
{
"fides_key": "user.genetic",
"fides_key": "user.health_and_medical.genetic",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Genetic Data",
Expand Down Expand Up @@ -550,7 +550,7 @@
"active": true
},
{
"fides_key": "user.political_opinion",
"fides_key": "user.demographic.political_opinion",
"organization_fides_key": "default_organization",
"tags": null,
"name": "Political Opinion",
Expand Down
2 changes: 1 addition & 1 deletion clients/admin-ui/cypress/fixtures/taxonomy/data_uses.json
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
"active": true
},
{
"fides_key": "improve.system",
"fides_key": "functional.service.improve",
"organization_fides_key": "default_organization",
"tags": null,
"name": "System",
Expand Down
2 changes: 1 addition & 1 deletion clients/fides-js/__tests__/lib/cookie.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ describe("makeConsentDefaultsLegacy", () => {
{
cookieKeys: ["default_true"],
default: true,
fidesDataUseKey: "improve.system",
fidesDataUseKey: "functional.service.improve",
},
{
cookieKeys: ["default_false"],
Expand Down
Loading
Loading