[Backend] Save and Retrieve Privacy Preferences by Fides User Device Id #3132

pattisdr · 2023-04-23T21:26:17Z

Closes #2838
❗ Contains migration; check downrev before merge
❗ Dependent on #3131

Description Of Changes

Adds two new endpoints to Save and Retrieve Privacy Preferences with respect to a fides_user_device_id that was in the browser instead of just a verified identity like an email or a phone number
Start saving current and historical privacy preferences with respect to an optional fides user device id
Allow current and historical preferences to be saved with respect to one or both of a verified provided identity and a fides user device provided identity id simultaneously
If preferences are saved with respect to a verified provided identity and a fides user device id simultaneously, consolidate the current preferences into single records
Expose the new fides user device id in the historical report
Enforce on the backend that a privacy preference must either be saved with respect to a verified provided identity or a fides user device id provided identity (or both!) but we require at least one.

Code Changes

Add new indexed FK column on the current record: CurrentPrivacyPreference.fides_user_device_provided_identity_id. A unique constraint added here so we maintain one "current" record for each "fides_user_provided_identity_id" and each "privacy_notice"
Add new indexed FK column on the historical record: PrivacyPreferenceHistory.fides_user_device_provided_identity_id
Add hashed field PrivacyPreferenceHistory.hashed_fides_user_device field so we can search historical records by this device id more easily
Cache the encrypted value PrivacyPreferenceHistory.fides_user_device directly on this field so we can surface for reporting purposes without having to join in the ProvidedIdentity table.
Add new method get_or_create_fides_user_device_id_provided_identity to get and create provided identity records for device ides and keeping ProvidedIdentities for a fides user device id unique when saved for the purposes of consent. We separately create these records for privacy requests, and those are not expected to be unique.
Update existing endpoint for saving privacy preferences in the verified identity flow to allow a fides user device id to optionally be saved alongside these preferences
Pull out a new method _save_privacy_preferences_for_identities to be shared between the existing endpoint that saves privacy preferences with respect to a verified identity and the new endpoint that only saves preferences with respect to a fides user device id.
Add the two new endpoints: PATCH /privacy-preferences and GET /privacy-preferences?fides_user_device_id=XXXXXX

Steps to Confirm

In postman, create a client with permission to save privacy notices.
Use the Privacy Notices > Create Privacy Notices endpoint to create a starting set of notices. Note one of the "privacy_notice_history_id"s that are created
Under the Privacy Preferences > Save Privacy Preferences for Device Id collection, save a preference with respect to a privacy_notice_history_id you created above. Verify response matches what you'd expect
Get saved Privacy Preferences with respect to device id under the Privacy Preferences > Privacy Preferences by Fides User Device Id collection
Access the historical report via the PrivacyPReferences > Get Historical Preferences collection and verify that fides user device id shows up under "fides_user_device_id"

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

cypress · 2023-04-23T21:48:35Z

Passing run #1555 ↗︎

0	3	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `e69e85e` into `01ea9b4`...
Project: fides	Commit: `51865cc4d9 ℹ️`
Status: Passed	Duration: 00:36 💡
Started: Apr 25, 2023 3:49 AM	Ended: Apr 25, 2023 3:49 AM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

codecov · 2023-04-23T22:43:24Z

Codecov Report

Patch coverage: 98.91% and project coverage change: +0.03 🎉

Comparison is base (01ea9b4) 87.48% compared to head (e69e85e) 87.51%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3132      +/-   ##
==========================================
+ Coverage   87.48%   87.51%   +0.03%     
==========================================
  Files         309      309              
  Lines       17924    17993      +69     
  Branches     2325     2337      +12     
==========================================
+ Hits        15680    15747      +67     
- Misses       1820     1821       +1     
- Partials      424      425       +1

Impacted Files	Coverage Δ
...s/api/v1/endpoints/privacy_preference_endpoints.py	`99.21% <97.77%> (-0.79%)`	⬇️
src/fides/api/ops/api/v1/urn_registry.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/models/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/schemas/privacy_preference.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/schemas/redis_cache.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/util/consent_util.py	`100.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

pattisdr · 2023-04-24T00:55:41Z

@seanpreston requesting your review since I'm saving privacy preferences here with respect to the new style PrivacyPreferenceHistory and CurrentPrivacyPreference records and you are working on saving this with respect to the old-style Consent records. This is built off of your ProvidedIdentity migration, and I also add a method or two that might be useful for your work.

@eastandwestwind I was hoping to get your review on this as well as it's the start of the backend for being able to save and retrieve privacy preferences from things like overlays.

…vice id provided identity, not just a verified identity like an email or phone number. - Start tracking an optional fides_user_device_provided_identity for PrivacyPreferenceHistory and CurrentPrivacyPreference records. - At least one provided identity must be supplied, but you can also supply both. - If both a verified identity (like email or phone number) and a fides user device id are supplied, consolidate any "current preferences" saved with respect to these records into the same records. - For the "verified identity flow" endpoint that will be called through the privacy center, also allow a fides user device id to be saved, in addition to that verified identity with respect to the privacy preferences. - Store encrypted versions of the fides user device id so we can surface in a report, as well as a hash of the record for filtering. Also have a FK to the provided identity table so we can enforce uniqueness for current privacy preferences. - Add some more code comments to clarify field purposes on the historical record

…s user device id alone, outside of the "verified identity" workflow. This endpoint might be called from an overlay or consent banner. Extract the existing code for saving preferences for a verified identity to be shared with saving preferences for a fides user device id.

…fides user device id

- Fix some other tests to require new backend validation that require preferences must be saved with respect to a specific provided identity id - Add new db annotations - Update the changelog - Update the postman collection

… provided identity.

eastandwestwind

Great work on this @pattisdr ! I've gone through your testing steps and confirmed that everything worked as expected for me.

I can send everything expected in PATCH {{host}}/privacy-preferences with no issues, from FE for now. We can definitely refactor some into BE later.

src/fides/api/ops/models/privacy_preference.py

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

eastandwestwind · 2023-04-24T23:31:49Z

One more question as I'm looking through docs- will we need an endpoint to retrieve privacy preference by user id (email / phone)? This can be a follow-up of course depending on need, specifically in privacy center, when an identity is known, but it was a first time visitor.

pattisdr · 2023-04-25T00:31:02Z

One more question as I'm looking through docs- will we need an endpoint to retrieve privacy preference by user id (email / phone)? This can be a follow-up of course depending on need, specifically in privacy center, when an identity is known, but it was a first time visitor.

@eastandwestwind I am not sure I understand your question.

I do have this separate endpoint for retrieving privacy preferences by an email or phone number that's very similar to the request we make in the privacy center now except this returns the new-style privacy preferences. This endpoint requires you follow that flow to verify your identity.

eastandwestwind · 2023-04-25T00:39:10Z

@pattisdr this was what I was looking for, thanks!

- Move new get_or_create_fides_user_device_id_provided_identity and get_fides_user_device_id_provided_identity into a consent util. - Update copy paste error in get privacy preferences docstring.

pattisdr mentioned this pull request Apr 23, 2023

Refactor New Privacy Preference Endpoints to Examine Request Headers #3133

Closed

pattisdr changed the title ~~[WIP] Save and Retrieve Privacy Preferences by Fides User Device Id~~ Save and Retrieve Privacy Preferences by Fides User Device Id Apr 23, 2023

pattisdr marked this pull request as ready for review April 23, 2023 23:13

pattisdr commented Apr 24, 2023

View reviewed changes

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py Show resolved Hide resolved

pattisdr requested review from seanpreston and eastandwestwind April 24, 2023 00:53

Base automatically changed from sp/807/fides-device-id to main April 24, 2023 13:33

pattisdr added 6 commits April 24, 2023 09:06

Add an endpoint to get current privacy preferences with respect to a …

bc78d77

…fides user device id

Expose the fides user device id in the historical report.

11a741e

- Fix some other tests to require new backend validation that require preferences must be saved with respect to a specific provided identity id - Add new db annotations - Update the changelog - Update the postman collection

Formatting

b3c8366

Improve test coverage on the method that fetches fides user device id…

b82b6ea

… provided identity.

pattisdr force-pushed the fides_2838_user_preferences_by_device_id branch from 7e5f060 to b82b6ea Compare April 24, 2023 14:12

pattisdr changed the title ~~Save and Retrieve Privacy Preferences by Fides User Device Id~~ [Backend] Save and Retrieve Privacy Preferences by Fides User Device Id Apr 24, 2023

eastandwestwind approved these changes Apr 24, 2023

View reviewed changes

src/fides/api/ops/models/privacy_preference.py Show resolved Hide resolved

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py Outdated Show resolved Hide resolved

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py Outdated Show resolved Hide resolved

eastandwestwind reviewed Apr 24, 2023

View reviewed changes

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py Outdated Show resolved Hide resolved

pattisdr added 2 commits April 24, 2023 21:52

Merge branch 'main' into fides_2838_user_preferences_by_device_id

f14098a

Bump downrev.

e69e85e

- Move new get_or_create_fides_user_device_id_provided_identity and get_fides_user_device_id_provided_identity into a consent util. - Update copy paste error in get privacy preferences docstring.

pattisdr merged commit 61cbd96 into main Apr 25, 2023

pattisdr deleted the fides_2838_user_preferences_by_device_id branch April 25, 2023 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backend] Save and Retrieve Privacy Preferences by Fides User Device Id #3132

[Backend] Save and Retrieve Privacy Preferences by Fides User Device Id #3132

pattisdr commented Apr 23, 2023 •

edited

Loading

cypress bot commented Apr 23, 2023 •

edited

Loading

codecov bot commented Apr 23, 2023 •

edited

Loading

pattisdr commented Apr 24, 2023

eastandwestwind left a comment

eastandwestwind commented Apr 24, 2023

pattisdr commented Apr 25, 2023

eastandwestwind commented Apr 25, 2023

[Backend] Save and Retrieve Privacy Preferences by Fides User Device Id #3132

[Backend] Save and Retrieve Privacy Preferences by Fides User Device Id #3132

Conversation

pattisdr commented Apr 23, 2023 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

cypress bot commented Apr 23, 2023 • edited Loading

Passing run #1555 ↗︎

codecov bot commented Apr 23, 2023 • edited Loading

Codecov Report

pattisdr commented Apr 24, 2023

eastandwestwind left a comment

Choose a reason for hiding this comment

eastandwestwind commented Apr 24, 2023

pattisdr commented Apr 25, 2023

eastandwestwind commented Apr 25, 2023

pattisdr commented Apr 23, 2023 •

edited

Loading

cypress bot commented Apr 23, 2023 •

edited

Loading

codecov bot commented Apr 23, 2023 •

edited

Loading