Executing Consent Requests via the Privacy Request Execution Layer [#2146] #2125

pattisdr · 2023-01-04T05:59:07Z

Closes #2146
👉 Note: Requesting to merge into consent-propagation feature branch, not main
✅ Note: Half of this PR are new tests, actual code change here is about ~500 lines.

Description Of Changes

This PR adds the foundation to execute consent requests via the privacy request execution framework. When consent preferences are changed, a PrivacyRequest is created (with a Policy > Consent Rule attached) to propagate those consent preferences to third party services, server-side.

ConsentRequest and Consent records are still created per usual.

POC Changes

Updates config.json to reflect new consent defaults: Adds default_consent_policy as the policy_key for consent requests, and also adds executable=True to all the default consentOptions
Seeds the db with a default consent Policy and a default Consent Rule (action_type=consent) on startup
Adds a DB schema migration that adds an optional PrivacyRequest.executable_consent_preferences column where we store the user's desired consent preferences that are executable (useful for record keeping/retries, etc, storing the current state at the time the request was submitted). Also add an optional consentrequest.privacy_request_id FK to link whether a PrivacyRequest was queued to execute consent preferences
PATCH {{host}}/consent-request/{{consent_request_id}}/preferences now loads the config.json file as a last step to see which preferences are executable and filters out consent preferences that are not. It passes on those executable consent preferences to create and queue a PrivacyRequest to propagate executable consent preferences to third-party systems.
Update frequently-used method _get_consent_request_and_provided_identity to return a tuple containing both the ConsentRequest and ProvidedIdentity since there are cases where we also want the original ConsentRequest
Add new method DatasetConfig.get_dataset_with_stubbed_collection that creates a src.fides.api.ops.graph.config.Dataset from a DatasetConfig. It ignores any collections defined and just creates a single Collection representing the Dataset as a whole. This will be used to build the Consent Graph where we just want one node per Dataset instead of one node per Collection.
Request execution has multiple steps. We add a consent step after access and erasure, and only run consent if the attached Policy has a consent rule. We're also more careful about only running access and erasure steps if we have access and erasure rules. I'm also limiting the graph so nodes must be of SaasConnector-type, but this is subject to change as we further refine how the graph is going to be built.
Execution layer adds multiple consent-related methods to parallel existing methods for access and erasure. Namely, run_consent_request builds a very simple graph, where each Dataset has one node representing it. None of the nodes are dependent upon each other. Every node just takes in identity_data and every node outputs whether the consent request on that node was successful.
Saas connector mocks making a consent request but it only creates logs for now.

Steps to Confirm

2023-01-10 01:21:47.708 [INFO] (saas_connector:run_consent_request:421): Mocking consent request - actual logic should go here for saas connectors
2023-01-10 01:21:47.709 [INFO] (saas_connector:run_consent_request:426): Demo only! Testing available consent params! identity_data: {'email': 'dawn@ethyca.com'}, executable_preferences: [{'use': 'advertising', 'opt_in': False}, {'use': 'advertising.first_party', 'opt_in': True}, {'use': 'improve', 'opt_in': False}]

Update the config.json in the privacy center to set some of the executable options to False. And follow the steps above. When updating consent preferences, note the log:

2023-01-10 01:24:30.649 [INFO] (consent_request_endpoints:load_executable_consent_options:246): Consent option: 'advertising' is not executable.

The queued privacy request also doesn't have the advertising preference saved.

Screenshots

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md

Illustrating that we can execute consent requests as privacy requests and share the same execution layer.

src/fides/api/ctl/database/seed.py

src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

pattisdr · 2023-01-04T06:38:52Z

src/fides/api/ops/task/graph_task.py

+        output: bool = self.connector.run_consent_request(
+            self.traversal_node, self.resources.policy, self.resources.request, identity
+        )
+        self.log_end(ActionType.consent)


The parallel GraphTask.erasure_request (and access_request) also caches that the erasure was completed on this node so if we restart from failure, we don't re-attempt this node. This is used to update the graph on retry.

self.resources.cache_erasure( f"{self.key}", output )

There's a lot of good reasons to not try to re-run an erasure request on a collection if it's already been completed, but I assume it is okay to retry a request for consent.

We can revisit if needed later.

src/fides/api/ops/models/policy.py

src/fides/api/ops/models/privacy_request.py

src/fides/api/ctl/database/seed.py

src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

src/fides/api/ops/service/connectors/saas_connector.py

src/fides/api/ops/service/privacy_request/request_runner_service.py

src/fides/api/ops/service/connectors/saas_connector.py

# Conflicts: # src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

… this at the GraphTask level instead of the individual connector level.

…y" to be the data use.

…s where there is at most one node per Dataset. This is dependent on knowing how consent request details would be stored on saas configs. I am assuming they are stored at the dataset level somewhere, not the collection level. - Instead of passing in a DatasetGraph with all the Datasets and all the Collections to run_consent_request, instead pass in a Datasets with a single stubbed Collection with no links to other collections. - Update `run_consent_request` to not use traverse.traverse. Instead, built a custom dictionary by hand that just has one node for every dataset, mapped to a tuple of the function to run (GraphTask.consent_request and the identity data). The nodes have no dependencies. Each node just takes in identity data. There are more details to be worked out here.

pattisdr · 2023-01-04T18:03:10Z

Note: commit 4fc9ec7 experiments with how to create a simpler sort of graph for ConsentRequests. It creates a graph with at most one stubbed collection per dataset type, as I imagine consent requests will largely be requests against the larger service instead of to multiple specific collections.

I may be wrong on these assumptions so the easiest way forward here is to understand where we're going to define how to talk to a third party service. Once I know how/where that's defined, then I can build a graph with that information. The larger point here is that we're flexible on how we build the graph for consent requests. (It would be useful if we can still get them into the Dataset > Collection structure)

If you want to play around with a graph that has all the nodes instead of one node per dataset, try the previous commit.

seanpreston · 2023-01-05T16:45:30Z

Some initial comments:

Note postgres and mongo collections have skipped logs

I'm not sure where I stand with this — if the collection is skipped on one hand it's good to show that it was visited, but in a pure consent request these aren't relevant to the overall traversal and so if the user is expecting them to be skipped entirely, it could be more confusing to be showing them. We could always investigate display options to mitigate, e.g. greying them out or highlighting they were skipped more obviously from the view somehow.

PATCH {{host}}/consent-request/{{consent_request_id}}/preferences now creates and queues a separate PrivacyRequest to execute those consent preferences

This will need to happen for specific consent prefs when a flag is set in the post data.

where we're going to define how to talk to a third party service

I think it's safe to say this will be inside the a saas connector yaml, whether that's consent only for one connector, or interlinked with an existing saas connector for now.

src/fides/api/ops/models/datasetconfig.py

pattisdr · 2023-01-08T19:41:24Z

I'm not sure where I stand with this — if the collection is skipped on one hand it's good to show that it was visited, but in a pure consent request these aren't relevant to the overall traversal and so if the user is expecting them to be skipped entirely, it could be more confusing to be showing them.

@seanpreston totally agree, it is not a problem to make the graph more targeted. Adam and I have been talking about different ways to do this, what can immediately be eliminated from the graph.

EDIT: newer commits are only adding DatasetConfigs that are of saas type, but this is subject to change. we have lots of flexibility here.

pattisdr · 2023-01-08T19:43:33Z

PATCH {{host}}/consent-request/{{consent_request_id}}/preferences now creates and queues a separate PrivacyRequest to execute those consent preferences

This will need to happen for specific consent prefs when a flag is set in the post data.

@seanpreston currently this is checked at the GraphTask level - when we go to execute the task, we filter out which preferences are not executable.

EDIT: Latest commit is filtering out consent preferences that are not executable before the the privacy request is queued only being given executable preferences

…commit.

…t creation. Remove test that consent actions are not supported.

…e executable. - Instead, store in the config.json file. - Load whether consent options are executable directly from config.json file in the backend - Additionally, only persist the consent preferences that are executable to the PrivacyRequest. - Rename privacyrequest.consent_preferences to privacyrequest.executable_consent_preferences - Allow an optional Policy key to be passed in when setting consent preferences. If no policy key is passed in, we use the default consent policy.

…ent preferences into its own method as I've made "set_consent_preferences" very long.

…that are executable.

- Adjust how I'm creating a graph with one node to return a Dataset with one stubbed Collection regardless of whether Collections exist on the current node. - Only add saas datasets to the graph for now, but this is subject to change.

…ivacy request execution framework as a means to propagate consent preferences. Subject to change as certain details are tweaked. - Break some code out into individual functions that are more self-contained and more easily tested. - Add dataset annotations for new db fields. - Break out consent model tests into their own file, separate from privacy request model tests.

pattisdr · 2023-01-10T00:57:25Z

.fides/db_dataset.yml

+    - name: executable_consent_preferences
+      data_categories: [ system.operations ]
+      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified


Here and above we're just adding new annotations for our fides dataset

pattisdr · 2023-01-10T00:58:14Z

clients/privacy-center/config/config.json

        "url": "https://example.com/privacy#analytics",
        "default": true,
        "highlight": false,
-        "cookieKeys": []
+        "cookieKeys": [],
+        "executable": true


Going ahead and adding executable:true here for all the consentOptions. The backend loads this file directly to have one source of truth.

agreed offline that we should be setting executable:false as a default

agreed offline that in initial implementation, we'll only really support a single data use being executable:true and its preference that will drive whether an opt_in or opt_out request is executed for all connectors with consent requests enabled

agreed offline that the backend will not load this file, and instead the front-end (i.e. privacy center) will fully determine which data uses are executable

thanks for summarizing Adam 👍

pattisdr · 2023-01-10T00:58:58Z

clients/privacy-center/config/config.json

@@ -37,6 +37,7 @@
      "email": "required",
      "phone": "optional"
    },
+    "policy_key": "default_consent_policy",


This is not yet being used and should be passed to set_consent_preferences in #2122

pattisdr · 2023-01-10T01:03:04Z

src/fides/api/ctl/database/seed.py

+
+        log.info("Creating: Default Consent Policy")
+        consent_policy = Policy.create_or_update(
+            db=db_session,
+            data={
+                "name": "Default Consent Policy",
+                "key": DEFAULT_CONSENT_POLICY,
+                "execution_timeframe": 45,
+                "client_id": client_id,
+            },
+        )
+
+        log.info("Creating: Default Consent Rule")
+        Rule.create_or_update(
+            db=db_session,
+            data={
+                "action_type": ActionType.consent.value,
+                "name": "Default Consent Rule",
+                "key": DEFAULT_CONSENT_RULE,
+                "policy_id": consent_policy.id,
+                "client_id": client_id,
+            },
+        )
+


Here I just seed the database with a new default Policy with a Consent Rule attached. The primary purpose of this resource is to let the privacy request execution workflow know that this is a consent request! No RuleTargets are required for ConsentRules. Relevant configuration is pulled from the config.json file.

pattisdr · 2023-01-10T01:04:06Z

src/fides/api/ctl/migrations/versions/de456534dbda_add_privacyrequest_consent_preferences_.py

+    op.add_column(
+        "privacyrequest",
+        sa.Column(
+            "executable_consent_preferences",


This optional list of consent preferences persists to the PrivacyRequest what consent options we're updating, and in particular, just the executable ones. That way the execution workflow doesn't have to worry about what is and isn't executable, it can use the whole list.

pattisdr · 2023-01-10T01:10:13Z

src/fides/api/ops/models/datasetconfig.py

+    def get_dataset_with_stubbed_collection(self) -> Dataset:
+        """
+        Return a Dataset with a single mock Collection for use in building a graph
+        where we only want one node per dataset, instead of one node per collection.  Note that
+        the expectation is that there would be no dependencies between nodes on the eventual graph, and the graph
+        doesn't require information stored at the collection-level.
+
+        The single Collection will be the resource that gets practically added to the graph, but the intent
+        is that this single node represents the overall Dataset, and will execute Dataset-level requests,
+        not Collection-level requests.
+        """
+        dataset_graph: Dataset = self.get_graph()
+        stubbed_collection = Collection(name=dataset_graph.name, fields=[], after=set())
+
+        dataset_graph.collections = [stubbed_collection]


For now, I am creating a Dataset with a single stubbed Collection, assuming all the information to create the request will exist at the Dataset level, and I don't need information on Collections.

This is what allows us to later build a graph with one node per dataset.

pattisdr · 2023-01-10T01:11:25Z

src/fides/api/ops/models/policy.py

-    if action_type in [ActionType.consent.value, ActionType.update.value]:
+    if action_type in [ActionType.update.value]:


We already technically had consent rules, we're just allowing ourselves to use them!

pattisdr · 2023-01-10T01:13:13Z

src/fides/api/ops/service/connectors/saas_connector.py

+    def run_consent_request(
+        self,
+        node: TraversalNode,
+        policy: Policy,
+        privacy_request: PrivacyRequest,
+        identity_data: Dict[str, Any],
+        executable_preferences: List[Consent],
+    ) -> bool:
+        """Execute a consent request. Return whether the consent request to the third party succeeded.
+
+        Executable_preferences have already been filtered to just consent preferences the customer has deemed executable.
+
+        Return True if 200 OK
+        """
+        logger.info(
+            "Mocking consent request - actual logic should go here for saas connectors",
+            node.address.value,
+        )
+
+        logger.info(
+            "Demo only! Testing available consent params! identity_data: {}, executable_preferences: {}",
+            identity_data,
+            [
+                {"use": preference.data_use, "opt_in": preference.opt_in}
+                for preference in executable_preferences
+            ],
+        )
+        return True
+


Currently I am adding a run_consent_request method to the SaasConnector. This is where we'd make requests to third-party endpoints to propagate consent. Extra logging is added here to demonstrate what is available but should be removed before release.

pattisdr · 2023-01-10T01:13:47Z

src/fides/api/ops/service/privacy_request/request_runner_service.py

+        if policy.get_rules_for_action(
+            action_type=ActionType.consent
+        ) and can_run_checkpoint(
+            request_checkpoint=CurrentStep.consent,
+            from_checkpoint=resume_step,
+        ):
+
+            await run_consent_request(
+                privacy_request=privacy_request,
+                policy=policy,
+                graph=build_consent_dataset_graph(datasets),
+                connection_configs=connection_configs,
+                identity=identity_data,
+                session=session,
+            )
+


Adding an additional step to the existing request runner, similar to how we have steps above for access and erasure.

pattisdr · 2023-01-10T01:33:38Z

src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

+    identity = Identity()
+    setattr(
+        identity,
+        provided_identity.field_name.value,  # type:ignore[attr-defined]
+        provided_identity.encrypted_value["value"],  # type:ignore[index]
+    )  # Pull the information on the ProvidedIdentity for the ConsentRequest to pass along to create a PrivacyRequest


Note that a separate ProvidedIdentity is created for the PrivacyRequest for now. So there's one for the ConsentRequest too but I think this is okay to start.

…ilures are properly caught cause the privacy request status to update.

src/fides/api/ops/models/policy.py

src/fides/api/ops/service/privacy_request/request_runner_service.py

adamsachs · 2023-01-10T16:30:24Z

i left some comments inline that mainly summarize what's been discussed offline as next steps. generally, though, this looks really good to me! i think once those follow up points have been addressed, i'll re-review, but i expect this should be just about ready to go.

really nice work finding a relatively quick/simple framework integration here that gives us a solid foundation for future consent propagation support.

…preferences to be more generic as things are evolving here. - Adjust rule validation logic for when trying to create a new consent rule and adding to a Policy that already has a consent rule.

pattisdr · 2023-01-10T17:47:46Z

Discussed with Adam, we're going to go ahead and merge this, as it is the foundation for the consent propagation work. (merging into consent-propagation, not main.) Further changes needed are reticketed.

Quick POC -

979dc25

Illustrating that we can execute consent requests as privacy requests and share the same execution layer.

pattisdr added the do not merge Please don't merge yet, bad things will happen if you do label Jan 4, 2023

Adjust seed logging.

d63c01a

pattisdr commented Jan 4, 2023

View reviewed changes

src/fides/api/ops/service/privacy_request/request_runner_service.py Outdated Show resolved Hide resolved

pattisdr commented Jan 4, 2023

View reviewed changes

src/fides/api/ops/service/connectors/saas_connector.py Outdated Show resolved Hide resolved

pattisdr added 4 commits January 4, 2023 09:16

Merge branch 'main' into consent-as-privacy-request-poc

d894ce6

# Conflicts: # src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

Pass in executable_preferences to Connector.run_consent_request doing…

cd138a1

… this at the GraphTask level instead of the individual connector level.

Get rid of redundant columns on the RuleUse table. We can use the "ke…

423deb5

…y" to be the data use.

pattisdr changed the title ~~[POC][Draft]Executing Consent Requests via the Execution Layer~~ [POC][Draft]Executing Consent Requests via the Execution Layer [#2099] Jan 4, 2023

seanpreston reviewed Jan 5, 2023

View reviewed changes

src/fides/api/ops/models/datasetconfig.py Show resolved Hide resolved

seanpreston mentioned this pull request Jan 5, 2023

Generate + make third-party consent API requests #2144

Closed

pattisdr added 7 commits January 8, 2023 15:38

Merge branch 'main' into consent-as-privacy-request-poc

2b2d737

Bump downrev after main merge and get lint checks passing on current …

f685a10

…commit.

Fix bug on whether we're adding consent_preferences to privacy reques…

0f24e95

…t creation. Remove test that consent actions are not supported.

Separate out the logic to queue the privacy request to propagate cons…

2881a36

…ent preferences into its own method as I've made "set_consent_preferences" very long.

Only create a privacy request if there are saved consent preferences …

64ae2f6

…that are executable.

Make some adjustments to the consent graph.

332351c

- Adjust how I'm creating a graph with one node to return a Dataset with one stubbed Collection regardless of whether Collections exist on the current node. - Only add saas datasets to the graph for now, but this is subject to change.

pattisdr changed the base branch from main to consent-propagation January 9, 2023 19:14

pattisdr removed the do not merge Please don't merge yet, bad things will happen if you do label Jan 9, 2023

pattisdr added the run unsafe ci checks Runs fideops-related CI checks that require sensitive credentials label Jan 10, 2023

pattisdr marked this pull request as ready for review January 10, 2023 01:17

pattisdr changed the title ~~[POC][Draft]Executing Consent Requests via the Execution Layer [#2099]~~ Executing Consent Requests via the Privacy Request Execution Layer [#2099][#2146] Jan 10, 2023

pattisdr commented Jan 10, 2023

View reviewed changes

pattisdr linked an issue Jan 10, 2023 that may be closed by this pull request

Foundation for Propagating Consent Requests via the Privacy Request workflow (POC cleanup) #2146

Closed

pattisdr changed the title ~~Executing Consent Requests via the Privacy Request Execution Layer [#2099][#2146]~~ Executing Consent Requests via the Privacy Request Execution Layer [#2146] Jan 10, 2023

pattisdr self-assigned this Jan 10, 2023

pattisdr requested review from adamsachs, allisonking and NevilleS January 10, 2023 01:39

Move where we run the consent graph so it is in the try/except and fa…

6137d72

…ilures are properly caught cause the privacy request status to update.

pattisdr removed the run unsafe ci checks Runs fideops-related CI checks that require sensitive credentials label Jan 10, 2023

Fix mandrill misspelling.

34f5eeb

adamsachs reviewed Jan 10, 2023

View reviewed changes

src/fides/api/ops/models/policy.py Outdated Show resolved Hide resolved

adamsachs reviewed Jan 10, 2023

View reviewed changes

src/fides/api/ops/service/privacy_request/request_runner_service.py Show resolved Hide resolved

pattisdr mentioned this pull request Jan 10, 2023

Configure the Privacy Center with which (if any) consent type should be “executable” using connectors #2122

Closed

Rename privacyrequest.executable_consent_preferences back to consent_…

1bfb101

…preferences to be more generic as things are evolving here. - Adjust rule validation logic for when trying to create a new consent rule and adding to a Policy that already has a consent rule.

pattisdr merged commit fd0c6cf into consent-propagation Jan 10, 2023

pattisdr deleted the consent-as-privacy-request-poc branch January 10, 2023 17:58

pattisdr mentioned this pull request Jan 17, 2023

Consent Propagation Feature #2255

Merged

14 tasks

pattisdr mentioned this pull request Mar 31, 2023

DynamoDB Connector - Consent Policy Spike #2938

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Executing Consent Requests via the Privacy Request Execution Layer [#2146] #2125

Executing Consent Requests via the Privacy Request Execution Layer [#2146] #2125

pattisdr commented Jan 4, 2023 •

edited

Loading

pattisdr Jan 4, 2023 •

edited

Loading

pattisdr commented Jan 4, 2023 •

edited

Loading

seanpreston commented Jan 5, 2023 •

edited

Loading

pattisdr commented Jan 8, 2023 •

edited

Loading

pattisdr commented Jan 8, 2023 •

edited

Loading

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

adamsachs Jan 10, 2023 •

edited

Loading

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023

pattisdr Jan 10, 2023 •

edited

Loading

adamsachs commented Jan 10, 2023

pattisdr commented Jan 10, 2023

		if action_type in [ActionType.consent.value, ActionType.update.value]:
		if action_type in [ActionType.update.value]:

Executing Consent Requests via the Privacy Request Execution Layer [#2146] #2125

Executing Consent Requests via the Privacy Request Execution Layer [#2146] #2125

Conversation

pattisdr commented Jan 4, 2023 • edited Loading

Description Of Changes

POC Changes

Steps to Confirm

Screenshots

Pre-Merge Checklist

pattisdr Jan 4, 2023 • edited Loading

Choose a reason for hiding this comment

pattisdr commented Jan 4, 2023 • edited Loading

seanpreston commented Jan 5, 2023 • edited Loading

pattisdr commented Jan 8, 2023 • edited Loading

pattisdr commented Jan 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamsachs Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr Jan 10, 2023 • edited Loading

Choose a reason for hiding this comment

adamsachs commented Jan 10, 2023

pattisdr commented Jan 10, 2023

pattisdr commented Jan 4, 2023 •

edited

Loading

pattisdr Jan 4, 2023 •

edited

Loading

pattisdr commented Jan 4, 2023 •

edited

Loading

seanpreston commented Jan 5, 2023 •

edited

Loading

pattisdr commented Jan 8, 2023 •

edited

Loading

pattisdr commented Jan 8, 2023 •

edited

Loading

adamsachs Jan 10, 2023 •

edited

Loading

pattisdr Jan 10, 2023 •

edited

Loading