Sovrn Email Consent Connector

[pattisdr]

Closes #2394
❗ UI doesn't yet have handling for this new connector type; do not merge
❗ Contains migration; check if downrev needs to be bumped before merge

Code Changes

Database

• Added a new sovrn connection type enum
• Added a new Privacy Request Status of awaiting_consent_email_send
• Added a new PrivacyRequest.awaiting_consent_email_send_at field
• Added a new providedidentitytype: sovrn

Email types

• Added a new CONSENT_REQUEST_EMAIL_FULFILLMENT template and associated schemas to describe what needs to be supplied to populate that email

Connector (Connectors contain the logic to reach out to other services)

• Added a generic GenericEmailConsentConnector that could be used for any consent email type connector. (This is not yet exposed as an option).
• Created a SovrnConsentConnector that uses the GenericEmailConsentConnector. These are new LimitedConnector types that don't interact directly with datasets, and just need to define how to make test connections.

ConnectionConfig (ConnectionConfigs describe the details needed to configure a service)

• Created a new sovrn connectionconfig that needs test_email_address, and advanced_settings supplied.

Scheduler

• Added a new scheduled task: initiate_scheduled_batch_consent_email_send that runs weekly on Mondays 12:00 PM ET to see if there are any consent emails that need to be sent and batches all the data accumulated in the last week into a single email per connector.

Connection Type

• Added a new "email" connection type - we currently have db, saas, and manual

Steps to Confirm

• Mark a consent preference as executable in your privacy center:
  • Update your test env config fides/data/test_env/privacy_center_config/config.json so advertising.first_party is True
• Temporarily edit the scheduler to send consent emails every minute instead of once a week
  • Go to the consent email batch service fides/api/ops/service/privacy_request/consent_email_batch_service.py::initiate_scheduled_batch_consent_email_send and replace existing job with:

        scheduler.add_job(
             func=send_consent_email_batch,
             kwargs={},
             id=BATCH_CONSENT_EMAIL_SEND,
            coalesce=False,
            replace_existing=True,
            trigger="cron",
            minute='*/1',
            hour='*'
      )

• Add a ljt_readerID cookie to the privacy center with document.cookie = "ljt_readerID=fides_test_reader_id;";
• Bring up the test env nox -s "fides_env(dev)" (make sure your privacy center changes take too. You may need to do nox -s "build(privacy-center)" and docker-compose up -d --no-deps fides-pc if not.
• Set up mailgun via the API (you'll need credentials)
  • POST {{host}}/messaging/config/

      {
        "key": "{{mailgun_config_key}}",
       "name": "mailgun",
       "service_type": "mailgun",
       "details": {
       "domain": "{{mailgun_domain}}"
       }
   }

  • PUT {{host}}/messaging/config/{{mailgun_config_key}}/secret

       {
           "mailgun_api_key": "{{mailgun_api_key}}"
       }

• Add a sovrn consent email connector via the API. Set up both the recipient email address and test email address as YOUR EMAIL ADDRESS, not a Sovrn email address. This connector sends live emails to third party services.
• Verify you've received the test email that was sent after you saved the secrets:
  
• Verify email send task is starting in terminal (but no emails are getting sent)

2023-02-09 15:38:00.067 [INFO] (consent_email_batch_service:send_consent_email_batch:172): Skipping batch consent email send with status: no_applicable_privacy_requests```

• Go to the privacy center http://localhost:3001/ to manage your consent preferences and opt in or out of email marketing
• Approve the privacy request and verify it briefly goes into a state of "awaiting consent email send"

• Email for the privacy request should be sent shortly
  
• Verify privacy request is put in a completed state afterwards
• Verify audit logs for the email send were added

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

Description Of Changes

Largely this adds a new Consent Email Connector (Sovrn) with plenty of foundation that can be reused for generic or vendor-specific email connectors in the future. Instead of sending the emails immediately, relevant data is batched together and sent in a single email to the vendor weekly.

• Consent Privacy Request Execution can exit early with an awaiting consent email send state if certain conditions are met:
  • Privacy Request must be a consent type
  • Privacy request must have consent preferences saved
  • There must be at least one consent email type connector configured
  • We must have collected at least one relevant user identity type for one of those connectors.
• There's a new scheduled task initiate_scheduled_batch_consent_email_send that handles the weekly email send with APScheduler
  • Once weekly, this task starts and looks for privacy requests in awaiting consent email send status. It then batches all the relevant user data together across all these privacy requests for each type of email consent connector.
  • Privacy requests are skipped if they don't have consent preferences saved or the correct user identities.
  • One email is sent per connector
• After the email send, we add audit logs to each privacy request for each email sent if applicable
• We put each privacy request in a temporary paused state
• We requeue each privacy request from the post-webhooks step to complete further processing (which runs any post webhooks, and sends relevant user notifications, and then marks the privacy request as complete).

[codecov]

Codecov Report

Base: 85.95% // Head: 86.26% // Increases project coverage by +0.30% 🎉

Coverage data is based on head (aed1dff) compared to base (1807df5).
Patch coverage: 97.56% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2543      +/-   ##
==========================================
+ Coverage   85.95%   86.26%   +0.30%     
==========================================
  Files         285      289       +4     
  Lines       15540    15823     +283     
  Branches     1959     1997      +38     
==========================================
+ Hits        13358    13650     +292     
+ Misses       1799     1785      -14     
- Partials      383      388       +5     

Impacted Files	Coverage Δ
src/fides/api/ops/tasks/__init__.py	79.41% <ø> (ø)
.../service/privacy_request/request_runner_service.py	77.06% <85.00%> (+2.77%)	⬆️
src/fides/api/ops/models/privacy_request.py	95.94% <87.50%> (-0.19%)	⬇️
src/fides/api/ops/schemas/messaging/messaging.py	98.69% <94.73%> (-0.57%)	⬇️
..._configuration/connection_secrets_email_consent.py	95.12% <95.12%> (ø)
src/fides/api/main.py	80.26% <100.00%> (+0.26%)	⬆️
...ides/api/ops/email_templates/get_email_template.py	64.70% <100.00%> (+2.20%)	⬆️
...rc/fides/api/ops/email_templates/template_names.py	100.00% <100.00%> (ø)
src/fides/api/ops/models/connectionconfig.py	95.83% <100.00%> (+0.04%)	⬆️
src/fides/api/ops/models/policy.py	86.90% <100.00%> (+0.04%)	⬆️
... and 12 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[cypress]

Passing run #167 ↗︎

0

3

0

0

0

Details:

Merge aed1dff into 1807df5...
Project: fides	Commit: 7b2b1f1bc8 ℹ️
Status: Passed	Duration: 00:34 💡
Started: Feb 16, 2023 4:56 PM	Ended: Feb 16, 2023 4:57 PM

_{This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.}

[allisonking]

whew, awesome work!! tested it with your instructions and everything worked as expected 💯

[pattisdr]

Failing test is unrelated to this work and being addressed separately here: https://github.com/ethyca/fides/pull/2596/files#r1107779266