Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add taxonomy entity form #1019

Merged
merged 15 commits into from
Aug 26, 2022
Merged

Add taxonomy entity form #1019

merged 15 commits into from
Aug 26, 2022

Conversation

allisonking
Copy link
Contributor

@allisonking allisonking commented Aug 24, 2022

Closes #860

Code Changes

  • Rename data-categories.slice --> taxonomy.slice and update related imports
  • Add create mutation to each taxonomy related slice
  • Add setting if the create form is open to redux
  • Derive the parent key from the fides key when adding a new fides key. This wasn't in the designs, but I think it's a nice UX
  • Extend TaxonomyFormBase to also be able to handle creating (mostly copy changes depending on if it is an edit or create form)
  • Add create calls to taxonomy hooks to bring everything together
  • Fixes some incorrect nullish coalescing (learned that empty strings do not nullish coalesce!)
  • Removes parent_key from Data Subject (only just realized it doesn't exist on that field)
  • Cypress tests

Steps to Confirm

  • Try adding taxonomy entities!
    Note: Data Subject requires rights/strategy right now. I think this will be fixed by something in Delete taxonomy UI #1006, but I'll double check once both PR's are merged.

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation Updated:
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md

Description Of Changes

Screen.Recording.2022-08-24.at.4.41.49.PM.mov

@allisonking allisonking marked this pull request as ready for review August 25, 2022 00:44
@allisonking allisonking requested a review from a team August 25, 2022 00:44
Copy link
Contributor

@ssangervasi ssangervasi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice test coverage!

@allisonking allisonking merged commit ba09cec into main Aug 26, 2022
@allisonking allisonking deleted the aking-860-add-taxonomy branch August 26, 2022 16:23
ThomasLaPiana added a commit that referenced this pull request Sep 2, 2022
* Bump next-auth from 4.9.0 to 4.10.3 in /clients/ops/privacy-center (#1014)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.9.0 to 4.10.3.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.9.0...next-auth@v4.10.3)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Bump fideslib (#1092)

* Bump fideslib to fix docs auth issue

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 983 Adds infra for email config and dispatch (#1059)

* Email config mvp crud / db layer

* adds name to email config model

* remove unintended changes

* gets POC working

* removes org name config var, updates crud endpoints to better handle supporting only 1 config, formatting

* updates postman collection, adds to changelog, bumps downrev on migration

* sort

* formatting

* use correct response model

* Delete base.py

* update migration annotation

* Create a Saas Connector from a Template [#814] (#1076)

* Starting point for SaaS connector templates

* Fix imports from restructuring.

* Get happy path working for instantiate connector from template endpoint.

* Remove updating connector instances for now - out of scope.

* Test nonexistent templates, secrets validation, instance key / fides key already exists.

* Create DatasetConfigs and ConnectionConfigs instead of create_or_update in the template endpoint.  Don't save ConnectionConfig until secrets are validated.

* Add the other saas connectors to the registry and update their configs and datasets with instance_fides_key.

- Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted.

* Update the fides_keys in the existing saas configs and dataset yamls to have brackets around the "instance_fides_key" to indicate these will be replaced.

Update the fides_key definition to allow "<instance_fides_key>" with brackets specifically to pass validation.

* Fix a side effect on a separate endpoint that returns the types of secrets that should be supplied for a given connector.  Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected.

* - Update CHANGELOG
- Add new endpoint to postman collection
- Add drafts doc.
- Update old response body in docs for connection types.

* Replace the <instance_fides_key> with a properly formatted fides_key in the saas fixtures.

* If DatasetConfig creation fails, delete the recently created ConnectionConfig.

* Address some of the saas integration tests where I've changed the fides_key.

* Fix typos.

* Fix typo.

* Fix unrelated bug where hubspot dataset has new datacategories with user-* data categories after the fideslang update, so they would show up if the user picked a "user" data category.

* Respond to CR.

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

* Update the "instantiate_connection_from_template" to return a portion of both the connection config and the dataset. (#1105)

* Adds `AuditLog` and `ExecutionLog`s to `seed_test_data` command (#1097)

* Reduce Idle Connections from Health Checks [#1102] (#1107)

* Don't create a new engine as part of running the health checks and share a single engine across the application, including for the health checks.  Currently we're using the default pool_size and max_overflow.

* Update changelog.

* Fix that health checks are still supposed to run, even if the database is disabled.

* Need to yield instead -  'generator' object has no attribute 'query'

* Escape redis user and password (#1104)

* Escape redis user and password

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* pass in analytics id env to worker (#1113)

* pass in analytics id env to worker

* changelog

* Adds tests for email endpoints and dispatch service (#1112)

* Endpoint to Verify User Identity [#1062] (#1111)

* Add an endpoint to verify a user's identity before queuing the privacy request provided it doesn't need separate manual approval by a system admin.

- Add a new PrivacyRequest.identity_verified_at timestamp
- Add a new PrivacyRequestStatus - "identity_unverified".
- Add methods to cache the verification code in Redis for comparison with a default ttl of 10 minutes

* - Fix linting/copy-paste errors.
- Update changelog.
- Add endpoint to postman

* Add new keys to response bodies.

* Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists.

* Revert "Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists."

This reverts commit 40fcf6d119135d08a6d3ecfc40c5d73846bf2205.

* Bump hashicorp/vault-action from 2.4.1 to 2.4.2 (#1119)

Bumps [hashicorp/vault-action](https://github.com/hashicorp/vault-action) from 2.4.1 to 2.4.2.
- [Release notes](https://github.com/hashicorp/vault-action/releases)
- [Changelog](https://github.com/hashicorp/vault-action/blob/main/CHANGELOG.md)
- [Commits](hashicorp/vault-action@v2.4.1...v2.4.2)

---
updated-dependencies:
- dependency-name: hashicorp/vault-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi[all] from 0.79.0 to 0.79.1 (#1120)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.0 to 0.79.1.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](fastapi/fastapi@0.79.0...0.79.1)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-urllib3 from 1.26.22 to 1.26.23 (#1117)

Bumps [types-urllib3](https://github.com/python/typeshed) from 1.26.22 to 1.26.23.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-urllib3
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump nox from 2022.1.7 to 2022.8.7 (#1118)

Bumps [nox](https://github.com/wntrblm/nox) from 2022.1.7 to 2022.8.7.
- [Release notes](https://github.com/wntrblm/nox/releases)
- [Changelog](https://github.com/wntrblm/nox/blob/main/CHANGELOG.md)
- [Commits](wntrblm/nox@2022.1.7...2022.8.7)

---
updated-dependencies:
- dependency-name: nox
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 14.0.0 to 14.1.0 (#1122)

Bumps [faker](https://github.com/joke2k/faker) from 14.0.0 to 14.1.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](joke2k/faker@v14.0.0...v14.1.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump sqlalchemy-redshift from 0.8.10 to 0.8.11 (#1121)

Bumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.10 to 0.8.11.
- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)
- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)
- [Commits](sqlalchemy-redshift/sqlalchemy-redshift@0.8.10...0.8.11)

---
updated-dependencies:
- dependency-name: sqlalchemy-redshift
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* POC for dynamic routing (#1033)

* Add initial POC for dynamic routing

* fix a couple lints

* Fix mypy lint

* making pylint happy

* Remove log

* Fix another pylint issue

* Add docstring

* Update index if check

* Handle nested nextjs routes

* Update changelog

* Add Sass connector configuration (#1099)

* Sass Connecter feature development

761 - Add a Connection - SaaS connector's configuration parameters
984 - Saas Connector configuration - left navigation to toggle between connection params and dataset config
985 - SaaS Connector Configuration - Testing a Connection

* Skipping unit test temporarily

* Update flags.json file

By default, turning off the createNewConnection flag which is still under development.

* Updated CHANGELOG.md file

* Resolved ESLint issues

* Resolved npm build issue

* Resolved npm build issue

* Updated Saas connector configuration

* Updated Saas connector configuration

* Removed unnecessary import statement

* add new privilege for creating SaaS connectors to user management interface

* add connections read as a privilege

* Applied code review feedback

* Updated the CSS visibility of the CircleHelpIcon component

* Added toast success when a user creates a Saas config

* Resolved ESLint issue

Co-authored-by: Sean Preston <sean@ethyca.com>

* 1128-Add Retry button back into the subject request detail view (#1131)

* 1128-Add Retry button back into the subject request detail view

* Updated CHANGELOG.md file

* provide a way to give invited users the resume permission

Co-authored-by: Sean Preston <sean@ethyca.com>

* Fix localhost setting for non-endpoint analytics calls [#1029] (#1130)

* Set local_host to None for non-endpoint analytics calls.  These are logging various tasks coming out of celery.

* Update Changelog.

* Add email_templates module (#1123)

* Add email_templates module

* run isort

* Add unit tests

* Update ttl calculation

* Add ttl minutes test

* fix lint issues

* fix pylint issue

* fix pylint issue

* fix isort

* Update template constant

* Update changelog

* fix lints

* Add jinja to requirements.txt

* update templates directory

* update unit test

* Update imports

* fix issue template path

* Add templates to manifest

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Formatting Issues in Webhooks Documentation (#1114)

* Fix typo in `derived_identity`

* Update CHANGELOG.md

* Missed PR Link

* replaced `<>` with `{}`

In the live version of the docs, the `<>`s were being stripped from our code example titles. I've replaced them with `{}` to align with some of the other pages I've seen.

* Send Identity Verification Email [#1010] (#1115)

* If identity verification required, send email to the user with the verification code.

* Adjust the identity_verification_required autouse fixture, and add an autouse override for just the tests where we want to turn on identity verification.

* Add starting docs and updating the changelog.

Start with identity_verification_required set to False for now until all the related pieces are in.

* Update some of the docstrings.

* Add unverified status color in the FE.

* Add new privacy request status to types and constants.

* Restore trailing comma.

* Update identity_verification_required to subject_identity_verification_required for clarity.

* Adjust email_body_params to accommodate new template.

Co-authored-by: Sean Preston <sean@ethyca.com>

* Support case-insensitive connection type search [Unticketed] (#1133)

* Make connection type search case-insensitive.

* Update changelog.

* Add option to login as root user from config (#1116)


Co-authored-by: Paul Sanders <pau@ethyca.com>

* Move logging to %-style formatting [#837] (#1132)

* Add a new Pii class and use it to wrap arguments not already wrapped with NonPii in those logs that are currently using %-style formatting.

* Switch logging formatting to %-style instead of f-string.

* Continue to address lingering f string instances, and wrap some arguments in Pii, such as raw exceptions.

* Remove NotPii class and update tests.

* Adjust errors made in %-style conversion.

* Remove accidental Pii on print statements, update some PII wrappings.

* Adjust string formatting of newly added log.

* Update Changelog.

* Fix missed closing curly brackets.

* Remove missed curly brackets.

* Add future annotations for backwards compatibility (#1136)

* Add future annotations for backwards compatibility

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Release 1.7.1 (#1141)

* update changelog

* update ERD

* make Hubspot test more robust

* Fix docs build in CI (#1138)

* Fix docs build in CI

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* fix command syntax (#1143)

* Prevent modifying default taxonomy fields (#990)

* Add is_default field to taxonomy sql models (#976)

* Add is_default field to taxonomy sql models

* Update changelog

* Autoformat alembic migration file

* Update dataset.yml with is_default

* Bump fideslang version

Update changelog

Autoformat alembic migration file

Update dataset.yml with is_default

Bump fideslang version

* Prevent modifying default taxonomy on update

Add docstring

* Move is_default check to routes/crud.py

Revert database/crud.py changes

* Handle forbidding when given a list of fides keys

* Add test

* Handle attempting to modify is_default field

* Update changelog

* Debug failing test

* Add another print debugging stmt

* Revert "Add another print debugging stmt"

This reverts commit 527bd03.

* Revert "Debug failing test"

This reverts commit daf3ca8.

* Scope resources_dict to function

* Clean up changelog

* Allow modifying defaults but not is_default

* Handle case where checking for new upsert

* Refactor to put tests in a class

* Handle upserting is_default

* hook up API request to logout action [Unticketed] (#1139)

* hook up API request to logout action

* run format

* update import orer

* add lint:fix

* Update hubspot users and owners configurations (#1091)

* Update CHANGELOG.md

* Delete taxonomy UI (#1006)

* Add delete call to slices

* Hook up delete button

* Add result handling

* Only show delete on nodes without children

* Add tests for delete

* Update changelog

* Render action buttons as a prop to AccordionTree

* Fix import consistency

* Rename onEdit and onDelete

* Clear edit entity on delete

* Use TreeNode type

* Update cypress fixtures (#1022)

* Update fixtures

* Update tests based on updated fixtures

* Add taxonomy entity form (#1019)

* Rename data-categories.slice --> taxonomy.slice

* Add active taxonomy type to store

* Add create mutation to slices

* Hook up create to form

* Fixup form UX

* Derive parent key from fides key

* Conditionally render parent key field

* Add tests for adding taxonomy entities

* Clean up

* Update changelog

* Simplify setting add state

* Add test for showing either add or create form

* Derive isCreate from status of fides key

* SaaS Connector Template Creation Fix: Integer fides_key (#1166)

* Use quotes when replacing <"instance_fides_key"> in the saas config and dataset config files to force a string.

* Update CHANGELOG.

* Start a new "email" ConnectionConfig type [#1134] (#1142)

* Start a new "email" ConnectionConfig type.

* Hide "email" type from the get_connection_types endpoint for now, as the email connector isn't fleshed out yet.

* Update CHANGELOG.

* Simplify by sending one email to start?

* Update request body in postman collection.

* Fix CHANGELOG formatting.

* Revert "SaaS Connector Template Creation Fix: Integer fides_key (#1166)" (#1171)

This reverts commit 19fbe93.

* Fix analytics opt out environment variable name (#1170)

* 1004 OAuth2 client credentials flow (#1159)

* update Subject Request detail page (#1164)

* [#495] clarify additions to subject request event log

* [#863] note retry capabilities in subject requests

* changelog

* fidesops.toml

* changelog

Co-authored-by: Cole Garbo <colegarbo@Ethycas-MacBook-Pro.local>

* Bump fastapi[all] from 0.79.1 to 0.81.0 (#1178)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.1 to 0.81.0.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](fastapi/fastapi@0.79.1...0.81.0)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add check for is_default before rendering delete button (#1023)

* Add check for is_default before rendering delete button

* Update changelog

* Update test data to include is_default

* Add cypress tests

* Build docker image for privacy center (#1165)

* Build docker image for privacy center

* Update docs and add docker publish to workflow

* Update CHANGELOG

* Move privacy center docker publish to its own workflow

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Update `StorageConfig` to handle multiple auth methods (#1140)

* Initial automatic impl

* Lints

* disable pylint error

* Fix unit tests

* fix imports

* Update fixture

* Fix config tests

* Covert type to enum

* Add new unit tests

* Update changelong

* Update docs

* Update phone number

* patch task scheduler

* remove pylint ignore and add exception

* Update docs

* 1009 id verification required endpoint (#1221)

* Adds identity verification config endpoint

* adds test, reuse existing config var, adds postman collection

* update changelog

* add type ignore

* move ignore line

* Update src/fidesops/ops/graph/config.py

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* remove duplicate lines in method

* another dupe line

* remove scope, update order of classmethod decorator

* remove dependencies on endpoint

* unused imports

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* fix reference in data categories upgrade migration (#1223)

* Fix mypy error (#1030)

* Fix mypy error

* Update CHANGELOG

* Remove unused import

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Publish on every release (#1226)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Make log send async (#1174)

* Make log send async

* Update CHANGELOG

* Add async to additional fideslog calls

* WIP

* Fix issue with async function is celery

* Make __send work with name mangling

* Remove extra await

* Await coroutines in tests

* Remove analytics id

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 1106 saas config shopify access endpoints (#1220)

* Custom label for user defined taxonomy fields (#1027)

* Add renderTag prop

* Add cypress tests

* Update changelog

* Adds email communications guide to docs, exposes other hidden guides in nav bar (#1233)

* adds email communications guide to docs, exposes other hidden guides in nav bar

* update changelog

* email docs copyedits

* remove dupe items from nav

Co-authored-by: Cole <cole@ethyca.com>

* Add boolean fields and use them in taxonomy forms (#1028)

* Add CustomRadioGroup as a form input

* Use new radio group input for taxonomy forms

* Update extra form fields prop to be a function

* Cast string boolean back to real boolean

* Tests

* Update changelog

* Clean up some type comparisons

* Revert === undefined since the fields can actually be null

* Fix version number (#1232)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* remove exclude_unset=True to return clean diff (#1026)

* remove exclude_unset=True to return clean diff

* changelog

* don't stop the test suite when there is a failure

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Merge project changes again

* fix pylint issue

* Bump pylint from 2.14.5 to 2.15.0 (#1182)

* Bump pylint from 2.14.5 to 2.15.0

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.14.5 to 2.15.0.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Commits](pylint-dev/pylint@v2.14.5...v2.15.0)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Ignore missing-timeout warning

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Sanders <paul@ethyca.com>

* Bump fideslib (#1236)

* Bump fideslib

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Email Connector: Build Masking Instructions  (#1168)

* Pass in input_data to erasure requests, and not just access requests, so it can be used for the email connector, which won't have any rows returned from an access request.

- Add an EmailConnector.build_masking_instructions method with a draft of data needed to instruct the user how to query/mask/what fields to mask on their end.

* Have the EmailConnector.mask_data  cache the raw details of what needs to be masked in Redis. We'll use this to send one email at the end for each "email"-based dataset at end, instead of sending one email for each collection.

Reuse some of the caching code created for manual connectors / failed privacy requests where similar to the EmailConnectors, we have some separate action that is required on a given collection.  Rename to make more generic.

* Remove restriction that a ManualAction needs a get or update value.  The manual action could just be locating data for another collection downstream.

Cache email template details, even if there are no actions needed on that specific collection,

* Update the expected number of collections in the email dataset.

* build_masking_instructions is not required to return a ManualAction.

* Reconcile this test with the work to make log send asynchronous.

Co-authored-by: Sean Preston <sean@ethyca.com>

* Adds email scopes to postman collection (#1241)

* Adds email scopes to postman collection

* update changelog

* Bump black from 22.6.0 to 22.8.0 (#1238)

Bumps [black](https://github.com/psf/black) from 22.6.0 to 22.8.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](psf/black@22.6.0...22.8.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Rename fidesops-privacy-center image to fides-privacy-center (#1237)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump eslint from 8.9.0 to 8.23.0 in /clients/ops/privacy-center (#1180)

Bumps [eslint](https://github.com/eslint/eslint) from 8.9.0 to 8.23.0.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)
- [Commits](eslint/eslint@v8.9.0...v8.23.0)

---
updated-dependencies:
- dependency-name: eslint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* #1008 User  identity verification flow (#1231)

* Update lint commands and eslint ignore

* Run lints and refactor privacy cards

* Refactor modal

* Finish verification flow

* get config from server

* Update changelong

* Fix test failures

* Format file

* Mock out route

* Format file

* Add code resending

* Update test to use hostUrl

* Add headers util function and PrivacyRequestStatus status enum

* Bump @chakra-ui/utils from 1.10.4 to 2.0.9 in /clients/ops/admin-ui (#1145)

Bumps [@chakra-ui/utils](https://github.com/chakra-ui/chakra-ui/tree/HEAD/packages/utils) from 1.10.4 to 2.0.9.
- [Release notes](https://github.com/chakra-ui/chakra-ui/releases)
- [Changelog](https://github.com/chakra-ui/chakra-ui/blob/main/packages/utils/CHANGELOG.md)
- [Commits](https://github.com/chakra-ui/chakra-ui/commits/@chakra-ui/utils@2.0.9/packages/utils)

---
updated-dependencies:
- dependency-name: "@chakra-ui/utils"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix broken ctl tests

* static checks passing

* merge the updated ops files into their ctl counterparts

* remove the deprecated apply command

* get the server running

* get static checks passing

* fix ops unit tests except for two new identity verification tests

* fix the failing identity_verification tests

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sean Preston <sean@ethyca.com>
Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Catherine Smith <eastandwestwind@gmail.com>
Co-authored-by: Adrian Galvan <adrian@ethyca.com>
Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>
Co-authored-by: Andrew Jackson <andrew.c.j1995@gmail.com>
Co-authored-by: chriscalhoun1974 <68459950+chriscalhoun1974@users.noreply.github.com>
Co-authored-by: Robert Keyser <39230492+RobertKeyser@users.noreply.github.com>
Co-authored-by: Allison King <allisonjuliaking@gmail.com>
Co-authored-by: Eduardo Armendariz <eduardo.armendariz13@gmail.com>
Co-authored-by: Kelsey Thomas <101993653+Kelsey-Ethyca@users.noreply.github.com>
Co-authored-by: Cole Isaac <82131455+conceptualshark@users.noreply.github.com>
Co-authored-by: Cole Garbo <colegarbo@Ethycas-MacBook-Pro.local>
Co-authored-by: HamzaWaseemOnBench <104363089+HamzaWaseemOnBench@users.noreply.github.com>
Co-authored-by: Cole <cole@ethyca.com>
Co-authored-by: Steve Murphy <steven.d.murphy@gmail.com>
Co-authored-by: Paul Sanders <paul@ethyca.com>
ThomasLaPiana added a commit that referenced this pull request Sep 23, 2022
* Prevent modifying default taxonomy fields (#990)

* Add is_default field to taxonomy sql models (#976)

* Add is_default field to taxonomy sql models

* Update changelog

* Autoformat alembic migration file

* Update dataset.yml with is_default

* Bump fideslang version

Update changelog

Autoformat alembic migration file

Update dataset.yml with is_default

Bump fideslang version

* Prevent modifying default taxonomy on update

Add docstring

* Move is_default check to routes/crud.py

Revert database/crud.py changes

* Handle forbidding when given a list of fides keys

* Add test

* Handle attempting to modify is_default field

* Update changelog

* Debug failing test

* Add another print debugging stmt

* Revert "Add another print debugging stmt"

This reverts commit 527bd03.

* Revert "Debug failing test"

This reverts commit daf3ca8.

* Scope resources_dict to function

* Clean up changelog

* Allow modifying defaults but not is_default

* Handle case where checking for new upsert

* Refactor to put tests in a class

* Handle upserting is_default

* Delete taxonomy UI (#1006)

* Add delete call to slices

* Hook up delete button

* Add result handling

* Only show delete on nodes without children

* Add tests for delete

* Update changelog

* Render action buttons as a prop to AccordionTree

* Fix import consistency

* Rename onEdit and onDelete

* Clear edit entity on delete

* Use TreeNode type

* Update cypress fixtures (#1022)

* Update fixtures

* Update tests based on updated fixtures

* Add taxonomy entity form (#1019)

* Rename data-categories.slice --> taxonomy.slice

* Add active taxonomy type to store

* Add create mutation to slices

* Hook up create to form

* Fixup form UX

* Derive parent key from fides key

* Conditionally render parent key field

* Add tests for adding taxonomy entities

* Clean up

* Update changelog

* Simplify setting add state

* Add test for showing either add or create form

* Derive isCreate from status of fides key

* Add check for is_default before rendering delete button (#1023)

* Add check for is_default before rendering delete button

* Update changelog

* Update test data to include is_default

* Add cypress tests

* Fix mypy error (#1030)

* Fix mypy error

* Update CHANGELOG

* Remove unused import

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Custom label for user defined taxonomy fields (#1027)

* Add renderTag prop

* Add cypress tests

* Update changelog

* Add boolean fields and use them in taxonomy forms (#1028)

* Add CustomRadioGroup as a form input

* Use new radio group input for taxonomy forms

* Update extra form fields prop to be a function

* Cast string boolean back to real boolean

* Tests

* Update changelog

* Clean up some type comparisons

* Revert === undefined since the fields can actually be null

* remove exclude_unset=True to return clean diff (#1026)

* remove exclude_unset=True to return clean diff

* changelog

* don't stop the test suite when there is a failure

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Set pydantic < 1.10.0 to fix CI issues with fideslang functions (#1045)

Should be able to revert after fideslang figures out why the latest
pydantic causes failures

* [fidesctl-plus #78] cross app navigation (#1037)

* fctl/nav: Move NavBar into nav directory

* fctl/nav: Decouple Header and NavBar

* fctl/nav: Extract NavButton presentational component

* fctl/nav: Extract NavLink component

* fctl/zones: Generic utility for configuring app zones

This change brings in some zone handling code from fidesctl-plus and makes it more generic so that
we can use the same pattern in both apps.

* fctl/nav: Use zone-aware nav links

* fctl/nav: Replace Jest nav tests with Cypress tests

This change switches moves our nav routing tests into Cypress instead of Jest.
The React test renderer was having trouble integrating with Next's dynamic import,
which was going to require a complex solution. Instead, by moving the nav tests into
Cypress we can test the real router state without relying on mocks.

This will prove even more useful in a later commit when we update what links are available
based on whether the API says we are in fidesctl-plus.

* fctl/features: Query /plus/health API to determine if Plus features should be shown

* Update changelog

* 1.8.3 (#1050)

* Prepare changelog for 1.8.3 release

* Fixup misattributed changelog items

* Add fix for pydantic version

* noxfiles: Session for building fidesctl python package (#1047)

* noxfiles: Session for building fctl python package

* fix a pylint error

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* [942] fctl/api: Serve static files using route maps (#1046)

* fctl/api: Adapt generate_route_file_map from Ops and unit test it

* fctl/api: Serve static files using route maps

This includes both packaged (pip installed) and local build files, depending on
what is available.

* Update changelog

* Fix truncated evaluation error messages (#1053)

* Fix truncated evaluation error messages

* add a test for the evaluations output

* fix mypy error

* update changelog

* Bump next-auth from 4.9.0 to 4.10.3 in /clients/ctl/admin-ui (#1025)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.9.0 to 4.10.3.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.9.0...next-auth@v4.10.3)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Allison King <allisonjuliaking@gmail.com>

* Cascade delete taxonomy children (#1042)

* Configure pytest for async functions

* Refactor FixtureRequest so it can be shared across tests

* Add test for deleting children taxonomy

* Add logic to cascade delete

* Update changelog

* Update UI to allow deleting parent taxonomy fields with a warning

* Clean up

* Update cypress tests

* Initial systems management page (#1054)

* Rename SystemsTable to SystemsGrid

* Add BorderGrid component to abstract out grid

* Build out SystemCard

* Add search feature

* Pull SystemCard out into its own component

* Add cypress tests for system management page

* Update changelog

* Fix nav bar test

* 1.8.4 (#1061)

* update changelog

* remove phantom bullet point

* Alter taxonomy upsert behavior (#1040)

* Configure pytest for async functions

* Add tests for changing updating default taxonomy

* Change upsert behavior to append

* Update changelog

* Address PR comments

* Replace upsert with create

* Remove unused import

* Remove disabled Nav Buttons (#1067)

* Update NavBar.tsx

* remove "more" from the navbar

* fix a linting error

* remove cypress tests for disabled links

* Form to add a system via yaml (#1062)

* Add new system page

* Refactor YamlForm so it can be reused

* Fixup some types in existing system slice

* Add SystemYamlForm

* Add cypress test for system

* Update changelog

* Fix empty state and remove ellipsis for now

* Remove tests on more actions button

* Move changelog items to Unreleased

* ui/dataset: "Classify" toggle for fidesctl-plus (#1057)

* ui/dataset: Spacing improvements for generate form

* ui/inputs: CustomSwitch component for switches

* ui/dataset: "Classify" toggle for fidesctl-plus

* ui/dataset: Cypress test when classify is available

* Update changelog

* [1058] ui/dataset: Confirmation modal to kick off classify (#1069)

* ui/featuers: Extract Plus API into its own slice

* ui/dataset: Refactor dataset creation chain of mutations

This should make it easier to extend the sequence of API calls to add Classify.
Before, the structure made it hard to identify the orders in which mutations were called,
and which object was the generated (temporary) vs persisted dataset. Now the generate
and create functions return their results or an error string.

I also made a change to how the error message is shown: instead of always assigning
the message to the form's URL field, it's shown in a error toast. Now that we have
multiple fields on this form, it wasn't clear the URL is going to be responsible
for any errors.

* ui/plus: Mock implementation of the classify API

* ui/dataset: Request classify if toggled

* ui/dataset: Confirmation modal to kick off classify

* ui/dataset: Cypress test for starting classify

* ui/dataset: Clear active dataset on un-mount instead of mount

This change makes it possible for the classify flow to highlight the newly-created
dataset when we navigate back to the table. Previously, the active dataset was cleared
by index page, which made preserving the active set between routes impossible. Now
we only clear when we leave a datset's view, which gives us the same experience and
the new feature.

* Update changelog

* Scaffold manual system flow (#1068)

* Refactor generate type to boolean

* Add ManualSystemFlow

* Add ConfigureSteps

* Make button bigger

* Update changelog

* Remove UI features for WIP elements

* [1073] ui/dataset: Show status badge for datasets using classification results (#1074)

* ui/plus: Classifications grouped by dataset fides key

* ui/dataset: Show status badge for datasets using classification results

* ui/dataset: Cypress tests for classified table

* Update changelog

* ui/state: Deduplicate query results that were being stored in state slices (#1083)

* ui/dedup-state: Systems unused

* ui/dedup-state: Datasets unused

* ui/dedup-state: Organization hydrate unused

* ui/dedup-state: Taxonomy DataCategories query

* ui/dedup-state: DataSubjects query

* ui/dedup-state: DataQualifiers query

* ui/dedup-state: DataUses query

* Fix header help link (#1078)

* Fix header help link

* update changelog

* Reuse config wizard forms for adding a system (#1072)

* Consolidate tooltips in DescribeSystemsForm

* Add form to manual system flow

* Add validation schema for DescribeSystemsForm

* Fix onBlur handlers of select fields

* Refactor DescribeSystemsForm

* Refactor PrivacyDeclarationForm

* Continue refactoring PrivacyDeclarationForm

* Refactor ReviewSystemForm

* Refactor SuccessPage

* Update changelog

* Refactor ReviewSystemForm to use a grid ReviewItem

* Add cypress test for flow

* Try to fix flaky test

* Ensure we stub taxonomy items

* Refactor PrivacyDeclarationAccordion to match designs more

* Fix adding another declaration when name field is blank

* Rename SuccessPage --> SystemRegisterSuccess

* Pass system object through props

* Update tests

* UI to delete a system (#1085)

* Implement delete system feature

* Add NotFoundError

* Add cypress tests

* Update changelog

* Refactor errors to ErrorDetails

* Separate 'next' and 'add' logic in PrivacyDeclarationForm (#1086)

* Separate continue and add logic

* Update tests

* Update changelog

* New fields on system forms (#1082)

* Extend DescribeSystemsForm

* Extend PrivacyDeclarationForm

* Add joint controller and data protection impact assessment

* Move system dependency field to abridged form

* Prepare data protection impact assessment for payload

* Add cypress tests

* Extend ReviewSystemForm

* Add tests for extended review form

* Fix adding another declaration when name field is blank

* Rename SuccessPage --> SystemRegisterSuccess

* Pass system object through props

* Update tests

* Update changelog

* Refactor ReviewSystemForm

* Move config wizard system forms to system directory (#1097)

* Move system forms to system directory

* Refactor form layout components into its own file

* Rename files from form --> step

* Update changelog

* ui/datasets: Refactor selectors (#1087)

* ui/dataset: Track dataset by fides key instead of object copy and nulls

Previously we've stored a copy of the dataset returned by the get-by-key query,
but only every use its key property. With this change we only store the key and
use the object from the query directly.

As part of this, I also converted a lot of `null` to `undefined`. There different
opinions on this, but undefined is generally more useful in a TS codebase because:

1. It plays nicely with optional function arguments.
2. It can be represented by a single question mark in interfaces. (See the State
   changes in this commit.)
3. It doesn't have null's `typeof null === "object"` confusion.

* ui/dataset: Single state for edit drawer

* ui/dataset: Extract collection lookups to selector

* ui/dataset: Extract field lookups to selector

* ui/dataset: Extract field types into Cell component

* ui/cypress: Consistent test fides key

* Bump fideslang to 1.3.0 (#1103)

* Bump fideslang to 1.3.0

* Add `egress` and `ingress` to `ctl_systems`

* Update `CHANGELOG.md`

* Prepare 1.8.5 release (#1107)

* Bump fideslang to 1.3.0

* Add `egress` and `ingress` to `ctl_systems`

* Update `CHANGELOG.md`

* Prepare `CHANGELOG.md` for v1.8.5

* upgrade pymysql to version 1.0.2 (#1094)

* Edit system UI (#1096)

* Set activeSystem in system.slice

* Allow form fields to take undefined

* Extend DescribeSystemForm to be able to edit

* Allow editing system from its card

* Update changelog

* Pull the form part of privacy declaration into its own component

* Further refactor PrivacyDeclarationForm to support editing

* Pull common intercepts out into a stubs file

* Add cypress test for editing

* Reserve dataset_references for unabridged forms

* Make sure not to override privacy declaration

* Explain allowing undefined for form inputs

* Change routing behavior to go back on cancel

* Fix error message

* Update CHANGELOG.md

* included more changes from fidesctl

* fix static checks

* manual updates from ctl UI

* formatting

* change line endings to fix prettier error

* fix docker build issue

* merge alembic heads for fidesctl merge and get tests parsing

* don't include the worker as a dependency for docker-compose fides

* remove worker flag from default fides service

* fix static checks

* get webserver running again

* reenable the worker for the docker-compose service

* enable redis for the compose service

* clean command is absolute in its destruction

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Allison King <allisonjuliaking@gmail.com>
Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Steve Murphy <steven.d.murphy@gmail.com>
Co-authored-by: Sebastian Sangervasi <2236777+ssangervasi@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Phil Salant <PSalant726@users.noreply.github.com>
ThomasLaPiana pushed a commit that referenced this pull request Sep 26, 2022
* add pagination back to connection types endpoints

* check for pagination in system_type search

Co-authored-by: Paul Sanders <pau@ethyca.com>
ThomasLaPiana added a commit that referenced this pull request Sep 27, 2022
* Delete Datastore [Frontend] (#683)

* Link Delete Connection menu item with confirmation modal, patterned off of user delete modal.

* Update changelog.

* Vertically center modals.

* Add a spinner while collection is being deleted. Only close modal after successful deletion.

* Prevent closing modal while we're still making the delete request.

* 546 - [Admin UI] Change "Policy Name" to "Request Type" on SR list page (#696)

* 546 - [Admin UI] Change "Policy Name" to "Request Type" on SR list page

* Rollback file change

* Updated CHANGELOG.md file

Co-authored-by: Christopher Calhoun <chris@ethyca.com>

* Enable/Disable Datastores [Frontend] (#693)

* Add the ability to enable/disable a connectionconfig.

* Fix other location to prevent from closing modal while in progress.

* Update changelog.

* 659 Add Postgres and Redis to health endpoint (#690)

* adds db and redis to health endpoint, untested

* updates typing, suppresses sqlalchemy and alembic logs for healthchecks

* adds to makefile, updates docs and tests

* format

* adds check for cache enabled, update tests

* format

* standardize health endpoint structure

* Execute Privacy Requests with Celery (#621)

 Updates the way privacy requests are dispatched into processing from a background process into a Celery task

* 512 db redis health (#686)

* catches exceptions from db/redis and provides better errors on startup

* move health endpoint changes to separate branch

* update changelog

* catch appropriate redis err, stop app upon db and cache connection failure

* adds check for cache enabled

* Datastore Connection Filtering (#691)

* Refactor routes into enums and create connections page

* Test switching back to double quotes

* Convert back to double quotes

* Add placeholder connection filters

* Set up api scaffolding

* Get basic grid going

* Initial grid card styling

* Fix simple eslint issues

* Add development config back in

* Finish draft of card

* Add working test button and landing page

* Add pagination and small fixes

* Fix testing issues

* Add auth tests for datastore connection page

* run formatter

* Update changelog

* update the create_test_data command to add connectionconfigs

* Disable create buttons & fix text overflow

* Update filter dropdown values

* Fix test timestamp bug

* Remove development variable

* Add working filter dropdowns

* Add outside click hook & polish things

* Fix imports

* Update changelog

* Update button hover color

* remove commented out code

* fix typo

* Remove Saas Option

* Fix welcome screen bug

* Remove edit button

* Fix lint and formatting issues

* removes commented-out code

Co-authored-by: Sean Preston <sean@ethyca.com>
Co-authored-by: eastandwestwind <eastandwestwind@gmail.com>

* update domain -> host config references in stripe and sentry test infrastructure (#698)

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>
Co-authored-by: Adrian Galvan <adriang430@gmail.com>

* Clean up postman collection (#704)

* Revoke a Pending Privacy Request [#525] (#592)

* Add the ability to cancel a pending task.  The celery task is not actually cancelled yet.

- Track cancel reason, datetime cancelled, and add a new cancelled status.

* Add drp revoke request to postman collection.

* Add drp revoke docs.

* Update down_rev after rebase.

* Fix incorrect check.

* Restore new canceled state.

* Check that the privacy request is not canceled right before starting execution.  This is really our last chance to check before we start executing the graph in dask.    The use case here might be it was canceled shortly after it was approved.

* Attempt to revoke a queued celery task if we cancel it before it starts executing.

* Prettier.

* Changelog updated.

* Add a few unit tests around how triggering the run_privacy_request_task with a cancelled task id doesn't do anything and how you can't approve a canceled privacy request.

* Fix SQLAlchemy logging to console - logging in migration propagates to the rest of the application.

* Refresh session instead of creating a new one.

* Add 200 character limit.

* Add some assertions that db.refresh is doing what we think it's doing.

* Update CHANGELOG.md, bump to `fideslib==2.1.0`  (#705)

* bump version of fideslib

* updates changelog pre-release

* Update datastore connection filters to support clearing (#701)

* Update connection filters to clear

* Use new common header function for analytics

* Update prepareHeaders function to make type checker happy

* Update changelog

* Management UI updates (#702)

* reorder ui docs

* admi ui

* subject request detail finalization, datastore stub

* config edit

* comment out docs not implemented

* remove tracking

* changelog

* remove stray comment, add Canceled status

* Remove stray hyphen (#709)

* Reduce docker image size (#707)

* Reduce docker image size

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump next-auth from 4.3.2 to 4.5.0 in /clients/privacy-center (#694)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.3.2 to 4.5.0.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.3.2...next-auth@v4.5.0)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update tutorial directions to install fidesdemo from the root directory (#715)

* update tutorial directions to install fidesdemo from the root directory

* changelog

* [SaaS Connector] Salesforce (access) (#676)

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>
Co-authored-by: Hamza W <hamza@Hamzas-MacBook-Pro.local>

* Parallelize CI safe checks (#717)

* Parallelize CI safe checks

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Make reading of environment variables case insensitive (#712)

* Make reading of environment variables case insensitive

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Add dependabot (#718)

* Add dependabot

* Update CHANGELOG

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump requests-mock from 1.8.0 to 1.9.3 (#732)

Bumps [requests-mock](https://github.com/jamielennox/requests-mock) from 1.8.0 to 1.9.3.
- [Release notes](https://github.com/jamielennox/requests-mock/releases)
- [Commits](https://github.com/jamielennox/requests-mock/compare/1.8.0...1.9.3)

---
updated-dependencies:
- dependency-name: requests-mock
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump packaging from 20.9 to 21.3 (#733)

Bumps [packaging](https://github.com/pypa/packaging) from 20.9 to 21.3.
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pypa/packaging/compare/20.9...21.3)

---
updated-dependencies:
- dependency-name: packaging
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-redis from 4.2.6 to 4.3.2 (#729)

Bumps [types-redis](https://github.com/python/typeshed) from 4.2.6 to 4.3.2.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-redis
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 8.12.1 to 13.14.0 (#730)

Bumps [faker](https://github.com/joke2k/faker) from 8.12.1 to 13.14.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v8.12.1...v13.14.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-python from 2 to 4 (#724)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v2...v4)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/checkout from 2 to 3 (#723)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github/codeql-action from 1 to 2 (#725)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 1 to 2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump docker/login-action from 1 to 2 (#722)

Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-node from 2 to 3 (#726)

Bumps [actions/setup-node](https://github.com/actions/setup-node) from 2 to 3.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [#716] update datastore documentation (#742)

* Adds `fidesops worker` command to start the Celery worker [#663] (#673)

* add fidesops worker cmd

* rename Dockerfiles, adds worker specific dockerfile with no frontend

* formatting

* update container names, add separate dockerfile for worker

* pass worker args through to worker_main

* rename containers fidesops -> webserver, celery -> worker

* fix imports

* add return type

* update dockerfile

* describe changelog

* specify same argv for all celery instantiation

* add logging to cli

* add back .egg

* specify dockerfile to build from

* 708 - fix console warning in disable connections (#750)

* removes unneeded code in disable connection

* Bump fideslib from version 2.1.0 to 2.1.1 (#721)

* Bump fideslib from version 2.1.0 to 2.1.1

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Password Hashing update (#749)

* Fix no such container error with docker-compose (#758)

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* 1.6.1 release (#763)

* Update CHANGELOG for release

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Bump psycopg2-binary from 2.9.1 to 2.9.3 (#783)

Bumps [psycopg2-binary](https://github.com/psycopg/psycopg2) from 2.9.1 to 2.9.3.
- [Release notes](https://github.com/psycopg/psycopg2/releases)
- [Changelog](https://github.com/psycopg/psycopg2/blob/master/NEWS)
- [Commits](https://github.com/psycopg/psycopg2/commits)

---
updated-dependencies:
- dependency-name: psycopg2-binary
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2021.10.0 to 2022.6.1 (#781)

Bumps [dask](https://github.com/dask/dask) from 2021.10.0 to 2022.6.1.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2021.10.0...2022.6.1)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump alembic from 1.6.5 to 1.8.0 (#780)

Bumps [alembic](https://github.com/sqlalchemy/alembic) from 1.6.5 to 1.8.0.
- [Release notes](https://github.com/sqlalchemy/alembic/releases)
- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/alembic/commits)

---
updated-dependencies:
- dependency-name: alembic
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump sqlalchemy-bigquery from 1.3.0 to 1.4.4 (#779)

Bumps [sqlalchemy-bigquery](https://github.com/googleapis/python-bigquery-sqlalchemy) from 1.3.0 to 1.4.4.
- [Release notes](https://github.com/googleapis/python-bigquery-sqlalchemy/releases)
- [Changelog](https://github.com/googleapis/python-bigquery-sqlalchemy/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/python-bigquery-sqlalchemy/compare/v1.3.0...v1.4.4)

---
updated-dependencies:
- dependency-name: sqlalchemy-bigquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pytest-cov from 2.11.1 to 3.0.0 (#787)

Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.11.1 to 3.0.0.
- [Release notes](https://github.com/pytest-dev/pytest-cov/releases)
- [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-cov/compare/v2.11.1...v3.0.0)

---
updated-dependencies:
- dependency-name: pytest-cov
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump apscheduler from 3.8.0 to 3.9.1 (#789)

Bumps [apscheduler](https://github.com/agronholm/apscheduler) from 3.8.0 to 3.9.1.
- [Release notes](https://github.com/agronholm/apscheduler/releases)
- [Changelog](https://github.com/agronholm/apscheduler/blob/3.9.1/docs/versionhistory.rst)
- [Commits](https://github.com/agronholm/apscheduler/compare/3.8.0...3.9.1)

---
updated-dependencies:
- dependency-name: apscheduler
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-pyyaml from 6.0.8 to 6.0.9 (#791)

Bumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.8 to 6.0.9.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pyyaml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [#681] add documentation on fideslog use (#751)

* [#681] add documentation on fideslog use

* changelog

* update fidesops use wording

* Fideslib models (#700)

* WIP

* WIP

* WIP

* WIP

* Use available exceptions from fideslib

* Fix failing tests

* Fix policy tests

* Remove debugging code

* Fix failing tests

* Fix failing tests

* Fix failiing tests

* Run black and isort

* Make pylint in docker happy

* Clean up migrations

* Move downgrade point of table renames

* Remove Dockerfile temp workaround and fix pylint errors

* Fix failing tests

* Remove jwt.py

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 660 Add support for multiple statuses to be selected for filtering subject requests  (#802)

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

* Resolve issue with MyPy seeing files in fidesops as missing imports (#719)


Co-authored-by: Paul Sanders <pau@ethyca.com>

* 249 saas connector zendesk ticket erasure (#775)

Co-authored-by: Hamza W <hamza@Hamzas-MacBook-Pro.local>
Co-authored-by: Adrian Galvan <adriang430@gmail.com>

* Fixing `check-migration` command (#806)

* Fix issue requiring separate install of snowflake-connector-python (#807)

* Fix issue requiring separate install of snowflake-connector-python

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 513 - [Admin UI] Update Subject Request status filter to be a multise… (#764)

Co-authored-by: Christopher Calhoun <chris@ethyca.com>

* 706 Adds SaaS connection type to SaaS yaml config (#748)

* Adds Saas type to saas yaml config

* To allow data migration that queries on a connectiontype enum to work, update previous schema migrations that used "alter type" to update connection type to rename the enum type, create a new enum with the new types, and then delete the old enum type.

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

* Make `worker` node optional (#770)

* default fidesops to running the worker and webserver on same container

* default to using a worker, add docker config for worker

* update changelog

* USE_DEDICATED_WORKER -> WORKER_ENABLED

* add basic descriptions for celery vars to docs

* remove unused import

* add Make command for a server + worker

* GET Available Connectors [#706] (#768)

* Adds Saas type to saas yaml config

* alter postman collection

* updates changelog

* lint fixes

* Add endpoint to surface all available connectors including database options and saas options.

* Exclude custom and manual types from list of available connectors.

- Add docs and postman collection.

* Update changelog.

* Remove committed ANALYTICS_ID.

* Import ClientDetail from fideslib instead of fidesops.

* Fix import order.

Co-authored-by: eastandwestwind <eastandwestwind@gmail.com>

* Endpoint: Return Secrets for a Connector Type [#753] (#795)

* Adds Saas type to saas yaml config

* alter postman collection

* updates changelog

* lint fixes

* Add endpoint to surface all available connectors including database options and saas options.

* Exclude custom and manual types from list of available connectors.

- Add docs and postman collection.

* Update changelog.

* Add an endpoint to fetch the types of secrets that should be supplied for a given connection type.

- Relocate "load_config" which we use to load saas config yamls, now that we have another use case beyond unit tests.

* Dynamically override the SaaSSchema docstring for a given saas connector type, so the description isn't abstract.

- Update changelog
- Add docs
- Add endpoint to postman collection

* Add missing import.

* Add a request method to docs.

* Update docstring.

* Remove committed ANALYTICS_ID.

* Import ClientDetail from fideslib instead of fidesops.

* Fix import order.

* Restore removed items in changelog.

Co-authored-by: eastandwestwind <eastandwestwind@gmail.com>

* Add fixture to clear tables between test (#680)

* Add fixture to clear tables between test runs

* Update CHANGELOG

* Add missing ordering to customer_details logs query in test.

* update import path

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Replace user authentication routes with fideslib routes (#811)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* test fixing publish_docs ci action (#818)

* test fixing publish_docs ci action

* update branch target to

* try another commit to see if publish_docs is triggered

* revert change to branch target

* allow publish docs on test branch (#819)

* allow publish docs on test branch

* remove main

* trying new empty commit

* reverting to main

* Temporarly disable paths

* Revert temporary test

* Add makefile

* pull latest

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Update Celery config defaults (#808)

* update celery config defaults, set to redis settings if none provided

* rename REDIS_CONNECTION_URL to CONNECTION_URL

* add var to config reference

* make DB index optional, use f'string

* adds support for redis USER

* add default for db_index in format string

* set correct default for USER

* Bump next-auth from 4.5.0 to 4.9.0 in /clients/privacy-center (#823)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.5.0 to 4.9.0.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.5.0...next-auth@v4.9.0)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fideslib to handle base64 encoded password (#820)

* Handle hashed password

* Update CHANGELOG

* Fix failing test and use UserPasswordReset schema from fideslib

* Restore fidesops.toml

Co-authored-by: Paul Sanders <pau@ethyca.com>

* update local deployment for serving the ui [#644] (#827)

* update local deployment for serving the ui

* changelog

* Skip Masking of Uvicorn Logs [#766] (#831)

* Update get_fides_log_record_factory to skip masking of all uvicorn logs.

* Update changelog.

* 832 - Create new user gives HTTP 422 Unprocessable Entity exception (#833)

* 832 - [User Management] Create new user gives HTTP 422 Unprocessable Entity exception

* Updated CHANGELOG.md file

* [Admin UI] Change Login Page Wording (#774)

* sign in to instead of sign into

Changing the wording of the admin ui page to improve grammatical accuracy.

* fides admin-ui wording changelog

* added link to pr

* fix typo on subject identities

* updated changelog

* typo

* Fix bug in client with no scopes (#830)

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: eastandwestwind <eastandwestwind@gmail.com>

* 599 clipboard icon (#838)

* replace clipboard icon

* temp fix for auth

* revert last change

* changelog

* adds concurrency to unsafe check jobs (#835)

* Reduce docker image size (#846)

* Reduce docker image size

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* [#743] Store provided identity data in application database (#834)

* adds identity fields to PrivacyRequest model

* store identity data inside database

* update changelog

* add identities in test data command

* store identities provided via the DRP creation endpoint

* black + isort

* store provided identity data in request creation from onetrust

* remove deprecated migration

* adds new provided identity table

* use new provided identity table

* add docstring, remove comment

* update DRP privacy request creation to use ProvidedIdentity model

* update identity creation in test data command

* use persisted identity in OneTrust

* update test to use persisted identity

* isort update

* use enums

* optionally receive a salt in hash_value cmd

* use a constant salt for provided identity hashing

* remove import

* use typehints

* update typedef

* use enum in dict

* Bump faker from 13.14.0 to 13.15.0 (#848)

Bumps [faker](https://github.com/joke2k/faker) from 13.14.0 to 13.15.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v13.14.0...v13.15.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump unidecode from 1.2.0 to 1.3.4 (#849)

Bumps [unidecode](https://github.com/kmike/text-unidecode) from 1.2.0 to 1.3.4.
- [Release notes](https://github.com/kmike/text-unidecode/releases)
- [Commits](https://github.com/kmike/text-unidecode/commits)

---
updated-dependencies:
- dependency-name: unidecode
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update fastapi-pagination[sqlalchemy] requirement (#852)

Updates the requirements on [fastapi-pagination[sqlalchemy]](https://github.com/uriyyo/fastapi-pagination) to permit the latest version.
- [Release notes](https://github.com/uriyyo/fastapi-pagination/releases)
- [Commits](https://github.com/uriyyo/fastapi-pagination/compare/0.8.3...0.9.3)

---
updated-dependencies:
- dependency-name: fastapi-pagination[sqlalchemy]
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump black from 22.3.0 to 22.6.0 (#855)

Bumps [black](https://github.com/psf/black) from 22.3.0 to 22.6.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.3.0...22.6.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updating SaaSType enum (#857)

* Serve UI from root (#720)

* Refactor user management UI code (#839)

* Refactor pages to use the same Layout

* Add UserForm.tsx

* Add Yup validation schema

* Integrate refactored form to New route

* Styling

* WIP refactor EditUserForm

* Fix EditUserForm typescript errors

* Refactor handleSubmit

* Small fixes to EditUserForm

* Make call to get scopes earlier so form will be populated on first render

* Update changelog

* Search for `PrivacyRequest` based on hashed identity [#765] (#847)

* adds identity fields to PrivacyRequest model

* store identity data inside database

* update changelog

* add identities in test data command

* store identities provided via the DRP creation endpoint

* black + isort

* store provided identity data in request creation from onetrust

* remove deprecated migration

* adds new provided identity table

* use new provided identity table

* add docstring, remove comment

* update DRP privacy request creation to use ProvidedIdentity model

* update identity creation in test data command

* use persisted identity in OneTrust

* update test to use persisted identity

* isort update

* use enums

* optionally receive a salt in hash_value cmd

* use a constant salt for provided identity hashing

* remove import

* use typehints

* update typedef

* use enum in dict

* test for exact match search

* added exact match search to request status api

* import order

* update CHANGELOG

* documentation reorganization and page standardization (#858)

* copy changes to fix docs outlining [#427] and standardize formatting [DOX-216]

* Fix `create_test_data` (#862)

* Fix create_test_data

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Bump pre-commit from 2.9.3 to 2.20.0 (#853)

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pre-commit/pre-commit/compare/v2.9.3...v2.20.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Return persisted identities in `get_request_status` view (#860)

* 671 - [Datastore Management] Include Icon with datastore connection (FE) (#845)

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

* Updated CHANGELOG.md file

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

1. Updated connector svg images
2. Added default style to Spinner component

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Deleted unnecessary file

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Updated MySQL and Outreach .svg files

* Renamed connectors folder to connector-logos

* Resolved misspelling error in import statement

* Code review feedback

* Code review feedback

* Populate dataset (#844)

* Correct test name for mypy in safe_pr_checks.yml (#875)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Adds `celery.toml` for loading custom Celery config [#821] (#865)

* adds option to configure EVENT_QUEUE_PREFIX for celery

* provide the option to specify a default queue name too

* update celery config to load in from its own config toml file

* updates changelog

* update value for event_queue_prefix

* test celery config overrides

* include config_path arg

* add type def

* add config path to execution settings

* correct values

* add celery configuration to docs (#872)

* update config reference

* additional documentation on celery configs [#755]

* add celery.toml reference

* define optional celery config and overrides

* reword config links

* link to lowercase settings

* v1.6.2 Release Checklist (#881)

* updates changelog

* update ERD

* use star

* splits IMAGE_NAME into COMPOSE_SERVICE_NAME for docker compose services as naming has diverged (#884)

* belated changelog push (#885)

* Move root-level docker files into docker/ subdir (#877)

* Move root-level docker files into docker/ subdir

* move all of the compose files

* remove root aux compose files, refactor away the no-db and worker compose files

* unify the app and worker dockerfiles into a single file that leverages build stages

* move python scripts into a subdir, fix paths in compose integration files

* fix the script tests

* use python sleep instead of system sleep, fix script paths

* remove the analytics_id that accicentally got committed

* updated changelog

* move the sample sql data to a subdir of docker/ so it can be mounted

* update the teardown command and fix the integration files

* fix more path typos

* more desperate tweaks

* fix mysql/mariadb/mongo tests

* added an additional build step if mssql not there, all tests passing

* fix an accidental lowercasing

* Apply suggestions from code review

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Update CHANGELOG.md

Co-authored-by: Paul Sanders <psanders1@gmail.com>

Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>

* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern (#873)

* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern

* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern

Refactored the Datastore Management column border layout design to match Figma.

* Updated CHANGELOG.md file

* 45 - [Datastore Management] Keep datastore cards in 1/3 screen pattern

* 745 - [Datastore Management] Keep datastore cards in 1/3 screen pattern

* Fixed import statement

* 793 upgrade password hash (#876)

* Update config.py

* Update crypto imports to fideslib and remove legacy file and tests

* Update Identity salt

* checkpoint, lowercasing all of the things

* update more config values to lowercase

* fix linting errors, fix config validators

* update the configs to use lowercase keys

* lowercase the allowed keys

* Update create_test_data.py

* bump fideslib version

* Lowercase config variables

* Fix linting issues

* Fix some test failures

* Fix application fixtures

* Remove old celery config options

* Sort script imports

* Update changelog

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Feat: Vault for secrets (#869)

* Replace config/gh secrets with secrets from vault

* sorting and update changelog

* fix path for importing test helpers

* more sorting

* run ci again with empty commit

* update unsafe_pr_checks

* allow for no vault vars for unit tests, adds back back for saas config toml vars

* check for client

* sort

* init client to None

* fix imports

* fix pytest markers to better indicate which tests rely on actual secrets

* Bump pandas from 1.3.3 to 1.4.3 (#896)

Bumps [pandas](https://github.com/pandas-dev/pandas) from 1.3.3 to 1.4.3.
- [Release notes](https://github.com/pandas-dev/pandas/releases)
- [Changelog](https://github.com/pandas-dev/pandas/blob/main/RELEASE.md)
- [Commits](https://github.com/pandas-dev/pandas/compare/v1.3.3...v1.4.3)

---
updated-dependencies:
- dependency-name: pandas
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add dependabot label to dependabot PRs (#898)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* [SaaS Connector] Salesforce (erasure) (#888)

* 252 saas connector sendgrid (#883)

* initial sendgrid saas connector integration. access only, contacts only

* erasure (update) support for sendgrid

* remove unused imports

* update sendgrid test fixture to expect 404 response status code because of ignore_errors enhancement

* Fixing import order and cleaning up the retry logic for consistency

* add sendgrid env var support to makefile and unsafe_pr_checks config. remove DELETE endpoint per PR comments

* Added delete endpoint for contacts

* Fixing data_path for contacts endpoint

* Reverting search query to improve performance and avoid server timeouts

* Updated delete endpoint request, used request instead of SaaSRequest in tests

* updated imports after check suggestion

* Updated code after review

* Removed unused variables, imports

* Restoring Makefile

* Fixed import cryptographic_util error

* Misc fixes

* Updated Changelog file

* Updated Changelog for unreleased section and pulled main

* Updated Changelog and added Sendgrid in unreleased section

* Updated Changelog and added Sendgrid in unreleased section with link

* Updated Changelog and added Sendgrid in added section after Adam's suggestion

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>
Co-authored-by: Adam Sachs <adam@Adams-MBP.attlocal.net>
Co-authored-by: Adrian Galvan <adriang430@gmail.com>
Co-authored-by: Hamza W <hamza@Hamzas-MacBook-Pro.local>
Co-authored-by: Adrian Galvan <adrian@ethyca.com>

* 747 - Users should be able to click on the full field of a dropdown-type filter to open up the dropdown (#903)

* 747 - Users should be able to click on the full field of a dropdown-type filter to open up the dropdown

* Disabled eslint import/extensions rule for certain files with an alias path in the import statement

* Updated CHANGELOG.md file

* Resolved jest unit tests from failing

* Removed eslint-disable import/extensions declarations

* Bump types-redis from 4.3.2 to 4.3.4 (#895)

Bumps [types-redis](https://github.com/python/typeshed) from 4.3.2 to 4.3.4.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-redis
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Adds saas config base info to connection config responses (#904)

* adds saas config base info to connection config responses

* update changelog

* add properties to base class

* use diff naming to avoid recursion

* update saas_config prop for unit tests

* another occurance of saas config in unit test

* use pydantic alias for type field

* sort

* revert change to saas type naming

* unused import

* init parent class so that props are avail on self

* revert to using props

* Adding privacy_request_id placeholder (#911)

* Bump mypy from 0.961 to 0.971 (#914)

Bumps [mypy](https://github.com/python/mypy) from 0.961 to 0.971.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.961...v0.971)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pydash from 5.0.2 to 5.1.0 (#920)

Bumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0.
- [Release notes](https://github.com/dgilland/pydash/releases)
- [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/dgilland/pydash/compare/v5.0.2...v5.1.0)

---
updated-dependencies:
- dependency-name: pydash
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update boto3 requirement from ~=1.18.14 to ~=1.24.34 (#917)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.18.14...1.24.34)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.6.1 to 2022.7.0 (#915)

Bumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.6.1...2022.7.0)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix integration set up scripts for postgres and mariadb - casing has changed for config variables. (#921)

* zendesk and salesforce connection docs (#908)

* Adobe Campaign access and erasure (#905)

* Updated tutorial to match latest fidesdemo (#772)

* Correct build arg variable name (#925)

* Correct build arg variable name

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Default `FIDESOPS__ADMIN_UI__ENABLED` to `True` (#936)

* serve AdminUI by default

* updates changelog

* Update python docker base image from slim-buster to slim-bullseye (#928)

* Update python docker base image from slim-buster to slim-bullseye

* Update CHANGELOG

* Remove ipython from dev-requirements.txt

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Update boto3 requirement from ~=1.24.34 to ~=1.24.36 (#939)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.34...1.24.36)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 13.15.0 to 13.15.1 (#941)

Bumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v13.15.0...v13.15.1)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-ujson from 5.2.0 to 5.4.0 (#947)

Bumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-ujson
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add db vs saas to connection type api (#937)

* add db vs saas to connection type api

* adds changelog line

* mypy

* fix test

* format

* more tests

* formatting

* adds system type query param

* format

* adjust test saas search

* correct query param

* Use Nox as the build tool instead of Make (#919)

* Create noxfile.py

* update the dockerfile with more stages

* add GitPython as a dev requirement (used by nox builds)

* add the noxfiles, all docker builds work

* fix the "make compose-build" command

* remove the worker docker stage due to it being redundant with prod

* create a unified compose file for integrations

* update the password prompt to be accurate

* copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python

* add create_user and seed_test_data to the nox utils

* docs commands work

* get the generic dev command working

* add db commands to utils

* clean up mypy configuration in pyproject.toml and remove config from setup.cfg

* simplify some of the CI targets and start updating the pytest targets

* cleanup run_infrastructure

* update ci_suite and other CI nox targets

* add nox as a dev-requirement

* get dev commands working, sans quickstart

* tweak to the compose_down constant

* get the new pylint target passing

* remove analytics_id

* updated the changelog

* Apply suggestions from code review

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* fix the xenon command

* add a few tweaks to the worker dev command

* fix an import issue

* update dockerignore

* move mssql to the bottom of the datastore list

Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>

* 863 - Retry a DSR (FE) (#938)

* Update docs docker base image from slim-buster to slim-bullseye (#949)

* Changed Debian base image

* Update the python version to be the same as used in the fidesops app image

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Updated changelog

Co-authored-by: Dave Quinlan <dave@ethyca.com>
Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Experimenting with fixes for the failing MSSQL CI tests (#918)

* Change docker password environment variable to MSSQL_SA_PASSWORD

* Revert password environment variable name

* Set MSSQL user to root in docker-compose

* Revert setting root user in docker-compose

* Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION

* Add healthcheck to mssql compose file

* Modify healthcheck command

* Revert healthcheck

* Try for more loging information

* Another try for logging

* Another try for logging

* Try running only mssql tests to avoid timeout

* Revert mssql only flag

* Extend time out to try to get logs

* Revert extra logging and extended timeout

* Set network mode to host

* Make mssql run on its own

* Remove network from docker-compose and only run mssql in integration

* Increase integration test logging

* Revert mssql only

* Use cache for docker

* Fix workflow error

* Fix workflow error

* Fix target

* Fix make traget

* Fix make traget

* Revert cache

* Verify that mssql is running from pytest fixture

* Add restart to mssql

* Revert rester in docker-compose

* Revert wait for mssql in pytest fixture

* Wait for mssql to be ready before adding test data

* Add count of retries

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Delete custom GitHub issue templates (#955)

We've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE

Removing the templates from this repo allows it to automatically pull in the organization templates for consistency.

* Bump sqlalchemy-redshift from 0.8.8 to 0.8.10 (#940)

Bumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.8 to 0.8.10.
- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)
- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)
- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.8...0.8.10)

---
updated-dependencies:
- dependency-name: sqlalchemy-redshift
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Move tests into an "ops" subdir (#935)

* Create __init__.py

* move all of the test files down one dir into `ops`

* update test paths

* update paths where needed

* update the changelog

* Dispatch a repository event on new published releases (#945)

* Add release dispatch event action

* Update `CHANGELOG.md`

* Reduce # of clients connected to the application db [#810] (#944)

* Reduce number of open connections:

- Limit task concurrency to two per worker.
- Create one Engine per celery process which opens up a connection pool.  Create one Session per celery process and use that session across privacy requests.
- Close the session after the privacy request has finished executing.  This just resets the session and returns connections back to the pool. It can be reused.
- Remove unnecessary places where session is closed manually because the session is being used as a context manager and is already closed through that.
- Pass the same Session that the privacy request is using through to TaskResources to be re-used to create ExecutionLogs instead of opening up a new Session.
- Don't close the session when passing it into the Execution Log, wait until the entire privacy request is complete/exited.

* Define "self" for run_privacy_task - it's the task itself.

For mypy's benefits, define that the session is a context manager.

* Make a session non-optional for graph_task.run_access_request, graph_task.run_erasure, and for instantiating taskResources

* Use missing db fixture.

* Add missing db resource.

* Update test to reflect new behavior that disabling a datasource while a request is in progress can cause related collections to be skipped once the current session is expired and the connection config has the most recent state.

Because the same Session that is being used to run the PrivacyRequest is now being used for ExecutionLogs, the process of saving an ExecutionLog runs a session.commit() which expires the Session and causes the ConnectionConfig to have the most recent state the next time it is accessed.

* Update CHANGELOG.

* enable worker by default in our dockerfile (#958)

* add extra steps to make clean (#767)

* Push `dev` image on pushes to `main` (#956)

* Update publish_to_dockerhub.yml

* add a dev step and use nox

* update the changelog

* Move Client Code into an `ops` subdir (#964)

* Move Client Code into an `ops` subdir

* move all of the files

* update the dockerfile

* update package.json

* update codepaths for workflow tests

* Update the changelog and docs references

* Update .github/dependabot.yaml

* Bump gitpython from 3.1 to 3.1.27 (#971)

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1 to 3.1.27.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.0...3.1.27)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix the `nox -s dev` command not spinning up the webserver (#959)

* Update dev_nox.py

* spin up the app before running a container shell

* [#927, #929, #930] sendgrid, adobe, outreach connector docs (#951)

* Bump sqlalchemy-utils from 0.37.8 to 0.38.3 (#968)

Bumps [sqlalchemy-utils](https://github.com/kvesteri/sqlalchemy-utils) from 0.37.8 to 0.38.3.
- [Release notes](https://github.com/kvesteri/sqlalchemy-utils/releases)
- [Changelog](https://github.com/kvesteri/sqlalchemy-utils/blob/master/CHANGES.rst)
- [Commits](https://github.com/kvesteri/sqlalchemy-utils/compare/0.37.8...0.38.3)

---
updated-dependencies:
- dependency-name: sqlalchemy-utils
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pyodbc from 4.0.32 to 4.0.34 (#980)

Bumps [pyodbc](https://github.com/mkleehammer/pyodbc) from 4.0.32 to 4.0.34.
- [Release notes](https://github.com/mkleehammer/pyodbc/releases)
- [Commits](https://github.com/mkleehammer/pyodbc/compare/4.0.32...4.0.34)

---
updated-dependencies:
- dependency-name: pyodbc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.7.0 to 2022.7.1 (#967)

Bumps [dask](https://github.com/dask/dask) from 2022.7.0 to 2022.7.1.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.7.0...2022.7.1)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump alembic from 1.8.0 to 1.8.1 (#989)

Bumps [alembic](https://github.com/sqlalchemy/alembic) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/sqlalchemy/alembic/releases)
- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/alembic/commits)

---
updated-dependencies:
- dependency-name: alembic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix erroneous values in the Outreach config (#988)

* Add documentation for new nox commands (#981)

* Add documentation for new nox commands

* changelog

* missed make to nox edits

* review edits

* update `nox -s dev` to not open a shell, add it as a posarg option

* Apply suggestions from code review

* remove the analytics id

* update typo

* remove extra numbering

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Bump types-toml from 0.10.7 to 0.10.8 (#998)

Bumps [types-toml](https://github.com/python/typeshed) from 0.10.7 to 0.10.8.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-toml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fideslog from 1.2.1 to 1.2.2 (#996)

Bumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/ethyca/fideslog/releases)
- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: fideslog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Added Choose your connection feature (#987)

* Added Choose your connection feature

760-Add a Connection - Select a connector to configure (front end)
866-Add a Connection - Front End layout structure

* Updated CHANGELOG.md file

* Fix lint issues

* Fix build issue

* Hide overflow

Co-authored-by: Andrew Jackson <andrew.c.j1995@gmail.com>

* saas request overrides (#986)

* initial cut of saas request overrides. include mailchimp as an example and test case. minor refactor of some of the saas request execution to enable smoother override

* fix rebase issue by moving saas override tests into ops subdir

* import path updates to resolve conflicts caused by rebase

* add session parameter into graph task calls to fix saas override integration tests caused by rebase

* update changelog

* tweaks to saas connector overrides and associated tests

* expose override factory register as module variable for clenaer decorator calls

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>

* Update OAuth strategy to be able to perform local testing (#962)

* Update boto3 requirement from ~=1.24.36 to ~=1.24.42 (#1001)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.36...1.24.42)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi[all] from 0.78.0 to 0.79.0 (#1002)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.78.0 to 0.79.0.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.78.0...0.79.0)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update config reference to use lowercase values (#952)

* [#909] update config reference to use lowercase values

* changelog

* missed uppercase variables

* update usages of False to false in connector docs

* Added Auth0 Connector (#991)

* add pagination back to connection types endpoints (#1019)

* add pagination back to connection types endpoints

* check for pagination in system_type search

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Subject Request Events and Logs Section (#1018)

* Fix small issue with eslint config

* Get initial drawer working

* Add jest config to eslintignore

* Fix small css issue

* Refactor components and get initial functionality

* Fix small logic error

* Fix couple of bugs and format code

* Conditinally display error tag

* Fix issues with merge

* Format and lint

* Sort imports

* Update to new solution

* Format code

* Update changelog

Co-authored-by: Sean Preston <sean@ethyca.com>

* First draft of OAuth documentation (#963)

* Send Errored Requests / Reprocessed Requests Info to FidesLog [#754] (#993)

* Add a method to format a representation of the graph for caching in Redis and a separate method to build a summary of the differences in the graph when a privacy request is rerun to fideslog.

- Adds FieldAddress.from_string method.

* Add methods to cache a representation of the access graph when it is built and a separate method to retrieve it from the cache.

- Also add a method to build a "rerun_access_graph" AnalyticsEvent for fideslog where applicable.

* When running the access portion of the privacy request, log stats about a rerun and then cache the current access graph.

- Give a different prefix when we're caching the access graph to not get mixed up with access request results.

* Show skipped new edges that are directly upstream of completed nodes instead. These edges are intentionally dropped from the new graph on rerun, so want to surface this count.

* Log if a privacy request fails during the "erasure" step of privacy request execution.  Even though the access step is not rerun here, compare the previously cached access graph with the access graph that would have been run to determine what data has changed.

* Add missing session variables.

* Send an event to Fideslog when privacy request execution fails.

* Add missed session - bad merge.

* Update changelog.

* Update docstrings.

* Remove copy/paste comment.

* Respond to CR comments.

* Currently AnalyticsEvent.local_host cannot be None.

* Update the compose file and workflows to expect an already-built image (#966)

* Update the compose file and workflows to be in line with fidesctl

* update the compose file to look for a specific image

* rename files and update the safe PR checks

* add check_migrations to the ci checks and nox

* fix the failing PR checks

* fix CI failures

* update the unsafe checks workflow

* update run_infrastructure to use the compose service name

* remove the makefile and the old run_infra script

* bump pylint version, pin isort, fix issues

* update pytest setup path

* update the changelog

* make OPS_TEST_DIR a constant

* fix nox missing vars

* specify that the nox imports are relative imports

* remove relative import paths

* run isort

Co-authored-by: Sean Preston <sean@ethyca.com>

* Create `AuditLog` on privacy request approval (#1038)

* Create approval audit logs

* Add tests

* disable a pylint lint

* Update changelog

* Update privacy_request fixture

* Updating Salesforce to use OAuth2 authentication code flow (#1039)

* Removing saas_config.toml (#1043)

* Bump types-pyyaml from 6.0.9 to 6.0.11 (#1047)

Bumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.9 to 6.0.11.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pyyaml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump docker/build-push-action from 2 to 3 (#1044)

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.7.1 to 2022.8.0 (#1046)

Bumps [dask](https://github.com/dask/dask) from 2022.7.1 to 2022.8.0.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.7.1...2022.8.0)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update boto3 requirement from ~=1.24.42 to ~=1.24.46 (#1045)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [C…
ThomasLaPiana added a commit that referenced this pull request Oct 6, 2022
* Bump pydash from 5.0.2 to 5.1.0 (#920)

Bumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0.
- [Release notes](https://github.com/dgilland/pydash/releases)
- [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/dgilland/pydash/compare/v5.0.2...v5.1.0)

---
updated-dependencies:
- dependency-name: pydash
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update boto3 requirement from ~=1.18.14 to ~=1.24.34 (#917)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.18.14...1.24.34)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.6.1 to 2022.7.0 (#915)

Bumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.6.1...2022.7.0)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix integration set up scripts for postgres and mariadb - casing has changed for config variables. (#921)

* zendesk and salesforce connection docs (#908)

* Adobe Campaign access and erasure (#905)

* Updated tutorial to match latest fidesdemo (#772)

* Correct build arg variable name (#925)

* Correct build arg variable name

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Default `FIDESOPS__ADMIN_UI__ENABLED` to `True` (#936)

* serve AdminUI by default

* updates changelog

* Update python docker base image from slim-buster to slim-bullseye (#928)

* Update python docker base image from slim-buster to slim-bullseye

* Update CHANGELOG

* Remove ipython from dev-requirements.txt

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Update boto3 requirement from ~=1.24.34 to ~=1.24.36 (#939)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.34...1.24.36)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 13.15.0 to 13.15.1 (#941)

Bumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v13.15.0...v13.15.1)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-ujson from 5.2.0 to 5.4.0 (#947)

Bumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-ujson
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add db vs saas to connection type api (#937)

* add db vs saas to connection type api

* adds changelog line

* mypy

* fix test

* format

* more tests

* formatting

* adds system type query param

* format

* adjust test saas search

* correct query param

* Use Nox as the build tool instead of Make (#919)

* Create noxfile.py

* update the dockerfile with more stages

* add GitPython as a dev requirement (used by nox builds)

* add the noxfiles, all docker builds work

* fix the "make compose-build" command

* remove the worker docker stage due to it being redundant with prod

* create a unified compose file for integrations

* update the password prompt to be accurate

* copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python

* add create_user and seed_test_data to the nox utils

* docs commands work

* get the generic dev command working

* add db commands to utils

* clean up mypy configuration in pyproject.toml and remove config from setup.cfg

* simplify some of the CI targets and start updating the pytest targets

* cleanup run_infrastructure

* update ci_suite and other CI nox targets

* add nox as a dev-requirement

* get dev commands working, sans quickstart

* tweak to the compose_down constant

* get the new pylint target passing

* remove analytics_id

* updated the changelog

* Apply suggestions from code review

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* fix the xenon command

* add a few tweaks to the worker dev command

* fix an import issue

* update dockerignore

* move mssql to the bottom of the datastore list

Co-authored-by: Paul Sanders <psanders1@gmail.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>

* 863 - Retry a DSR (FE) (#938)

* Update docs docker base image from slim-buster to slim-bullseye (#949)

* Changed Debian base image

* Update the python version to be the same as used in the fidesops app image

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Updated changelog

Co-authored-by: Dave Quinlan <dave@ethyca.com>
Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Experimenting with fixes for the failing MSSQL CI tests (#918)

* Change docker password environment variable to MSSQL_SA_PASSWORD

* Revert password environment variable name

* Set MSSQL user to root in docker-compose

* Revert setting root user in docker-compose

* Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION

* Add healthcheck to mssql compose file

* Modify healthcheck command

* Revert healthcheck

* Try for more loging information

* Another try for logging

* Another try for logging

* Try running only mssql tests to avoid timeout

* Revert mssql only flag

* Extend time out to try to get logs

* Revert extra logging and extended timeout

* Set network mode to host

* Make mssql run on its own

* Remove network from docker-compose and only run mssql in integration

* Increase integration test logging

* Revert mssql only

* Use cache for docker

* Fix workflow error

* Fix workflow error

* Fix target

* Fix make traget

* Fix make traget

* Revert cache

* Verify that mssql is running from pytest fixture

* Add restart to mssql

* Revert rester in docker-compose

* Revert wait for mssql in pytest fixture

* Wait for mssql to be ready before adding test data

* Add count of retries

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Delete custom GitHub issue templates (#955)

We've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE

Removing the templates from this repo allows it to automatically pull in the organization templates for consistency.

* Bump sqlalchemy-redshift from 0.8.8 to 0.8.10 (#940)

Bumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.8 to 0.8.10.
- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)
- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)
- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.8...0.8.10)

---
updated-dependencies:
- dependency-name: sqlalchemy-redshift
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Move tests into an "ops" subdir (#935)

* Create __init__.py

* move all of the test files down one dir into `ops`

* update test paths

* update paths where needed

* update the changelog

* Dispatch a repository event on new published releases (#945)

* Add release dispatch event action

* Update `CHANGELOG.md`

* Reduce # of clients connected to the application db [#810] (#944)

* Reduce number of open connections:

- Limit task concurrency to two per worker.
- Create one Engine per celery process which opens up a connection pool.  Create one Session per celery process and use that session across privacy requests.
- Close the session after the privacy request has finished executing.  This just resets the session and returns connections back to the pool. It can be reused.
- Remove unnecessary places where session is closed manually because the session is being used as a context manager and is already closed through that.
- Pass the same Session that the privacy request is using through to TaskResources to be re-used to create ExecutionLogs instead of opening up a new Session.
- Don't close the session when passing it into the Execution Log, wait until the entire privacy request is complete/exited.

* Define "self" for run_privacy_task - it's the task itself.

For mypy's benefits, define that the session is a context manager.

* Make a session non-optional for graph_task.run_access_request, graph_task.run_erasure, and for instantiating taskResources

* Use missing db fixture.

* Add missing db resource.

* Update test to reflect new behavior that disabling a datasource while a request is in progress can cause related collections to be skipped once the current session is expired and the connection config has the most recent state.

Because the same Session that is being used to run the PrivacyRequest is now being used for ExecutionLogs, the process of saving an ExecutionLog runs a session.commit() which expires the Session and causes the ConnectionConfig to have the most recent state the next time it is accessed.

* Update CHANGELOG.

* enable worker by default in our dockerfile (#958)

* add extra steps to make clean (#767)

* Push `dev` image on pushes to `main` (#956)

* Update publish_to_dockerhub.yml

* add a dev step and use nox

* update the changelog

* Move Client Code into an `ops` subdir (#964)

* Move Client Code into an `ops` subdir

* move all of the files

* update the dockerfile

* update package.json

* update codepaths for workflow tests

* Update the changelog and docs references

* Update .github/dependabot.yaml

* Bump gitpython from 3.1 to 3.1.27 (#971)

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1 to 3.1.27.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.0...3.1.27)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix the `nox -s dev` command not spinning up the webserver (#959)

* Update dev_nox.py

* spin up the app before running a container shell

* [#927, #929, #930] sendgrid, adobe, outreach connector docs (#951)

* Bump sqlalchemy-utils from 0.37.8 to 0.38.3 (#968)

Bumps [sqlalchemy-utils](https://github.com/kvesteri/sqlalchemy-utils) from 0.37.8 to 0.38.3.
- [Release notes](https://github.com/kvesteri/sqlalchemy-utils/releases)
- [Changelog](https://github.com/kvesteri/sqlalchemy-utils/blob/master/CHANGES.rst)
- [Commits](https://github.com/kvesteri/sqlalchemy-utils/compare/0.37.8...0.38.3)

---
updated-dependencies:
- dependency-name: sqlalchemy-utils
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump pyodbc from 4.0.32 to 4.0.34 (#980)

Bumps [pyodbc](https://github.com/mkleehammer/pyodbc) from 4.0.32 to 4.0.34.
- [Release notes](https://github.com/mkleehammer/pyodbc/releases)
- [Commits](https://github.com/mkleehammer/pyodbc/compare/4.0.32...4.0.34)

---
updated-dependencies:
- dependency-name: pyodbc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.7.0 to 2022.7.1 (#967)

Bumps [dask](https://github.com/dask/dask) from 2022.7.0 to 2022.7.1.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.7.0...2022.7.1)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump alembic from 1.8.0 to 1.8.1 (#989)

Bumps [alembic](https://github.com/sqlalchemy/alembic) from 1.8.0 to 1.8.1.
- [Release notes](https://github.com/sqlalchemy/alembic/releases)
- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/alembic/commits)

---
updated-dependencies:
- dependency-name: alembic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix erroneous values in the Outreach config (#988)

* Add documentation for new nox commands (#981)

* Add documentation for new nox commands

* changelog

* missed make to nox edits

* review edits

* update `nox -s dev` to not open a shell, add it as a posarg option

* Apply suggestions from code review

* remove the analytics id

* update typo

* remove extra numbering

Co-authored-by: Thomas <thomas.lapiana+github@pm.me>

* Bump types-toml from 0.10.7 to 0.10.8 (#998)

Bumps [types-toml](https://github.com/python/typeshed) from 0.10.7 to 0.10.8.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-toml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fideslog from 1.2.1 to 1.2.2 (#996)

Bumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/ethyca/fideslog/releases)
- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: fideslog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Added Choose your connection feature (#987)

* Added Choose your connection feature

760-Add a Connection - Select a connector to configure (front end)
866-Add a Connection - Front End layout structure

* Updated CHANGELOG.md file

* Fix lint issues

* Fix build issue

* Hide overflow

Co-authored-by: Andrew Jackson <andrew.c.j1995@gmail.com>

* saas request overrides (#986)

* initial cut of saas request overrides. include mailchimp as an example and test case. minor refactor of some of the saas request execution to enable smoother override

* fix rebase issue by moving saas override tests into ops subdir

* import path updates to resolve conflicts caused by rebase

* add session parameter into graph task calls to fix saas override integration tests caused by rebase

* update changelog

* tweaks to saas connector overrides and associated tests

* expose override factory register as module variable for clenaer decorator calls

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>

* Update OAuth strategy to be able to perform local testing (#962)

* Update boto3 requirement from ~=1.24.36 to ~=1.24.42 (#1001)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.36...1.24.42)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi[all] from 0.78.0 to 0.79.0 (#1002)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.78.0 to 0.79.0.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.78.0...0.79.0)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update config reference to use lowercase values (#952)

* [#909] update config reference to use lowercase values

* changelog

* missed uppercase variables

* update usages of False to false in connector docs

* Added Auth0 Connector (#991)

* add pagination back to connection types endpoints (#1019)

* add pagination back to connection types endpoints

* check for pagination in system_type search

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Subject Request Events and Logs Section (#1018)

* Fix small issue with eslint config

* Get initial drawer working

* Add jest config to eslintignore

* Fix small css issue

* Refactor components and get initial functionality

* Fix small logic error

* Fix couple of bugs and format code

* Conditinally display error tag

* Fix issues with merge

* Format and lint

* Sort imports

* Update to new solution

* Format code

* Update changelog

Co-authored-by: Sean Preston <sean@ethyca.com>

* First draft of OAuth documentation (#963)

* Send Errored Requests / Reprocessed Requests Info to FidesLog [#754] (#993)

* Add a method to format a representation of the graph for caching in Redis and a separate method to build a summary of the differences in the graph when a privacy request is rerun to fideslog.

- Adds FieldAddress.from_string method.

* Add methods to cache a representation of the access graph when it is built and a separate method to retrieve it from the cache.

- Also add a method to build a "rerun_access_graph" AnalyticsEvent for fideslog where applicable.

* When running the access portion of the privacy request, log stats about a rerun and then cache the current access graph.

- Give a different prefix when we're caching the access graph to not get mixed up with access request results.

* Show skipped new edges that are directly upstream of completed nodes instead. These edges are intentionally dropped from the new graph on rerun, so want to surface this count.

* Log if a privacy request fails during the "erasure" step of privacy request execution.  Even though the access step is not rerun here, compare the previously cached access graph with the access graph that would have been run to determine what data has changed.

* Add missing session variables.

* Send an event to Fideslog when privacy request execution fails.

* Add missed session - bad merge.

* Update changelog.

* Update docstrings.

* Remove copy/paste comment.

* Respond to CR comments.

* Currently AnalyticsEvent.local_host cannot be None.

* Update the compose file and workflows to expect an already-built image (#966)

* Update the compose file and workflows to be in line with fidesctl

* update the compose file to look for a specific image

* rename files and update the safe PR checks

* add check_migrations to the ci checks and nox

* fix the failing PR checks

* fix CI failures

* update the unsafe checks workflow

* update run_infrastructure to use the compose service name

* remove the makefile and the old run_infra script

* bump pylint version, pin isort, fix issues

* update pytest setup path

* update the changelog

* make OPS_TEST_DIR a constant

* fix nox missing vars

* specify that the nox imports are relative imports

* remove relative import paths

* run isort

Co-authored-by: Sean Preston <sean@ethyca.com>

* Create `AuditLog` on privacy request approval (#1038)

* Create approval audit logs

* Add tests

* disable a pylint lint

* Update changelog

* Update privacy_request fixture

* Updating Salesforce to use OAuth2 authentication code flow (#1039)

* Removing saas_config.toml (#1043)

* Bump types-pyyaml from 6.0.9 to 6.0.11 (#1047)

Bumps [types-pyyaml](https://github.com/python/typeshed) from 6.0.9 to 6.0.11.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pyyaml
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump docker/build-push-action from 2 to 3 (#1044)

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump dask from 2022.7.1 to 2022.8.0 (#1046)

Bumps [dask](https://github.com/dask/dask) from 2022.7.1 to 2022.8.0.
- [Release notes](https://github.com/dask/dask/releases)
- [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md)
- [Commits](https://github.com/dask/dask/compare/2022.7.1...2022.8.0)

---
updated-dependencies:
- dependency-name: dask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update boto3 requirement from ~=1.24.42 to ~=1.24.46 (#1045)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.42...1.24.46)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Refactor static checks to run outside of Docker (#1053)

* Refactor static checks to run outside of Docker

* run static checks outside of docker, reusing cached virtual envs

* update the changelog

* Create CHANGELOG.md

* Update CHANGELOG.md

* update the static checks in the workflows

* remove the static check dependency on "build"

* Create finished audit log for privacy requests (#1040)

* Add finished AuditLog

* Bump fideslib

* Update test

* Fix format lints

* Lowercase encoding

* Update changelog

* Fix issue with migration downgrade

* Sort lists in test data tests

* format file

* Update quickstart to use docker-compose and docker network for all commands (#1056)

* Bump snowflake-sqlalchemy from 1.3.2 to 1.3.4 (#1051)

Bumps [snowflake-sqlalchemy](https://github.com/snowflakedb/snowflake-sqlalchemy) from 1.3.2 to 1.3.4.
- [Release notes](https://github.com/snowflakedb/snowflake-sqlalchemy/releases)
- [Commits](https://github.com/snowflakedb/snowflake-sqlalchemy/commits)

---
updated-dependencies:
- dependency-name: snowflake-sqlalchemy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* updates changelog (#1063)

* Reorganize application code (#1058)

* Reorganize application code

* first round of moving everything and updating the import paths

* checkpoint, more path updates

* more path updates

* fix imports

* fix isort

* fix mypy, isort and setup.py issues

* fix unsafe checks build step running on any label

* update the changelog

* delete temp files

* Update docs/fidesops/docs/development/contributing_details.md

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* update more file references

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* Patch versioneer to allow editable installs (#1070)

* Patch versioneer to allow editable installs

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Preserving headers in SaaSRequestParams during pagination (#1069)

* Add setuptools to dev-requirements to fix versioneer error (#1072)

* Add setuptools to dev-requirements to fix versioneer error and revert patch

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump fideslang to 1.1.0 (#890)

* Bump fideslang to 1.1.0

* find/replace user.provided.identifiable -> user

* remove derived mentions

* don't remove volumes on teardown

* update address fields

* replace user.derived and user.provided -> user

* fix two more tests

* fix pylint errors

* fix integration tests

* fix failing mongo tasks

* update the changelog

* fix the failing mongo task test

* another mongo task fix

* more mongo task fixes

* Revert test back to two addresses being masked.

* Update mongo array access test to reflect that underlying dataset has changed, and policy has changed, so more fields are returned.

* add the noxfiles

* update the dockerfile and get the nox docker commands working

* Revert "update the dockerfile and get the nox docker commands working"

This reverts commit 4b98c62163d419996977b7bb2dd17d181aac2f07.

* remove noxfiles

* updates from comments

* Update test

* Add migration

* Update categories in test config files

* Fix data categories

* Fix more data categories

* Change user.provided.nonidentifiable to user

* Update migraiton with review suggestions

* Run black

* Add more logging to migration

* Increment counter

* fix migration conflict

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>
Co-authored-by: Paul Sanders <pau@ethyca.com>

* 797 datadog (#1060)

* 515 SaaS connector Logi ID (#1074)

* Combine Execution and Audit Logs in Request Status Endpoint [#1024] (#1068)

* Update the request status endpoint, so when the verbose query param is used and execution logs are embedded, also return audit logs.

Execution Logs are created at the collection level while audit logs are for the overall privacy request level, so most fields returned for audit logs are None.
Logs are also grouped at the dataset level here, so give the audit logs a fake dataset name for display purposes, for example, "Request approved".

* Update CHANGELOG and update docs to reflect that audit logs are included in a verbose request status response.

* Notify fidesdemo on new releases (#1075)

* Notify fidesdemo on new releases

* Update `CHANGELOG.md`

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump pytest from 6.2.2 to 7.1.2 (#1081)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.2 to 7.1.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/6.2.2...7.1.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 13.15.1 to 14.0.0 (#1080)

Bumps [faker](https://github.com/joke2k/faker) from 13.15.1 to 14.0.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v13.15.1...v14.0.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/setup-python from 3 to 4 (#1078)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 4.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fideslog from 1.2.2 to 1.2.3 (#1079)

Bumps [fideslog](https://github.com/ethyca/fideslog) from 1.2.2 to 1.2.3.
- [Release notes](https://github.com/ethyca/fideslog/releases)
- [Commits](https://github.com/ethyca/fideslog/compare/v1.2.2...v1.2.3)

---
updated-dependencies:
- dependency-name: fideslog
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-urllib3 from 1.26.15 to 1.26.22 (#1084)

Bumps [types-urllib3](https://github.com/python/typeshed) from 1.26.15 to 1.26.22.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-urllib3
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fideslang from 1.1.0 to 1.2.0 (#1085)

Bumps [fideslang](https://github.com/ethyca/fideslang) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/ethyca/fideslang/releases)
- [Changelog](https://github.com/ethyca/fideslang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ethyca/fideslang/compare/1.1.0...1.2.0)

---
updated-dependencies:
- dependency-name: fideslang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump next-auth from 4.9.0 to 4.10.3 in /clients/ops/privacy-center (#1014)

Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.9.0 to 4.10.3.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/next-auth@v4.9.0...next-auth@v4.10.3)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* Bump fideslib (#1092)

* Bump fideslib to fix docs auth issue

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 983 Adds infra for email config and dispatch (#1059)

* Email config mvp crud / db layer

* adds name to email config model

* remove unintended changes

* gets POC working

* removes org name config var, updates crud endpoints to better handle supporting only 1 config, formatting

* updates postman collection, adds to changelog, bumps downrev on migration

* sort

* formatting

* use correct response model

* Delete base.py

* update migration annotation

* Create a Saas Connector from a Template [#814] (#1076)

* Starting point for SaaS connector templates

* Fix imports from restructuring.

* Get happy path working for instantiate connector from template endpoint.

* Remove updating connector instances for now - out of scope.

* Test nonexistent templates, secrets validation, instance key / fides key already exists.

* Create DatasetConfigs and ConnectionConfigs instead of create_or_update in the template endpoint.  Don't save ConnectionConfig until secrets are validated.

* Add the other saas connectors to the registry and update their configs and datasets with instance_fides_key.

- Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted.

* Update the fides_keys in the existing saas configs and dataset yamls to have brackets around the "instance_fides_key" to indicate these will be replaced.

Update the fides_key definition to allow "<instance_fides_key>" with brackets specifically to pass validation.

* Fix a side effect on a separate endpoint that returns the types of secrets that should be supplied for a given connector.  Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected.

* - Update CHANGELOG
- Add new endpoint to postman collection
- Add drafts doc.
- Update old response body in docs for connection types.

* Replace the <instance_fides_key> with a properly formatted fides_key in the saas fixtures.

* If DatasetConfig creation fails, delete the recently created ConnectionConfig.

* Address some of the saas integration tests where I've changed the fides_key.

* Fix typos.

* Fix typo.

* Fix unrelated bug where hubspot dataset has new datacategories with user-* data categories after the fideslang update, so they would show up if the user picked a "user" data category.

* Respond to CR.

Co-authored-by: Dawn Pattison <pattisdr@users.noreply.github.com>

* Update the "instantiate_connection_from_template" to return a portion of both the connection config and the dataset. (#1105)

* Adds `AuditLog` and `ExecutionLog`s to `seed_test_data` command (#1097)

* Reduce Idle Connections from Health Checks [#1102] (#1107)

* Don't create a new engine as part of running the health checks and share a single engine across the application, including for the health checks.  Currently we're using the default pool_size and max_overflow.

* Update changelog.

* Fix that health checks are still supposed to run, even if the database is disabled.

* Need to yield instead -  'generator' object has no attribute 'query'

* Escape redis user and password (#1104)

* Escape redis user and password

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* pass in analytics id env to worker (#1113)

* pass in analytics id env to worker

* changelog

* Adds tests for email endpoints and dispatch service (#1112)

* Endpoint to Verify User Identity [#1062] (#1111)

* Add an endpoint to verify a user's identity before queuing the privacy request provided it doesn't need separate manual approval by a system admin.

- Add a new PrivacyRequest.identity_verified_at timestamp
- Add a new PrivacyRequestStatus - "identity_unverified".
- Add methods to cache the verification code in Redis for comparison with a default ttl of 10 minutes

* - Fix linting/copy-paste errors.
- Update changelog.
- Add endpoint to postman

* Add new keys to response bodies.

* Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists.

* Revert "Instead of using a new VerificationCode schema, use the SubjectIdentityVerificationBodyParams that already exists."

This reverts commit 40fcf6d119135d08a6d3ecfc40c5d73846bf2205.

* Bump hashicorp/vault-action from 2.4.1 to 2.4.2 (#1119)

Bumps [hashicorp/vault-action](https://github.com/hashicorp/vault-action) from 2.4.1 to 2.4.2.
- [Release notes](https://github.com/hashicorp/vault-action/releases)
- [Changelog](https://github.com/hashicorp/vault-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/vault-action/compare/v2.4.1...v2.4.2)

---
updated-dependencies:
- dependency-name: hashicorp/vault-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi[all] from 0.79.0 to 0.79.1 (#1120)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.0 to 0.79.1.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.79.0...0.79.1)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump types-urllib3 from 1.26.22 to 1.26.23 (#1117)

Bumps [types-urllib3](https://github.com/python/typeshed) from 1.26.22 to 1.26.23.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-urllib3
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump nox from 2022.1.7 to 2022.8.7 (#1118)

Bumps [nox](https://github.com/wntrblm/nox) from 2022.1.7 to 2022.8.7.
- [Release notes](https://github.com/wntrblm/nox/releases)
- [Changelog](https://github.com/wntrblm/nox/blob/main/CHANGELOG.md)
- [Commits](https://github.com/wntrblm/nox/compare/2022.1.7...2022.8.7)

---
updated-dependencies:
- dependency-name: nox
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump faker from 14.0.0 to 14.1.0 (#1122)

Bumps [faker](https://github.com/joke2k/faker) from 14.0.0 to 14.1.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v14.0.0...v14.1.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump sqlalchemy-redshift from 0.8.10 to 0.8.11 (#1121)

Bumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.10 to 0.8.11.
- [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases)
- [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst)
- [Commits](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/compare/0.8.10...0.8.11)

---
updated-dependencies:
- dependency-name: sqlalchemy-redshift
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* POC for dynamic routing (#1033)

* Add initial POC for dynamic routing

* fix a couple lints

* Fix mypy lint

* making pylint happy

* Remove log

* Fix another pylint issue

* Add docstring

* Update index if check

* Handle nested nextjs routes

* Update changelog

* Add Sass connector configuration (#1099)

* Sass Connecter feature development

761 - Add a Connection - SaaS connector's configuration parameters
984 - Saas Connector configuration - left navigation to toggle between connection params and dataset config
985 - SaaS Connector Configuration - Testing a Connection

* Skipping unit test temporarily

* Update flags.json file

By default, turning off the createNewConnection flag which is still under development.

* Updated CHANGELOG.md file

* Resolved ESLint issues

* Resolved npm build issue

* Resolved npm build issue

* Updated Saas connector configuration

* Updated Saas connector configuration

* Removed unnecessary import statement

* add new privilege for creating SaaS connectors to user management interface

* add connections read as a privilege

* Applied code review feedback

* Updated the CSS visibility of the CircleHelpIcon component

* Added toast success when a user creates a Saas config

* Resolved ESLint issue

Co-authored-by: Sean Preston <sean@ethyca.com>

* 1128-Add Retry button back into the subject request detail view (#1131)

* 1128-Add Retry button back into the subject request detail view

* Updated CHANGELOG.md file

* provide a way to give invited users the resume permission

Co-authored-by: Sean Preston <sean@ethyca.com>

* Fix localhost setting for non-endpoint analytics calls [#1029] (#1130)

* Set local_host to None for non-endpoint analytics calls.  These are logging various tasks coming out of celery.

* Update Changelog.

* Add email_templates module (#1123)

* Add email_templates module

* run isort

* Add unit tests

* Update ttl calculation

* Add ttl minutes test

* fix lint issues

* fix pylint issue

* fix pylint issue

* fix isort

* Update template constant

* Update changelog

* fix lints

* Add jinja to requirements.txt

* update templates directory

* update unit test

* Update imports

* fix issue template path

* Add templates to manifest

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Formatting Issues in Webhooks Documentation (#1114)

* Fix typo in `derived_identity`

* Update CHANGELOG.md

* Missed PR Link

* replaced `<>` with `{}`

In the live version of the docs, the `<>`s were being stripped from our code example titles. I've replaced them with `{}` to align with some of the other pages I've seen.

* Send Identity Verification Email [#1010] (#1115)

* If identity verification required, send email to the user with the verification code.

* Adjust the identity_verification_required autouse fixture, and add an autouse override for just the tests where we want to turn on identity verification.

* Add starting docs and updating the changelog.

Start with identity_verification_required set to False for now until all the related pieces are in.

* Update some of the docstrings.

* Add unverified status color in the FE.

* Add new privacy request status to types and constants.

* Restore trailing comma.

* Update identity_verification_required to subject_identity_verification_required for clarity.

* Adjust email_body_params to accommodate new template.

Co-authored-by: Sean Preston <sean@ethyca.com>

* Support case-insensitive connection type search [Unticketed] (#1133)

* Make connection type search case-insensitive.

* Update changelog.

* Add option to login as root user from config (#1116)


Co-authored-by: Paul Sanders <pau@ethyca.com>

* Move logging to %-style formatting [#837] (#1132)

* Add a new Pii class and use it to wrap arguments not already wrapped with NonPii in those logs that are currently using %-style formatting.

* Switch logging formatting to %-style instead of f-string.

* Continue to address lingering f string instances, and wrap some arguments in Pii, such as raw exceptions.

* Remove NotPii class and update tests.

* Adjust errors made in %-style conversion.

* Remove accidental Pii on print statements, update some PII wrappings.

* Adjust string formatting of newly added log.

* Update Changelog.

* Fix missed closing curly brackets.

* Remove missed curly brackets.

* Add future annotations for backwards compatibility (#1136)

* Add future annotations for backwards compatibility

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Release 1.7.1 (#1141)

* update changelog

* update ERD

* make Hubspot test more robust

* Fix docs build in CI (#1138)

* Fix docs build in CI

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>
Co-authored-by: Sean Preston <sean@ethyca.com>

* fix command syntax (#1143)

* hook up API request to logout action [Unticketed] (#1139)

* hook up API request to logout action

* run format

* update import orer

* add lint:fix

* Update hubspot users and owners configurations (#1091)

* Update CHANGELOG.md

* SaaS Connector Template Creation Fix: Integer fides_key (#1166)

* Use quotes when replacing <"instance_fides_key"> in the saas config and dataset config files to force a string.

* Update CHANGELOG.

* Start a new "email" ConnectionConfig type [#1134] (#1142)

* Start a new "email" ConnectionConfig type.

* Hide "email" type from the get_connection_types endpoint for now, as the email connector isn't fleshed out yet.

* Update CHANGELOG.

* Simplify by sending one email to start?

* Update request body in postman collection.

* Fix CHANGELOG formatting.

* Revert "SaaS Connector Template Creation Fix: Integer fides_key (#1166)" (#1171)

This reverts commit 28b6317fc7bf86019eb6559824226dc261511a95.

* Fix analytics opt out environment variable name (#1170)

* 1004 OAuth2 client credentials flow (#1159)

* update Subject Request detail page (#1164)

* [#495] clarify additions to subject request event log

* [#863] note retry capabilities in subject requests

* changelog

* fidesops.toml

* changelog

Co-authored-by: Cole Garbo <colegarbo@Ethycas-MacBook-Pro.local>

* Bump fastapi[all] from 0.79.1 to 0.81.0 (#1178)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.79.1 to 0.81.0.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.79.1...0.81.0)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Build docker image for privacy center (#1165)

* Build docker image for privacy center

* Update docs and add docker publish to workflow

* Update CHANGELOG

* Move privacy center docker publish to its own workflow

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Update `StorageConfig` to handle multiple auth methods (#1140)

* Initial automatic impl

* Lints

* disable pylint error

* Fix unit tests

* fix imports

* Update fixture

* Fix config tests

* Covert type to enum

* Add new unit tests

* Update changelong

* Update docs

* Update phone number

* patch task scheduler

* remove pylint ignore and add exception

* Update docs

* 1009 id verification required endpoint (#1221)

* Adds identity verification config endpoint

* adds test, reuse existing config var, adds postman collection

* update changelog

* add type ignore

* move ignore line

* Update src/fidesops/ops/graph/config.py

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* remove duplicate lines in method

* another dupe line

* remove scope, update order of classmethod decorator

* remove dependencies on endpoint

* unused imports

Co-authored-by: Paul Sanders <psanders1@gmail.com>

* fix reference in data categories upgrade migration (#1223)

* Publish on every release (#1226)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Make log send async (#1174)

* Make log send async

* Update CHANGELOG

* Add async to additional fideslog calls

* WIP

* Fix issue with async function is celery

* Make __send work with name mangling

* Remove extra await

* Await coroutines in tests

* Remove analytics id

Co-authored-by: Paul Sanders <pau@ethyca.com>

* 1106 saas config shopify access endpoints (#1220)

* Adds email communications guide to docs, exposes other hidden guides in nav bar (#1233)

* adds email communications guide to docs, exposes other hidden guides in nav bar

* update changelog

* email docs copyedits

* remove dupe items from nav

Co-authored-by: Cole <cole@ethyca.com>

* Fix version number (#1232)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump pylint from 2.14.5 to 2.15.0 (#1182)

* Bump pylint from 2.14.5 to 2.15.0

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.14.5 to 2.15.0.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.14.5...v2.15.0)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Ignore missing-timeout warning

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Sanders <paul@ethyca.com>

* Bump fideslib (#1236)

* Bump fideslib

* Update CHANGELOG

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Email Connector: Build Masking Instructions  (#1168)

* Pass in input_data to erasure requests, and not just access requests, so it can be used for the email connector, which won't have any rows returned from an access request.

- Add an EmailConnector.build_masking_instructions method with a draft of data needed to instruct the user how to query/mask/what fields to mask on their end.

* Have the EmailConnector.mask_data  cache the raw details of what needs to be masked in Redis. We'll use this to send one email at the end for each "email"-based dataset at end, instead of sending one email for each collection.

Reuse some of the caching code created for manual connectors / failed privacy requests where similar to the EmailConnectors, we have some separate action that is required on a given collection.  Rename to make more generic.

* Remove restriction that a ManualAction needs a get or update value.  The manual action could just be locating data for another collection downstream.

Cache email template details, even if there are no actions needed on that specific collection,

* Update the expected number of collections in the email dataset.

* build_masking_instructions is not required to return a ManualAction.

* Reconcile this test with the work to make log send asynchronous.

Co-authored-by: Sean Preston <sean@ethyca.com>

* Adds email scopes to postman collection (#1241)

* Adds email scopes to postman collection

* update changelog

* Bump black from 22.6.0 to 22.8.0 (#1238)

Bumps [black](https://github.com/psf/black) from 22.6.0 to 22.8.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.6.0...22.8.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Rename fidesops-privacy-center image to fides-privacy-center (#1237)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Bump eslint from 8.9.0 to 8.23.0 in /clients/ops/privacy-center (#1180)

Bumps [eslint](https://github.com/eslint/eslint) from 8.9.0 to 8.23.0.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)
- [Commits](https://github.com/eslint/eslint/compare/v8.9.0...v8.23.0)

---
updated-dependencies:
- dependency-name: eslint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* #1008 User  identity verification flow (#1231)

* Update lint commands and eslint ignore

* Run lints and refactor privacy cards

* Refactor modal

* Finish verification flow

* get config from server

* Update changelong

* Fix test failures

* Format file

* Mock out route

* Format file

* Add code resending

* Update test to use hostUrl

* Add headers util function and PrivacyRequestStatus status enum

* Bump @chakra-ui/utils from 1.10.4 to 2.0.9 in /clients/ops/admin-ui (#1145)

Bumps [@chakra-ui/utils](https://github.com/chakra-ui/chakra-ui/tree/HEAD/packages/utils) from 1.10.4 to 2.0.9.
- [Release notes](https://github.com/chakra-ui/chakra-ui/releases)
- [Changelog](https://github.com/chakra-ui/chakra-ui/blob/main/packages/utils/CHANGELOG.md)
- [Commits](https://github.com/chakra-ui/chakra-ui/commits/@chakra-ui/utils@2.0.9/packages/utils)

---
updated-dependencies:
- dependency-name: "@chakra-ui/utils"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Allows worker to start successfully in both dev and dev_with_worker (#1250)

* Allows worker to start successfully in both dev and dev_with_worker

* update changelog

* formatting

* Add `execution_timeframe` to `Policy` model and schema (#1244)

* Add execution_timeframe column to model and schema

* Fix test failures

* Fix Policy patch route

* Update Policy patch test

* Run black and isort lints

* Update changelog

* fix typo

Co-authored-by: Sean Preston <sean@ethyca.com>

* Fix for pytest-asyncio bug (#1260)

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Clean up docker build (#1252)

* Add files to .dockerignore

* Add config path to docker-compose

* Mount volume for build check

* Fix pylint error

* Update CHANGELOG

* Fix failing external test

* Remove worker env var from docker-compose

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Email Connector: Send Email with Erasure Instructions [#1158] (#1246)

* Send an email for each email-based dataset at the end of privacy request execution.

- Add a migration to create a new audit log type.  Create an audit log for the email send.
-  Throw an exception for email-based connectors and catch to override the default execution log.
- Add a draft of an email template
- Connect sending a "test email" with dummy data.  A fidesops admin could configure to check their email config was working.

* Add more "checkpoints" to privacy request execution - these are locations from which we can resume privacy request execution without having to run from the beginning.

- Add more options to CurrentStep Enum
- Cache the checkpoint if an email send fails, so we can retry from the same step.

* Don't send an email if the connection config is read only or there are no updates to be applied to any of the collections on the dataset.

* Don't assume there's a collection when building "resume" details. A failed privacy request can be resumed outside of the traversal.

* Add a first draft of docs for setting up an email connector.

* Moves the email connector send method to the email connector file.

* Update mock location.

* Bump downrev.

* update email connector guides

* correct link, broken sentence

* Create a new EmailRequestFulfillmentBodyParams type to be used once the cached email details are extracted by dataset.

* Fix missed test.

Co-authored-by: Cole <cole@ethyca.com>

* Fix download link (#1264)

* Fix download link

* Update changelog

* Add `due_date` to `PrivacyRequest` model (#1259)

* Add `due_date` to `PrivacyRequest` model

* Add `due_date` test and run lints

* Update CHANGELOG.md

* Fix test failures

* Fix type error

* Fix mypy issue

* Switch to `strptime`

* Fix migration down revision

* Move date format string into constant

* Refactor strategy instantiation for more extensitiliby (#1254)

* Instantiate strategies via abstract Strategy base class

A generalized Strategy abstract base class provides generalized getter methods
that instantiate strategy subclasses (implementations).
These methods rely on the builtin __subclasses__() method to identify Strategy subclasses,
which allows for more dynamic and extensible strategy implementation, removing the need
for a hardcoded enumeration of supported Strategy implementations.
Abstract strategy types inherit from this new abstract base class,
and strategy subclasses (implementations) must provide `name` and `configuration_model` attributes
that are leveraged by new instantiation mechanism in the abstract base class.

* Update get_description() to be a class rather than static method

This allows the method to leverage the new `name` class variable rather than
relying on a static constant variable.

* Remove strategy factories and update references

Strategy factories are no longer needed with refactored Strategy getters.
Update the uses (references) of strategy factories throughout the codebase
to now rely on the new Strategy getters.
Strategy subclasses (implementations) now need to be imported explicitly
in __init__.py's because they used to be imported in factory modules.
Also remove the old MaskingStrategy registration/factory mechanisms.

* Remove strategy name constants

Now that the abstract Strategy base class enforces implementation subclasses
to have a `name` class attribute, this attribute should be relied upon rather than
the arbitrary name constants declared previously.
The get_strategy_name() abstract method is also superfluous, as the `name`
class attribute can be used as a standardized way to retrieve the strategy name.

* Remove get_configuration_model() abstract method

The generalized strategy getter now relies upon the `configuration_model`
class variable that's on each Strategy. Therefore we no longer need the
get_configuration_model() getter on each Strategy subclass.

* Update MaskingStrategy docs with new Strategy functionality

* Update changelog

* Improve recursion in _find_all_strategy_subclasses

* Fix recursion bug when finding all strategies

Update associated tests to make sure the recursion is properly tested

* Tweak conditional for falsy check

* Make get_strategies endpoint test more robust

* Fix typo in documentation

Co-authored-by: Adam Sachs <adam@Adams-MacBook-Pro.local>

* Make admin ui work when volumes are mounted (#1266)

* Make admin ui work when volumes are mounted

* Update changelog

* Fix path issue with hard refresh

* Update from code review

Co-authored-by: Paul Sanders <pau@ethyca.com>

* Update the Erasure Request Email Fulfillment template [#1265] (#1270)

* Update email template for email request fulfilment.

* Formatting changes.

* Bump types-redis from 4.3.4 to 4.3.20 (#1255)

Bumps [types-redis](https://github.com/python/typeshed) from 4.3.4 to 4.3.20.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-redis
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update boto3 requirement from ~=1.24.46 to ~=1.24.68 (#1272)

Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](https://github.com/boto/boto3/compare/1.24.46...1.24.68)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump requests-mock from 1.9.3 to 1.10.0 (#1276)

Bumps [requests-mock](https://github.com/jamielennox/requests-mock) from 1.9.3 to 1.10.0.
- [Release notes](https://github.com/jamielennox/requests-mock/releases)
- [Commits](https://github.com/jamielennox/requests-mock/compare/1.9.3...1.10.0)

---
updated-dependencies:
- dependency-name: requests-mock
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fastapi[all] from 0.81.0 to 0.82.0 (#1277)

Bumps [fastapi[all]](https://github.com/tiangolo/fastapi) from 0.81.0 to 0.82.0.
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.81.0...0.82.0)

---
updated-dependencies:
- dependency-name: fastapi[all]
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add button to add a taxonomy entity
2 participants