Connector dataset dropdown

[allisonking]

Closes #2114

Code Changes

• Adds a dataset dropdown next to the existing yaml editor
• Allows the dataset dropdown to link a dataset to a connector config
• Additional confirmation button for overwriting datasets in the yaml editor
• Fix copy (system yaml --> dataset YAML)
• Cypress tests

Steps to Confirm

• nox -s test_env
• Configure a connector dataset

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

Description Of Changes

Using the dropdown

Screen.Recording.2023-01-10.at.7.01.39.PM.mov

Using the yaml editor

Screen.Recording.2023-01-10.at.7.04.22.PM.mov

[allisonking]

Failing test seems unrelated?

[pattisdr]

yep this is unrelated @allisonking, I saw this on my last unified-fides-resources PR but I don't think it's on main. After we merge your branch in, I'll get unified-fides-resources up-to-date with main and investigate there.

[pattisdr]

This feels a little buggy to me in places. This branch isn't up-to-date with unified-fides-resources that has the improved upsert validation, but that is handy here so it's easy to see where we might add better error handling:

1. Start from scratch, delete datasetconfigs, connectionconfigs, and ctl_datasets. Refresh. It makes sense that there's no dropdown to connect my first connection config to an existing dataset, but the save button stacked on the cancel button looks off

2) Copy and paste a dataset (mysql is a good one) that doesn't have an `organization_fides_key` or `data_qualifier` in it. You could also send in a bad data category. We get failed requests but the UI looks like we never submitted a request at all. (In the background a ctl_dataset has been created, but the ops datasetconfig was not created)

error_handling_patch_datasetconfig.mov

3. Editing an existing dataset that was good to something invalid (add bad data category) also puts the UI in a weird state

4. When saving existing dataset yaml you've added a helpful confirmation that you're about to overwrite a dataset. This is not necessarily true if the user is also editing the fides_key.
   Below, the only edit I made was to change postgres_example_test_dataset fides_key to updated_fides_key. This will create a new ctl_dataset titled updated_fides_key, and link it to the DatasetConfig.

5. When configuring an existing dataset in the connector workflow, it took me awhile to figure out how to edit directly. I needed to unselect the current dataset.

[allisonking]

Thanks for the thorough testing @pattisdr ! 🙏

1. This should be fixed. It's a little weird code-wise, but the new designs don't have a cancel button anymore so I don't think we have to worry too much about it since it'll go away soon.
   

2 and 3. I'm having trouble reproducing these—I just merged in unified-fides-resources which like you said throws the proper errors now, so I wonder if that's letting me see good errors instead of getting stuck in weird states? Can you let me know if you're still seeing these problems, and if so, with what dataset? (I tried out mysql_example_test_dataset.yml which seemed to work ok)

4. Good catch, should be fixed!
5. Yeah, this was the decision we came to when we were trying to decide on behavior without design. The good news is this will go away with the new design that's currently in the works

[pattisdr]

2 and 3. I'm having trouble reproducing these—I just merged in unified-fides-resources which like you said throws the proper errors now, so I wonder if that's letting me see good errors instead of getting stuck in weird states? Can you let me know if you're still seeing these problems, and if so, with what dataset? (I tried out mysql_example_test_dataset.yml which seemed to work ok)

OK after some more digging I found what bad request is causing this: it's if a GET http://localhost:3000/api/v1/dataset throws an error because it's returning a bad dataset. It's harder to create a bad dataset after you've merged in unified-fides-resources.

To reproduce, you could force the backend to throw an error, for example:

1. Go to src/fides/api/ctl/routes/crud.py
2. In the list endpoint async def ls:, raise an exception first thing Exception()

This mimics what would happen if a user had a bad dataset already saved.

[allisonking]

Thanks for such thorough testing! Should be ready for another review now 🤞

[pattisdr]

wonderful, testing now!

[pattisdr]

OK this UI is feeling pretty good now and holding up under a lot of scenarios I'm testing. Thanks for your nice changes here! There's just a couple of things I saw on this last round but I think it's about ready to go.

[pattisdr]

@allisonking one more thing I just thought of I forgot to test: creating saas connectors. One big difference with saas connectors is that we create them from a template. After the user enters their connector parameters and clicks save, the connection config, dataset config, and dataset are all created in the background for them.

There's a small thing where it's hard to save the dataset when you create multiple saas connectors.

• Create a mailchimp connector
• Enter connector parameters (just fake data is fine)
• On configure dataset click save in the dropdown
• Create a segment connector
• Enter connector parameters
• On configure dataset, the segment dataset is not in the dropdown. I can click "Save" here, but it's confusing because there's only "Select" in the dropdown. I can't edit the yaml either: by toggling to another dataset, then back to Select, you can edit the yaml. I suspect the "Select" is tied to the actual dataset here behind the scenes.

Screen.Recording.2023-01-12.at.12.35.55.PM.mov

[allisonking]

Oh man, that's a good catch I never would have looked for. This should fix it. The change really should have only been the one line (to add "Datasets" to the list of resources in the cache to be invalidated when a new saas connector is created), but due to some tech debt I had to combine the relevant redux slices to use the same base API first.

[pattisdr]

Alright this looks really good to me @allisonking thanks for your hard work here 🏆 Failing test is unrelated like we discussed and I'll look into this after you merge this branch into unified-fides-resources.