Add a new Cypress E2E test runner that executes against a fully configured environment

[NevilleS]

Description

In some previous releases, we worked on building out test setup scripts (#1291) and a local manual test environment (#1292), which gives us the ability to do true E2E manual tests to confirm all features are working as expected when running together.

Our existing Cypress suites for the admin-ui and privacy-center are great, but they are isolated to their specific applications and tend to stub out their backends for performance. Therefore, there's still an open need for an E2E test runner to use for regression tests on an ongoing basis.

Once the runner is configured, let's put it to work by defining a core smoke test for releases. We should ensure this can run against our local test env first, but I want to immediately start using this against some hosted staging & demo environments in the short-term, so let's ensure that any URLs are easy to configure and override (e.g. http://localhost:3000 will quickly become something like https://fides.fides-staging.ethyca.com/)

In addition, let's signup for Cypress Cloud and publish results there, it's a great product!

Acceptance Criteria

• MUST implement a new Cypress runner to run E2E tests, targeting the local "manual test" environment (see Create a single nox command to setup & run a manual test environment #1292)
• MUST support targeting different hosts via environment variables, so we can run against staging / demo / etc.
• MUST have a nox session (e.g. nox -s e2e_test) that runs the Cypress suite in headless mode
• MUST configure the test reporter to publish results to our Cypress Cloud account
• MUST write a single Cypress E2E test that runs a simple smoke test covering:
  1. Confirm can login to the Admin UI
  2. Confirm that the Postgres & Mongo connectors are configured
  3. Confirm that the Privacy Center can be accessed
  4. Submit an access request via the Privacy Center
  5. Approve the access request via the Admin UI
  6. Confirm the access request succeeds

[ssangervasi]

One note from this convo:

not running against the live backend in those E2E tests is a bug, not a feature. we’ve scoped out work to fix that here

I am a fan of FE+BE integrated regression test. However, I would say the 80/20 rule for cypress tests is that 80% should be using stubbed endpoints. Developers should be live-running the Cypress tests for a feature when changing it. An actual BE is usually bad for this because:

• It will be slow
• Create/Deletes are going to get out of sync
• The data that is useful for regressions is a separate concern from what is useful for covering a UI feature
• The BE might not even be updated yet (developing features in tandem like Classify UI).

I think this ticket is pointing in the right direction: we should have a distinct subset of test that don't stub anything. I just want to make it clear the goal isn't to convert all cypress tests to run against a live API.

[rsilvery]

@NevilleS , @seanpreston are we going to complete this in this quarter? Trying to update Aha dashboard.

[NevilleS]

@ssangervasi totally fair - I've updated the title & description here to be more appropriate - you're right that we shouldn't need to replace the existing suites, and in fact an E2E runner is really just one more to add.

[NevilleS]

@rsilvery sorry for not replying last month- we deprioritized this work last year to focus elsewhere, but I'd like to include it again now.

[allisonking]

MUST implement a new Cypress runner to run E2E tests, targeting the local "manual test" environment (see #1292)

We'll have to figure out where we want these tests to live in the repo. Normally they hang out next to the UI they are testing, though in this case there are multiple UIs they are testing. It might still make sense to have them live with admin-ui since that's where most of the code is, just in a separate folder. And then the same thing for datamap-ui for plus-only features... Or it might work to have it just live at its own folder in clients. That's what this blog post does, although their use case is slightly different in that they aren't trying to go between apps within their tests

A trickier thing might be actually getting Cypress to talk to different URLs. We've been hurt by their cross origin prevention when the app tries to navigate between zones. It looks like they might handle this now though, for the purpose of customers being able to log in via something like Authy

It's probably worth POC-ing this part alone to make sure we can hit all the apps we need against nox -s test_env without error.

The rest of the requirements look doable to me provided the first case works out 👍 I'd estimate the POC taking a day or so, depending on how flexible the cypress configuration is. Then the rest of the requirements should be able to be done without much hiccup 🤞

Is another requirement to have this running in CI, or do we only want to run it manually? Also excited to try out cypress cloud!

[seanpreston]

Commenting to flag that this work is blocked by #2385

[allisonking]

Closed by #2241