CI: Front-End Test Framework

[kdumontnu]

Feature Description

I will continue to update this over time and ultimately aim to run a vote in the next few days if a clear contender doesn't emerge.

As discussed in #18302 and #14639, gitea is in need of a visual testing framework to prevent changes from breaking existing parts of the UI.

Requirements:

• Need to render JS, HTML, Vue, and CSS to detect if differences occur between accepted and proposed changes
• Need to have a method to store accepted state of UI (i.e. screenshot)

Considerations:

• Cross platform and mobile testing would be ideal, but might add too many permutations
• The ability to screencap specific UI elements would also be helpful
• Whatever we select we need to figure out where to save the screenshot files.
  • We can't really commit images directly to the repo because it would simply be too much data
  • One option is LFS, but we don't really want to add a new dev dependency
  • Some of the testing frameworks have a "Dashboard" that should host the screencap images automatically in a database (ex. Cypress). Need to verify this.
  • Can we use git notes to link to an S3/Minio bucket which contains the test images? Has anyone tried something like this?
• Does this play well with JS testing framework (ie. Jest)

Options

Tool	Description	License	Visual Test	JS Assertions	Pros	Cons
Cypress.io			Y	Y
Playwrite			Y	Y
TestCafe	End-to-end testing, simplified		N			No visual testing
Percy
BackStopJS			Y	?

TBD

I haven't included Selenium here, as I figured that would require us building our own testing framework, which, given all of the options above, I don't think is necessary. If someone has examples of a good implementation I can include it.

Questions

• Can we use the existing integration test framework to emulate an instance of Gitea or do we need to spin up a new gitea instance in CI to run the visual tests? For example, we can have a separate Drone pipeline run in parallel that starts an instance of gitea and runs the front-end tests.
• How many permutations do we need / can we afford? For ex. Firefox, Chrome, Safari, Mobile + we should test on Windows/Ubuntu/Alpine? Will need to follow up with some tests.

Roadmap

TBD

[silverwind]

So at this point, I'm not sure if visual testing really is the way to go. Besides the problem with screenshot storage, I can see it becoming a nuisance when doing far-reaching styling changes.

I have to say thought, I have no experience with visual testing so maybe my fears are unwarrented. To me it sounds like visual testing is probably something best done on a external service that takes care of screenshot storage and diff presentation, I don't know if such a service exists.

I think it it'd overall be more valuable to have coded UI integration tests that perform common actions like operate on issues/pr, create commits on the web editor, which could be described using a headless browser like playwright which integrates well with jest.

The only complexity we face is that we need the backend available during these tests, but I'm certain this can be achieved with some setup code in jest.

[wxiaoguang]

Same feeling with @silverwind , I am afraid pixel-level UI comparing is too heavy and it would lead to lower willing to do UI refactoring or improvement.

[kdumontnu]

I think it it'd overall be more valuable to have coded UI integration tests that perform common actions like operate on issues/pr, create commits on the web editor, which could be described using a headless browser like playwright which integrates well with jest.

pixel-level UI comparing is too heavy

I agree that coded assertion testing, like Playwright, Mocha, etc. is a requirement. It will help us catch issues with templates and javascript/Vue, especially for the JQuery port.

However, the thing it won't catch is CSS issues. I imaging migrating from Fomantic to Tailwind would be incredibly difficult without that. I guess it's acceptable to have some CSS bugs, but I think it would be really nice for the creators of a CSS class to say "it should look like this", very quickly be able to test it on multiple browsers, and make sure no one accidentally breaks it in the future.

I think it makes sense to do a proof of concept to see if visual testing would be too heavy (at least we'll learn). If we can get 80% of the benefit with 20% of the effort from having just something like Playwright then I could definitely be convinced. If people like something like Cypress or equivalent even for just the assertion tests, we could still add visual in the future.

[kdumontnu]

@silverwind @wxiaoguang I tried to get Cypress set up and it just felt too heavy. Install took a while and it wasn't clear how managing screenshots would work. In the PR above I took a swing at setting up Playwright. It seemed a lot more straightforward.

It does support screenshots in case we want to take the leap on adding that in the future (I still think LFS would be the way to go, but that convo could be had later).

Feel free to take a look - happy for some feedback.

[silverwind]

Interesting article: https://docusaurus.io/blog/upgrading-frontend-dependencies-with-confidence-using-visual-regression-testing