Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running CircleCI render tests on Windows, Mac and Firefox #12268

Merged
merged 129 commits into from
Dec 13, 2022

Conversation

SnailBones
Copy link
Contributor

@SnailBones SnailBones commented Sep 29, 2022

Closes https://github.com/mapbox/gl-js-team/issues/480

Adds three new runs in CircleCI:

  • test-render-macos-chrome
  • test-render-windows-chrome
  • test-render-linux-firefox

Windows and Mac tests are running on virtual machines.

Failing tests on Windows and Mac are due to bugs including #7331. Using --use-gl-desktop lets these tests pass on Mac but also causes a different set of failing tests due to missing globes, inconsistent stars and fill extrusion patterns (as was the reason for disabling this locally in #12090).

The same tests skipped on Windows and Mac formerly failed on Linux without the --use-gl-desktop flag (added in #10389) as seen in this run without the flag.

I've taken the following approach to updating "allowed" properties:

  • If the images are visually indistinguishable and dif is suitably low (< 1% for small images and less for larger or detail-oriented ones), raise the "allowed" threshold just above the dif.
  • Otherwise, add to a platform-specific ignore file.
  • For cases where appearance is still good but the dif is higher, we may want to consider introducing platform-specific "allowed" values, but I've avoided that for now to minimize the scope of this PR.

Previous runs tended to finish in 5-6 minutes. All of the new runs added take longer:

  • Firefox: ~8m
  • Mac: ~14m
  • Windows: ~20m

To address this, I've implemented parallel runs on Mac and Windows. This has required some restructuring of how reading testfiles work. If a tests-to-run.txt file is present in the root directory, the test runner will read tests from that file instead of running all tests that match a glob.

There's likely room to optimize the CI config through caching. but 90% of the time spent on Windows is running tests. A more effective way to speed up CI outputs would be splitting it into two (or more) runs, with each running half the tests.

Issues tracking new tests in ignores:

Launch Checklist

  • Operating systems
    • Windows
    • MacOS
  • Browsers:
    • Firefox
  • Update (default?) "allowed" values so tests pass
  • Open issues tracking all platform-specific test failures
  • Performance:
    • [] Cache yarn and other slower setup steps when possible (Caching yarn cache makes runs slower on Windows)
    • Parallelize test runs to bring run time to < 10 minutes

@SnailBones SnailBones added testing 💯 skip changelog Used for PRs that do not need a changelog entry labels Sep 29, 2022
@SnailBones SnailBones changed the title Running CircleCi render tests on Windows and Mac virtual machines Running CircleCI render tests on Windows and Mac virtual machines Sep 30, 2022
@@ -2,8 +2,7 @@
"version": 8,
"metadata": {
"test": {
"height": 256,
"allowed": 0.00025
"height": 256
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was incorrectly reverted in #10562 and was passing due to a high "allowed." By removing the allowed and correcting the test we should now correctly catch a regression to order.

@SnailBones SnailBones merged commit f0d57f5 into main Dec 13, 2022
@SnailBones SnailBones deleted the aidan/windows-ci branch December 13, 2022 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip changelog Used for PRs that do not need a changelog entry testing 💯
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants