Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Experiment] Try Playwright #34089

Closed
wants to merge 44 commits into from
Closed

[Experiment] Try Playwright #34089

wants to merge 44 commits into from

Conversation

WunderBart
Copy link
Member

@WunderBart WunderBart commented Aug 16, 2021

Howdy! 👋

I'd like to make a case for migrating our E2E tests to Playwright. I figured the best way to do that would be to list its advantages and do a bit of refactoring so that we can actually see those advantages in action. This PR is not meant to be merged! The Playwright implementation here is very basic and just a subset of tests has been refactored for experiment/demo purposes, although all those refactors would be valid if cherry-picked into proper PR(s). So, without further ado...

Why should we drop Puppeteer in favor of Playwright?

In general, because it's easier to write stable tests with it. There are a few reasons for that, so let me start with

The API

It's almost identical. If you're used to Puppeteer you'll have no trouble switching to Playwright. <It's worth noting here that it would also mean a low-cost refactor!> . The high syntax similarity itself doesn't mean it will get easier though, so let's move to

The Auto-Waiting Mechanism

I think this is the #1 reason for the stability improvement over Puppeteer. From the documentation:

Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. (...)

For example, the page.click( selector ) method waits for the target element to be attached, visible, stable, receive events, and enabled before clicking the actual element. In practice, that means:

  • No need to perform any additional presence checks, e.g.

    const elementHandle = await page.waitForSelector( selector );
    await elementHandle.click();
    
    // becomes
    
    await page.click( selector );

    Check out the demo refactor commits (i.e. 70eba4e) for more practical examples.

  • No need to disable CSS animation or enforce the reduced motion.

    This is thanks to the stability check, which makes sure the element has a bounding box and is not moving before the action is performed. I think this is a big one because it would allow us to fully test the application, including the CSS animations. For the purpose of demonstrating that, I've deactivated the gutenberg-test-plugin-disables-the-css-animations plugin and removed the enforced reduced motion in this PR. At this point, you should be seeing passing Playwright tests with animations enabled in the Checks tab.

    And finally...

  • Less code / Faster scripting.

The Advanced Selectors Support

I think this one's the #1 reason for making the tests easier to write and follow. From my experience, one of the most important good practices for writing resilient E2E tests is prioritizing the user-facing attributes as they change less often than CSS classes. There isn't a more user-facing attribute than the element's inner text, and that's why the wide range of supported text selectors, together with layout-based selectors in Playwright win the game for me. For example, this is how we can fill the media URL in the Image block by combining text and layout-based selectors:

await page.fill( 'input:below( :text( "Current media URL:" ) )', imageUrl );

...while with Puppeteer, we need to rely on CSS selectors or create complicated XPath selectors. For example, this is how we're currently selecting an active Inserter element:

`//*[contains(@class, "components-autocomplete__result") and contains(@class, "is-selected") and contains(text(), 'List')]`

With Playwright it would simply be:

'.components-autocomplete__results.is-selected:text-is( "List" )'

Not only it encourages good practices but it also makes the element selection more straightforward and easier to follow. Big win, if you ask me.

There's one observation I wanted to mention here. I'm not sure why it was decided to use snapshots as an expected output comparison, but I suppose that it's mostly because of how convenient they are in this regard. I think though that there's one disadvantage in how we're using them that makes this method a bit unreliable — there's no waiting for the UI update before the edited post content is taken for snapshot comparison. The most erratic suite that demonstrated this issue for me was the List block one. The UI was not keeping up, at least not in a consistent manner, and the tests were failing intermittently. Among other corrections, it needed a bit of time before the UI updates in order to make a snapshot comparison. That time can be easily applied by using the waitForSelector method that would wait for the expected output. The reason why I'm mentioning it here is that with Playwright it's much easier to verify the produced content using the aforementioned selectors. If you check out the List block demo refactor, you'll see that the snapshot comparisons aren't really needed at all when we verify the content using simple text matchers. Unless of course, we do want to compare the full markup, which I'm not sure is a good idea since it can contain a lot of unrelated information, doesn't say much inside the test itself, and can be difficult to follow. Snapshots comparisons can be useful of course, e.g. for strict comparison against multiline content, but IMO it should almost always be done after the waiting for the expected UI has been done. Just a good practice in my opinion. What do you think?

Let's move on to the next point.

The Browser Support

This is a short one - Playwright supports all of them. 😄

The Debugging

I think that the Playwright's built-in Inspector deserves a mention here. It's quite intuitive and has some neat features like stepping through the script or recording the actions to a script (codegen). I think it's great to have that kind of functionality OOTB.

Screen.Recording.2021-08-17.at.19.41.11.mov

You can check it out yourself by running PWDEBUG=1 npm run test-e2e on this branch.

The Documentation

Also something worth mentioning in my opinion, as it's easy to navigate and really well written. Please take a few minutes and check it out to see for yourself: https://playwright.dev/docs/intro. Maybe you'll find even more reasons to switch to Playwright. 🤷

What can I do with this branch?

You can run some tests locally in a few different ways. Before you do, make sure your wp-env is up and running and then:

# Run headlessly all the tests refactored in this PR:
npm run test:e2e 

# Run the same thing headfully:
PLAYWRIGHT_HEADLESS=false npm run test:e2e

# Run headfully in slow-motion:
PLAYWRIGHT_HEADLESS=false PLAYWRIGHT_SLOMO=200 npm run test:e2e

# Run a specific suite:
# (Only blocks/* tests have been refactored to run with Playwright!)
npm run test:e2e packages/e2e-tests/specs/editor/blocks/image.test.js

# Run in the debugging mode:
# (You can put `await page.pause()` anywhere in the code to pause the execution.)
PWDEBUG=1 npm run test:e2e

# Run with verbose logs:
DEBUG=pw:api npm run test:e2e

See https://playwright.dev/docs/debug#run-in-debug-mode for more information on Playwright debugging.


I know it got quite lengthy, but I really wanted to make a good case for Playwright, and I hope I did. Writing good E2E tests is often a struggle, and I think that if there's a chance of improving that, especially with such a low cost, then it should be done. I would be happy to work on this task if we decide to move it forward. 🙌

@WunderBart WunderBart force-pushed the try/playwright branch 2 times, most recently from ff6604b to b649299 Compare August 16, 2021 15:36
@WunderBart WunderBart self-assigned this Aug 16, 2021
@WunderBart WunderBart marked this pull request as ready for review August 16, 2021 16:30
@gziolo gziolo added [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Technical Prototype Offers a technical exploration into an idea as an example of what's possible labels Aug 17, 2021
@gziolo
Copy link
Member

gziolo commented Aug 17, 2021

The interesting part is that in this PR some of the issues with existing tests are exposed. I'm looking forward to seeing more details about the benefits of moving to Playwright. Thank you for investing your time investigating steps necessary to use Playwright instead of Puppeteer.

To get us there we will have to update also @wordpress/scripts (impacts 3rd party projects) and WordPress core to use the same testing framework so it's rather a complex process that requires a good rollout plan.

@WunderBart
Copy link
Member Author

To get us there we will have to update also @wordpress/scripts (impacts 3rd party projects) and WordPress core to use the same testing framework so it's rather a complex process that requires a good rollout plan.

Thanks for the heads up! I was aware of the @wordpress/scripts but I didn't know about the core part. Thankfully, it doesn't seem like there would be a lot to update, though! 🤞

@WunderBart WunderBart force-pushed the try/playwright branch 2 times, most recently from b973ea6 to b8d0566 Compare August 17, 2021 12:12
@github-actions
Copy link

github-actions bot commented Aug 17, 2021

Size Change: 0 B

Total Size: 1.06 MB

ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 931 B
build/admin-manifest/index.min.js 1.09 kB
build/annotations/index.min.js 2.7 kB
build/api-fetch/index.min.js 2.21 kB
build/autop/index.min.js 2.08 kB
build/blob/index.min.js 459 B
build/block-directory/index.min.js 6.19 kB
build/block-directory/style-rtl.css 1.01 kB
build/block-directory/style.css 1.01 kB
build/block-editor/default-editor-styles-rtl.css 378 B
build/block-editor/default-editor-styles.css 378 B
build/block-editor/index.min.js 134 kB
build/block-editor/style-rtl.css 13.9 kB
build/block-editor/style.css 13.9 kB
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 65 B
build/block-library/blocks/archives/style.css 65 B
build/block-library/blocks/audio/editor-rtl.css 58 B
build/block-library/blocks/audio/editor.css 58 B
build/block-library/blocks/audio/style-rtl.css 111 B
build/block-library/blocks/audio/style.css 111 B
build/block-library/blocks/audio/theme-rtl.css 125 B
build/block-library/blocks/audio/theme.css 125 B
build/block-library/blocks/block/editor-rtl.css 161 B
build/block-library/blocks/block/editor.css 161 B
build/block-library/blocks/button/editor-rtl.css 474 B
build/block-library/blocks/button/editor.css 474 B
build/block-library/blocks/button/style-rtl.css 600 B
build/block-library/blocks/button/style.css 600 B
build/block-library/blocks/buttons/editor-rtl.css 315 B
build/block-library/blocks/buttons/editor.css 315 B
build/block-library/blocks/buttons/style-rtl.css 370 B
build/block-library/blocks/buttons/style.css 370 B
build/block-library/blocks/calendar/style-rtl.css 207 B
build/block-library/blocks/calendar/style.css 207 B
build/block-library/blocks/categories/editor-rtl.css 84 B
build/block-library/blocks/categories/editor.css 83 B
build/block-library/blocks/categories/style-rtl.css 79 B
build/block-library/blocks/categories/style.css 79 B
build/block-library/blocks/code/style-rtl.css 90 B
build/block-library/blocks/code/style.css 90 B
build/block-library/blocks/code/theme-rtl.css 131 B
build/block-library/blocks/code/theme.css 131 B
build/block-library/blocks/columns/editor-rtl.css 206 B
build/block-library/blocks/columns/editor.css 205 B
build/block-library/blocks/columns/style-rtl.css 497 B
build/block-library/blocks/columns/style.css 496 B
build/block-library/blocks/cover/editor-rtl.css 666 B
build/block-library/blocks/cover/editor.css 670 B
build/block-library/blocks/cover/style-rtl.css 1.23 kB
build/block-library/blocks/cover/style.css 1.23 kB
build/block-library/blocks/embed/editor-rtl.css 488 B
build/block-library/blocks/embed/editor.css 488 B
build/block-library/blocks/embed/style-rtl.css 417 B
build/block-library/blocks/embed/style.css 417 B
build/block-library/blocks/embed/theme-rtl.css 124 B
build/block-library/blocks/embed/theme.css 124 B
build/block-library/blocks/file/editor-rtl.css 300 B
build/block-library/blocks/file/editor.css 300 B
build/block-library/blocks/file/style-rtl.css 255 B
build/block-library/blocks/file/style.css 255 B
build/block-library/blocks/file/view.min.js 322 B
build/block-library/blocks/freeform/editor-rtl.css 2.44 kB
build/block-library/blocks/freeform/editor.css 2.44 kB
build/block-library/blocks/gallery/editor-rtl.css 977 B
build/block-library/blocks/gallery/editor.css 982 B
build/block-library/blocks/gallery/style-rtl.css 1.6 kB
build/block-library/blocks/gallery/style.css 1.59 kB
build/block-library/blocks/gallery/theme-rtl.css 122 B
build/block-library/blocks/gallery/theme.css 122 B
build/block-library/blocks/group/editor-rtl.css 159 B
build/block-library/blocks/group/editor.css 159 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 78 B
build/block-library/blocks/group/theme.css 78 B
build/block-library/blocks/heading/style-rtl.css 114 B
build/block-library/blocks/heading/style.css 114 B
build/block-library/blocks/home-link/style-rtl.css 247 B
build/block-library/blocks/home-link/style.css 247 B
build/block-library/blocks/html/editor-rtl.css 332 B
build/block-library/blocks/html/editor.css 333 B
build/block-library/blocks/image/editor-rtl.css 728 B
build/block-library/blocks/image/editor.css 728 B
build/block-library/blocks/image/style-rtl.css 482 B
build/block-library/blocks/image/style.css 487 B
build/block-library/blocks/image/theme-rtl.css 124 B
build/block-library/blocks/image/theme.css 124 B
build/block-library/blocks/latest-comments/style-rtl.css 284 B
build/block-library/blocks/latest-comments/style.css 284 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B
build/block-library/blocks/latest-posts/editor.css 137 B
build/block-library/blocks/latest-posts/style-rtl.css 528 B
build/block-library/blocks/latest-posts/style.css 527 B
build/block-library/blocks/list/style-rtl.css 94 B
build/block-library/blocks/list/style.css 94 B
build/block-library/blocks/media-text/editor-rtl.css 266 B
build/block-library/blocks/media-text/editor.css 263 B
build/block-library/blocks/media-text/style-rtl.css 488 B
build/block-library/blocks/media-text/style.css 485 B
build/block-library/blocks/more/editor-rtl.css 431 B
build/block-library/blocks/more/editor.css 431 B
build/block-library/blocks/navigation-link/editor-rtl.css 568 B
build/block-library/blocks/navigation-link/editor.css 570 B
build/block-library/blocks/navigation-link/style-rtl.css 94 B
build/block-library/blocks/navigation-link/style.css 94 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 300 B
build/block-library/blocks/navigation-submenu/editor.css 299 B
build/block-library/blocks/navigation-submenu/style-rtl.css 195 B
build/block-library/blocks/navigation-submenu/style.css 195 B
build/block-library/blocks/navigation-submenu/view.min.js 343 B
build/block-library/blocks/navigation/editor-rtl.css 1.72 kB
build/block-library/blocks/navigation/editor.css 1.72 kB
build/block-library/blocks/navigation/style-rtl.css 1.48 kB
build/block-library/blocks/navigation/style.css 1.47 kB
build/block-library/blocks/navigation/view.min.js 2.74 kB
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 377 B
build/block-library/blocks/page-list/editor.css 377 B
build/block-library/blocks/page-list/style-rtl.css 198 B
build/block-library/blocks/page-list/style.css 198 B
build/block-library/blocks/paragraph/editor-rtl.css 157 B
build/block-library/blocks/paragraph/editor.css 157 B
build/block-library/blocks/paragraph/style-rtl.css 273 B
build/block-library/blocks/paragraph/style.css 273 B
build/block-library/blocks/post-author/editor-rtl.css 210 B
build/block-library/blocks/post-author/editor.css 210 B
build/block-library/blocks/post-author/style-rtl.css 182 B
build/block-library/blocks/post-author/style.css 181 B
build/block-library/blocks/post-comments-form/style-rtl.css 140 B
build/block-library/blocks/post-comments-form/style.css 140 B
build/block-library/blocks/post-comments/style-rtl.css 360 B
build/block-library/blocks/post-comments/style.css 359 B
build/block-library/blocks/post-content/editor-rtl.css 138 B
build/block-library/blocks/post-content/editor.css 138 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B
build/block-library/blocks/post-excerpt/editor.css 73 B
build/block-library/blocks/post-excerpt/style-rtl.css 69 B
build/block-library/blocks/post-excerpt/style.css 69 B
build/block-library/blocks/post-featured-image/editor-rtl.css 398 B
build/block-library/blocks/post-featured-image/editor.css 398 B
build/block-library/blocks/post-featured-image/style-rtl.css 143 B
build/block-library/blocks/post-featured-image/style.css 143 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 378 B
build/block-library/blocks/post-template/style.css 379 B
build/block-library/blocks/post-terms/style-rtl.css 73 B
build/block-library/blocks/post-terms/style.css 73 B
build/block-library/blocks/post-title/style-rtl.css 60 B
build/block-library/blocks/post-title/style.css 60 B
build/block-library/blocks/preformatted/style-rtl.css 103 B
build/block-library/blocks/preformatted/style.css 103 B
build/block-library/blocks/pullquote/editor-rtl.css 198 B
build/block-library/blocks/pullquote/editor.css 198 B
build/block-library/blocks/pullquote/style-rtl.css 378 B
build/block-library/blocks/pullquote/style.css 378 B
build/block-library/blocks/pullquote/theme-rtl.css 167 B
build/block-library/blocks/pullquote/theme.css 167 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 262 B
build/block-library/blocks/query-pagination/editor.css 255 B
build/block-library/blocks/query-pagination/style-rtl.css 234 B
build/block-library/blocks/query-pagination/style.css 231 B
build/block-library/blocks/query-title/editor-rtl.css 85 B
build/block-library/blocks/query-title/editor.css 85 B
build/block-library/blocks/query/editor-rtl.css 131 B
build/block-library/blocks/query/editor.css 132 B
build/block-library/blocks/quote/style-rtl.css 187 B
build/block-library/blocks/quote/style.css 187 B
build/block-library/blocks/quote/theme-rtl.css 220 B
build/block-library/blocks/quote/theme.css 222 B
build/block-library/blocks/rss/editor-rtl.css 202 B
build/block-library/blocks/rss/editor.css 204 B
build/block-library/blocks/rss/style-rtl.css 289 B
build/block-library/blocks/rss/style.css 288 B
build/block-library/blocks/search/editor-rtl.css 165 B
build/block-library/blocks/search/editor.css 165 B
build/block-library/blocks/search/style-rtl.css 374 B
build/block-library/blocks/search/style.css 375 B
build/block-library/blocks/search/theme-rtl.css 64 B
build/block-library/blocks/search/theme.css 64 B
build/block-library/blocks/separator/editor-rtl.css 99 B
build/block-library/blocks/separator/editor.css 99 B
build/block-library/blocks/separator/style-rtl.css 250 B
build/block-library/blocks/separator/style.css 250 B
build/block-library/blocks/separator/theme-rtl.css 172 B
build/block-library/blocks/separator/theme.css 172 B
build/block-library/blocks/shortcode/editor-rtl.css 474 B
build/block-library/blocks/shortcode/editor.css 474 B
build/block-library/blocks/site-logo/editor-rtl.css 462 B
build/block-library/blocks/site-logo/editor.css 464 B
build/block-library/blocks/site-logo/style-rtl.css 153 B
build/block-library/blocks/site-logo/style.css 153 B
build/block-library/blocks/site-tagline/editor-rtl.css 86 B
build/block-library/blocks/site-tagline/editor.css 86 B
build/block-library/blocks/site-title/editor-rtl.css 84 B
build/block-library/blocks/site-title/editor.css 84 B
build/block-library/blocks/social-link/editor-rtl.css 165 B
build/block-library/blocks/social-link/editor.css 165 B
build/block-library/blocks/social-links/editor-rtl.css 812 B
build/block-library/blocks/social-links/editor.css 811 B
build/block-library/blocks/social-links/style-rtl.css 1.3 kB
build/block-library/blocks/social-links/style.css 1.3 kB
build/block-library/blocks/spacer/editor-rtl.css 307 B
build/block-library/blocks/spacer/editor.css 307 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 471 B
build/block-library/blocks/table/editor.css 472 B
build/block-library/blocks/table/style-rtl.css 481 B
build/block-library/blocks/table/style.css 481 B
build/block-library/blocks/table/theme-rtl.css 188 B
build/block-library/blocks/table/theme.css 188 B
build/block-library/blocks/tag-cloud/style-rtl.css 146 B
build/block-library/blocks/tag-cloud/style.css 146 B
build/block-library/blocks/template-part/editor-rtl.css 636 B
build/block-library/blocks/template-part/editor.css 635 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/term-description/editor-rtl.css 90 B
build/block-library/blocks/term-description/editor.css 90 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 87 B
build/block-library/blocks/verse/style.css 87 B
build/block-library/blocks/video/editor-rtl.css 571 B
build/block-library/blocks/video/editor.css 572 B
build/block-library/blocks/video/style-rtl.css 173 B
build/block-library/blocks/video/style.css 173 B
build/block-library/blocks/video/theme-rtl.css 124 B
build/block-library/blocks/video/theme.css 124 B
build/block-library/common-rtl.css 853 B
build/block-library/common.css 849 B
build/block-library/editor-rtl.css 9.76 kB
build/block-library/editor.css 9.75 kB
build/block-library/index.min.js 146 kB
build/block-library/reset-rtl.css 536 B
build/block-library/reset.css 536 B
build/block-library/style-rtl.css 10.3 kB
build/block-library/style.css 10.3 kB
build/block-library/theme-rtl.css 665 B
build/block-library/theme.css 669 B
build/block-serialization-default-parser/index.min.js 1.09 kB
build/block-serialization-spec-parser/index.min.js 2.79 kB
build/blocks/index.min.js 45.7 kB
build/components/index.min.js 210 kB
build/components/style-rtl.css 15.9 kB
build/components/style.css 15.9 kB
build/compose/index.min.js 10.3 kB
build/core-data/index.min.js 12.3 kB
build/customize-widgets/index.min.js 11.1 kB
build/customize-widgets/style-rtl.css 1.5 kB
build/customize-widgets/style.css 1.49 kB
build/data-controls/index.min.js 614 B
build/data/index.min.js 7.1 kB
build/date/index.min.js 31.5 kB
build/deprecated/index.min.js 428 B
build/dom-ready/index.min.js 304 B
build/dom/index.min.js 4.45 kB
build/edit-navigation/index.min.js 15.3 kB
build/edit-navigation/style-rtl.css 3.71 kB
build/edit-navigation/style.css 3.7 kB
build/edit-post/classic-rtl.css 492 B
build/edit-post/classic.css 494 B
build/edit-post/index.min.js 29 kB
build/edit-post/style-rtl.css 7.19 kB
build/edit-post/style.css 7.18 kB
build/edit-site/index.min.js 28.4 kB
build/edit-site/style-rtl.css 5.36 kB
build/edit-site/style.css 5.36 kB
build/edit-widgets/index.min.js 15.7 kB
build/edit-widgets/style-rtl.css 4.1 kB
build/edit-widgets/style.css 4.1 kB
build/editor/index.min.js 37.4 kB
build/editor/style-rtl.css 3.76 kB
build/editor/style.css 3.75 kB
build/element/index.min.js 3.17 kB
build/escape-html/index.min.js 517 B
build/format-library/index.min.js 5.68 kB
build/format-library/style-rtl.css 571 B
build/format-library/style.css 571 B
build/hooks/index.min.js 1.55 kB
build/html-entities/index.min.js 424 B
build/i18n/index.min.js 3.6 kB
build/is-shallow-equal/index.min.js 501 B
build/keyboard-shortcuts/index.min.js 1.72 kB
build/keycodes/index.min.js 1.3 kB
build/list-reusable-blocks/index.min.js 1.85 kB
build/list-reusable-blocks/style-rtl.css 838 B
build/list-reusable-blocks/style.css 838 B
build/media-utils/index.min.js 2.92 kB
build/notices/index.min.js 845 B
build/nux/index.min.js 2.03 kB
build/nux/style-rtl.css 747 B
build/nux/style.css 743 B
build/plugins/index.min.js 1.83 kB
build/primitives/index.min.js 921 B
build/priority-queue/index.min.js 582 B
build/react-i18n/index.min.js 671 B
build/redux-routine/index.min.js 2.63 kB
build/reusable-blocks/index.min.js 2.28 kB
build/reusable-blocks/style-rtl.css 256 B
build/reusable-blocks/style.css 256 B
build/rich-text/index.min.js 10.6 kB
build/server-side-render/index.min.js 1.5 kB
build/shortcode/index.min.js 1.48 kB
build/token-list/index.min.js 562 B
build/url/index.min.js 1.74 kB
build/viewport/index.min.js 1.02 kB
build/warning/index.min.js 248 B
build/widgets/index.min.js 7.11 kB
build/widgets/style-rtl.css 1.16 kB
build/widgets/style.css 1.16 kB
build/wordcount/index.min.js 1.04 kB

compressed-size-action

@WunderBart WunderBart force-pushed the try/playwright branch 4 times, most recently from 2000d6d to 444a3d7 Compare August 17, 2021 13:17
@gziolo
Copy link
Member

gziolo commented Aug 17, 2021

As far as I can tell CI runs tests with Puppeteer:

Screen Shot 2021-08-17 at 15 37 24

@WunderBart
Copy link
Member Author

As far as I can tell CI runs tests with Puppeteer:

I think it just uses Puppeteer's Chromium. It would break if Puppeteer was used due to the migrated API. You can check that locally by running PWDEBUG=1 npm run test-e2e - it will run headfully with the Playwright's debugging tools open.

@gziolo
Copy link
Member

gziolo commented Aug 17, 2021

As far as I can tell CI runs tests with Puppeteer:

OK, I think you are correct. We have this logic that installs Chromium when the tests are run for the first time rather than during npm install. We will need the same handling for Playwright.

@WunderBart
Copy link
Member Author

WunderBart commented Aug 17, 2021

One of the tests started failing after the last rebase with trunk. Will take a look tomorrow.

packages/e2e-tests/jest.playwright.config.js Show resolved Hide resolved
@@ -0,0 +1,27 @@
module.exports = {
preset: 'jest-playwright-preset',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll lose the failure artifacts feature if we just use the default preset.

@WunderBart WunderBart changed the title Try Playwright [Experiment] Try Playwright Aug 18, 2021
- Type in directly when searching for block.
- Wait for the block to be stable when focused as the layout shifts when the Inserter closes.
- Remove unnecessary waitForInserterCloseAndContentFocus call because Playwright's auto-waiting does that automagically.
- Remove explicit `waits` and remove the unnecessary scrollIntoView call (all thanks to auto-waiting)
Doesn't really seem like we need it anymore?
(Removed accidentally)
It's standing in the way of the Replace button because the test image is really small.
@WunderBart
Copy link
Member Author

WunderBart commented Oct 1, 2021

Regarding some concerns about how testing with CSS animations would impact the speed, here's some data I've collected from running the migrated specs:

Test File Puppeteer Playwright: Animation OFF Playwright: Animation ON Animation Δt
buttons.test.js 12,12s 12,53s 14,5s 1,97s
classic.test.js 13,71s 10,27s 11,56s 1,29s
code.test.js 10,07s 9,66s 11,24s 1,57s
columns.test.js 7,58s 7,79s 8,85s 1,06s
cover.test.js 7,65s 8,2s 9,06s 0,86s
gallery.test.js 11,0s 12,15s 14,61s 2,46s
group.test.js 10,25s 10,13s 11,27s 1,14s
heading.test.js 17,48s 16,43s 19,09s 2,66s
html.test.js 9,77s 8,97s 10,4s 1,43s
image.test.js 34,12s 43,21s 45,86s 2,65s
list.test.js 77,46s 75,4s 82,53s 7,13s
missing.test.js 7,6s 6,94s 7,31s 0,37s
paragraph.test.js 6,62s 6,77s 7,41s 0,64s
post-title.test.js 8,04s 8,32s 9,39s 1,07s
preformatted.test.js 11,54s 11,22s 13,16s 1,95s
query.test.js 17,5s 17,48s 19,7s 2,22s
quote.test.js 49,44s 49,34s 52,55s 3,2s
separator.test.js 6,2s 6,2s 6,93s 0,72s
site-title.test.js 14,01s 13,79s 15,48s 1,69s
spacer.test.js 7,55s 8,22s 9,22s 1,0s
table.test.js 23,38s 26,68s 26,41s -0,27s
Total time: 363,09s 369,68s 406,5s 36,82s

It's hard to say how much more time will animations add when all the tests are run (124 suites in total). Basing on the numbers above, we can probably expect at least a few minutes. I'd say the trade-off would still be worth it but we can always decide later, of course. Just thought it would be interesting to see the actual numbers from this POC :)

@WunderBart
Copy link
Member Author

I'm gathering some more feedback on this proposal via the Make Test blog: https://make.wordpress.org/test/2021/10/04/proposal-migrate-e2e-to-playwright/

@WunderBart WunderBart closed this Jul 18, 2022
@gziolo gziolo deleted the try/playwright branch July 18, 2022 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Type] Technical Prototype Offers a technical exploration into an idea as an example of what's possible
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants