Boost: Re-enable Critical CSS and Concatenate E2E tests #44105

LiamSarsfield · 2025-06-26T14:07:23Z

Addresses HOG-188: Enable Critical CSS E2E tests

Proposed changes:

This PR re-enables 2 out of 7 Jetpack Boost e2e test suites that were temporarily disabled in commit 0eb34d93ca7522549dcec7ea35c09d4f39ef5406 due to flakiness:

Re-enabled test suites:

Jetpack Boost - Critical CSS (specs/critical-css)
Jetpack Boost - Concatenate JS and CSS (specs/concatenate)

Test suites still disabled (to be addressed in separate PRs):

Jetpack Boost - Base (specs/base)
Jetpack Boost - Modules (specs/modules)
Jetpack Boost - Page Cache (specs/page-cache)
Jetpack Boost - Image CDN (specs/image-cdn)
Jetpack Boost - Image Guide (specs/image-guide)

Jetpack product discussion

N/A

Does this pull request change what data or activity we track or use?

No

How this PR resolves the failing tests:

1. Extended timeout for Critical CSS generation

Increased timeout from 3 to 4 minutes for Critical CSS meta info visibility
Provides more reliable waiting for CSS generation to complete

2. Added error simulation plugin

Created e2e-critical-css-force-errors.php plugin to simulate Critical CSS generation errors
Enables more controlled testing of error scenarios
Plugin is properly activated/deactivated in test lifecycle

3. Improved test flow and assertions

Removed flaky progress indicator assertions that were timing-dependent
Improved Critical CSS advanced recommendations test flow
Better test setup with proper plugin management

4. Removed temporary test.skip() calls

Re-enabled tests that were individually skipped in various test files
Tests now run reliably with the implemented fixes

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Testing instructions:

From the projects/plugins/boost/tests/e2e directory:

Via ngrok (Recommended, but requires a subscription)

Replace $tunnel_url with your actual ngrok URL

Install E2E test dependencies:
```
pnpm install
```

Start the Docker environment and tunnel:

pnpm run env:up && TUNNEL_URL=$tunnel_url pnpm run tunnel:up

Start your ngrok tunnel

ngrok http --url=$tunnel_url http://localhost:8889/

Run the specific re-enabled test suites:

 TUNNEL_URL=$tunnel_url pnpm run test:run specs/critical-css
 TUNNEL_URL=$tunnel_url pnpm run test:run specs/concatenate

Via localtunnel (Not recommended as it may be slow)

Install E2E test dependencies:
```
pnpm install
```
Start the Docker environment and tunnel:
```
pnpm run env:up && pnpm run tunnel:up
```

Run the specific re-enabled test suites:

 pnpm run test: run specs/critical-css
 pnpm run test: run specs/concatenate

Expected behavior:

All Critical CSS tests should pass without skipping or timeout errors
Concatenate JS and CSS tests should run reliably and complete successfully
The new e2e-critical-css-force-errors.php plugin should be properly activated/deactivated during tests
Test execution should complete within the extended 4-minute timeout for Critical CSS generation
No test.skip() calls should remain in the re-enabled test files
Tests should pass consistently when run multiple times

Verification steps:

Check that tests are running in CI:
- The re-enabled test suites should appear in the E2E test matrix
- CI should execute both "Jetpack Boost - Critical CSS" and "Jetpack Boost - Concatenate JS and CSS" jobs
Verify error simulation works:
- Critical CSS tests should properly handle simulated errors via the force-errors plugin
- Plugin should be automatically managed (activated/deactivated) during test lifecycle
Test stability:
- Run each test suite 3-5 times to ensure consistency
- No intermittent failures or timing-related issues should occur

github-actions · 2025-06-26T14:08:24Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Review, ...).
✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

Follow this PR Review Process:

Ensure all required checks appearing at the bottom of this PR are passing.
Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
You can use GitHub's Reviewers functionality to request a review.
When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

Boost plugin:

No scheduled milestone found for this plugin.

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

jp-launch-control · 2025-06-26T14:14:13Z

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

- Introduced a new plugin `e2e-critical-css-force-errors` to simulate errors in Critical CSS Advanced Recommendations. - Updated e2e tests to activate the new plugin during the Critical CSS module tests. - Added the new plugin to the Docker configuration for proper testing environment setup.

… the 'Regenerate' button. This improves the test coverage for user interactions within the Critical CSS advanced recommendations.

…per activation/deactivation of the e2e-critical-css-force-errors plugin in tests. This enhances the reliability of the Critical CSS module tests.

…s and ensure visibility of critical CSS meta information. This streamlines the test flow and focuses on essential assertions.

LiamSarsfield · 2025-06-30T10:47:45Z

projects/plugins/boost/tests/e2e/lib/pages/wp-admin/JetpackBoostPage.js

@@ -103,7 +103,7 @@ export default class JetpackBoostPage extends WpPage {

 	async waitForCriticalCssMetaInfoVisibility() {
 		const selector = '[data-testid="critical-css-meta"]';
-		return this.waitForElementToBeVisible( selector, 3 * 60 * 1000 );
+		return this.waitForElementToBeVisible( selector, 4 * 60 * 1000 );


This function is executed when waiting for the local Critical CSS generation to finish. I've given it an extra minute in case the local instance cannot finish generating within 3 minutes.

I hope that's enough...

Out of scope for this, just thoughts.

Seeing this made me wonder if we shouldn't just limit the generation to the home page. After all, we just want to see that it actually works. We don't want to generate for the whole website...

That's a great idea, I might discuss this with you offline regarding the lift of this. Or create a Linear issue for Triage for this if waiting for the Critical CSS continues to be flaky.

kraftbj · 2025-06-30T20:19:48Z

I'm still seeing a lot of timeouts for me. I believe I saw some chatter about localtunnel vs other solutions. I'm curious if we need to look at that layer. Let me know if you need my logs.

A few React imports managed to get incorrectly switched to types rather than full imports. This should fix them.

LiamSarsfield · 2025-06-30T22:02:24Z

I'm still seeing a lot of timeouts for me. I believe I saw some chatter about localtunnel vs other solutions. I'm curious if we need to look at that layer. Let me know if you need my logs.

@kraftbj
There was discussions here: pc9hqz-2IT-p2 / #36582 regarding moving fully over to ngrok but it didn't gain traction from what I understand. I'm also seeing the same timeout results, but I deemed it due to my geographic region which was also mentioned in the post pc9hqz-2IT-p2#why-is-localtunnel-slow. Though if it's also slow for you to the point they are timing out, I think it's worth having that conversation again. If you have an ngrok subscription, does it have the same issues?
To use ngrok, assuming the e2e testing environment is already up and running, and reset/fresh:

pnpm run tunnel: down

ngrok http --url=$TUNNEL_URL http://localhost:8889/

TUNNEL_URL=$TUNNEL_URL pnpm run tunnel: up

pnpm env:reset && TUNNEL_URL=$TUNNEL_URL pnpm test: run specs/critical-css

If this works for you, then I think we should look at potentially recommending another service/ngrok over localtunnel to run the tests locally (potentially even remotely to speed up builds).

dilirity

I still find it weird that tests are flaky... Considering this is local critical css generation, even on local tunnel things should happen fast since it's localhost when it's running the process...

I really hope the changes you made finally address the flakiness 🤞🏻

dilirity · 2025-07-02T07:45:09Z

projects/plugins/boost/tests/e2e/lib/pages/wp-admin/JetpackBoostPage.js

@@ -103,7 +103,7 @@ export default class JetpackBoostPage extends WpPage {

 	async waitForCriticalCssMetaInfoVisibility() {
 		const selector = '[data-testid="critical-css-meta"]';
-		return this.waitForElementToBeVisible( selector, 3 * 60 * 1000 );
+		return this.waitForElementToBeVisible( selector, 4 * 60 * 1000 );


I hope that's enough...

Out of scope for this, just thoughts.

Seeing this made me wonder if we shouldn't just limit the generation to the home page. After all, we just want to see that it actually works. We don't want to generate for the whole website...

projects/plugins/boost/tests/e2e/specs/critical-css/critical-css.test.js

LiamSarsfield added 2 commits June 26, 2025 15:04

Remove flaky test skips from Boost plugin end-to-end tests

2af5c15

changelog

3d09341

LiamSarsfield changed the title ~~Fix/boost/e2e-tests~~ Boost: Attempting to fix flaky e2e tests Jun 26, 2025

github-actions bot added [Plugin] Boost A feature to speed up the site and improve performance. [Status] In Progress [Tests] Includes Tests E2E Tests labels Jun 26, 2025

github-actions bot added the [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. label Jun 26, 2025

Adding back Boost e2e tests to matrix

0595bd3

github-actions bot added the Actions GitHub actions used to automate some of the work around releases and repository management label Jun 26, 2025

LiamSarsfield added the [Type] Janitorial label Jun 26, 2025

LiamSarsfield added 2 commits June 26, 2025 17:59

Wait for Critical CSS to be done

920ac1a

github-actions bot added the Docker label Jun 27, 2025

LiamSarsfield added 10 commits June 27, 2025 13:08

Merge remote-tracking branch 'origin/trunk' into fix/boost/e2e-tests

f90c661

- Make PHP linting happy

814143e

Debug workflow

d9e3143

Make phpcs happy

f290ad5

Enhance e2e test for Critical CSS module by adding a click action for…

cb835f4

… the 'Regenerate' button. This improves the test coverage for user interactions within the Critical CSS advanced recommendations.

Increase timeout for critical CSS meta info visibility and ensure pro…

7518607

…per activation/deactivation of the e2e-critical-css-force-errors plugin in tests. This enhances the reliability of the Critical CSS module tests.

Refactor Critical CSS module tests to remove progress indicator check…

f3ed9d4

…s and ensure visibility of critical CSS meta information. This streamlines the test flow and focuses on essential assertions.

Remove tests that are still flaky

47fe9a1

Remove e2e tests that are still failing

696c390

Merge branch 'trunk' into fix/boost/e2e-tests

9d8ca22

LiamSarsfield changed the title ~~Boost: Attempting to fix flaky e2e tests~~ Boost: Re-enable Critical CSS and Concatenate E2E tests Jun 30, 2025

LiamSarsfield added 2 commits June 30, 2025 11:18

Update changelog

0e85250

Merge branch 'trunk' into fix/boost/e2e-tests

383c1f8

LiamSarsfield commented Jun 30, 2025

View reviewed changes

LiamSarsfield requested review from kraftbj, dilirity and a team June 30, 2025 10:57

LiamSarsfield marked this pull request as ready for review June 30, 2025 10:57

Merge branch 'trunk' into fix/boost/e2e-tests

ce90ef8

LiamSarsfield mentioned this pull request Jun 30, 2025

Boost: Re-enable Base/Common E2E tests #44140

Merged

3 tasks

Present tense for changelog

695e54d

LiamSarsfield added [Status] Needs Review This PR is ready for review. and removed [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. [Status] In Progress labels Jun 30, 2025

LiamSarsfield mentioned this pull request Jun 30, 2025

Boost: Re-enable Module E2E tests #44141

Draft

3 tasks

Merge branch 'trunk' into fix/boost/e2e-tests

4bf350d

LiamSarsfield and others added 3 commits June 30, 2025 21:30

Merge branch 'trunk' into fix/boost/e2e-tests

2d4fb8e

Fix some runtime errors after #44136

24184ff

A few React imports managed to get incorrectly switched to types rather than full imports. This should fix them.

Merge branch 'fix/boost-after-44136' into fix/boost/e2e-tests

bc4bed1

github-actions bot added the [Package] Jetpack mu wpcom WordPress.com Features label Jun 30, 2025

LiamSarsfield removed the [Package] Jetpack mu wpcom WordPress.com Features label Jun 30, 2025

LiamSarsfield added 4 commits June 30, 2025 23:03

Merge branch 'trunk' into fix/boost/e2e-tests

9faaf38

Merge branch 'trunk' into fix/boost/e2e-tests

f681fd9

Merge branch 'trunk' into fix/boost/e2e-tests

a81bc8f

Merge branch 'trunk' into fix/boost/e2e-tests

abb7300

kraftbj approved these changes Jul 1, 2025

View reviewed changes

dilirity approved these changes Jul 2, 2025

View reviewed changes

Merge branch 'trunk' into fix/boost/e2e-tests

af41442

LiamSarsfield merged commit 760d8ca into trunk Jul 2, 2025
66 checks passed

LiamSarsfield deleted the fix/boost/e2e-tests branch July 2, 2025 08:11

github-actions bot removed the [Status] Needs Review This PR is ready for review. label Jul 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Boost: Re-enable Critical CSS and Concatenate E2E tests #44105

Boost: Re-enable Critical CSS and Concatenate E2E tests #44105

Uh oh!

LiamSarsfield commented Jun 26, 2025 •

edited by kraftbj

Loading

Uh oh!

github-actions bot commented Jun 26, 2025 •

edited

Loading

Uh oh!

jp-launch-control bot commented Jun 26, 2025 •

edited

Loading

Uh oh!

LiamSarsfield Jun 30, 2025 •

edited

Loading

Uh oh!

dilirity Jul 2, 2025

Uh oh!

LiamSarsfield Jul 2, 2025

Uh oh!

kraftbj commented Jun 30, 2025

Uh oh!

LiamSarsfield commented Jun 30, 2025

Uh oh!

dilirity left a comment

Uh oh!

dilirity Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Boost: Re-enable Critical CSS and Concatenate E2E tests #44105

Boost: Re-enable Critical CSS and Concatenate E2E tests #44105

Uh oh!

Conversation

LiamSarsfield commented Jun 26, 2025 • edited by kraftbj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes:

Re-enabled test suites:

Test suites still disabled (to be addressed in separate PRs):

Jetpack product discussion

Does this pull request change what data or activity we track or use?

How this PR resolves the failing tests:

1. Extended timeout for Critical CSS generation

2. Added error simulation plugin

3. Improved test flow and assertions

4. Removed temporary test.skip() calls

Other information:

Testing instructions:

Via ngrok (Recommended, but requires a subscription)

Via localtunnel (Not recommended as it may be slow)

Expected behavior:

Verification steps:

Uh oh!

github-actions bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jp-launch-control bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Summary

Uh oh!

LiamSarsfield Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dilirity Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

LiamSarsfield Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

kraftbj commented Jun 30, 2025

Uh oh!

LiamSarsfield commented Jun 30, 2025

Uh oh!

dilirity left a comment

Choose a reason for hiding this comment

Uh oh!

dilirity Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiamSarsfield commented Jun 26, 2025 •

edited by kraftbj

Loading

github-actions bot commented Jun 26, 2025 •

edited

Loading

jp-launch-control bot commented Jun 26, 2025 •

edited

Loading

LiamSarsfield Jun 30, 2025 •

edited

Loading