Wrong detection of browser idle in applied vs simulated slowdown which affects TTI

[filrak]

Hey! I already had a call with one of your colleagues some time ago where we confirmed the issue.

The case is - things like https://web.dev/ that have underlyimg lighthouse (probably with applied slowdown) measure TTI wrongly. To be more precise they don't detect browser idle at the right moment therefore things that are prefetched in the background are considered still impacting TTI.

This ends up with a huge difference in real-world performance vs LH result in our product that is highly relying on prefetching. The only "working" configuration is "Simulated 4G" seems to measure it accurately.

Since many clients rely on LH score it's much much harder to properly pitch them and tell that product is fast when the score says something totally different.

Provide the steps to reproduce

1. Run LH on https://demo.vuestorefront.io/ locally simulated 4g
2. Run the same audit on applied 4g and web.dev/measure

What is the current behavior?

TTI is measured wrongly and LH result highly differs between simulated and applied 4G (usually around 5x vs 8x)

What is the expected behavior?

Browser idle should be detected in the right moment before prefetching happens.

Environment Information

• Affected Channels: DevTools and external sources
• Lighthouse version: latest
• Node.js version:
• Operating System: Tested in W10 and MacOS Mojave

Related issues

[filrak]

Might be related to: #9483

[patrickhulce]

Thanks for filing @filrak! Would you mind elaborating a bit? Specifically...

they don't detect browser idle at the right moment

What is the right moment? And who did you speak with that "confirmed" that the current strategy is incorrect?

LH result highly differs between simulated and applied 4G

The runs I'm seeing differ by about 14 points with TTI varying by ~2 seconds.

While on the high end of score variance, this is relatively within the expected range of volatile performance measurements. You can read more about the factors affecting performance metric variance in our explainer.

properly pitch them and tell that product is fast when the score says something totally different.

The applied throttling performance metrics are what users actually experience on a cold load, slow 4G, mid-tier mobile phone. Speed Index in particular is the worst of the metrics on this page as the images take ~8 seconds to load. Is this the same experience the team is aware of when considering the product to be fast? I ask because if the frustration comes from the usage of a slow 4G, mid-tier mobile phone as our model, that's a very different discussion than "the metrics are computed incorrectly".

[filrak]

Hey!, Thanks for the answer. I think I need to clarify a few things from my message that may be misleading. Sorry for inconvenience .

First of all I wrongly communicated the thing about difference with "applied" and "sinulated".
The main problem is not a local Lighthouse audit but the one made by your external tools like web.dev

page speed insights (in polish but location remains the same, TTI 21 sec)

The assumption came from a fact that during a call with your friend from Google (which name I forgot because it was some time ago but I'll come back once I find it in a mailbox) we measured it locally and he suspected the problem might be in the applied 4g setting because we spotted a significant difference between those two and he admitted that prefetching might be the reason of this. If you're saying it's an acceptable difference then it's not the case and the problem applies only to those external tools using LH under the hood.

Then I would like to know what could cause such difference between your external tools and local runs regarding TTI. To me it seems like a bug but I might be terribly wrong aswell.

[connorjclark]

@patrickhulce here's the trace from LR: https://gist.github.com/connorjclark/e9a44c9d3a2434929041034e34e15d30

[patrickhulce]

Ah thanks for clarifying @filrak! And thanks @connorjclark for the raw data!

The difference between these two environments for your page comes down to the lack of support for HTTP/2 on the infrastructure that web.dev runs on. Because the site delivers lots of files it is particularly punished by the lack of H2 support there. This bug is captured by #7326 as well as internally assigned to the appropriate team. Feel free to subscribe to that issue for any progress updates, but I'll go ahead and close this one as a duplicate for now.

[filrak]

Thank you very much for the clarification :)