Wildly inconsistent results in headless mode

[rdsubhas]

Provide the steps to reproduce

1. Run without headless mode: lighthouse --view "https://www.omio.com.br" a few times

2. Run in headless mode: lighthouse --view --chrome-flags="--headless" "https://www.omio.com.br"

What is the current behavior?

More than 300% difference between headless and non-headless modes.

What is the expected behavior?

Regular lighthouse variance, less than 40%.

A lot of community/CI tools run using puppeteer, which defaults to chrome headless mode. So would be useful to at least have prominent documentation saying headless mode is not recommended.

Environment Information

• Affected Channels: CLI
• Lighthouse version: 5.6.0
• Chrome version: 78.0.3904.108
• Node.js version: v10.16.2
• Operating System: MacOS 10.14.6

Related issues

• headless mode causes extreme variations in lighthouse, allow override sitespeedio/plugin-lighthouse#38

[patrickhulce]

Thanks for filing @rdsubhas! That URL returns a bot-specific interstitial page with status code 405 when requested by headless Chrome. If the page is intentionally varying its output, then obviously you'll get variable results when measuring it.

Example Screenshot

[rdsubhas]

Ah great thanks!