Evaluate browser metrics against webvital metrics

[ankur22]

• Ensure we’re not working with duplicate metrics (prefixed browser_). We should gather the measurements from one source and not both for the same metric.
• We need to take a look at the existing browser metrics and evaluate whether they are still needed when we have the core webvitals.
• Work with Faro team to ensure that our core webvitals measurement values are the same as the webvitals measured by Faro.

Current browser and webvital metrics from the summary at the end of the test:

    browser_dom_content_loaded.......: avg=108.7ms  min=10µs     med=952µs    max=802.96ms p(75)=1.88ms   p(90)=330.51ms p(95)=566.73ms
     browser_first_meaningful_paint...: avg=1.27s    min=132.07ms med=1.27s    max=2.41s    p(75)=1.84s    p(90)=2.18s    p(95)=2.3s    
     browser_first_paint..............: avg=122.45ms min=112.84ms med=122.45ms max=132.07ms p(75)=127.26ms p(90)=130.15ms p(95)=131.11ms
     browser_loaded...................: avg=312.03ms min=74µs     med=1.73ms   max=1.39s    p(75)=357.65ms p(90)=933.52ms p(95)=1.16s   
     data_received....................: 18 MB 3.2 MB/s
     data_sent........................: 57 kB 10 kB/s
     http_req_connecting..............: avg=6.55ms   min=0s       med=0s       max=176ms    p(75)=0s       p(90)=29.4ms   p(95)=37.09ms 
     http_req_duration................: avg=129.33ms min=341µs    med=120.22ms max=476.59ms p(75)=212.86ms p(90)=241.5ms  p(95)=281.22ms
     http_req_receiving...............: avg=75.53ms  min=0s       med=50.5ms   max=440ms    p(75)=117ms    p(90)=172.1ms  p(95)=192.55ms
     http_req_sending.................: avg=0s       min=0s       med=0s       max=0s       p(75)=0s       p(90)=0s       p(95)=0s      
     http_req_tls_handshaking.........: avg=3.92ms   min=0s       med=0s       max=90ms     p(75)=0s       p(90)=18.3ms   p(95)=22ms    
     http_reqs........................: 130   23.199591/s
     iteration_duration...............: avg=5.6s     min=5.6s     med=5.6s     max=5.6s     p(75)=5.6s     p(90)=5.6s     p(95)=5.6s    
     iterations.......................: 1     0.178458/s
     vus..............................: 1     min=1       max=1
     vus_max..........................: 1     min=1       max=1
     web_vitals.......................:       
       cumulative_layout_shift........: avg=0.003211 min=0.003211 med=0.003211 max=0.003211 p(75)=0.003211 p(90)=0.003211 p(95)=0.003211
         ratings......................: good=1 needs_improvement=0 poor=0
       first_contentful_paint.........: avg=1.1s     min=223.3ms  med=1.1s     max=1.99s    p(75)=1.54s    p(90)=1.81s    p(95)=1.9s    
         ratings......................: good=1 needs_improvement=1 poor=0
       first_input_delay..............: avg=46.5ms   min=46.5ms   med=46.5ms   max=46.5ms   p(75)=46.5ms   p(90)=46.5ms   p(95)=46.5ms  
         ratings......................: good=1 needs_improvement=0 poor=0
       interaction_to_next_paint......: avg=64ms     min=64ms     med=64ms     max=64ms     p(75)=64ms     p(90)=64ms     p(95)=64ms    
         ratings......................: good=1 needs_improvement=0 poor=0
       largest_content_paint..........: avg=383.3ms  min=383.3ms  med=383.3ms  max=383.3ms  p(75)=383.3ms  p(90)=383.3ms  p(95)=383.3ms 
         ratings......................: good=1 needs_improvement=0 poor=0
       time_to_first_byte.............: avg=215.35ms min=89.4ms   med=215.35ms max=341.3ms  p(75)=278.32ms p(90)=316.11ms p(95)=328.7ms 
         ratings......................: good=2 needs_improvement=0 poor=0

[ankur22]

Existing Metrics (before web vitals)

browser_first_contentful_paint

This is a duplicate of webvital_first_contentful_paint, so we should remove it. This was removed but accidentally added back in after some confusion during a merge conflict when merging #838 into main.

We want to work with the web vital measurements instead of the CDP measurements so that we're consistent with the rest of the web vital metrics. The CDP measurement is considerably smaller than what we get back from the js web vital library, but if we compare with another industry standard tool (lighthouse) the first_contentful_paint measurement from the JS library matches up with what lighthouse shows:

From lighthouse:

From xk6-browser (both CDP and web vital versions):

[ankur22]

browser_first_paint

This metric measures the time it takes for anything to be rendered to the screen. It's usually a background colour. It has generally been superseded by webvital_first_contentful_paint. Lighthouse also doesn't use the first_paint metric to evaluate the performance of the website either.

Is it valuable information to know when the first_paint took place vs the first_contentful_paint? If there's a large gap between the two, then would that help resolve an issue in the frontend?

[ankur22]

browser_dom_content_loaded

You can find more information for this metric here. In summary, it's when the html file for the page has been loaded, but non of the dependencies. This is prior to when the rendering takes place.

Similar to browser_first_paint, is it valuable to know how long it take to load the html file and compare that against the rendering of the page? Maybe a large browser_dom_content_loaded value could surface that the html file is too large and should be refactored?

Lighthouse doesn't display this metric, although Chrome does display this in the network tab in the inspection window.

[ankur22]

browser_loaded

You can find more information on this metric here. In summary, this is fired when the html file and its dependencies (including content in iframes) have loaded, but not started rendering.

Is it useful information to gather to work out how long the dependencies take (browser_dom_content_loaded will needed to be negated)? A large browser_loaded value could show us that there are too many dependencies or they're too large?

Lighthouse doesn't display this metric, although Chrome does display this in the network tab in the inspection window.

[ka3de]

Nice work @ankur22 !
There is a significant difference on first_contentful_paint between CDP and WebVitals script.
One question. I understand in the end we decided to not compare with Faro by now as it's still a project in a preliminary state?

[ankur22]

One question. I understand in the end we decided to not compare with Faro by now as it's still a project in a preliminary state?

Good point, i've not concluded on that yet. I did take a look at the Faro code, and they also use the same library (import) to calculate web vitals (setup code), so the results should match. I'll do some tests though and share the results in this issue.

[ka3de]

Good point, i've not concluded on that yet. I did take a look at the Faro code, and they also use the same library (import) to calculate web vitals (setup code), so the results should match. I'll do some tests though and share the results in this issue.

Got it! Well if they use the same library I guess results should be coherent and we're good to go 👍 Thanks!

[ankur22]

After experimenting with faro and k6-browser, i've found no difference in the web vital measurements, which was expected since they use the same library. Here is the result of the final experiment:

Result from Faro:

Result from k6-browser: