Rework the way we currently detect regressions in build time metrics #40076
Labels
Comments
|
/cc @Karm (mandrel), @ebullient (metrics), @galderz (mandrel), @jmartisk (metrics) |
|
Go for it! Lets spot those frogs boiling earlier :) |
|
I love this idea! |
|
So IIUC, the same checks, but with alerts that won't block merging, and a nice infra with lots of history to help investigation? Sounds great, +1 :) |
This was referenced May 27, 2024
zakkak
added a commit
to zakkak/quarkus
that referenced
this issue
Jun 11, 2024
As part of quarkusio#40076 is now implemented and we are gathering the necessary stats in a database there is no longer the need for these tests to exist. Note that automated alerting in case of regressions is still not setup but we now have a way to go back in time and see which (few) patches resulted in the regression, even if we don't spot it immediately.
zakkak
added a commit
to zakkak/quarkus
that referenced
this issue
Jun 12, 2024
As part of quarkusio#40076 is now implemented and we are gathering the necessary stats in a database there is no longer the need for these tests to exist. Note that automated alerting in case of regressions is still not setup but we now have a way to go back in time and see which (few) patches resulted in the regression, even if we don't spot it immediately.
holly-cummins
pushed a commit
to holly-cummins/quarkus
that referenced
this issue
Jul 31, 2024
As part of quarkusio#40076 is now implemented and we are gathering the necessary stats in a database there is no longer the need for these tests to exist. Note that automated alerting in case of regressions is still not setup but we now have a way to go back in time and see which (few) patches resulted in the regression, even if we don't spot it immediately.
|
Status update: ✔️ Data are being collected from CI runs on main (see #39784) |
danielsoro
pushed a commit
to danielsoro/quarkus
that referenced
this issue
Sep 20, 2024
As part of quarkusio#40076 is now implemented and we are gathering the necessary stats in a database there is no longer the need for these tests to exist. Note that automated alerting in case of regressions is still not setup but we now have a way to go back in time and see which (few) patches resulted in the regression, even if we don't spot it immediately.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
#36108 was an attempt to detect regressions in native builds when certain metrics are outside a given range. Unfortunately this doesn't seem to work well in practice. The main reason seems to be that multiple PRs gradually increase the metrics without hitting the threshold. Then a new PR that happens to increase the metrics a bit more triggers a failure. Although this PR might not be responsible for the total increase (that resulted in hitting the threshold) it is the one being blocked.
Implementation ideas
A thought we had within the mandrel team (cc @Karm @jerboaa) and we are working towards it is the following.
We would like to start collecting data from Quarkus CI runs (initially from runs on
mainand lately probably from PRs as well). This will allow us to observe the change over time (as show in #39674 (comment)) instead of just when we hit a threshold.Next we would ideally like to feed these data to a tool with anomaly detection (possibly https://horreum.hyperfoil.io/) in order to get automated alerts when something seems wrong. That could be:
Related PRs:
The text was updated successfully, but these errors were encountered: