[Alerting] Opening an alerting flyout is very slow

[sorenlouv]

Background
When opening the alerting flyout there is a significant delay before the user can interact with the ui. This is quite painful if a user wants to create multiple alerts.

It turns out that the request to api/alerts/_health is blocking and thus causes the flyout component to hang until it has been loaded. Other requests do not load in parallel and the user has to wait additional time for those to load (eg. actions) afterwards.
This means that in my example below it takes more than 4 seconds (2.33s + 1.40s) from the user opens the flyout until they can interact with the actions.

Question
It would be very helpful if the health check could become non-blocking, and thus allow the alerting flyout to render while it is loading.
I understand that this will cause some flickering for those users that do not have TLS enabled etc but this will be a benefit for the large majority of users who has alerting setup correctly and just want to interact with the alerting component.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

I switched to a different cluster that turns out to be even slower than the previous one:

It takes almost 20 seconds from the user clicks a button until the alerting flyout is ready.
Again, the health endpoint is particularly slow. While there might be room for optimizing this I still thing the most important is to make it non-blocking.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[ymao1]

@sqren @elastic/kibana-alerting-services

Do we know why the health check is returning so slowly? It seems like it might be better to solve that underlying issue rather than switching to a non-blocking health endpoint? I wasn't able to reproduce the slowness locally so I introduced a 5-second delay and switched to a non-blocking health check. Even showing the create alert form for 5 seconds and then showing the TLS message seems like a distracting user experience to me:

I imagine if the health check is taking 20 seconds, a user would have enough time to fill out a couple of fields and maybe even try to save before being told that they need an encryption key. Or, as you mentioned, this would only happen to a small minority of users? What do you think?

[sorenlouv]

Even showing the create alert form for 5 seconds and then showing the TLS message seems like a distracting user experience to me:

I think this is the wrong scenario to optimise for. Users who don't have TLS enabled won't be able to create alerts. Why would they keep coming back to the alerting flyout day in, day out, multiple times a day? They'll open the flyout once, figure out they need to enable TLS and either enable it or decide against it and never come back.
The decision to wait for the TLS check is hurting the 99% of use cases who have already setup TLS and just want to create an alert.

[sorenlouv]

I imagine if the health check is taking 20 seconds, a user would have enough time to fill out a couple of fields

Yes, this is my point. Instead of blocking the ui we can let the user start filling out the alert while we perform the check in the background.

and maybe even try to save before being told that they need an encryption key.

The save button could be disabled with a text on hover letting the user know that we are performing a background check.

[ymao1]

Gotcha. Sounds like very few users would encounter the weird UX so I'm fine with that.