[APM] Metric-powered correlations

[sorenlouv]

The correlations feature in APM is currently based on transaction events and will therefore become inaccurate as the sampling rate decreases.
Transaction metrics events doesn't only retain very few dimensions and can therefore not be used as a direct replacement for transaction events.

Together with the ML team we should investigate whether correlations should be based on a hybrid model that uses both transactions and metrics, and how that can be done.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sophiec20]

The metric data has a limited set of fields that it summarises information by; therefore there is minimal benefit in using it for correlations as it stands. In past conversations we proposed to hold off until such time that is becomes possible for users to a) configure which fields they can summarise by and b) to have improved workflow/guidance on how to extend data collection for useful labelled fields. Once this is available, then hybrid correlations would give speed ups to initial results, although final results would be unaffected.

A statistically sound sampling strategy would mean that correlations remains accurate enough for root cause analysis. If the motivation for this work is mainly accuracy, then we should additionally consider some work to assess the optimal configuration/implementation for sampling.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.