[Lens] All layers are cleared when switching from multi-layer XY chart to any other XY chart

[cchaos]

Kibana version: 7.9/Master

Describe the bug:

Lens should be able to handle switching between XY chart types when there are multiple layers, but instead it tells me it's not going to work, then when I do it anyway, all layers are gone and the chart switcher makes it look like it never switched.

Steps to reproduce:

1. Create any XY type of chart (i.e. Bar)
2. Add a second layer
3. Try switching to a different type of XY chart (i.e. Line)

Screenshots (if relevant):

https://share.getcloudapp.com/4gujG7nm

Errors in browser console (if relevant):

No errors thrown anywhere

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[flash1293]

This might be caused by #64613

Chart switches among sub types use the suggestion system now but the suggestion system can’t handle multiple layers (at least that was the case in late 2019, maybe I missed a change till then). @wylieconlon can we extend the suggestion system to handle multiple layers correctly in this specific scenario?

Also seems like this scenario is worth a functional test.

[wylieconlon]

It looks like this issue is happening in all XY charts that include a second layer, whether or not the second layer has data. The issue is not a crash, because if you click the "add layer" button in the right panel things go back to normal.

I also checked and the "most correct" suggestion is actually generated by both the datasource and visualization. I think the issue is mostly about how we sort and filter suggestions in the chart switcher. I think the only way to solve this issue is to add more information to the suggestions request so that the datasource and visualization can be more aware that the suggestion will be used for chart-switching purposes.

[wylieconlon]

I realized I hadn't addressed all the points you made @flash1293:

1. Yes, layers are handled in suggestions using a parameter keptLayerIds. The coordination of adding and removing layers is now done by the editor, making it more reliable.
2. This definitely needs a functional test.