Support for Elasticsearch point-in-time API

[lukasolson]

In 7.10, Elasticsearch released the point-in-time API. This is a meta issue to discuss where/how we can use the API to enhance the experience in Kibana.

A couple of places where we might benefit from using the PIT API:

• Terms aggregation "other" bucket ([data.search.aggs] Use point in time searches for "Other" bucket #79863)
• In Dashboard so that all panels query the same snapshot of data
• In Discover when we send multiple (separate) queries for the date histogram, hits, and count (Split Discover query into three #69134)

Steps

• Close the previous session [Search] Review searches clean up #105441
• Split out the monitoring tasks
• Create a PIT before sending any search requests in a session, send the PIT id with the search requests
• Clear PIT when clearing a session (in the cleanup task)

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[lukeelmers]

In 7.12, saved objects find supports using a PIT with search_after for paging through results (on the server only). I believe this is the first place we've used PIT in Kibana.

Also as a general PSA, we chatted with the ES team again to learn more about performance implications in using PIT. Overall the guidance is still that they're low impact, however we learned that keeping a PIT open unnecessarily can create a fair amount of overhead if your index is changing frequently as it can delay the ability to perform a merge process to free up resources -- So it is always preferred to explicitly close a PIT when you are done with it rather than waiting for the keep_alive to expire.

[lizozom]

I suggest we implement PIT as part of Search Sessions, as a mean of synchronizing all of the searches in that session.
This would also be helpful for client side caching (all requests are cached with their PIT).

If we do so, we could sent a PIT start request when we receive the first search for the current session (batching and holding on to any additional requests until we get a PIT id) and then sending all requests with the PIT id set.

We should also think about the keepalives and cleaning up the PITs when all searches are done.

[lukeelmers]

We should also think about the keepalives and cleaning up the PITs when all searches are done.

++ I would definitely recommend doing some performance testing if you end up using PIT for search sessions and need to keep PITs alive for long-running async searches. Since it still introduces some overhead in ES, it would be good to verify that a bunch of search sessions piling up and opening a ton of long-lived PITs doesn't make things worse.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[lukasolson]

Related: #134232

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.