TSDB: Provide a mechanism for users to go back to old working dashboards.

[lalit-satapathy]

Summary
If user has an working custom dashboard that gets broken (because it had some wrong aggregation on counters) after enabling TSDB. By disabling TSDB (either by a package upgrade or manual disablement), user should get the dashboard to working state. This gives users an ability to go back to a working state without disrupting the usage.

Current understanding based on offline discussion:
User realises that dashboard isn't loading and then downgrades. Which means another backing index is created (non tsdb). If the visualization/dashboard loads and the tsdb backing index is included in the query then an error can still occur (i think shard level error).

TBD
@lalit-satapathy to create a testing scenario for this.

CC: @martijnvg @ruflin

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[lalit-satapathy]

duplicate of #96170

[ruflin]

The expected behaviour here would be that the query still works, but for the time span where TSDB was enabled, the visualisation would be empty. @lalit-satapathy Is this accurate?

[lalit-satapathy]

The expected behaviour here would be that the query still works, but for the time span where TSDB was enabled, the visualisation would be empty. @lalit-satapathy Is this accurate?

Planning to verify the issue once in a real testcase.

[mlunadia]

The expected behaviour here would be that the query still works, but for the time span where TSDB was enabled, the visualisation would be empty. @lalit-satapathy Is this accurate?

@ruflin On conversation with the TSDB team last week they confirmed that any time range including a portion of data from a TSDB index would not work and Kibana has no automatic mechanism to exclude the TSDB index from the query.
If a user faced issues and decided to disable TSDB, any broken dashboards would only work for time ranges pre-TSDB and post-TSDB.
There were some suggestions of manually handling the data to fix the issue it but seemed like it implied a lot of manual work.

[martijnvg]

Let's move this conversation to #96170