Remove legacy /api/status endpoint response format

[joshdover]

In #76054, we added a new status format that reflects the architecture of the Kibana Platform. We still report using the old format by default and use only return the new format when the v8format=true query parameter is set.

For 8.0, we need to remove support for the legacy format, but we should continue to accept the v8format query parameter, but ignore it, for smooth upgrades. This change should not be backported to the 7.x branch.

In theory, we could continue to support the new and old formats and just change the default. This could ease some upgrade scenarios.

@ycombinator do you know if the kibana metricbeats module would need this compatibility in order to support smooth upgrades to 8.x?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[ycombinator]

Thanks for the ping, @joshdover. Yes, the kibana Metricbeat module will be impacted by this change. We should make the necessary changes to it in preparation for the legacy /api/status response format going away in 8.0.

cc: @sayden (he is currently the primary developer on Elastic Stack beats modules but I'm happy to answer any questions).

[afharo]

Should we add a Deprecation warning in the Upgrade assistant somehow?

Maybe, we can add it only when we get a request to GET /api/status without ?v8format=true once we've completed #77624. What do you think?

[lukeelmers]

For 8.0, we need to remove support for the legacy format, but we should continue to accept the v8format query parameter, but ignore it, for smooth upgrades. This change should not be backported to the 7.x branch.

I discussed this with @stacey-gammon & @kobelb, and since we don't have any telemetry indicating usage of this API, their preference is to take this a step further and introduce something like v7format which still preserves the old format. At the same time, we can also add telemetry to measure usage of the flag over time and then get rid of it in 8.x.

So to summarize:

• Continue accepting v8format in 8.0
• Add v7format in 8.0 as a workaround for folks who want to upgrade but keep the old structure. This would essentially be deprecated from the beginning.
• Introduce telemetry to track usage of v7format flag
• Completely remove v7format flag when usage is sufficiently low

Should we add a Deprecation warning in the Upgrade assistant somehow?

Maybe, we can add it only when we get a request to GET /api/status without ?v8format=true once we've completed #77624. What do you think?

Yeah, it wouldn't be bulletproof as it couldn't survive a server restart and the UA wouldn't show the warning unless the route was hit before UA was loaded. I wonder if just writing a deprecation log would be acceptable in this case?

Otherwise AFAIK we don't really have another solution for using the deprecations service for REST APIs. Something that could be improved by #105692 though.

[mshustov]

Completely remove v7format flag when usage is sufficiently low

@lukeelmers shouldn't we track v8format query param usage as well to remove it later?

[lukeelmers]

shouldn't we track v8format query param usage as well

@mshustov That's a good point, as we were focused on tracking v7format (and thus the eventual removal of the legacy format), we hadn't discussed what to do with v8format.

As it is currently implemented v8format will default to true if it isn't provided, and v7format will default to false. So keeping that flag around is basically no maintenance for us, aside from the need for it to remain in the schema. In this regard, it probably wouldn't be too painful to keep at around for a long time before deprecating (rule of thumb that has been discussed lately is ~2.5 years or < 1% telemetry usage).

But adding the telemetry for it would also be trivial, so I went ahead and created a follow-up task for it: #112519