Adds plugin version sweep background job

[downsrob]

Issue #, if available:
#207

Description of changes:
Index Management currently skips all job executions when there are two differing versions of Index Management on the cluster. The plugin currently does this by performing a NodesInfoRequest to get and compare plugin versions whenever there is a node added or a new cluster, and set a flag, SkipExecution, to true when there are multiple plugin versions. We have seen cases where the SkipExecution flag is still set to true even though the upgrade process (early ES 7.x to later ES 7.x) has finished and the cluster is on the latest version w/ all nodes containing the same version of IM plugin.

From analyzing the code, we can see race conditions that would allow multiple requests to overwrite each other in the wrong order. Though the cluster changed events would come in order, the NodesInfoRequests may actually overwrite the flag out of order.

To resolve this race condition, this PR adds a background job which will run every five minutes to poll the plugin versions if the flag is currently set to true.

This is an alternative strategy to #423 and is also entirely by Stevan Buzejic, @stevanbz, I am just raising the PR for an early review.

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #434 (47b7a24) into main (39be4e3) will increase coverage by 0.00%.
The diff coverage is 78.26%.

@@            Coverage Diff            @@
##               main     #434   +/-   ##
=========================================
  Coverage     75.94%   75.95%           
- Complexity     2480     2492   +12     
=========================================
  Files           315      316    +1     
  Lines         14500    14547   +47     
  Branches       2243     2248    +5     
=========================================
+ Hits          11012    11049   +37     
- Misses         2239     2246    +7     
- Partials       1249     1252    +3     

Impacted Files	Coverage Δ
...exstatemanagement/PluginVersionSweepCoordinator.kt	69.69% <69.69%> (ø)
...pensearch/indexmanagement/IndexManagementPlugin.kt	90.00% <100.00%> (+0.11%)	⬆️
...exmanagement/indexstatemanagement/SkipExecution.kt	61.29% <100.00%> (-5.38%)	⬇️
...exstatemanagement/settings/ManagedIndexSettings.kt	98.49% <100.00%> (+0.05%)	⬆️
...ment/indexstatemanagement/util/RestHandlerUtils.kt	88.88% <0.00%> (-11.12%)	⬇️
...arch/indexmanagement/rollup/RollupSearchService.kt	57.40% <0.00%> (-3.71%)	⬇️
...exstatemanagement/resthandler/RestExplainAction.kt	100.00% <0.00%> (ø)
.../opensearch/indexmanagement/rollup/model/Rollup.kt	86.04% <0.00%> (+0.46%)	⬆️
...management/rollup/interceptor/RollupInterceptor.kt	80.15% <0.00%> (+0.79%)	⬆️
...t/resthandler/RestRetryFailedManagedIndexAction.kt	88.00% <0.00%> (+1.04%)	⬆️
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[bowenlan-amzn]

A general questions:

1. can we disable the trigger logic in skipExecution since we now have this background loop.
   trigger logic I am referring to

override fun clusterChanged(event: ClusterChangedEvent) {
        if (event.nodesChanged() || event.isNewCluster) {
            sweepISMPluginVersion()
        }
    }

in SkipExecution

[bowenlan-amzn]

No need to have this if we are adding a new setting

[stevanbz]

Makes sense. Tnx!

[bowenlan-amzn]

Do we need to cancel this job or let it run forever?

[bowenlan-amzn]

Is this same as sweepSkipPeriod? If so, should we use sweepSkipPeriod instead?

[stevanbz]

A general questions:

1. can we disable the trigger logic in skipExecution since we now have this background loop.
   trigger logic I am referring to

override fun clusterChanged(event: ClusterChangedEvent) {
        if (event.nodesChanged() || event.isNewCluster) {
            sweepISMPluginVersion()
        }
    }

in SkipExecution

Good question. And you are right - I am thinking the same. SkipExecution class should do only sweepISMPluginVersion, while the caller class will be responsible for triggering the request.

So, my proposal is:

Caller class, PluginVersionSweepCoordinator, will listen for cluster changed events and will be responsible for calling the sweepISM method. This class already has a scheduled job that can be canceled optionally (ie. if the skip flag is being set to true).

ie.

 override fun clusterChanged(event: ClusterChangedEvent) {
        if (event.nodesChanged() || event.isNewCluster) {
            skipExecution.sweepISMPluginVersion()
            initBackgroundSweepISMPluginVersionExecution()
        }
    }