[BUG] IndexNodeLeftDelayedTimeOut Setting NOT getting honoured when ExistingShardsAllocatorBatchMode is enabled.

[gargharsh3134]

Describe the bug

For an index with greater than or equal to 2 replica shards, the INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING is not getting honoured when EXISTING_SHARDS_ALLOCATOR_BATCH_MODE is enabled. When the nodes (on which the replica shards are allocated) drop from the cluster, the replica shards get allocated to different nodes instead of being delayed for the time specified in the index setting.

This incorrect allocation only occurs when more than 1 replica shards of a shardID are unassigned due to node drops. In the batch mode enabled setting, allocation decision is being made and executed for only one out all the replica shards belonging to a shardID, and thereby the left over replica shards are not getting marked as ignored during ReplicaShardBatchAllocator run. The subsequent run of BalancedShardAllocator (which runs after ReplicaShardBatchAllocator) ends up allocating those unassigned replica shards, which should instead have been delayed had the decision been taken and executed.

OpenSearch/server/src/main/java/org/opensearch/gateway/ShardsBatchGatewayAllocator.java

Lines 215 to 220 in 581fcd2

	} else {
	batchIdToStoreShardBatch.values()
	.stream()
	.filter(batch -> batchesToAssign.contains(batch.batchId))
	.forEach(batch -> replicaBatchShardAllocator.allocateUnassignedBatch(batch.getBatchedShardRoutings(), allocation));
	}

Related component

Cluster Manager

To Reproduce

1. Create a cluster with 6 Nodes.
2. Create an index with 1 primary shard and 3 replica shards.
3. Set a high value (60m) of INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING for the index created as part of step 2.
4. Stop 2 out of the 3 nodes having replica shards assigned.
5. The cluster would turn green, and both the replica shards will end up getting assigned.

Expected behavior

The replica shards should remain unassigned for the duration specified in the index's INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@gargharsh3134 Thanks for creating this issue, could you create a pull request to address?

[gargharsh3134]

@peternied Fix will be tracked as part of #13748 and the integration test to replicate this behaviour has been added as part of #13813.
Thanks!