[BUG] org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush and org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush are flaky

[BhumikaSaini-Amazon]

Describe the bug
org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush and org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush are flaky.

Tried running few iterations on local as well and can see some iterations failing:

 % ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush" -Dtests.iters=10

...

Tests with failures:
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:8BF3BD337C691C5D]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:51A2BEF2DAD5496]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:34F4800A41530FBF]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:E9CFD6F89DE78204]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:D711B27E8FD79202]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:4BA22804ACC306AC]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush {seed=[5317206DA34465B3:790ECD6ADB0811F6]}

10 tests completed, 7 failed

> Task :server:internalClusterTest FAILED

FAILURE: Build failed with an exception.

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush" -Dtests.iters=10

...

Tests with failures:
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush {seed=[FF04F2F4FC9F72CF:EA4393212FE7E6C8]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush {seed=[FF04F2F4FC9F72CF:64AA05FD7E23AE03]}
 - org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush {seed=[FF04F2F4FC9F72CF:19852BB5B3D53591]}

10 tests completed, 3 failed

> Task :server:internalClusterTest FAILED

FAILURE: Build failed with an exception.

Also reported in #8279

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush" -Dtests.seed=7F3F53D37D69C46C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nl-BE -Dtests.timezone=America/Belize -Druntime.java=20

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithoutInvokeFlush" -Dtests.seed=7F3F53D37D69C46C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nl-BE -Dtests.timezone=America/Belize -Druntime.java=20

Expected behavior
The tests should pass/fail deterministically.

Additional context
Example of a failing build: https://build.ci.opensearch.org/job/gradle-check/20013/

[sachinpkale]

Muted as part of: #8484

Keeping the issue open till we fix these tests.

[gbbafna]

PR link #9190