[BUG] SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled is flaky

[peternied]

Describe the bug

Test log data: https://build.ci.opensearch.org/job/gradle-check/33984/testReport/

Assertion:

java.lang.AssertionError: There are still [6] in-flight contexts. The first one's creation site is listed as the cause of this exception.

Log entries

[2024-02-20T20:37:50,701][INFO ][o.o.c.r.FailAwareWeightedRouting] [node_t2] [test][9]: Fail open executed due to exception
org.opensearch.transport.NodeNotConnectedException: [node_t1][127.0.0.1:33757] Node not connected

[2024-02-20T20:37:50,702][INFO ][o.o.c.r.FailAwareWeightedRouting] [node_t2] [test][0]: Fail open executed due to exception
org.opensearch.transport.NodeDisconnectedException: [node_t1][127.0.0.1:33757][indices:data/read/search[phase/query]] disconnected

Related component

Search

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled" -Dtests.seed=AC9C2A81373045B9 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-JO -Dtests.timezone=Europe/Lisbon -Druntime.java=21

Expected behavior

Tests are reliable

Additional Details

No response

[peternied]

[Triage - attendees 1 2 3 4 5]
Thanks for filing