[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

reta · 2023-11-28T19:19:27Z

Describe the bug
The test case org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky:

org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache {p0={"search.concurrent_segment_search.enabled":"true"}}

java.lang.AssertionError: expected:<[1, 1, 0]> but was:<[0, 2, 0]>
	at __randomizedtesting.SeedInfo.seed([5507014708D300D0:F76E1CAB764B5B4E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.opensearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:649)
	at org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache(IndicesRequestCacheIT.java:635)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.IndicesRequestCacheIT" -Dtests.method="testProfileDisableCache {p0={"search.concurrent_segment_search.enabled":"true"}}"

Expected behavior
The test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/30517/testReport/junit/org.opensearch.indices/IndicesRequestCacheIT/testProfileDisableCache__p0___search_concurrent_segment_search_enabled___true___/

The text was updated successfully, but these errors were encountered:

peternied · 2023-11-30T17:40:41Z

Thanks for filing this issue

neetikasinghal · 2024-01-12T23:15:02Z

I am not able to reproduce this in 500 runs yet.

neetikasinghal · 2024-01-12T23:51:43Z

Unable to reproduce it within 2000 runs as well. Also tried with the random seed mentioned in the build failure above.

neetikasinghal · 2024-01-16T23:56:25Z

Unable to reproduce this within 3500 runs, I will close this issue for now. We can re-open it back if encounter this failure back.

reta · 2024-03-20T21:56:51Z

It is back:

java.lang.AssertionError: expected:<[1, 1, 0]> but was:<[0, 2, 0]>
	at __randomizedtesting.SeedInfo.seed([71AF33CAB99ACC83:D3C62E26C702971D]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.opensearch.indices.IndicesRequestCacheIT.assertCacheState(IndicesRequestCacheIT.java:690)
	at org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache(IndicesRequestCacheIT.java:637)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:369)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

https://build.ci.opensearch.org/job/gradle-check/35517/testReport/junit/org.opensearch.indices/IndicesRequestCacheIT/testProfileDisableCache__p0___opensearch_experimental_feature_pluggable_caching_enabled___false___/

jed326 · 2024-03-27T22:47:34Z

This looks like the same category as #12308.
The failure here doesn't look related to concurrent search, @sgup432 mind taking a look at this one to see if it's related to the pluggable caching changes?

These are very frequent offenders and are causing a lot of pain in CI. Related issues: opensearch-project#11374 opensearch-project#12308 opensearch-project#13540 opensearch-project#13503 opensearch-project#13711 opensearch-project#13949 opensearch-project#13437 opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 Signed-off-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit fbe048f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - #11374 - #12308 - #13540 - #13503 - #13711 - #13949 - #13437 - #13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

reta · 2024-06-19T16:08:11Z

Closing in favour of #14288

…opensearch-project#14082) These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 (cherry picked from commit fbe048f) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

These are very frequent offenders and are causing a lot of pain in CI. Related issues: - opensearch-project#11374 - opensearch-project#12308 - opensearch-project#13540 - opensearch-project#13503 - opensearch-project#13711 - opensearch-project#13949 - opensearch-project#13437 - opensearch-project#13600 Signed-off-by: Andrew Ross <andrross@amazon.com>

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Nov 28, 2023

reta mentioned this issue Nov 28, 2023

[AUTOCUT] Gradle Check Failure on push to main #11370

Closed

ankitkala assigned yigithub Dec 16, 2023

yigithub assigned neetikasinghal and unassigned yigithub Jan 12, 2024

neetikasinghal closed this as completed Jan 16, 2024

jed326 mentioned this issue Feb 14, 2024

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testCacheWithInvalidation is flaky #12308

Closed

reta reopened this Mar 20, 2024

github-actions bot added the untriaged label Mar 20, 2024

reta mentioned this issue Mar 20, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #12806

Closed

msfroh removed the untriaged label Mar 27, 2024

jed326 unassigned neetikasinghal Mar 27, 2024

reta mentioned this issue Apr 16, 2024

[AUTOCUT] Gradle Check Failure on push to main #13202

Closed

ltaragi mentioned this issue Apr 24, 2024

Take shallow copy snapshot only when remote store compatibility mode is STRICT #13309

Merged

8 tasks

reta mentioned this issue Apr 29, 2024

[Backport] [2.x] Bump org.bouncycastle:bc-fips from 1.0.2.4 to 1.0.2.5 in /distribution/tools/plugin-cli (#13446) #13449

Merged

dblock mentioned this issue Apr 30, 2024

Increase cluster publish timeout in flaky test testStaleClusterManage… #13463

Merged

8 tasks

jed326 mentioned this issue May 1, 2024

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache {p0={"search.concurrent_segment_search.enabled":"true"}} is flaky #13479

Closed

bowenlan-amzn mentioned this issue May 4, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #13536

Closed

mgodwan removed Indexing & Search Indexing Indexing, Bulk Indexing and anything related to indexing Indexing:Replication Issues and PRs related to core replication framework eg segrep labels May 15, 2024

This was referenced May 24, 2024

Bump jackson from 2.17.0 to 2.17.1 #13817

Merged

Bump org.apache.xmlbeans:xmlbeans from 5.2.0 to 5.2.1 in /plugins/ingest-attachment #13839

Merged

andrross mentioned this issue Jun 7, 2024

Mute all flaky IndicesRequestCacheIT tests #14077

Merged

3 tasks

kiranprakash154 self-assigned this Jun 12, 2024

kiranprakash154 added the Search:Performance label Jun 12, 2024

kiranprakash154 mentioned this issue Jun 17, 2024

Fix Indices Request Cache ITs from flaking and timing out #14369

Merged

3 tasks

reta closed this as completed Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

reta commented Nov 28, 2023

peternied commented Nov 30, 2023

neetikasinghal commented Jan 12, 2024

neetikasinghal commented Jan 12, 2024

neetikasinghal commented Jan 16, 2024

reta commented Mar 20, 2024

jed326 commented Mar 27, 2024

reta commented Jun 19, 2024

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

[BUG] org.opensearch.indices.IndicesRequestCacheIT.testProfileDisableCache is flaky #11374

Comments

reta commented Nov 28, 2023

peternied commented Nov 30, 2023

neetikasinghal commented Jan 12, 2024

neetikasinghal commented Jan 12, 2024

neetikasinghal commented Jan 16, 2024

reta commented Mar 20, 2024

jed326 commented Mar 27, 2024

reta commented Jun 19, 2024