Adding multi node IT

[amitgalitz]

Description

adding multi node IT

Issues Resolved

resolves #354

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ohltyler]

CI is not running, is there a syntax error? I believe run shouldn't be tabbed in.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f8e822f) 72.14% compared to head (9e3abdc) 71.98%.
Report is 2 commits behind head on main.

❗ Current head 9e3abdc differs from pull request most recent head e860468. Consider uploading reports for the commit e860468 to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #360      +/-   ##
============================================
- Coverage     72.14%   71.98%   -0.17%     
+ Complexity      613      612       -1     
============================================
  Files            79       79              
  Lines          3070     3070              
  Branches        238      238              
============================================
- Hits           2215     2210       -5     
- Misses          751      755       +4     
- Partials        104      105       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

LGTM, although I don't think we need to have 7 copies of this! Could probably put a conditional on the multi-node test to only do for one if the JDKs.

[owaiskazi19]

@amitgalitz I see the multi node tests are failing with TimeoutException, do we have any idea about it?

[amitgalitz]

@amitgalitz I see the multi node tests are failing with TimeoutException, do we have any idea about it?

These should be fixed by latest code changes from Dan, however there might be still some left over issue that relates to the issues that Josh is working on. I will confirm this

[dbwiddis]

I see the multi node tests are failing with TimeoutException, do we have any idea about it?

Yeah let's get the single node tests passing consistently and see if it fixes these, if so merge away.

[owaiskazi19]

Yeah let's get the single node tests passing consistently and see if it fixes these, if so merge away.

Still fails with the same exception. Do we have to add a TimeoutSuite for the integ tests?

[dbwiddis]

Still fails with the same exception. Do we have to add a TimeoutSuite for the integ tests?

Looks to me like tests are running before the cluster is up and connected.

» WARN ][o.o.c.c.ClusterFormationFailureHelper] [integTest-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [integTest-0, integTest-1, integTest-2] to bootstrap a cluster: have discovered [{integTest-0}{Ea7SJKosQ9SDCLE0H1A3_Q}{qLyHPpOfRZyAmzYHo0KyNg}{127.0.0.1}{127.0.0.1:41569}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{integTest-0}{Ea7SJKosQ9SDCLE0H1A3_Q}{qLyHPpOfRZyAmzYHo0KyNg}{127.0.0.1}{127.0.0.1:41569}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

[owaiskazi19]

Suggested change

	if: matrix.java == 21
	if: ${{ matrix.java == 21 }}

[joshpalis]

Shouldn't it be if: ${{ matrix.java }} == 21

[amitgalitz]

current code works, multi node only ran on jdk21

need to fix failures

[amitgalitz]

getting lots of time out exceptions locally now too, I didn't see this before, so trying out a few different commits locally to see if I can locate any big differences

[owaiskazi19]

Can you make this 1000 and try again for mac-os?

[joshpalis]

Seems removing the wipreAllIndices method resolved most of the issues we're seeing. All but 1 multi-node test passed

1> [2024-01-12T04:01:49,541][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionAgentFrameworkWorkflow] before test
  1> [2024-01-12T04:01:49,544][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionAgentFrameworkWorkflow] initializing REST clients against [http://[::1]:36829, http://127.0.0.1:41505,/ http://[::1]:46719, http://127.0.0.1:41221,/ http://[::1]:41357, http://127.0.0.1:46585]/
  1> [2024-01-12T04:02:27,917][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionAgentFrameworkWorkflow] after test
  1> [2024-01-12T04:02:27,922][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testSearchWorkflows] before test
  1> [2024-01-12T04:02:28,156][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testSearchWorkflows] after test
  1> [2024-01-12T04:02:28,160][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionLocalModelWorkflow] before test
  1> [2024-01-12T04:06:41,908][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionLocalModelWorkflow] after test
  1> [2024-01-12T04:06:41,913][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionRemoteModelWorkflow] before test
  1> [2024-01-12T04:11:16,175][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionRemoteModelWorkflow] after test

Tests with failures:
 - org.opensearch.flowframework.rest.FlowFrameworkRestApiIT.testCreateAndProvisionRemoteModelWorkflow

5 tests completed, 1 failed

[amitgalitz]

closing for #416