Add infra support and helper function to create real datasource

[derek-ho]

Description

Currently, there is no common support for creating real datasource connection to other opensearch instances. This adds this support in a helper function that other plugins can call when setting up their own multi datasources tests. In github workflow, it will spin up a cluster with no security running on port 9201 that it can connect to. TODO: add cluster supports for security enabled (basic auth and sigv4 auth). This is currently slightly blocked on opensearch-project/OpenSearch-Dashboards#4524

Issues Resolved

Fix: #1230

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[derek-ho]

@BionIT can you review/suggest direction this should go? Any plugins that can use this POC to write some tests for the repo to establish a pattern?

[kavilla]

Does this need backports?

[wanglam]

@derek-ho How can we add test data to the added data sources? For ml-commons, we may need to call /_plugins/_ml/models related API to construct test models. Is there any examples to do it?

[BionIT]

Thanks @derek-ho for this change! Sorry for the late reply! Pulling in @zhongnansu to take a look at the PR as well, since this can save the efforts for other plugins as well

cc @seraphjiang

[zhongnansu]

thanks for the change! left some comments

[derek-ho]

Dashboard failures are coming from notification dashboards:

curl: (7) Failed to connect to localhost port 5601 after 0 ms: Connection refused
{"type":"log","@timestamp":"2024-04-19T18:52:41Z","tags":["fatal","root"],"pid":2551,"message":"Error: Cannot find module '../../public/utils/constants'\nRequire stack:\n- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/plugins/notificationsDashboards/server/routes/configRoutes.js\n- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/plugins/notificationsDashboards/server/routes/index.js\n- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/plugins/notificationsDashboards/server/plugin.js\n- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/plugins/notificationsDashboards/server/index.js\n- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/src/core/server/
 FATAL  Error: Cannot find module '../../public/utils/constants'
Require stack:
- /home/runner/work/opensearch-dashboards-functional-test/opensearch-dashboards-functional-test/opensearch-dashboards-3.0.0/plugins/notificationsDashboards/server/routes/configRoutes.js
- ```

[BionIT]

@CCongWang @tianleh Could you help to review this PR? Thanks in advance!

[derek-ho]

@derek-ho How can we add test data to the added data sources? For ml-commons, we may need to call /_plugins/_ml/models related API to construct test models. Is there any examples to do it?

@wanglam I modified this PR to download the full distribution. See the helper function I introduced and the README. Once you get the id of the created datasource via the helper function, you should be able to make a regular request with cy.request(ENDPOINT?dataSourceId=${id}), or whichever way the ML plugin implemented request routing for the external datasource, or via UI and cypress actions.

[wanglam]

@derek-ho How can we add test data to the added data sources? For ml-commons, we may need to call /_plugins/_ml/models related API to construct test models. Is there any examples to do it?

@wanglam I modified this PR to download the full distribution. See the helper function I introduced and the README. Once you get the id of the created datasource via the helper function, you should be able to make a regular request with cy.request(ENDPOINT?dataSourceId=${id}), or whichever way the ML plugin implemented request routing for the external datasource, or via UI and cypress actions.

We need to call our opensearch plugin directly in our functional tests repo. Could we store the endpoint to the cypress.json? So we can reference the environment variable in cy.request.