Use latest artifacts for OpenSearch and OpenSearch Security Plugin during integration tests

[davidlago]

Right now we build the testing container with OpenSearch + security for our integration tests from their respective .zip files we get from artifacts.opensearch.org. These were not being kept up to date with every bundle run and having stale ones was causing issues that we temporarily fixed here by pointing to artifacts that we get from a specific run of our project's CI.

After this issue gets closed, modify our integration test workflow to point to the latest URLs.

[peternied]

Is there a reason that the security dashboard cannot use the distribution opensearchstaging/opensearch container as the base? It is automatically updated on successful builds.

It might be worthwhile making that container more usable if there are gaps.

[davidlago]

Good point. I believe the issue was that as we add things to the manifest (including other plugins) successful builds of the container image might take a bit. This pipeline builds the minimal set we need (core + security + dashboards) to test dashboards plugin.

[peternied]

This pipeline builds the minimal set we need

What about changing the distribution to be the 'minimal' set as this plugin gets released as part of that distribution?

If there are hang-ups around stability, predictability for using the distribution docker image, I think all plugins would benefit if they were resolved, we should make sure to have tracking issues.

[davidlago]

Not sure I'm doing a good job at explaining this (and I could be totally wrong!), but as we get started with a new version build, the docker images might take a bit to stabilize as more plugins get added etc, and that could break our CI (I believe this happened for 1.3 at some point).

Another thing is if this is actually a bug or a feature, being a forcing factor to get our distributions stable early :)

[peternied]

Ah, I am thinking about the release cycle, not including the full development into scope. Let me take another stab, when a new OpenSearch version is created there are different levels of plugin support:

1. No plugins, its only OpenSearch-min
2. There are some plugins
3. All plugins are included

During these different times how should plugins like Security-Dashboard-Plugin be able to build and test PRs? It has a tight coupling with the security plugin and that plugin is not ready to consume via the distribution because that process takes a multiple days/weeks to bootstrap.

That sound more accurate?

[davidlago]

yeah! for us is purely an IT thing... ITs running against even the most basic of the distributions. And as you know, we have a tight version check between plugins and backend, so it's not like we can use the previous distribution.

[peternied]

opensearch-project/opensearch-build#1569 has been merged, we should be able to update the build scripts

[DarshitChanpura]

This issue has already been fixed in 1.3 branch with PR: #925

It will be fixed in main branch with this PR: #927