Increase jvm heap size of test cluster from default 1G to 2G for integration tests.

[rishabh6788]

Description

The OpenSearch cluster spun up by integration tests uses the defaults for jvm parameters, which only allocates 1G of heap memory. We have had issues with our integration tests where the tests failed due lack of memory, such as OOM errors and circuit_breaker_exceptions.

This PR increases the heap size of the test OS cluster to 2G.
This will work for tar and zip distribution types but may not work for rpm and deb distributions as the config files for them are located in the root directories.
This will only throw permission error exception for rpm and deb, and test will still execute with 1G heap size. We can revisit this post 2.14 release to proper handle permissions in rpm and deb.

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peterzhuamazon]

Hi @rishabh6788, you can make changes here on both deb and rpm to give permissions to jvm yml:
https://github.com/opensearch-project/opensearch-build/blob/main/src/test_workflow/integ_test/distribution_rpm.py#L50-L52

Thanks.

[rishabh6788]

Hi @rishabh6788, you can make changes here on both deb and rpm to give permissions to jvm yml: https://github.com/opensearch-project/opensearch-build/blob/main/src/test_workflow/integ_test/distribution_rpm.py#L50-L52

Thanks.

made required changes to make jvm.options file editable for rpm and deb integ-test runs.
Tested locally and cluster is coming up 2G heap.
@peterzhuamazon

[joshuali925]

Is this breaking OSD tests? i got chmod: cannot access '/etc/opensearch-dashboards/jvm.options': No such file or directory

$ ./test.sh integ-test manifests/2.14.0/opensearch-dashboards-2.14.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.14.0/9743/linux/x64/deb opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.14.0/7621/linux/x64/deb --component observabilityDashboards
...
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch Dashboards, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch Dashboards configuration files, we recommend you add the user that runs these tools to the 'opensearch-dashboards' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
chmod: cannot access '/etc/opensearch-dashboards/jvm.options': No such file or directory
2024-05-06 17:46:06 INFO     Process is not started
2024-05-06 17:46:06 INFO     Removing /tmp/tmpxq_8qazh
Traceback (most recent call last):
  File "/home/ubuntu/projects/opensearch-build/src/test_workflow/test_cluster.py", line 61, in create
    cluster.start()
  File "/home/ubuntu/projects/opensearch-build/src/test_workflow/test_cluster.py", line 78, in start
    service.start()
  File "/home/ubuntu/projects/opensearch-build/src/test_workflow/integ_test/service_opensearch_dashboards.py", line 44, in start
    self.dist.install(self.download())
  File "/home/ubuntu/projects/opensearch-build/src/test_workflow/integ_test/distribution_deb.py", line 58, in install
    subprocess.check_call(deb_install_cmd, cwd=self.work_dir, shell=True)
  File "/home/ubuntu/.asdf/installs/python/3.9.15/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'sudo dpkg --purge opensearch-dashboards && sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! dpkg --install /tmp/tmpxq_8qazh/local-test-cluster/opensearch-dashboards-2.14.0-linux-x64.deb && sudo chmod 0666 /etc/opensearch-dashboards/opensearch_dashboards.yml /etc/opensearch-dashboards/jvm.options && sudo chmod 0755 /etc/opensearch-dashboards /var/log/opensearch-dashboards && sudo usermod -a -G opensearch-dashboards `whoami` && sudo usermod -a -G adm `whoami`' returned non-zero exit status 1.

[zelinh]

This PR seems to break all OSD integ tests for deb & rpm. ^^
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5936/pipeline
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5938/pipeline

[peterzhuamazon]

We are trying to fix this now.
Thanks.