Backport 1097, 1302, 1435, 1441, 1445, 1430

[engechas]

Description of changes:

• Cherry-pick change to use zipArchive for notifications
• Update BWC tests to use same structure/arguments as main
• Update BWC version to 2.3.0
• Backport optimize doc-level monitor workflow for index patterns #1097, optimize doc-level monitor execution workflow for datastreams #1302, Add jvm aware setting and max num docs settings for batching docs for percolate queries #1435, optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc #1441, optimize sequence number calculation and reduce search requests in doc level monitor execution #1445, Clean up doc level queries on dry run #1430

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 82.40575% with 98 lines in your changes are missing coverage. Please review.

Project coverage is 76.22%. Comparing base (5f738e3) to head (be34ced).
Report is 10 commits behind head on 2.5.

Files	Patch %	Lines
.../opensearch/alerting/DocumentLevelMonitorRunner.kt	80.20%	36 Missing and 23 partials ⚠️
...opensearch/alerting/util/DocLevelMonitorQueries.kt	84.84%	7 Missing and 18 partials ⚠️
.../org/opensearch/alerting/MonitorMetadataService.kt	37.50%	0 Missing and 5 partials ⚠️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt	72.22%	1 Missing and 4 partials ⚠️
...in/org/opensearch/alerting/MonitorRunnerService.kt	89.47%	2 Missing ⚠️
.../alerting/transport/TransportIndexMonitorAction.kt	66.66%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.5    #1470      +/-   ##
============================================
+ Coverage     75.76%   76.22%   +0.45%     
  Complexity      116      116              
============================================
  Files           125      125              
  Lines          6851     7320     +469     
  Branches       1028     1104      +76     
============================================
+ Hits           5191     5580     +389     
- Misses         1132     1178      +46     
- Partials        528      562      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.