Promote min and bundle artifacts to release-candidate after every bundle build #1378

zylzulu · 2021-12-17T21:31:21Z

Signed-off-by: Yilin Zhang yilizhan@amazon.com

Description

Uploads min and bundle artifacts for both opensearch and dashboards to the below location on every build:

https://artifacts.opensearch.org/release-candidates/core/opensearch/<version>/opensearch-min-<version>-linux-x64.tar.gz
https://artifacts.opensearch.org/release-candidates/bundle/opensearch/<version>/opensearch-<version>-linux-x64.tar.gz
https://artifacts.opensearch.org/release-candidates/core/opensearch-dashboards/<version>/opensearch-dashboards-min-<version>-linux-x64.tar.gz
https://artifacts.opensearch.org/release-candidates/bundle/opensearch-dashboards/<version>/opensearch-dashboards-<version>-linux-x64.tar.gz

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

codecov-commenter · 2021-12-17T21:33:48Z

Codecov Report

Merging #1378 (f683cc0) into main (09f65b3) will decrease coverage by 0.06%.
The diff coverage is 82.35%.

@@             Coverage Diff              @@
##               main    #1378      +/-   ##
============================================
- Coverage     94.45%   94.38%   -0.07%     
  Complexity       11       11              
============================================
  Files           140      140              
  Lines          3085     3101      +16     
  Branches         10       10              
============================================
+ Hits           2914     2927      +13     
- Misses          164      167       +3     
  Partials          7        7

Impacted Files	Coverage Δ
tests/jenkins/jobs/BuildManifest_Jenkinsfile	`100.00% <ø> (ø)`
src/jenkins/BuildManifest.groovy	`88.00% <82.35%> (-12.00%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09f65b3...f683cc0. Read the comment docs.

vars/uploadArtifacts.groovy

peternied · 2021-12-17T22:54:15Z

Please make sure that the new output will handle these cases:

.../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-x64.tar.gz
.../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-arm64.tar.gz
.../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-x64-latest.tar.gz
.../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-arm64-latest.tar.gz

I think we've got the first case, but we need the snapshot ones too

vars/uploadArtifacts.groovy

zelinh · 2021-12-18T00:49:53Z

src/jenkins/BuildManifest.groovy

+    public String getPackageName(String productName) {
+        String packagePrefix = [
+            productName,
+            this.build.version,
+            this.build.platform,
+            this.build.architecture,       
+        ].join('/')
+        return packagePrefix + '.tar.gz'
+    }


Can we use the name with version, platform, architecture and distribution from here?

dblock · 2021-12-29T20:10:19Z

Can we try to move this forward, please?

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

dblock

Add tests for the new functions at least, please.

src/jenkins/BuildManifest.groovy

…oduct name from manifest Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

src/jenkins/BuildManifest.groovy

dblock · 2021-12-30T14:17:56Z

@zylzulu will review when the build is green and the CR is ready

src/jenkins/BuildManifest.groovy

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

… into artifacts_snapshot_publish

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya · 2022-01-07T19:57:45Z

Please make sure that the new output will handle these cases:

* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-x64.tar.gz

* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-arm64.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-x64-latest.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-arm64-latest.tar.gz

I think we've got the first case, but we need the snapshot ones too

Hi @peternied , @dblock,
currently the min and bundle are being uploaded to build/test-release-candidate folder (I'll change to release-candidates before I merge my code. Not making any code changes since I can use this for testing ).
Regarding snapshots from where do I retrieve them?
Also I believe snapshots only applies to min here

dblock · 2022-01-07T20:11:20Z

Please make sure that the new output will handle these cases:
* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-x64.tar.gz

* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-arm64.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-x64-latest.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-arm64-latest.tar.gz
I think we've got the first case, but we need the snapshot ones too
Hi @peternied , @dblock, currently the min and bundle are being uploaded to build/test-release-candidate folder (I'll change to release-candidates before I merge my code. Not making any code changes since I can use this for testing ). Regarding snapshots from where do I retrieve them? Also I believe snapshots only applies to min here

Check out https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L50, which calls publish-snapshot to Maven. You'll probably want to publish around there.

gaiksaya · 2022-01-07T23:45:15Z

Please make sure that the new output will handle these cases:
* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-x64.tar.gz

* .../release-candidates/core/opensearch/1.2.2/opensearch-min-1.2.2-linux-arm64.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-x64-latest.tar.gz

* .../snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-arm64-latest.tar.gz
I think we've got the first case, but we need the snapshot ones too
Hi @peternied , @dblock, currently the min and bundle are being uploaded to build/test-release-candidate folder (I'll change to release-candidates before I merge my code. Not making any code changes since I can use this for testing ). Regarding snapshots from where do I retrieve them? Also I believe snapshots only applies to min here
Check out https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L50, which calls publish-snapshot to Maven. You'll probably want to publish around there.

Looking at the above code:
The buildManifest just builds the snapshot but does not upload it anywhere. Its the publish-snapshot that uploads the artifacts to maven.

https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L45-L47

Should we directly upload from here (jenkins local) to snapshots (prod) s3? Also what about arm64? While I executed the jenkinfile to see the generated artifacts, I did not see anything generated for arm64 here.
OR Can we separate this problem from the release-candidates one?
Also talked to @mch2 seems like we have 2 possible solutions:

Rename the above received artifact to snapshots as it guarantees that user is running tests against an artifact that will get released
Build x64 (already doing) and arm64 snapshots and then upload to s3 (either just to prod or both staging and prod).

dblock · 2022-01-10T12:48:18Z

@gaiksaya I think it's perfectly OK to do the snapshot uploading as part of another PR, and we can discuss options there; (2) seems like the better thing to do

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya · 2022-01-10T18:38:05Z

@gaiksaya I think it's perfectly OK to do the snapshot uploading as part of another PR, and we can discuss options there; (2) seems like the better thing to do

Created the issue #1442 to address the min snapshots uploading part

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

src/jenkins/BuildManifest.groovy

tests/jenkins/jobs/BuildManifest_Jenkinsfile

vars/uploadArtifacts.groovy

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

src/jenkins/BuildManifest.groovy

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

…dle build (opensearch-project#1378) * Promote to release candidate after every bundle build Signed-off-by: Yilin Zhang <yilizhan@amazon.com> Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com>

Promote to release candidate after every bundle build

604237d

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

zylzulu requested review from dblock, peternied, abhinavGupta16, gaiksaya, peterzhuamazon, bbarani and zelinh December 17, 2021 21:31

dblock requested changes Dec 17, 2021

View reviewed changes

vars/uploadArtifacts.groovy Outdated Show resolved Hide resolved

vars/uploadArtifacts.groovy Outdated Show resolved Hide resolved

dblock mentioned this pull request Dec 17, 2021

Publish build artifacts to correct locations from snapshot builds #1261

Closed

zelinh reviewed Dec 18, 2021

View reviewed changes

vars/uploadArtifacts.groovy Outdated Show resolved Hide resolved

zelinh reviewed Dec 18, 2021

View reviewed changes

Extend BuildManifest to know components.artifacts.dist

2f135b2

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

zylzulu requested review from dblock and zelinh December 29, 2021 21:03

dblock reviewed Dec 29, 2021

View reviewed changes

src/jenkins/BuildManifest.groovy Show resolved Hide resolved

src/jenkins/BuildManifest.groovy Outdated Show resolved Hide resolved

Increase robustness by raising error when size of dist = 0 and get pr…

0bb33fb

…oduct name from manifest Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

dblock reviewed Dec 29, 2021

View reviewed changes

src/jenkins/BuildManifest.groovy Outdated Show resolved Hide resolved

zylzulu requested a review from dblock December 29, 2021 23:19

dblock reviewed Dec 30, 2021

View reviewed changes

src/jenkins/BuildManifest.groovy Outdated Show resolved Hide resolved

correct typo and move the getPackageName to the build class

3415c52

Signed-off-by: Yilin Zhang <yilizhan@amazon.com>

gaiksaya removed request for abhinavGupta16 and bbarani January 5, 2022 02:31

gaiksaya added 2 commits January 4, 2022 18:39

Merge branch 'main' of github.com:opensearch-project/opensearch-build…

4ecc27f

… into artifacts_snapshot_publish

Fix exception

ce83a9c

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya removed their request for review January 5, 2022 03:03

gaiksaya added 4 commits January 6, 2022 19:14

Refactor

9f14d2c

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Refactor

25ee117

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Refactor

8938eb8

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Fix conflicts

cb34ea9

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya self-assigned this Jan 7, 2022

davidlago mentioned this pull request Jan 7, 2022

Use latest artifacts for OpenSearch and OpenSearch Security Plugin during integration tests opensearch-project/security-dashboards-plugin#886

Closed

gaiksaya mentioned this pull request Jan 10, 2022

Upload min targ.gz snapshots to publicly accessible bucket #1442

Closed

Reset the test env and correct the upload path

8d9f5ce

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya marked this pull request as ready for review January 10, 2022 18:38

gaiksaya requested a review from a team as a code owner January 10, 2022 18:38

Fix test based on new final path

3cde236

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

dblock requested changes Jan 10, 2022

View reviewed changes

Refactor

f683cc0

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya requested a review from dblock January 10, 2022 23:08

dblock requested changes Jan 11, 2022

View reviewed changes

src/jenkins/BuildManifest.groovy Outdated Show resolved Hide resolved

Generalize the code

434f5da

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya changed the title ~~Promote to release-candidate after every bundle build~~ Promote min and bundle artifacts to release-candidate after every bundle build Jan 11, 2022

gaiksaya requested a review from dblock January 11, 2022 23:19

dblock approved these changes Jan 11, 2022

View reviewed changes

dblock requested a review from a team January 11, 2022 23:34

peterzhuamazon approved these changes Jan 12, 2022

View reviewed changes

gaiksaya merged commit d0ac211 into opensearch-project:main Jan 12, 2022

This was referenced Jan 12, 2022

Publish built artifacts publicly to release-candidates/ folder #453

Closed

Support snapshot uploading to s3 for every build #1472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promote min and bundle artifacts to release-candidate after every bundle build #1378

Promote min and bundle artifacts to release-candidate after every bundle build #1378

zylzulu commented Dec 17, 2021 •

edited by gaiksaya

Loading

codecov-commenter commented Dec 17, 2021 •

edited

Loading

peternied commented Dec 17, 2021

zelinh Dec 18, 2021

dblock commented Dec 29, 2021

dblock left a comment

dblock commented Dec 30, 2021

gaiksaya commented Jan 7, 2022 •

edited

Loading

dblock commented Jan 7, 2022

gaiksaya commented Jan 7, 2022 •

edited

Loading

dblock commented Jan 10, 2022

gaiksaya commented Jan 10, 2022

Promote min and bundle artifacts to release-candidate after every bundle build #1378

Promote min and bundle artifacts to release-candidate after every bundle build #1378

Conversation

zylzulu commented Dec 17, 2021 • edited by gaiksaya Loading

Description

Check List

codecov-commenter commented Dec 17, 2021 • edited Loading

Codecov Report

peternied commented Dec 17, 2021

zelinh Dec 18, 2021

Choose a reason for hiding this comment

dblock commented Dec 29, 2021

dblock left a comment

Choose a reason for hiding this comment

dblock commented Dec 30, 2021

gaiksaya commented Jan 7, 2022 • edited Loading

dblock commented Jan 7, 2022

gaiksaya commented Jan 7, 2022 • edited Loading

dblock commented Jan 10, 2022

gaiksaya commented Jan 10, 2022

zylzulu commented Dec 17, 2021 •

edited by gaiksaya

Loading

codecov-commenter commented Dec 17, 2021 •

edited

Loading

gaiksaya commented Jan 7, 2022 •

edited

Loading

gaiksaya commented Jan 7, 2022 •

edited

Loading