Speed up CI Build #996

markmandel · 2019-08-13T18:38:38Z

Run more things in parallel.
Cache C++ build output, as that is the longer path.

agones-bot · 2019-08-13T19:04:24Z

Build Succeeded 👏

Build Id: 2fe02d02-92ed-479e-85ea-70725daac70e

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-1d1dd4d
image: gcr.io/agones-images/agones-sdk:1.0.0-1d1dd4d
image: gcr.io/agones-images/agones-ping:1.0.0-1d1dd4d
Linux C++ SDK (build): agonessdk-1.0.0-1d1dd4d-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-1d1dd4d.zip

A preview of the website (the last 30 builds are retained):

https://1d1dd4d-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-1d1dd4d

roberthbailey · 2019-08-13T19:07:45Z

cloudbuild.yaml

 - name: "make-docker"
  id: push-images
  waitFor:
-    - build
+    - build-images
  dir: "build"
  args: [ "-j", "3", "push"]


We build 4 images now, so this can be -j 4.

cloudbuild.yaml

markmandel · 2019-08-13T19:12:53Z

First pass seemed to scrape off 3 minutes. Not as much as I would like.

markmandel · 2019-08-13T19:25:17Z

@googlebot rescan

agones-bot · 2019-08-13T20:11:42Z

Build Succeeded 👏

Build Id: 7af13475-17bb-403c-9826-f24aa1d7e186

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-eb92bdf
image: gcr.io/agones-images/agones-sdk:1.0.0-eb92bdf
image: gcr.io/agones-images/agones-ping:1.0.0-eb92bdf
Linux C++ SDK (build): agonessdk-1.0.0-eb92bdf-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-eb92bdf.zip

A preview of the website (the last 30 builds are retained):

https://eb92bdf-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-eb92bdf

- Run more things in parallel. - Cache C++ build output, as that is the longer path.

agones-bot · 2019-08-13T20:36:18Z

Build Succeeded 👏

Build Id: 8662c7b1-7c90-464e-8cb2-c6a20a4f91fa

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-9e658c6
image: gcr.io/agones-images/agones-sdk:1.0.0-9e658c6
image: gcr.io/agones-images/agones-ping:1.0.0-9e658c6
Linux C++ SDK (build): agonessdk-1.0.0-9e658c6-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-9e658c6.zip

A preview of the website (the last 30 builds are retained):

https://9e658c6-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-9e658c6

markmandel · 2019-08-13T20:42:12Z

That's better - a 14 minute build.

aLekSer · 2019-08-13T20:58:44Z

Cool gain in execution time.
However there is a real chance (saw it on a cloudbuild run) that steps tests and run-sdk-conformance-tests would overlap with fail result.
It should occur when SDK conformance would remove ./node_modules and make tests would use node_modules (
Step #15 - "tests": /usr/bin/make run-sdk-command COMMAND=test SDK_FOLDER=node in particular)
Removing node_modules performed by this command:

agones/build/includes/sdk.mk

Line 140 in c5fa6ef

$(MAKE) run-sdk-command COMMAND=clean

You could see from logs that Step 15 tests and Step 21 SDK conformance runs simultaneously:
https://console.cloud.google.com/cloud-build/builds/8662c7b1-7c90-464e-8cb2-c6a20a4f91fa?project=agones-images

markmandel · 2019-08-13T21:02:47Z

Why does it need to be removed? Could we not gitignore it instead?

aLekSer · 2019-08-13T21:05:03Z

we could, I need to recap pros and cons as well as reason behind adding clean step for Node JS.

markmandel · 2019-08-13T21:45:05Z

Looking at the code - it's already .gitignored. So I can remove the clean step. Ran it locally, and git did not complain about anything not being added 👍

agones-bot · 2019-08-13T22:00:38Z

Build Succeeded 👏

Build Id: 96ff6b54-bcd0-41e9-abb7-a412b0dc0a01

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-92c9bf4
image: gcr.io/agones-images/agones-sdk:1.0.0-92c9bf4
image: gcr.io/agones-images/agones-ping:1.0.0-92c9bf4
Linux C++ SDK (build): agonessdk-1.0.0-92c9bf4-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-92c9bf4.zip

A preview of the website (the last 30 builds are retained):

https://92c9bf4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-92c9bf4

markmandel · 2019-08-13T22:09:16Z

One thing I don't understand. I upped the e2e test parrallel count to 64 (from 32), no increase in performance. Seems odd.

agones-bot · 2019-08-13T22:37:46Z

Build Succeeded 👏

Build Id: 7720793f-ba41-41dd-b787-6a2971a3c797

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-5148373
image: gcr.io/agones-images/agones-sdk:1.0.0-5148373
image: gcr.io/agones-images/agones-ping:1.0.0-5148373
Linux C++ SDK (build): agonessdk-1.0.0-5148373-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-5148373.zip

A preview of the website (the last 30 builds are retained):

https://5148373-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-5148373

markmandel · 2019-08-13T22:41:29Z

Just removed 4 minutes from the e2e test!

13 minute run!

markmandel · 2019-08-13T22:45:23Z

test/e2e/fleet_test.go

@@ -56,7 +56,10 @@ func TestFleetScaleUpEditAndScaleDown(t *testing.T) {
 	fixtures := []bool{true, false}

 	for _, usePatch := range fixtures {
+		usePatch := usePatch


I figure someone will ask about this:
https://gist.github.com/posener/92a55c4cd441fc5e5e85f27bca008721

aLekSer · 2019-08-14T08:27:03Z

build/includes/sdk.mk

@@ -137,7 +137,6 @@ run-sdk-conformance-no-build: ensure-build-sdk-image
 run-sdk-conformance-test:
 	$(MAKE) run-sdk-command COMMAND=build-sdk-test
 	$(MAKE) run-sdk-conformance-no-build
-	$(MAKE) run-sdk-command COMMAND=clean


Note that there is also a clean step in Rust. Will check if Rust package files are gitignored.
I suggest adding a new make target for clean. Otherwise it would be not visible from sdk.mk that we have a special way to remove package files.

Rust target dir have appropriate .gitignore. https://github.com/googleforgames/agones/blob/master/test/sdk/rust/.gitignore
So the only request for adding new target for clean left.

Could be:

clean-sdk-conformance-tests: $(MAKE) run-all-sdk-command COMMAND=clean

Created a PR linked to this.

agones-bot · 2019-08-14T17:49:58Z

Build Succeeded 👏

Build Id: cac641e8-737e-48b4-b702-79b23f9d1e31

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-c94a333
image: gcr.io/agones-images/agones-sdk:1.0.0-c94a333
image: gcr.io/agones-images/agones-ping:1.0.0-c94a333
Linux C++ SDK (build): agonessdk-1.0.0-c94a333-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-c94a333.zip

A preview of the website (the last 30 builds are retained):

https://c94a333-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/996/head:pr_996 && git checkout pr_996
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-c94a333

markmandel added kind/cleanup Refactoring code, fixing up documentation, etc area/build-tools Development tooling. I.e. pretty much everything in the `build` directory. area/tests Unit tests, e2e tests, anything to make sure things don't break labels Aug 13, 2019

roberthbailey reviewed Aug 13, 2019

View reviewed changes

markmandel force-pushed the ci/go-go-go-faster branch from 1d1dd4d to eb92bdf Compare August 13, 2019 19:42

Speed up CI Build

9e658c6

- Run more things in parallel. - Cache C++ build output, as that is the longer path.

markmandel force-pushed the ci/go-go-go-faster branch from eb92bdf to 9e658c6 Compare August 13, 2019 20:21

markmandel marked this pull request as ready for review August 13, 2019 20:43

- Remove clean from run-sdk-conformance-test to remove race condition.

92c9bf4

- Allowing more e2e tests to run in parrallel

5148373

markmandel commented Aug 13, 2019

View reviewed changes

aLekSer reviewed Aug 14, 2019

View reviewed changes

aLekSer mentioned this pull request Aug 14, 2019

Adding clean target for SDK conformance tests #997

Merged

roberthbailey approved these changes Aug 14, 2019

View reviewed changes

Merge branch 'master' into ci/go-go-go-faster

c94a333

markmandel merged commit 56b9a6e into googleforgames:master Aug 14, 2019

markmandel added this to the 1.0 milestone Aug 14, 2019

markmandel deleted the ci/go-go-go-faster branch August 14, 2019 23:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up CI Build #996

Speed up CI Build #996

markmandel commented Aug 13, 2019 •

edited

Loading

agones-bot commented Aug 13, 2019

roberthbailey Aug 13, 2019

markmandel commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019

aLekSer commented Aug 13, 2019 •

edited

Loading

markmandel commented Aug 13, 2019

aLekSer commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019 •

edited

Loading

markmandel Aug 13, 2019

aLekSer Aug 14, 2019 •

edited

Loading

aLekSer Aug 14, 2019

aLekSer Aug 14, 2019

agones-bot commented Aug 14, 2019

Speed up CI Build #996

Speed up CI Build #996

Conversation

markmandel commented Aug 13, 2019 • edited Loading

agones-bot commented Aug 13, 2019

roberthbailey Aug 13, 2019

Choose a reason for hiding this comment

markmandel commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019

aLekSer commented Aug 13, 2019 • edited Loading

markmandel commented Aug 13, 2019

aLekSer commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019

agones-bot commented Aug 13, 2019

markmandel commented Aug 13, 2019 • edited Loading

markmandel Aug 13, 2019

Choose a reason for hiding this comment

aLekSer Aug 14, 2019 • edited Loading

Choose a reason for hiding this comment

aLekSer Aug 14, 2019

Choose a reason for hiding this comment

aLekSer Aug 14, 2019

Choose a reason for hiding this comment

agones-bot commented Aug 14, 2019

markmandel commented Aug 13, 2019 •

edited

Loading

aLekSer commented Aug 13, 2019 •

edited

Loading

markmandel commented Aug 13, 2019 •

edited

Loading

aLekSer Aug 14, 2019 •

edited

Loading