Add automation to report on recent build flakes #3012

zmerlynn · 2023-03-02T01:05:22Z

Adds build/report/report.go, which generates a dated report and an index.html that redirects to it.

You can see it live at https://agones-build-reports.storage.googleapis.com/index.html, which redirects (today) to https://agones-build-reports.storage.googleapis.com/2023-03-02.html. The formatting is terrible, apologies for the web1.0 vibe.

Also adds a cloudbuild.yaml to run it. I opted to strip the build down to just the gcloud build runner because I was having difficulty with authentication with the normal build container - plus it's way heavier to build.

I tested this cloudbuild.yaml using a new trigger that is currently pulling from the PR branch: https://console.cloud.google.com/cloud-build/triggers;region=global/edit/5ac7d2ff-e190-412b-924b-67fe75e57de5?project=agones-images
You can see a recent run here: https://console.cloud.google.com/cloud-build/builds;region=global/c4ed1f1b-28aa-424a-94f8-b02724bdc235?project=agones-images

Also updates vendoring to pull in the cloudbuild package (plus upgrades).

agones-bot · 2023-03-02T01:11:08Z

Build Failed 😱

Build Id: 416fc094-2e25-4508-950b-721b6b6b8cd2

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T01:45:27Z

Build Failed 😱

Build Id: 1f8631c4-a848-4ab7-b209-8a87ed092555

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T01:50:22Z

Build Failed 😱

Build Id: 7b7396c1-ce30-400a-a42b-83f57d82150e

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T01:56:41Z

Build Failed 😱

Build Id: bf41603a-e517-4b91-8b32-f8fe4d51d3f6

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T02:13:13Z

Build Failed 😱

Build Id: 94e4f7db-afc4-4000-806e-18fdda657dc8

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T02:42:39Z

Build Failed 😱

Build Id: b6c8d5d7-95f3-40b8-ae94-2ff17904a7af

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-02T02:54:28Z

Build Failed 😱

Build Id: 16cf63a3-965d-4cd9-a774-a79134e39fdd

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

gongmax · 2023-03-03T00:52:59Z

Wow, much nicer now!

google-oss-prow · 2023-03-03T00:53:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: gongmax, zmerlynn
Once this PR has been reviewed and has the lgtm label, please ask for approval from markmandel. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

markmandel · 2023-03-03T01:09:28Z

That is really cool!

markmandel · 2023-03-03T01:10:37Z

Suggestion on enhancement: Show which step fails. That way we can at least see some patterns in which step is the failure (although not the failure itself)

zmerlynn · 2023-03-03T01:58:41Z

Suggestion on enhancement: Show which step fails. That way we can at least see some patterns in which step is the failure (although not the failure itself)

Yup. A lot of this was just to get some sort of framework in place to extend it. It's a starting place.

I've been thinking we should allow people to annotate somehow, but it requires a level of interactivity I'm certainly not competent at. It would be nice to be able to audit the builds and note where it failed to detect trends, like I did manually a few weeks ago. I may hack something up in a csv file that plugs in here, as a starting point.

agones-bot · 2023-03-03T02:06:25Z

Build Failed 😱

Build Id: e58158ca-f4ed-4008-93ba-182bd91ea9cf

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-03T04:30:59Z

Build Succeeded 👏

Build Id: c3d8e057-8414-4294-82fc-7f5b9c509830

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-acf95c9-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-acf95c9-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-acf95c9-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-acf95c9-amd64
Linux C++ SDK (build): agonessdk-1.31.0-acf95c9-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-acf95c9-amd64.zip

A preview of the website (the last 30 builds are retained):

https://acf95c9-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3012/head:pr_3012 && git checkout pr_3012
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-acf95c9-amd64

build/Makefile

aimuz · 2023-03-03T06:14:22Z

build/report/report.go

+	for sha := range success {
+		buildCount++
+		if ids, ok := failure[sha]; ok {
+			flakeCount++
+			for _, id := range ids {
+				flakes = append(flakes, flake{
+					ID:         id,
+					CreateTime: idTime[id].Format(time.RFC3339),
+				})
+			}
+		}
+	}


Suggested change

for sha := range success {

buildCount++

if ids, ok := failure[sha]; ok {

flakeCount++

for _, id := range ids {

flakes = append(flakes, flake{

ID: id,

CreateTime: idTime[id].Format(time.RFC3339),

})

}

}

}

buildCount := len(success)

for sha := range success {

ids, ok := failure[sha]; ok

if !ok {

continue

}

flakeCount++

for _, id := range ids {

flakes = append(flakes, flake{

ID: id,

CreateTime: idTime[id].Format(time.RFC3339),

})

}

}

A suggestion, I don't know if it would be better, at least not so deep

Thanks for the suggestion. I decided to change methodology and just count all the flakes and report the flake ratio, which makes this less complicated anyways.

Adds build/report/report.go, which generates a dated report and an index.html that redirects to it. You can see it live at https://agones-build-reports.storage.googleapis.com/index.html The formatting is terrible, apologies for the web1.0 vibe. Adds a cloudbuild.yaml to run it. I opted to strip the build down to just the gcloud build runner because I was having difficulty with authentication with the normal build container - plus it's _way_ heavier to build.

google-oss-prow · 2023-03-03T15:50:55Z

New changes are detected. LGTM label has been removed.

zmerlynn · 2023-03-03T15:51:21Z

https://agones-build-reports.storage.googleapis.com/test/2023-03-03-new8.html has the new output, and a description of the methodology at the bottom.

agones-bot · 2023-03-03T16:01:05Z

Build Failed 😱

Build Id: 9a0577f9-1c26-4f86-8640-27b9271f57e2

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-03T17:31:07Z

Build Failed 😱

Build Id: 7489ca6e-3bf8-43cf-b19e-a9c2c959f618

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-03T18:08:17Z

Build Failed 😱

Build Id: dc286ac4-58ec-4346-92c0-bf5ce2046d1e

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-03T19:25:00Z

Build Succeeded 👏

Build Id: 1e53572f-0263-4bda-8dfd-4fb47508e8e6

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-ad692ca-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-ad692ca-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-ad692ca-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-ad692ca-amd64
Linux C++ SDK (build): agonessdk-1.31.0-ad692ca-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-ad692ca-amd64.zip

A preview of the website (the last 30 builds are retained):

https://ad692ca-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3012/head:pr_3012 && git checkout pr_3012
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-ad692ca-amd64

google-oss-prow bot added the do-not-merge/work-in-progress label Mar 2, 2023

google-oss-prow bot requested review from dzlier-gcp and EricFortin March 2, 2023 01:05

google-oss-prow bot added the size/L label Mar 2, 2023

zmerlynn removed request for EricFortin and dzlier-gcp March 2, 2023 01:05

zmerlynn force-pushed the add-flake-report branch from b4bc080 to 854317a Compare March 2, 2023 01:40

zmerlynn force-pushed the add-flake-report branch from 854317a to 19a6fbe Compare March 2, 2023 01:46

zmerlynn force-pushed the add-flake-report branch from 19a6fbe to b1e6613 Compare March 2, 2023 01:50

zmerlynn force-pushed the add-flake-report branch from b1e6613 to a0fd95c Compare March 2, 2023 02:07

zmerlynn force-pushed the add-flake-report branch from fffcfb3 to 3837a59 Compare March 2, 2023 17:04

google-oss-prow bot added size/XXL and removed size/L labels Mar 2, 2023

zmerlynn force-pushed the add-flake-report branch from 3837a59 to 2d69f08 Compare March 2, 2023 17:13

zmerlynn changed the title ~~WIP: Add flake report script / build~~ Add automation to report on recent build flakes Mar 2, 2023

zmerlynn marked this pull request as ready for review March 2, 2023 17:19

google-oss-prow bot removed the do-not-merge/work-in-progress label Mar 2, 2023

google-oss-prow bot requested review from aLekSer and cyriltovena March 2, 2023 17:19

zmerlynn assigned gongmax and markmandel Mar 2, 2023

zmerlynn removed request for aLekSer and cyriltovena March 2, 2023 17:21

gongmax approved these changes Mar 3, 2023

View reviewed changes

google-oss-prow bot added the lgtm label Mar 3, 2023

aimuz reviewed Mar 3, 2023

View reviewed changes

zmerlynn added 6 commits March 3, 2023 15:50

Add make update-go-deps to update all vendored deps, and run it

d253d91

Address review comments, get it back working :/

6bdd77d

Actually add test

64d71d3

Tweak appearance a bit, add simplecss

7864708

Change methology flightly

a85ff8f

zmerlynn force-pushed the add-flake-report branch from acf95c9 to a85ff8f Compare March 3, 2023 15:50

google-oss-prow bot removed the lgtm label Mar 3, 2023

zmerlynn mentioned this pull request Mar 3, 2023

Flake in TestControllerSyncGameServer/Creating_a_new_GameServer #3016

Closed

zmerlynn enabled auto-merge (squash) March 3, 2023 17:05

Fix open <b>

ad692ca

zmerlynn merged commit 7671529 into googleforgames:main Mar 3, 2023

Kalaiselvi84 added the kind/feature New features for Agones label Apr 10, 2023

Kalaiselvi84 added this to the 1.31.0 milestone Apr 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automation to report on recent build flakes #3012

Add automation to report on recent build flakes #3012

zmerlynn commented Mar 2, 2023 •

edited

Loading

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

gongmax commented Mar 3, 2023

google-oss-prow bot commented Mar 3, 2023

markmandel commented Mar 3, 2023

markmandel commented Mar 3, 2023

zmerlynn commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

aimuz Mar 3, 2023

zmerlynn Mar 3, 2023

google-oss-prow bot commented Mar 3, 2023

zmerlynn commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

Add automation to report on recent build flakes #3012

Add automation to report on recent build flakes #3012

Conversation

zmerlynn commented Mar 2, 2023 • edited Loading

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

agones-bot commented Mar 2, 2023

gongmax commented Mar 3, 2023

google-oss-prow bot commented Mar 3, 2023

markmandel commented Mar 3, 2023

markmandel commented Mar 3, 2023

zmerlynn commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

aimuz Mar 3, 2023

Choose a reason for hiding this comment

zmerlynn Mar 3, 2023

Choose a reason for hiding this comment

google-oss-prow bot commented Mar 3, 2023

zmerlynn commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

agones-bot commented Mar 3, 2023

zmerlynn commented Mar 2, 2023 •

edited

Loading