Flakiness: TestGameServerAllocationDeletionOnUnAllocate #1328

markmandel · 2020-02-08T08:07:01Z

Realised that Go test will compile and run tests in parallel per file,
which means we could shutdown the controller while an allocation is in
process - which means, it will fail completely, as the Kubernetes API
cannot reach the controller's http endpoint for allocation.

Split the controller tests into a separate package which could be run
with -parallel=1 so that there can be no race condition.

Also did some grouping of e2e test functionality to reduce code
repetition.

Closes #1326

agones-bot · 2020-02-08T08:23:10Z

Build Succeeded 👏

Build Id: 38d4ccc6-d19f-4f1f-89ba-1120c2416aab

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-90c0f81
image: gcr.io/agones-images/agones-sdk:1.4.0-90c0f81
image: gcr.io/agones-images/agones-ping:1.4.0-90c0f81
Linux C++ SDK (build): agonessdk-1.4.0-90c0f81-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-90c0f81.zip

A preview of the website (the last 30 builds are retained):

https://90c0f81-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-90c0f81

agones-bot · 2020-02-08T14:13:10Z

Build Succeeded 👏

Build Id: c7752cf5-faf8-449f-afae-eeb9056c45f1

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-90c0f81
image: gcr.io/agones-images/agones-sdk:1.4.0-90c0f81
image: gcr.io/agones-images/agones-ping:1.4.0-90c0f81
Linux C++ SDK (build): agonessdk-1.4.0-90c0f81-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-90c0f81.zip

A preview of the website (the last 30 builds are retained):

https://90c0f81-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-90c0f81

agones-bot · 2020-02-08T14:39:34Z

Build Succeeded 👏

Build Id: 56c9a35f-385e-411e-8eb6-cb78fff75d40

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-90c0f81
image: gcr.io/agones-images/agones-sdk:1.4.0-90c0f81
image: gcr.io/agones-images/agones-ping:1.4.0-90c0f81
Linux C++ SDK (build): agonessdk-1.4.0-90c0f81-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-90c0f81.zip

A preview of the website (the last 30 builds are retained):

https://90c0f81-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-90c0f81

agones-bot · 2020-02-08T15:13:03Z

Build Failed 😱

Build Id: 7e56763e-881f-42b6-9716-60df13350e64

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2020-02-08T16:16:20Z

Got bit by different flaky bug #1276 - running again.

agones-bot · 2020-02-08T16:32:22Z

Build Succeeded 👏

Build Id: 91da735d-9b25-46c4-8b66-538aa59682ff

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-c613c8c
image: gcr.io/agones-images/agones-sdk:1.4.0-c613c8c
image: gcr.io/agones-images/agones-ping:1.4.0-c613c8c
Linux C++ SDK (build): agonessdk-1.4.0-c613c8c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-c613c8c.zip

A preview of the website (the last 30 builds are retained):

https://c613c8c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-c613c8c

markmandel · 2020-02-08T16:59:23Z

Running a bunch of tests to make sure it's working, and if that passes, will cleanup and submit.

agones-bot · 2020-02-08T17:14:51Z

Build Succeeded 👏

Build Id: d4871666-6079-4b34-b11e-0aeb86433870

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-c613c8c
image: gcr.io/agones-images/agones-sdk:1.4.0-c613c8c
image: gcr.io/agones-images/agones-ping:1.4.0-c613c8c
Linux C++ SDK (build): agonessdk-1.4.0-c613c8c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-c613c8c.zip

A preview of the website (the last 30 builds are retained):

https://c613c8c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-c613c8c

agones-bot · 2020-02-08T17:49:02Z

Build Succeeded 👏

Build Id: 6b119ad8-3c68-4f60-b748-95aef181f5b6

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-c613c8c
image: gcr.io/agones-images/agones-sdk:1.4.0-c613c8c
image: gcr.io/agones-images/agones-ping:1.4.0-c613c8c
Linux C++ SDK (build): agonessdk-1.4.0-c613c8c-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-c613c8c.zip

A preview of the website (the last 30 builds are retained):

https://c613c8c-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-c613c8c

agones-bot · 2020-02-08T22:38:21Z

Build Failed 😱

Build Id: 3affb86f-ed11-49d7-93cf-01d915a01f02

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-02-08T22:58:33Z

Build Succeeded 👏

Build Id: 0226c81d-675e-4b13-bbea-65a8787eb1a9

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-569daf4
image: gcr.io/agones-images/agones-sdk:1.4.0-569daf4
image: gcr.io/agones-images/agones-ping:1.4.0-569daf4
Linux C++ SDK (build): agonessdk-1.4.0-569daf4-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-569daf4.zip

A preview of the website (the last 30 builds are retained):

https://569daf4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-569daf4

pooneh-m · 2020-02-10T22:45:27Z

test/e2e/controller/missing_test.go

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-package e2e
+package controller


Can we rename the file name to something else? missing_test.go sounds like it includes the tests that were missing from test coverage :D

crash_test.go ? That better?

Tests where the controller crashes for whatever reason? WDYT?

Thanks, much better!

SGTM! Will make the change.

pooneh-m · 2020-02-10T22:57:03Z

test/e2e/controller/missing_test.go

 // getAgonesControllerPods returns all the Agones controller pods
 func getAgonesControllerPods() (*corev1.PodList, error) {
 	opts := metav1.ListOptions{LabelSelector: labels.Set{"agones.dev/role": "controller"}.String()}
 	return framework.KubeClient.CoreV1().Pods("agones-system").List(opts)
 }
+
+func defaultGameServer(namespace string) *agonesv1.GameServer {


Can we refactor this instead?

Yeah I thought about centralising that too - moving it into framework maybe. Wasn't sure if 2 duplication was enough to warrant it. If you think so, I'm happy to do it.

I assume that is what you mean here?

Correct, thanks for the clarification. framework.go should be a good place or you can add a new file such as testhelper and centralize the common helper methods there?

Sounds good - I like the idea of testhelper.go or similar (I think we have a previous pattern for this somewhere, maybe, I can copy).

google-oss-robot · 2020-02-10T23:01:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: markmandel, pooneh-m

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [markmandel,pooneh-m]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Realised that Go test will compile and run tests in parallel per file, which means we could shutdown the controller while an allocation is in process - which means, it will fail completely, as the Kubernetes API cannot reach the controller's http endpoint for allocation. Split the controller tests into a separate package which could be run with -parallel=1 so that there can be no race condition. Also did some grouping of e2e test functionality to reduce code repetition. Closes googleforgames#1326

google-oss-robot · 2020-02-11T01:56:27Z

New changes are detected. LGTM label has been removed.

agones-bot · 2020-02-11T02:11:47Z

Build Failed 😱

Build Id: ae943e9a-ac4b-4da6-965f-cff6d3f585c3

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-02-11T02:29:50Z

Build Succeeded 👏

Build Id: dd3a7cd5-cf28-47e3-9a06-64dcbb4a4a61

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.4.0-b94b0f0
image: gcr.io/agones-images/agones-sdk:1.4.0-b94b0f0
image: gcr.io/agones-images/agones-ping:1.4.0-b94b0f0
Linux C++ SDK (build): agonessdk-1.4.0-b94b0f0-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.4.0-b94b0f0.zip

A preview of the website (the last 30 builds are retained):

https://b94b0f0-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1328/head:pr_1328 && git checkout pr_1328
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.4.0-b94b0f0

Realised that Go test will compile and run tests in parallel per file, which means we could shutdown the controller while an allocation is in process - which means, it will fail completely, as the Kubernetes API cannot reach the controller's http endpoint for allocation. Split the controller tests into a separate package which could be run with -parallel=1 so that there can be no race condition. Also did some grouping of e2e test functionality to reduce code repetition. Closes googleforgames#1326

markmandel added kind/bug These are bugs. area/tests Unit tests, e2e tests, anything to make sure things don't break labels Feb 8, 2020

google-oss-robot added the do-not-merge/work-in-progress label Feb 8, 2020

google-oss-robot requested review from cyriltovena and dzlier-gcp February 8, 2020 08:07

google-oss-robot added approved size/L labels Feb 8, 2020

markmandel linked an issue Feb 8, 2020 that may be closed by this pull request

Flaky: TestGameServerAllocationDeletionOnUnAllocate #1326

Closed

markmandel force-pushed the flaky/TestGameServerAllocationDeletionOnUnAllocate branch from c613c8c to 6574c55 Compare February 8, 2020 22:09

markmandel marked this pull request as ready for review February 8, 2020 22:09

google-oss-robot removed the do-not-merge/work-in-progress label Feb 8, 2020

markmandel force-pushed the flaky/TestGameServerAllocationDeletionOnUnAllocate branch from 6574c55 to 569daf4 Compare February 8, 2020 22:23

markmandel removed the request for review from dzlier-gcp February 10, 2020 21:44

pooneh-m requested review from pooneh-m and removed request for cyriltovena February 10, 2020 22:30

pooneh-m reviewed Feb 10, 2020

View reviewed changes

pooneh-m approved these changes Feb 10, 2020

View reviewed changes

google-oss-robot assigned pooneh-m Feb 10, 2020

google-oss-robot added the lgtm label Feb 10, 2020

markmandel force-pushed the flaky/TestGameServerAllocationDeletionOnUnAllocate branch from 569daf4 to b94b0f0 Compare February 11, 2020 01:56

google-oss-robot removed the lgtm label Feb 11, 2020

markmandel merged commit f998c8e into googleforgames:master Feb 11, 2020

markmandel added this to the 1.4.0 milestone Feb 11, 2020

markmandel deleted the flaky/TestGameServerAllocationDeletionOnUnAllocate branch February 13, 2020 18:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flakiness: TestGameServerAllocationDeletionOnUnAllocate #1328

Flakiness: TestGameServerAllocationDeletionOnUnAllocate #1328

markmandel commented Feb 8, 2020 •

edited

Loading

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

markmandel commented Feb 8, 2020

agones-bot commented Feb 8, 2020

markmandel commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

pooneh-m Feb 10, 2020

markmandel Feb 10, 2020 •

edited

Loading

pooneh-m Feb 10, 2020

markmandel Feb 10, 2020

pooneh-m Feb 10, 2020

markmandel Feb 10, 2020

pooneh-m Feb 10, 2020

markmandel Feb 10, 2020

google-oss-robot commented Feb 10, 2020

google-oss-robot commented Feb 11, 2020

agones-bot commented Feb 11, 2020

agones-bot commented Feb 11, 2020

Flakiness: TestGameServerAllocationDeletionOnUnAllocate #1328

Flakiness: TestGameServerAllocationDeletionOnUnAllocate #1328

Conversation

markmandel commented Feb 8, 2020 • edited Loading

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

markmandel commented Feb 8, 2020

agones-bot commented Feb 8, 2020

markmandel commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

agones-bot commented Feb 8, 2020

pooneh-m Feb 10, 2020

Choose a reason for hiding this comment

markmandel Feb 10, 2020 • edited Loading

Choose a reason for hiding this comment

pooneh-m Feb 10, 2020

Choose a reason for hiding this comment

markmandel Feb 10, 2020

Choose a reason for hiding this comment

pooneh-m Feb 10, 2020

Choose a reason for hiding this comment

markmandel Feb 10, 2020

Choose a reason for hiding this comment

pooneh-m Feb 10, 2020

Choose a reason for hiding this comment

markmandel Feb 10, 2020

Choose a reason for hiding this comment

google-oss-robot commented Feb 10, 2020

google-oss-robot commented Feb 11, 2020

agones-bot commented Feb 11, 2020

agones-bot commented Feb 11, 2020

markmandel commented Feb 8, 2020 •

edited

Loading

markmandel Feb 10, 2020 •

edited

Loading