Add leader election feature to `agones-controller` #3025

chiayi · 2023-03-13T22:20:52Z

WIP PR.

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

/kind hotfix

What this PR does / Why we need it:
Leader Election feature for agones-controller

Which issue(s) this PR fixes:
Work on #2797

Special notes for your reviewer:

agones-bot · 2023-03-13T22:27:06Z

Build Failed 😱

Build Id: b57b23a3-75d0-433a-80d7-eed5198019e8

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

cmd/controller/main.go

zmerlynn · 2023-03-13T23:02:35Z

cmd/controller/main.go

@@ -433,6 +521,22 @@ type httpServer struct {
 	http.ServeMux
 }

+func buildConfig(kubeconfig string) (*rest.Config, error) {


figure out if we need this? we presumably construct a client elsewhere and that would need similar logic?

agones-bot · 2023-03-23T19:55:50Z

Build Failed 😱

Build Id: 89d1e47b-0f4a-4993-80be-83f85012b9ea

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-24T18:13:12Z

Build Failed 😱

Build Id: d27d4de6-044f-40ab-9b72-cee323a97b6c

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-24T18:28:50Z

Build Failed 😱

Build Id: 534f5263-129f-4fac-a5b9-bee19e0fa7f4

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-24T19:09:50Z

Build Failed 😱

Build Id: 85cefdce-d12c-47a8-b4d0-4ac9670e0aac

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-24T19:21:56Z

Build Failed 😱

Build Id: 6aed74b5-9b8a-4558-90aa-9ee2fe165470

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-24T21:08:09Z

Build Failed 😱

Build Id: 12fc2ca5-599a-4ea1-ae04-56c015edb82a

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

cmd/controller/main.go

cmd/extensions/pprof.go

install/helm/agones/defaultfeaturegates.yaml

zmerlynn · 2023-03-27T16:50:38Z

install/helm/agones/values.yaml

@@ -94,6 +94,7 @@ agones:
      failureThreshold: 3
      timeoutSeconds: 1
    allocationBatchWaitTime: 500ms
+    replicas: 1


I think we should derive this somehow from the feature gate status, but allow it to be forced one way or the other. Here's my thought:

We add a boolean --leader-elected flag to controller to enable/disable leader election. This acts as the ultimate chicken-switch.

This is not a typed structure, so we could have "replicas: derived-from-feature-gate" here, and then later, we apply the following logic:

If replicas is not an int:

If SplitControllerAndExtensions is enabled:

derive replicas as 2

Else (SplitControllerAndExtensions is disabled):

derive replicas as 1

Else (replicas is an int)

replicas is the value set

if SplitControllerAndExtensions is not enabled:

validate replicas == 1

Then, using the above logic, if derived replicas is >1, we enable --leader-election and enable the PDB.

If you want to add a function for derived replicas, you can define it in helpers and use it elsewhere (it's a little fussy because it effectively defines a template, but it works). Example:

agones/install/helm/agones/templates/_helpers.tpl

Lines 25 to 41 in 5ddbe60

{{/*

Create a default fully qualified app name.

We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).

If release name contains chart name it will be used as a full name.

*/}}

{{- define "agones.fullname" -}}

{{- if .Values.fullnameOverride -}}

{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}

{{- else -}}

{{- $name := default .Chart.Name .Values.nameOverride -}}

{{- if contains $name .Release.Name -}}

{{- .Release.Name | trunc 63 | trimSuffix "-" -}}

{{- else -}}

{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}

{{- end -}}

{{- end -}}

{{- end -}}

install/helm/agones/templates/controller.yaml

agones-bot · 2023-03-27T17:20:05Z

Build Failed 😱

Build Id: 00cf887c-a8fe-4700-9f7d-a28e239cac92

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-27T22:12:10Z

Build Succeeded 👏

Build Id: c5fc03ce-c25a-4f7e-a75f-1edc2d4c59d7

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-e8311ab-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-e8311ab-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-e8311ab-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-e8311ab-amd64
Linux C++ SDK (build): agonessdk-1.31.0-e8311ab-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-e8311ab-amd64.zip

A preview of the website (the last 30 builds are retained):

https://e8311ab-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3025/head:pr_3025 && git checkout pr_3025
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-e8311ab-amd64

zmerlynn

Good job! Mostly just nits this time around.

cmd/controller/main.go

agones-bot · 2023-03-27T23:41:37Z

Build Succeeded 👏

Build Id: e8cfd135-a2db-4ef9-96f8-5a2af7c0f88b

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-f967b5d-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-f967b5d-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-f967b5d-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-f967b5d-amd64
Linux C++ SDK (build): agonessdk-1.31.0-f967b5d-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-f967b5d-amd64.zip

A preview of the website (the last 30 builds are retained):

https://f967b5d-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3025/head:pr_3025 && git checkout pr_3025
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-f967b5d-amd64

agones-bot · 2023-03-28T01:06:38Z

Build Succeeded 👏

Build Id: f2eca970-b964-41db-990b-3862be55af9c

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-62d1bf2-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-62d1bf2-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-62d1bf2-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-62d1bf2-amd64
Linux C++ SDK (build): agonessdk-1.31.0-62d1bf2-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-62d1bf2-amd64.zip

A preview of the website (the last 30 builds are retained):

https://62d1bf2-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3025/head:pr_3025 && git checkout pr_3025
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-62d1bf2-amd64

zmerlynn · 2023-03-28T17:33:36Z

cmd/controller/main.go

-	logger.Info("Shut down agones controllers")
+		<-ctx.Done()
+		logger.Info("Shut down agones controllers")
+	})


Nice, exactly what I was hoping to see.

google-oss-prow · 2023-03-28T17:34:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chiayi, zmerlynn
Once this PR has been reviewed and has the lgtm label, please assign markmandel for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow · 2023-03-28T17:36:37Z

New changes are detected. LGTM label has been removed.

agones-bot · 2023-03-28T19:45:32Z

Build Failed 😱

Build Id: 5d5b3027-cb86-4b73-8786-4635c7389169

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-28T22:50:43Z

Build Failed 😱

Build Id: 2b5a9971-8fa4-45eb-9887-3f4b49c4545f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2023-03-29T16:48:47Z

Build Succeeded 👏

Build Id: b0ff3cf4-4f6c-4dd9-9913-6f85d1763310

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.31.0-40e33cc-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.31.0-40e33cc-linux-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.31.0-40e33cc-amd64
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.31.0-40e33cc-amd64
Linux C++ SDK (build): agonessdk-1.31.0-40e33cc-amd64-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.31.0-40e33cc-amd64.zip

A preview of the website (the last 30 builds are retained):

https://40e33cc-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/3025/head:pr_3025 && git checkout pr_3025
helm install agones ./install/helm/agones --namespace agones-system --agones.image.release=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.31.0-40e33cc-amd64

Adds --leader-election flag, which enabled leader election amongst controller replicas. Helm changes to follow.

google-oss-prow bot added the do-not-merge/work-in-progress label Mar 13, 2023

google-oss-prow bot requested review from aLekSer and roberthbailey March 13, 2023 22:20

chiayi requested review from zmerlynn and removed request for aLekSer and roberthbailey March 13, 2023 22:20

google-oss-prow bot added the size/L label Mar 13, 2023

zmerlynn reviewed Mar 13, 2023

View reviewed changes

chiayi force-pushed the agoneshabranch branch from 19ad845 to c57f731 Compare March 23, 2023 19:49

chiayi force-pushed the agoneshabranch branch from c57f731 to 39dac45 Compare March 24, 2023 14:50

google-oss-prow bot added size/XXL and removed size/L labels Mar 24, 2023

chiayi force-pushed the agoneshabranch branch 2 times, most recently from c91457f to ae2a5c3 Compare March 24, 2023 16:54

chiayi force-pushed the agoneshabranch branch from ae2a5c3 to 15a49a5 Compare March 24, 2023 18:19

chiayi force-pushed the agoneshabranch branch from 15a49a5 to 5aff406 Compare March 24, 2023 18:54

chiayi force-pushed the agoneshabranch branch from 5aff406 to dd382cf Compare March 24, 2023 19:12

chiayi force-pushed the agoneshabranch branch from dd382cf to 7bab608 Compare March 24, 2023 20:47

chiayi changed the title ~~WIP push~~ Add leader election feature to agones-controller Mar 24, 2023

zmerlynn reviewed Mar 27, 2023

View reviewed changes

chiayi force-pushed the agoneshabranch branch from 7bab608 to 4b84915 Compare March 27, 2023 17:01

chiayi force-pushed the agoneshabranch branch from 4b84915 to b7f7594 Compare March 27, 2023 21:12

chiayi force-pushed the agoneshabranch branch from b7f7594 to e8311ab Compare March 27, 2023 21:24

chiayi force-pushed the agoneshabranch branch from e8311ab to f967b5d Compare March 27, 2023 22:54

zmerlynn reviewed Mar 27, 2023

View reviewed changes

chiayi force-pushed the agoneshabranch branch from f967b5d to 62d1bf2 Compare March 27, 2023 23:46

chiayi marked this pull request as ready for review March 28, 2023 16:57

google-oss-prow bot removed the do-not-merge/work-in-progress label Mar 28, 2023

google-oss-prow bot requested review from markmandel and roberthbailey March 28, 2023 16:57

zmerlynn approved these changes Mar 28, 2023

View reviewed changes

google-oss-prow bot assigned zmerlynn Mar 28, 2023

google-oss-prow bot added the lgtm label Mar 28, 2023

zmerlynn force-pushed the agoneshabranch branch from 62d1bf2 to a506de5 Compare March 28, 2023 17:36

google-oss-prow bot removed the lgtm label Mar 28, 2023

zmerlynn enabled auto-merge (squash) March 28, 2023 17:37

Add leader election to controller

40e33cc

auto-merge was automatically disabled March 28, 2023 21:09
Head branch was pushed to by a user without write access

chiayi force-pushed the agoneshabranch branch from a506de5 to 40e33cc Compare March 28, 2023 21:09

zmerlynn merged commit 7aace06 into googleforgames:main Mar 29, 2023

Kalaiselvi84 pushed a commit to Kalaiselvi84/agones that referenced this pull request Mar 31, 2023

Add leader election to controller (googleforgames#3025)

1fd4b2a

Adds --leader-election flag, which enabled leader election amongst controller replicas. Helm changes to follow.

Kalaiselvi84 added the kind/feature New features for Agones label Apr 10, 2023

Kalaiselvi84 added this to the 1.31.0 milestone Apr 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add leader election feature to `agones-controller` #3025

Add leader election feature to `agones-controller` #3025

chiayi commented Mar 13, 2023 •

edited

Loading

agones-bot commented Mar 13, 2023

zmerlynn Mar 13, 2023

agones-bot commented Mar 23, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

zmerlynn Mar 27, 2023

agones-bot commented Mar 27, 2023

agones-bot commented Mar 27, 2023

zmerlynn left a comment

agones-bot commented Mar 27, 2023

agones-bot commented Mar 28, 2023

zmerlynn Mar 28, 2023

google-oss-prow bot commented Mar 28, 2023

google-oss-prow bot commented Mar 28, 2023

agones-bot commented Mar 28, 2023

agones-bot commented Mar 28, 2023

agones-bot commented Mar 29, 2023

	{{/*
	Create a default fully qualified app name.
	We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
	If release name contains chart name it will be used as a full name.
	*/}}
	{{- define "agones.fullname" -}}
	{{- if .Values.fullnameOverride -}}
	{{- .Values.fullnameOverride \| trunc 63 \| trimSuffix "-" -}}
	{{- else -}}
	{{- $name := default .Chart.Name .Values.nameOverride -}}
	{{- if contains $name .Release.Name -}}
	{{- .Release.Name \| trunc 63 \| trimSuffix "-" -}}
	{{- else -}}
	{{- printf "%s-%s" .Release.Name $name \| trunc 63 \| trimSuffix "-" -}}
	{{- end -}}
	{{- end -}}
	{{- end -}}

Add leader election feature to agones-controller #3025

Add leader election feature to agones-controller #3025

Conversation

chiayi commented Mar 13, 2023 • edited Loading

agones-bot commented Mar 13, 2023

zmerlynn Mar 13, 2023

Choose a reason for hiding this comment

agones-bot commented Mar 23, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

agones-bot commented Mar 24, 2023

zmerlynn Mar 27, 2023

Choose a reason for hiding this comment

agones-bot commented Mar 27, 2023

agones-bot commented Mar 27, 2023

zmerlynn left a comment

Choose a reason for hiding this comment

agones-bot commented Mar 27, 2023

agones-bot commented Mar 28, 2023

zmerlynn Mar 28, 2023

Choose a reason for hiding this comment

google-oss-prow bot commented Mar 28, 2023

google-oss-prow bot commented Mar 28, 2023

agones-bot commented Mar 28, 2023

agones-bot commented Mar 28, 2023

agones-bot commented Mar 29, 2023

Add leader election feature to `agones-controller` #3025

Add leader election feature to `agones-controller` #3025

chiayi commented Mar 13, 2023 •

edited

Loading