From e700d5bd0c5039f77a919bacb59bb30016fc0bb3 Mon Sep 17 00:00:00 2001 From: Zach Loafman Date: Tue, 29 Nov 2022 09:11:25 -0800 Subject: [PATCH] LifecycleContract alpha: Implement (immutable) scale subresource, add pdb (#2807) This PR, under the `LifecycleContract` feature gate suggested in #2794 * Adds a scale resource to `GameServer` by adding an `immutableReplicas` field, which has a `default`, `min` and `max` of 1. Having a `scale` subresource lets us define a `PodDisruptionBudget` that can be set to `maxUnavailable: 0%` [1]. * Adds a PDB per namespace with label selector `agones.dev/safe-to-evict: "false"`, which nothing yet adds. * Adds a mechanism to get feature gate values in Helm by using: {{- $featureGates := include "agones.featureGates" . | fromYaml }} * Cleanup / documentation of feature gate mechanisms After this PR, it's possible to define a fleet with the label and have all `GameServer` pods protected by a `PodDisruptionBudget`, e.g.: ``` $ kubectl scale fleet/fleet-example --replicas=5 fleet.agones.dev/fleet-example scaled $ kubectl describe pdb Name: agones-gameserver-safe-to-evict-false Namespace: default Max unavailable: 0% Selector: agones.dev/safe-to-evict=false Status: Allowed disruptions: 0 Current: 4 Desired: 5 Total: 5 Events: ``` Additionally, because min/max/default are 1, Kubernetes enforces the immutability for us: ``` $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=1 gameserver.agones.dev/fleet-example-k6dfs-6m5nq scaled $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=2 The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 2: spec.immutableReplicas in body should be less than or equal to 1 $ kubectl scale gs/fleet-example-k6dfs-6m5nq --replicas=0 The GameServer "fleet-example-k6dfs-6m5nq" is invalid: spec.immutableReplicas: Invalid value: 0: spec.immutableReplicas in body should be greater than or equal to 1 ``` The only artifact of this addition is a new field in the Spec/Status named `immutableReplicas`, in the Kubernetes object. This field is not present in the in-memory representation for `GameServer`, nor is it present in `etcd` (by defaulting rules). The field is visible on `describe` or `get -oyaml`, but is otherwise ignored. [1] https://kubernetes.io/docs/tasks/run-application/configure-pdb/#identify-an-application-to-protect --- build/Makefile | 3 +- cloudbuild.yaml | 3 +- install/helm/agones/defaultfeaturegates.yaml | 29 ++++++++ install/helm/agones/templates/_helpers.tpl | 24 +++++++ .../templates/crds/_gameserverspecschema.yaml | 8 +++ .../templates/crds/_gameserverstatus.yaml | 8 +++ install/helm/agones/templates/crds/fleet.yaml | 3 +- .../agones/templates/crds/gameserver.yaml | 17 ++++- .../agones/templates/crds/gameserverset.yaml | 3 +- install/helm/agones/templates/pdb.yaml | 30 ++++++++ install/yaml/install.yaml | 17 ++++- pkg/apis/agones/v1/gameserver.go | 2 + pkg/util/runtime/features.go | 70 +++++++++++++++---- site/content/en/docs/Guides/feature-stages.md | 9 +-- 14 files changed, 199 insertions(+), 27 deletions(-) create mode 100644 install/helm/agones/defaultfeaturegates.yaml create mode 100644 install/helm/agones/templates/pdb.yaml diff --git a/build/Makefile b/build/Makefile index cb7038383a..34c625f758 100644 --- a/build/Makefile +++ b/build/Makefile @@ -67,7 +67,8 @@ KIND_CONTAINER_NAME=$(KIND_PROFILE)-control-plane # Game Server image to use while doing end-to-end tests GS_TEST_IMAGE ?= gcr.io/agones-images/simple-game-server:0.14 -ALPHA_FEATURE_GATES ?= "PlayerTracking=true&PlayerAllocationFilter=true&SDKGracefulTermination=true&ResetMetricsOnDelete=true" +# Enable all alpha feature gates. Keep in sync with `false` (alpha) entries in pkg/util/runtime/features.go:featureDefaults +ALPHA_FEATURE_GATES ?= "LifecycleContract=true&PlayerAllocationFilter=true&PlayerTracking=true&ResetMetricsOnDelete=true&SDKGracefulTermination=true&Example=true" # Build with Windows support WITH_WINDOWS=1 diff --git a/cloudbuild.yaml b/cloudbuild.yaml index af17a5f021..2799444fcb 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -241,10 +241,11 @@ steps: # # Run the e2e tests with FeatureGates inverted compared to Stable # +# Keep in sync with (the inverse of) pkg/util/runtime/features.go:featureDefaults - name: 'e2e-runner' args: - - 'PlayerTracking=true&StateAllocationFilter=false&PlayerAllocationFilter=true&SDKGracefulTermination=true&CustomFasSyncInterval=false&ResetMetricsOnDelete=true' + - 'CustomFasSyncInterval=false&StateAllocationFilter=false&LifecycleContract=true&PlayerAllocationFilter=true&PlayerTracking=true&ResetMetricsOnDelete=true&SDKGracefulTermination=true&Example=true' - 'e2e-test-cluster' - "${_REGISTRY}" id: e2e-feature-gates diff --git a/install/helm/agones/defaultfeaturegates.yaml b/install/helm/agones/defaultfeaturegates.yaml new file mode 100644 index 0000000000..1daf815591 --- /dev/null +++ b/install/helm/agones/defaultfeaturegates.yaml @@ -0,0 +1,29 @@ +# Copyright 2022 Google LLC All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Default values for feature gates. Keep in sync with pkg/util/runtime/features.go:featureDefaults + +# Beta features +CustomFasSyncInterval: true +StateAllocationFilter: true + +# Alpha features +LifecycleContract: false +PlayerAllocationFilter: false +PlayerTracking: false +ResetMetricsOnDelete: false +SDKGracefulTermination: false + +# Example feature +Example: false \ No newline at end of file diff --git a/install/helm/agones/templates/_helpers.tpl b/install/helm/agones/templates/_helpers.tpl index 090c2b11ce..badca7649c 100644 --- a/install/helm/agones/templates/_helpers.tpl +++ b/install/helm/agones/templates/_helpers.tpl @@ -46,3 +46,27 @@ Create chart name and version as used by the chart label. {{- define "agones.chart" -}} {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} {{- end -}} + +{{/* +Creates a YAML object representing known feature gates. Can then be used as: + {{- $featureGates := include "agones.featureGates" . | fromYaml }} +then you can check a feature gate with e.g. $featureGates.Example + +Implemented by intentionally duplicating YAML - later keys take precedence. +So we start with defaultfeaturegates.yaml and then splat intentionally set +feature gates. In the process, we validate that the feature gate is known. +*/}} +{{- define "agones.featureGates" -}} +{{- .Files.Get "defaultfeaturegates.yaml" -}} +{{- if $.Values.agones.featureGates }} +{{- $gates := .Files.Get "defaultfeaturegates.yaml" | fromYaml }} +{{- range splitList "&" $.Values.agones.featureGates }} +{{- $f := splitn "=" 2 . -}} +{{- if hasKey $gates $f._0 }} +{{$f._0}}: {{$f._1}} +{{- else -}} +{{- printf "Unknown feature gate %q" $f._0 | fail -}} +{{- end -}} +{{- end -}} +{{- end -}} +{{- end -}} \ No newline at end of file diff --git a/install/helm/agones/templates/crds/_gameserverspecschema.yaml b/install/helm/agones/templates/crds/_gameserverspecschema.yaml index 89f1b4da50..46b665343e 100644 --- a/install/helm/agones/templates/crds/_gameserverspecschema.yaml +++ b/install/helm/agones/templates/crds/_gameserverspecschema.yaml @@ -153,4 +153,12 @@ properties: type: integer title: The initial player capacity of this Game Server minimum: 0 +{{- if .featureLifecycleContract }} + immutableReplicas: + type: integer + title: Immutable count of Pods to a GameServer. Always 1. (Implementation detail of implementing the Scale subresource.) + default: 1 + minimum: 1 + maximum: 1 +{{- end }} {{- end }} diff --git a/install/helm/agones/templates/crds/_gameserverstatus.yaml b/install/helm/agones/templates/crds/_gameserverstatus.yaml index e415022cbc..009110082d 100644 --- a/install/helm/agones/templates/crds/_gameserverstatus.yaml +++ b/install/helm/agones/templates/crds/_gameserverstatus.yaml @@ -65,4 +65,12 @@ status: nullable: true items: type: string +{{- if .featureLifecycleContract }} + immutableReplicas: + type: integer + title: Immutable count of Pods to a GameServer. Always 1. (Implementation detail of implementing the Scale subresource.) + default: 1 + minimum: 1 + maximum: 1 +{{- end}} {{- end}} diff --git a/install/helm/agones/templates/crds/fleet.yaml b/install/helm/agones/templates/crds/fleet.yaml index 0f4c8d6506..d9fc99b6eb 100644 --- a/install/helm/agones/templates/crds/fleet.yaml +++ b/install/helm/agones/templates/crds/fleet.yaml @@ -13,6 +13,7 @@ # limitations under the License. {{- if .Values.agones.crds.install }} +{{- $featureGates := include "agones.featureGates" . | fromYaml }} apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition @@ -99,7 +100,7 @@ spec: - type: integer - type: string template: - {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }} + {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }} {{- include "gameserver.schema" $data | indent 17 }} status: description: 'FleetStatus is the status of a Fleet. More info: diff --git a/install/helm/agones/templates/crds/gameserver.yaml b/install/helm/agones/templates/crds/gameserver.yaml index a2373f6a85..2c13e0788e 100644 --- a/install/helm/agones/templates/crds/gameserver.yaml +++ b/install/helm/agones/templates/crds/gameserver.yaml @@ -13,6 +13,7 @@ # limitations under the License. {{- if .Values.agones.crds.install }} +{{- $featureGates := include "agones.featureGates" . | fromYaml }} apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition @@ -55,7 +56,17 @@ spec: type: date schema: openAPIV3Schema: - {{- $data := dict "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }} - {{- include "gameserver.schema" . | indent 9 }} - {{- include "gameserver.status" . | indent 11 }} # in an include, as it's easier to align + {{- $data := dict "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }} + {{- include "gameserver.schema" $data | indent 9 }}{{- /* include the schema then status, as it's easier to align */ -}} + {{- include "gameserver.status" $data | indent 11 }} +{{- if $featureGates.LifecycleContract }} + subresources: + # scale enables the scale subresource. We can't actually scale GameServers, but this allows + # for the use of PodDisruptionBudget (PDB) without having to use a PDB per Pod. + scale: + # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas. + specReplicasPath: .spec.immutableReplicas + # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas. + statusReplicasPath: .status.immutableReplicas +{{- end }} {{- end }} diff --git a/install/helm/agones/templates/crds/gameserverset.yaml b/install/helm/agones/templates/crds/gameserverset.yaml index 37cea6f171..f9340b36e2 100644 --- a/install/helm/agones/templates/crds/gameserverset.yaml +++ b/install/helm/agones/templates/crds/gameserverset.yaml @@ -13,6 +13,7 @@ # limitations under the License. {{- if .Values.agones.crds.install }} +{{- $featureGates := include "agones.featureGates" . | fromYaml }} apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition @@ -79,7 +80,7 @@ spec: - Packed - Distributed template: - {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields }} + {{- $data := dict "metadata" true "podPreserveUnknownFields" .Values.gameservers.podPreserveUnknownFields "featureLifecycleContract" $featureGates.LifecycleContract }} {{- include "gameserver.schema" $data | indent 18 }} status: description: 'GameServerSetStatus is the status of a GameServerSet. More info: diff --git a/install/helm/agones/templates/pdb.yaml b/install/helm/agones/templates/pdb.yaml new file mode 100644 index 0000000000..e73bf1edf0 --- /dev/null +++ b/install/helm/agones/templates/pdb.yaml @@ -0,0 +1,30 @@ +# Copyright 2022 Google LLC All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- $featureGates := include "agones.featureGates" . | fromYaml }} +{{- if $featureGates.LifecycleContract }} +{{- range .Values.gameservers.namespaces }} +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: agones-gameserver-safe-to-evict-false + namespace: {{ . }} +spec: + maxUnavailable: 0% + selector: + matchLabels: + agones.dev/safe-to-evict: "false" +{{- end }} +{{- end }} diff --git a/install/yaml/install.yaml b/install/yaml/install.yaml index 0ebbfa86d7..9bc9fe5662 100644 --- a/install/yaml/install.yaml +++ b/install/yaml/install.yaml @@ -9518,7 +9518,7 @@ spec: type: array nullable: true items: - type: string # in an include, as it's easier to align + type: string --- # Source: agones/templates/crds/gameserverallocationpolicy.yaml # Copyright 2019 Google LLC All Rights Reserved. @@ -15009,6 +15009,21 @@ spec: # See the License for the specific language governing permissions and # limitations under the License. --- +# Source: agones/templates/pdb.yaml +# Copyright 2022 Google LLC All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--- # Source: agones/templates/extensions.yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration diff --git a/pkg/apis/agones/v1/gameserver.go b/pkg/apis/agones/v1/gameserver.go index f9e7ea3ed6..61bc96222e 100644 --- a/pkg/apis/agones/v1/gameserver.go +++ b/pkg/apis/agones/v1/gameserver.go @@ -161,6 +161,7 @@ type GameServerSpec struct { // (Alpha, PlayerTracking feature flag) Players provides the configuration for player tracking features. // +optional Players *PlayersSpec `json:"players,omitempty"` + // immutableReplicas is present in gameservers.agones.dev but omitted here (it's always 1). } // PlayersSpec tracks the initial player capacity @@ -233,6 +234,7 @@ type GameServerStatus struct { // [FeatureFlag:PlayerTracking] // +optional Players *PlayerStatus `json:"players"` + // immutableReplicas is present in gameservers.agones.dev but omitted here (it's always 1). } // GameServerStatusPort shows the port that was allocated to a diff --git a/pkg/util/runtime/features.go b/pkg/util/runtime/features.go index a80a7ea8a9..4d078f2ed9 100644 --- a/pkg/util/runtime/features.go +++ b/pkg/util/runtime/features.go @@ -28,42 +28,82 @@ const ( // FeatureGateFlag is a name of a command line flag, which turns on specific tests for FeatureGates FeatureGateFlag = "feature-gates" - // FeatureExample is an example feature gate flag, used for testing and demonstrative purposes - FeatureExample Feature = "Example" + //////////////// + // Beta features - // FeaturePlayerTracking is a feature flag to enable/disable player tracking features. - FeaturePlayerTracking Feature = "PlayerTracking" + // FeatureCustomFasSyncInterval is a feature flag that enables a custom FleetAutoscaler resync interval + FeatureCustomFasSyncInterval Feature = "CustomFasSyncInterval" // FeatureStateAllocationFilter is a feature flag that enables state filtering on Allocation. FeatureStateAllocationFilter Feature = "StateAllocationFilter" + //////////////// + // Alpha features + + // FeatureLifecycleContract enables the `lifecycleContract` API to specify disruption tolerance. + FeatureLifecycleContract Feature = "LifecycleContract" + // FeaturePlayerAllocationFilter is a feature flag that enables the ability for Allocations to filter based on // player capacity. FeaturePlayerAllocationFilter Feature = "PlayerAllocationFilter" - // FeatureCustomFasSyncInterval is a feature flag that enables custom the FleetAutoscaler rsync interval - FeatureCustomFasSyncInterval Feature = "CustomFasSyncInterval" - - // FeatureSDKGracefulTermination is a feature flag that enables SDK to support gracefulTermination - FeatureSDKGracefulTermination Feature = "SDKGracefulTermination" + // FeaturePlayerTracking is a feature flag to enable/disable player tracking features. + FeaturePlayerTracking Feature = "PlayerTracking" // FeatureResetMetricsOnDelete is a feature flag that tells the metrics service to unregister and register // relevant metric views to reset their state immediately when an Agones resource is deleted. FeatureResetMetricsOnDelete Feature = "ResetMetricsOnDelete" + + // FeatureSDKGracefulTermination is a feature flag that enables SDK to support gracefulTermination + FeatureSDKGracefulTermination Feature = "SDKGracefulTermination" + + //////////////// + // Example feature + + // FeatureExample is an example feature gate flag, used for testing and demonstrative purposes + FeatureExample Feature = "Example" ) var ( // featureDefaults is a map of all Feature Gates that are // operational in Agones, and what their default configuration is. - // alpha features are disabled. + // alpha features are disabled by default; beta features are enabled. + // + // To add a new alpha feature: + // * add a const above + // * add it to `featureDefaults` + // * add it to install/helm/agones/defaultfeaturegates.yaml + // * add it to `ALPHA_FEATURE_GATES` in build/Makefile + // * add the inverse to the e2e-runner config in cloudbuild.yaml + // * add it to site/content/en/docs/Guides/feature-stages.md + // + // To promote a feature from alpha->beta: + // * move from `false` to `true` in `featureDefaults` + // * move from `false` to `true` in install/helm/agones/defaultfeaturegates.yaml + // * remove from `ALPHA_FEATURE_GATES` in build/Makefile + // * invert in the e2e-runner config in cloudbuild.yaml + // * change the value in site/content/en/docs/Guides/feature-stages.md + // + // To promote a feature from beta->GA: + // * remove all places consuming the feature gate and fold logic to true + // * consider cleanup - often folding a gate to true allows refactoring + // * invert the "new alpha feature" steps above + // + // In each of these, keep the feature sorted by descending maturity then alphabetical featureDefaults = map[Feature]bool{ - FeatureExample: true, - FeaturePlayerTracking: false, - FeatureStateAllocationFilter: true, + // Beta features + FeatureCustomFasSyncInterval: true, + FeatureStateAllocationFilter: true, + + // Alpha features + FeatureLifecycleContract: false, FeaturePlayerAllocationFilter: false, - FeatureCustomFasSyncInterval: true, - FeatureSDKGracefulTermination: false, + FeaturePlayerTracking: false, FeatureResetMetricsOnDelete: false, + FeatureSDKGracefulTermination: false, + + // Example feature + FeatureExample: false, } // featureGates is the storage of what features are enabled diff --git a/site/content/en/docs/Guides/feature-stages.md b/site/content/en/docs/Guides/feature-stages.md index 46c12264a9..8952a9abd1 100644 --- a/site/content/en/docs/Guides/feature-stages.md +++ b/site/content/en/docs/Guides/feature-stages.md @@ -22,17 +22,18 @@ A feature within Agones can be in `Alpha`, `Beta` or `Stable` stage. `Alpha` and `Beta` features can be enabled or disabled through the `agones.featureGates` configuration option that can be found in the [Helm configuration]({{< ref "/docs/Installation/Install Agones/helm.md#configuration" >}}) documentation. -The current set of `alpha` and `beta` feature gates are: +The current set of `alpha` and `beta` feature gates: | Feature Name | Gate | Default | Stage | Since | |-----------------------------------------------------------------------------------------------------------------------|--------------------------|----------|---------|--------| -| Example Gate (not in use) | `Example` | Disabled | None | 0.13.0 | -| [Player Tracking]({{< ref "/docs/Guides/player-tracking.md" >}}) | `PlayerTracking` | Disabled | `Alpha` | 1.6.0 | | [Custom resync period for FleetAutoscaler](https://github.com/googleforgames/agones/issues/1955) | `CustomFasSyncInterval` | Enabled | `Beta` | 1.25.0 | | [GameServer state filtering on GameServerAllocations](https://github.com/googleforgames/agones/issues/1239) | `StateAllocationFilter` | Enabled | `Beta` | 1.26.0 | +| [Lifecycle Contracts](https://github.com/googleforgames/agones/issues/2794) | `LifecycleContract` | Disabled | `Alpha` | 1.28.0 | | [GameServer player capacity filtering on GameServerAllocations](https://github.com/googleforgames/agones/issues/1239) | `PlayerAllocationFilter` | Disabled | `Alpha` | 1.14.0 | -| [Graceful Termination for GameServer SDK](https://github.com/googleforgames/agones/pull/2205) | `SDKGracefulTermination` | Disabled | `Alpha` | 1.18.0 | +| [Player Tracking]({{< ref "/docs/Guides/player-tracking.md" >}}) | `PlayerTracking` | Disabled | `Alpha` | 1.6.0 | | [Reset Metric Export on Fleet / Autoscaler deletion]({{% relref "./metrics.md#dropping-metric-labels" %}}) | `ResetMetricsOnDelete` | Disabled | `Alpha` | 1.26.0 | +| [Graceful Termination for GameServer SDK](https://github.com/googleforgames/agones/pull/2205) | `SDKGracefulTermination` | Disabled | `Alpha` | 1.18.0 | +| Example Gate (not in use) | `Example` | Disabled | None | 0.13.0 | {{< alert title="Note" color="info" >}} If you aren't sure if Feature Flags have been set correctly, have a look at the