Game Server health checking

[markmandel]

Design

Health Configuration

apiVersion: "stable.agon.io/v1alpha1"
kind: GameServer
metadata:
  name: "simple-udp"
spec:
  portPolicy: "static"
  containerPort: 7654
  hostPort: 7777
  # new health section
  health:
    # defaults to false, but can be set to true
    disabled: false
    # If the `Health()` function doesn't get called at least once every timeout seconds, then
    # the game server is not healthy. Defaults to "5"
    periodSeconds: 3
    # Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1.
    failureThreshold: 3
    # Number of seconds after the container has started before health check is initiated. Defaults to 5 seconds
    initialDelaySeconds: 5
  template:
    spec:
      containers:
      - name: simple-udp
        image: gcr.io/agon-images/udp-server:0.1

SDK API

SDK.Health()

The Health() function on the SDK object ill need to be called regularly below the timeout threshold time to be considered healthy.

Failure

1. If any of the backing Pod containers fails for any reason before the GameServer moves to Ready then, it should restart as per the restartPolicy (which defaults to "Always")
2. If the GameServer Pod fails (for any reason) after the Ready state, then it doesn't restart, but moves the GameServer to an Unhealthy state - and then it's up to the managing code to determine what to do at that point.
3. If the SDK sidecar fails, then it should always be restarted.

Outside scope

• The GameServer Pod fails to be scheduled for any reason, such as lack of resources (cpu, memory) or a being allocated an unavailable port - this will be managed in a separate ticket

Implementation

gRPC

Health is a unidirectional stream from the gameserver client -> the sidecar. The sidecar will update the State to UnHealthy if it doesn't receive a healthcheck event within the allotted time.

Kubernetes

(Theory - need investigation)

1. The sidecar will proxy the liveness check for the GameServer container through a gshealthz url endpoint. It will track Health() messages and if they drop below the set threshhold, return a 500.
2. Once the GameServer is Ready, then this will always return 200 - which (in theory) should mean that Kubernetes will never restart the GameServer container.