Add liveness check to pkg/gameservers/controller.

[enocom]

Original Context.

I don't know if this is possible, but what would determine if this controller is healthy is that the goroutines generated in this code are still running and haven't exited in any way.

[markmandel]

@enocom mind if I edit this ticket to include the full details? Just in case anyone else happens to want to pick this up? (Unless you want to expand this out, since I know you had ideas)

[enocom]

@markmandel Please do. I think this issue might be good for first-time contributors, too.

[markmandel]

Some interesting notes here:

• The health check handler is passed in here
• The actual goroutines are run in the workerqueue. so there is likely opportunity to standardise this across anything that uses a workerqueue.

Final thought - for a readiness check, readiness can be turned on as soon as the WebHook that powers the MutationWebhookConfiguration is ready

[markmandel]

@enocom did you have some thoughts on how you were thinking of tracking if the goroutines were still active? (or maybe a PR 😁 )

[enocom]

I imagine adding a Registrar, which in functionality would be similar to a sync.WaitGroup. We pass the Registrar to anything that starts a goroutine and upon starting the goroutine, we maybe call registrar.Inc() with a corresponding defer registrar.Dec() for when the goroutine exits. Then the liveness check could simply query the Registrar to see if all registered goroutines are still running.

Thoughts?

[markmandel]

Works for me. Although the Inc(), Dec() and Count() would need to be threadsafe. I'm assuming a sync.RWMutex ?

Thought - would this be part of a Registrar, or maybe just part of the WorkerQueue?

[enocom]

There's some question of where this responsibility belongs.

Insofar as implementation, sync.RWMutex works and maybe even a sync.Mutex too. Just depends where the hot paths may be.