Write Concept for Cluster Automation

[srueg]

Context

Currently we do multiple automated steps when a new cluster is created: a new git repo is created for the catalog, a token is stored in Vault, etc. This will be further extended in the future, for example to save arbitrary secrets in Vault (projectsyn/lieutenant-operator#79), create some CRs to automatically provision cluster via hive or Crossplane, set up authentication provider (e.g. Keycloak), Configure OpsGenie, etc.

Until now all of these features are implemented directly in the lieutenant-operator. Some of these automations are more generic than others and some will most likely only every be used by VSHN.
To make this more extensible and provide a scalable approach to add more automations as the need arises, we need a concept (SDD?) to define where and how such functionality should be implemented.

It should also be part of the concept how to leverage existing operators whenever possible. This includes for example the hive operator or the keycloak-operator.

Some Ideas

• Create a new operator for each feature (i.e. Vault operator, Keycloak operator, etc.)
• Create a new operator for all the VSHN specific features (Keycloak, OpsGenie)
• Implement the features in lieutenant-operator but in a generic way, like we currently do with the GitRepos (GitLab, GitHub, Gitea)

Alternatives

Continue to implement all functionality within the lieutenant-operator. This has the downside that it will become a complex operator which needs to support many different use-cases and might be too VSHN specific.

[tobru]

It should also be part of the concept how to leverage existing operators whenever possible. This includes for example the hive operator or the keycloak-operator.

One way to achieve this could be to leverage the upcoming Carpenter Operator, described in SDD 0023 - Managed Services Controller. When a Lieutenant cluster object is created with some conditions, Carpenter could create a new object out of it based on well defined ConditionalObjects object.