Add kustomise template for hints auto discover

[MichaelKatsoulis]

What does this PR do?

This PR adds a Kustomize template to enable hints based auto discovery by default in standalone elastic agent.
It mainly adds a initcontainer block in elastic agent daemonset.
For all the rest resources needed for the deployment of elastic agent it relies on the resources from the default kustomize templates.

Why is it important?

To enable users of the Observability Kubernetes quickstart to make use of hints auto discovery without the need of configuring and redeploying their manifests.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Follow instructions of https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-kustomize/default#remote-usage-of-kustomize-templates

Use https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover for the kustomization file link.

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/3760

[mergify]

This pull request does not have a backport label. Could you fix it @MichaelKatsoulis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[pkoutsovasilis]

Could you extend the integration k8s tests by adding a new one with hints enabled? it would be nice to check that the agent and all components are starting and reporting healthy with this feature enabled

[gizas]

Which tests are you referring to Panos?

[pkoutsovasilis]

I had in mind these tests https://github.com/elastic/elastic-agent/blob/main/testing/integration/kubernetes_agent_standalone_test.go#L63 🙂

[gizas]

Thank you !! I wanted to double check as we dont actually own them.

But I created an issue here: #5660. We can discuss internally for prioritisation

[pkoutsovasilis]

AFAICT the k8s ownership is shared, and the agent repo now has k8s integration tests. So ideally anything that has to do with different k8s deployments should be covered by a test but sure discuss #5660 and try to couple this integration test with this PR

[swiatekm]

I'm not sure I fully understand the intent of this PR, but shouldn't we also be modifying https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml?

[gizas]

@swiatekm no the main standalone manifest should not be affected

The users should run a command like the following and make an installation of agent+k8s integration+system integration+kube-state-metrics up and running:

kubectl kustomize https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover | sed -e "s/JUFQSV9LRVkl/UTWamNtcGFkdz09/g" -e "s/%ES_HOST%/https:\/\/e56ee37c53644df6940a8ae217ab5960.us-central1.gcp.cloud.es.io:443/g" -e "s/%CA_TRUSTED%/ca_trusted_fingerprint/g" -e "s/%ONBOARDING_ID%/12345/g" | kubectl apply -f

Please read some more info here

[gizas]

Tested successfully the PR with a current Elastic Cloud setup.

Prereq: Go to Stack Management > API Keys and create a new key

Important use the beats format key

Create the encoded value by running:

echo -n oB1lTZIBdpMiLQUgtFM4:Z1bs7_ufTuOj5O8pg0Sxmg | base64
b0IxbFRaSUJkcE1pTFFVZ3RGTTQ6WjFiczdfdWZUdU9qNU84cGcwU3htZw==

Then run the following:

kubectl kustomize https://github.com/MichaelKatsoulis/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/ksm-hints/elastic-agent-standalone\?ref\=kustomize_templates_hints_autodiscover | sed -e "s/JUFQSV9LRVkl/b0IxbFRaSUJkcE1pTFFVZ3RGTTQ6WjFiczdfdWZUdU9qNU84cGcwU3htZw==/g" -e "s/%ES_HOST%/https:\/\/<CloudURL>:443/g" -e "/{CA_TRUSTED}/c\ "  > k2.yaml

Replace the image referneces:
9.0 > 8.15
9.0.0 > 8.15.0

Then run kubectl apply -f k2.yaml

Metrics and logs collection is successfull. I will run some more tests with autodiscovery templates and annotations in pods.
But main fucntionality is ok

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[gizas]

Continuation of testing, the autodiscovery works as expected:

Once the pod gets annotated with:

annotations:
        co.elastic.hints/enabled: 'true'
        co.elastic.hints/package: "nginx"

The datastream changes and no duplication happens:

I can see two issues with current template:

• We need to comment out / remove the dataset: kubernetes.container_logs that is produced
  • Discuss if we should keep audit.logs
• We need to uncomment following lines

providers.kubernetes:
      hints.enabled: true
      hints.default_container_logs: true

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube