ci: remove apm-beats-update

[v1v]

https://github.com/elastic/apm-server/blob/main/.ci/jobs/update-beats-mbp.yml\#L5 could be the way to do it

What does this PR do?

Remove pipeline to validate whether lib changes could break the APM Server

Why is it important?

Jenkins is deprecated and the CI for elastic/beats will be managed by the ingest team. https://github.com/elastic/apm-server/blob/main/.ci/jobs/update-beats-mbp.yml#L5 could be the alternative even though changes are validated after merging them

@axw , what are your thoughts? Should we rely on update-beats-mbp.yml only for now? (I know we are migrating to GH actions, that job will eventually be migrated). Or is it better to migrate this job too?

@cmacknz , if this job is needed, then it might be required to be migrated to Buildkite? Or maybe GitHub actions could be used instead for this particular use case?

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-05-09T07:25:06.135+0000

• Duration: 13 min 5 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

In the past apm-server has been broken by libbeat library changes, and it wasn't obvious to the libbeat maintainers because apm-server is in a separate repo. It's been quite a while since we saw breaking API changes, excluding Agent V2-related API changes which were communicated ahead of time. I'm not sure how often this CI step has prevented breakage before it reached us though - that's kind of the point. @cmacknz do you know?

[cmacknz]

I'm not sure how often this CI step has prevented breakage before it reached us though - that's kind of the point. @cmacknz do you know?

I don't remember this catching anything recently, but besides agent V2 we haven't made many major changes so the risk of this happening was low.

We could live without this, but it definitely adds the risk that we miss something this automation would have helped us catch.

I think it is valuable to port this workflow to Buildkite as an example of how to trigger automatic updates like this. There will be other repositories for new Elastic Agent inputs outside of Beats in the future and a reference job for how to automatically track the upstream dependency would be helpful.

[v1v]

I think it is valuable to port this workflow to Buildkite as an example of how to trigger automatic updates like this. There will be other repositories for new Elastic Agent inputs outside of Beats in the future and a reference job for how to automatically track the upstream dependency would be helpful.

Perfect! I'll keep this open until the migration has done in Buildkite. Who is leading the migration to Buildkite so I can help with if needed?

[cmacknz]

Who is leading the migration to Buildkite so I can help with if needed?

@mrodm has been tracking the pieces that we need to migrate.

[joshdover]

Given this has been broken for 9 months, any opposition to just merging this now and re-implementing it later on Buildkite?

[joshdover]

Merging this now. If anyone disagrees, we can revert it.