Add telemetry to bundler

[garryod]

Add telemetry (logs + metrics + traces) to bundler using tracing with export in OpenTelemetry format via opentelemetry-otlp.

Collecting & visualising results with:

• Jaeger for traces
• Prometheus for metrics

[tpoliaw]

Again, I'll leave the yaml config to someone who knows what it's doing but the code change looks ok.

OpenTelemetry looks widely used enough to be a good choice for the exporting side, but I'm not familiar enough with the downstream platforms to really comment on them.

What does Jaeger offer over something like Grafana that's used elsewhere at Diamond? Are we going to end up with an individual monitoring system for each service or is the aim that they're all going to be combined into a central system at some point?

[tpoliaw]

Is there any way of parameterising these values rather than repeating them in several places/files?

[garryod]

otelcol & jaeger both support env var substitution in their configs, so I could put them in the docker-compose. Unfortunately prometheus doesn't

[garryod]

What does Jaeger offer over something like Grafana that's used elsewhere at Diamond? Are we going to end up with an individual monitoring system for each service or is the aim that they're all going to be combined into a central system at some point?

Jaeger faciliates distributed tracing, as opposed to metrics visualisation. That is to say it provides a view of 'events' each of which can be drilled down into giving a waterfall of spans which were involved in this event and logs which were captured in those spans, even across service boundaries.

To really make distributed tracing powerful there needs to be a single instance which can correlate spans from desperate services, so I would envisage having one instance (cluster for HA) which all services would push their tracing data to.

[tpoliaw]

I thought grafana offered tracing visualisation as well but looking at it, it requires an external data source (such as jaeger) to work. I have no preferences one way or the other, I'm just not sure of what each part is doing. Where does prometheus fit in?

[garryod]

Jaeger and Prometheus both serve effectively the same role, they act as a storage backend and query engine for traces and metrics respectively. Grafana can then provide visualisations of this data

[DiamondJoseph]

If Peter is happy with the Rust, the devcontainer changes look sensible; presumably we have an equivalent standard logging deployment which handles the jaeger?

[garryod]

If Peter is happy with the Rust, the devcontainer changes look sensible; presumably we have an equivalent standard logging deployment which handles the jaeger?

Nothing in the deployment yet, and no recommended good practice for how to do this, kind of using this project (and XChem) as the test bed for OTEL & Jaeger with the view to spin up some Diamond wide infra to handle traces & metrics

[DiamondJoseph]

spin up some Diamond wide infra to handle traces & metrics

Either way, the monitoring deployment lives outside of the deployment it's monitoring, so that's a future problem