Support FindService && FindOperations

[swartz-k]

Is your feature request related to a problem? Please describe.
While using 16686 for query traces, Serive && Operations are not querable.

Describe the solution you'd like
Usr 16686 Jaeger with tempo grpc plugin as Jaeger total experience.

Describe alternatives you've considered
Support Memory Index Label & Boltdb shipper for multi info query.
I have done this like below.

Additional context

1. In ingester instance add github.com/cortexproject/cortex/pkg/ingester/index to cache Service and Operation in method modules/ingester/instance.go:284 getOrCreateTrace
2. In tempo.proto add struct such as GetServicesRequest GetOperationsRequest

[joe-elliott]

Thanks for the feature request. I'm unsure how I feel about this. It's kind of a clever shortcut to allow for querying this information from Tempo. For clarity, we intend to eventually add search to Tempo, but we intend to do it by augmenting the blocks in a way that they become searchable.

I feel like there's limited value in returning services and operations if broader search capabilities are not present, but I'm certainly open to be persuaded otherwise. Also, I would be very hesitant to persist this information to the backend since it may complicate future attempts to add search, but perhaps an in-memory/ephemeral option would be feasible.

[swartz-k]

Yes. I just do it as a demo to research wheter it possible to support query traces in a broader way.
I had tested it by add an app which has about 500qps and total 999999 requests, per request create about 5 spans, those will cost memory about 10Gi. Also in memory way, search traces with multi ingester pods may have some problems.
As you said, Add search to Tempo is a better way to users and also make Tempo suitable for more situation.
Should we do some work about design docs for Tempo Searching like Loki does?

[joe-elliott]

Internally we have only the seed of a "design doc". Really just a scratchpad with ideas/challenges/initial thoughts on adding search to Tempo. We will discuss at our next meeting about whether we wan to make this one public. It's less of a design doc and more of a scratchpad.

[swartz-k]

Thanks a lot. I will do some works about that too. Hope to make Tempo Better together.

[joe-elliott]

So if the question we are trying to answer is: What services and operations are pushing to Tempo I think I would prefer using metrics.

Once this is merged:
open-telemetry/opentelemetry-collector-contrib#2012

I think it would be a better option than shoehorning support for these queries into Tempo. We can add support for this process in the Grafana cloud agent if you are using that.

[strowi]

Hi,

trying to migrate from jaeger -> tempo i stumbled upon this. Trying and testing again the demo - looking at 16686 without any traces showing up and traces in Grafana not being searchable like jaeger.

It might be a good thing to put a note into the examples/docs for people coming from jaeger not to get frustrated by this.

[yeya24]

Hi, @joe-elliott, now the span metrics processor is added to the Grafana agent. How should we proceed with supporting searching in Tempo?

Another question is that now tempo-query is deprecated. But only Jaeger UI supports searching, so we still need to use the tempo-query component?

[joe-elliott]

How should we proceed with supporting searching in Tempo?

We have written up a number of design docs on this. I believe the first pass will use some kind of limited index along with heavy parallelization. We intend to get more heavily invested in this after Grafanacon.

But only Jaeger UI supports searching, so we still need to use the tempo-query component?

When we add search to Tempo we will also add UI support in Grafana.

[mapno]

How should we proceed with supporting searching in Tempo?

We have written up a number of design docs on this. I believe the first pass will use some kind of limited index along with heavy parallelization. We intend to get more heavily invested in this after Grafanacon.

Hi @yeya24, in the meantime, we've added automatic logging to the agent (grafana/agent#551) as an alternative to search in Tempo. With automatic logging, you log a line per span/process/root span which then can be fully searched in Loki. This feature is in its early stages, so any feedback is very welcome.