tempo-mixin: add panel for Envoy Proxy sidecar, update tanka example … #1137
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and jsonnet dependencies Changes: ### Tempo / Writes dashboard Add a panel with Envoy metrics. This panel is to be used when the gateway is run with an Envoy Proxy sidecar (for HTTP2 load balancing). The metrics from Envoy expose the gRPC status codes, which are essential pieces of information when debugging ingress issues. The gateway only reports HTTP statuses, which is deceptive since a 200 OK HTTP packet might contain a gRPC message with an error. Example: if a request is denied by the ingester because it hits the limit, the distributor will return a HTTP 200 OK message containg a gRPC message with status 9 FAILED PRECONDITION. I've also added a text panel listing the gRPC status codes since these aren't common knowledge. It links to this doc: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md ### tanka example - update prometheus config to always add `cluster` label, the tempo-mixin dashboards depend on this label - enable search in Grafana and Tempo - use a relative link to tempo-mixin, this makes it easier to quickly test changes I've also run `jb update` in tempo-mixin and tk examples, hence the large amount of vendor file being changed.
kvrhdn
requested review from
annanay25,
dgzlopes,
joe-elliott,
mapno and
mdisibio
as code owners
|
Btw, the end result looks a little bit like this:
Interesting things to note here:
|
annanay25
approved these changes
Nov 26, 2021
|
|
||
| Code | Number | Description | ||
| ---|---|--- | ||
| OK | 0 | Not an error; returned on success. |
There was a problem hiding this comment.
Nice, very useful addition.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Changes:
Tempo / Writes dashboard
Add a panel with Envoy metrics. This panel is to be used when the gateway is run with an Envoy Proxy sidecar (for HTTP2 load balancing).
The metrics from Envoy expose the gRPC status codes, which are essential pieces of information when debugging ingress issues. The gateway only reports HTTP statuses, which is deceptive since a 200 OK HTTP packet might contain a gRPC message with an error.
Example: if a request is denied by the ingester because it hits the limit, the distributor will return a HTTP 200 OK message containing a gRPC message with status 9 FAILED PRECONDITION.
I've also added a text panel listing the gRPC status codes since these aren't common knowledge.
It links to this doc: https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
tanka example
clusterlabel, the tempo-mixin dashboards depend on this labelI've also run
jb updatein tempo-mixin and tk examples, hence the large amount of vendor file being changed.Which issue(s) this PR fixes:
Fixes #Checklist
Tests updatedDocumentation addedCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]