Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SearchTagValuesV2 use protobuf internally instead of json to reduce latency #3731

Merged
merged 4 commits into from
May 31, 2024

Conversation

mdisibio
Copy link
Contributor

@mdisibio mdisibio commented May 29, 2024

What this PR does:
The frontend communicates with the queriers using json, and this is becomes a bottleneck for large autocomplete lookups. This PR switches /api/v2/search/tag/.../values to use http/protobuf instead. Testing locally and in a larger cell on a large lookup, the savings are around 30% user-facing latency to Grafana. Timings: 15s -> 10s, 5s -> 3s. We like graphs, so here is one showing the second set:

image

The encoding is implemented optionally where the querier will reply with proto only if the request header is Accept: application/protobuf. Else it defaults to json. This helps with compatibility during rollouts of old/new pods, and also curl-ability.

A couple future steps:

(1) It might be worth swapping over the remaining apis with large payloads like search/tags and metrics. But didn't want to expand scope.

(2) Even better would be preserve the protobuf all the way to Grafana like we do for trace lookup, but there is work needed to connect the response/request through the new async pipeline in the frontend, so the final response can be built with the appropriate encoding.

Which issue(s) this PR fixes:
Fixes n/a

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

modules/querier/http.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants