Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make search case sensitive #1553

Merged
merged 10 commits into from
Jul 12, 2022
Merged

Conversation

mdisibio
Copy link
Contributor

@mdisibio mdisibio commented Jul 6, 2022

What this PR does:
This PR makes all search-related activity (including tag lookups) case-sensitive. An original intent of this was to reduce cardinality and resource usage of ingester-search, which is extra data stored next to the actual block. Therefore this change will increase usage some, but the amount depends on the traffic. Parquet backend search will be more efficient due to the removal of ToLower. v2 is unaffected (was already case-sensitive).

An increase in resources at the ingester is likely to cause more occurrence of #1258 so there is an attempt to fix it. We weren't able to conclusively determine the cause of such enormous buffers, but it seems related to tags that are very large. Changes:

  1. Only store the first 1024 bytes of tags recorded in flatbuffers. This loses some search-ability, but seems like a worthwhile trade off to keep data sizes down.
  2. There is a soft cap of ~900MB on the size of buffers generated in tempofb. After hitting that limit it will stop writing further tags, and any remaining (ignored) data won't be searchable. The rationale for this number is explained in the comments, but I want to point out that this is still an enormous buffer, and unlikely to work well, but at least it won't panic, and will make further investigation easier.

Which issue(s) this PR fixes:
Fixes #1547 #1258

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@mdisibio mdisibio changed the title 1547 search case sensitive Make search case sensitive Jul 8, 2022
@mdisibio mdisibio marked this pull request as ready for review July 11, 2022 17:25
@mdisibio mdisibio merged commit dd15c00 into grafana:main Jul 12, 2022
@mdisibio mdisibio deleted the 1547-search-case-sensitive branch April 25, 2023 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make all search case-sensitive
2 participants