Compactor: Partially drop traces that exceed max_bytes_per_trace
#1317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
joe-elliott
requested review from
annanay25,
mdisibio,
dgzlopes,
mapno,
kvrhdn and
zalegrala
as code owners
joe-elliott
changed the title
max_bytes_per_tracemax_bytes_per_trace
Signed-off-by: Joe Elliott <number101010@gmail.com>
mdisibio
reviewed
Mar 2, 2022
mdisibio
reviewed
Mar 2, 2022
Signed-off-by: Joe Elliott <number101010@gmail.com>
joe-elliott
changed the title
max_bytes_per_tracemax_bytes_per_trace
mdisibio
approved these changes
Mar 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Updates the compactor to not write an object back into a block that exceeds the per tenant
max_bytes_per_trace. Moves thetempo_discarded_spans_totalmetric into a shared location for both the distributor and the compactor to use. The distributor now adds a spans discarded reason oftrace_too_large_to_compact.A lot of subtle choices were made when creating this PR and ultimately I erred on the side of submitting a minimal change, but please review with the following in mind:
The compactor will go through the work of creating the trace even when it will just choose to throw it away immediately afterward. I chose to do this because we are only seeing issues when traces grow into the 100s of MBs. This is occurring on tenants with significantly lower
max_bytes_per_trace. This change will catch and discard huge traces before they become a problem.After throwing a combined trace away the compactor will attempt to count the number of spans it threw away, but this number is almost certainly overcounted. The compactor simply counts all the spans in the discarded objects even though it's likely that most of those spans are in the returned object. The only way I think to make this value correct would be fairly invasive changes into the ObjectCombiner. I felt like that complexity would not be worth it, but feel free to advise otherwise.
Which issue(s) this PR fixes:
Fixes #976
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]