Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Increase in errors due to no active logging span #8510

Closed
clokep opened this issue Oct 9, 2020 · 2 comments · Fixed by #8567
Closed

Increase in errors due to no active logging span #8510

clokep opened this issue Oct 9, 2020 · 2 comments · Fixed by #8567
Assignees
@erikjohnston
Labels
A-Logging Synapse's logs (structured or otherwise). Not metrics. z-bug (Deprecated Label)

Comments

@clokep
Copy link
Member

clokep commented Oct 9, 2020

Starting on Oct 1 (which corresponds with a re-deploy of matrix.org) we've been seeing a lot more of the following error:

There was no active span when trying to inject the span into a byte dict. Did you forget to start one or did a context slip?

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/15795/
@clokep clokep added z-bug (Deprecated Label) A-Logging Synapse's logs (structured or otherwise). Not metrics. labels Oct 9, 2020
@clokep clokep self-assigned this Oct 9, 2020
@clokep clokep changed the title Increase in logging span errors Increase in errors due to no active logging span Oct 9, 2020
@clokep
Copy link
Member Author

clokep commented Oct 9, 2020

I believe that this is a regression in the range of bbb7ca1f157f43250a2373b194c7dee3dfbfca80 to 7b6f857aa996ec4dde368e11115bf1684770d78f.

@clokep
Copy link
Member Author

clokep commented Oct 9, 2020

Based on the message, it seems this is from inject_active_span_byte_dict, which is used in only two spots:

synapse/synapse/http/matrixfederationclient.py

Line 408 in 1c26243

inject_active_span_byte_dict(headers_dict, request.destination)

synapse/synapse/replication/http/_base.py

Line 205 in 866c84d

inject_active_span_byte_dict(headers, None, check_destination=False)

I've seen this on workers not handling federation, which makes me guess this is from replication?

@richvdh richvdh mentioned this issue Oct 12, 2020
Open
@clokep clokep removed their assignment Oct 14, 2020
@erikjohnston erikjohnston self-assigned this Oct 16, 2020
erikjohnston added a commit that referenced this issue Oct 16, 2020
@erikjohnston
Start an opentracing span for background processes.
f9c54e6 
This should reduce the number of `There was no active span` errors we
see.

Fixes #8510.
@erikjohnston erikjohnston mentioned this issue Oct 16, 2020
Merged
erikjohnston added a commit that referenced this issue Oct 19, 2020
@erikjohnston
Start an opentracing span for background processes. (#8567)
1fcdbeb 
This should reduce the number of `There was no active span` errors we
see.

Fixes #8510.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@erikjohnston erikjohnston

Labels
A-Logging Synapse's logs (structured or otherwise). Not metrics. z-bug (Deprecated Label)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Start an opentracing span for background processes. matrix-org/synapse
2 participants
@clokep @erikjohnston