Export logs only if the level/logger/target is enabled.

[lalitb]

This PR is for discussion in the community meeting. As of now, the pipeline to emit the logs using the tokio-tracing library with otel-sdk is as follows (using user_events exporter as an example)

error!("login failed") -> Layer::on_event() -> create LogRecord -> logger.emit() -> create LogData -> processor.emit() -> user_events_exporter::export()
-> [Linux Kernel ring buffer] <- perf listener

So the event is written to the Linux kernel ring buffer, from where the perf listener will read the events. The event is written to the kernel buffer even when there is no listener reading the ring buffer. Further, the listener can be configured only to collect (say) critical errors/warnings, and ignore debug/trace events. This means that the application will be spending ~1000 ns to emit the events which are eventually dropped if there is no listener configured to read that event. This overhead can be avoided if it is possible for the tokio-tracing log appender to emit the events only when a listener is attached or the listener is configured to read that type of event.
The proposal is to add a new method exporter::event_enabled() in Exporter. This method will return true if the exporter is configured to read that type of event. The type here could be a combination of level + logger-name. This method can then be called from tokio-tracing log appender as below:

error("login failed") -> Layer::event_enabled() -> logger::event_enabled(level, target) -> processor::event_enabled(level, target, logger_name) -> exporter::event_enabled(level, target, logger_name)

-> False, drop the event ( this has ~40 ns of overhead)
-> True, emit the event ( this has ~1000 ns of overhead)

So, event_enabled method would also be added in the logger and processor to eventually enable calling this method from the exporter. As this method is not part of otel specs, we can add it behind the feature flag as below:

#[cfg(feature = "logs_level_enabled")]
fn event_enabled(&self, level : crate::logs::Severity) -> bool {
   . . .
 }

So to summarise - for scenarios where the exporter is only configured to receive specific types of events, we can get rid of all the unnecessary processing for the other events. These performance gains are significant for the system applications written in C/C++ and Rust.

[lalitb]

Two alternatives apart from the existing proposal were discussed in today's meeting :

• Configure the event_enabled callback as part of opentelemetry_sdk::Logger, similar to how IdGenerator / Samplers are configured for Tracer. This callback would result in events getting dropped, or generate non-recordable events (similar to non-recordable spans).
• @shaun-cox - You had some suggestions to create the log equivalent for TraceContextExt. If you could elaborate on it further here.

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: -0.2% ⚠️

Comparison is base (8975aa3) 49.7% compared to head (4316f89) 49.6%.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1147     +/-   ##
=======================================
- Coverage   49.7%   49.6%   -0.2%     
=======================================
  Files        165     165             
  Lines      20598   20649     +51     
=======================================
  Hits       10253   10253             
- Misses     10345   10396     +51     

Files Changed	Coverage Δ
opentelemetry-api/src/global/logs.rs	0.0% <0.0%> (ø)
opentelemetry-api/src/logs/logger.rs	0.0% <ø> (ø)
opentelemetry-api/src/logs/noop.rs	0.0% <0.0%> (ø)
opentelemetry-appender-log/src/lib.rs	0.0% <0.0%> (ø)
opentelemetry-appender-tracing/src/layer.rs	0.0% <0.0%> (ø)
opentelemetry-sdk/src/export/logs/mod.rs	0.0% <0.0%> (ø)
opentelemetry-sdk/src/logs/log_emitter.rs	0.0% <0.0%> (ø)
opentelemetry-sdk/src/logs/log_processor.rs	0.0% <0.0%> (ø)
...pentelemetry-user-events-logs/src/logs/exporter.rs	0.0% <0.0%> (ø)
...ser-events-logs/src/logs/reentrant_logprocessor.rs	0.0% <0.0%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TommyCpp]

Looks good to me. We can experiment this under the feature flag.

[lalitb]

This PR is ready now. As discussed in community meeting, the logs can be enabled/disabled in exporter (and in future in LoggerProvider through closure if required) using level, target and logger-name. All the code is within feature flag. As of now, flag is default enabled for user_events exporter and tracing appender, so that the changes can be tested in CI. Let me know if there is any concern.

EDIT - Have also added the event enabled check for the logger appender

[cijothomas]

LGTM.
We need to evolve the parameters for the is_enabled further based on needs. The current "target", "name" seems to be good enough to start with.

[jtescher]

looks reasonable

[lalitb]

@jtescher - Can we include this in 0.20.0 release?