Implement the BatchingProcessor

[MrAlias]

Part of #5063

Adds a feature complete implementation of the BatchingProcessor. There remains performance improvements to the addressed as follow-up tasks (tracked below)

Design

At a high-level, the BatchingProcessor is designed to provide the highest throughput of log Records possible while being compatible with OpenTelemetry. The entry point of log Records is the OnEmit method. This method is designed to receive Records as fast as possible while still honoring shutdown commands. All Records received are enqueued to a queue.

In order to block OnEmit as little as possible, a separate "poll" goroutine is spawned at the creation of a BatchingProcessor. This goroutine is responsible for batching the queue at regular polled intervals, or when it is directly signaled to.

To keep the polling goroutine from backing up, all batches it makes are exported with a bufferedExporter. This exporter allows the poll goroutine to enqueue an export payload that will be handled in a separate goroutine dedicated to the export. This asynchronous behavior allows the poll goroutine to maintain accurate interval polling.

  __BatchingProcessor__     __Poll Goroutine__     __Export Goroutine__
||                     || ||                  || ||                    ||
||          ********** || ||                  || ||     **********     ||
|| Records=>* OnEmit * || ||   | - ticker     || ||     * export *     ||
||          ********** || ||   | - trigger    || ||     **********     ||
||             ||      || ||   |              || ||         ||         ||
||             ||      || ||   |              || ||         ||         ||
||   __________\/___   || ||   |***********   || ||   ______/\_______  ||
||  (____queue______)>=||=||===|*  batch  *===||=||=>[_export_buffer_] ||
||                     || ||   |***********   || ||                    ||
||_____________________|| ||__________________|| ||____________________||

Release valve

When the BatchingProcessor receives more Records than it is able to batch and export it needs a way to not block the user application still sending Records. The "release valve" that prevents this is the queue records are stored to. This queue is backed by a ring buffer. It will overwrite the oldest records first when writes to OnEmit are made faster than the queue can be batched.

Shutdown and ForceFlush

The asynchronous design of the BatchingProcessor that allows for high throughput needs to also incorporate the synchronous nature of calls to Shutdown and ForceFlush.

To handle the flushing operation for both of these operations, the queue type can be explicitly flushed. The data returned can be sent to the bufferExporter's Export method to synchronously be exported (along with flushing any pending exports).

The shutdown state of the processor is managed at a high-level using an atomic.Bool that is checked by all exported methods before the preform operations.

Enabled

This updates the Enabled method to return false when the processor has been shutdown.

Follow up tasks

Tracked in #5063

• Pool export payloads
• Add option for users to configure the exporter buffer
• Add custom implementation of "container/ring"
• Add benchmarks
• Ensure a BatchingProcessor created without using NewBatchingProcessor does not panic

[codecov]

Codecov Report

Attention: Patch coverage is 98.75000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 84.5%. Comparing base (fe3de70) to head (47b77c6).

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5093     +/-   ##
=======================================
+ Coverage   84.3%   84.5%   +0.1%     
=======================================
  Files        258     258             
  Lines      17042   17110     +68     
=======================================
+ Hits       14382   14466     +84     
+ Misses      2361    2345     -16     
  Partials     299     299             

Files	Coverage Δ
sdk/log/exporter.go	98.1% <100.0%> (+0.1%)	⬆️
sdk/log/batch.go	99.4% <98.5%> (+16.0%)	⬆️

... and 1 file with indirect coverage changes

All my blocking comments are resolved. I will do my best to do another look on Thursday.

[MrAlias]

Plan is to wait until Monday to merge. @dashpole and @MadVikingGod are planning to take a look at this.

[MrAlias]

@dashpole does this look good to you?

[MrAlias]

@dashpole: planning to merge given this is blocking other work.