Reduce memory overhead of decision log event buffering

[tsandall]

Currently, decision log events are serialized to JSON before being buffered. This introduces significant overhead and reduces the number of events that can be buffered. We should investigate an alternative in-memory representation for decision log events that increases the number of events that can be buffered.

Example scenario:

• 1,000 decisions per second
• 5 minute upload interval

Sample decision log event:

{
    "labels": {
      "app": "my-example-app",
      "id": "1780d507-aea2-45cc-ae50-fa153c8e4a5a"
    },
    "decision_id": "4ca636c1-55e4-417a-b1d8-4aceb67960d1",
    "revision": "W3sibCI6InN5cy9jYXRhbG9nIiwicyI6NDA3MX1d",
    "path": "http/example/authz/allow",
    "input": {
      "method": "GET",
      "path": "/salary/bob"
    },
    "result": "true",
    "requested_by": "[::1]:59943",
    "timestamp": "2018-01-01T00:00:00.000000Z"
}

Some notes about the event structure:

• labels do not change
• revision and path are likely the same across many events
• decision_id is globally unique
• timestamp is unique but may be relative to previous decision
• input and result may be same across events but structure is not easily known to OPA

One alternative would be to compress the events before buffering them. If there's a streaming compressor that supports memory limits that would be easy enough to integrate.