HBase change event producer

The HBase change event producer is a sub-type of the Aesop AbstractEventProducer class that listens to HBase WAL edits using the NGData HBase SEP library. The implementation is self-contained in the Aesop Maven module : hbase-producer

This change event producer sets up a NGData SepConsumer that listens to WAL edits and implements the EventListener interface to receive call-backs when a list of SepEvent instances are available for consumption. It uses a number of worker threads to process SEP events. This implementation uses a default value of 1 worker thread for these reasons:

• The work done by the worker is to transform each SEP event to the Avro defined change data instance which is entirely a CPU bound operation and append it to the Relay's Event buffer, a memory update operation. A single thread will therefore be able to process millions of such events quite easily.
• The events appended to the Relay's Event buffer need to have SCNs in a monotonically increasing order. Using multiple worker threads can affect ordering and result in exceptions.

The EventListener implementation uses a configured SepEventMapper implementation to transform WAL edit data contained in SEPEvent instances to the Avro schema defined change data. See PersonSepEventMapper in the sample-hbase-relay module for an implementation of SEP Event mapping to the sample Person Avro change data type.

This producer uses the HBase KeyValue(retrieved from the SEPEvent) instance's timestamp as the SCN for the change event. This may not be a good system change number that uniquely identifies the transaction. An alternative like this recommendation : Application Generated SCN is under consideration.