MySQL change event producer

The MySql Producer relies on Open Replicator. It uses the Open Replicator to read MySql bin logs and convert them into change events. The Open Replicator project has been forked to https://github.com/Flipkart/open-replicator for the following reasons:

• Enable support for MySql version 5.6 Bin Log while the original version supports only version 5.5 event formats.
• Support for parsing Checksum introduced in version 5.6.

The implementation details of this producer is as follows:

• MySql Producer registers a Replication Listener with Open Replicator. A Call back method is invoked on the Replication Listener when events are processed by the Open Replicator.
• Various Event Processors are registered in MySql producer to handle specific event types viz. Insert, Update, Delete, InsertV2, UpdateV2, DeleteV2 for DML and various type of DDL events.
• For DML events
  • Transaction Manager handles these events. The Transaction Manager groups events under a Transaction and manages life cycle of the Transaction until it is committed. Once the Transaction is committed, the Transaction Manager adds the events into Event Buffer.
• For DDL (Currently only Alter is supported) events
  • The AVRO Schema Generator is called to regenerate the Schema for the table and update the schema registry in Relay.
  • Propagation of the DDL events to Clients is not yet supported.

The MySql producer generates SCN by a combination of the bin log sql file number and the location within the bin log. In the resulting SCN, the binlog file number occupies the high 32 bits and the location occupies the low 32 bits.

• Example if the bin log file is 1 and the location within the bin log file is 4 (from start of bin log file) then SCN would be (1<<32)|4. In practice the same can be derived using (1<<32)+4.

Generational relay SCN generator

The bin log offset based SCN generator for the relay does not support MySQL mastership transfers. This is because the bin log file number and offset can be quite different when a transfer happens. The resulting SCN number can even be earlier to the last seen SCN on the relay if the bin log file number is lower than current MySQL master. Different SCN generation approaches are discussed in the Databus Wiki. Aesop has a Generational SCN Generator that supports mastership transfers by checkpointing bin log host and change generation information.

The generated relay SCN has the format : high 16 bits derived from generation, next 16 bits derived from location reference of local SCN, low 32 bits is the offset inside the file.

Checkpoints are currently on file system and not shared. A sample configuration to use this SCN generator in the MySQL event producer is as follows:

<property name="scnGenerator">
    <bean id="generationalSCNGeneratorFactory" class="com.flipkart.aesop.runtime.producer.impl.GenerationalSCNGenerator">
        <property name="relayLogicalSourceName" value="com.flipkart.aesop.events.ortest.Person"/>
        <property name="checkpointPersistenceProvider">
            <bean id="checkpointPersistenceProviderFactory" class="com.flipkart.aesop.runtime.producer.impl.FileSystemCPProviderFactory">
                <property name="checkpointDir" value="../../../../sandbox/gen_SCN_CP_directory"/>
            </bean>
        </property>		    			    	
    </bean>
</property>