Using Haraka Outbound with a Custom Queue Driver

[bahdcoder]

Using Haraka Outbound with a Custom Queue Driver

Hello team, I am currently exploring using Haraka with a custom queue driver and not the default disk queue driver.

For example, using Redis / Bull MQ as a queue driver as we can easily scale horizontally across servers, kill workers while still making sure we don't lose anything on disk and other Haraka worker servers can pick up a job from the queue if one worker fails.

A custom queue driver could also help us set up better monitoring of the entire queue system when Haraka is running across multiple servers.

I dug into the code to learn more about Outbound, and it seems to be very tightly coupled with the file based queue system.

What I tried

I tried exploring using a plugin to save an incoming email as a Redis / Bull MQ job, and this worked (thank you plugins !).

Next I tried pulling jobs from my Redis / Bull MQ queue (in a plugin) and sending those email jobs using outbound.send_email(from, to, contents), but it seems this method just queues them to disk when it calls process_delivery.

I also explored using HMailItem directly but it also seems tightly coupled to _qfile.

I would please love to get some help / information writing a plugin that overrides the default disk queue behaviour.

Please let me know if I am completely off track as I just learned about the project.

Thank you very much