Queue Options

Description

When creating the Queue object you can supply options to customize the way the queue works. Each one of the Queue object options is discussed below. See the Queue Constructor document for more detail.

Option	Type	Default	Version
name	String	rjqJobList
databaseInitDelay	Integer	1000 ms (1 sec)
queryRunOptions	Boolean	{ readMode: 'majority' }
changeFeed	Boolean	true
concurrency	Integer	1
masterInterval	Integer	310000 ms (5 min 10 sec)
limitJobLogs	Integer	1000 log entries	v3.1.0
removeFinishedJobs	Bool / Int	15552000000 ms (180 days)

Options Example

This example is to show all the options being customized:

const Queue = require('rethinkdb-job-queue')

const cxnOptions = {
  host: 'proddb.domain.com',
  port: 10000,
  db: 'JobQueue'
}

const qOptions = {
  name: 'VideoProcess',
  masterInterval: 1000000,
  changeFeed: true,
  concurrency: 50,
  removeFinishedJobs: false
}

const q = new Queue(cxnOptions, qOptions)

Queue name Option

Default: rjqJobList

Valid: Any string value. Don't use spaces.

The name option defines the name of the Queue, the Queue object, and the Table created in RethinkDB. You can create multiple Queue objects with different names causing multiple Tables to be created in the RethinkDB database.

Important: It is highly recommended not to set the name option to the same name of an existing table in your database. The Queue.drop API call will drop the Table from the database. Always use a name related to the jobs you are processing.

name Example

This example creates two queues using the default options except for the name. One of the queues is called 'registration' and the other 'video'. This example will create two Tables in the RethinkDB database called 'registration' and 'video'.

const Queue = require('rethinkdb-job-queue')
const regoQ = new Queue(null, { name: 'registration' })
const videoProcessQ = new Queue(null, { name: 'video' })

Queue databaseInitDelay Option

Default: 1000 milliseconds

Valid: Any positive number. Keep it under 10 seconds though.

If you instantiate a Queue object with options pointing to a database or table that does not exist, rethinkdb-job-queue will create the database, table, and indexes for you. This is a great feature, however it causes problems if multiple Queue objects are instantiated at the same time. It will cause two databases or tables with the same name to be created within the same RethinkDB instance.

To mitigate this issue there is a delay before the databases resources are created. This delay comprises of a fixed portion and a random portion. The databaseInitDelay option is exposing the fixed portion of the initialization delay. The random portion is hard coded and will be between zero and one second.

This option is ignored if the Queue object is a Queue Master. Only the hard coded delay will apply.

databaseInitDelay Example

This example will set the databaseInitDelay to five seconds causing the initialization delay to be equal to five seconds plus a random value between zero and one second.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { databaseInitDelay: 5000 })

Queue queryRunOptions Option

Default: { readMode: 'majority' }

Valid: JavaScript object with valid properties.

The queryRunOptions option was added to expose access to the run options for queries sent to the RethinkDB database.

The default object sets the readMode value to majority to improve the database consistency.

readMode Performance Consideration

The default value of the queryRunOptions readMode option improves consistency of data access at the expense of performance. Consider changing the readMode to single if your use case is either of the following;

• You are only using a single server in your RethinkDB cluster.
• Your jobs take many seconds to complete and your concurrency is set to 1.

If you do change the run option readMode to single and you experience the same job being processed by multiple Queue workers, change it back to the default majority value.

queryRunOptions Example

This example changes the default queryRunOptions from readMode: majority to readMode: single to improve performance on a single database cluster.

const Queue = require('rethinkdb-job-queue')
const runOptions = { readMode: 'single' }
const Q = new Queue(null, { queryRunOptions: runOptions })

Queue changeFeed Option

Default: true

Valid: true or false.

If enabled the changeFeed option will cause the Queue object to raise events for the entire queue rather than just the local Queue object.

Behind the scenes a RethinkDB change feed is created for the whole of the queue Table in the database. Once the changes are received by the Queue object, they are checked to ensure they are not local changes. If the changes are detected as local changes (a change made by self) they are ignored because the events will be raised by the Queue object. If the changes are not local then the relevant event is raised.

See the Queue Events document for more detail.

changeFeed Example

This example simply disables the changeFeed option.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { changeFeed: false })

Queue concurrency Option

Default: 1

Valid: Any positive Integer.

The Queue object can process many jobs at once. Keep in mind the blocking nature of long running tasks. If your jobs are external and will not block the node process, feel free to push the concurrency value through the roof.

concurrency Example

This example sets the concurrency option to 100. If more than 100 jobs are available on the queue then 100 will be processed at once.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { concurrency: 100 })

Queue masterInterval Option

Default: 310000 milliseconds (5 min 10 sec)

Valid: Enable with a positive Integer. Disable with false or 0.

The masterInterval option determines how often the Queue Master review process is carried out against the queue. Please refer to the Queue Master document for more detail.

The default value of 310000 milliseconds was chosen because it is 10 seconds past the job timeout default value of 300000 milliseconds. This is to improve detection of failed nodejs processes on queue startup.

Warning: Setting the masterInterval value too low will cause excessive database load. Every time this period lapses queries are run against the database. Keep this value as high as possible.

masterInterval Example

This example shows setting the masterInterval to 1 minute to ensure the queue is kept clean and failed jobs get processed.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { masterInterval: 600000 })

Queue limitJobLogs Option

Default: 1000 maximum number of log entries for each job

Valid: Any positive integer. Do not set to 0.

Using the Job Repeat option or some other variant of processing, a Jobs log entries will continue to grow in number. This Queue option will limit the Jobs log entries to the value specified.

It is important to note that the log limit is only applied during the job completion or failure. If some other processing is applied to a single job and it is never completed or failed the log limit will not apply.

If the number of log entries on a job exceeds the limitJobLogs value and the job is completed or failed, the most recent log entries are retained up to the limitJobLogs value. There is also a log truncation entry added to the existing log entries.

limitJobLogs Example

This example shows setting the limitJobLogs option to 50.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { limitJobLogs: 50})

If the above limit is reached on a job the following log entry is added:

{
  "data":  "Retaining 50 log entries" ,
  "date": Mon Mar 20 2017 00:10:33 GMT+00:00 ,
  "message":  "Job logs have been truncated" ,
  "processCount": 2 ,
  "queueId":  "WebDev:rjqJobQueueTests:jobCompleted:43870:d0d30257-6c2b-4553-9779-4a3c2e97fae2" ,
  "retryCount": 0 ,
  "status":  "waiting" ,
  "type":  "information"
}

Queue removeFinishedJobs Option

Default: 15552000000 milliseconds (180 days)

Valid: true, false or a positive Integer.

Important: Setting the removeFinishedJobs option to a positive integer is only valid for a Queue Master.

The removeFinishedJobs option is used to control how the queue removes jobs which are considered to be finished from the database.

A job is considered finished when it has a status of either completed, cancelled, or terminated. See the Job Status document for more detail.

Finished jobs will be permanently removed from the database based on the values you assign to removeFinishedJobs.

This is one of the more flexible options available on Queue objects. You have three values you can assign to the removeFinishedJobs option.

Value 1: true

Setting the removeFinishedJobs option to true will cause jobs to be removed from the queue immediately when they are finished. They will no longer be in the database.

This value is valid for all Queue objects.

Value 2: false or 0

Setting the removeFinishedJobs option to false or 0 will prevent the queue from ever removing jobs from the database.

This value is valid for all Queue objects.

Value 3: Integer as number of milliseconds

By setting the removeFinishedJobs option to a number such as 15552000000, the Queue Master review process will permanently remove jobs that were finished 180 days ago.

This value is only valid for Queue Master Queue objects. If you set this on a non-master Queue object it will operate the same as setting the value to false.

removeFinishedJobs Example

This example shows setting removeFinishedJobs option to false causing the queue to never remove jobs from the database:

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { removeFinishedJobs: false })

This example shows setting the removeFinishedJobs option to 30 days on the Queue Master. The Queue Master review process will remove finished jobs when they are a month old.

const Queue = require('rethinkdb-job-queue')
const q = new Queue(null, { removeFinishedJobs: 2592000000 })