Job.updateProgress

Method Signature

Job.updateProgress(percent)

Parameter: percent Number

• An Integer between 0 and 100 representing percentage processed.

Returns: Promise => Job

Example:

job.updateProgress(30).catch(err => console.error(err))

Description

The Job.updateProgress method carries out two very important tasks; inform the queue of the jobs progress, and resetting the job timeout algorithm.

Job Progress

Some jobs within your code may get processed so quickly that updating the progress of the job in the queue is pointless. Other jobs may take many minutes or more to process. On these long jobs it may be worth informing the queue that the job is progressing nicely.

By calling the code job.updateProgress(38) a database update query is performed changing the progress value of the job in the queue backing table to a value of 38 percent. If there are other Queue objects assigned to the same queue, and they have the changeFeed option enabled, they will be alerted to the progress of the jobs.

Warning: It is very important to be aware that EVERY call to Job.updateProgress is writing to the RethinkDB database. Consider only updating the progress every 10 seconds or every minute.

Do you need to update the progress of the job anyway? Maybe not. If none of the other Queue objects you have distributed need to know about the progress of each job then you can ignore updating the progress.

Hang on. There is more to Job.updateProgress than just informing the queue of the jobs progression. Keep reading below.

Reset Job Timeout

The other major function of calling Job.updateProgress is resetting the job timeout algorithm.

Why should I care about resetting the job timeout?

When you create jobs in rethinkdb-job-queue you can configure a job timeout option. This value is set to 300000 milliseconds by default which is 5 minutes.

If you are processing lots of long running jobs that have a varying degree of time to complete, it is possible for a job to be processed past its configured timeout value.

If this happens the Queue object processing your job will update the job to a status of 'failed' and depending on the job options will be delayed for retry.

This is a bad situation to occur and can easily be avoided by calling Job.updateProgress periodically during the job processing within the timeout period.

How does it reset the timeout?

There are two ways a job can be detected as 'failed' due to job timeout; the timeout algorithm with the Queue object, and the Queue Master review process.

In an active Queue object when it starts processing a job the setTimeout JavaScript function is called. Its job is to detect a failed job due to the handling function taking too long to process the job. This could be due to an external failure etc. This is also why you should configure the timeout value for your jobs well past the expected time the jobs will take to be processed.

When Job.updateProgress is called the setTimeout function is cleared and recreated.

The Queue Master review process will also detect jobs that have run past their timeout value. As noted above, when calling Job.updateProgress the database gets updated with the new progress value. This update query also updates the dateEnable value. This prevents the review process from marking the job with a 'failed' status.

Summary

Even if you don't need to inform other Queue object about your jobs progress, it is a good idea to periodically call Job.updateProgress to reset the timeout for the job.

Another option would be to set the job timeout value so high, an hour for example, that if a job gets detected as timed out, it truly is a failed job.

Examples

Note: The following examples are ignoring the async nature of the fictional modules. The fictional modules should be using a callback or promise base mechanism.

This example updates the job progress based on a fictional task object called foo. The foo module supports an update event.

const Queue = require('rethinkdb-job-queue')
const q = new Queue()
const foo = require('some-strange-task-module')

const job = q.createJob({ data: 'Something Important' })

q.process((job, next) => {
    try {

      foo.on('update', (queueId, jobId, percent) => {
        return job.updateProgress(percent)
      }

      let result = foo.process(job.data)

      next(null, result)
    } catch (err) {
      console.error(err)
      next(err)
    }
})

return q.addJob(job).catch(err => console.error(err))

This example updates the job progress based on a fictional task object called bar. The bar module does not support an update event however it does have a status property. This example is not changing the percent value and is only used to reset the job timeout algorithm.

const Queue = require('rethinkdb-job-queue')
const q = new Queue()
const bar = require('some-strange-task-module')

const job = q.createJob({ data: 'Something Important' })

q.process((job, next) => {
    try {

      setInterval(() => {
        if (bar.status === 'running') {
          job.updateProgress(0)
        }
      }, 10000)

      let result = bar.process(job.data)

      next(null, result)
    } catch (err) {
      console.error(err)
      next(err)
    }
})

return q.addJob(job).catch(err => console.error(err))