Meteor Background Jobs Take the load off

Note: If you understand the theory behind background jobs and want to skip to the coding part of the tutorial, click here.

Background Jobs

At some point, your app may need a new feature that involves some heavy lifting by the CPU. A common example is creating thumbnails for images. A user may upload 10 files, and for each file a thumbnail needs to be made. You could do this in the request/response lifecycle (straight after the user has uploaded the images), but this would mean the user would have to wait a while without a response from the server (I know Meteor comes with latency compensation, but it still is not ideal). It also would hold up Meteor from handling other clients.

A bakery

To understand this better, lets imagine our application as a bakery with one employee at the front serving our customers, and one in a back office. You can think of the front employee as your application, serving customers with their food requests. One customer might make an order for 20 loaves of bread, which need to be cooked in the back office (or maybe I should call it kitchen). The front employee can't waste their time cooking the loaves - other customers won't be able to be served. So she/he delegates the job to the back room and tells the customer to come back later.

This is essentailly what we are doing with our application. Spinning up another Node process is kind of like giving us another human worker. In saying this, your Meteor application doesn't actually work with with just one thread (or human if using the bakery analogy) - deep down, the slow I/O tasks are delegated to other threads. You can read more about it this article I wrote: Where does Meteor fit in?.

In terms of actual threads doing work - here is how I visualise it:

In the above picture we can see how CPU intensive tasks like creating thumbnails can slowdown throughput (rate of requests being processed).

Meteor workers package

There are a couple of packages that allow you to run background jobs, namely Meteor Job Collection and Meteor Workers. I ended up going with Meteor Workers because of its self contained deployment:

"We wanted to be able to deploy our app just like we always have and not have a separate deployment for the application handling the jobs."

Usually background jobs require you to set up separate processes to work through the jobs. This can take time and creates more overhead. With Meteor Workers, we still just run meteor and along with our app starting we will see a separate background worker process (or multiple, depending on our configuration), start along with it.

This "monq" worker will do our background jobs for us in a separate thread to the one running our Meteor app code.

The code

Let's try it out. Firstly, create a Meteor app, add the package, then start the app:

meteor create background-workers
cd background-workers
meteor add differential:workers
meteor

Let's make it so when you click the "Click me" button it creates a simple background job.

To create a job we must define what happens in the job, and then add the job to the queue.

Coffeescript

Note that the Meteor Worker README code is in Coffeescript, mainly because they require you to extend the Job 'class'. Coffeescript makes it easy to perform inheritence, which is hard in Javascript due to it's prototypical nature. Basically you have a Job 'class', which comes with some default functionality and functions. You're required to extend this 'Job' class, and override the handleJob() function. If we decide to go the Javascript route we'd need an extend() function that mimics Object.create(), which is kind of annoying (read this if you're interested).

So for simplicity we'll use Coffeescript. I actually highly recommend Coffeescript and use it on all my projects (personal and for professionally) - you can learn it very fast and overall it's more enjoyable and makes you write code faster. I've been writing my Meteor book code in Javascript (because my Coffeescript tutorials were getting some hate) and found it very painful going back to Javascript after writing in Coffeescript). Anyway back to the code.

Add the coffeescript package:

meteor add coffeescript

Now rename background-workers.js to background-workers.coffee and rewrite it like this:

Now when you click the button, you'll see your handleJob() function's console.log() appear in the server logs:

Some things to note:

• We use a method to create the job. The method is only defined on the server because it will give an undefined error if run on the console
• The @ symbol at the front of @TestJob is necessary. It is equivalent to this.TestJob, which adds it as a property of the global root object (on the client it would be window, on Node it's just called global). Meteor Workers actually goes through every property of global looking for properties ending in "Job".
• While your handleJob() function is being run in a separate thread, it still has access to your Meteor app - so you can use your collections and Meteor libraries

This actually is all you really need to know for creating background jobs - you just put your CPU intensive code in the handleJob function. In terms of where to put them, I usually put them in server/jobs/, with each job getting it's own file.

Recurring jobs

Sometimes you want your job to just run every so often, like every hour or so. Meteor Workers allows this too. Let's make ours run every 5 seconds:

• @setupCron needs the @ symbol. So it's set as a property of the TestJob object, not its instances
• The parser variable passed as a parameter of @setupCron is a later.js parse object, which allows you to specify the time period in a cool way

Debugging

Looking inside the package we can see it creates a new collection called jobs:

# meteor-workers/collections/jobs.coffee
Jobs = new Mongo.Collection "jobs"

So at any time you can peek in the database and check what jobs are in there:

meteor mongo
db.jobs.find()

Any errors will be shown in these documents along with stack trace.

Configuration

Configuration is done using the Meteor.settings API. The easiest way is to use a separate settings.json file in the root of your project:

For this to take effect you have to run your app like this: meteor --settings settings.json. This creates 2 processes with up to 10 workers (threads).

The end

So overall, background jobs aren't really that complicated, mainly thanks to the Meteor Workers package and it's underlying Node packages (like Monq).