Chapman: Building a Distributed Job Queue in MongoDB

55455771d6771dfd58860695d3e597e6?s=47 rick446
June 21, 2013

Chapman: Building a Distributed Job Queue in MongoDB

When you're building a web application, you want to respond to every request as quickly as possible. The usual approach is to use an asynchronous job queue like Sidekiq, Resque, Celery, RQ, or a number of other frameworks to handle those tasks outside the request/response cycle in a separate 'worker' process. Unfortunately, many of these frameworks either require the deployment of Redis, RabbitMQ, or some other request broker, or they resort to polling a database for new work to do. Chapman is a distributed task queue built on MongoDB that avoids gratuitous polling, using capped collections and tailable cursors to provide notifications of incoming work. Inspired by Celery, Chapman also supports task graphs, where multiple tasks that depend on each other can be executed by the system asynchronously. Come learn how is using MongoDB and Chapman to handle its core data processing needs.



June 21, 2013