Chapman: Building a High-Performance Distributed Task Service with MongoDB

55455771d6771dfd58860695d3e597e6?s=47 rick446
June 02, 2015

Chapman: Building a High-Performance Distributed Task Service with MongoDB

When you're building a web application, you want to respond to every request as quickly as possible. The usual approach is to use an asynchronous job queue like Sidekiq, Resque, Celery, RQ, or a number of other frameworks to handle those tasks outside the request/response cycle in a separate 'worker' process. Unfortunately, many of these frameworks either require the deployment of Redis, RabbitMQ, or some other request broker, or they resort to polling a database for new work to do. Chapman is a distributed task queue built on MongoDB that avoids gratuitous polling, using tailable cursors with the oplog to provide notifications of incoming work. Inspired by Celery, Chapman also supports task graphs, where multiple tasks that depend on each other can be executed by the system asynchronously. Come learn how is using MongoDB and Chapman to handle its core data processing needs.



June 02, 2015