Queue Everything and Please Everyone

Slide 1

Slide 1 text

Queue Everything & Please Everyone * * * Vaidik Kapoor

Slide 2

Slide 2 text

Who am I? ➔ Undergraduate student. ➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor ➔ Twitter: @vaidikkapoor ➔ Github: vaidikkp ➔ Web: vaidikkapoor.info

Slide 3

Slide 3 text

Web Applications

Slide 4

Slide 4 text

Responsiveness

Slide 5

Slide 5 text

Why do everything at once? Task 1 Task 2 Task 3 Task N Client Request Response

Slide 6

Slide 6 text

Break the Request-Response Cycle Task 1 Task 2 Task 3 Task N Client Request Response

Slide 7

Slide 7 text

Meet Queues

Slide 8

Slide 8 text

What can Queues help with? ➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches ➔ Anything that you want to offload off your server. ➔ Improve the overall User Experience

Slide 9

Slide 9 text

Redis + HotQueue / PyRes

Slide 10

Slide 10 text

Redis + HotQueue PRODUCER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) queue.put(message) CONSUMER / WORKER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) while True: message = queue.get() # do something awesome

Slide 11

Slide 11 text

Redis + PyRes ➔ Python clone of Github's Resque ➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information ➔ Comes with Monitoring System written in Ruby

Slide 12

Slide 12 text

Redis + PyRes PRODUCER: from pyres import ResQ r = Resq() class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshot r.enqueue(WebPage, 'http://python.org') CONSUMER: ./pyres_worker screenies

Slide 13

Slide 13 text

Message Queue A system for enabling asynchronous processing of discrete tasks.

Slide 14

Slide 14 text

What can MQs help with? ➔ Everything that generic Queues can help with. ➔ Increase Reliability ➔ Cron Jobs (Celery)

Slide 15

Slide 15 text

Available MQ Solutions ➔ RabbitMQ ➔ Amazon Simple Queue ➔ Apache MQ ➔ Gearman ➔ Starling ➔ OpenAMQ ➔ Sun Java Message Queue System

Slide 16

Slide 16 text

Message Queue Protocols ➔ AMQP: Advanced Message Queue Protocol ➔ JMS: Java Messaging Service ➔ STOMP: Streaming Text Oriented Messaging Protocol

Slide 17

Slide 17 text

Criteria for Broker Selection ➔ Difficulty in Recovery ➔ Relatively low level of required maintenance ➔ Ease of Deployment ➔ Durability ➔ Persistence ➔ Community Support ➔ Cluster Support ➔ What language is it written in?

Slide 18

Slide 18 text

RabbitMQ ➔ A Message Broker ➔ Implements AMQP

Slide 19

Slide 19 text

Pika & Other AMQP Libraries

Slide 20

Slide 20 text

Celery ➔ An amazing Task Manager ➔ Batteries Included ➔ Uses RabbitMQ as broker (or almost anything) ➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid ➔ Supports multiprocessing. ➔ Distribute tasks easily. ➔ Makes life easy

Slide 21

Slide 21 text

Celery Example from celery.task import task @task(): def take_screenshot(url): # magical code to take screenshot comes here

Slide 22

Slide 22 text

How to put it all together?

Slide 23

Slide 23 text

Web App Broker Task Manager Worker Server (s) Database The General Setup

Slide 24

Slide 24 text

Then what?

Slide 25

Slide 25 text

Notify Your Users

Slide 26

Slide 26 text

Django's Messaging System

Slide 27

Slide 27 text

Flask's Flashes

Slide 28

Slide 28 text

Poll using AJAX

Slide 29

Slide 29 text

WebSockets

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Things to Remember

Slide 32

Slide 32 text

Isolate Isolate code, make reusable components.

Slide 33

Slide 33 text

Recycle Remove Request Dependency

Slide 34

Slide 34 text

Unit Testing?

Slide 35

Slide 35 text

Review ➔ Choose a system according to needs ➔ Build a robust system ➔ Integrate seamlessly with your UI ➔ Isolate & Recycle: make it a habit ➔ Improve the overall UX! ➔ Please everyone, even yourself!

Slide 36

Slide 36 text

References ➔ PyRes - http://pyres.readthedocs.org/ ➔ RabbitMQ - http://www.rabbitmq.com/ ➔ Celery - http://celeryproject.org/ ➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

Slide 37

Slide 37 text

Thank You!