Queue Everything and Please Everyone

9f2a1db7d95b36b91c2b97dab2941cf6?s=47 Vaidik Kapoor
September 15, 2012

Queue Everything and Please Everyone

The intent of this talk is to introduce the concepts of queueing in web development, tools at one’s disposal, some common patterns and how it all can be used to reduce time to generate user feedback.

9f2a1db7d95b36b91c2b97dab2941cf6?s=128

Vaidik Kapoor

September 15, 2012
Tweet

Transcript

  1. Queue Everything & Please Everyone * * * Vaidik Kapoor

  2. Who am I? ➔ Undergraduate student. ➔ Involved with: ✔

    Mozilla as a Contributor and Rep ✔ Drupal as a Contributor ➔ Twitter: @vaidikkapoor ➔ Github: vaidikkp ➔ Web: vaidikkapoor.info
  3. Web Applications

  4. Responsiveness

  5. Why do everything at once? Task 1 Task 2 Task

    3 Task N Client Request Response
  6. Break the Request-Response Cycle Task 1 Task 2 Task 3

    Task N Client Request Response
  7. Meet Queues

  8. What can Queues help with? ➔ Background Processing ✔ Data

    Processing ✔ Media Processing ✔ Updating Caches ➔ Anything that you want to offload off your server. ➔ Improve the overall User Experience
  9. Redis + HotQueue / PyRes

  10. Redis + HotQueue PRODUCER: from hotqueue import HotQueue queue =

    HotQueue("myqueue", host="localhost", port=6379, db=0) queue.put(message) CONSUMER / WORKER: from hotqueue import HotQueue queue = HotQueue("myqueue", host="localhost", port=6379, db=0) while True: message = queue.get() # do something awesome
  11. Redis + PyRes ➔ Python clone of Github's Resque ➔

    Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information ➔ Comes with Monitoring System written in Ruby
  12. Redis + PyRes PRODUCER: from pyres import ResQ r =

    Resq() class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshot r.enqueue(WebPage, 'http://python.org') CONSUMER: ./pyres_worker screenies
  13. Message Queue A system for enabling asynchronous processing of discrete

    tasks.
  14. What can MQs help with? ➔ Everything that generic Queues

    can help with. ➔ Increase Reliability ➔ Cron Jobs (Celery)
  15. Available MQ Solutions ➔ RabbitMQ ➔ Amazon Simple Queue ➔

    Apache MQ ➔ Gearman ➔ Starling ➔ OpenAMQ ➔ Sun Java Message Queue System
  16. Message Queue Protocols ➔ AMQP: Advanced Message Queue Protocol ➔

    JMS: Java Messaging Service ➔ STOMP: Streaming Text Oriented Messaging Protocol
  17. Criteria for Broker Selection ➔ Difficulty in Recovery ➔ Relatively

    low level of required maintenance ➔ Ease of Deployment ➔ Durability ➔ Persistence ➔ Community Support ➔ Cluster Support ➔ What language is it written in?
  18. RabbitMQ ➔ A Message Broker ➔ Implements AMQP

  19. Pika & Other AMQP Libraries

  20. Celery ➔ An amazing Task Manager ➔ Batteries Included ➔

    Uses RabbitMQ as broker (or almost anything) ➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid ➔ Supports multiprocessing. ➔ Distribute tasks easily. ➔ Makes life easy
  21. Celery Example from celery.task import task @task(): def take_screenshot(url): #

    magical code to take screenshot comes here
  22. How to put it all together?

  23. Web App Broker Task Manager Worker Server (s) Database The

    General Setup
  24. Then what?

  25. Notify Your Users

  26. Django's Messaging System

  27. Flask's Flashes

  28. Poll using AJAX

  29. WebSockets

  30. Email

  31. Things to Remember

  32. Isolate Isolate code, make reusable components.

  33. Recycle Remove Request Dependency

  34. Unit Testing?

  35. Review ➔ Choose a system according to needs ➔ Build

    a robust system ➔ Integrate seamlessly with your UI ➔ Isolate & Recycle: make it a habit ➔ Improve the overall UX! ➔ Please everyone, even yourself!
  36. References ➔ PyRes - http://pyres.readthedocs.org/ ➔ RabbitMQ - http://www.rabbitmq.com/ ➔

    Celery - http://celeryproject.org/ ➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  37. Thank You!