Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ZendCon 2012 - Message Queues : A Primer

ZendCon 2012 - Message Queues : A Primer

Message queues are everywhere and with the many popular options it can be hard to figure out which one you should use or where you might want to use a message queue. This talk will go through each of the standardized protocols; take a look at different message queues in action with PHP and give you an idea which one you might want to utilize for your next project.


Mike Willbanks

October 24, 2012

More Decks by Mike Willbanks

Other Decks in Technology


  1. Message Queues : A Primer Mike Willbanks | Barnes &

  2. 2 •  Talk  Slides will be posted after the talk.

    •  Me  Sr. Web Architect Manager at NOOK Developer  Open Source Contributor  Where you can find me: •  Twitter: mwillbanks G+: Mike Willbanks •  IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com •  GitHub: https://github.com/mwillbanks Housekeeping…
  3. 3 • Message Queues? • Protocols and Software • Picking a Message Queue

    • Best Practices Agenda
  4. Message Queues?

  5. 5 A Definition “Message queues and mailboxes are software-engineering components

    used for interprocess communication, or for inter- thread communication within the same process. They use a queue for messaging – the passing of control or of content.”
  6. 6 What is messaging? “Messaging describes the sending and receiving

    of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”
  7. 7 General Anatomy Task Producer Consumer Messages Messages Messages Producer

    creates a message and pushes it to the queue; the consumer reads from the queue and processes the message.
  8. 8 l  Pub/Sub l  FIFO buffer l  Push / Pull

    l  A way to communicate between applications / systems. l  A way to decouple components. l  A way to offload work. Describing Message Queues
  9. 9 l  Offload Heavy Work l  Integration with Legacy Systems

    l  Asynchronous Processing l  Parallel Processing l  Process consistency l  Scalability Why to use a Message Queue
  10. 10 l  “Write programs that work together.” l  “Do it

    in the background.” Unix Foundations
  11. 11 Why it matters l  Web systems need to be

    geared to run things asynchronously. l  Distribution of load l  System integrity
  12. 12 Enterprise Integration Patterns Understanding messaging patterns will greatly assist

    you in building out great systems for messaging. http://www.eaipatterns.com/
  13. Common Messaging Patterns

  14. 14 Publish-Subscribe

  15. 15 Request-Reply

  16. 16 Message Router

  17. Message queue examples •  You’ve seen them before; they are

    used in most applications to help them scale.
  18. 18

  19. 19

  20. 20

  21. Generic Use Cases •  When to make use of message

    queues in case you’re wondering.
  22. 22 Notifications Email, SMS, Push Messaging….

  23. 23 Photo Processing Thumbnails, Resizing, Watermarking, Converting…

  24. 24 Video Processing Resampling, Audio Overlay, Type Conversion…

  25. 25 Analytics Web Server Logs, Log Aggregation, PHP Errors, etc.

  26. 26 Integrations Save local first; push second.

  27. Protocols •  AMQP •  STOMP •  XMPP •  Vendor Specific

  28. 28 AMQP Advanced Message Queuing Protocol

  29. 29 l  AMQP Working Group (Community and Vendor) l  Platform

    agnostic protocol. l  Completely open, interoperable and broadly applicable. l  Many severs available and many client libraries. Overview of AMQP
  30. 30 How it Works Producer, Exchange, Queue, Consumer

  31. 31 l  AMQP utilizes exchanges, queues and bindings. l  An

    exchange are routers with routing tables. l  A binding defines the routing rules. l  A queue is where the messages wait for a consumer. How it Works
  32. 32 Exchanges

  33. 33 l  Fanout Exchange l  No routing keys involved. Any

    message that is sent to the exchange is sent to all queues bound to that exchange. l  Direct Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly. l  Topic Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern. Understanding Exchanges
  34. 34 Implementations www.rabbitmq.com Very popular and common message queue owned

    by VMware. qpid.apache.org Long standing project; apache foundation. www.openamq.org Long standing project; ZeroMQ partner, no news since 2009.
  35. 35 l  An exchange, queue and bindings must be defined

    first. Publishing can then commence after. l  Create the queue l  Create the exchange l  Bind to the queue. Building a Queue
  36. 36 l  Default behavior is no persistence. l  How important

    are the messages? l  Just about all items have a level of persistence if you would like them to survive on reboot. l  Mark exchanges, queues and messages as DURABLE. Persistence?
  37. 37 l  Extension compatible with AMQP specification 0-9-1. l  pecl

    install amqp PECL AMQP
  38. 38 AMQP Client

  39. 39 AMQP Worker

  40. 40 STOMP Simple (or Streaming) Text Orientated Messaging Protocol

  41. 41 Overview l  Simple protocol l  Behaviors follow very simple

    commands. l  Most message queues can communicate over STOMP.
  42. 42 How It Works l  When you send in a

    message, you tell it which queue to go to. l  When you subscribe you request a queue. Connect Send Disconnect /queue/ msg P H P S T O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack
  43. 43 Sever Implementations activemq.apache.org One of the oldest message queues

    existing; a apache foundation project activemq.apache.org/apollo Next generation ActiveMQ www.rabbitmq.com Very popular and common message queue owned by VMware. www.jboss.org/hornetq Supported by Red Hat Middle Ware division, picking up steam.
  44. 44 l  pecl install stomp l  That was easy J

    PECL Stomp
  45. 45 STOMP Client

  46. 46 STOMP Worker

  47. 47 XMPP Extensible Messaging and Presence Protocol (Although not really

    a “Message Queue”)
  48. 48 Overview l  Best for real-time data. l  Leveraging pub/sub

    can turn it into more of a generic message system. l  Multiple libraries l  JAXL - https://github.com/abhinavsingh/JAXL l  Xmpp - https://github.com/alexmace/Xmpp
  49. 49 XEP-0060: Publish-Subscribe l  Specification for implementing Publish Subscribe models.

    l  Extension to the original XMPP specification. Publish Subscribe Sub1 Sub2 Sub3 to, id, message from, to, id, message
  50. 50 Publish

  51. 51 Message

  52. Special Message Queues •  Various others; including Gearman, ZeroMQ, etc.

  53. 53 l  There are job servers available that are more

    flexible or more specific. l  Extreme Flexibility l  Job Severs l  Cloud Messaging Overview
  54. 54 ZeroMQ The ultimate in message queue flexibility. Socket library

    that acts as a concurrency framework. Contains a PHP extension.
  55. 55 Several Types of Queues Request / Reply Publish /

    Subscribe Parallel Pipeline Fair Queuing
  56. 56 ZeroMQ Client Example

  57. 57 ZeroMQ Worker Example

  58. 58 l  Application framework for farming out work. l  Job

    sever for asynchronous or synchronous messages. Gearman
  59. 59 l  Pass a job to the job server l 

    Worker receives the job and processes l  Ability to persist if enabled; default is in-memory. Gearman
  60. 60 Gearman Client Example

  61. 61 Gearman Worker Example

  62. 62 l  Asynchronous Job Queue l  Good scheduling system (aka

    delays). l  User land PHP clients. Beanstalkd
  63. 63 l  Pass a job to the job server l 

    Worker receives the job and processes l  Ability to persist to binlog; default is in-memory Beanstalkd Application Code Beanstalkd Client API (Pheanstalk) Beanstalkd Server Beanstalkd Worker API (Pheanstalk) Worker Application Code
  64. 64 Pheanstalk Client

  65. 65 Pheanstalk Worker

  66. Picking a Message Queue •  What you want to look

  67. 67 The Trifecta Scale Performance Durability Messages Horizontal? Vertical? In-Memory?

    Persistence? Replication? Raw Speed? # Messages? Messages Messages Messages Messages Messages Messages Messages Messages
  68. 68 Standards •  A recognized standard?  AMQP? STOMP? XMPP? • 

    How many developers?  Will an unfortunate event kill off the product? •  Undocumented protocol?  Forget about it!
  69. 69 Delivery Policies Store Message Saved? Process Message ACK? Receive

    Message TTL?
  70. 70 Message Policies •  Handling in-active messages? •  Messages that

    failed processing? •  Time to live? •  Retry? •  Ability to check message status?
  71. 71 Routing Policies •  Fanout? •  Direct? •  Topic? • 

    Broadcast? •  Etc?
  72. 72 Security Policies •  Require authentication?  LDAP/AD integration? •  Connection

    Restrictions? •  Require logging?  Compliance  Intrusion Detection
  73. Best Practices •  Rules of the road my friends…

  74. 74 Messages •  Formatting  JSON or XML are great options.

    •  Please no serialized PHP objects. •  Message Size  Only as large as necessary. •  Don’t send a binary object through the queue.
  75. 75 Workers •  Dumb as possible!  Single message type • 

    Easier to scale.  Single operation •  Easy to debug and far more flexible.
  76. 76 PHP Daemons •  Prevent Memory Leaks  Detection  Cycle Workers

    •  Handle Signals  Properly shutdown!  Watch out for OS service kills •  Sleeping is good J
  77. 77

  78. 78 Environment Integration Web Server Database Worker Server Message Queue

    Web Server -> Message Queue Worker Server -> Message Queue
  79. 79 High Availability Web Server Database Worker Server Message Queue

    Message Queue Worker Server Send to 1 Insert multiple message queue servers + multiple worker nodes. Each worker node can connect to as many message queue servers as necessary.
  80. 80 Supervisord Process monitoring, logging and more!

  81. 81 Installing Supervisord • Requires: python-setuptools sudo easy_install supervisor! • Generate Configuration

    sudo echo_supervisord_conf > /etc/supervisord.conf! !
  82. 82

  83. 83 •  These slides will be posted to SlideShare &

    SpeakerDeck.   SpeakerDeck: http://speakerdeck.com/u/mwillbanks   Slideshare: http://www.slideshare.net/mwillbanks   Twitter: mwillbanks   G+: Mike Willbanks   IRC (freenode): mwillbanks   Blog: http://blog.digitalstruct.com   GitHub: https://github.com/mwillbanks Questions?