Upgrade to Pro — share decks privately, control downloads, hide ads and more …

International PHP Conference - Fall 2012 : Message Queues - A Primer

International PHP Conference - Fall 2012 : Message Queues - A Primer

A primer to implementing message queues; the protocols and how to work with the various message queue systems.

Mike Willbanks

October 15, 2012
Tweet

More Decks by Mike Willbanks

Other Decks in Programming

Transcript

  1. Mike Willbanks | Barnes & Noble
    Message Queues : A Primer

    View Slide

  2. Housekeeping…
    •  Talk
    –  Slides will be posted after the talk.
    •  Me
    –  Sr. Web Architect Manager at NOOK
    Developer
    –  Prior MNPHP Organizer
    –  Open Source Contributor
    –  Where you can find me:
    •  Twitter: mwillbanks G+: Mike Willbanks
    •  IRC (freenode): mwillbanks Blog:
    http://blog.digitalstruct.com
    •  GitHub: https://github.com/mwillbanks

    View Slide

  3. Agenda
    •  Message Queues?
    •  Protocols and Software
    •  Picking a Message Queue
    •  Best Practices

    View Slide

  4. A Definition
    “Message queues and mailboxes are software-engineering
    components used for interprocess communication, or for inter-
    thread communication within the same process. They use a queue
    for messaging – the passing of control or of content.”

    View Slide

  5. What is messaging?
    “Messaging describes the sending and receiving of data (in the
    form of messages) between systems. Messages are exchanged
    between programs or applications, similar to the way people
    communicate by email but with guarantees on delivery, speed,
    security and the absence of spam.”

    View Slide

  6. General Anatomy
    Task
    Producer Consumer
    Messages
    Messages
    Messages
    Producer creates a message and pushes it to the queue; the
    consumer reads from the queue and processes the message.

    View Slide

  7. l 
    Pub/Sub
    l 
    FIFO buffer
    l 
    Push / Pull
    l 
    A way to communicate between
    applications / systems.
    l 
    A way to decouple components.
    l 
    A way to offload work.
    Describing Message Queues

    View Slide

  8. l 
    Offload Heavy Work
    l 
    Integration with Legacy Systems
    l 
    Asynchronous Processing
    l 
    Parallel Processing
    l 
    Process consistency
    l 
    Scalability
    Why to use a Message Queue

    View Slide

  9. Unix Foundations
    l 
    “Write programs that work together.”
    l 
    “Do it in the background.”

    View Slide

  10. l 
    Web systems need to be geared to run things
    asynchronously.
    l 
    Distribution of load
    l 
    System integrity
    Why it matters

    View Slide

  11. MESSAGE QUEUE EXAMPLES
    You’ve seen them before; they are used in most applications to
    help them scale.

    View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. GENERIC USE CASES
    When to make use of message queues in case you’re wondering.

    View Slide

  16. Notifications
    Email, SMS, Push Messaging….

    View Slide

  17. Photo Processing
    Thumbnails, Resizing, Watermarking, Converting…

    View Slide

  18. Video Processing
    Resampling, Audio Overlay, Type Conversion…

    View Slide

  19. Analytics
    Web Server Logs, Log Aggregation, PHP Errors, etc.

    View Slide

  20. Integrations
    Save local first; push second.

    View Slide

  21. PROTOCOLS
    AMQP
    STOMP
    XMPP
    Vendor Specific

    View Slide

  22. AMQP
    Advanced Message Queuing Protocol

    View Slide

  23. l 
    AMQP Working Group (Community and
    Vendor)
    l 
    Platform agnostic protocol.
    l 
    Completely open, interoperable and broadly
    applicable.
    l 
    Many severs available and many client
    libraries.
    Overview of AMQP

    View Slide

  24. How it Works
    Producer, Exchange, Queue, Consumer

    View Slide

  25. l 
    AMQP utilizes exchanges, queues and
    bindings.
    l 
    An exchange are routers with routing tables.
    l 
    A binding defines the routing rules.
    l 
    A queue is where the messages wait for a
    consumer.
    How it Works

    View Slide

  26. Exchanges

    View Slide

  27. l 
    Fanout Exchange
    l 
    No routing keys involved. Any message that is
    sent to the exchange is sent to all queues bound
    to that exchange.
    l 
    Direct Exchange
    l 
    Routing keys involved. A queue binds to the
    exchange to request messages that match a
    routing key exactly.
    l 
    Topic Exchange
    l 
    Routing keys involved. A queue binds to the
    exchange to request messages that match a
    routing key pattern.
    Understanding Exchanges

    View Slide

  28. Implementations
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    qpid.apache.org
    Long standing project; apache foundation.
    www.openamq.org
    Long standing project; ZeroMQ partner, no news since
    2009.

    View Slide

  29. l 
    An exchange, queue and bindings must be defined
    first. Publishing can then commence after.
    l 
    Create the queue
    l 
    Create the exchange
    l 
    Bind to the queue.
    Building a Queue

    View Slide

  30. l 
    Default behavior is no persistence.
    l 
    How important are the messages?
    l 
    Just about all items have a level of persistence if you
    would like them to survive on reboot.
    l 
    Mark exchanges, queues and messages as DURABLE.
    Persistence?

    View Slide

  31. l 
    Extension compatible with AMQP specification
    0-9-1.
    l 
    pecl install amqp
    PECL AMQP

    View Slide

  32. AMQP Client

    View Slide

  33. AMQP Worker

    View Slide

  34. STOMP
    Simple (or Streaming) Text Orientated Messaging Protocol

    View Slide

  35. Overview
    l 
    Simple protocol
    l 
    Behaviors follow very simple commands.
    l 
    Most message queues can communicate over
    STOMP.

    View Slide

  36. How It Works
    l 
    When you send in a message, you tell it which
    queue to go to.
    l 
    When you subscribe you request a queue.
    Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack

    View Slide

  37. Sever Implementations
    activemq.apache.org
    One of the oldest message queues existing;
    a apache foundation project
    activemq.apache.org/apollo
    Next generation ActiveMQ
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    www.jboss.org/hornetq
    Supported by Red Hat Middle Ware division, picking
    up steam.

    View Slide

  38. l 
    pecl install stomp
    l 
    That was easy J
    PECL Stomp

    View Slide

  39. STOMP Client

    View Slide

  40. STOMP Worker

    View Slide

  41. XMPP
    Extensible Messaging and Presence Protocol
    (Although not really a “Message Queue”)

    View Slide

  42. Overview
    l 
    Best for real-time data.
    l 
    Leveraging pub/sub can turn it into more of a generic message
    system.
    l 
    Multiple libraries
    l 
    JAXL - https://github.com/abhinavsingh/JAXL
    l 
    Xmpp - https://github.com/alexmace/Xmpp

    View Slide

  43. XEP-0060: Publish-Subscribe
    l 
    Specification for implementing Publish Subscribe
    models.
    l 
    Extension to the original XMPP specification.
    Publish
    Subscribe
    Sub1 Sub2 Sub3
    to, id,
    message
    from, to, id,
    message

    View Slide

  44. Publish

    View Slide

  45. Message

    View Slide

  46. SPECIAL MESSAGE QUEUES
    Various others; including Gearman, ZeroMQ, etc.

    View Slide

  47. l 
    There are job servers available that are more
    flexible or more specific.
    l 
    Extreme Flexibility
    l 
    Job Severs
    l 
    Cloud Messaging
    Overview

    View Slide

  48. ZeroMQ
    The ultimate in message queue flexibility.
    Socket library that acts as a concurrency framework.
    Contains a PHP extension.

    View Slide

  49. Several Types of Queues
    Request / Reply
    Publish / Subscribe
    Parallel Pipeline
    Fair Queuing
    And more…

    View Slide

  50. ZeroMQ Client Example

    View Slide

  51. ZeroMQ Worker Example

    View Slide

  52. l 
    Application framework for farming out work.
    l 
    Job sever for asynchronous or synchronous
    messages.
    Gearman

    View Slide

  53. l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist if enabled; default is in-memory.
    Gearman

    View Slide

  54. Gearman Client Example

    View Slide

  55. Gearman Worker Example

    View Slide

  56. l 
    Asynchronous Job Queue
    l 
    Good scheduling system (aka delays).
    l 
    User land PHP clients.
    Beanstalkd

    View Slide

  57. l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist to binlog; default is in-memory
    Beanstalkd
    Application Code
    Beanstalkd Client API
    (Pheanstalk)
    Beanstalkd Server
    Beanstalkd Worker API
    (Pheanstalk)
    Worker Application Code

    View Slide

  58. Pheanstalk Client

    View Slide

  59. Pheanstalk Worker

    View Slide

  60. PICKING A MESSAGE QUEUE
    What you want to look for.

    View Slide

  61. The Trifecta
    Scale
    Performance Durability
    Messages
    Horizontal?
    Vertical?
    In-Memory?
    Persistence?
    Replication?
    Raw Speed?
    # Messages?
    Messages Messages
    Messages
    Messages
    Messages
    Messages Messages
    Messages

    View Slide

  62. Standards
    •  A recognized standard?
    – AMQP? STOMP? XMPP?
    •  How many developers?
    – Will an unfortunate event kill off the
    product?
    •  Undocumented protocol?
    – Forget about it!

    View Slide

  63. Delivery Policies
    Store Message Saved?
    Process Message ACK?
    Receive Message TTL?

    View Slide

  64. Message Policies
    •  Handling in-active messages?
    •  Messages that failed processing?
    •  Time to live?
    •  Retry?
    •  Ability to check message status?

    View Slide

  65. Routing Policies
    •  Fanout?
    •  Direct?
    •  Topic?
    •  Broadcast?
    •  Etc?

    View Slide

  66. Security Policies
    •  Require authentication?
    – LDAP/AD integration?
    •  Connection Restrictions?
    •  Require logging?
    – Compliance
    – Intrusion Detection

    View Slide

  67. BEST PRACTICES
    Rules of the road my friends…

    View Slide

  68. Messages
    •  Formatting
    – JSON or XML are great options.
    •  Please no serialized PHP objects.
    •  Message Size
    – Only as large as necessary.
    •  Don’t send a binary object through the queue.

    View Slide

  69. Workers
    •  Dumb as possible!
    – Single message type
    •  Easier to scale.
    – Single operation
    •  Easy to debug and far more flexible.

    View Slide

  70. PHP Daemons
    •  Prevent Memory Leaks
    – Detection
    – Cycle Workers
    •  Handle Signals
    – Properly shutdown!
    – Watch out for OS service kills
    •  Sleeping is good J

    View Slide

  71. View Slide

  72. Environment Integration
    Web Server
    Database
    Worker Server
    Message Queue
    Web Server -> Message Queue
    Worker Server -> Message Queue

    View Slide

  73. High Availability
    Web Server
    Database
    Worker Server
    Message Queue
    Message Queue
    Worker Server
    Send to 1
    Insert multiple message queue servers + multiple worker nodes.
    Each worker node can connect to as many message queue servers
    as necessary.

    View Slide

  74. Supervisord
    Process monitoring, logging and more!

    View Slide

  75. Installing Supervisord
    •  Requires: python-setuptools
    sudo easy_install supervisor!
    •  Generate Configuration
    sudo echo_supervisord_conf > /etc/
    supervisord.conf!
    !

    View Slide

  76. View Slide

  77. QUESTIONS?
    These slides will be posted to SlideShare & SpeakerDeck.
    SpeakerDeck: http://speakerdeck.com/u/mwillbanks
    Slideshare: http://www.slideshare.net/mwillbanks
    Twitter: mwillbanks
    G+: Mike Willbanks
    IRC (freenode): mwillbanks
    Blog: http://blog.digitalstruct.com
    GitHub: https://github.com/mwillbanks

    View Slide