Upgrade to Pro — share decks privately, control downloads, hide ads and more …

International PHP Conference - Fall 2012 : Message Queues - A Primer

International PHP Conference - Fall 2012 : Message Queues - A Primer

A primer to implementing message queues; the protocols and how to work with the various message queue systems.

Mike Willbanks

October 15, 2012
Tweet

More Decks by Mike Willbanks

Other Decks in Programming

Transcript

  1. Mike Willbanks | Barnes & Noble
    Message Queues : A Primer

    View full-size slide

  2. Housekeeping…
    •  Talk
    –  Slides will be posted after the talk.
    •  Me
    –  Sr. Web Architect Manager at NOOK
    Developer
    –  Prior MNPHP Organizer
    –  Open Source Contributor
    –  Where you can find me:
    •  Twitter: mwillbanks G+: Mike Willbanks
    •  IRC (freenode): mwillbanks Blog:
    http://blog.digitalstruct.com
    •  GitHub: https://github.com/mwillbanks

    View full-size slide

  3. Agenda
    •  Message Queues?
    •  Protocols and Software
    •  Picking a Message Queue
    •  Best Practices

    View full-size slide

  4. A Definition
    “Message queues and mailboxes are software-engineering
    components used for interprocess communication, or for inter-
    thread communication within the same process. They use a queue
    for messaging – the passing of control or of content.”

    View full-size slide

  5. What is messaging?
    “Messaging describes the sending and receiving of data (in the
    form of messages) between systems. Messages are exchanged
    between programs or applications, similar to the way people
    communicate by email but with guarantees on delivery, speed,
    security and the absence of spam.”

    View full-size slide

  6. General Anatomy
    Task
    Producer Consumer
    Messages
    Messages
    Messages
    Producer creates a message and pushes it to the queue; the
    consumer reads from the queue and processes the message.

    View full-size slide

  7. l 
    Pub/Sub
    l 
    FIFO buffer
    l 
    Push / Pull
    l 
    A way to communicate between
    applications / systems.
    l 
    A way to decouple components.
    l 
    A way to offload work.
    Describing Message Queues

    View full-size slide

  8. l 
    Offload Heavy Work
    l 
    Integration with Legacy Systems
    l 
    Asynchronous Processing
    l 
    Parallel Processing
    l 
    Process consistency
    l 
    Scalability
    Why to use a Message Queue

    View full-size slide

  9. Unix Foundations
    l 
    “Write programs that work together.”
    l 
    “Do it in the background.”

    View full-size slide

  10. l 
    Web systems need to be geared to run things
    asynchronously.
    l 
    Distribution of load
    l 
    System integrity
    Why it matters

    View full-size slide

  11. MESSAGE QUEUE EXAMPLES
    You’ve seen them before; they are used in most applications to
    help them scale.

    View full-size slide

  12. GENERIC USE CASES
    When to make use of message queues in case you’re wondering.

    View full-size slide

  13. Notifications
    Email, SMS, Push Messaging….

    View full-size slide

  14. Photo Processing
    Thumbnails, Resizing, Watermarking, Converting…

    View full-size slide

  15. Video Processing
    Resampling, Audio Overlay, Type Conversion…

    View full-size slide

  16. Analytics
    Web Server Logs, Log Aggregation, PHP Errors, etc.

    View full-size slide

  17. Integrations
    Save local first; push second.

    View full-size slide

  18. PROTOCOLS
    AMQP
    STOMP
    XMPP
    Vendor Specific

    View full-size slide

  19. AMQP
    Advanced Message Queuing Protocol

    View full-size slide

  20. l 
    AMQP Working Group (Community and
    Vendor)
    l 
    Platform agnostic protocol.
    l 
    Completely open, interoperable and broadly
    applicable.
    l 
    Many severs available and many client
    libraries.
    Overview of AMQP

    View full-size slide

  21. How it Works
    Producer, Exchange, Queue, Consumer

    View full-size slide

  22. l 
    AMQP utilizes exchanges, queues and
    bindings.
    l 
    An exchange are routers with routing tables.
    l 
    A binding defines the routing rules.
    l 
    A queue is where the messages wait for a
    consumer.
    How it Works

    View full-size slide

  23. l 
    Fanout Exchange
    l 
    No routing keys involved. Any message that is
    sent to the exchange is sent to all queues bound
    to that exchange.
    l 
    Direct Exchange
    l 
    Routing keys involved. A queue binds to the
    exchange to request messages that match a
    routing key exactly.
    l 
    Topic Exchange
    l 
    Routing keys involved. A queue binds to the
    exchange to request messages that match a
    routing key pattern.
    Understanding Exchanges

    View full-size slide

  24. Implementations
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    qpid.apache.org
    Long standing project; apache foundation.
    www.openamq.org
    Long standing project; ZeroMQ partner, no news since
    2009.

    View full-size slide

  25. l 
    An exchange, queue and bindings must be defined
    first. Publishing can then commence after.
    l 
    Create the queue
    l 
    Create the exchange
    l 
    Bind to the queue.
    Building a Queue

    View full-size slide

  26. l 
    Default behavior is no persistence.
    l 
    How important are the messages?
    l 
    Just about all items have a level of persistence if you
    would like them to survive on reboot.
    l 
    Mark exchanges, queues and messages as DURABLE.
    Persistence?

    View full-size slide

  27. l 
    Extension compatible with AMQP specification
    0-9-1.
    l 
    pecl install amqp
    PECL AMQP

    View full-size slide

  28. STOMP
    Simple (or Streaming) Text Orientated Messaging Protocol

    View full-size slide

  29. Overview
    l 
    Simple protocol
    l 
    Behaviors follow very simple commands.
    l 
    Most message queues can communicate over
    STOMP.

    View full-size slide

  30. How It Works
    l 
    When you send in a message, you tell it which
    queue to go to.
    l 
    When you subscribe you request a queue.
    Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack

    View full-size slide

  31. Sever Implementations
    activemq.apache.org
    One of the oldest message queues existing;
    a apache foundation project
    activemq.apache.org/apollo
    Next generation ActiveMQ
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    www.jboss.org/hornetq
    Supported by Red Hat Middle Ware division, picking
    up steam.

    View full-size slide

  32. l 
    pecl install stomp
    l 
    That was easy J
    PECL Stomp

    View full-size slide

  33. STOMP Client

    View full-size slide

  34. STOMP Worker

    View full-size slide

  35. XMPP
    Extensible Messaging and Presence Protocol
    (Although not really a “Message Queue”)

    View full-size slide

  36. Overview
    l 
    Best for real-time data.
    l 
    Leveraging pub/sub can turn it into more of a generic message
    system.
    l 
    Multiple libraries
    l 
    JAXL - https://github.com/abhinavsingh/JAXL
    l 
    Xmpp - https://github.com/alexmace/Xmpp

    View full-size slide

  37. XEP-0060: Publish-Subscribe
    l 
    Specification for implementing Publish Subscribe
    models.
    l 
    Extension to the original XMPP specification.
    Publish
    Subscribe
    Sub1 Sub2 Sub3
    to, id,
    message
    from, to, id,
    message

    View full-size slide

  38. SPECIAL MESSAGE QUEUES
    Various others; including Gearman, ZeroMQ, etc.

    View full-size slide

  39. l 
    There are job servers available that are more
    flexible or more specific.
    l 
    Extreme Flexibility
    l 
    Job Severs
    l 
    Cloud Messaging
    Overview

    View full-size slide

  40. ZeroMQ
    The ultimate in message queue flexibility.
    Socket library that acts as a concurrency framework.
    Contains a PHP extension.

    View full-size slide

  41. Several Types of Queues
    Request / Reply
    Publish / Subscribe
    Parallel Pipeline
    Fair Queuing
    And more…

    View full-size slide

  42. ZeroMQ Client Example

    View full-size slide

  43. ZeroMQ Worker Example

    View full-size slide

  44. l 
    Application framework for farming out work.
    l 
    Job sever for asynchronous or synchronous
    messages.
    Gearman

    View full-size slide

  45. l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist if enabled; default is in-memory.
    Gearman

    View full-size slide

  46. Gearman Client Example

    View full-size slide

  47. Gearman Worker Example

    View full-size slide

  48. l 
    Asynchronous Job Queue
    l 
    Good scheduling system (aka delays).
    l 
    User land PHP clients.
    Beanstalkd

    View full-size slide

  49. l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist to binlog; default is in-memory
    Beanstalkd
    Application Code
    Beanstalkd Client API
    (Pheanstalk)
    Beanstalkd Server
    Beanstalkd Worker API
    (Pheanstalk)
    Worker Application Code

    View full-size slide

  50. Pheanstalk Client

    View full-size slide

  51. Pheanstalk Worker

    View full-size slide

  52. PICKING A MESSAGE QUEUE
    What you want to look for.

    View full-size slide

  53. The Trifecta
    Scale
    Performance Durability
    Messages
    Horizontal?
    Vertical?
    In-Memory?
    Persistence?
    Replication?
    Raw Speed?
    # Messages?
    Messages Messages
    Messages
    Messages
    Messages
    Messages Messages
    Messages

    View full-size slide

  54. Standards
    •  A recognized standard?
    – AMQP? STOMP? XMPP?
    •  How many developers?
    – Will an unfortunate event kill off the
    product?
    •  Undocumented protocol?
    – Forget about it!

    View full-size slide

  55. Delivery Policies
    Store Message Saved?
    Process Message ACK?
    Receive Message TTL?

    View full-size slide

  56. Message Policies
    •  Handling in-active messages?
    •  Messages that failed processing?
    •  Time to live?
    •  Retry?
    •  Ability to check message status?

    View full-size slide

  57. Routing Policies
    •  Fanout?
    •  Direct?
    •  Topic?
    •  Broadcast?
    •  Etc?

    View full-size slide

  58. Security Policies
    •  Require authentication?
    – LDAP/AD integration?
    •  Connection Restrictions?
    •  Require logging?
    – Compliance
    – Intrusion Detection

    View full-size slide

  59. BEST PRACTICES
    Rules of the road my friends…

    View full-size slide

  60. Messages
    •  Formatting
    – JSON or XML are great options.
    •  Please no serialized PHP objects.
    •  Message Size
    – Only as large as necessary.
    •  Don’t send a binary object through the queue.

    View full-size slide

  61. Workers
    •  Dumb as possible!
    – Single message type
    •  Easier to scale.
    – Single operation
    •  Easy to debug and far more flexible.

    View full-size slide

  62. PHP Daemons
    •  Prevent Memory Leaks
    – Detection
    – Cycle Workers
    •  Handle Signals
    – Properly shutdown!
    – Watch out for OS service kills
    •  Sleeping is good J

    View full-size slide

  63. Environment Integration
    Web Server
    Database
    Worker Server
    Message Queue
    Web Server -> Message Queue
    Worker Server -> Message Queue

    View full-size slide

  64. High Availability
    Web Server
    Database
    Worker Server
    Message Queue
    Message Queue
    Worker Server
    Send to 1
    Insert multiple message queue servers + multiple worker nodes.
    Each worker node can connect to as many message queue servers
    as necessary.

    View full-size slide

  65. Supervisord
    Process monitoring, logging and more!

    View full-size slide

  66. Installing Supervisord
    •  Requires: python-setuptools
    sudo easy_install supervisor!
    •  Generate Configuration
    sudo echo_supervisord_conf > /etc/
    supervisord.conf!
    !

    View full-size slide

  67. QUESTIONS?
    These slides will be posted to SlideShare & SpeakerDeck.
    SpeakerDeck: http://speakerdeck.com/u/mwillbanks
    Slideshare: http://www.slideshare.net/mwillbanks
    Twitter: mwillbanks
    G+: Mike Willbanks
    IRC (freenode): mwillbanks
    Blog: http://blog.digitalstruct.com
    GitHub: https://github.com/mwillbanks

    View full-size slide