ZendCon 2012 - Message Queues : A Primer

Message queues are everywhere and with the many popular options it can be hard to figure out which one you should use or where you might want to use a message queue. This talk will go through each of the standardized protocols; take a look at different message queues in action with PHP and give you an idea which one you might want to utilize for your next project.

Transcript

1. Message Queues : A
   Primer
   Mike Willbanks | Barnes & Noble
   

   View Slide

2. 2
   •  Talk
    Slides will be posted after the talk.
   •  Me
    Sr. Web Architect Manager at NOOK Developer
    Open Source Contributor
    Where you can find me:
   •  Twitter: mwillbanks G+: Mike Willbanks
   •  IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com
   •  GitHub: https://github.com/mwillbanks
   Housekeeping…
   

   View Slide

3. 3
   • Message Queues?
   • Protocols and Software
   • Picking a Message Queue
   • Best Practices
   Agenda
   

   View Slide

4. Message Queues?
   

   View Slide

5. 5
   A Definition
   “Message queues and mailboxes are software-engineering
   components used for interprocess communication, or for inter-
   thread communication within the same process. They use a
   queue for messaging – the passing of control or of content.”
   

   View Slide

6. 6
   What is messaging?
   “Messaging describes the sending and receiving of data (in the
   form of messages) between systems. Messages are exchanged
   between programs or applications, similar to the way people
   communicate by email but with guarantees on delivery, speed,
   security and the absence of spam.”
   

   View Slide

7. 7
   General Anatomy
   Task
   Producer Consumer
   Messages
   Messages
   Messages
   Producer creates a message and pushes it to the queue; the
   consumer reads from the queue and processes the message.
   

   View Slide

8. 8
   l 
   Pub/Sub
   l 
   FIFO buffer
   l 
   Push / Pull
   l 
   A way to communicate between applications / systems.
   l 
   A way to decouple components.
   l 
   A way to offload work.
   Describing Message Queues
   

   View Slide

9. 9
   l 
   Offload Heavy Work
   l 
   Integration with Legacy Systems
   l 
   Asynchronous Processing
   l 
   Parallel Processing
   l 
   Process consistency
   l 
   Scalability
   Why to use a Message Queue
   

   View Slide

10. 10
    l 
    “Write programs that work together.”
    l 
    “Do it in the background.”
    Unix Foundations
    

    View Slide

11. 11
    Why it matters
    l 
    Web systems need to be geared to run things
    asynchronously.
    l 
    Distribution of load
    l 
    System integrity
    

    View Slide

12. 12
    Enterprise Integration Patterns
    Understanding messaging patterns will greatly
    assist you in building out great systems for
    messaging.
    http://www.eaipatterns.com/
    

    View Slide

13. Common Messaging Patterns
    

    View Slide

14. 14
    Publish-Subscribe
    

    View Slide

15. 15
    Request-Reply
    

    View Slide

16. 16
    Message Router
    

    View Slide

17. Message queue examples
    •  You’ve seen them before; they are used in most
    applications to help them scale.
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. Generic Use Cases
    •  When to make use of message queues in case you’re
    wondering.
    

    View Slide

22. 22
    Notifications
    Email, SMS, Push Messaging….
    

    View Slide

23. 23
    Photo Processing
    Thumbnails, Resizing, Watermarking, Converting…
    

    View Slide

24. 24
    Video Processing
    Resampling, Audio Overlay, Type Conversion…
    

    View Slide

25. 25
    Analytics
    Web Server Logs, Log Aggregation, PHP Errors, etc.
    

    View Slide

26. 26
    Integrations
    Save local first; push second.
    

    View Slide

27. Protocols
    •  AMQP
    •  STOMP
    •  XMPP
    •  Vendor Specific
    

    View Slide

28. 28
    AMQP
    Advanced Message Queuing Protocol
    

    View Slide

29. 29
    l 
    AMQP Working Group (Community and
    Vendor)
    l 
    Platform agnostic protocol.
    l 
    Completely open, interoperable and broadly
    applicable.
    l 
    Many severs available and many client
    libraries.
    Overview of AMQP
    

    View Slide

30. 30
    How it Works
    Producer, Exchange, Queue, Consumer
    

    View Slide

31. 31
    l 
    AMQP utilizes exchanges, queues and
    bindings.
    l 
    An exchange are routers with routing tables.
    l 
    A binding defines the routing rules.
    l 
    A queue is where the messages wait for a
    consumer.
    How it Works
    

    View Slide

32. 32
    Exchanges
    

    View Slide

33. 33
    l 
    Fanout Exchange
    l 
    No routing keys involved. Any message that is sent to the
    exchange is sent to all queues bound to that exchange.
    l 
    Direct Exchange
    l 
    Routing keys involved. A queue binds to the exchange to
    request messages that match a routing key exactly.
    l 
    Topic Exchange
    l 
    Routing keys involved. A queue binds to the exchange to
    request messages that match a routing key pattern.
    Understanding Exchanges
    

    View Slide

34. 34
    Implementations
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    qpid.apache.org
    Long standing project; apache foundation.
    www.openamq.org
    Long standing project; ZeroMQ partner, no news since
    2009.
    

    View Slide

35. 35
    l 
    An exchange, queue and bindings must be defined
    first. Publishing can then commence after.
    l 
    Create the queue
    l 
    Create the exchange
    l 
    Bind to the queue.
    Building a Queue
    

    View Slide

36. 36
    l 
    Default behavior is no persistence.
    l 
    How important are the messages?
    l 
    Just about all items have a level of persistence if you
    would like them to survive on reboot.
    l 
    Mark exchanges, queues and messages as DURABLE.
    Persistence?
    

    View Slide

37. 37
    l 
    Extension compatible with AMQP specification
    0-9-1.
    l 
    pecl install amqp
    PECL AMQP
    

    View Slide

38. 38
    AMQP Client
    

    View Slide

39. 39
    AMQP Worker
    

    View Slide

40. 40
    STOMP
    Simple (or Streaming) Text Orientated Messaging Protocol
    

    View Slide

41. 41
    Overview
    l 
    Simple protocol
    l 
    Behaviors follow very simple commands.
    l 
    Most message queues can communicate over
    STOMP.
    

    View Slide

42. 42
    How It Works
    l 
    When you send in a message, you tell it which queue to
    go to.
    l 
    When you subscribe you request a queue.
    Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack
    

    View Slide

43. 43
    Sever Implementations
    activemq.apache.org
    One of the oldest message queues existing;
    a apache foundation project
    activemq.apache.org/apollo
    Next generation ActiveMQ
    www.rabbitmq.com
    Very popular and common message queue owned by
    VMware.
    www.jboss.org/hornetq
    Supported by Red Hat Middle Ware division, picking
    up steam.
    

    View Slide

44. 44
    l 
    pecl install stomp
    l 
    That was easy J
    PECL Stomp
    

    View Slide

45. 45
    STOMP Client
    

    View Slide

46. 46
    STOMP Worker
    

    View Slide

47. 47
    XMPP
    Extensible Messaging and Presence Protocol
    (Although not really a “Message Queue”)
    

    View Slide

48. 48
    Overview
    l 
    Best for real-time data.
    l 
    Leveraging pub/sub can turn it into more of a generic message
    system.
    l 
    Multiple libraries
    l 
    JAXL - https://github.com/abhinavsingh/JAXL
    l 
    Xmpp - https://github.com/alexmace/Xmpp
    

    View Slide

49. 49
    XEP-0060: Publish-Subscribe
    l 
    Specification for implementing Publish Subscribe
    models.
    l 
    Extension to the original XMPP specification.
    Publish
    Subscribe
    Sub1 Sub2 Sub3
    to, id,
    message
    from, to, id,
    message
    

    View Slide

50. 50
    Publish
    

    View Slide

51. 51
    Message
    

    View Slide

52. Special Message Queues
    •  Various others; including Gearman, ZeroMQ, etc.
    

    View Slide

53. 53
    l 
    There are job servers available that are more
    flexible or more specific.
    l 
    Extreme Flexibility
    l 
    Job Severs
    l 
    Cloud Messaging
    Overview
    

    View Slide

54. 54
    ZeroMQ
    The ultimate in message queue flexibility.
    Socket library that acts as a concurrency framework.
    Contains a PHP extension.
    

    View Slide

55. 55
    Several Types of Queues
    Request / Reply
    Publish / Subscribe
    Parallel Pipeline
    Fair Queuing
    

    View Slide

56. 56
    ZeroMQ Client Example
    

    View Slide

57. 57
    ZeroMQ Worker Example
    

    View Slide

58. 58
    l 
    Application framework for farming out work.
    l 
    Job sever for asynchronous or synchronous messages.
    Gearman
    

    View Slide

59. 59
    l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist if enabled; default is in-memory.
    Gearman
    

    View Slide

60. 60
    Gearman Client Example
    

    View Slide

61. 61
    Gearman Worker Example
    

    View Slide

62. 62
    l 
    Asynchronous Job Queue
    l 
    Good scheduling system (aka delays).
    l 
    User land PHP clients.
    Beanstalkd
    

    View Slide

63. 63
    l 
    Pass a job to the job server
    l 
    Worker receives the job and processes
    l 
    Ability to persist to binlog; default is in-memory
    Beanstalkd
    Application Code
    Beanstalkd Client API
    (Pheanstalk)
    Beanstalkd Server
    Beanstalkd Worker API
    (Pheanstalk)
    Worker Application Code
    

    View Slide

64. 64
    Pheanstalk Client
    

    View Slide

65. 65
    Pheanstalk Worker
    

    View Slide

66. Picking a Message Queue
    •  What you want to look for.
    

    View Slide

67. 67
    The Trifecta
    Scale
    Performance Durability
    Messages
    Horizontal?
    Vertical?
    In-Memory?
    Persistence?
    Replication?
    Raw Speed?
    # Messages?
    Messages Messages
    Messages
    Messages
    Messages
    Messages Messages
    Messages
    

    View Slide

68. 68
    Standards
    •  A recognized standard?
     AMQP? STOMP? XMPP?
    •  How many developers?
     Will an unfortunate event kill off the product?
    •  Undocumented protocol?
     Forget about it!
    

    View Slide

69. 69
    Delivery Policies
    Store Message Saved?
    Process Message ACK?
    Receive Message TTL?
    

    View Slide

70. 70
    Message Policies
    •  Handling in-active messages?
    •  Messages that failed processing?
    •  Time to live?
    •  Retry?
    •  Ability to check message status?
    

    View Slide

71. 71
    Routing Policies
    •  Fanout?
    •  Direct?
    •  Topic?
    •  Broadcast?
    •  Etc?
    

    View Slide

72. 72
    Security Policies
    •  Require authentication?
     LDAP/AD integration?
    •  Connection Restrictions?
    •  Require logging?
     Compliance
     Intrusion Detection
    

    View Slide

73. Best Practices
    •  Rules of the road my friends…
    

    View Slide

74. 74
    Messages
    •  Formatting
     JSON or XML are great options.
    •  Please no serialized PHP objects.
    •  Message Size
     Only as large as necessary.
    •  Don’t send a binary object through the queue.
    

    View Slide

75. 75
    Workers
    •  Dumb as possible!
     Single message type
    •  Easier to scale.
     Single operation
    •  Easy to debug and far more flexible.
    

    View Slide

76. 76
    PHP Daemons
    •  Prevent Memory Leaks
     Detection
     Cycle Workers
    •  Handle Signals
     Properly shutdown!
     Watch out for OS service kills
    •  Sleeping is good J
    

    View Slide

77. View Slide

78. 78
    Environment Integration
    Web Server
    Database
    Worker Server
    Message Queue
    Web Server -> Message Queue
    Worker Server -> Message Queue
    

    View Slide

79. 79
    High Availability
    Web Server
    Database
    Worker Server
    Message Queue
    Message Queue
    Worker Server
    Send to 1
    Insert multiple message queue servers + multiple worker
    nodes. Each worker node can connect to as many message
    queue servers as necessary.
    

    View Slide

80. 80
    Supervisord
    Process monitoring, logging and more!
    

    View Slide

81. 81
    Installing Supervisord
    • Requires: python-setuptools
    sudo easy_install supervisor!
    • Generate Configuration
    sudo echo_supervisord_conf > /etc/supervisord.conf!
    !
    

    View Slide

82. View Slide

83. 83
    •  These slides will be posted to SlideShare & SpeakerDeck.
      SpeakerDeck: http://speakerdeck.com/u/mwillbanks
      Slideshare: http://www.slideshare.net/mwillbanks
      Twitter: mwillbanks
      G+: Mike Willbanks
      IRC (freenode): mwillbanks
      Blog: http://blog.digitalstruct.com
      GitHub: https://github.com/mwillbanks
    Questions?
    

    View Slide