International PHP Conference - Fall 2012 : Message Queues - A Primer

Transcript

1. Mike Willbanks | Barnes & Noble Message Queues : A

   Primer

2. Housekeeping… •  Talk –  Slides will be posted after the

   talk. •  Me –  Sr. Web Architect Manager at NOOK Developer –  Prior MNPHP Organizer –  Open Source Contributor –  Where you can find me: •  Twitter: mwillbanks G+: Mike Willbanks •  IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com •  GitHub: https://github.com/mwillbanks

3. Agenda •  Message Queues? •  Protocols and Software •  Picking

   a Message Queue •  Best Practices

4. A Definition “Message queues and mailboxes are software-engineering components used

   for interprocess communication, or for inter- thread communication within the same process. They use a queue for messaging – the passing of control or of content.”

5. What is messaging? “Messaging describes the sending and receiving of

   data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”

6. General Anatomy Task Producer Consumer Messages Messages Messages Producer creates

   a message and pushes it to the queue; the consumer reads from the queue and processes the message.

7. l  Pub/Sub l  FIFO buffer l  Push / Pull l 

   A way to communicate between applications / systems. l  A way to decouple components. l  A way to offload work. Describing Message Queues

8. l  Offload Heavy Work l  Integration with Legacy Systems l 

   Asynchronous Processing l  Parallel Processing l  Process consistency l  Scalability Why to use a Message Queue

9. Unix Foundations l  “Write programs that work together.” l  “Do

   it in the background.”

10. l  Web systems need to be geared to run things

    asynchronously. l  Distribution of load l  System integrity Why it matters

11. MESSAGE QUEUE EXAMPLES You’ve seen them before; they are used

    in most applications to help them scale.
12. None
13. None
14. None

15. GENERIC USE CASES When to make use of message queues

    in case you’re wondering.

16. Notifications Email, SMS, Push Messaging….

17. Photo Processing Thumbnails, Resizing, Watermarking, Converting…

18. Video Processing Resampling, Audio Overlay, Type Conversion…

19. Analytics Web Server Logs, Log Aggregation, PHP Errors, etc.

20. Integrations Save local first; push second.

21. PROTOCOLS AMQP STOMP XMPP Vendor Specific

22. AMQP Advanced Message Queuing Protocol

23. l  AMQP Working Group (Community and Vendor) l  Platform agnostic

    protocol. l  Completely open, interoperable and broadly applicable. l  Many severs available and many client libraries. Overview of AMQP

24. How it Works Producer, Exchange, Queue, Consumer

25. l  AMQP utilizes exchanges, queues and bindings. l  An exchange

    are routers with routing tables. l  A binding defines the routing rules. l  A queue is where the messages wait for a consumer. How it Works

26. Exchanges

27. l  Fanout Exchange l  No routing keys involved. Any message

    that is sent to the exchange is sent to all queues bound to that exchange. l  Direct Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly. l  Topic Exchange l  Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern. Understanding Exchanges

28. Implementations www.rabbitmq.com Very popular and common message queue owned by

    VMware. qpid.apache.org Long standing project; apache foundation. www.openamq.org Long standing project; ZeroMQ partner, no news since 2009.

29. l  An exchange, queue and bindings must be defined first.

    Publishing can then commence after. l  Create the queue l  Create the exchange l  Bind to the queue. Building a Queue

30. l  Default behavior is no persistence. l  How important are

    the messages? l  Just about all items have a level of persistence if you would like them to survive on reboot. l  Mark exchanges, queues and messages as DURABLE. Persistence?

31. l  Extension compatible with AMQP specification 0-9-1. l  pecl install

    amqp PECL AMQP

32. AMQP Client

33. AMQP Worker

34. STOMP Simple (or Streaming) Text Orientated Messaging Protocol

35. Overview l  Simple protocol l  Behaviors follow very simple commands.

    l  Most message queues can communicate over STOMP.

36. How It Works l  When you send in a message,

    you tell it which queue to go to. l  When you subscribe you request a queue. Connect Send Disconnect /queue/ msg P H P S T O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack

37. Sever Implementations activemq.apache.org One of the oldest message queues existing;

    a apache foundation project activemq.apache.org/apollo Next generation ActiveMQ www.rabbitmq.com Very popular and common message queue owned by VMware. www.jboss.org/hornetq Supported by Red Hat Middle Ware division, picking up steam.

38. l  pecl install stomp l  That was easy J PECL

    Stomp

39. STOMP Client

40. STOMP Worker

41. XMPP Extensible Messaging and Presence Protocol (Although not really a

    “Message Queue”)

42. Overview l  Best for real-time data. l  Leveraging pub/sub can

    turn it into more of a generic message system. l  Multiple libraries l  JAXL - https://github.com/abhinavsingh/JAXL l  Xmpp - https://github.com/alexmace/Xmpp

43. XEP-0060: Publish-Subscribe l  Specification for implementing Publish Subscribe models. l 

    Extension to the original XMPP specification. Publish Subscribe Sub1 Sub2 Sub3 to, id, message from, to, id, message

44. Publish

45. Message

46. SPECIAL MESSAGE QUEUES Various others; including Gearman, ZeroMQ, etc.

47. l  There are job servers available that are more flexible

    or more specific. l  Extreme Flexibility l  Job Severs l  Cloud Messaging Overview

48. ZeroMQ The ultimate in message queue flexibility. Socket library that

    acts as a concurrency framework. Contains a PHP extension.

49. Several Types of Queues Request / Reply Publish / Subscribe

    Parallel Pipeline Fair Queuing And more…

50. ZeroMQ Client Example

51. ZeroMQ Worker Example

52. l  Application framework for farming out work. l  Job sever

    for asynchronous or synchronous messages. Gearman

53. l  Pass a job to the job server l  Worker

    receives the job and processes l  Ability to persist if enabled; default is in-memory. Gearman

54. Gearman Client Example

55. Gearman Worker Example

56. l  Asynchronous Job Queue l  Good scheduling system (aka delays).

    l  User land PHP clients. Beanstalkd

57. l  Pass a job to the job server l  Worker

    receives the job and processes l  Ability to persist to binlog; default is in-memory Beanstalkd Application Code Beanstalkd Client API (Pheanstalk) Beanstalkd Server Beanstalkd Worker API (Pheanstalk) Worker Application Code

58. Pheanstalk Client

59. Pheanstalk Worker

60. PICKING A MESSAGE QUEUE What you want to look for.

61. The Trifecta Scale Performance Durability Messages Horizontal? Vertical? In-Memory? Persistence?

    Replication? Raw Speed? # Messages? Messages Messages Messages Messages Messages Messages Messages Messages

62. Standards •  A recognized standard? – AMQP? STOMP? XMPP? •  How

    many developers? – Will an unfortunate event kill off the product? •  Undocumented protocol? – Forget about it!

63. Delivery Policies Store Message Saved? Process Message ACK? Receive Message

    TTL?

64. Message Policies •  Handling in-active messages? •  Messages that failed

    processing? •  Time to live? •  Retry? •  Ability to check message status?

65. Routing Policies •  Fanout? •  Direct? •  Topic? •  Broadcast?

    •  Etc?

66. Security Policies •  Require authentication? – LDAP/AD integration? •  Connection Restrictions?

    •  Require logging? – Compliance – Intrusion Detection

67. BEST PRACTICES Rules of the road my friends…

68. Messages •  Formatting – JSON or XML are great options. • 

    Please no serialized PHP objects. •  Message Size – Only as large as necessary. •  Don’t send a binary object through the queue.

69. Workers •  Dumb as possible! – Single message type •  Easier

    to scale. – Single operation •  Easy to debug and far more flexible.

70. PHP Daemons •  Prevent Memory Leaks – Detection – Cycle Workers • 

    Handle Signals – Properly shutdown! – Watch out for OS service kills •  Sleeping is good J
71. None

72. Environment Integration Web Server Database Worker Server Message Queue Web

    Server -> Message Queue Worker Server -> Message Queue

73. High Availability Web Server Database Worker Server Message Queue Message

    Queue Worker Server Send to 1 Insert multiple message queue servers + multiple worker nodes. Each worker node can connect to as many message queue servers as necessary.

74. Supervisord Process monitoring, logging and more!

75. Installing Supervisord •  Requires: python-setuptools sudo easy_install supervisor! •  Generate

    Configuration sudo echo_supervisord_conf > /etc/ supervisord.conf! !
76. None

77. QUESTIONS? These slides will be posted to SlideShare & SpeakerDeck.

    SpeakerDeck: http://speakerdeck.com/u/mwillbanks Slideshare: http://www.slideshare.net/mwillbanks Twitter: mwillbanks G+: Mike Willbanks IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com GitHub: https://github.com/mwillbanks