Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Messaging Patterns With RabbitMQ

B3eb24dc767e178a2c7d67f1ee1af11f?s=47 Alvaro Videla
September 26, 2011

Messaging Patterns With RabbitMQ

B3eb24dc767e178a2c7d67f1ee1af11f?s=128

Alvaro Videla

September 26, 2011
Tweet

Transcript

  1. Messaging Patterns With RabbitMQ Álvaro Videla - Liip AG Tuesday,

    June 14, 2011
  2. About Me • Developer at Liip AG • Blog: http://videlalvaro.github.com/

    • Twitter: @old_sound Tuesday, June 14, 2011
  3. About Me Co-authoring RabbitMQ in Action http://bit.ly/rabbitmq Tuesday, June 14,

    2011
  4. Why Do I need Messaging? Tuesday, June 14, 2011

  5. An Example Tuesday, June 14, 2011

  6. Implement a Photo Gallery Tuesday, June 14, 2011

  7. Two Parts: Tuesday, June 14, 2011

  8. Pretty Simple Tuesday, June 14, 2011

  9. ‘Till new requirements arrive Tuesday, June 14, 2011

  10. The Product Owner Tuesday, June 14, 2011

  11. Can we also notify the user friends when she uploads

    a new image? Tuesday, June 14, 2011
  12. Can we also notify the user friends when she uploads

    a new image? I forgot to mention we need it for tomorrow… Tuesday, June 14, 2011
  13. The Social Media Guru Tuesday, June 14, 2011

  14. We need to give badges to users for each picture

    upload Tuesday, June 14, 2011
  15. We need to give badges to users for each picture

    upload and post uploads to Twitter Tuesday, June 14, 2011
  16. The Sysadmin Tuesday, June 14, 2011

  17. Dumb! You’re delivering full size images! The bandwidth bill has

    tripled! Tuesday, June 14, 2011
  18. Dumb! You’re delivering full size images! The bandwidth bill has

    tripled! We need this fixed for yesterday! Tuesday, June 14, 2011
  19. The Developer in the other team Tuesday, June 14, 2011

  20. I need to call your PHP stuff but from Python

    Tuesday, June 14, 2011
  21. I need to call your PHP stuff but from Python

    And also Java starting next week Tuesday, June 14, 2011
  22. The User Tuesday, June 14, 2011

  23. I don’t want to wait till your app resizes my

    image! Tuesday, June 14, 2011
  24. You Tuesday, June 14, 2011

  25. FML! Tuesday, June 14, 2011

  26. Let’s see the code evolution Tuesday, June 14, 2011

  27. %% image_controller handle('PUT', "/user/image", ReqData) -> image_handler:do_upload(ReqData:get_file()), ok. First Implementation:

    Tuesday, June 14, 2011
  28. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    resize_image(Image), ok. Second Implementation: Tuesday, June 14, 2011
  29. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    resize_image(Image), notify_friends(ReqData:get_user()), ok. Third Implementation: Tuesday, June 14, 2011
  30. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), ok. Fourth Implementation: Tuesday, June 14, 2011
  31. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), tweet_new_image(User, Image), ok. Final Implementation: Tuesday, June 14, 2011
  32. Can our code scale to new requirements? Tuesday, June 14,

    2011
  33. What if Tuesday, June 14, 2011

  34. What if • We need to speed up image conversion

    Tuesday, June 14, 2011
  35. What if • We need to speed up image conversion

    • User notification has to be sent by email Tuesday, June 14, 2011
  36. What if • We need to speed up image conversion

    • User notification has to be sent by email • Stop tweeting about new images Tuesday, June 14, 2011
  37. What if • We need to speed up image conversion

    • User notification has to be sent by email • Stop tweeting about new images • Resize in different formats Tuesday, June 14, 2011
  38. Can we do better? Tuesday, June 14, 2011

  39. Sure. Using messaging Tuesday, June 14, 2011

  40. Design Publish / Subscribe Pattern Tuesday, June 14, 2011

  41. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: Tuesday, June 14, 2011
  42. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). Tuesday, June 14, 2011
  43. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). Tuesday, June 14, 2011
  44. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()),

    Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). %% resizer on('new_image', Msg) -> resize_image(Msg.image). Tuesday, June 14, 2011
  45. Second Implementation: Tuesday, June 14, 2011

  46. Second Implementation: %% there’s none. Tuesday, June 14, 2011

  47. Messaging Tuesday, June 14, 2011

  48. Messaging • Share data across processes Tuesday, June 14, 2011

  49. Messaging • Share data across processes • Processes can be

    part of different apps Tuesday, June 14, 2011
  50. Messaging • Share data across processes • Processes can be

    part of different apps • Apps can live in different machines Tuesday, June 14, 2011
  51. Messaging • Share data across processes • Processes can be

    part of different apps • Apps can live in different machines • Communication is Asynchronous Tuesday, June 14, 2011
  52. Main Concepts Tuesday, June 14, 2011

  53. Main Concepts • Messages are sent by Producers Tuesday, June

    14, 2011
  54. Main Concepts • Messages are sent by Producers • Messages

    are delivered to Consumers Tuesday, June 14, 2011
  55. Main Concepts • Messages are sent by Producers • Messages

    are delivered to Consumers • Messages goes through a Channel Tuesday, June 14, 2011
  56. Messaging and RabbitMQ Tuesday, June 14, 2011

  57. What is RabbitMQ? Tuesday, June 14, 2011

  58. RabbitMQ • Enterprise Messaging System • Open Source MPL •

    Written in Erlang/OTP • Commercial Support • Messaging via AMQP Tuesday, June 14, 2011
  59. Features • Reliable and High Scalable • Easy To install

    • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1 Tuesday, June 14, 2011
  60. Client Libraries • Java • .NET/C# • Erlang • Ruby,

    Python, PHP, Perl, AS3, Lisp, Scala, Clojure, Haskell Tuesday, June 14, 2011
  61. AMQP • Advanced Message Queuing Protocol • Suits Interoperability •

    Completely Open Protocol • Binary Protocol Tuesday, June 14, 2011
  62. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html Tuesday, June 14, 2011

  63. AMQP Model • Exchanges • Message Queues • Bindings •

    Rules for binding them Tuesday, June 14, 2011
  64. Exchange Types • Fanout • Direct • Topic Tuesday, June

    14, 2011
  65. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.html Tuesday, June 14, 2011

  66. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.html Tuesday, June 14, 2011

  67. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.html Tuesday, June 14, 2011

  68. Messaging Patterns Tuesday, June 14, 2011

  69. There are many messaging patterns http://www.eaipatterns.com/ Tuesday, June 14, 2011

  70. Basic Patterns Tuesday, June 14, 2011

  71. Competing Consumers How can a messaging client process multiple messages

    concurrently? Tuesday, June 14, 2011
  72. Competing Consumers Create multiple Competing Consumers on a single channel

    so that the consumers can process multiple messages concurrently. Tuesday, June 14, 2011
  73. Competing Consumers Tuesday, June 14, 2011

  74. Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type =

    <<"direct">>, durable = true}, #'queue.declare'{queue = Queue, durable = false}, #'queue.bind'{queue = Queue, exchange = Exchange}. publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
  75. Consumer Code init_consumer(Exchange, Queue) -> init(Exchange, Queue), #'basic.consume'{queue = Queue}.

    on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> do_something_with_msg(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
  76. Publish/Subscribe How can the sender broadcast an event to all

    interested receivers? Tuesday, June 14, 2011
  77. Publish/Subscribe Send the event on a Publish-Subscribe Channel, which delivers

    a copy of a particular event to each receiver. Tuesday, June 14, 2011
  78. Publish/Subscribe Tuesday, June 14, 2011

  79. Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type =

    <<"fanout">>, %% different type durable = true} %% same as before ... publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
  80. Consumer Code A init_consumer(Exchange, ResizeImageQueue) -> init(Exchange, ResizeImageQueue), #'basic.consume'{queue =

    ResizeImageQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> resize_message(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
  81. Consumer Code B init_consumer(Exchange, NotifyFriendsQueue) -> init(Exchange, NotifyFriendsQueue), #'basic.consume'{queue =

    NotifyFriendsQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> notify_friends(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
  82. Consumer Code C init_consumer(Exchange, LogImageUpload) -> init(Exchange, LogImageUpload), #'basic.consume'{queue =

    LogImageUpload}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> log_image_upload(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
  83. Request/Reply When an application sends a message, how can it

    get a response from the receiver? Tuesday, June 14, 2011
  84. Request/Reply Send a pair of Request-Reply messages, each on its

    own channel. Tuesday, June 14, 2011
  85. Request/Reply Tuesday, June 14, 2011

  86. Return Address How does a replier know where to send

    the reply? Tuesday, June 14, 2011
  87. Return Address The request message should contain a Return Address

    that indicates where to send the reply message. Tuesday, June 14, 2011
  88. Return Address Tuesday, June 14, 2011

  89. Correlation Identifier How does a requestor that has received a

    reply know which request this is the reply for? Tuesday, June 14, 2011
  90. Correlation Identifier Each reply message should contain a Correlation Identifier,

    a unique identifier that indicates which request message this reply is for. Tuesday, June 14, 2011
  91. Correlation Identifier Tuesday, June 14, 2011

  92. Putting it all together Tuesday, June 14, 2011

  93. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive =

    true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. Tuesday, June 14, 2011
  94. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive =

    true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
  95. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive =

    true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, do_something_with_reply(Payload). Tuesday, June 14, 2011
  96. RPC Server on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) ->

    CorrelationId = Props.correlation_id, ReplyTo = Props.reply_to, Reply = process_request(Payload), NewProps = #'P_basic'{correlation_id = CorrelationId}, publish("", %% anonymous exchange #amqp_msg{props = NewProps, payload = Reply}, ReplyTo). %% routing key Tuesday, June 14, 2011
  97. Advanced Patterns Tuesday, June 14, 2011

  98. Control Bus How can we effectively administer a messaging system

    that is distributed across multiple platforms and a wide geographic area? Tuesday, June 14, 2011
  99. Control Bus Use a Control Bus to manage an enterprise

    integration system. Tuesday, June 14, 2011
  100. Control Bus • Send Configuration Messages • Start/Stop Services •

    Inject Test Messages • Collect Statistics Tuesday, June 14, 2011
  101. Control Bus Tuesday, June 14, 2011

  102. Control Bus Make Services “Control Bus” Enabled Tuesday, June 14,

    2011
  103. Detour How can you route a message through intermediate steps

    to perform validation, testing or debugging functions? Tuesday, June 14, 2011
  104. Detour Construct a Detour with a context-based router controlled via

    the Control Bus. In one state the router routes incoming messages through additional steps while in the other it routes messages directly to the destination channel. Tuesday, June 14, 2011
  105. Detour Tuesday, June 14, 2011

  106. Wire Tap How do you inspect messages that travel on

    a point-to-point channel? Tuesday, June 14, 2011
  107. Wire Tap Insert a simple Recipient List into the channel

    that publishes each incoming message to the main channel and a secondary channel. Tuesday, June 14, 2011
  108. Wire Tap Tuesday, June 14, 2011

  109. Smart Proxy How can you track messages on a service

    that publishes reply messages to the Return Address specified by the requestor? Tuesday, June 14, 2011
  110. Smart Proxy Use a Smart Proxy to store the Return

    Address supplied by the original requestor and replace it with the address of the Smart Proxy. When the service sends the reply message route it to the original Return Address. Tuesday, June 14, 2011
  111. Smart Proxy Tuesday, June 14, 2011

  112. Credits Pattern graphics and description taken from: http://www.eaipatterns.com/ Tuesday, June

    14, 2011
  113. Thanks! @old_sound http://vimeo.com/user1169087 http://www.slideshare.net/old_sound Tuesday, June 14, 2011