Messaging Patterns With RabbitMQ

Transcript

1. Messaging Patterns With RabbitMQ Álvaro Videla - Liip AG Tuesday, June 14, 2011
2. About Me • Developer at Liip AG • Blog: http://videlalvaro.github.com/ • Twitter: @old_sound Tuesday, June 14, 2011
3. About Me Co-authoring RabbitMQ in Action http://bit.ly/rabbitmq Tuesday, June 14, 2011
4. Why Do I need Messaging? Tuesday, June 14, 2011
5. An Example Tuesday, June 14, 2011
6. Implement a Photo Gallery Tuesday, June 14, 2011
7. Two Parts: Tuesday, June 14, 2011
8. Pretty Simple Tuesday, June 14, 2011
9. ‘Till new requirements arrive Tuesday, June 14, 2011
10. The Product Owner Tuesday, June 14, 2011
11. Can we also notify the user friends when she uploads a new image? Tuesday, June 14, 2011
12. Can we also notify the user friends when she uploads a new image? I forgot to mention we need it for tomorrow… Tuesday, June 14, 2011
13. The Social Media Guru Tuesday, June 14, 2011
14. We need to give badges to users for each picture upload Tuesday, June 14, 2011
15. We need to give badges to users for each picture upload and post uploads to Twitter Tuesday, June 14, 2011
16. The Sysadmin Tuesday, June 14, 2011
17. Dumb! You’re delivering full size images! The bandwidth bill has tripled! Tuesday, June 14, 2011
18. Dumb! You’re delivering full size images! The bandwidth bill has tripled! We need this fixed for yesterday! Tuesday, June 14, 2011
19. The Developer in the other team Tuesday, June 14, 2011
20. I need to call your PHP stuff but from Python Tuesday, June 14, 2011
21. I need to call your PHP stuff but from Python And also Java starting next week Tuesday, June 14, 2011
22. The User Tuesday, June 14, 2011
23. I don’t want to wait till your app resizes my image! Tuesday, June 14, 2011
24. You Tuesday, June 14, 2011
25. FML! Tuesday, June 14, 2011
26. Let’s see the code evolution Tuesday, June 14, 2011
27. %% image_controller handle('PUT', "/user/image", ReqData) -> image_handler:do_upload(ReqData:get_file()), ok. First Implementation: Tuesday, June 14, 2011
28. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), ok. Second Implementation: Tuesday, June 14, 2011
29. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), ok. Third Implementation: Tuesday, June 14, 2011
30. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), ok. Fourth Implementation: Tuesday, June 14, 2011
31. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), tweet_new_image(User, Image), ok. Final Implementation: Tuesday, June 14, 2011
32. Can our code scale to new requirements? Tuesday, June 14, 2011
33. What if Tuesday, June 14, 2011
34. What if • We need to speed up image conversion Tuesday, June 14, 2011
35. What if • We need to speed up image conversion • User notification has to be sent by email Tuesday, June 14, 2011
36. What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new images Tuesday, June 14, 2011
37. What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new images • Resize in different formats Tuesday, June 14, 2011
38. Can we do better? Tuesday, June 14, 2011
39. Sure. Using messaging Tuesday, June 14, 2011
40. Design Publish / Subscribe Pattern Tuesday, June 14, 2011
41. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: Tuesday, June 14, 2011
42. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). Tuesday, June 14, 2011
43. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). Tuesday, June 14, 2011
44. %% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). %% resizer on('new_image', Msg) -> resize_image(Msg.image). Tuesday, June 14, 2011
45. Second Implementation: Tuesday, June 14, 2011
46. Second Implementation: %% there’s none. Tuesday, June 14, 2011
47. Messaging Tuesday, June 14, 2011
48. Messaging • Share data across processes Tuesday, June 14, 2011
49. Messaging • Share data across processes • Processes can be part of different apps Tuesday, June 14, 2011
50. Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machines Tuesday, June 14, 2011
51. Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machines • Communication is Asynchronous Tuesday, June 14, 2011
52. Main Concepts Tuesday, June 14, 2011
53. Main Concepts • Messages are sent by Producers Tuesday, June 14, 2011
54. Main Concepts • Messages are sent by Producers • Messages are delivered to Consumers Tuesday, June 14, 2011
55. Main Concepts • Messages are sent by Producers • Messages are delivered to Consumers • Messages goes through a Channel Tuesday, June 14, 2011
56. Messaging and RabbitMQ Tuesday, June 14, 2011
57. What is RabbitMQ? Tuesday, June 14, 2011
58. RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial Support • Messaging via AMQP Tuesday, June 14, 2011
59. Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1 Tuesday, June 14, 2011
60. Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, Haskell Tuesday, June 14, 2011
61. AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary Protocol Tuesday, June 14, 2011
62. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html Tuesday, June 14, 2011
63. AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding them Tuesday, June 14, 2011
64. Exchange Types • Fanout • Direct • Topic Tuesday, June 14, 2011
65. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.html Tuesday, June 14, 2011
66. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.html Tuesday, June 14, 2011
67. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.html Tuesday, June 14, 2011
68. Messaging Patterns Tuesday, June 14, 2011
69. There are many messaging patterns http://www.eaipatterns.com/ Tuesday, June 14, 2011
70. Basic Patterns Tuesday, June 14, 2011
71. Competing Consumers How can a messaging client process multiple messages concurrently? Tuesday, June 14, 2011
72. Competing Consumers Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently. Tuesday, June 14, 2011
73. Competing Consumers Tuesday, June 14, 2011
74. Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type = <<"direct">>, durable = true}, #'queue.declare'{queue = Queue, durable = false}, #'queue.bind'{queue = Queue, exchange = Exchange}. publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
75. Consumer Code init_consumer(Exchange, Queue) -> init(Exchange, Queue), #'basic.consume'{queue = Queue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> do_something_with_msg(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
76. Publish/Subscribe How can the sender broadcast an event to all interested receivers? Tuesday, June 14, 2011
77. Publish/Subscribe Send the event on a Publish-Subscribe Channel, which delivers a copy of a particular event to each receiver. Tuesday, June 14, 2011
78. Publish/Subscribe Tuesday, June 14, 2011
79. Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type = <<"fanout">>, %% different type durable = true} %% same as before ... publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
80. Consumer Code A init_consumer(Exchange, ResizeImageQueue) -> init(Exchange, ResizeImageQueue), #'basic.consume'{queue = ResizeImageQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> resize_message(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
81. Consumer Code B init_consumer(Exchange, NotifyFriendsQueue) -> init(Exchange, NotifyFriendsQueue), #'basic.consume'{queue = NotifyFriendsQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> notify_friends(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
82. Consumer Code C init_consumer(Exchange, LogImageUpload) -> init(Exchange, LogImageUpload), #'basic.consume'{queue = LogImageUpload}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> log_image_upload(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011
83. Request/Reply When an application sends a message, how can it get a response from the receiver? Tuesday, June 14, 2011
84. Request/Reply Send a pair of Request-Reply messages, each on its own channel. Tuesday, June 14, 2011
85. Request/Reply Tuesday, June 14, 2011
86. Return Address How does a replier know where to send the reply? Tuesday, June 14, 2011
87. Return Address The request message should contain a Return Address that indicates where to send the reply message. Tuesday, June 14, 2011
88. Return Address Tuesday, June 14, 2011
89. Correlation Identifier How does a requestor that has received a reply know which request this is the reply for? Tuesday, June 14, 2011
90. Correlation Identifier Each reply message should contain a Correlation Identifier, a unique identifier that indicates which request message this reply is for. Tuesday, June 14, 2011
91. Correlation Identifier Tuesday, June 14, 2011
92. Putting it all together Tuesday, June 14, 2011
93. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. Tuesday, June 14, 2011
94. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011
95. RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, do_something_with_reply(Payload). Tuesday, June 14, 2011
96. RPC Server on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, ReplyTo = Props.reply_to, Reply = process_request(Payload), NewProps = #'P_basic'{correlation_id = CorrelationId}, publish("", %% anonymous exchange #amqp_msg{props = NewProps, payload = Reply}, ReplyTo). %% routing key Tuesday, June 14, 2011
97. Advanced Patterns Tuesday, June 14, 2011
98. Control Bus How can we effectively administer a messaging system that is distributed across multiple platforms and a wide geographic area? Tuesday, June 14, 2011
99. Control Bus Use a Control Bus to manage an enterprise integration system. Tuesday, June 14, 2011
100. Control Bus • Send Configuration Messages • Start/Stop Services • Inject Test Messages • Collect Statistics Tuesday, June 14, 2011
101. Control Bus Tuesday, June 14, 2011
102. Control Bus Make Services “Control Bus” Enabled Tuesday, June 14, 2011
103. Detour How can you route a message through intermediate steps to perform validation, testing or debugging functions? Tuesday, June 14, 2011
104. Detour Construct a Detour with a context-based router controlled via the Control Bus. In one state the router routes incoming messages through additional steps while in the other it routes messages directly to the destination channel. Tuesday, June 14, 2011
105. Detour Tuesday, June 14, 2011
106. Wire Tap How do you inspect messages that travel on a point-to-point channel? Tuesday, June 14, 2011
107. Wire Tap Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel. Tuesday, June 14, 2011
108. Wire Tap Tuesday, June 14, 2011
109. Smart Proxy How can you track messages on a service that publishes reply messages to the Return Address specified by the requestor? Tuesday, June 14, 2011
110. Smart Proxy Use a Smart Proxy to store the Return Address supplied by the original requestor and replace it with the address of the Smart Proxy. When the service sends the reply message route it to the original Return Address. Tuesday, June 14, 2011
111. Smart Proxy Tuesday, June 14, 2011
112. Credits Pattern graphics and description taken from: http://www.eaipatterns.com/ Tuesday, June 14, 2011
113. Thanks! @old_sound http://vimeo.com/user1169087 http://www.slideshare.net/old_sound Tuesday, June 14, 2011