Tweet
Tweet

Slide 1

Slide 1 text

Messaging Patterns With RabbitMQ Álvaro Videla - Liip AG Tuesday, June 14, 2011

Slide 2

Slide 2 text

About Me • Developer at Liip AG • Blog: http://videlalvaro.github.com/ • Twitter: @old_sound Tuesday, June 14, 2011

Slide 3

Slide 3 text

About Me Co-authoring RabbitMQ in Action http://bit.ly/rabbitmq Tuesday, June 14, 2011

Slide 4

Slide 4 text

Why Do I need Messaging? Tuesday, June 14, 2011

Slide 5

Slide 5 text

An Example Tuesday, June 14, 2011

Slide 6

Slide 6 text

Implement a Photo Gallery Tuesday, June 14, 2011

Slide 7

Slide 7 text

Two Parts: Tuesday, June 14, 2011

Slide 8

Slide 8 text

Pretty Simple Tuesday, June 14, 2011

Slide 9

Slide 9 text

‘Till new requirements arrive Tuesday, June 14, 2011

Slide 10

Slide 10 text

The Product Owner Tuesday, June 14, 2011

Slide 11

Slide 11 text

Can we also notify the user friends when she uploads a new image? Tuesday, June 14, 2011

Slide 12

Slide 12 text

Can we also notify the user friends when she uploads a new image? I forgot to mention we need it for tomorrow… Tuesday, June 14, 2011

Slide 13

Slide 13 text

The Social Media Guru Tuesday, June 14, 2011

Slide 14

Slide 14 text

We need to give badges to users for each picture upload Tuesday, June 14, 2011

Slide 15

Slide 15 text

We need to give badges to users for each picture upload and post uploads to Twitter Tuesday, June 14, 2011

Slide 16

Slide 16 text

The Sysadmin Tuesday, June 14, 2011

Slide 17

Slide 17 text

Dumb! You’re delivering full size images! The bandwidth bill has tripled! Tuesday, June 14, 2011

Slide 18

Slide 18 text

Dumb! You’re delivering full size images! The bandwidth bill has tripled! We need this fixed for yesterday! Tuesday, June 14, 2011

Slide 19

Slide 19 text

The Developer in the other team Tuesday, June 14, 2011

Slide 20

Slide 20 text

I need to call your PHP stuff but from Python Tuesday, June 14, 2011

Slide 21

Slide 21 text

I need to call your PHP stuff but from Python And also Java starting next week Tuesday, June 14, 2011

Slide 22

Slide 22 text

The User Tuesday, June 14, 2011

Slide 23

Slide 23 text

I don’t want to wait till your app resizes my image! Tuesday, June 14, 2011

Slide 24

Slide 24 text

You Tuesday, June 14, 2011

Slide 25

Slide 25 text

FML! Tuesday, June 14, 2011

Slide 26

Slide 26 text

Let’s see the code evolution Tuesday, June 14, 2011

Slide 27

Slide 27 text

%% image_controller handle('PUT', "/user/image", ReqData) -> image_handler:do_upload(ReqData:get_file()), ok. First Implementation: Tuesday, June 14, 2011

Slide 28

Slide 28 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), ok. Second Implementation: Tuesday, June 14, 2011

Slide 29

Slide 29 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), ok. Third Implementation: Tuesday, June 14, 2011

Slide 30

Slide 30 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), ok. Fourth Implementation: Tuesday, June 14, 2011

Slide 31

Slide 31 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), resize_image(Image), notify_friends(ReqData:get_user()), add_points_to_user(ReqData:get_user()), tweet_new_image(User, Image), ok. Final Implementation: Tuesday, June 14, 2011

Slide 32

Slide 32 text

Can our code scale to new requirements? Tuesday, June 14, 2011

Slide 33

Slide 33 text

What if Tuesday, June 14, 2011

Slide 34

Slide 34 text

What if • We need to speed up image conversion Tuesday, June 14, 2011

Slide 35

Slide 35 text

What if • We need to speed up image conversion • User notification has to be sent by email Tuesday, June 14, 2011

Slide 36

Slide 36 text

What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new images Tuesday, June 14, 2011

Slide 37

Slide 37 text

What if • We need to speed up image conversion • User notification has to be sent by email • Stop tweeting about new images • Resize in different formats Tuesday, June 14, 2011

Slide 38

Slide 38 text

Can we do better? Tuesday, June 14, 2011

Slide 39

Slide 39 text

Sure. Using messaging Tuesday, June 14, 2011

Slide 40

Slide 40 text

Design Publish / Subscribe Pattern Tuesday, June 14, 2011

Slide 41

Slide 41 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: Tuesday, June 14, 2011

Slide 42

Slide 42 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). Tuesday, June 14, 2011

Slide 43

Slide 43 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). Tuesday, June 14, 2011

Slide 44

Slide 44 text

%% image_controller handle('PUT', "/user/image", ReqData) -> {ok, Image} = image_handler:do_upload(ReqData:get_file()), Msg = #msg{user = ReqData:get_user(), image = Image}, publish_message('new_image', Msg). First Implementation: %% friends notifier on('new_image', Msg) -> notify_friends(Msg.user, Msg.image). %% points manager on('new_image', Msg) -> add_points(Msg.user, 'new_image'). %% resizer on('new_image', Msg) -> resize_image(Msg.image). Tuesday, June 14, 2011

Slide 45

Slide 45 text

Second Implementation: Tuesday, June 14, 2011

Slide 46

Slide 46 text

Second Implementation: %% there’s none. Tuesday, June 14, 2011

Slide 47

Slide 47 text

Messaging Tuesday, June 14, 2011

Slide 48

Slide 48 text

Messaging • Share data across processes Tuesday, June 14, 2011

Slide 49

Slide 49 text

Messaging • Share data across processes • Processes can be part of different apps Tuesday, June 14, 2011

Slide 50

Slide 50 text

Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machines Tuesday, June 14, 2011

Slide 51

Slide 51 text

Messaging • Share data across processes • Processes can be part of different apps • Apps can live in different machines • Communication is Asynchronous Tuesday, June 14, 2011

Slide 52

Slide 52 text

Main Concepts Tuesday, June 14, 2011

Slide 53

Slide 53 text

Main Concepts • Messages are sent by Producers Tuesday, June 14, 2011

Slide 54

Slide 54 text

Main Concepts • Messages are sent by Producers • Messages are delivered to Consumers Tuesday, June 14, 2011

Slide 55

Slide 55 text

Main Concepts • Messages are sent by Producers • Messages are delivered to Consumers • Messages goes through a Channel Tuesday, June 14, 2011

Slide 56

Slide 56 text

Messaging and RabbitMQ Tuesday, June 14, 2011

Slide 57

Slide 57 text

What is RabbitMQ? Tuesday, June 14, 2011

Slide 58

Slide 58 text

RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial Support • Messaging via AMQP Tuesday, June 14, 2011

Slide 59

Slide 59 text

Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1 Tuesday, June 14, 2011

Slide 60

Slide 60 text

Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, Haskell Tuesday, June 14, 2011

Slide 61

Slide 61 text

AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary Protocol Tuesday, June 14, 2011

Slide 62

Slide 62 text

Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html Tuesday, June 14, 2011

Slide 63

Slide 63 text

AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding them Tuesday, June 14, 2011

Slide 64

Slide 64 text

Exchange Types • Fanout • Direct • Topic Tuesday, June 14, 2011

Slide 65

Slide 65 text

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.html Tuesday, June 14, 2011

Slide 66

Slide 66 text

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.html Tuesday, June 14, 2011

Slide 67

Slide 67 text

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.html Tuesday, June 14, 2011

Slide 68

Slide 68 text

Messaging Patterns Tuesday, June 14, 2011

Slide 69

Slide 69 text

There are many messaging patterns http://www.eaipatterns.com/ Tuesday, June 14, 2011

Slide 70

Slide 70 text

Basic Patterns Tuesday, June 14, 2011

Slide 71

Slide 71 text

Competing Consumers How can a messaging client process multiple messages concurrently? Tuesday, June 14, 2011

Slide 72

Slide 72 text

Competing Consumers Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently. Tuesday, June 14, 2011

Slide 73

Slide 73 text

Competing Consumers Tuesday, June 14, 2011

Slide 74

Slide 74 text

Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type = <<"direct">>, durable = true}, #'queue.declare'{queue = Queue, durable = false}, #'queue.bind'{queue = Queue, exchange = Exchange}. publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011

Slide 75

Slide 75 text

Consumer Code init_consumer(Exchange, Queue) -> init(Exchange, Queue), #'basic.consume'{queue = Queue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> do_something_with_msg(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011

Slide 76

Slide 76 text

Publish/Subscribe How can the sender broadcast an event to all interested receivers? Tuesday, June 14, 2011

Slide 77

Slide 77 text

Publish/Subscribe Send the event on a Publish-Subscribe Channel, which delivers a copy of a particular event to each receiver. Tuesday, June 14, 2011

Slide 78

Slide 78 text

Publish/Subscribe Tuesday, June 14, 2011

Slide 79

Slide 79 text

Publisher Code init(Exchange, Queue) -> #'exchange.declare'{exchange = Exchange, type = <<"fanout">>, %% different type durable = true} %% same as before ... publish_msg(Exchange, Payload) -> Props = #'P_basic'{content_type = <<"application/json">>, delivery_mode = 2}, %% persistent publish(Exchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011

Slide 80

Slide 80 text

Consumer Code A init_consumer(Exchange, ResizeImageQueue) -> init(Exchange, ResizeImageQueue), #'basic.consume'{queue = ResizeImageQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> resize_message(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011

Slide 81

Slide 81 text

Consumer Code B init_consumer(Exchange, NotifyFriendsQueue) -> init(Exchange, NotifyFriendsQueue), #'basic.consume'{queue = NotifyFriendsQueue}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> notify_friends(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011

Slide 82

Slide 82 text

Consumer Code C init_consumer(Exchange, LogImageUpload) -> init(Exchange, LogImageUpload), #'basic.consume'{queue = LogImageUpload}. on(#'basic.deliver'{delivery_tag = DeliveryTag}, #amqp_msg{} = Msg) -> log_image_upload(Msg), #'basic.ack'{delivery_tag = DeliveryTag}. Tuesday, June 14, 2011

Slide 83

Slide 83 text

Request/Reply When an application sends a message, how can it get a response from the receiver? Tuesday, June 14, 2011

Slide 84

Slide 84 text

Request/Reply Send a pair of Request-Reply messages, each on its own channel. Tuesday, June 14, 2011

Slide 85

Slide 85 text

Request/Reply Tuesday, June 14, 2011

Slide 86

Slide 86 text

Return Address How does a replier know where to send the reply? Tuesday, June 14, 2011

Slide 87

Slide 87 text

Return Address The request message should contain a Return Address that indicates where to send the reply message. Tuesday, June 14, 2011

Slide 88

Slide 88 text

Return Address Tuesday, June 14, 2011

Slide 89

Slide 89 text

Correlation Identifier How does a requestor that has received a reply know which request this is the reply for? Tuesday, June 14, 2011

Slide 90

Slide 90 text

Correlation Identifier Each reply message should contain a Correlation Identifier, a unique identifier that indicates which request message this reply is for. Tuesday, June 14, 2011

Slide 91

Slide 91 text

Correlation Identifier Tuesday, June 14, 2011

Slide 92

Slide 92 text

Putting it all together Tuesday, June 14, 2011

Slide 93

Slide 93 text

RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. Tuesday, June 14, 2011

Slide 94

Slide 94 text

RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). Tuesday, June 14, 2011

Slide 95

Slide 95 text

RPC Client init() -> #'queue.declare_ok'{queue = SelfQueue} = #'queue.declare'{exclusive = true, auto_delete = true}, #'basic.consume'{queue = SelfQueue, no_ack = true}, SelfQueue. request(Payload, RequestId) -> Props = #'P_basic'{correlation_id = RequestId, reply_to = SelfQueue}, publish(ServerExchange, #amqp_msg{props = Props, payload = Payload}). on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, do_something_with_reply(Payload). Tuesday, June 14, 2011

Slide 96

Slide 96 text

RPC Server on(#'basic.deliver'{}, #amqp_msg{props = Props, payload = Payload}) -> CorrelationId = Props.correlation_id, ReplyTo = Props.reply_to, Reply = process_request(Payload), NewProps = #'P_basic'{correlation_id = CorrelationId}, publish("", %% anonymous exchange #amqp_msg{props = NewProps, payload = Reply}, ReplyTo). %% routing key Tuesday, June 14, 2011

Slide 97

Slide 97 text

Advanced Patterns Tuesday, June 14, 2011

Slide 98

Slide 98 text

Control Bus How can we effectively administer a messaging system that is distributed across multiple platforms and a wide geographic area? Tuesday, June 14, 2011

Slide 99

Slide 99 text

Control Bus Use a Control Bus to manage an enterprise integration system. Tuesday, June 14, 2011

Slide 100

Slide 100 text

Control Bus • Send Configuration Messages • Start/Stop Services • Inject Test Messages • Collect Statistics Tuesday, June 14, 2011

Slide 101

Slide 101 text

Control Bus Tuesday, June 14, 2011

Slide 102

Slide 102 text

Control Bus Make Services “Control Bus” Enabled Tuesday, June 14, 2011

Slide 103

Slide 103 text

Detour How can you route a message through intermediate steps to perform validation, testing or debugging functions? Tuesday, June 14, 2011

Slide 104

Slide 104 text

Detour Construct a Detour with a context-based router controlled via the Control Bus. In one state the router routes incoming messages through additional steps while in the other it routes messages directly to the destination channel. Tuesday, June 14, 2011

Slide 105

Slide 105 text

Detour Tuesday, June 14, 2011

Slide 106

Slide 106 text

Wire Tap How do you inspect messages that travel on a point-to-point channel? Tuesday, June 14, 2011

Slide 107

Slide 107 text

Wire Tap Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel. Tuesday, June 14, 2011

Slide 108

Slide 108 text

Wire Tap Tuesday, June 14, 2011

Slide 109

Slide 109 text

Smart Proxy How can you track messages on a service that publishes reply messages to the Return Address specified by the requestor? Tuesday, June 14, 2011

Slide 110

Slide 110 text

Smart Proxy Use a Smart Proxy to store the Return Address supplied by the original requestor and replace it with the address of the Smart Proxy. When the service sends the reply message route it to the original Return Address. Tuesday, June 14, 2011

Slide 111

Slide 111 text

Smart Proxy Tuesday, June 14, 2011

Slide 112

Slide 112 text

Credits Pattern graphics and description taken from: http://www.eaipatterns.com/ Tuesday, June 14, 2011

Slide 113

Slide 113 text

Thanks! @old_sound http://vimeo.com/user1169087 http://www.slideshare.net/old_sound Tuesday, June 14, 2011