Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Messaging Patterns With RabbitMQ

Alvaro Videla
September 26, 2011

Messaging Patterns With RabbitMQ

Alvaro Videla

September 26, 2011
Tweet

More Decks by Alvaro Videla

Other Decks in Programming

Transcript

  1. Messaging Patterns
    With RabbitMQ
    Álvaro Videla - Liip AG
    Tuesday, June 14, 2011

    View Slide

  2. About Me
    • Developer at Liip AG
    • Blog: http://videlalvaro.github.com/
    • Twitter: @old_sound
    Tuesday, June 14, 2011

    View Slide

  3. About Me
    Co-authoring
    RabbitMQ in Action
    http://bit.ly/rabbitmq
    Tuesday, June 14, 2011

    View Slide

  4. Why Do I need
    Messaging?
    Tuesday, June 14, 2011

    View Slide

  5. An Example
    Tuesday, June 14, 2011

    View Slide

  6. Implement a
    Photo Gallery
    Tuesday, June 14, 2011

    View Slide

  7. Two Parts:
    Tuesday, June 14, 2011

    View Slide

  8. Pretty Simple
    Tuesday, June 14, 2011

    View Slide

  9. ‘Till new
    requirements arrive
    Tuesday, June 14, 2011

    View Slide

  10. The Product Owner
    Tuesday, June 14, 2011

    View Slide

  11. Can we also notify the
    user friends when she
    uploads a new image?
    Tuesday, June 14, 2011

    View Slide

  12. Can we also notify the
    user friends when she
    uploads a new image?
    I forgot to mention we need it for tomorrow…
    Tuesday, June 14, 2011

    View Slide

  13. The Social Media Guru
    Tuesday, June 14, 2011

    View Slide

  14. We need to give badges
    to users for each
    picture upload
    Tuesday, June 14, 2011

    View Slide

  15. We need to give badges
    to users for each
    picture upload
    and post uploads to Twitter
    Tuesday, June 14, 2011

    View Slide

  16. The Sysadmin
    Tuesday, June 14, 2011

    View Slide

  17. Dumb! You’re delivering
    full size images!
    The bandwidth bill has
    tripled!
    Tuesday, June 14, 2011

    View Slide

  18. Dumb! You’re delivering
    full size images!
    The bandwidth bill has
    tripled!
    We need this fixed for yesterday!
    Tuesday, June 14, 2011

    View Slide

  19. The Developer in the
    other team
    Tuesday, June 14, 2011

    View Slide

  20. I need to call your PHP
    stuff but from Python
    Tuesday, June 14, 2011

    View Slide

  21. I need to call your PHP
    stuff but from Python
    And also Java starting next week
    Tuesday, June 14, 2011

    View Slide

  22. The User
    Tuesday, June 14, 2011

    View Slide

  23. I don’t want to wait
    till your app resizes
    my image!
    Tuesday, June 14, 2011

    View Slide

  24. You
    Tuesday, June 14, 2011

    View Slide

  25. FML!
    Tuesday, June 14, 2011

    View Slide

  26. Let’s see the
    code evolution
    Tuesday, June 14, 2011

    View Slide

  27. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    image_handler:do_upload(ReqData:get_file()),
    ok.
    First Implementation:
    Tuesday, June 14, 2011

    View Slide

  28. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    resize_image(Image),
    ok.
    Second Implementation:
    Tuesday, June 14, 2011

    View Slide

  29. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    resize_image(Image),
    notify_friends(ReqData:get_user()),
    ok.
    Third Implementation:
    Tuesday, June 14, 2011

    View Slide

  30. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    resize_image(Image),
    notify_friends(ReqData:get_user()),
    add_points_to_user(ReqData:get_user()),
    ok.
    Fourth Implementation:
    Tuesday, June 14, 2011

    View Slide

  31. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    resize_image(Image),
    notify_friends(ReqData:get_user()),
    add_points_to_user(ReqData:get_user()),
    tweet_new_image(User, Image),
    ok.
    Final Implementation:
    Tuesday, June 14, 2011

    View Slide

  32. Can our code scale to
    new requirements?
    Tuesday, June 14, 2011

    View Slide

  33. What if
    Tuesday, June 14, 2011

    View Slide

  34. What if
    • We need to speed up image conversion
    Tuesday, June 14, 2011

    View Slide

  35. What if
    • We need to speed up image conversion
    • User notification has to be sent by email
    Tuesday, June 14, 2011

    View Slide

  36. What if
    • We need to speed up image conversion
    • User notification has to be sent by email
    • Stop tweeting about new images
    Tuesday, June 14, 2011

    View Slide

  37. What if
    • We need to speed up image conversion
    • User notification has to be sent by email
    • Stop tweeting about new images
    • Resize in different formats
    Tuesday, June 14, 2011

    View Slide

  38. Can we do better?
    Tuesday, June 14, 2011

    View Slide

  39. Sure.
    Using messaging
    Tuesday, June 14, 2011

    View Slide

  40. Design
    Publish / Subscribe Pattern
    Tuesday, June 14, 2011

    View Slide

  41. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    Msg = #msg{user = ReqData:get_user(), image = Image},
    publish_message('new_image', Msg).
    First Implementation:
    Tuesday, June 14, 2011

    View Slide

  42. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    Msg = #msg{user = ReqData:get_user(), image = Image},
    publish_message('new_image', Msg).
    First Implementation:
    %% friends notifier
    on('new_image', Msg) ->
    notify_friends(Msg.user, Msg.image).
    Tuesday, June 14, 2011

    View Slide

  43. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    Msg = #msg{user = ReqData:get_user(), image = Image},
    publish_message('new_image', Msg).
    First Implementation:
    %% friends notifier
    on('new_image', Msg) ->
    notify_friends(Msg.user, Msg.image).
    %% points manager
    on('new_image', Msg) ->
    add_points(Msg.user, 'new_image').
    Tuesday, June 14, 2011

    View Slide

  44. %% image_controller
    handle('PUT', "/user/image", ReqData) ->
    {ok, Image} = image_handler:do_upload(ReqData:get_file()),
    Msg = #msg{user = ReqData:get_user(), image = Image},
    publish_message('new_image', Msg).
    First Implementation:
    %% friends notifier
    on('new_image', Msg) ->
    notify_friends(Msg.user, Msg.image).
    %% points manager
    on('new_image', Msg) ->
    add_points(Msg.user, 'new_image').
    %% resizer
    on('new_image', Msg) ->
    resize_image(Msg.image).
    Tuesday, June 14, 2011

    View Slide

  45. Second Implementation:
    Tuesday, June 14, 2011

    View Slide

  46. Second Implementation:
    %% there’s none.
    Tuesday, June 14, 2011

    View Slide

  47. Messaging
    Tuesday, June 14, 2011

    View Slide

  48. Messaging
    • Share data across processes
    Tuesday, June 14, 2011

    View Slide

  49. Messaging
    • Share data across processes
    • Processes can be part of different apps
    Tuesday, June 14, 2011

    View Slide

  50. Messaging
    • Share data across processes
    • Processes can be part of different apps
    • Apps can live in different machines
    Tuesday, June 14, 2011

    View Slide

  51. Messaging
    • Share data across processes
    • Processes can be part of different apps
    • Apps can live in different machines
    • Communication is Asynchronous
    Tuesday, June 14, 2011

    View Slide

  52. Main Concepts
    Tuesday, June 14, 2011

    View Slide

  53. Main Concepts
    • Messages are sent by Producers
    Tuesday, June 14, 2011

    View Slide

  54. Main Concepts
    • Messages are sent by Producers
    • Messages are delivered to Consumers
    Tuesday, June 14, 2011

    View Slide

  55. Main Concepts
    • Messages are sent by Producers
    • Messages are delivered to Consumers
    • Messages goes through a Channel
    Tuesday, June 14, 2011

    View Slide

  56. Messaging
    and
    RabbitMQ
    Tuesday, June 14, 2011

    View Slide

  57. What is RabbitMQ?
    Tuesday, June 14, 2011

    View Slide

  58. RabbitMQ
    • Enterprise Messaging System
    • Open Source MPL
    • Written in Erlang/OTP
    • Commercial Support
    • Messaging via AMQP
    Tuesday, June 14, 2011

    View Slide

  59. Features
    • Reliable and High Scalable
    • Easy To install
    • Easy To Cluster
    • Runs on: Windows, Solaris, Linux, OSX
    • AMQP 0.8 - 0.9.1
    Tuesday, June 14, 2011

    View Slide

  60. Client Libraries
    • Java
    • .NET/C#
    • Erlang
    • Ruby, Python, PHP, Perl, AS3, Lisp, Scala,
    Clojure, Haskell
    Tuesday, June 14, 2011

    View Slide

  61. AMQP
    • Advanced Message Queuing Protocol
    • Suits Interoperability
    • Completely Open Protocol
    • Binary Protocol
    Tuesday, June 14, 2011

    View Slide

  62. Message Flow
    http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html
    Tuesday, June 14, 2011

    View Slide

  63. AMQP Model
    • Exchanges
    • Message Queues
    • Bindings
    • Rules for binding them
    Tuesday, June 14, 2011

    View Slide

  64. Exchange Types
    • Fanout
    • Direct
    • Topic
    Tuesday, June 14, 2011

    View Slide

  65. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
    Fanout_Exchange.html
    Tuesday, June 14, 2011

    View Slide

  66. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
    Direct_Exchange.html
    Tuesday, June 14, 2011

    View Slide

  67. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts-
    Topic_Exchange.html
    Tuesday, June 14, 2011

    View Slide

  68. Messaging Patterns
    Tuesday, June 14, 2011

    View Slide

  69. There are many
    messaging patterns
    http://www.eaipatterns.com/
    Tuesday, June 14, 2011

    View Slide

  70. Basic Patterns
    Tuesday, June 14, 2011

    View Slide

  71. Competing Consumers
    How can a messaging
    client process multiple
    messages concurrently?
    Tuesday, June 14, 2011

    View Slide

  72. Competing Consumers
    Create multiple Competing
    Consumers on a single channel
    so that the consumers can
    process multiple messages
    concurrently.
    Tuesday, June 14, 2011

    View Slide

  73. Competing Consumers
    Tuesday, June 14, 2011

    View Slide

  74. Publisher Code
    init(Exchange, Queue) ->
    #'exchange.declare'{exchange = Exchange,
    type = <>,
    durable = true},
    #'queue.declare'{queue = Queue, durable = false},
    #'queue.bind'{queue = Queue, exchange = Exchange}.
    publish_msg(Exchange, Payload) ->
    Props = #'P_basic'{content_type = <>,
    delivery_mode = 2}, %% persistent
    publish(Exchange, #amqp_msg{props = Props, payload = Payload}).
    Tuesday, June 14, 2011

    View Slide

  75. Consumer Code
    init_consumer(Exchange, Queue) ->
    init(Exchange, Queue),
    #'basic.consume'{queue = Queue}.
    on(#'basic.deliver'{delivery_tag = DeliveryTag},
    #amqp_msg{} = Msg) ->
    do_something_with_msg(Msg),
    #'basic.ack'{delivery_tag = DeliveryTag}.
    Tuesday, June 14, 2011

    View Slide

  76. Publish/Subscribe
    How can the sender
    broadcast an event to all
    interested receivers?
    Tuesday, June 14, 2011

    View Slide

  77. Publish/Subscribe
    Send the event on a Publish-Subscribe
    Channel, which delivers a copy of a
    particular event to each receiver.
    Tuesday, June 14, 2011

    View Slide

  78. Publish/Subscribe
    Tuesday, June 14, 2011

    View Slide

  79. Publisher Code
    init(Exchange, Queue) ->
    #'exchange.declare'{exchange = Exchange,
    type = <>, %% different type
    durable = true}
    %% same as before ...
    publish_msg(Exchange, Payload) ->
    Props = #'P_basic'{content_type = <>,
    delivery_mode = 2}, %% persistent
    publish(Exchange, #amqp_msg{props = Props, payload = Payload}).
    Tuesday, June 14, 2011

    View Slide

  80. Consumer Code A
    init_consumer(Exchange, ResizeImageQueue) ->
    init(Exchange, ResizeImageQueue),
    #'basic.consume'{queue = ResizeImageQueue}.
    on(#'basic.deliver'{delivery_tag = DeliveryTag},
    #amqp_msg{} = Msg) ->
    resize_message(Msg),
    #'basic.ack'{delivery_tag = DeliveryTag}.
    Tuesday, June 14, 2011

    View Slide

  81. Consumer Code B
    init_consumer(Exchange, NotifyFriendsQueue) ->
    init(Exchange, NotifyFriendsQueue),
    #'basic.consume'{queue = NotifyFriendsQueue}.
    on(#'basic.deliver'{delivery_tag = DeliveryTag},
    #amqp_msg{} = Msg) ->
    notify_friends(Msg),
    #'basic.ack'{delivery_tag = DeliveryTag}.
    Tuesday, June 14, 2011

    View Slide

  82. Consumer Code C
    init_consumer(Exchange, LogImageUpload) ->
    init(Exchange, LogImageUpload),
    #'basic.consume'{queue = LogImageUpload}.
    on(#'basic.deliver'{delivery_tag = DeliveryTag},
    #amqp_msg{} = Msg) ->
    log_image_upload(Msg),
    #'basic.ack'{delivery_tag = DeliveryTag}.
    Tuesday, June 14, 2011

    View Slide

  83. Request/Reply
    When an application sends a message,
    how can it get a response from the
    receiver?
    Tuesday, June 14, 2011

    View Slide

  84. Request/Reply
    Send a pair of Request-Reply
    messages, each on its own channel.
    Tuesday, June 14, 2011

    View Slide

  85. Request/Reply
    Tuesday, June 14, 2011

    View Slide

  86. Return Address
    How does a replier know where to
    send the reply?
    Tuesday, June 14, 2011

    View Slide

  87. Return Address
    The request message should contain a
    Return Address that indicates where to send
    the reply message.
    Tuesday, June 14, 2011

    View Slide

  88. Return Address
    Tuesday, June 14, 2011

    View Slide

  89. Correlation Identifier
    How does a requestor that has
    received a reply know which
    request this is the reply for?
    Tuesday, June 14, 2011

    View Slide

  90. Correlation Identifier
    Each reply message should contain a Correlation
    Identifier, a unique identifier that indicates which
    request message this reply is for.
    Tuesday, June 14, 2011

    View Slide

  91. Correlation Identifier
    Tuesday, June 14, 2011

    View Slide

  92. Putting it all together
    Tuesday, June 14, 2011

    View Slide

  93. RPC Client
    init() ->
    #'queue.declare_ok'{queue = SelfQueue} =
    #'queue.declare'{exclusive = true, auto_delete = true},
    #'basic.consume'{queue = SelfQueue, no_ack = true},
    SelfQueue.
    Tuesday, June 14, 2011

    View Slide

  94. RPC Client
    init() ->
    #'queue.declare_ok'{queue = SelfQueue} =
    #'queue.declare'{exclusive = true, auto_delete = true},
    #'basic.consume'{queue = SelfQueue, no_ack = true},
    SelfQueue.
    request(Payload, RequestId) ->
    Props = #'P_basic'{correlation_id = RequestId,
    reply_to = SelfQueue},
    publish(ServerExchange, #amqp_msg{props = Props,
    payload = Payload}).
    Tuesday, June 14, 2011

    View Slide

  95. RPC Client
    init() ->
    #'queue.declare_ok'{queue = SelfQueue} =
    #'queue.declare'{exclusive = true, auto_delete = true},
    #'basic.consume'{queue = SelfQueue, no_ack = true},
    SelfQueue.
    request(Payload, RequestId) ->
    Props = #'P_basic'{correlation_id = RequestId,
    reply_to = SelfQueue},
    publish(ServerExchange, #amqp_msg{props = Props,
    payload = Payload}).
    on(#'basic.deliver'{},
    #amqp_msg{props = Props, payload = Payload}) ->
    CorrelationId = Props.correlation_id,
    do_something_with_reply(Payload).
    Tuesday, June 14, 2011

    View Slide

  96. RPC Server
    on(#'basic.deliver'{},
    #amqp_msg{props = Props, payload = Payload}) ->
    CorrelationId = Props.correlation_id,
    ReplyTo = Props.reply_to,
    Reply = process_request(Payload),
    NewProps = #'P_basic'{correlation_id = CorrelationId},
    publish("", %% anonymous exchange
    #amqp_msg{props = NewProps,
    payload = Reply},
    ReplyTo). %% routing key
    Tuesday, June 14, 2011

    View Slide

  97. Advanced Patterns
    Tuesday, June 14, 2011

    View Slide

  98. Control Bus
    How can we effectively administer a messaging
    system that is distributed across multiple
    platforms and a wide geographic area?
    Tuesday, June 14, 2011

    View Slide

  99. Control Bus
    Use a Control Bus to
    manage an enterprise
    integration system.
    Tuesday, June 14, 2011

    View Slide

  100. Control Bus
    • Send Configuration Messages
    • Start/Stop Services
    • Inject Test Messages
    • Collect Statistics
    Tuesday, June 14, 2011

    View Slide

  101. Control Bus
    Tuesday, June 14, 2011

    View Slide

  102. Control Bus
    Make Services
    “Control Bus” Enabled
    Tuesday, June 14, 2011

    View Slide

  103. Detour
    How can you route a message through
    intermediate steps to perform validation,
    testing or debugging functions?
    Tuesday, June 14, 2011

    View Slide

  104. Detour
    Construct a Detour with a context-based router
    controlled via the Control Bus.
    In one state the router routes incoming messages
    through additional steps while in the other it routes
    messages directly to the destination channel.
    Tuesday, June 14, 2011

    View Slide

  105. Detour
    Tuesday, June 14, 2011

    View Slide

  106. Wire Tap
    How do you inspect messages that
    travel on a point-to-point channel?
    Tuesday, June 14, 2011

    View Slide

  107. Wire Tap
    Insert a simple Recipient List into the channel that
    publishes each incoming message to the main
    channel and a secondary channel.
    Tuesday, June 14, 2011

    View Slide

  108. Wire Tap
    Tuesday, June 14, 2011

    View Slide

  109. Smart Proxy
    How can you track messages on a service that
    publishes reply messages to the Return Address
    specified by the requestor?
    Tuesday, June 14, 2011

    View Slide

  110. Smart Proxy
    Use a Smart Proxy to store the Return
    Address supplied by the original
    requestor and replace it with the
    address of the Smart Proxy.
    When the service sends the reply
    message route it to the original Return
    Address.
    Tuesday, June 14, 2011

    View Slide

  111. Smart Proxy
    Tuesday, June 14, 2011

    View Slide

  112. Credits
    Pattern graphics and description taken from:
    http://www.eaipatterns.com/
    Tuesday, June 14, 2011

    View Slide

  113. Thanks!
    @old_sound
    http://vimeo.com/user1169087
    http://www.slideshare.net/old_sound
    Tuesday, June 14, 2011

    View Slide