Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decouple all the things: Asynchronous messaging keeps it simple

Decouple all the things: Asynchronous messaging keeps it simple

Talk at Codemotion Berlin 2016

Kerstin Puschke

October 24, 2016
Tweet

More Decks by Kerstin Puschke

Other Decks in Technology

Transcript

  1. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 2 Machine to Machine Communication
  2. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 3 418 I’m a teapot
  3. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 4 Improving maintainablity and stability with asynchronous communication
  4. Index Synchronous Communication Asynchronous Messaging with AMQP Publish - Subscribe

    Task Queue Code & Demo Summary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 5
  5. Synchronous Communication and its disadvantages Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 6
  6. Updates need to propagate from the owner of the data

    to its clients Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 7 inform about update business partners business partner out of business orders support contracts
  7. downside of REST: client misses update if temporarily down Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 8 business partners orders support contracts
  8. downside of REST: app needs to know its clients Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 9 business partners orders support contracts invoices
  9. Updates need to propagate Example from xing.com Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 10 member profile (cv) new job title events …
  10. Asynchronous Messaging using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 11
  11. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 12 Advanced Message Queuing Protocol
  12. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 13 “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the cloud.” amqp.org/about/what Bingo, anyone?
  13. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 14 “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol
  14. Interoperability Decouple all the things / Kerstin Puschke / @titanoboa42

    / Codemotion Berlin 2016 15 message broker message consumer message producer
  15. Messages contain payload and routing key •  Routing Key • 

    Payload -  application data -  any form, any encoding… •  …and more -  Structured app-specific data (properties / attributes) -  Headers, Annotations -  … Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 16
  16. Message Flow Decouple all the things / Kerstin Puschke /

    @titanoboa42 / Codemotion Berlin 2016 17 Producer Exchange Queue Consumer P Message Broker X Q C
  17. No information lost if consumer down Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 18 Producer Exchange Queue Consumer P Message Broker X Q C
  18. Applications are in power Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 19 Producer Exchange Queue Consumer P Message Broker X Q C
  19. Producer does not have to know consumers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 20 Producer Exchange Queue Consumer P X Q2 Q1 C1 C2
  20. Publish Subscribe using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 21
  21. Propagating updates Clients have different needs – use different messages

    Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 22 owner of updated data owner of related data mirror cache
  22. Example message Routing key <…>.user.deleted Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 23 { “user_id” : 42 }
  23. Example message Routing key <…>.profile.updated Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 24 { “user_id” : 42, “fields” : [ “country”, “city” ] }
  24. Flexible routing via different exchange types •  Direct Exchange - 

    routes to all queues whose binding_key equals routing_key •  Topic Exchange -  routing on multiple attributes -  dot-separated words as routing keys, plus wildcards •  Header Exchange -  routing on multiple attributes based on headers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 25
  25. Flexible routing via different exchange types •  Fanout Exchange - 

    broadcasts to every queue it has a binding for -  ignores routing key and queue name Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 26
  26. Don’t include updated data Routing key <…>.profile.updated Payload Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 27 { “user_id” : 42, “fields” : [ “country”, “city” ] }
  27. No guaranteed order – aim for commutativity •  Order of

    messages not guarantueed •  Commutativity •  Complement with synchronous API (e.g. REST) -  most recent data -  leading system -  message consumers follow up with REST if necessary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 28
  28. No exactly-once delivery aim for idem-potent messages •  Duplicated messages

    -  possible e.g. if producer re-sends message after connection failure -  use idem-potent message handling Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 29
  29. Consumer does not have to know producers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 30 Producer Exchange Queue Consumer X P2 P1 Q C
  30. Centralized tracking of decentralized events Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 31 X P2 P1 Q C event to be tracked tracking counter ++
  31. Avoid n-m routing Keep breaking changes manageable Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 32 X Q2 Q1 P2 P1 P1 P2 C1 C1 C2 C2
  32. Notifications / Events producer state change Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 33 X Q C event state changed P
  33. Task / Command consumer state to change Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 34 X Q C event state change P
  34. Task Queue using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 35
  35. Run tasks asynchronously Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 36 X Q C image upload user facing image processing P
  36. Improve response time Avoid downtimes Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 37 X Q C image upload user facing image processing P
  37. Scaling on the fly Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 38 X Q migration trigger data migration P C1 C2
  38. Migration without downtime Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 39 X Q C old system user facing new system P
  39. Test without impacting user Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 40 X Q C old system user facing shadow calls P
  40. Example using the Advanced Message Queuing Protocol in Ruby Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 41
  41. Ruby example Connect to the broker require "bunny" connection =

    Bunny.new connection.start channel = connection.create_channel Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 42
  42. Ruby example Create queue task_queue = channel.queue( "awesome_task_queue", durable: true

    ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 43
  43. Ruby example Publish message msg = "Hello “+Time.now.strftime('%T') channel.default_exchange.publish( msg,

    routing_key: task_queue.name, persistent: true ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 44
  44. Ruby example Consume message task_queue.subscribe(block: true) do |delivery_info, properties, payload|

    puts "Received '#{payload}'” sleep 2 # hard work end Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 45
  45. In a nutshell •  Interoperability •  Improve response times • 

    Avoid downtimes •  Scale on the fly •  Simplify code •  Easy to get started Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 47
  46. Decouple all the things Asynchronous messaging keeps it simple Kerstin

    Puschke @titanoboa42 Codemotion Berlin 2016 We‘re hiring xing.com/careers