Decouple all the things: Asynchronous messaging keeps it simple

Decouple all the things: Asynchronous messaging keeps it simple

Talk at Codemotion Berlin 2016

5e8e44a4f6632772c47925006aff31d9?s=128

Kerstin Puschke

October 24, 2016
Tweet

Transcript

  1. Decouple all the things Asynchronous messaging keeps it simple Kerstin

    Puschke @titanoboa42 Codemotion Berlin 2016
  2. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 2 Machine to Machine Communication
  3. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 3 418 I’m a teapot
  4. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 4 Improving maintainablity and stability with asynchronous communication
  5. Index Synchronous Communication Asynchronous Messaging with AMQP Publish - Subscribe

    Task Queue Code & Demo Summary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 5
  6. Synchronous Communication and its disadvantages Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 6
  7. Updates need to propagate from the owner of the data

    to its clients Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 7 inform about update business partners business partner out of business orders support contracts
  8. downside of REST: client misses update if temporarily down Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 8 business partners orders support contracts
  9. downside of REST: app needs to know its clients Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 9 business partners orders support contracts invoices
  10. Updates need to propagate Example from xing.com Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 10 member profile (cv) new job title events …
  11. Asynchronous Messaging using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 11
  12. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 12 Advanced Message Queuing Protocol
  13. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 13 “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the cloud.” amqp.org/about/what Bingo, anyone?
  14. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 14 “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol
  15. Interoperability Decouple all the things / Kerstin Puschke / @titanoboa42

    / Codemotion Berlin 2016 15 message broker message consumer message producer
  16. Messages contain payload and routing key •  Routing Key • 

    Payload -  application data -  any form, any encoding… •  …and more -  Structured app-specific data (properties / attributes) -  Headers, Annotations -  … Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 16
  17. Message Flow Decouple all the things / Kerstin Puschke /

    @titanoboa42 / Codemotion Berlin 2016 17 Producer Exchange Queue Consumer P Message Broker X Q C
  18. No information lost if consumer down Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 18 Producer Exchange Queue Consumer P Message Broker X Q C
  19. Applications are in power Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 19 Producer Exchange Queue Consumer P Message Broker X Q C
  20. Producer does not have to know consumers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 20 Producer Exchange Queue Consumer P X Q2 Q1 C1 C2
  21. Publish Subscribe using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 21
  22. Propagating updates Clients have different needs – use different messages

    Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 22 owner of updated data owner of related data mirror cache
  23. Example message Routing key <…>.user.deleted Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 23 { “user_id” : 42 }
  24. Example message Routing key <…>.profile.updated Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 24 { “user_id” : 42, “fields” : [ “country”, “city” ] }
  25. Flexible routing via different exchange types •  Direct Exchange - 

    routes to all queues whose binding_key equals routing_key •  Topic Exchange -  routing on multiple attributes -  dot-separated words as routing keys, plus wildcards •  Header Exchange -  routing on multiple attributes based on headers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 25
  26. Flexible routing via different exchange types •  Fanout Exchange - 

    broadcasts to every queue it has a binding for -  ignores routing key and queue name Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 26
  27. Don’t include updated data Routing key <…>.profile.updated Payload Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 27 { “user_id” : 42, “fields” : [ “country”, “city” ] }
  28. No guaranteed order – aim for commutativity •  Order of

    messages not guarantueed •  Commutativity •  Complement with synchronous API (e.g. REST) -  most recent data -  leading system -  message consumers follow up with REST if necessary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 28
  29. No exactly-once delivery aim for idem-potent messages •  Duplicated messages

    -  possible e.g. if producer re-sends message after connection failure -  use idem-potent message handling Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 29
  30. Consumer does not have to know producers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 30 Producer Exchange Queue Consumer X P2 P1 Q C
  31. Centralized tracking of decentralized events Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 31 X P2 P1 Q C event to be tracked tracking counter ++
  32. Avoid n-m routing Keep breaking changes manageable Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 32 X Q2 Q1 P2 P1 P1 P2 C1 C1 C2 C2
  33. Notifications / Events producer state change Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 33 X Q C event state changed P
  34. Task / Command consumer state to change Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 34 X Q C event state change P
  35. Task Queue using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 35
  36. Run tasks asynchronously Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 36 X Q C image upload user facing image processing P
  37. Improve response time Avoid downtimes Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 37 X Q C image upload user facing image processing P
  38. Scaling on the fly Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 38 X Q migration trigger data migration P C1 C2
  39. Migration without downtime Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 39 X Q C old system user facing new system P
  40. Test without impacting user Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 40 X Q C old system user facing shadow calls P
  41. Example using the Advanced Message Queuing Protocol in Ruby Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 41
  42. Ruby example Connect to the broker require "bunny" connection =

    Bunny.new connection.start channel = connection.create_channel Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 42
  43. Ruby example Create queue task_queue = channel.queue( "awesome_task_queue", durable: true

    ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 43
  44. Ruby example Publish message msg = "Hello “+Time.now.strftime('%T') channel.default_exchange.publish( msg,

    routing_key: task_queue.name, persistent: true ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 44
  45. Ruby example Consume message task_queue.subscribe(block: true) do |delivery_info, properties, payload|

    puts "Received '#{payload}'” sleep 2 # hard work end Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 45
  46. Demo Decouple all the things / Kerstin Puschke / @titanoboa42

    / Codemotion Berlin 2016 46
  47. In a nutshell •  Interoperability •  Improve response times • 

    Avoid downtimes •  Scale on the fly •  Simplify code •  Easy to get started Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 47
  48. Decouple all the things Asynchronous messaging keeps it simple Kerstin

    Puschke @titanoboa42 Codemotion Berlin 2016 We‘re hiring xing.com/careers