Decouple all the things: Asynchronous messaging keeps it simple

Transcript

1. Decouple all the things Asynchronous messaging keeps it simple Kerstin

   Puschke @titanoboa42 Codemotion Berlin 2016

2. Decouple all the things / Kerstin Puschke / @titanoboa42 /

   Codemotion Berlin 2016 2 Machine to Machine Communication

3. Decouple all the things / Kerstin Puschke / @titanoboa42 /

   Codemotion Berlin 2016 3 418 I’m a teapot

4. Decouple all the things / Kerstin Puschke / @titanoboa42 /

   Codemotion Berlin 2016 4 Improving maintainablity and stability with asynchronous communication

5. Index Synchronous Communication Asynchronous Messaging with AMQP Publish - Subscribe

   Task Queue Code & Demo Summary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 5

6. Synchronous Communication and its disadvantages Decouple all the things /

   Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 6

7. Updates need to propagate from the owner of the data

   to its clients Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 7 inform about update business partners business partner out of business orders support contracts

8. downside of REST: client misses update if temporarily down Decouple

   all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 8 business partners orders support contracts

9. downside of REST: app needs to know its clients Decouple

   all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 9 business partners orders support contracts invoices

10. Updates need to propagate Example from xing.com Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 10 member profile (cv) new job title events …

11. Asynchronous Messaging using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 11

12. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 12 Advanced Message Queuing Protocol

13. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 13 “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the cloud.” amqp.org/about/what Bingo, anyone?

14. Decouple all the things / Kerstin Puschke / @titanoboa42 /

    Codemotion Berlin 2016 14 “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol

15. Interoperability Decouple all the things / Kerstin Puschke / @titanoboa42

    / Codemotion Berlin 2016 15 message broker message consumer message producer

16. Messages contain payload and routing key •  Routing Key • 

    Payload -  application data -  any form, any encoding… •  …and more -  Structured app-specific data (properties / attributes) -  Headers, Annotations -  … Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 16

17. Message Flow Decouple all the things / Kerstin Puschke /

    @titanoboa42 / Codemotion Berlin 2016 17 Producer Exchange Queue Consumer P Message Broker X Q C

18. No information lost if consumer down Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 18 Producer Exchange Queue Consumer P Message Broker X Q C

19. Applications are in power Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 19 Producer Exchange Queue Consumer P Message Broker X Q C

20. Producer does not have to know consumers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 20 Producer Exchange Queue Consumer P X Q2 Q1 C1 C2

21. Publish Subscribe using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 21

22. Propagating updates Clients have different needs – use different messages

    Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 22 owner of updated data owner of related data mirror cache

23. Example message Routing key <…>.user.deleted Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 23 { “user_id” : 42 }

24. Example message Routing key <…>.profile.updated Payload Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 24 { “user_id” : 42, “fields” : [ “country”, “city” ] }

25. Flexible routing via different exchange types •  Direct Exchange - 

    routes to all queues whose binding_key equals routing_key •  Topic Exchange -  routing on multiple attributes -  dot-separated words as routing keys, plus wildcards •  Header Exchange -  routing on multiple attributes based on headers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 25

26. Flexible routing via different exchange types •  Fanout Exchange - 

    broadcasts to every queue it has a binding for -  ignores routing key and queue name Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 26

27. Don’t include updated data Routing key <…>.profile.updated Payload Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 27 { “user_id” : 42, “fields” : [ “country”, “city” ] }

28. No guaranteed order – aim for commutativity •  Order of

    messages not guarantueed •  Commutativity •  Complement with synchronous API (e.g. REST) -  most recent data -  leading system -  message consumers follow up with REST if necessary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 28

29. No exactly-once delivery aim for idem-potent messages •  Duplicated messages

    -  possible e.g. if producer re-sends message after connection failure -  use idem-potent message handling Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 29

30. Consumer does not have to know producers Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 30 Producer Exchange Queue Consumer X P2 P1 Q C

31. Centralized tracking of decentralized events Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 31 X P2 P1 Q C event to be tracked tracking counter ++

32. Avoid n-m routing Keep breaking changes manageable Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 32 X Q2 Q1 P2 P1 P1 P2 C1 C1 C2 C2

33. Notifications / Events producer state change Decouple all the things

    / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 33 X Q C event state changed P

34. Task / Command consumer state to change Decouple all the

    things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 34 X Q C event state change P

35. Task Queue using the Advanced Message Queuing Protocol Decouple all

    the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 35

36. Run tasks asynchronously Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 36 X Q C image upload user facing image processing P

37. Improve response time Avoid downtimes Decouple all the things /

    Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 37 X Q C image upload user facing image processing P

38. Scaling on the fly Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 38 X Q migration trigger data migration P C1 C2

39. Migration without downtime Decouple all the things / Kerstin Puschke

    / @titanoboa42 / Codemotion Berlin 2016 39 X Q C old system user facing new system P

40. Test without impacting user Decouple all the things / Kerstin

    Puschke / @titanoboa42 / Codemotion Berlin 2016 40 X Q C old system user facing shadow calls P

41. Example using the Advanced Message Queuing Protocol in Ruby Decouple

    all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 41

42. Ruby example Connect to the broker require "bunny" connection =

    Bunny.new connection.start channel = connection.create_channel Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 42

43. Ruby example Create queue task_queue = channel.queue( "awesome_task_queue", durable: true

    ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 43

44. Ruby example Publish message msg = "Hello “+Time.now.strftime('%T') channel.default_exchange.publish( msg,

    routing_key: task_queue.name, persistent: true ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 44

45. Ruby example Consume message task_queue.subscribe(block: true) do |delivery_info, properties, payload|

    puts "Received '#{payload}'” sleep 2 # hard work end Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 45

46. Demo Decouple all the things / Kerstin Puschke / @titanoboa42

    / Codemotion Berlin 2016 46

47. In a nutshell •  Interoperability •  Improve response times • 

    Avoid downtimes •  Scale on the fly •  Simplify code •  Easy to get started Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 47

48. Decouple all the things Asynchronous messaging keeps it simple Kerstin

    Puschke @titanoboa42 Codemotion Berlin 2016 We‘re hiring xing.com/careers