Decouple all the things: Asynchronous messaging keeps it simple

Talk at Codemotion Berlin 2016

Transcript

1. Decouple all the things Asynchronous messaging keeps it simple Kerstin Puschke @titanoboa42 Codemotion Berlin 2016
2. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 2 Machine to Machine Communication
3. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 3 418 I’m a teapot
4. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 4 Improving maintainablity and stability with asynchronous communication
5. Index Synchronous Communication Asynchronous Messaging with AMQP Publish - Subscribe Task Queue Code & Demo Summary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 5
6. Synchronous Communication and its disadvantages Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 6
7. Updates need to propagate from the owner of the data to its clients Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 7 inform about update business partners business partner out of business orders support contracts
8. downside of REST: client misses update if temporarily down Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 8 business partners orders support contracts
9. downside of REST: app needs to know its clients Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 9 business partners orders support contracts invoices
10. Updates need to propagate Example from xing.com Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 10 member profile (cv) new job title events …
11. Asynchronous Messaging using the Advanced Message Queuing Protocol Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 11
12. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 12 Advanced Message Queuing Protocol
13. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 13 “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the cloud.” amqp.org/about/what Bingo, anyone?
14. Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 14 “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol
15. Interoperability Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 15 message broker message consumer message producer
16. Messages contain payload and routing key •  Routing Key •  Payload -  application data -  any form, any encoding… •  …and more -  Structured app-specific data (properties / attributes) -  Headers, Annotations -  … Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 16
17. Message Flow Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 17 Producer Exchange Queue Consumer P Message Broker X Q C
18. No information lost if consumer down Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 18 Producer Exchange Queue Consumer P Message Broker X Q C
19. Applications are in power Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 19 Producer Exchange Queue Consumer P Message Broker X Q C
20. Producer does not have to know consumers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 20 Producer Exchange Queue Consumer P X Q2 Q1 C1 C2
21. Publish Subscribe using the Advanced Message Queuing Protocol Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 21
22. Propagating updates Clients have different needs – use different messages Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 22 owner of updated data owner of related data mirror cache
23. Example message Routing key <…>.user.deleted Payload Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 23 { “user_id” : 42 }
24. Example message Routing key <…>.profile.updated Payload Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 24 { “user_id” : 42, “fields” : [ “country”, “city” ] }
25. Flexible routing via different exchange types •  Direct Exchange -  routes to all queues whose binding_key equals routing_key •  Topic Exchange -  routing on multiple attributes -  dot-separated words as routing keys, plus wildcards •  Header Exchange -  routing on multiple attributes based on headers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 25
26. Flexible routing via different exchange types •  Fanout Exchange -  broadcasts to every queue it has a binding for -  ignores routing key and queue name Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 26
27. Don’t include updated data Routing key <…>.profile.updated Payload Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 27 { “user_id” : 42, “fields” : [ “country”, “city” ] }
28. No guaranteed order – aim for commutativity •  Order of messages not guarantueed •  Commutativity •  Complement with synchronous API (e.g. REST) -  most recent data -  leading system -  message consumers follow up with REST if necessary Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 28
29. No exactly-once delivery aim for idem-potent messages •  Duplicated messages -  possible e.g. if producer re-sends message after connection failure -  use idem-potent message handling Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 29
30. Consumer does not have to know producers Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 30 Producer Exchange Queue Consumer X P2 P1 Q C
31. Centralized tracking of decentralized events Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 31 X P2 P1 Q C event to be tracked tracking counter ++
32. Avoid n-m routing Keep breaking changes manageable Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 32 X Q2 Q1 P2 P1 P1 P2 C1 C1 C2 C2
33. Notifications / Events producer state change Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 33 X Q C event state changed P
34. Task / Command consumer state to change Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 34 X Q C event state change P
35. Task Queue using the Advanced Message Queuing Protocol Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 35
36. Run tasks asynchronously Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 36 X Q C image upload user facing image processing P
37. Improve response time Avoid downtimes Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 37 X Q C image upload user facing image processing P
38. Scaling on the fly Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 38 X Q migration trigger data migration P C1 C2
39. Migration without downtime Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 39 X Q C old system user facing new system P
40. Test without impacting user Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 40 X Q C old system user facing shadow calls P
41. Example using the Advanced Message Queuing Protocol in Ruby Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 41
42. Ruby example Connect to the broker require "bunny" connection = Bunny.new connection.start channel = connection.create_channel Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 42
43. Ruby example Create queue task_queue = channel.queue( "awesome_task_queue", durable: true ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 43
44. Ruby example Publish message msg = "Hello “+Time.now.strftime('%T') channel.default_exchange.publish( msg, routing_key: task_queue.name, persistent: true ) Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 44
45. Ruby example Consume message task_queue.subscribe(block: true) do |delivery_info, properties, payload| puts "Received '#{payload}'” sleep 2 # hard work end Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 45
46. Demo Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 46
47. In a nutshell •  Interoperability •  Improve response times •  Avoid downtimes •  Scale on the fly •  Simplify code •  Easy to get started Decouple all the things / Kerstin Puschke / @titanoboa42 / Codemotion Berlin 2016 47
48. Decouple all the things Asynchronous messaging keeps it simple Kerstin Puschke @titanoboa42 Codemotion Berlin 2016 We‘re hiring xing.com/careers