Decouple all the things - asynchronous messaging keeps it simple

Transcript

1. Decouple all the things Asynchronous messaging keeps it simple YAPC::EU

   2015

2. Communication is Exhausting

3. RSVP

4. Missing Replies

5. Ambigous Replies

6. Ambigous Reply 6 accepts X regrets Guests: all my kids

   Name:

7. Ambigous Reply 7 accepts X regrets X Guests: Name: Ada

8. No RSVP required?

9. Machine-to-Machine Communication

10. Timeout?

11. 418 I’m a Teapot

12. Improve Maintainability and Stability by adding Asynchronous Communication

13. Outline Disadvantages of Synchronous Communication AMQP Publish-Subscribe Task Queue Code

    Example & Live Demo Summary 13

14. Disadvantages of Synchronous Communication

15. inform about update Master data updates need to propagate 15

    customer data customer out of business orders support contracts

16. downside of REST: client misses update if temporarily down 16

    customer data orders support contracts

17. downside of REST: master data app needs to know its

    clients 17 customer data orders support contracts invoices

18. xing.com – core updates have to propagate 18 core user

    data jobs events …

19. Async. messaging to propagate master data updates 19 core user

    data events … jobs

20. Asynchronous Messaging Advanced Message Queuing Protocol

21. amqp.org/about/what “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables

    “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the Cloud.”

22. “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol

    “open standard application layer protocol for message-oriented middleware”

23. Interoperability 23 message broker consumer producer

24. Message Flow 24 P Message Broker X Q C Producer

    Exchange Queue Consumer

25. Messages contain payload and routing key 25 Routing Key Payload

    •  application data – any form, any encoding… …and more •  Structured app-specific data (properties / attributes) •  Headers •  Annotations •  …

26. Applications are in power 26 X Q Producer Exchange Queue

    Consumer P C

27. No information lost if consumer down 27 X Q Producer

    Exchange Queue Consumer P C

28. Add consumers without touching producer 28 P X Q2 Q1

    C1 C2 Producer Exchange Queue Consumer

29. Publish - Subscribe

30. Different messages for different client needs 30 master data client

    Owns additional user data related to master data •  User account aka master data record removed Caches master data •  If master data updated or removed Mirrors master data •  Updated master data

31. Example Message routing key <...>user.deleted payload 31 { “user_id”: 42

    }

32. Example Message routing key <...>profile.updated payload 32 { “user_id”: 42,

    “fields”: [“last_name”] }

33. Consumer subscribes, publisher does not notice 33 P X Q2

    Q1 C1 C2

34. Why not include updated data? routing key <...>profile.updated payload 34

    { “user_id”: 42, “fields”: [“last_name”] }

35. No guaranteed order – aim for commutativity Order of messages

    not guarantueed Commutativity Complement with synchronous API (e.g. REST) •  most recent data •  leading system •  message consumers follow up with REST if necessary Small payload •  easier on the broker •  just enough data for consumer to decide about making REST request 35

36. No exactly-once delivery – aim for idem-potent messages Duplicated messages

    •  possible e.g. if producer re-sends message after connection failure •  use idem-potent message handling Lost messages •  acceptable for usecase? •  e.g. consider rebuilding mirror / cache occasionally 36

37. Add producers without touching the consumer 37 Producer Exchange Queue

    Consumer P1 P2 X Q C

38. Centralized tracking of decentralized events 38 P1 P2 X Q

    C event to be tracked tracking counter ++

39. Avoid n-m routing, keep breaking changes manageable 39 P1 P2

    X Q2 Q1 C1 C2 P1 P2 C2 C1

40. Notifications / Events - producer state changed 40 Message Broker

    X Q P C event / state change

41. Task / Command – consumer state to change 41 Message

    Broker X Q P C event / state change

42. Task Queue

43. Run tasks asynchronously 43 Message Broker X Q P C

    image upload user facing image processing

44. Run tasks asynchronously routing key <...>picture.create payload 44 { “user_id”:

    42, <…cropping info…> }

45. Improve response time, avoid downtimes, gain reliability 45 Message Broker

    X Q P C image upload user facing image processing

46. Scaling on the fly 46 X Q C C P

    migration trigger data migration

47. Migration without downtime 47 Message Broker X Q P C

    Old system New system

48. Test without impacting users 48 Message Broker X Q P

    C old app user facing shadow calls to new app

49. Hello, World! Code Example & Live Demo

50. Flexible Routing via different Exchange Types Direct Exchange •  routes

    to all queues whose binding_key equals routing_key Fanout Exchange •  broadcasts to every queue it has a binding for •  ignores routing key and queue name Topic Exchange •  dot-separated words as routing keys, plus wildcards •  routing on multiple attributes Header Exchange •  based on headers, not routing keys •  routing on multiple attributes 50

51. In a nutshell •  Interoperability •  Improve response times • 

    Avoid downtimes •  Robustness & Reliability •  Scale on the fly •  Reduce complexity •  Simplify code 51

52. Decouple All the Things!

53. Who am I Contact •  http://www.kpuschke.eu •  twitter: @titanoboa42 • 

    https://www.xing.com/profile/ Kerstin_Puschke Senior Software Engineer at XING Hamburg •  xing.com social network for business professionals •  debian, perl, javascript, ruby on rails, mysql, redis, riak, . . . We’re hiring! http://corporate.xing.com/ english/company/careers- at-xing/

54. www.xing.com Thank you for your attention!