Decouple all the things - asynchronous messaging keeps it simple

Decouple all the things - asynchronous messaging keeps it simple

Talk at arrrrcamp 2015

5e8e44a4f6632772c47925006aff31d9?s=128

Kerstin Puschke

September 30, 2015
Tweet

Transcript

  1. Decouple all the things Asynchronous messaging keeps it simple Arrrrcamp

    2015
  2. Communication is Exhausting

  3. RSVP

  4. Ambigous Reply 4 accepts X regrets Guests: all my kids

    Name:
  5. Ambigous Reply 5 accepts X regrets X Guests: Name: Ada

  6. No RSVP required?

  7. Machine-to-Machine Communication

  8. 418 I’m a Teapot

  9. Improve Maintainability and Stability by adding Asynchronous Communication

  10. Outline Disadvantages of Synchronous Communication AMQP Publish-Subscribe Task Queue Code

    Example & Live Demo Summary 10
  11. Disadvantages of Synchronous Communication

  12. inform about update Master data updates need to propagate 12

    business partners master data business partner out of business orders support contracts
  13. downside of REST: client misses update if temporarily down 13

    master data orders support contracts
  14. downside of REST: master data app needs to know its

    clients 14 master data orders support contracts invoices
  15. xing.com – core updates have to propagate 15 core user

    data jobs events …
  16. Asynchronous Messaging Advanced Message Queuing Protocol

  17. amqp.org/about/what “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables

    “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet “The capable, commoditized, multi-vendor communications ecosystem which AMQP enables creates opportunities for commerce and innovation which can transform the way business is done on the Internet, and in the Cloud.”
  18. “open standard application layer protocol for message-oriented middleware” https://en.wikipedia.org/wiki/ Advanced_Message_Queuing_Protocol

    “open standard application layer protocol for message-oriented middleware”
  19. Interoperability 19 message broker message consumer message producer

  20. Message Flow 20 P Message Broker X Q C Producer

    Exchange Queue Consumer
  21. Messages contain payload and routing key 21 Routing Key Payload

    •  application data – any form, any encoding… …and more •  Structured app-specific data (properties / attributes) •  Headers •  Annotations •  …
  22. Applications are in power 22 X Q Producer Exchange Queue

    Consumer P C
  23. No information lost if consumer down 23 X Q Producer

    Exchange Queue Consumer P C
  24. Add consumers without touching producer 24 P X Q2 Q1

    C1 C2 Producer Exchange Queue Consumer
  25. Publish - Subscribe

  26. Different messages for different client needs 26 master data client

    Owns additional data related to master data •  User account aka master data record removed Caches master data •  If master data updated or removed Mirrors master data •  Updated master data
  27. Example Message routing key <...>user.deleted payload 27 { “user_id”: 42

    }
  28. Example Message routing key <...>profile.updated payload 28 { “user_id”: 42,

    “fields”: [“last_name”] }
  29. Consumer subscribes, publisher does not notice 29 P X Q2

    Q1 C1 C2
  30. Why not include updated data? routing key <...>profile.updated payload 30

    { “user_id”: 42, “fields”: [“last_name”] }
  31. No guaranteed order – aim for commutativity Order of messages

    not guarantueed Commutativity Complement with synchronous API (e.g. REST) •  most recent data •  leading system •  message consumers follow up with REST if necessary Small payload •  easier on the broker •  just enough data for consumer to decide about making REST request 31
  32. No exactly-once delivery – aim for idem-potent messages Duplicated messages

    •  possible e.g. if producer re-sends message after connection failure •  use idem-potent message handling 32
  33. Add producers without touching the consumer 33 Producer Exchange Queue

    Consumer P1 P2 X Q C
  34. Centralized tracking of decentralized events 34 P1 P2 X Q

    C event to be tracked tracking counter ++
  35. Avoid n-m routing, keep breaking changes manageable 35 P1 P2

    X Q2 Q1 C1 C2 P1 P2 C2 C1
  36. Notifications / Events - producer state changed 36 Message Broker

    X Q P C event / state change
  37. Task / Command – consumer state to change 37 Message

    Broker X Q P C event / state change
  38. Task Queue

  39. Run tasks asynchronously 39 Message Broker X Q P C

    image upload user facing image processing
  40. Improve response time, avoid downtimes, gain reliability 40 Message Broker

    X Q P C image upload user facing image processing
  41. Scaling on the fly 41 X Q C C P

    migration trigger data migration
  42. Migration without downtime 42 Message Broker X Q P C

    Old system New system
  43. Test without impacting users 43 Message Broker X Q P

    C old app user facing shadow calls to new app
  44. Hello, World! Code Example & Live Demo

  45. Flexible Routing via different Exchange Types Direct Exchange •  routes

    to all queues whose binding_key equals routing_key Fanout Exchange •  broadcasts to every queue it has a binding for •  ignores routing key and queue name Topic Exchange •  dot-separated words as routing keys, plus wildcards •  routing on multiple attributes Header Exchange •  based on headers, not routing keys •  routing on multiple attributes 45
  46. In a nutshell •  Interoperability •  Improve response times • 

    Avoid downtimes •  Robustness & Reliability •  Scale on the fly •  Reduce complexity •  Simplify code 46
  47. Decouple all your things!

  48. Who am I Contact •  http://www.kpuschke.eu •  twitter: @titanoboa42 • 

    https://www.xing.com/profile/ Kerstin_Puschke Senior Software Engineer at XING Hamburg •  xing.com social network for business professionals •  debian, perl, javascript, ruby on rails, mysql, redis, riak, . . . We’re hiring! http://corporate.xing.com/ english/company/careers- at-xing/
  49. www.xing.com Thank you for your attention!