Decouple all the things - asynchronous messaging keeps it simple

Talk at arrrrcamp 2015

Transcript

1. Decouple all the things
   Asynchronous messaging keeps it simple
   Arrrrcamp 2015
   

   View Slide

2. Communication is Exhausting
   

   View Slide

3. RSVP
   

   View Slide

4. Ambigous Reply
   4
   accepts
   X
   regrets
   Guests:
   all my kids
   Name:
   

   View Slide

5. Ambigous Reply
   5
   accepts
   X
   regrets
   X
   Guests:
   Name:
   Ada
   

   View Slide

6. No RSVP required?
   

   View Slide

7. Machine-to-Machine Communication
   

   View Slide

8. 418 I’m a Teapot
   

   View Slide

9. Improve Maintainability and Stability by adding
   Asynchronous Communication
   

   View Slide

10. Outline
    Disadvantages of Synchronous Communication
    AMQP
    Publish-Subscribe
    Task Queue
    Code Example & Live Demo
    Summary
    10
    

    View Slide

11. Disadvantages of Synchronous Communication
    

    View Slide

12. inform about
    update
    Master data updates need to propagate
    12
    business partners
    master data
    business partner
    out of business
    orders
    support
    contracts
    

    View Slide

13. downside of REST:
    client misses update if temporarily down
    13
    master data
    orders
    support
    contracts
    

    View Slide

14. downside of REST:
    master data app needs to know its clients
    14
    master data
    orders
    support
    contracts
    invoices
    

    View Slide

15. xing.com – core updates have to propagate
    15
    core
    user data
    jobs
    events
    …
    

    View Slide

16. Asynchronous Messaging
    Advanced Message Queuing Protocol
    

    View Slide

17. amqp.org/about/what
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet, and in the
    Cloud.”
    

    View Slide

18. “open standard application layer protocol for
    message-oriented middleware”
    https://en.wikipedia.org/wiki/
    Advanced_Message_Queuing_Protocol
    “open standard application layer protocol for
    message-oriented middleware”
    

    View Slide

19. Interoperability
    19
    message
    broker
    message
    consumer
    message
    producer
    

    View Slide

20. Message Flow
    20
    P
    Message Broker
    X Q C
    Producer
    Exchange
    Queue
    Consumer
    

    View Slide

21. Messages contain payload and routing key
    21
    Routing Key
    Payload
    •  application data
    – any form, any encoding…
    …and more
    •  Structured app-specific data (properties / attributes)
    •  Headers
    •  Annotations
    •  …
    

    View Slide

22. Applications are in power
    22
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C
    

    View Slide

23. No information lost if consumer down
    23
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C
    

    View Slide

24. Add consumers without touching producer
    24
    P X
    Q2
    Q1 C1
    C2
    Producer
    Exchange
    Queue
    Consumer
    

    View Slide

25. Publish - Subscribe
    

    View Slide

26. Different messages for different client needs
    26
    master
    data
    client
    Owns additional data related to master data
    •  User account aka master data record removed
    Caches master data
    •  If master data updated or removed
    Mirrors master data
    •  Updated master data
    

    View Slide

27. Example Message
    routing key
    <...>user.deleted
    payload
    27
    {
    “user_id”: 42
    }
    

    View Slide

28. Example Message
    routing key
    <...>profile.updated
    payload
    28
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }
    

    View Slide

29. Consumer subscribes, publisher does not notice
    29
    P X
    Q2
    Q1 C1
    C2
    

    View Slide

30. Why not include updated data?
    routing key
    <...>profile.updated
    payload
    30
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }
    

    View Slide

31. No guaranteed order – aim for commutativity
    Order of messages not guarantueed
    Commutativity
    Complement with synchronous API (e.g. REST)
    •  most recent data
    •  leading system
    •  message consumers follow up with REST if necessary
    Small payload
    •  easier on the broker
    •  just enough data for consumer to decide about making REST request
    31
    

    View Slide

32. No exactly-once delivery – aim for idem-potent messages
    Duplicated messages
    •  possible e.g. if producer re-sends message after connection failure
    •  use idem-potent message handling
    32
    

    View Slide

33. Add producers without touching the consumer
    33
    Producer
    Exchange
    Queue
    Consumer
    P1
    P2
    X Q C
    

    View Slide

34. Centralized tracking of decentralized events
    34
    P1
    P2
    X Q C
    event to be
    tracked
    tracking
    counter ++
    

    View Slide

35. Avoid n-m routing, keep breaking changes manageable
    35
    P1
    P2
    X
    Q2
    Q1 C1
    C2
    P1
    P2 C2
    C1
    

    View Slide

36. Notifications / Events - producer state changed
    36
    Message Broker
    X Q
    P C
    event /
    state change
    

    View Slide

37. Task / Command – consumer state to change
    37
    Message Broker
    X Q
    P C
    event / state
    change
    

    View Slide

38. Task Queue
    

    View Slide

39. Run tasks asynchronously
    39
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing
    

    View Slide

40. Improve response time, avoid downtimes, gain reliability
    40
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing
    

    View Slide

41. Scaling on the fly
    41
    X Q
    C
    C
    P
    migration
    trigger
    data
    migration
    

    View Slide

42. Migration without downtime
    42
    Message Broker
    X Q
    P C
    Old system New system
    

    View Slide

43. Test without impacting users
    43
    Message Broker
    X Q
    P C
    old app
    user facing
    shadow calls
    to new app
    

    View Slide

44. Hello, World!
    Code Example & Live Demo
    

    View Slide

45. Flexible Routing via different Exchange Types
    Direct Exchange
    •  routes to all queues whose binding_key equals routing_key
    Fanout Exchange
    •  broadcasts to every queue it has a binding for
    •  ignores routing key and queue name
    Topic Exchange
    •  dot-separated words as routing keys, plus wildcards
    •  routing on multiple attributes
    Header Exchange
    •  based on headers, not routing keys
    •  routing on multiple attributes
    45
    

    View Slide

46. In a nutshell
    •  Interoperability
    •  Improve response times
    •  Avoid downtimes
    •  Robustness & Reliability
    •  Scale on the fly
    •  Reduce complexity
    •  Simplify code
    46
    

    View Slide

47. Decouple all your things!
    

    View Slide

48. Who am I
    Contact
    •  http://www.kpuschke.eu
    •  twitter: @titanoboa42
    •  https://www.xing.com/profile/
    Kerstin_Puschke
    Senior Software Engineer
    at XING Hamburg
    •  xing.com
    social network for business
    professionals
    •  debian, perl, javascript, ruby on
    rails, mysql, redis, riak, . . .
    We’re hiring!
    http://corporate.xing.com/
    english/company/careers-
    at-xing/
    

    View Slide

49. www.xing.com
    Thank you
    for your
    attention!
    

    View Slide