Decouple all the things - Asynchronous messaging keeps it simple

Talk at DevconTLV, March 2016

Transcript

1. Decouple all the things
   Asynchronous messaging keeps it simple
   DevconTLV 2016
   

   View Slide

2. Machine-to-Machine Communication
   

   View Slide

3. 418 I’m a Teapot
   

   View Slide

4. Improve Maintainability and Stability by adding
   Asynchronous Communication
   

   View Slide

5. Outline
   Disadvantages of Synchronous Communication
   AMQP
   Publish-Subscribe
   Task Queue
   Summary
   5
   

   View Slide

6. Disadvantages of Synchronous Communication
   

   View Slide

7. inform about
   update
   Updates need to propagate
   7
   business partners
   business partner
   out of business
   orders
   support
   contracts
   

   View Slide

8. downside of REST:
   client misses update if temporarily down
   8
   business
   partners
   orders
   support
   contracts
   

   View Slide

9. downside of REST:
   app needs to know its clients
   9
   business
   partners
   orders
   support
   contracts
   invoices
   

   View Slide

10. xing.com – core updates have to propagate
    10
    user profile
    data
    jobs
    events
    …
    

    View Slide

11. Asynchronous Messaging
    Advanced Message Queuing Protocol
    

    View Slide

12. amqp.org/about/what
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet, and in the
    Cloud.”
    

    View Slide

13. “open standard application layer protocol for
    message-oriented middleware”
    https://en.wikipedia.org/wiki/
    Advanced_Message_Queuing_Protocol
    “open standard application layer protocol for
    message-oriented middleware”
    

    View Slide

14. Interoperability
    14
    message
    broker
    message
    consumer
    message
    producer
    

    View Slide

15. Messages contain payload and routing key
    15
    Routing Key
    Payload
    •  application data
    – any form, any encoding…
    …and more
    •  Structured app-specific data (properties / attributes)
    •  Headers
    •  Annotations
    •  …
    

    View Slide

16. Message Flow
    16
    P
    Message Broker
    X Q C
    Producer
    Exchange
    Queue
    Consumer
    

    View Slide

17. No information lost if consumer down
    17
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C
    

    View Slide

18. Applications are in power
    18
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C
    

    View Slide

19. Consumer subscribes, without touching producer
    19
    P X
    Q2
    Q1 C1
    C2
    Producer
    Exchange
    Queue
    Consumer
    

    View Slide

20. Publish - Subscribe
    

    View Slide

21. Example Message
    routing key
    <...>user.deleted
    payload
    21
    {
    “user_id”: 42
    }
    

    View Slide

22. Example Message
    routing key
    <...>profile.updated
    payload
    22
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }
    

    View Slide

23. Why not include updated data?
    routing key
    <...>profile.updated
    payload
    23
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }
    

    View Slide

24. No guaranteed order – aim for commutativity
    Order of messages not guarantueed
    Commutativity
    Complement with synchronous API (e.g. REST)
    •  most recent data
    •  leading system
    •  message consumers follow up with REST if necessary
    Small payload
    •  easier on the broker
    24
    

    View Slide

25. No exactly-once delivery – aim for idem-potent messages
    Duplicated messages
    •  possible e.g. if producer re-sends message after connection failure
    •  use idem-potent message handling
    25
    

    View Slide

26. Add producers without touching the consumer
    26
    Producer
    Exchange
    Queue
    Consumer
    P1
    P2
    X Q C
    

    View Slide

27. Centralized tracking of decentralized events
    27
    P1
    P2
    X Q C
    event to be
    tracked
    tracking
    counter ++
    

    View Slide

28. Avoid n-m routing, keep breaking changes manageable
    28
    P1
    P2
    X
    Q2
    Q1 C1
    C2
    P1
    P2 C2
    C1
    

    View Slide

29. Notifications / Events - producer state changed
    29
    Message Broker
    X Q
    P C
    event /
    state change
    

    View Slide

30. Task / Command – consumer state to change
    30
    Message Broker
    X Q
    P C
    event / state
    change
    

    View Slide

31. Task Queue
    

    View Slide

32. Run tasks asynchronously
    32
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing
    

    View Slide

33. Improve response time, avoid downtimes
    33
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing
    

    View Slide

34. Scaling on the fly
    34
    X Q
    C
    C
    P
    migration
    trigger
    data
    migration
    

    View Slide

35. Migration without downtime
    35
    Message Broker
    X Q
    P C
    old system
    user facing
    new system
    

    View Slide

36. Test without impacting users
    36
    Message Broker
    X Q
    P C
    old app
    user facing
    shadow calls
    to new app
    

    View Slide

37. In a nutshell
    •  Interoperability
    •  Improve response times
    •  Avoid downtimes
    •  Scale on the fly
    •  Reduce complexity
    •  Simplify code
    •  Easy to get started
    37
    

    View Slide

38. Decouple all the things!
    

    View Slide

39. Who am I
    Contact
    •  http://www.kpuschke.eu
    •  twitter: @titanoboa42
    •  https://www.xing.com/profile/
    Kerstin_Puschke
    Senior Software Engineer
    at XING Hamburg
    •  xing.com
    social network for business
    professionals
    •  debian, perl, javascript, ruby on
    rails, mysql, redis, riak, . . .
    We’re hiring in Hamburg,
    Barcelona,…
    http://corporate.xing.com/
    english/company/careers-
    at-xing/
    The image cannot be displayed. Your
    computer may not have enough memory
    to open the image, or the image may have
    been corrupted. Restart your computer,
    and then open the file again. If the red x
    still appears, you may have to delete the
    image and then insert it again.
    

    View Slide

40. www.xing.com
    Thank you
    for your
    attention!
    

    View Slide

41. Flexible Routing via different Exchange Types
    Direct Exchange
    •  routes to all queues whose binding_key equals routing_key
    Fanout Exchange
    •  broadcasts to every queue it has a binding for
    •  ignores routing key and queue name
    Topic Exchange
    •  dot-separated words as routing keys, plus wildcards
    •  routing on multiple attributes
    Header Exchange
    •  based on headers, not routing keys
    •  routing on multiple attributes
    41
    

    View Slide