Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decouple all the things - Asynchronous messaging keeps it simple

Decouple all the things - Asynchronous messaging keeps it simple

Talk at DevconTLV, March 2016

Kerstin Puschke

March 22, 2016
Tweet

More Decks by Kerstin Puschke

Other Decks in Technology

Transcript

  1. Decouple all the things
    Asynchronous messaging keeps it simple
    DevconTLV 2016

    View full-size slide

  2. Machine-to-Machine Communication

    View full-size slide

  3. 418 I’m a Teapot

    View full-size slide

  4. Improve Maintainability and Stability by adding
    Asynchronous Communication

    View full-size slide

  5. Outline
    Disadvantages of Synchronous Communication
    AMQP
    Publish-Subscribe
    Task Queue
    Summary
    5

    View full-size slide

  6. Disadvantages of Synchronous Communication

    View full-size slide

  7. inform about
    update
    Updates need to propagate
    7
    business partners
    business partner
    out of business
    orders
    support
    contracts

    View full-size slide

  8. downside of REST:
    client misses update if temporarily down
    8
    business
    partners
    orders
    support
    contracts

    View full-size slide

  9. downside of REST:
    app needs to know its clients
    9
    business
    partners
    orders
    support
    contracts
    invoices

    View full-size slide

  10. xing.com – core updates have to propagate
    10
    user profile
    data
    jobs
    events

    View full-size slide

  11. Asynchronous Messaging
    Advanced Message Queuing Protocol

    View full-size slide

  12. amqp.org/about/what
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet, and in the
    Cloud.”

    View full-size slide

  13. “open standard application layer protocol for
    message-oriented middleware”
    https://en.wikipedia.org/wiki/
    Advanced_Message_Queuing_Protocol
    “open standard application layer protocol for
    message-oriented middleware”

    View full-size slide

  14. Interoperability
    14
    message
    broker
    message
    consumer
    message
    producer

    View full-size slide

  15. Messages contain payload and routing key
    15
    Routing Key
    Payload
    •  application data
    – any form, any encoding…
    …and more
    •  Structured app-specific data (properties / attributes)
    •  Headers
    •  Annotations
    •  …

    View full-size slide

  16. Message Flow
    16
    P
    Message Broker
    X Q C
    Producer
    Exchange
    Queue
    Consumer

    View full-size slide

  17. No information lost if consumer down
    17
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View full-size slide

  18. Applications are in power
    18
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View full-size slide

  19. Consumer subscribes, without touching producer
    19
    P X
    Q2
    Q1 C1
    C2
    Producer
    Exchange
    Queue
    Consumer

    View full-size slide

  20. Publish - Subscribe

    View full-size slide

  21. Example Message
    routing key
    <...>user.deleted
    payload
    21
    {
    “user_id”: 42
    }

    View full-size slide

  22. Example Message
    routing key
    <...>profile.updated
    payload
    22
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View full-size slide

  23. Why not include updated data?
    routing key
    <...>profile.updated
    payload
    23
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View full-size slide

  24. No guaranteed order – aim for commutativity
    Order of messages not guarantueed
    Commutativity
    Complement with synchronous API (e.g. REST)
    •  most recent data
    •  leading system
    •  message consumers follow up with REST if necessary
    Small payload
    •  easier on the broker
    24

    View full-size slide

  25. No exactly-once delivery – aim for idem-potent messages
    Duplicated messages
    •  possible e.g. if producer re-sends message after connection failure
    •  use idem-potent message handling
    25

    View full-size slide

  26. Add producers without touching the consumer
    26
    Producer
    Exchange
    Queue
    Consumer
    P1
    P2
    X Q C

    View full-size slide

  27. Centralized tracking of decentralized events
    27
    P1
    P2
    X Q C
    event to be
    tracked
    tracking
    counter ++

    View full-size slide

  28. Avoid n-m routing, keep breaking changes manageable
    28
    P1
    P2
    X
    Q2
    Q1 C1
    C2
    P1
    P2 C2
    C1

    View full-size slide

  29. Notifications / Events - producer state changed
    29
    Message Broker
    X Q
    P C
    event /
    state change

    View full-size slide

  30. Task / Command – consumer state to change
    30
    Message Broker
    X Q
    P C
    event / state
    change

    View full-size slide

  31. Run tasks asynchronously
    32
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View full-size slide

  32. Improve response time, avoid downtimes
    33
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View full-size slide

  33. Scaling on the fly
    34
    X Q
    C
    C
    P
    migration
    trigger
    data
    migration

    View full-size slide

  34. Migration without downtime
    35
    Message Broker
    X Q
    P C
    old system
    user facing
    new system

    View full-size slide

  35. Test without impacting users
    36
    Message Broker
    X Q
    P C
    old app
    user facing
    shadow calls
    to new app

    View full-size slide

  36. In a nutshell
    •  Interoperability
    •  Improve response times
    •  Avoid downtimes
    •  Scale on the fly
    •  Reduce complexity
    •  Simplify code
    •  Easy to get started
    37

    View full-size slide

  37. Decouple all the things!

    View full-size slide

  38. Who am I
    Contact
    •  http://www.kpuschke.eu
    •  twitter: @titanoboa42
    •  https://www.xing.com/profile/
    Kerstin_Puschke
    Senior Software Engineer
    at XING Hamburg
    •  xing.com
    social network for business
    professionals
    •  debian, perl, javascript, ruby on
    rails, mysql, redis, riak, . . .
    We’re hiring in Hamburg,
    Barcelona,…
    http://corporate.xing.com/
    english/company/careers-
    at-xing/
    The image cannot be displayed. Your
    computer may not have enough memory
    to open the image, or the image may have
    been corrupted. Restart your computer,
    and then open the file again. If the red x
    still appears, you may have to delete the
    image and then insert it again.

    View full-size slide

  39. www.xing.com
    Thank you
    for your
    attention!

    View full-size slide

  40. Flexible Routing via different Exchange Types
    Direct Exchange
    •  routes to all queues whose binding_key equals routing_key
    Fanout Exchange
    •  broadcasts to every queue it has a binding for
    •  ignores routing key and queue name
    Topic Exchange
    •  dot-separated words as routing keys, plus wildcards
    •  routing on multiple attributes
    Header Exchange
    •  based on headers, not routing keys
    •  routing on multiple attributes
    41

    View full-size slide