$30 off During Our Annual Pro Sale. View Details »

Decouple all the things - asynchronous messaging keeps it simple

Decouple all the things - asynchronous messaging keeps it simple

Talk at arrrrcamp 2015

Kerstin Puschke

September 30, 2015
Tweet

More Decks by Kerstin Puschke

Other Decks in Technology

Transcript

  1. Decouple all the things
    Asynchronous messaging keeps it simple
    Arrrrcamp 2015

    View Slide

  2. Communication is Exhausting

    View Slide

  3. RSVP

    View Slide

  4. Ambigous Reply
    4
    accepts
    X
    regrets
    Guests:
    all my kids
    Name:

    View Slide

  5. Ambigous Reply
    5
    accepts
    X
    regrets
    X
    Guests:
    Name:
    Ada

    View Slide

  6. No RSVP required?

    View Slide

  7. Machine-to-Machine Communication

    View Slide

  8. 418 I’m a Teapot

    View Slide

  9. Improve Maintainability and Stability by adding
    Asynchronous Communication

    View Slide

  10. Outline
    Disadvantages of Synchronous Communication
    AMQP
    Publish-Subscribe
    Task Queue
    Code Example & Live Demo
    Summary
    10

    View Slide

  11. Disadvantages of Synchronous Communication

    View Slide

  12. inform about
    update
    Master data updates need to propagate
    12
    business partners
    master data
    business partner
    out of business
    orders
    support
    contracts

    View Slide

  13. downside of REST:
    client misses update if temporarily down
    13
    master data
    orders
    support
    contracts

    View Slide

  14. downside of REST:
    master data app needs to know its clients
    14
    master data
    orders
    support
    contracts
    invoices

    View Slide

  15. xing.com – core updates have to propagate
    15
    core
    user data
    jobs
    events

    View Slide

  16. Asynchronous Messaging
    Advanced Message Queuing Protocol

    View Slide

  17. amqp.org/about/what
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet, and in the
    Cloud.”

    View Slide

  18. “open standard application layer protocol for
    message-oriented middleware”
    https://en.wikipedia.org/wiki/
    Advanced_Message_Queuing_Protocol
    “open standard application layer protocol for
    message-oriented middleware”

    View Slide

  19. Interoperability
    19
    message
    broker
    message
    consumer
    message
    producer

    View Slide

  20. Message Flow
    20
    P
    Message Broker
    X Q C
    Producer
    Exchange
    Queue
    Consumer

    View Slide

  21. Messages contain payload and routing key
    21
    Routing Key
    Payload
    •  application data
    – any form, any encoding…
    …and more
    •  Structured app-specific data (properties / attributes)
    •  Headers
    •  Annotations
    •  …

    View Slide

  22. Applications are in power
    22
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View Slide

  23. No information lost if consumer down
    23
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View Slide

  24. Add consumers without touching producer
    24
    P X
    Q2
    Q1 C1
    C2
    Producer
    Exchange
    Queue
    Consumer

    View Slide

  25. Publish - Subscribe

    View Slide

  26. Different messages for different client needs
    26
    master
    data
    client
    Owns additional data related to master data
    •  User account aka master data record removed
    Caches master data
    •  If master data updated or removed
    Mirrors master data
    •  Updated master data

    View Slide

  27. Example Message
    routing key
    <...>user.deleted
    payload
    27
    {
    “user_id”: 42
    }

    View Slide

  28. Example Message
    routing key
    <...>profile.updated
    payload
    28
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View Slide

  29. Consumer subscribes, publisher does not notice
    29
    P X
    Q2
    Q1 C1
    C2

    View Slide

  30. Why not include updated data?
    routing key
    <...>profile.updated
    payload
    30
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View Slide

  31. No guaranteed order – aim for commutativity
    Order of messages not guarantueed
    Commutativity
    Complement with synchronous API (e.g. REST)
    •  most recent data
    •  leading system
    •  message consumers follow up with REST if necessary
    Small payload
    •  easier on the broker
    •  just enough data for consumer to decide about making REST request
    31

    View Slide

  32. No exactly-once delivery – aim for idem-potent messages
    Duplicated messages
    •  possible e.g. if producer re-sends message after connection failure
    •  use idem-potent message handling
    32

    View Slide

  33. Add producers without touching the consumer
    33
    Producer
    Exchange
    Queue
    Consumer
    P1
    P2
    X Q C

    View Slide

  34. Centralized tracking of decentralized events
    34
    P1
    P2
    X Q C
    event to be
    tracked
    tracking
    counter ++

    View Slide

  35. Avoid n-m routing, keep breaking changes manageable
    35
    P1
    P2
    X
    Q2
    Q1 C1
    C2
    P1
    P2 C2
    C1

    View Slide

  36. Notifications / Events - producer state changed
    36
    Message Broker
    X Q
    P C
    event /
    state change

    View Slide

  37. Task / Command – consumer state to change
    37
    Message Broker
    X Q
    P C
    event / state
    change

    View Slide

  38. Task Queue

    View Slide

  39. Run tasks asynchronously
    39
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View Slide

  40. Improve response time, avoid downtimes, gain reliability
    40
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View Slide

  41. Scaling on the fly
    41
    X Q
    C
    C
    P
    migration
    trigger
    data
    migration

    View Slide

  42. Migration without downtime
    42
    Message Broker
    X Q
    P C
    Old system New system

    View Slide

  43. Test without impacting users
    43
    Message Broker
    X Q
    P C
    old app
    user facing
    shadow calls
    to new app

    View Slide

  44. Hello, World!
    Code Example & Live Demo

    View Slide

  45. Flexible Routing via different Exchange Types
    Direct Exchange
    •  routes to all queues whose binding_key equals routing_key
    Fanout Exchange
    •  broadcasts to every queue it has a binding for
    •  ignores routing key and queue name
    Topic Exchange
    •  dot-separated words as routing keys, plus wildcards
    •  routing on multiple attributes
    Header Exchange
    •  based on headers, not routing keys
    •  routing on multiple attributes
    45

    View Slide

  46. In a nutshell
    •  Interoperability
    •  Improve response times
    •  Avoid downtimes
    •  Robustness & Reliability
    •  Scale on the fly
    •  Reduce complexity
    •  Simplify code
    46

    View Slide

  47. Decouple all your things!

    View Slide

  48. Who am I
    Contact
    •  http://www.kpuschke.eu
    •  twitter: @titanoboa42
    •  https://www.xing.com/profile/
    Kerstin_Puschke
    Senior Software Engineer
    at XING Hamburg
    •  xing.com
    social network for business
    professionals
    •  debian, perl, javascript, ruby on
    rails, mysql, redis, riak, . . .
    We’re hiring!
    http://corporate.xing.com/
    english/company/careers-
    at-xing/

    View Slide

  49. www.xing.com
    Thank you
    for your
    attention!

    View Slide