Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decouple all the things - asynchronous messaging keeps it simple

Decouple all the things - asynchronous messaging keeps it simple

Talk at YAPC::EU 2015

Kerstin Puschke

September 04, 2015
Tweet

More Decks by Kerstin Puschke

Other Decks in Technology

Transcript

  1. Decouple all the things
    Asynchronous messaging keeps it simple
    YAPC::EU 2015

    View full-size slide

  2. Communication is Exhausting

    View full-size slide

  3. Missing Replies

    View full-size slide

  4. Ambigous Replies

    View full-size slide

  5. Ambigous Reply
    6
    accepts
    X
    regrets
    Guests:
    all my kids
    Name:

    View full-size slide

  6. Ambigous Reply
    7
    accepts
    X
    regrets
    X
    Guests:
    Name:
    Ada

    View full-size slide

  7. No RSVP required?

    View full-size slide

  8. Machine-to-Machine Communication

    View full-size slide

  9. 418 I’m a Teapot

    View full-size slide

  10. Improve Maintainability and Stability by adding
    Asynchronous Communication

    View full-size slide

  11. Outline
    Disadvantages of Synchronous Communication
    AMQP
    Publish-Subscribe
    Task Queue
    Code Example & Live Demo
    Summary
    13

    View full-size slide

  12. Disadvantages of Synchronous Communication

    View full-size slide

  13. inform about
    update
    Master data updates need to propagate
    15
    customer
    data
    customer out
    of business
    orders
    support
    contracts

    View full-size slide

  14. downside of REST:
    client misses update if temporarily down
    16
    customer
    data
    orders
    support
    contracts

    View full-size slide

  15. downside of REST:
    master data app needs to know its clients
    17
    customer
    data
    orders
    support
    contracts
    invoices

    View full-size slide

  16. xing.com – core updates have to propagate
    18
    core
    user data
    jobs
    events

    View full-size slide

  17. Async. messaging to propagate master data updates
    19
    core
    user data
    events

    jobs

    View full-size slide

  18. Asynchronous Messaging
    Advanced Message Queuing Protocol

    View full-size slide

  19. amqp.org/about/what
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet
    “The capable, commoditized, multi-vendor
    communications ecosystem which AMQP
    enables creates opportunities for commerce
    and innovation which can transform the way
    business is done on the Internet, and in the
    Cloud.”

    View full-size slide

  20. “open standard application layer protocol for
    message-oriented middleware”
    https://en.wikipedia.org/wiki/
    Advanced_Message_Queuing_Protocol
    “open standard application layer protocol for
    message-oriented middleware”

    View full-size slide

  21. Interoperability
    23
    message
    broker
    consumer
    producer

    View full-size slide

  22. Message Flow
    24
    P
    Message Broker
    X Q C
    Producer
    Exchange
    Queue
    Consumer

    View full-size slide

  23. Messages contain payload and routing key
    25
    Routing Key
    Payload
    •  application data
    – any form, any encoding…
    …and more
    •  Structured app-specific data (properties / attributes)
    •  Headers
    •  Annotations
    •  …

    View full-size slide

  24. Applications are in power
    26
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View full-size slide

  25. No information lost if consumer down
    27
    X Q
    Producer
    Exchange
    Queue
    Consumer
    P C

    View full-size slide

  26. Add consumers without touching producer
    28
    P X
    Q2
    Q1 C1
    C2
    Producer
    Exchange
    Queue
    Consumer

    View full-size slide

  27. Publish - Subscribe

    View full-size slide

  28. Different messages for different client needs
    30
    master
    data
    client
    Owns additional user data related to master data
    •  User account aka master data record removed
    Caches master data
    •  If master data updated or removed
    Mirrors master data
    •  Updated master data

    View full-size slide

  29. Example Message
    routing key
    <...>user.deleted
    payload
    31
    {
    “user_id”: 42
    }

    View full-size slide

  30. Example Message
    routing key
    <...>profile.updated
    payload
    32
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View full-size slide

  31. Consumer subscribes, publisher does not notice
    33
    P X
    Q2
    Q1 C1
    C2

    View full-size slide

  32. Why not include updated data?
    routing key
    <...>profile.updated
    payload
    34
    {
    “user_id”: 42,
    “fields”: [“last_name”]
    }

    View full-size slide

  33. No guaranteed order – aim for commutativity
    Order of messages not guarantueed
    Commutativity
    Complement with synchronous API (e.g. REST)
    •  most recent data
    •  leading system
    •  message consumers follow up with REST if necessary
    Small payload
    •  easier on the broker
    •  just enough data for consumer to decide about making REST request
    35

    View full-size slide

  34. No exactly-once delivery – aim for idem-potent messages
    Duplicated messages
    •  possible e.g. if producer re-sends message after connection failure
    •  use idem-potent message handling
    Lost messages
    •  acceptable for usecase?
    •  e.g. consider rebuilding mirror / cache occasionally
    36

    View full-size slide

  35. Add producers without touching the consumer
    37
    Producer
    Exchange
    Queue
    Consumer
    P1
    P2
    X Q C

    View full-size slide

  36. Centralized tracking of decentralized events
    38
    P1
    P2
    X Q C
    event to be
    tracked
    tracking
    counter ++

    View full-size slide

  37. Avoid n-m routing, keep breaking changes manageable
    39
    P1
    P2
    X
    Q2
    Q1 C1
    C2
    P1
    P2 C2
    C1

    View full-size slide

  38. Notifications / Events - producer state changed
    40
    Message Broker
    X Q
    P C
    event /
    state change

    View full-size slide

  39. Task / Command – consumer state to change
    41
    Message Broker
    X Q
    P C
    event / state
    change

    View full-size slide

  40. Run tasks asynchronously
    43
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View full-size slide

  41. Run tasks asynchronously
    routing key
    <...>picture.create
    payload
    44
    {
    “user_id”: 42,
    <…cropping info…>
    }

    View full-size slide

  42. Improve response time, avoid downtimes, gain reliability
    45
    Message Broker
    X Q
    P C
    image upload
    user facing
    image
    processing

    View full-size slide

  43. Scaling on the fly
    46
    X Q
    C
    C
    P
    migration
    trigger
    data
    migration

    View full-size slide

  44. Migration without downtime
    47
    Message Broker
    X Q
    P C
    Old system New system

    View full-size slide

  45. Test without impacting users
    48
    Message Broker
    X Q
    P C
    old app
    user facing
    shadow calls
    to new app

    View full-size slide

  46. Hello, World!
    Code Example & Live Demo

    View full-size slide

  47. Flexible Routing via different Exchange Types
    Direct Exchange
    •  routes to all queues whose binding_key equals routing_key
    Fanout Exchange
    •  broadcasts to every queue it has a binding for
    •  ignores routing key and queue name
    Topic Exchange
    •  dot-separated words as routing keys, plus wildcards
    •  routing on multiple attributes
    Header Exchange
    •  based on headers, not routing keys
    •  routing on multiple attributes
    50

    View full-size slide

  48. In a nutshell
    •  Interoperability
    •  Improve response times
    •  Avoid downtimes
    •  Robustness & Reliability
    •  Scale on the fly
    •  Reduce complexity
    •  Simplify code
    51

    View full-size slide

  49. Decouple All the Things!

    View full-size slide

  50. Who am I
    Contact
    •  http://www.kpuschke.eu
    •  twitter: @titanoboa42
    •  https://www.xing.com/profile/
    Kerstin_Puschke
    Senior Software Engineer
    at XING Hamburg
    •  xing.com
    social network for business
    professionals
    •  debian, perl, javascript, ruby on
    rails, mysql, redis, riak, . . .
    We’re hiring!
    http://corporate.xing.com/
    english/company/careers-
    at-xing/

    View full-size slide

  51. www.xing.com
    Thank you
    for your
    attention!

    View full-size slide