Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Message Queue is Dead, Long Live the Message Queue

The Message Queue is Dead, Long Live the Message Queue

Ramblings and thought experiments on message queues

Mathias Meyer

April 22, 2013
Tweet

More Decks by Mathias Meyer

Other Decks in Technology

Transcript

  1. the message queue
    is dead
    railsberry 2013, mathias meyer, @roidrage

    View full-size slide

  2. travis-ci.org

    View full-size slide

  3. the golden rules of message queues

    View full-size slide

  4. #1
    everything is a queue

    View full-size slide

  5. www
    email
    upload
    live updates

    View full-size slide

  6. rabbitmq
    activemq
    mqseries
    nsq
    beanstalkd
    sqs
    gearman
    qless
    kafka
    kestrel
    qpid
    sidekiq resque
    zeromq
    celery
    mqseries
    stormmq amqp
    jms
    hornetq
    swiftmq
    delayed_job

    View full-size slide

  7. publish/subscribe

    View full-size slide

  8. publish/subscribe

    View full-size slide

  9. log streaming
    databass

    View full-size slide

  10. unbounded queues

    View full-size slide

  11. #2
    a queue is either empty or infinitely full

    View full-size slide

  12. the art of capacity planning

    View full-size slide

  13. 50 msg * 60 seconds * 30 minutes
    = 90000 msg

    View full-size slide

  14. peak capacity: 120 msg/s

    View full-size slide

  15. 90000 msg / 120 msg/s = 12,5 min

    View full-size slide

  16. #3
    whenever something breaks,
    it's a queue backing up
    or dropping messages

    View full-size slide

  17. databass
    x x
    P
    x

    View full-size slide

  18. retry
    databass
    x x
    P
    x

    View full-size slide

  19. exponential backoff
    databass
    x x
    P
    x

    View full-size slide

  20. handling queues involves business decisions

    View full-size slide

  21. a thought experiment

    View full-size slide

  22. message queues introduce tight coupling

    View full-size slide

  23. hide the queue
    api

    View full-size slide

  24. the queue is dead,
    long live the queue

    View full-size slide

  25. no berries were harmed
    © railsberry

    View full-size slide