Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BYOMQ: Build Your Own Message Queue

BYOMQ: Build Your Own Message Queue

Why we [re]built IronMQ

Reed Allman

April 08, 2016
Tweet

More Decks by Reed Allman

Other Decks in Technology

Transcript

  1. BYOMQ: Why We [re]Built
    IronMQ
    Reed Allman
    @rdallman10

    View Slide

  2. Making [another!] Case for NIH
    Reed Allman
    @rdallman10

    View Slide

  3. Beating A Dead Horse [again!]
    Reed Allman
    @rdallman10

    View Slide

  4. There

    View Slide

  5. Are

    View Slide

  6. Too

    View Slide

  7. Many

    View Slide

  8. Options

    View Slide

  9. Me Showing .gifs for 30 Minutes
    Reed Allman
    @rdallman10

    View Slide

  10. View Slide

  11. NIH: Not Invented Here Syndrome
    A quick refresher:
    ...the tendency towards reinventing the wheel (reimplementing something that is
    already available) based on the belief that in-house developments are inherently better
    suited, more secure, more controlled, quicker to develop, and incur lower overall cost
    (including maintenance cost) than using existing implementations.

    View Slide

  12. Proof by Contradiction

    View Slide

  13. Throw this at ${USE_CASE}

    View Slide

  14. Out of the Box

    Massive scale on commodity hardware in hours of easy set up

    Battle tested, many bugs worked out, lots of docs, tools, guides, etc.

    Operate JVM, hard not to code on JVM

    Any and all problems associated with any of the boxes or how they fit together

    And these are open source! Many boxes are not and suffer vendor lock-in

    At mercy of those who understand it / time for fixing bugs, making improvements

    View Slide

  15. Could you do better?

    View Slide

  16. Different people come from different backgrounds and, based on that, find different
    tools useful. Ultimately you need to use the tools that let you get your job done most
    effectively, however that is defined in your particular case.
    - Russ Cox
    Go

    View Slide

  17. If it's a core business function -- do it yourself, no matter what.
    - Joel Spolsky
    Excel

    View Slide

  18. I felt that my team, which was supposed to be made up of distributed systems
    engineers, was really acting more as distributed system plumbers.
    - Jay Kreps
    Kafka

    View Slide

  19. View Slide

  20. Pain meow or later?

    View Slide

  21. To the Queues!

    View Slide

  22. Our Requirements:

    Easy to run (single binary ideal) in any environment & not hosted

    Persisted, consistent and performant

    Highly available -- fast recovery, no data loss, minimal downtime

    Horizontally scalable

    Timing based

    FIFO

    >= 1 time delivery

    Multi-Tenant capable+

    Preferably built in Go, HTTP interface

    View Slide

  23. ...Scalable, Low-Latency, Persistent MQ for Job Processing

    View Slide

  24. View Slide

  25. View Slide

  26. What, briefly

    Single Go binary per server

    RocksDB embedded to store all metadata and messages

    Built the distributed database - viewstamped replication, multi-master, gossip
    membership - scales across queues across nodes, auto-balanced

    Linear scaling factor of .98

    Recovery time < 300ms

    Simple HTTP interface, stupidly simple clients

    Enqueue / Dequeue avg latency < 10ms for < 4KB messages

    View Slide

  27. View Slide

  28. boom

    View Slide

  29. Results

    View Slide

  30. Full understanding and control makes for quick fixes, easy
    to add features

    View Slide

  31. Don’t have to build a daemon to add features, can bake
    everything in

    View Slide

  32. Takes a long time to get to stable

    View Slide

  33. Make all your own tools, plumb any data you want out

    View Slide

  34. Build exactly what you need, optimized for what you are
    doing

    View Slide

  35. Use the language(s) you know and love

    View Slide

  36. My attempt to say profound shit:
    Be a builder, not a plumber.

    View Slide

  37. Questions?

    View Slide