BYOMQ: Build Your Own Message Queue

Why we [re]built IronMQ

Transcript

1. BYOMQ: Why We [re]Built
   IronMQ
   Reed Allman
   @rdallman10
   

   View Slide

2. Making [another!] Case for NIH
   Reed Allman
   @rdallman10
   

   View Slide

3. Beating A Dead Horse [again!]
   Reed Allman
   @rdallman10
   

   View Slide

4. There
   

   View Slide

5. Are
   

   View Slide

6. Too
   

   View Slide

7. Many
   

   View Slide

8. Options
   

   View Slide

9. Me Showing .gifs for 30 Minutes
   Reed Allman
   @rdallman10
   

   View Slide

10. View Slide

11. NIH: Not Invented Here Syndrome
    A quick refresher:
    ...the tendency towards reinventing the wheel (reimplementing something that is
    already available) based on the belief that in-house developments are inherently better
    suited, more secure, more controlled, quicker to develop, and incur lower overall cost
    (including maintenance cost) than using existing implementations.
    

    View Slide

12. Proof by Contradiction
    

    View Slide

13. Throw this at ${USE_CASE}
    

    View Slide

14. Out of the Box
    ●
    Massive scale on commodity hardware in hours of easy set up
    ●
    Battle tested, many bugs worked out, lots of docs, tools, guides, etc.
    ●
    Operate JVM, hard not to code on JVM
    ●
    Any and all problems associated with any of the boxes or how they fit together
    ●
    And these are open source! Many boxes are not and suffer vendor lock-in
    ●
    At mercy of those who understand it / time for fixing bugs, making improvements
    

    View Slide

15. Could you do better?
    

    View Slide

16. Different people come from different backgrounds and, based on that, find different
    tools useful. Ultimately you need to use the tools that let you get your job done most
    effectively, however that is defined in your particular case.
    - Russ Cox
    Go
    

    View Slide

17. If it's a core business function -- do it yourself, no matter what.
    - Joel Spolsky
    Excel
    

    View Slide

18. I felt that my team, which was supposed to be made up of distributed systems
    engineers, was really acting more as distributed system plumbers.
    - Jay Kreps
    Kafka
    

    View Slide

19. View Slide

20. Pain meow or later?
    

    View Slide

21. To the Queues!
    

    View Slide

22. Our Requirements:
    ●
    Easy to run (single binary ideal) in any environment & not hosted
    ●
    Persisted, consistent and performant
    ●
    Highly available -- fast recovery, no data loss, minimal downtime
    ●
    Horizontally scalable
    ●
    Timing based
    ●
    FIFO
    ●
    >= 1 time delivery
    ●
    Multi-Tenant capable+
    ●
    Preferably built in Go, HTTP interface
    

    View Slide

23. ...Scalable, Low-Latency, Persistent MQ for Job Processing
    

    View Slide

24. View Slide

25. View Slide

26. What, briefly
    ●
    Single Go binary per server
    ●
    RocksDB embedded to store all metadata and messages
    ●
    Built the distributed database - viewstamped replication, multi-master, gossip
    membership - scales across queues across nodes, auto-balanced
    ●
    Linear scaling factor of .98
    ●
    Recovery time < 300ms
    ●
    Simple HTTP interface, stupidly simple clients
    ●
    Enqueue / Dequeue avg latency < 10ms for < 4KB messages
    

    View Slide

27. View Slide

28. boom
    

    View Slide

29. Results
    

    View Slide

30. Full understanding and control makes for quick fixes, easy
    to add features
    

    View Slide

31. Don’t have to build a daemon to add features, can bake
    everything in
    

    View Slide

32. Takes a long time to get to stable
    

    View Slide

33. Make all your own tools, plumb any data you want out
    

    View Slide

34. Build exactly what you need, optimized for what you are
    doing
    

    View Slide

35. Use the language(s) you know and love
    

    View Slide

36. My attempt to say profound shit:
    Be a builder, not a plumber.
    

    View Slide

37. Questions?
    

    View Slide