Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rocks rocks

Reed Allman
December 04, 2014

Rocks rocks

A high level overview with some details of different ways you can torment yourself trying to build a message queue on disk.

Talk: https://www.youtube.com/watch?v=HTjt6oj-RL4

Reed Allman

December 04, 2014

More Decks by Reed Allman

Other Decks in Technology


  1. Reed Allman Rock solid queues; and other bad puns

  2. Who are you?

  3. That slide telling you who I am and why you

    should listen to me: I work at Previously, I was an undergrad at Auburn There’s really no good reason to listen to me
  4. Now that we’ve gotten that out of the way

  5. War Eagle

  6. Why are you doing this?

  7. Some people don’t like it when you throw their things

    on the floor
  8. My thoughts:

  9. First, a legend: Enqueue: put a message on a queue

    Dequeue: pull a message off a queue, but don’t delete Delete/Ack: delete the message from the queue Timeout: After a timeout, a message goes back onto its queue Count: Increment or decrement queue size (can’t do tail-head)
  10. Stepping stones to Rocks

  11. None
  12. 1) MongoDB

  13. Turns out, it doesn’t scale

  14. None
  15. • message “windows” to simulate queues • global database lock

    • networks to cross
  16. Result: 300-500 msg/sec/queue peak

  17. 2) Leveldb

  18. No more funny message window business

  19. No weird locking service to escape db locks

  20. No network to cross

  21. None
  22. And then there were tests

  23. Write Stalls!

  24. Results: 0-2.5k msg/sec/queue

  25. None
  26. Now what?

  27. If only we could change some settings in Leveldb...

  28. fork?

  29. None
  30. 3) Rocksdb

  31. So many knobs (13 vs. 88)

  32. Which ones to turn?

  33. None
  34. Write stalls?

  35. Better, but still happening under high load

  36. Enter NewDefaultUniversalStyleCompactionOptions()

  37. *deep breath*

  38. Result: 4k msg/sec/queue

  39. None
  40. None
  41. We can do better

  42. high latency variance… for dequeue only

  43. None
  44. it.Seek(prefix) -> it.Seek(q.head)

  45. result: 10k msgs/sec/queue (and smooth latency)

  46. next!

  47. What’s wrong with this picture?

  48. ci.Put(“count”, counts++) // the gist -> ci.Merge(“count”, 1)

  49. Much better

  50. result: 26k msgs/sec/queue *other optimizations happened worth ++5k/s

  51. What’s next?

  52. None
  53. • Shard on queues • Gossip a little bit •

    Commit Log • Transactions • RPC • Proxies
  54. … probably all the stuff Andy just talked about

  55. We’re bottlenecked on http front end / GC now for

    per node throughput. Haven’t even tested rocks’ limits.
  56. None
  57. I started writing a fancy storage interface queue benchmarker thing

    but ran out of time. If for some reason these things interest you, it’s here: http://github.com/rdallman/rocksrocks
  58. Questions? Comments? Are you awake?