Rocks rocks

A high level overview with some details of different ways you can torment yourself trying to build a message queue on disk.

Talk: https://www.youtube.com/watch?v=HTjt6oj-RL4

Transcript

1. Reed Allman Rock solid queues; and other bad puns
2. Who are you?
3. That slide telling you who I am and why you should listen to me: I work at Previously, I was an undergrad at Auburn There’s really no good reason to listen to me
4. Now that we’ve gotten that out of the way
5. War Eagle
6. Why are you doing this?
7. Some people don’t like it when you throw their things on the floor
8. My thoughts:
9. First, a legend: Enqueue: put a message on a queue Dequeue: pull a message off a queue, but don’t delete Delete/Ack: delete the message from the queue Timeout: After a timeout, a message goes back onto its queue Count: Increment or decrement queue size (can’t do tail-head)
10. Stepping stones to Rocks
12. 1) MongoDB
13. Turns out, it doesn’t scale
15. • message “windows” to simulate queues • global database lock • networks to cross
16. Result: 300-500 msg/sec/queue peak
17. 2) Leveldb
18. No more funny message window business
19. No weird locking service to escape db locks
20. No network to cross
22. And then there were tests
23. Write Stalls!
24. Results: 0-2.5k msg/sec/queue
26. Now what?
27. If only we could change some settings in Leveldb...
28. fork?
30. 3) Rocksdb
31. So many knobs (13 vs. 88)
32. Which ones to turn?
34. Write stalls?
35. Better, but still happening under high load
36. Enter NewDefaultUniversalStyleCompactionOptions()
37. *deep breath*
38. Result: 4k msg/sec/queue
41. We can do better
42. high latency variance… for dequeue only
44. it.Seek(prefix) -> it.Seek(q.head)
45. result: 10k msgs/sec/queue (and smooth latency)
46. next!
47. What’s wrong with this picture?
48. ci.Put(“count”, counts++) // the gist -> ci.Merge(“count”, 1)
49. Much better
50. result: 26k msgs/sec/queue *other optimizations happened worth ++5k/s
51. What’s next?
53. • Shard on queues • Gossip a little bit • Commit Log • Transactions • RPC • Proxies
54. … probably all the stuff Andy just talked about
55. We’re bottlenecked on http front end / GC now for per node throughput. Haven’t even tested rocks’ limits.
57. I started writing a fancy storage interface queue benchmarker thing but ran out of time. If for some reason these things interest you, it’s here: http://github.com/rdallman/rocksrocks
58. Questions? Comments? Are you awake?