HanoiDB - and Other Riak Hacks

Keynote presentation by Kresten Krab Thorup from Erlang Workshop 2012, at ICFP Copenhagen. Code at http://github.com/krestenkrab/hanoidb

Transcript

1. HanoiDB –and other Riak Hacks Kresten Krab Thorup Trifork @drkrab
2. 2 Riak HanoiDB Erjang / Triq Common Medicine Card Riak Java Client DeltaZip Relai Sync iOS/ Android RiakMongo Adapter BTree Backend
3. 3 Riak HanoiDB Erjang / Triq Common Medicine Card Riak Java Client DeltaZip Relai iOS/ Android RiakMongo Adapter BTree Backend Sync
4. Common Medicine Card 4
5. Distributed Architecture 5 ▪ Availability: Run in multiple data centers ▪ Scalability: Prepare the system for expected growth
6. Data Kinds 6 Person Data ~5M entities read/write Audit Log ~5B entries mostly write Bitcask LevelDB
7. Riak Backends 7 Bitcask Keys in Memory Pure Erlang (almost) LevelDB Keys on Disk C++ (mostly) Both use log structured [append-only] files, and thus need to “garbage collect” old data files occasionally.
8. Random I/O 2.7 MB/s Sequential I/O 213 MB/s Price 0.3 $/MB Spinning Disk 100 x performance!
9. Random 60-300 MB/s Sequential 293-366 MB/s Price 1 $/MB Solid Disk seq io not much faster but 3x expensive
10. 0 100 200 300 400 Random Read Random Write Sequential Read Sequential Write HDD SDD
11. HanoiDB ▪ “LevelDB re-implemented in Erlang” ▪ Log-structured storage ▪ write-once B-trees ▪ organized in “doubling size” levels ▪ Incremental garbage collection ▪ log(N) worst-time insert/lookup
12. HanoiDB Levels 0 1 2 2level K/Vs × 2 log2(N/2) levels ordered key/values
13. HanoiDB lookup
14. HanoiDB compressed block Key-Sequence Datasets on Indelible Storage [Malcolm C. Easton, IBM J. RES. DEVELOP. VOL. 30 NO. 3 MAY 1986] meta data + bloom filter
15. HanoiDB 3 insert
16. HanoiDB 3 2 2, 3 merge
17. HanoiDB 2, 3
18. HanoiDB 7 2, 3
19. HanoiDB 7 1 2, 3
20. HanoiDB 7 1 2, 3 1, 7 merge
21. HanoiDB 2, 3 1, 7
22. HanoiDB 2, 3 1, 7 1, 2, 3, 7 merge
23. HanoiDB 1, 2, 3, 7
24. HanoiDB expensive !?
25. HanoiDB
26. HanoiDB merge merge merge for every insert, do one merge step at each level => guarantees “room” [1] J. L. Bentley and J. B. Saxe*. Decomposable searching problems I: Static-to-dynamic transformation. J. Algorithms 1(4):301–358, 1980. [2] M. H. Overmars* and J. van Leeuwen. Worst-case optimal insertion and deletion methods for decomposable searching problems. Inform. Process. Lett. 12:168–173, 1981.
27. HanoiDB the nursery “emulates” the top X levels with an in-memory structure and a transaction log nursery tx log level file append when full
28. hanoidb «gen_server» nursery API
29. hanoidb «gen_server» nursery hanoidb_level #8 «plain_fsm» A-8.data [random,read] top hanoidb_level #9 hanoidb_level #10 next next B-8.data [random,read] API process data structure file Legend lookup
30. Read Caching ▪ Hard to cache in Erlang! ▪ Live terms add GC load ▪ ETS impose copying cost ▪ Binaries-as-a-data structure ▪ Currently: No caching ▪ Spawn process for each lookup
31. hanoidb «gen_server» nursery hanoidb_level #8 «plain_fsm» A-8.data [random,read] hanoidb_merger «plain_fsm» merge_pid top hanoidb_level #9 hanoidb_merger merge_pid hanoidb_level #10 hanoidb_merger merge_pid next next B-8.data [random,read] X-8.data [seq,write] B-8.data [seq,read] A-8.data [seq,read] API nursery.data [append] nursery.log [append] same A+B files opened twice! insert
32. reader reader merger writer merger A-8.data [read_ahead] B-8.data [read_ahead] X-8.data [delayed_write]
33. hanoidb «gen_server» nursery hanoidb_level #8 «plain_fsm» A-8.data [random,read] top hanoidb_level #9 next B-8.data [random,read] API hanoidb_folder BF-8.data [seq,read] hanoidb_folder AF-8.data [seq,read] folding hanoidb_fold_merger {source_order, [pid()]} {fold_result, Key, Value} [back pressure] {level_results, pid(), [{Key,Value}*100]} [back pressure] hard link range fold
34. Comparison HanoiDB LevelDB
35. HanoiDB ▪ Pure Erlang = Easy to Improve & fix ▪ Low memory usage ▪ 2000 lines Erlang vs 30.000 lines C++ ▪ CRC/corruption strategies; Full Riak 2i support; expiry support
36. Erlang ▪ Erlang’s I/O system does all the heavy lifting; writing this with “async callbacks” would have been a pain. ▪ Many small processes governing files/ resources ▪ Error handling for process composites
37. Erlang: Be Maestro of IOPS @drkrab
38. Trifork We do Professional Services for Riak. Call us!