Why You Shouldn't Write a Database

Transcript

1. Why You Shouldn’t Write a Database Ben Johnson

2. Let’s define “database”

3. low-level LevelDB, BerkeleyDB, LMDB

4. Direct interface to OS
 Files, pages, & blocks Responsible for

   data integrity Typically key/value

5. high-level SQLite, MySQL, PostgreSQL, Mongo, InfluxDB, etc

6. Builds on low-level stores
 Rows, tables, indexes Interfaces with end

   user Relational, document, time series

7. Me

8. SkyDB ReportifyDB BoltDB InfluxDB

9. There are n+1 reasons not to write a database

10. #1. High barrier to entry

11. How to write a database

12. None

13. legit

14. sorta, but not quite

15. seriously?

16. suggests using XML

17. nope

18. Available resources?

19. Research papers!

20. Research papers! Narrowly focused (indexing, storage, locks)

21. Research papers! High level (little or no code)

22. Research papers! Assumes a Ph.D

23. Read source code!

24. Read source code! MySQL >1M SLOC

25. Read source code! Even small databases are 10KLOC+

26. There is no Writing Databases 101

27. #2. Debugging Sucks

28. Bugs are catastrophic Cause corruption, loss of data integrity

29. Bugs are catastrophic Users with data loss are very unhappy

30. Find a good hex editor

31. Find a good hex editor (I use Hex Fiend)

32. None

33. This is not what a database looks like

34. None

35. THIS is what a database looks like

36. Debugging w/o data Users usually can’t release their data

37. #3. Tradeoffs

38. Users don’t understand tradeoffs

39. None

40. Read Optimized vs Write Optimized

41. CPU Bound
 vs
 IO Bound

42. https://gist.github.com/jboner/2841832

43. Features are liabilities It’s not if you have bugs, it’s

    how many

44. #4. Limited Community

45. Very few people who have written a production database

46. TONS of people who will tell you why your database

    sucks

47. Databases are hard

48. The End

49. The End (Just kidding!)

50. Not scared off yet?

51. Only 2 reasons to write a database:

52. #1. To learn

53. Data Integrity Indexing Isolation levels Recovery Prefetching Parser & Lexers

    Replication Snapshotting Transactions Materialized View Referential Integrity Query Compilation Query Planning Query Optimization Serializability Write Ahead Log Memory Profiling Redo/Undo Log Snapshot Isolation Recovery Two-Phase Commit Quorums

54. #2. Gain efficiency (For a specific use case)

55. The more generic your database, the fewer assumptions you can

    make

56. Go write a database!

57. The End @benbjohnson

58. Questions? @benbjohnson