Why You Shouldn't Write a Database

Talk given at the October 2015 Denver Go meetup.

Transcript

1. Why You Shouldn’t
   Write a Database
   Ben Johnson
   

   View Slide

2. Let’s define
   “database”
   

   View Slide

3. low-level
   LevelDB, BerkeleyDB, LMDB
   

   View Slide

4. Direct interface to OS

   Files, pages, & blocks
   Responsible for data integrity
   Typically key/value
   

   View Slide

5. high-level
   SQLite, MySQL, PostgreSQL, Mongo, InfluxDB, etc
   

   View Slide

6. Builds on low-level stores

   Rows, tables, indexes
   Interfaces with end user
   Relational, document, time series
   

   View Slide

7. Me
   

   View Slide

8. SkyDB
   ReportifyDB
   BoltDB
   InfluxDB
   

   View Slide

9. There are n+1 reasons
   not to write a database
   

   View Slide

10. #1. High barrier to entry
    

    View Slide

11. How to write a database
    

    View Slide

12. View Slide

13. legit
    

    View Slide

14. sorta,
    but not quite
    

    View Slide

15. seriously?
    

    View Slide

16. suggests
    using XML
    

    View Slide

17. nope
    

    View Slide

18. Available resources?
    

    View Slide

19. Research papers!
    

    View Slide

20. Research papers!
    Narrowly focused (indexing, storage, locks)
    

    View Slide

21. Research papers!
    High level (little or no code)
    

    View Slide

22. Research papers!
    Assumes a Ph.D
    

    View Slide

23. Read source code!
    

    View Slide

24. Read source code!
    MySQL >1M SLOC
    

    View Slide

25. Read source code!
    Even small databases are 10KLOC+
    

    View Slide

26. There is no
    Writing Databases 101
    

    View Slide

27. #2. Debugging Sucks
    

    View Slide

28. Bugs are catastrophic
    Cause corruption, loss of data integrity
    

    View Slide

29. Bugs are catastrophic
    Users with data loss are very unhappy
    

    View Slide

30. Find a good hex editor
    

    View Slide

31. Find a good hex editor
    (I use Hex Fiend)
    

    View Slide

32. View Slide

33. This is not what a
    database looks like
    

    View Slide

34. View Slide

35. THIS is what a
    database looks like
    

    View Slide

36. Debugging w/o data
    Users usually can’t release their data
    

    View Slide

37. #3. Tradeoffs
    

    View Slide

38. Users don’t
    understand tradeoffs
    

    View Slide

39. View Slide

40. Read Optimized
    vs
    Write Optimized
    

    View Slide

41. CPU Bound

    vs

    IO Bound
    

    View Slide

42. https://gist.github.com/jboner/2841832
    

    View Slide

43. Features are liabilities
    It’s not if you have bugs, it’s how many
    

    View Slide

44. #4. Limited Community
    

    View Slide

45. Very few people who
    have written a
    production database
    

    View Slide

46. TONS of people who
    will tell you why
    your database sucks
    

    View Slide

47. Databases are hard
    

    View Slide

48. The End
    

    View Slide

49. The End
    (Just kidding!)
    

    View Slide

50. Not scared off yet?
    

    View Slide

51. Only 2 reasons to
    write a database:
    

    View Slide

52. #1. To learn
    

    View Slide

53. Data Integrity
    Indexing
    Isolation levels
    Recovery
    Prefetching
    Parser & Lexers
    Replication
    Snapshotting
    Transactions
    Materialized View
    Referential Integrity
    Query Compilation
    Query Planning
    Query Optimization
    Serializability
    Write Ahead Log
    Memory Profiling
    Redo/Undo Log
    Snapshot Isolation
    Recovery
    Two-Phase Commit
    Quorums
    

    View Slide

54. #2. Gain efficiency
    (For a specific use case)
    

    View Slide

55. The more generic your database,
    the fewer assumptions you can make
    

    View Slide

56. Go write a database!
    

    View Slide

57. The End
    @benbjohnson
    

    View Slide

58. Questions?
    @benbjohnson
    

    View Slide