Why You Shouldn't Write a Database

6c76488dff9b5d9a872dff88f008f88e?s=47 benbjohnson
October 22, 2015

Why You Shouldn't Write a Database

Talk given at the October 2015 Denver Go meetup.

6c76488dff9b5d9a872dff88f008f88e?s=128

benbjohnson

October 22, 2015
Tweet

Transcript

  1. Why You Shouldn’t Write a Database Ben Johnson

  2. Let’s define “database”

  3. low-level LevelDB, BerkeleyDB, LMDB

  4. Direct interface to OS
 Files, pages, & blocks Responsible for

    data integrity Typically key/value
  5. high-level SQLite, MySQL, PostgreSQL, Mongo, InfluxDB, etc

  6. Builds on low-level stores
 Rows, tables, indexes Interfaces with end

    user Relational, document, time series
  7. Me

  8. SkyDB ReportifyDB BoltDB InfluxDB

  9. There are n+1 reasons not to write a database

  10. #1. High barrier to entry

  11. How to write a database

  12. None
  13. legit

  14. sorta, but not quite

  15. seriously?

  16. suggests using XML

  17. nope

  18. Available resources?

  19. Research papers!

  20. Research papers! Narrowly focused (indexing, storage, locks)

  21. Research papers! High level (little or no code)

  22. Research papers! Assumes a Ph.D

  23. Read source code!

  24. Read source code! MySQL >1M SLOC

  25. Read source code! Even small databases are 10KLOC+

  26. There is no Writing Databases 101

  27. #2. Debugging Sucks

  28. Bugs are catastrophic Cause corruption, loss of data integrity

  29. Bugs are catastrophic Users with data loss are very unhappy

  30. Find a good hex editor

  31. Find a good hex editor (I use Hex Fiend)

  32. None
  33. This is not what a database looks like

  34. None
  35. THIS is what a database looks like

  36. Debugging w/o data Users usually can’t release their data

  37. #3. Tradeoffs

  38. Users don’t understand tradeoffs

  39. None
  40. Read Optimized vs Write Optimized

  41. CPU Bound
 vs
 IO Bound

  42. https://gist.github.com/jboner/2841832

  43. Features are liabilities It’s not if you have bugs, it’s

    how many
  44. #4. Limited Community

  45. Very few people who have written a production database

  46. TONS of people who will tell you why your database

    sucks
  47. Databases are hard

  48. The End

  49. The End (Just kidding!)

  50. Not scared off yet?

  51. Only 2 reasons to write a database:

  52. #1. To learn

  53. Data Integrity Indexing Isolation levels Recovery Prefetching Parser & Lexers

    Replication Snapshotting Transactions Materialized View Referential Integrity Query Compilation Query Planning Query Optimization Serializability Write Ahead Log Memory Profiling Redo/Undo Log Snapshot Isolation Recovery Two-Phase Commit Quorums
  54. #2. Gain efficiency (For a specific use case)

  55. The more generic your database, the fewer assumptions you can

    make
  56. Go write a database!

  57. The End @benbjohnson

  58. Questions? @benbjohnson