Why You Shouldn't Write a Database

Talk given at the October 2015 Denver Go meetup.

Transcript

1. Why You Shouldn’t Write a Database Ben Johnson
2. Let’s define “database”
3. low-level LevelDB, BerkeleyDB, LMDB
4. Direct interface to OS
 Files, pages, & blocks Responsible for data integrity Typically key/value
5. high-level SQLite, MySQL, PostgreSQL, Mongo, InfluxDB, etc
6. Builds on low-level stores
 Rows, tables, indexes Interfaces with end user Relational, document, time series
7. Me
8. SkyDB ReportifyDB BoltDB InfluxDB
9. There are n+1 reasons not to write a database
10. #1. High barrier to entry
11. How to write a database
13. legit
14. sorta, but not quite
15. seriously?
16. suggests using XML
17. nope
18. Available resources?
19. Research papers!
20. Research papers! Narrowly focused (indexing, storage, locks)
21. Research papers! High level (little or no code)
22. Research papers! Assumes a Ph.D
23. Read source code!
24. Read source code! MySQL >1M SLOC
25. Read source code! Even small databases are 10KLOC+
26. There is no Writing Databases 101
27. #2. Debugging Sucks
28. Bugs are catastrophic Cause corruption, loss of data integrity
29. Bugs are catastrophic Users with data loss are very unhappy
30. Find a good hex editor
31. Find a good hex editor (I use Hex Fiend)
33. This is not what a database looks like
35. THIS is what a database looks like
36. Debugging w/o data Users usually can’t release their data
37. #3. Tradeoffs
38. Users don’t understand tradeoffs
40. Read Optimized vs Write Optimized
41. CPU Bound
 vs
 IO Bound
42. https://gist.github.com/jboner/2841832
43. Features are liabilities It’s not if you have bugs, it’s how many
44. #4. Limited Community
45. Very few people who have written a production database
46. TONS of people who will tell you why your database sucks
47. Databases are hard
48. The End
49. The End (Just kidding!)
50. Not scared off yet?
51. Only 2 reasons to write a database:
52. #1. To learn
53. Data Integrity Indexing Isolation levels Recovery Prefetching Parser & Lexers Replication Snapshotting Transactions Materialized View Referential Integrity Query Compilation Query Planning Query Optimization Serializability Write Ahead Log Memory Profiling Redo/Undo Log Snapshot Isolation Recovery Two-Phase Commit Quorums
54. #2. Gain efficiency (For a specific use case)
55. The more generic your database, the fewer assumptions you can make
56. Go write a database!
57. The End @benbjohnson
58. Questions? @benbjohnson