Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why You Shouldn't Write a Database

benbjohnson
October 22, 2015

Why You Shouldn't Write a Database

Talk given at the October 2015 Denver Go meetup.

benbjohnson

October 22, 2015
Tweet

More Decks by benbjohnson

Other Decks in Technology

Transcript

  1. Why You Shouldn’t
    Write a Database
    Ben Johnson

    View Slide

  2. Let’s define
    “database”

    View Slide

  3. low-level
    LevelDB, BerkeleyDB, LMDB

    View Slide

  4. Direct interface to OS

    Files, pages, & blocks
    Responsible for data integrity
    Typically key/value

    View Slide

  5. high-level
    SQLite, MySQL, PostgreSQL, Mongo, InfluxDB, etc

    View Slide

  6. Builds on low-level stores

    Rows, tables, indexes
    Interfaces with end user
    Relational, document, time series

    View Slide

  7. Me

    View Slide

  8. SkyDB
    ReportifyDB
    BoltDB
    InfluxDB

    View Slide

  9. There are n+1 reasons
    not to write a database

    View Slide

  10. #1. High barrier to entry

    View Slide

  11. How to write a database

    View Slide

  12. View Slide

  13. legit

    View Slide

  14. sorta,
    but not quite

    View Slide

  15. seriously?

    View Slide

  16. suggests
    using XML

    View Slide

  17. nope

    View Slide

  18. Available resources?

    View Slide

  19. Research papers!

    View Slide

  20. Research papers!
    Narrowly focused (indexing, storage, locks)

    View Slide

  21. Research papers!
    High level (little or no code)

    View Slide

  22. Research papers!
    Assumes a Ph.D

    View Slide

  23. Read source code!

    View Slide

  24. Read source code!
    MySQL >1M SLOC

    View Slide

  25. Read source code!
    Even small databases are 10KLOC+

    View Slide

  26. There is no
    Writing Databases 101

    View Slide

  27. #2. Debugging Sucks

    View Slide

  28. Bugs are catastrophic
    Cause corruption, loss of data integrity

    View Slide

  29. Bugs are catastrophic
    Users with data loss are very unhappy

    View Slide

  30. Find a good hex editor

    View Slide

  31. Find a good hex editor
    (I use Hex Fiend)

    View Slide

  32. View Slide

  33. This is not what a
    database looks like

    View Slide

  34. View Slide

  35. THIS is what a
    database looks like

    View Slide

  36. Debugging w/o data
    Users usually can’t release their data

    View Slide

  37. #3. Tradeoffs

    View Slide

  38. Users don’t
    understand tradeoffs

    View Slide

  39. View Slide

  40. Read Optimized
    vs
    Write Optimized

    View Slide

  41. CPU Bound

    vs

    IO Bound

    View Slide

  42. https://gist.github.com/jboner/2841832

    View Slide

  43. Features are liabilities
    It’s not if you have bugs, it’s how many

    View Slide

  44. #4. Limited Community

    View Slide

  45. Very few people who
    have written a
    production database

    View Slide

  46. TONS of people who
    will tell you why
    your database sucks

    View Slide

  47. Databases are hard

    View Slide

  48. The End

    View Slide

  49. The End
    (Just kidding!)

    View Slide

  50. Not scared off yet?

    View Slide

  51. Only 2 reasons to
    write a database:

    View Slide

  52. #1. To learn

    View Slide

  53. Data Integrity
    Indexing
    Isolation levels
    Recovery
    Prefetching
    Parser & Lexers
    Replication
    Snapshotting
    Transactions
    Materialized View
    Referential Integrity
    Query Compilation
    Query Planning
    Query Optimization
    Serializability
    Write Ahead Log
    Memory Profiling
    Redo/Undo Log
    Snapshot Isolation
    Recovery
    Two-Phase Commit
    Quorums

    View Slide

  54. #2. Gain efficiency
    (For a specific use case)

    View Slide

  55. The more generic your database,
    the fewer assumptions you can make

    View Slide

  56. Go write a database!

    View Slide

  57. The End
    @benbjohnson

    View Slide

  58. Questions?
    @benbjohnson

    View Slide