Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Small Data: Storage For The Rest Of Us

Small Data: Storage For The Rest Of Us

A talk I gave at PyWaw Summit 2015.

Andrew Godwin

May 26, 2015

More Decks by Andrew Godwin

Other Decks in Programming


  1. Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF

  2. Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at

    Far too many hobbies
  3. BIG DATA What does it mean?

  4. BIG DATA What does it mean? What is 'big'?

  5. 1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?

  6. Scalable designs are a tradeoff: NOW LATER vs

  7. Small company? Agency? Focus on ease of change, not scalability

  8. You don't need to scale from day one But always

    leave yourself scaling points
  9. Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'

  10. Rapid development It's all about schema change overhead

  11. Explicit Schema ID int Name text Weight uint 1 2

    3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
  12. Silent Failure { "id": 342, "name": "David", "weight": 74, }

    { "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
  13. Continuous deployment It's 11pm. Do you know where your locks

  14. Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type

  15. Hardware choice ZOMG RUN IT ON THE CLOUD

  16. VMs are TERRIBLE at IO Up to 10x slowdown, even

    with VT-d.
  17. Memory is king Your database loves it. Don't let other

    apps steal it.
  18. Adding more power goes far Especially with PostgreSQL or read-only

  19. Scaling Breakpoints

  20. Sharding point Datasets paritioned by primary key

  21. Vertical split Entirely unrelated tables

  22. Denormalisation It's not free!

  23. Consistency leeway Can you take inconsistent views?

  24. Load Shapes

  25. Read-heavy Write-heavy Large size

  26. Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums

    Amazon Glacier Eventbrite Logging
  27. Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache

    / flat files Many indexes Fewer indexes
  28. Extremes

  29. Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency

    Extreme Size Sacrifice query time
  30. Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash

    Extreme Auditability True append only storage
  31. SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash

    Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
  32. Big Data isn't one thing It depends on type, size,

    complexity, throughput, latency...
  33. Focus on the current problems Future problems don't matter if

    you never get there
  34. Efficiency and iterating fast matters The smaller you are, the

    more time is worth
  35. Good architecture affects product You're not writing a system in

    a vacuum
  36. Thanks. Andrew Godwin @andrewgodwin