Small Data: Storage For The Rest Of Us

Transcript

1. Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF

   US

2. Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at

   Far too many hobbies

3. BIG DATA What does it mean?

4. BIG DATA What does it mean? What is 'big'?

5. 1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?

6. Scalable designs are a tradeoff: NOW LATER vs

7. Small company? Agency? Focus on ease of change, not scalability

8. You don't need to scale from day one But always

   leave yourself scaling points

9. Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'

10. Rapid development It's all about schema change overhead

11. Explicit Schema ID int Name text Weight uint 1 2

    3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }

12. Silent Failure { "id": 342, "name": "David", "weight": 74, }

    { "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }

13. Continuous deployment It's 11pm. Do you know where your locks

    are?

14. Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type

    changes

15. Hardware choice ZOMG RUN IT ON THE CLOUD

16. VMs are TERRIBLE at IO Up to 10x slowdown, even

    with VT-d.

17. Memory is king Your database loves it. Don't let other

    apps steal it.

18. Adding more power goes far Especially with PostgreSQL or read-only

    replicas

19. Scaling Breakpoints

20. Sharding point Datasets paritioned by primary key

21. Vertical split Entirely unrelated tables

22. Denormalisation It's not free!

23. Consistency leeway Can you take inconsistent views?

24. Load Shapes

25. Read-heavy Write-heavy Large size

26. Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums

    Amazon Glacier Eventbrite Logging

27. Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache

    / flat files Many indexes Fewer indexes

28. Extremes

29. Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency

    Extreme Size Sacrifice query time

30. Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash

    Extreme Auditability True append only storage

31. SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash

    Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature

32. Big Data isn't one thing It depends on type, size,

    complexity, throughput, latency...

33. Focus on the current problems Future problems don't matter if

    you never get there

34. Efficiency and iterating fast matters The smaller you are, the

    more time is worth

35. Good architecture affects product You're not writing a system in

    a vacuum

36. Thanks. Andrew Godwin @andrewgodwin