Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Small Data: Databases in the Real World

Small Data: Databases in the Real World

A talk I gave at PyCon AU 2014.

Andrew Godwin

August 04, 2014

More Decks by Andrew Godwin

Other Decks in Programming


  1. You don't need to scale from day one But always

    leave yourself scaling points
  2. Explicit Schema ID int Name text Weight uint 1 2

    3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
  3. Silent Failure { "id": 342, "name": "David", "weight": 74, }

    { "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
  4. Migration plan Implement consistent hashing on primary key Make large

    number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
  5. Migration plan Replicate database to new server Route split tables

    there, disable replication - or - Slowly backfill new datastore with fallback lookup
  6. Migration plan Add NULL fields to dependent tables App code

    to fetch and fill if not present Possibly prefill on save of new items
  7. Big Data isn't one thing It depends on type, size,

    complexity, throughput, latency...