Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Databases in the Real World
Search
Andrew Godwin
August 04, 2014
Programming
2
600
Small Data: Databases in the Real World
A talk I gave at PyCon AU 2014.
Andrew Godwin
August 04, 2014
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
320
Django Through The Years
andrewgodwin
0
210
Writing Maintainable Software At Scale
andrewgodwin
0
450
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
360
Async, Python, and the Future
andrewgodwin
2
670
How To Break Django: With Async
andrewgodwin
1
730
Taking Django's ORM Async
andrewgodwin
0
730
The Long Road To Asynchrony
andrewgodwin
0
660
The Scientist & The Engineer
andrewgodwin
1
770
Other Decks in Programming
See All in Programming
AWS Serverless Application Model入門_20250708
smatsuzaki
0
110
画像コンペでのベースラインモデルの育て方
tattaka
3
1.8k
生成AI、実際どう? - ニーリーの場合
nealle
0
140
実践 Dev Containers × Claude Code
touyu
1
220
AI OCR API on Lambdaを Datadogで可視化してみた
nealle
0
130
あのころの iPod を どうにか再生させたい
orumin
2
2.5k
Comparing decimals in Swift Testing
417_72ki
0
170
バイブコーディングの正体——AIエージェントはソフトウェア開発を変えるか?
stakaya
5
980
TanStack DB ~状態管理の新しい考え方~
bmthd
2
170
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
1
310
React 使いじゃなくても知っておきたい教養としての React
oukayuka
18
5.8k
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
lemonade_37
2
480
Featured
See All Featured
Being A Developer After 40
akosma
90
590k
Building Applications with DynamoDB
mza
96
6.6k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
6k
How to train your dragon (web standard)
notwaldorf
96
6.2k
Thoughts on Productivity
jonyablonski
69
4.8k
Done Done
chrislema
185
16k
Building Adaptive Systems
keathley
43
2.7k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
31
2.2k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA REAL WORLD DATABASES IN THE
Andrew Godwin Core Developer Senior Engineer
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
None
Sharding point Vertical split Consistency leeway
Sharding point Datasets paritioned by primary key
Migration plan Implement consistent hashing on primary key Make large
number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
Vertical split Entirely unrelated tables
Migration plan Replicate database to new server Route split tables
there, disable replication - or - Slowly backfill new datastore with fallback lookup
Denormalisation It's not free!
Migration plan Add NULL fields to dependent tables App code
to fetch and fill if not present Possibly prefill on save of new items
Consistency leeway Can you take inconsistent views?
Migration plan Change your site! Talk to your designers! Deliberately
introduce inconsistency!
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks! Andrew Godwin @andrewgodwin
[email protected]
are hiring!