Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Small Data: Storage For The Rest Of Us
Andrew Godwin
May 26, 2015
Programming
1
370
Small Data: Storage For The Rest Of Us
A talk I gave at PyWaw Summit 2015.
Andrew Godwin
May 26, 2015
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Writing Maintainable Software At Scale
andrewgodwin
0
190
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
150
Async, Python, and the Future
andrewgodwin
2
440
How To Break Django: With Async
andrewgodwin
1
420
Taking Django's ORM Async
andrewgodwin
0
400
The Long Road To Asynchrony
andrewgodwin
0
420
The Scientist & The Engineer
andrewgodwin
1
440
Pioneering Real-Time
andrewgodwin
0
190
Just Add Await: Retrofitting Async Into Django
andrewgodwin
2
1.2k
Other Decks in Programming
See All in Programming
TokyoR#103_DataProcessing
kilometer
0
520
Excelの助けを借りて楽にシナリオを作ろう
rpa_niiyama
0
260
Listかもしれない
irof
0
160
Use KMM to call the API of the National Tax Agency
akkeylab
0
290
LIFFで動く割り勘アプリTATEKAをリリースしてみた話
inoue2002
0
230
Azure Functionsをサクッと開発、サクッとデプロイ/vscodeconf2023-baba
nina01
1
330
和暦を正しく扱うための暦の話
nagise
10
4.8k
子育てとEMと転職と
_atsushisakai
1
330
tidy_rpart
bk_18
0
580
Most Valuable Bug(?) ~インシデント未遂から得た学び~
tatsumiakahori
0
140
社会人 20 年目エンジニア、発信で技術学びなおしてる話
e99h2121
1
140
How to Fight Production Incidents?
asatarin
0
170
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
451
31k
The Illustrated Children's Guide to Kubernetes
chrisshort
22
42k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
349
27k
BBQ
matthewcrist
75
8.1k
Making Projects Easy
brettharned
102
4.8k
The Cult of Friendly URLs
andyhume
68
5.1k
The Mythical Team-Month
searls
210
40k
The Invisible Side of Design
smashingmag
292
48k
Stop Working from a Prison Cell
hatefulcrawdad
263
18k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
101
6.2k
Adopting Sorbet at Scale
ufuk
65
7.8k
Into the Great Unknown - MozCon
thekraken
2
280
Transcript
Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF
US
Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at
Far too many hobbies
BIG DATA What does it mean?
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
Scaling Breakpoints
Sharding point Datasets paritioned by primary key
Vertical split Entirely unrelated tables
Denormalisation It's not free!
Consistency leeway Can you take inconsistent views?
Load Shapes
Read-heavy Write-heavy Large size
Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums
Amazon Glacier Eventbrite Logging
Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache
/ flat files Many indexes Fewer indexes
Extremes
Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency
Extreme Size Sacrifice query time
Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash
Extreme Auditability True append only storage
SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash
Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks. Andrew Godwin @andrewgodwin