Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Storage For The Rest Of Us
Search
Andrew Godwin
May 26, 2015
Programming
1
550
Small Data: Storage For The Rest Of Us
A talk I gave at PyWaw Summit 2015.
Andrew Godwin
May 26, 2015
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
260
Django Through The Years
andrewgodwin
0
160
Writing Maintainable Software At Scale
andrewgodwin
0
400
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
310
Async, Python, and the Future
andrewgodwin
2
610
How To Break Django: With Async
andrewgodwin
1
670
Taking Django's ORM Async
andrewgodwin
0
680
The Long Road To Asynchrony
andrewgodwin
0
590
The Scientist & The Engineer
andrewgodwin
1
700
Other Decks in Programming
See All in Programming
React 19でお手軽にCSS-in-JSを自作する
yukukotani
5
560
令和7年版 あなたが使ってよいフロントエンド機能とは
mugi_uno
10
4.9k
はてなにおけるfujiwara-wareの活用やecspressoのCI/CD構成 / Fujiwara Tech Conference 2025
cohalz
2
2.5k
非ブラウザランタイムとWeb標準 / Non-Browser Runtimes and Web Standards
petamoriken
0
430
ある日突然あなたが管理しているサーバーにDDoSが来たらどうなるでしょう?知ってるようで何も知らなかったDDoS攻撃と対策 #phpcon.2024
akase244
2
7.7k
Jaspr Dart Web Framework 박제창 @Devfest 2024
itsmedreamwalker
0
150
HTML/CSS超絶浅い説明
yuki0329
0
190
ErdMap: Thinking about a map for Rails applications
makicamel
1
540
return文におけるstd::moveについて
onihusube
1
1.4k
アクターシステムに頼らずEvent Sourcingする方法について
j5ik2o
6
700
VisionProで部屋の明るさを反映させるシェーダーを作った話
segur
0
100
生成AIでGitHubソースコード取得して仕様書を作成
shukob
0
630
Featured
See All Featured
Music & Morning Musume
bryan
46
6.3k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
26
1.9k
A Tale of Four Properties
chriscoyier
157
23k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
Fireside Chat
paigeccino
34
3.1k
For a Future-Friendly Web
brad_frost
176
9.5k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
30
2.1k
Producing Creativity
orderedlist
PRO
343
39k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF
US
Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at
Far too many hobbies
BIG DATA What does it mean?
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
Scaling Breakpoints
Sharding point Datasets paritioned by primary key
Vertical split Entirely unrelated tables
Denormalisation It's not free!
Consistency leeway Can you take inconsistent views?
Load Shapes
Read-heavy Write-heavy Large size
Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums
Amazon Glacier Eventbrite Logging
Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache
/ flat files Many indexes Fewer indexes
Extremes
Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency
Extreme Size Sacrifice query time
Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash
Extreme Auditability True append only storage
SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash
Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks. Andrew Godwin @andrewgodwin