Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Storage For The Rest Of Us
Search
Andrew Godwin
May 26, 2015
Programming
1
500
Small Data: Storage For The Rest Of Us
A talk I gave at PyWaw Summit 2015.
Andrew Godwin
May 26, 2015
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
230
Django Through The Years
andrewgodwin
0
110
Writing Maintainable Software At Scale
andrewgodwin
0
350
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
270
Async, Python, and the Future
andrewgodwin
2
560
How To Break Django: With Async
andrewgodwin
1
610
Taking Django's ORM Async
andrewgodwin
0
620
The Long Road To Asynchrony
andrewgodwin
0
540
The Scientist & The Engineer
andrewgodwin
1
630
Other Decks in Programming
See All in Programming
Temporalを取り巻く仕様を整理する
sajikix
0
110
Ruby メモリ管理 プログラミング
megmogmog1965
0
130
【Go言語】ジェネリクス
tomo1227
0
170
CSC307 Lecture 14
javiergs
PRO
0
220
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
Xcode 16のPreviewModifierと@Previewableを活用した効率的なプレビュー方法の考察
ojun9
2
160
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
Introduction to GitOps
hwchiu
0
110
「2024年版 Kotlin サーバーサイドプログラミング実践開発」の補講 〜O/Rマッパー編〜
n_takehata
2
260
Webエディタライブラリ 「CodeMirror」から学ぶ Webアプリ開発のテクニック
ryosukeigarashi
0
250
Namespace on read
tagomoris
2
370
SRE チーム立ち上げ前に考えたこと・取り組んだこと / Considerations and Preparations Before Establishing an SRE Team
mackey0225
3
320
Featured
See All Featured
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
The Power of CSS Pseudo Elements
geoffreycrofte
64
5.2k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
44
4.7k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
How to Ace a Technical Interview
jacobian
274
23k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
23
1.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
42
2.7k
5 minutes of I Can Smell Your CMS
philhawksworth
200
19k
A Tale of Four Properties
chriscoyier
155
22k
The MySQL Ecosystem @ GitHub 2015
samlambert
248
12k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
502
140k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
277
13k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF
US
Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at
Far too many hobbies
BIG DATA What does it mean?
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
Scaling Breakpoints
Sharding point Datasets paritioned by primary key
Vertical split Entirely unrelated tables
Denormalisation It's not free!
Consistency leeway Can you take inconsistent views?
Load Shapes
Read-heavy Write-heavy Large size
Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums
Amazon Glacier Eventbrite Logging
Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache
/ flat files Many indexes Fewer indexes
Extremes
Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency
Extreme Size Sacrifice query time
Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash
Extreme Auditability True append only storage
SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash
Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks. Andrew Godwin @andrewgodwin