Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Storage For The Rest Of Us
Search
Andrew Godwin
May 26, 2015
Programming
1
550
Small Data: Storage For The Rest Of Us
A talk I gave at PyWaw Summit 2015.
Andrew Godwin
May 26, 2015
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
270
Django Through The Years
andrewgodwin
0
170
Writing Maintainable Software At Scale
andrewgodwin
0
410
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
330
Async, Python, and the Future
andrewgodwin
2
620
How To Break Django: With Async
andrewgodwin
1
690
Taking Django's ORM Async
andrewgodwin
0
690
The Long Road To Asynchrony
andrewgodwin
0
620
The Scientist & The Engineer
andrewgodwin
1
720
Other Decks in Programming
See All in Programming
PHPのバージョンアップ時にも役立ったAST
matsuo_atsushi
0
230
kintone開発を効率化するためにチームで試した施策とその結果を大放出!
oguemon
0
320
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
1.2k
「個人開発マネタイズ大全」が教えてくれたこと
bani24884
1
290
Datadog Workflow Automation で圧倒的価値提供
showwin
1
300
sappoRo.R #12 初心者セッション
kosugitti
0
280
SwiftUI移行のためのインプレッショントラッキング基盤の構築
kokihirokawa
0
170
Drawing Heighway’s Dragon- Recursive Function Rewrite- From Imperative Style in Pascal 64 To Functional Style in Scala 3
philipschwarz
PRO
0
150
Domain-Driven Design (Tutorial)
hschwentner
13
22k
iOSでQRコード生成奮闘記
ktcryomm
2
120
Honoとフロントエンドの 型安全性について
yodaka
7
1.5k
Kotlinの開発でも AIをいい感じに使いたい / Making the Most of AI in Kotlin Development
kohii00
5
1.8k
Featured
See All Featured
Writing Fast Ruby
sferik
628
61k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Designing Experiences People Love
moore
140
23k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
440
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
Automating Front-end Workflow
addyosmani
1369
200k
Facilitating Awesome Meetings
lara
53
6.3k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
1.1k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Building an army of robots
kneath
303
45k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF
US
Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at
Far too many hobbies
BIG DATA What does it mean?
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
Scaling Breakpoints
Sharding point Datasets paritioned by primary key
Vertical split Entirely unrelated tables
Denormalisation It's not free!
Consistency leeway Can you take inconsistent views?
Load Shapes
Read-heavy Write-heavy Large size
Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums
Amazon Glacier Eventbrite Logging
Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache
/ flat files Many indexes Fewer indexes
Extremes
Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency
Extreme Size Sacrifice query time
Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash
Extreme Auditability True append only storage
SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash
Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks. Andrew Godwin @andrewgodwin