Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Databases in the Real World
Search
Andrew Godwin
August 04, 2014
Programming
2
490
Small Data: Databases in the Real World
A talk I gave at PyCon AU 2014.
Andrew Godwin
August 04, 2014
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
230
Django Through The Years
andrewgodwin
0
110
Writing Maintainable Software At Scale
andrewgodwin
0
350
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
270
Async, Python, and the Future
andrewgodwin
2
560
How To Break Django: With Async
andrewgodwin
1
610
Taking Django's ORM Async
andrewgodwin
0
620
The Long Road To Asynchrony
andrewgodwin
0
540
The Scientist & The Engineer
andrewgodwin
1
630
Other Decks in Programming
See All in Programming
MIERUNE BBQにおけるユーザー中心設計()
mierune
PRO
1
110
日付と正規化
megmogmog1965
0
140
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
Cloudflare Workers x AWS Lambdaの組み合わせユースケース / Cloudflare Workers x AWS Lambda Combination Use Case
seike460
PRO
2
310
CSC307 Lecture 11
javiergs
PRO
0
240
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
450
AHC035解説
terryu16
0
730
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
380
君たちはどうコードをレビューする (される) か / 大吉祥寺.pm
utgwkk
15
8.5k
Microservices rules (July 2024) : what good looks like
cer
PRO
0
1.6k
CSC307 Lecture 12
javiergs
PRO
0
220
[After Kotlin Fest 2024 LT Night @ Sansan] もっともっとKotlinを好きになる!K2 Compiler Pluginで遊んでみよう!
kitakkun
2
260
Featured
See All Featured
Done Done
chrislema
179
15k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
16
1.6k
Debugging Ruby Performance
tmm1
71
11k
Code Review Best Practice
trishagee
58
16k
10 Git Anti Patterns You Should be Aware of
lemiorhan
652
58k
The Invisible Customer
myddelton
117
13k
Making the Leap to Tech Lead
cromwellryan
127
8.7k
Designing the Hi-DPI Web
ddemaree
276
34k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.5k
How To Stay Up To Date on Web Technology
chriscoyier
784
250k
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA REAL WORLD DATABASES IN THE
Andrew Godwin Core Developer Senior Engineer
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
None
Sharding point Vertical split Consistency leeway
Sharding point Datasets paritioned by primary key
Migration plan Implement consistent hashing on primary key Make large
number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
Vertical split Entirely unrelated tables
Migration plan Replicate database to new server Route split tables
there, disable replication - or - Slowly backfill new datastore with fallback lookup
Denormalisation It's not free!
Migration plan Add NULL fields to dependent tables App code
to fetch and fill if not present Possibly prefill on save of new items
Consistency leeway Can you take inconsistent views?
Migration plan Change your site! Talk to your designers! Deliberately
introduce inconsistency!
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks! Andrew Godwin @andrewgodwin
[email protected]
are hiring!