Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Databases in the Real World
Search
Andrew Godwin
August 04, 2014
Programming
2
560
Small Data: Databases in the Real World
A talk I gave at PyCon AU 2014.
Andrew Godwin
August 04, 2014
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
280
Django Through The Years
andrewgodwin
0
180
Writing Maintainable Software At Scale
andrewgodwin
0
410
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
330
Async, Python, and the Future
andrewgodwin
2
630
How To Break Django: With Async
andrewgodwin
1
700
Taking Django's ORM Async
andrewgodwin
0
700
The Long Road To Asynchrony
andrewgodwin
0
620
The Scientist & The Engineer
andrewgodwin
1
730
Other Decks in Programming
See All in Programming
requirements with math
moony
0
480
PHPのガベージコレクションを深掘りしよう
rinchoku
0
220
体得しよう!RSA暗号の原理と解読
laysakura
3
460
ローコードサービスの進化のためのモノレポ移行
taro28
1
300
気がついたら子供が社会人になって 自分と同じモバイルアプリエンジニアになった件 / Parent-Child Engineers
koishi
0
250
研究開発と実装OSSと プロダクトの好循環 / A virtuous cycle of research and development implementation OSS and products
linyows
1
180
RCPと宣言型ポリシーについてのお話し
kokitamura
2
130
CTFのWebにおける⾼難易度問題について
hamayanhamayan
1
850
PHPでお金を扱う時、終わりのない 謎の1円調査の旅にでなくて済む方法
nakka
1
740
Go言語での実装を通して学ぶ、高速なベクトル検索を支えるクラスタリング技術/fukuokago-kmeans
monochromegane
1
120
Goで作るChrome Extensions / Fukuoka.go #21
n3xem
2
2.5k
신입 안드로이드 개발자의 AI 스타트업 생존기 (+ Native C++ Code를 Android에서 사용해보기)
dygames
0
460
Featured
See All Featured
How to Ace a Technical Interview
jacobian
276
23k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.2k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
How to train your dragon (web standard)
notwaldorf
91
5.9k
The Invisible Side of Design
smashingmag
299
50k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7.1k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA REAL WORLD DATABASES IN THE
Andrew Godwin Core Developer Senior Engineer
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
None
Sharding point Vertical split Consistency leeway
Sharding point Datasets paritioned by primary key
Migration plan Implement consistent hashing on primary key Make large
number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
Vertical split Entirely unrelated tables
Migration plan Replicate database to new server Route split tables
there, disable replication - or - Slowly backfill new datastore with fallback lookup
Denormalisation It's not free!
Migration plan Add NULL fields to dependent tables App code
to fetch and fill if not present Possibly prefill on save of new items
Consistency leeway Can you take inconsistent views?
Migration plan Change your site! Talk to your designers! Deliberately
introduce inconsistency!
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks! Andrew Godwin @andrewgodwin
[email protected]
are hiring!