Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Small Data: Databases in the Real World
Search
Andrew Godwin
August 04, 2014
Programming
2
460
Small Data: Databases in the Real World
A talk I gave at PyCon AU 2014.
Andrew Godwin
August 04, 2014
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
200
Django Through The Years
andrewgodwin
0
86
Writing Maintainable Software At Scale
andrewgodwin
0
330
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
240
Async, Python, and the Future
andrewgodwin
2
540
How To Break Django: With Async
andrewgodwin
1
570
Taking Django's ORM Async
andrewgodwin
0
580
The Long Road To Asynchrony
andrewgodwin
0
510
The Scientist & The Engineer
andrewgodwin
1
570
Other Decks in Programming
See All in Programming
R言語の環境構築と基礎 Tokyo.R 112
bob3bob3
0
260
今、知っておきたい! 生成AIエージェントの世界
elith
3
350
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
26
8.2k
新宿ダンジョンを可視化してみた
satoshi7190
2
240
Zero Waste, Radical Magic, and Italian Graft – Quarkus Efficiency Secrets
hollycummins
0
230
PHPはいつから死んでいるかの調査
chiroruxx
1
400
"config" ってなんだ? / What is "config"?
okashoi
0
240
Node.js v22 で変わること
yosuke_furukawa
PRO
9
3k
SwiftUIで使いやすいToastの作り方 / How to build a Toast system which is easy to use in SwiftUI
lovee
3
140
TYPO3 v13 – The road to LTS: What's new and new APIs
luisasofie_xoxo
0
200
Git Rebase
bkuhlmann
11
1.6k
0→1と1→10の狭間で Javaという技術選定を振り返る/Reflecting on the Decision to Choose Java Between Scaling from 0 to 1 and 1 to 10
jaguar_imo
2
380
Featured
See All Featured
BBQ
matthewcrist
80
8.8k
GitHub's CSS Performance
jonrohan
1025
450k
Design by the Numbers
sachag
274
18k
Practical Orchestrator
shlominoach
182
9.7k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
7
1k
VelocityConf: Rendering Performance Case Studies
addyosmani
320
23k
The Art of Programming - Codeland 2020
erikaheidi
42
12k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
116
18k
Done Done
chrislema
178
15k
Web Components: a chance to create the future
zenorocha
305
41k
The Language of Interfaces
destraynor
151
23k
Transcript
Andrew Godwin @andrewgodwin SMALL DATA REAL WORLD DATABASES IN THE
Andrew Godwin Core Developer Senior Engineer
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
None
Sharding point Vertical split Consistency leeway
Sharding point Datasets paritioned by primary key
Migration plan Implement consistent hashing on primary key Make large
number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
Vertical split Entirely unrelated tables
Migration plan Replicate database to new server Route split tables
there, disable replication - or - Slowly backfill new datastore with fallback lookup
Denormalisation It's not free!
Migration plan Add NULL fields to dependent tables App code
to fetch and fill if not present Possibly prefill on save of new items
Consistency leeway Can you take inconsistent views?
Migration plan Change your site! Talk to your designers! Deliberately
introduce inconsistency!
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks! Andrew Godwin @andrewgodwin
[email protected]
are hiring!