Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning to Build Distributed Systems the Hard Way
Search
Theo Hultberg
December 04, 2012
Programming
2
210
Learning to Build Distributed Systems the Hard Way
Presentation held at JDays 4 December, 2012
Theo Hultberg
December 04, 2012
Tweet
Share
More Decks by Theo Hultberg
See All by Theo Hultberg
Datalakes at AWS Summit Stockholm 2018
iconara
0
95
Building a CQL driver
iconara
0
57
Chasing the Elephant
iconara
0
92
Learning to Build Distributed Systems the Hard Way
iconara
1
140
Learning to Build Distributed Systems the Hard Way
iconara
3
5.1k
Concurrency and Distributed Systems in JRuby
iconara
3
660
A Guide to the Post Relational Revolution
iconara
4
5.4k
Standing on the Shoulders of Giants with JRuby
iconara
4
160
Shortcuts Around the Mistakes I've Made Scaling MongoDB
iconara
4
170
Other Decks in Programming
See All in Programming
Catch Up: Go Style Guide Update
andpad
0
170
CSC305 Lecture 04
javiergs
PRO
0
250
After go func(): Goroutines Through a Beginner’s Eye
97vaibhav
0
230
Let's Write a Train Tracking Algorithm
twocentstudios
0
220
AIで開発生産性を上げる個人とチームの取り組み
taniigo
0
130
階層構造を表現するデータ構造とリファクタリング 〜1年で10倍成長したプロダクトの変化と課題〜
yuhisatoxxx
3
920
AI Coding Meetup #3 - 導入セッション / ai-coding-meetup-3
izumin5210
0
590
開発生産性を上げるための生成AI活用術
starfish719
1
170
プロダクト開発をAI 1stに変革する〜SaaS is dead時代で生き残るために〜 / AI 1st Product Development
kobakei
0
490
CSC305 Lecture 02
javiergs
PRO
1
260
ネイティブ製ガントチャートUIを作って学ぶUICollectionViewLayoutの威力
jrsaruo
0
130
CSC509 Lecture 01
javiergs
PRO
1
430
Featured
See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
188
55k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3.1k
Done Done
chrislema
185
16k
Why Our Code Smells
bkeepers
PRO
339
57k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
The World Runs on Bad Software
bkeepers
PRO
71
11k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Transcript
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA
@iconara
speakerdeck.com/u/iconara (real time!)
Theo / @iconara
chief architect at BURT
let’s make online advertising a great experience
None
MAKING THIS
INTO THIS
HOW HARD CAN IT BE?
None
30K REQUESTS PER SECOND more than a billion requests per
day, over 1 TB raw data
ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of
millions of individual counters per day, plus counting uniques and visitor histories
IN REAL TIME or near real time, if you want
to be pedantic ×
HOW HARD CAN IT BE?
START WITH TWO OF EVERYTHING going from one to two
is the hardest, solve the scaling problem up front
START WITH TWO OF EVERYTHING you’ll solve the scaling problem,
and need less overcapacity THREE
GIVE A LOT OF THOUGHT TO KEYS AND IDS and
think about your queries first
MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random
a timestamp
CONSISTENCY IS OVERRATED don’t fear R + W < N
PRECOMPUTE ALL THE THINGS your users most likely don’t know
what they want, so why let them do ad hoc queries?
SEPARATE PROCESSING FROM STORAGE that way you can scale each
independently
PLAN HOW TO GET RID OF YOUR DATA deleting stuff
is harder than you might think × × × × × × ×
NoDB keep things streaming ×
DIVIDE THE LOAD big data systems are all about routing
and partitioning
RANDOM when you have no interdependencies between things it’s easy
to scale out
CONSISTENT when there are interdependencies you need to route using
some property of the objects, but make sure you get a uniform distribution
NUMEROLOGY
12
2 | 12 3 | 12 4 | 12 6
| 12
8 | 24 5 | 60
A DIVERSION ABOUT COUNTING TO 60 the reason why there’s
60 seconds to a minute, and 360 degrees to a circle × ×
3 SEGMENTS ON EACH FINGER = 12
3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON
OTHER HAND = 60
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
use multiples of 12 to scale without always having to
double
BLAH BLAH BLAH use multiples of 12 to scale without
always having to double
log2(366) ≈ 31
$-$ (ASCII code 36)-----
log2(366) ≈ 31
log2(366) ≈ 31 six characters 0-9, A-Z can represent 31
bits, which is kind of almost very close to four bytes
MEIHO0
MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
None
YOU CAN’T SCALE TO REAL TIME and don’t trust code
that doesn’t run continuously ×
DO YOU REALLY NEED A BACKUP? if you got 3x
replication over multiple availability zones, is that backup really worth it?
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of
things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
GÖTEBORG, DISTRIBUTED @gbgdistr
KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com