Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning to Build Distributed Systems the Hard Way
Search
Theo Hultberg
December 04, 2012
Programming
2
210
Learning to Build Distributed Systems the Hard Way
Presentation held at JDays 4 December, 2012
Theo Hultberg
December 04, 2012
Tweet
Share
More Decks by Theo Hultberg
See All by Theo Hultberg
Datalakes at AWS Summit Stockholm 2018
iconara
0
94
Building a CQL driver
iconara
0
57
Chasing the Elephant
iconara
0
92
Learning to Build Distributed Systems the Hard Way
iconara
1
140
Learning to Build Distributed Systems the Hard Way
iconara
3
5.1k
Concurrency and Distributed Systems in JRuby
iconara
3
660
A Guide to the Post Relational Revolution
iconara
4
5.4k
Standing on the Shoulders of Giants with JRuby
iconara
4
160
Shortcuts Around the Mistakes I've Made Scaling MongoDB
iconara
4
170
Other Decks in Programming
See All in Programming
MCP連携で加速するAI駆動開発/mcp integration accelerates ai-driven-development
bpstudy
0
310
kiroでゲームを作ってみた
iriikeita
0
180
オープンセミナー2025@広島「君はどこで動かすか?」アンケート結果
satoshi256kbyte
0
150
MLH State of the League: 2026 Season
theycallmeswift
0
100
ゲームの物理
fadis
5
1.2k
AIレビュアーをスケールさせるには / Scaling AI Reviewers
technuma
2
210
バイブコーディング × 設計思考
nogu66
0
120
Flutter로 Gemini와 MCP를 활용한 Agentic App 만들기 - 박제창 2025 I/O Extended Seoul
itsmedreamwalker
0
140
フロントエンドのmonorepo化と責務分離のリアーキテクト
kajitack
2
130
WebAssemblyインタプリタを書く ~Component Modelを添えて~
ruccho
1
870
Honoアップデート 2025年夏
yusukebe
1
810
Nuances on Kubernetes - RubyConf Taiwan 2025
envek
0
180
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
We Have a Design System, Now What?
morganepeng
53
7.7k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
For a Future-Friendly Web
brad_frost
179
9.9k
Building Applications with DynamoDB
mza
96
6.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Side Projects
sachag
455
43k
How to Think Like a Performance Engineer
csswizardry
25
1.8k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
Transcript
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA
@iconara
speakerdeck.com/u/iconara (real time!)
Theo / @iconara
chief architect at BURT
let’s make online advertising a great experience
None
MAKING THIS
INTO THIS
HOW HARD CAN IT BE?
None
30K REQUESTS PER SECOND more than a billion requests per
day, over 1 TB raw data
ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of
millions of individual counters per day, plus counting uniques and visitor histories
IN REAL TIME or near real time, if you want
to be pedantic ×
HOW HARD CAN IT BE?
START WITH TWO OF EVERYTHING going from one to two
is the hardest, solve the scaling problem up front
START WITH TWO OF EVERYTHING you’ll solve the scaling problem,
and need less overcapacity THREE
GIVE A LOT OF THOUGHT TO KEYS AND IDS and
think about your queries first
MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random
a timestamp
CONSISTENCY IS OVERRATED don’t fear R + W < N
PRECOMPUTE ALL THE THINGS your users most likely don’t know
what they want, so why let them do ad hoc queries?
SEPARATE PROCESSING FROM STORAGE that way you can scale each
independently
PLAN HOW TO GET RID OF YOUR DATA deleting stuff
is harder than you might think × × × × × × ×
NoDB keep things streaming ×
DIVIDE THE LOAD big data systems are all about routing
and partitioning
RANDOM when you have no interdependencies between things it’s easy
to scale out
CONSISTENT when there are interdependencies you need to route using
some property of the objects, but make sure you get a uniform distribution
NUMEROLOGY
12
2 | 12 3 | 12 4 | 12 6
| 12
8 | 24 5 | 60
A DIVERSION ABOUT COUNTING TO 60 the reason why there’s
60 seconds to a minute, and 360 degrees to a circle × ×
3 SEGMENTS ON EACH FINGER = 12
3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON
OTHER HAND = 60
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
use multiples of 12 to scale without always having to
double
BLAH BLAH BLAH use multiples of 12 to scale without
always having to double
log2(366) ≈ 31
$-$ (ASCII code 36)-----
log2(366) ≈ 31
log2(366) ≈ 31 six characters 0-9, A-Z can represent 31
bits, which is kind of almost very close to four bytes
MEIHO0
MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
None
YOU CAN’T SCALE TO REAL TIME and don’t trust code
that doesn’t run continuously ×
DO YOU REALLY NEED A BACKUP? if you got 3x
replication over multiple availability zones, is that backup really worth it?
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of
things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
GÖTEBORG, DISTRIBUTED @gbgdistr
KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com