Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning to Build Distributed Systems the Hard Way
Search
Theo Hultberg
December 04, 2012
Programming
2
200
Learning to Build Distributed Systems the Hard Way
Presentation held at JDays 4 December, 2012
Theo Hultberg
December 04, 2012
Tweet
Share
More Decks by Theo Hultberg
See All by Theo Hultberg
Datalakes at AWS Summit Stockholm 2018
iconara
0
73
Building a CQL driver
iconara
0
49
Chasing the Elephant
iconara
0
76
Learning to Build Distributed Systems the Hard Way
iconara
1
140
Learning to Build Distributed Systems the Hard Way
iconara
3
5.1k
Concurrency and Distributed Systems in JRuby
iconara
3
620
A Guide to the Post Relational Revolution
iconara
4
5.4k
Standing on the Shoulders of Giants with JRuby
iconara
4
150
Shortcuts Around the Mistakes I've Made Scaling MongoDB
iconara
4
160
Other Decks in Programming
See All in Programming
Embedding it into Ruby code
soutaro
2
550
RustでAWS Lambda functionをいい感じに書く
taiki45
2
160
チーム立ち上げにAWSを活用したらClaudeさんに褒められた話
mkdev10
3
230
The test code generator using static analysis and LLM
mikik0
1
330
哲学史とモデリング
tanakahisateru
4
440
The World is a Network (and We Are Just Nodes)
whatyouhide
0
110
mb_trim関数を作りました
youkidearitai
PRO
1
260
WebGLで始める コンピュータグラフィックス入門
heller77
0
400
TypeScript 関数型スタイルでバックエンド開発のリアル
naoya
49
16k
TypeScriptの型とパフォーマンス (TSKaigi 2024)
ypresto
15
5.3k
Adding Security to Microcontroller Ruby
sylph01
1
1.4k
欠陥を早期に発見するための Software Engineer in Test とその重要性 / What is Software Engineer in Test and How they works
orgachem
PRO
19
2.6k
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
21
2k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
22
1.4k
Making the Leap to Tech Lead
cromwellryan
125
8.6k
Adopting Sorbet at Scale
ufuk
69
8.6k
Fireside Chat
paigeccino
22
2.7k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
188
16k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
22
1.6k
What’s in a name? Adding method to the madness
productmarketing
PRO
17
2.7k
The Brand Is Dead. Long Live the Brand.
mthomps
49
31k
Design by the Numbers
sachag
274
18k
Optimizing for Happiness
mojombo
371
69k
The Power of CSS Pseudo Elements
geoffreycrofte
62
5k
Transcript
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA
@iconara
speakerdeck.com/u/iconara (real time!)
Theo / @iconara
chief architect at BURT
let’s make online advertising a great experience
None
MAKING THIS
INTO THIS
HOW HARD CAN IT BE?
None
30K REQUESTS PER SECOND more than a billion requests per
day, over 1 TB raw data
ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of
millions of individual counters per day, plus counting uniques and visitor histories
IN REAL TIME or near real time, if you want
to be pedantic ×
HOW HARD CAN IT BE?
START WITH TWO OF EVERYTHING going from one to two
is the hardest, solve the scaling problem up front
START WITH TWO OF EVERYTHING you’ll solve the scaling problem,
and need less overcapacity THREE
GIVE A LOT OF THOUGHT TO KEYS AND IDS and
think about your queries first
MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random
a timestamp
CONSISTENCY IS OVERRATED don’t fear R + W < N
PRECOMPUTE ALL THE THINGS your users most likely don’t know
what they want, so why let them do ad hoc queries?
SEPARATE PROCESSING FROM STORAGE that way you can scale each
independently
PLAN HOW TO GET RID OF YOUR DATA deleting stuff
is harder than you might think × × × × × × ×
NoDB keep things streaming ×
DIVIDE THE LOAD big data systems are all about routing
and partitioning
RANDOM when you have no interdependencies between things it’s easy
to scale out
CONSISTENT when there are interdependencies you need to route using
some property of the objects, but make sure you get a uniform distribution
NUMEROLOGY
12
2 | 12 3 | 12 4 | 12 6
| 12
8 | 24 5 | 60
A DIVERSION ABOUT COUNTING TO 60 the reason why there’s
60 seconds to a minute, and 360 degrees to a circle × ×
3 SEGMENTS ON EACH FINGER = 12
3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON
OTHER HAND = 60
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
use multiples of 12 to scale without always having to
double
BLAH BLAH BLAH use multiples of 12 to scale without
always having to double
log2(366) ≈ 31
$-$ (ASCII code 36)-----
log2(366) ≈ 31
log2(366) ≈ 31 six characters 0-9, A-Z can represent 31
bits, which is kind of almost very close to four bytes
MEIHO0
MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
None
YOU CAN’T SCALE TO REAL TIME and don’t trust code
that doesn’t run continuously ×
DO YOU REALLY NEED A BACKUP? if you got 3x
replication over multiple availability zones, is that backup really worth it?
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of
things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
GÖTEBORG, DISTRIBUTED @gbgdistr
KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com