Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning to Build Distributed Systems the Hard Way
Search
Theo Hultberg
December 04, 2012
Programming
2
210
Learning to Build Distributed Systems the Hard Way
Presentation held at JDays 4 December, 2012
Theo Hultberg
December 04, 2012
Tweet
Share
More Decks by Theo Hultberg
See All by Theo Hultberg
Datalakes at AWS Summit Stockholm 2018
iconara
0
95
Building a CQL driver
iconara
0
57
Chasing the Elephant
iconara
0
92
Learning to Build Distributed Systems the Hard Way
iconara
1
140
Learning to Build Distributed Systems the Hard Way
iconara
3
5.1k
Concurrency and Distributed Systems in JRuby
iconara
3
660
A Guide to the Post Relational Revolution
iconara
4
5.4k
Standing on the Shoulders of Giants with JRuby
iconara
4
160
Shortcuts Around the Mistakes I've Made Scaling MongoDB
iconara
4
170
Other Decks in Programming
See All in Programming
Developer Joy - The New Paradigm
hollycummins
1
330
XP, Testing and ninja testing ZOZ5
m_seki
3
790
モテるデスク環境
mozumasu
3
900
Cursorハンズオン実践!
eltociear
2
1.2k
チームの境界をブチ抜いていけ
tokai235
0
210
Claude CodeによるAI駆動開発の実践 〜そこから見えてきたこれからのプログラミング〜
iriikeita
0
320
SwiftDataを使って10万件のデータを読み書きする
akidon0000
0
230
Flutterで分数(Fraction)を表示する方法
koukimiura
0
140
大規模アプリのDIフレームワーク刷新戦略 ~過去最大規模の並行開発を止めずにアプリ全体に導入するまで~
mot_techtalk
1
470
AIと人間の共創開発!OSSで試行錯誤した開発スタイル
mae616
2
770
コードとあなたと私の距離 / The Distance Between Code, You, and I
hiro_y
0
190
『毎日の移動』を支えるGoバックエンド内製開発
yutautsugi
2
270
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
630
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
190
55k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
10
880
What's in a price? How to price your products and services
michaelherold
246
12k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
Keith and Marios Guide to Fast Websites
keithpitt
411
23k
Mobile First: as difficult as doing things right
swwweet
225
10k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.2k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
Transcript
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA
@iconara
speakerdeck.com/u/iconara (real time!)
Theo / @iconara
chief architect at BURT
let’s make online advertising a great experience
None
MAKING THIS
INTO THIS
HOW HARD CAN IT BE?
None
30K REQUESTS PER SECOND more than a billion requests per
day, over 1 TB raw data
ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of
millions of individual counters per day, plus counting uniques and visitor histories
IN REAL TIME or near real time, if you want
to be pedantic ×
HOW HARD CAN IT BE?
START WITH TWO OF EVERYTHING going from one to two
is the hardest, solve the scaling problem up front
START WITH TWO OF EVERYTHING you’ll solve the scaling problem,
and need less overcapacity THREE
GIVE A LOT OF THOUGHT TO KEYS AND IDS and
think about your queries first
MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random
a timestamp
CONSISTENCY IS OVERRATED don’t fear R + W < N
PRECOMPUTE ALL THE THINGS your users most likely don’t know
what they want, so why let them do ad hoc queries?
SEPARATE PROCESSING FROM STORAGE that way you can scale each
independently
PLAN HOW TO GET RID OF YOUR DATA deleting stuff
is harder than you might think × × × × × × ×
NoDB keep things streaming ×
DIVIDE THE LOAD big data systems are all about routing
and partitioning
RANDOM when you have no interdependencies between things it’s easy
to scale out
CONSISTENT when there are interdependencies you need to route using
some property of the objects, but make sure you get a uniform distribution
NUMEROLOGY
12
2 | 12 3 | 12 4 | 12 6
| 12
8 | 24 5 | 60
A DIVERSION ABOUT COUNTING TO 60 the reason why there’s
60 seconds to a minute, and 360 degrees to a circle × ×
3 SEGMENTS ON EACH FINGER = 12
3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON
OTHER HAND = 60
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
use multiples of 12 to scale without always having to
double
BLAH BLAH BLAH use multiples of 12 to scale without
always having to double
log2(366) ≈ 31
$-$ (ASCII code 36)-----
log2(366) ≈ 31
log2(366) ≈ 31 six characters 0-9, A-Z can represent 31
bits, which is kind of almost very close to four bytes
MEIHO0
MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
None
YOU CAN’T SCALE TO REAL TIME and don’t trust code
that doesn’t run continuously ×
DO YOU REALLY NEED A BACKUP? if you got 3x
replication over multiple availability zones, is that backup really worth it?
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of
things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
GÖTEBORG, DISTRIBUTED @gbgdistr
KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com