Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Learning to Build Distributed Systems the Hard Way
Search
Theo Hultberg
December 04, 2012
Programming
2
210
Learning to Build Distributed Systems the Hard Way
Presentation held at JDays 4 December, 2012
Theo Hultberg
December 04, 2012
Tweet
Share
More Decks by Theo Hultberg
See All by Theo Hultberg
Datalakes at AWS Summit Stockholm 2018
iconara
0
94
Building a CQL driver
iconara
0
57
Chasing the Elephant
iconara
0
92
Learning to Build Distributed Systems the Hard Way
iconara
1
140
Learning to Build Distributed Systems the Hard Way
iconara
3
5.1k
Concurrency and Distributed Systems in JRuby
iconara
3
660
A Guide to the Post Relational Revolution
iconara
4
5.4k
Standing on the Shoulders of Giants with JRuby
iconara
4
160
Shortcuts Around the Mistakes I've Made Scaling MongoDB
iconara
4
170
Other Decks in Programming
See All in Programming
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
lemonade_37
1
380
脱Riverpod?fqueryで考える、TanStack Queryライクなアーキテクチャの可能性
ostk0069
0
570
Android 15以上でPDFのテキスト検索を爆速開発!
tonionagauzzi
0
180
なぜあなたのオブザーバビリティ導入は頓挫するのか
ryota_hnk
4
530
Bedrock AgentCore ObservabilityによるAIエージェントの運用
licux
8
470
新しいモバイルアプリ勉強会(仮)について
uetyo
1
230
フロントエンドのパフォーマンスチューニング
koukimiura
6
2.3k
Understanding Kotlin Multiplatform
l2hyunwoo
0
240
DynamoDBは怖くない!〜テーブル設計の勘所とテスト戦略〜
hyamazaki
0
100
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ~継続的な取り組みを添えて~
takutakahashi
4
1.4k
코딩 에이전트 체크리스트: Claude Code ver.
nacyot
0
1k
可変性を制する設計: 構造と振る舞いから考える概念モデリングとその実装
a_suenami
8
1.1k
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Adopting Sorbet at Scale
ufuk
77
9.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
47
9.6k
Agile that works and the tools we love
rasmusluckow
329
21k
Visualization
eitanlees
146
16k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
400
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.7k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
The Cost Of JavaScript in 2023
addyosmani
51
8.7k
The Pragmatic Product Professional
lauravandoore
35
6.8k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Transcript
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA
@iconara
speakerdeck.com/u/iconara (real time!)
Theo / @iconara
chief architect at BURT
let’s make online advertising a great experience
None
MAKING THIS
INTO THIS
HOW HARD CAN IT BE?
None
30K REQUESTS PER SECOND more than a billion requests per
day, over 1 TB raw data
ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of
millions of individual counters per day, plus counting uniques and visitor histories
IN REAL TIME or near real time, if you want
to be pedantic ×
HOW HARD CAN IT BE?
START WITH TWO OF EVERYTHING going from one to two
is the hardest, solve the scaling problem up front
START WITH TWO OF EVERYTHING you’ll solve the scaling problem,
and need less overcapacity THREE
GIVE A LOT OF THOUGHT TO KEYS AND IDS and
think about your queries first
MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random
a timestamp
CONSISTENCY IS OVERRATED don’t fear R + W < N
PRECOMPUTE ALL THE THINGS your users most likely don’t know
what they want, so why let them do ad hoc queries?
SEPARATE PROCESSING FROM STORAGE that way you can scale each
independently
PLAN HOW TO GET RID OF YOUR DATA deleting stuff
is harder than you might think × × × × × × ×
NoDB keep things streaming ×
DIVIDE THE LOAD big data systems are all about routing
and partitioning
RANDOM when you have no interdependencies between things it’s easy
to scale out
CONSISTENT when there are interdependencies you need to route using
some property of the objects, but make sure you get a uniform distribution
NUMEROLOGY
12
2 | 12 3 | 12 4 | 12 6
| 12
8 | 24 5 | 60
A DIVERSION ABOUT COUNTING TO 60 the reason why there’s
60 seconds to a minute, and 360 degrees to a circle × ×
3 SEGMENTS ON EACH FINGER = 12
3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON
OTHER HAND = 60
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
12, 60, 120, 360 superior highly composite numbers
use multiples of 12 to scale without always having to
double
BLAH BLAH BLAH use multiples of 12 to scale without
always having to double
log2(366) ≈ 31
$-$ (ASCII code 36)-----
log2(366) ≈ 31
log2(366) ≈ 31 six characters 0-9, A-Z can represent 31
bits, which is kind of almost very close to four bytes
MEIHO0
MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
None
YOU CAN’T SCALE TO REAL TIME and don’t trust code
that doesn’t run continuously ×
DO YOU REALLY NEED A BACKUP? if you got 3x
replication over multiple availability zones, is that backup really worth it?
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of
things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
GÖTEBORG, DISTRIBUTED @gbgdistr
KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com