Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
There's no Clusterf*ck without a Cluster
Search
Dan Hopkins
April 19, 2014
Programming
1
190
There's no Clusterf*ck without a Cluster
Dan Hopkins
April 19, 2014
Tweet
Share
More Decks by Dan Hopkins
See All by Dan Hopkins
Actors: not just for movies anymore
danielhopkins
1
150
Other Decks in Programming
See All in Programming
Terraform やるなら公式スタイルガイドを読もう 〜重要項目 10選〜
hiyanger
13
3.1k
Comparing decimals in Swift Testing
417_72ki
0
170
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
1
280
AI時代のドメイン駆動設計-DDD実践におけるAI活用のあり方 / ddd-in-ai-era
minodriven
21
7.8k
ライブ配信サービスの インフラのジレンマ -マルチクラウドに至ったワケ-
mirrativ
1
230
A Gopher's Guide to Vibe Coding
danicat
0
150
Dart 参戦!!静的型付き言語界の隠れた実力者
kno3a87
0
200
フロントエンドのmonorepo化と責務分離のリアーキテクト
kajitack
2
110
パスタの技術
yusukebe
1
390
AIに安心して任せるためにTypeScriptで一意な型を作ろう
arfes0e2b3c
0
370
Understanding Kotlin Multiplatform
l2hyunwoo
0
260
バイブコーディングの正体——AIエージェントはソフトウェア開発を変えるか?
stakaya
5
940
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
70
11k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Facilitating Awesome Meetings
lara
55
6.5k
Side Projects
sachag
455
43k
BBQ
matthewcrist
89
9.8k
Faster Mobile Websites
deanohume
309
31k
Mobile First: as difficult as doing things right
swwweet
223
9.9k
Visualization
eitanlees
146
16k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
890
Embracing the Ebb and Flow
colly
86
4.8k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Transcript
There's No Clusterf*ck without a Cluster How @GoVictorOps went from
unicorns and broken to boring and working
Premature availabilization? • Connect you with your monitors • Harass
you when stuff breaks @boulderDanH
Availability is our DNA • Scala • Akka • Kafka
• Shard key
What is clustering?
An online encyclopedia says • Computers working together (appeals to
authority)
A dictionary says • clus·ter noun \ˈkləs-tər\ a number of
similar things that occur together (includes pronunciation for legitimacy)
Our definition • Who is currently in the cluster? •
Tell me when nodes are coming and going • High Availability / scaling
Requirements 1.0 1. Logical actor tree 2. Service discovery 3.
Lead me to success
Logical actor tree • Failover • Hand off
Service discovery • “cluster://user/victorops/broadcaster” ! “hello”
Tradeoffs are everywhere Vector clocks are totally cool Async consensus?
None
Implementation • Routers / Patterns • Native = Truth
Actor state • Easy and Tempting • Painful to unwind
None
None
What could go wrong? • Partitions are permanent • Want
some config? How about six! ◦ failure-detector.threshold x 2 ◦ failure-detector.min-std-deviation x 2 ◦ failure-detector.acceptable-heartbeat-pause x 2 • Hazelcast uses hazelcast.max.no.heartbeat.seconds • ZooKeeper uses “session timeout”
More picking on Akka • Logging during failures is sparse
• Remoting / Failure detection weren’t bulkheaded
Recap 1. Logical actor tree 2. Service discovery 3. Lead
me to success
Requirements 2.0 1. Member lists 2. Easy to configure, ability
to add machines w/o config 3. Pass remoting address around
None
What is Hazelcast? • Distributed maps & locks • Multicast
(IGMP)
Implementation akka.remote.quarantine-systems-for = "off" akka.remote.gate-invalid-addresses-for = 0 s src: akka-devel
• Publish Akka address using a map • Detect nodes joining / leaving cluster
• Multicast • In memory • Cluster Client • Member
list isn’t consistent across cluster What went wrong?
Recap on requirements 2.0 1. Member lists 2. Easy to
configure 3. Pass remoting address around
Requirements 3.0 1. Member list is consistent 2. Cluster clients
are first class
Cluster Membership • Consistent - Zk • Probably consistent -
Gossip • YOLO consistency - Hazelcast
… no seriously, this is the logo
What is ZooKeeper? • Clustered, consistent file system • API
is focused on building distributed concepts
Implementation • Cluster Membership = EPHEMERAL • Leader Election =
SEQUENTIAL • “Cluster” = EPHEMERAL_SEQUENTIAL • Store akka addresses in ephemeral nodes • Curator project
The Good • Reputation • Strong Consistency • Cluster clients
/ Service Discovery
What was / is hard? • Twitter’s Zk library •
External Cluster Manager
The final tally • Solid concept of membership • Keep
things simple • Log / Graph / Monitor everything
Questions?