There's no Clusterf*ck without a Cluster

1. There's No Clusterf*ck without a Cluster How @GoVictorOps went from unicorns and broken to boring and working
2. Premature availabilization? • Connect you with your monitors • Harass you when stuff breaks @boulderDanH
3. Availability is our DNA • Scala • Akka • Kafka • Shard key
4. What is clustering?
5. An online encyclopedia says • Computers working together (appeals to authority)
6. A dictionary says • clus·ter noun \ˈkləs-tər\ a number of similar things that occur together (includes pronunciation for legitimacy)
7. Our definition • Who is currently in the cluster? • Tell me when nodes are coming and going • High Availability / scaling
8. Requirements 1.0 1. Logical actor tree 2. Service discovery 3. Lead me to success
9. Logical actor tree • Failover • Hand off
10. Service discovery • “cluster://user/victorops/broadcaster” ! “hello”
11. Tradeoffs are everywhere Vector clocks are totally cool Async consensus?
13. Implementation • Routers / Patterns • Native = Truth
14. Actor state • Easy and Tempting • Painful to unwind
17. What could go wrong? • Partitions are permanent • Want some config? How about six! ◦ failure-detector.threshold x 2 ◦ failure-detector.min-std-deviation x 2 ◦ failure-detector.acceptable-heartbeat-pause x 2 • Hazelcast uses hazelcast.max.no.heartbeat.seconds • ZooKeeper uses “session timeout”
18. More picking on Akka • Logging during failures is sparse • Remoting / Failure detection weren’t bulkheaded
19. Recap 1. Logical actor tree 2. Service discovery 3. Lead me to success
20. Requirements 2.0 1. Member lists 2. Easy to configure, ability to add machines w/o config 3. Pass remoting address around
22. What is Hazelcast? • Distributed maps & locks • Multicast (IGMP)
23. Implementation akka.remote.quarantine-systems-for = "off" akka.remote.gate-invalid-addresses-for = 0 s src: akka-devel • Publish Akka address using a map • Detect nodes joining / leaving cluster
24. • Multicast • In memory • Cluster Client • Member list isn’t consistent across cluster What went wrong?
25. Recap on requirements 2.0 1. Member lists 2. Easy to configure 3. Pass remoting address around
26. Requirements 3.0 1. Member list is consistent 2. Cluster clients are first class
27. Cluster Membership • Consistent - Zk • Probably consistent - Gossip • YOLO consistency - Hazelcast
28. … no seriously, this is the logo
29. What is ZooKeeper? • Clustered, consistent file system • API is focused on building distributed concepts
30. Implementation • Cluster Membership = EPHEMERAL • Leader Election = SEQUENTIAL • “Cluster” = EPHEMERAL_SEQUENTIAL • Store akka addresses in ephemeral nodes • Curator project
31. The Good • Reputation • Strong Consistency • Cluster clients / Service Discovery
32. What was / is hard? • Twitter’s Zk library • External Cluster Manager
33. The final tally • Solid concept of membership • Keep things simple • Log / Graph / Monitor everything
34. Questions?