Lithium A split-brain resolver for Akka-Cluster Dennis van der Bij @MrDnx DennisVDB 

OMS • SwissBorg’s OMS (order management system) • Aggregates the prices of 4 crypto-exchanges • Best-execution 2 

OMS’ objectives • Best-execution • High availability 3 

OMS cluster Node-2 Node-3 Node-1 Node-4 Node-5 • Persistent actors • Singleton actors • … You are here S Super-important singleton 4 

Unreachable nodes Node-2 Node-3 Node-1 Node-4 Node-5 • S cannot be reached • Need to start S on a reachable node • Singleton actors are not migrated when nodes are unreachable S Partition A Partition B Dead or alive? 5 

Membership state • Leader chosen deterministically • Leader manages state transitions on convergence • Convergence cannot be reached with unreachable nodes • Eventually-perfect FD* • Nodes cannot become fully-fledged members or gracefully leave the cluster *Hayashibara, Naohiro, et al. "The φ accrual failure detector." Proceedings of the 23rd IEEE International Symposium on Reliable Distributed Systems, 2004 Joining Up Leaving Exiting Removed Down Leader Leader Leader Leader 6 

Remove from membership state Node-2 Node-3 Node-1 Node-2 Node-3 Node-1 S 7 

Remove from membership state Node-4 Node-5 S 8 

Split-brain Node-2 Node-3 Node-1 Node-4 Node-5 Network partition One cluster becomes two clusters S S 9 

Split-brain resolver • Prevent split-brains to happen in the 1st place • Pick only one partition that will survive - Survivor will down the unreachable nodes - Non-survivors will down themselves 10 

Existing solutions • Lightbend SBR - Multiple strategies - Multi-DC - Starting at $50’000 per year • Four OSS SBR’s - Two used in production, single strategy (MOIA) - Two others, multiple strategies (fail my tests) 11 

Lithium • Strategies - Static-quorum, keep-majority, keep-oldest, and keep-referee • Multi-datacenter support • Tests, tests, tests 12 

Static-quorum • Pick partition with at least N nodes • Downs the cluster: more than nodes, no partition with at least N nodes. 2N − 1 13 

Static-quorum 14 Node-2 Node-3 Node-1 Node-4 Node-5 N = 3 

Static-quorum 15 Node-2 Node-3 Node-1 Node-4 Node-5 Node-2 Node-3 Node-1 N = 3 

16 Keep-majority • Pick partition with a majority of nodes (or lowest address) • Downs the cluster: no partition with a majority 16 

Keep-majority 17 Node-2 Node-3 Node-1 Node-4 Node-5 

Keep-majority 18 Node-2 Node-3 Node-1 Node-4 Node-5 Node-2 Node-3 Node-1 

19 Keep-oldest • Pick partition containing the oldest member • Oldest member hosts the singleton instance • Nearly entire cluster is downed when oldest is alone 19 19 

Keep-oldest 20 Node-2 Node-3 Node-1 Node-4 Node-5 Oldest 

Keep-oldest 21 Node-2 Node-3 Node-1 Node-4 Node-5 Node-4 Node-5 Oldest 

22 Keep-referee • Pick the partition containing the “referee” node • Downs most of the cluster when the referee is alone 22 22 

Keep-referee 23 Node-2 Node-3 Node-1 Node-4 Node-5 Referee 

Keep-referee 24 Node-2 Node-3 Node-1 Node-4 Node-5 Node-4 Node-5 Referee 

Choosing a strategy 25 Use “role” to only take in account certain members 

How it works • Provide instance of DowningProvider • Each cluster member runs an instance of Lithium 26 

Demo 27 

Tests, tests, tests • ~70% LOCs are tests • Unit tests + property-based tests • “multi-jvm” tests 28 

Scenarios • Use property-based tests to detect edge-cases • Splits during membership changes 29 

Multi-JVM tests • Simulate a cluster locally • Split links between members programmatically • Observe how it gets resolved 30 

Demo 31 

Comparison Static-quorum Keep-majority Keep-oldest Keep-referee Multi-DC Lithium ARD N/A N/A N/A N/A SAD N/A ACCD N/A ADR N/A N/A N/A N/A 32 

https://github.com/SwissBorg/lithium/ 33 @MrDnx DennisVDB