Service Level Consensus - Paris DDD Meetup

Transcript

1. Service Level Consensus Building Clustered Event-driven services Paris Meetup 04/07/2017
2. About the speaker - Olivier Deheurles - 15+ years designing and developing real-time trading systems (finance & commodities) - Co-founder of Adaptive - Consultancy - We deliver event driven platforms for our clients - Offices in London, Barcelona, Montreal and New York - 120 people
3. This talk - Alternative architecture: Clustered Services - Quick overview of distributed consensus - Benefits - Q&As
4. The Cookie-Cutter approach Service Client Client Client Client Stateless Durable Transactional HTTP LOGIC DATA
5. How do we cope with failure? Service Client Client Client Client Stateless Durable Transactional Data-replication Service Load balancer Load balancer HTTP LOGIC DATA
6. Data replication Difficult problem: - Data must stay consistent between the database nodes - Data must be durable (persisted to disk) - Even if a server fails or the network fails → Distributed Consensus problem
7. Distributed Consensus Barbara Liskov 1988 Viewstamped Replication Turing Award 2008 Leslie Lamport 1989 Paxos Turing Award 2013 Diego Ongaro 2013 Raft
8. Some implementations Paxos - Google Chubby - Google Spanner - AWS (core distributed systems primitives) - Apache Mesos - Neo4j Viewstamped Replication - ? ZAB - ZooKeeper Raft - Etcd - Consul
9. Consensus Algorithms
10. What is it? - Protocol of interaction between multiple nodes - allow them to agree (reach consensus) on the state of the overall system
11. 2 nodes → Split brain ? ?
12. Server failure Tolerates fault of 1 node Tolerates fault of 2 nodes
13. Network failure 1 node partitioned from the other nodes 2 nodes partitioned from the other nodes
14. Core Concepts 1. Leader Election
15. Core Concepts 1. Leader Election Leader Follower Follower
16. Core Concepts 1. Leader Election Leader Follower Follower Client Client Client Client ...
17. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ...
18. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 1
19. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 1 1
20. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1
21. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1 Ack(1) Ack(1)
22. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1
23. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1 Commit(1) Commit(1)
24. Core Concepts 1. Leader Election 2. Log replication Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1 2 2 2
25. Log Replication - The leader builds a sequence of input messages (the log) - The sequence is replicated to each node Guaranteed even if - One of the servers fails - Network fails This architecture uses Command Sourcing
26. Core Concepts 1. Leader Election 2. Log replication 3. Deterministic Execution Leader Follower Follower Client Client Client Client ... Log 1 Log 1 Log 1
27. Core Concepts 1. Leader Election 2. Log replication 3. Deterministic Execution Leader Follower Follower Log 1 Log 1 Log 1 Consensus Module State Machine System Logic Deterministic Stateful Leader Election Log Replication Journaling
28. Deterministic Execution - Time - Never read system time, “inject” time - Random numbers - Use deterministic seed
29. Example: Distributed Hashmap
30. Leader (process) State Machine PutRequest GetRequest PutResponse GetResponse Consensus Module Message Handler Key Value Store Protocol - PutRequest(key, value) - PutResponse(success) - GetRequest(key) - GetResponse(value) 1 2 3
31. public class KeyValueStore { private Map<Object, Object> data = new HashMap<>(); void put(Object key, Object value) { data.put(key, value); } Object get(Object key) { return data.get(key); } } public class MessageHandler { private final KeyValueStore store; public Object accept(Object message) { if (message instanceof PutRequest) { PutRequest put = (PutRequest) message; store.put(put.getKey(), put.getValue()); return "OK"; } else if (message instanceof GetRequest) { GetRequest get = (GetRequest) message; return store.get(get.getKey()); } else { return "ERROR"; } } } Unpack messages Clean Model
32. What if we used Raft directly in our Service Layer?
33. Leader Follower Follower Client Client Client Client ... Clients Service Layer Service Logic Service Logic Service Logic
34. We’ve tried, here are the benefits...
35. Separation of Concerns Infrastructure - High Availability - Concurrency & Asynchrony - Durability (Raft log) - Messaging - Cluster client <> cluster - Cluster node <> cluster node You don’t have to build it yourself! - http://atomix.io/copycat/ and others State Machine - System logic & state - Single threaded code - Stateful - No Framework dependency You can focus on the domain problem and build a clean model
36. State and Logic in one place - Consistent state is guaranteed - The cluster can “own” all the required data - Request processed in-memory - Less code, synchronous, much more simple - Side-effect - Data and logic in one place yield great performance Service Client Stateless Durable Transactional LOGIC DATA Client LOGIC + DATA
37. No Database? The architecture is Command Sourced - Replay the log to restore state - Snapshot to trim the log Benefit: no database access code, ORM, etc...
38. Debuggability System crash - Download the Raft log - Load the same version of the code in an IDE - Replay - Same Exception, with debugger attached - Simple to reproduce bugs Application logs - Sequence of logs statements, no interleaving (single threaded code) - Simple to read
39. Resilience “out of the box” - The Raft infrastructure does all the heavy lifting - Resilience is not a concern of the service logic - Simple
40. Great environment to focus on the domain - Model your domain - POJOs - Data structures - Algorithms - No dependency on infrastructure
41. Event-driven 1. Client 1 sends a message 2. State-machine “reacts” to incoming message 3. State-machine “emits” messages to client 2 & 3 Leader Follower Follower Client 1 Client 2 Client 3 1 2 3 3 HTTP TCP UDP
42. Wrapping up - Widely applicable architecture - Deserves more attention! - Learn more - LMAX Architecture - Raft paper and website - White paper http://weareadaptive.com/blog
43. http://weareadaptive.com/careers/ careers@weareadaptive.com Thank you! Any questions?
44. Appendix
45. State too large for the cluster? Client Service 1 2 3 4 - Memory is cheap - 10s GIG RAM - Large heap => GC pauses => Azule - Or...
46. CQRS vs Clustered service (1/2) Client Write Model Event store Read Model Commands Events Events CQRS Event Sourcing Cluster service Client Commands Queries Queries Command Sourcing
47. CQRS vs Clustered service (2/2) Client Commands Queries Client Client Read Models Write Models Client Write Model Event store Read Model Commands Events Events CQRS Queries CQRS with Clustered service Updates
48. State Machine Message Emitters State Machine Hexagonal Architecture Consensus Module Message Handlers Application Threading model? Async? Logic Logic Application Messaging Q Q Single threaded Synchronous
49. Performance? - Copycat - ~ 5-10K msg/sec - Commodity hardware (AWS medium instances) - Specialised Raft infrastructure - > 100K msg/sec - 2 digits micro-second latencies - Kernel bypass - Fast SSDs - No allocation for managed runtimes - Etc.. - State-Machine - 10K msg/sec - Don’t do anything stupid - 100K-1M msg/sec: - Algo complexity: O(1), O(log(n)) - No/minimised allocation - Profiling - Perf testing - Never - I/O on state machine thread - CPU heavy work (crypto, etc) - Offload to separate thread / service
50. How to scale? Client Gateway 1 Gateway 2 Gateway 3 Gateway N ... Client Command Command Updates