The Path Towards Simplifying Consistency in Distributed Systems

Transcript

1. The Path Towards Simplifying Consistency in Distributed Systems
2. Caitie McCaffrey Distributed Systems Engineer caitiem.com @caitie
3. Distributed Systems Engineer Feral Concurrency Control Programmer Caitie McCaffrey
5. 2015 “Application-level Mechanisms for maintaining database integrity”
6. Halo 4 Statistics Service Halo 4 Statistics Service Record & Aggregate Statistics From Every Multiplayer Halo Game a User Plays
7. Halo 4 Statistics Service
8. Observability at Twitter Make Dashboards & Alerts Always Available
9. Cmd Line Tool Viz / Dashboad Alerting Svc Cuckoo-Read Cuckoo-Write Indexing Svc Relay Svc Twitter Front End Twitter Svc Twitter Statsite Twitter Svc Twitter Svc Scribe Collection Agent HDFS Manhattan Database Public Cloud Observability at Twitter
10. Notify Users When We Receive Abuse Reports
11. Notify Users If An Action is Taken
12. Report Events Stream Modification Events Stream Enrollment Events Stream Listener Service Daemons Service Executor Service Notification Events Stream Notification Service Manhattan Macaw
13. Feral Concurrency Control
14. Service Service Service Monoliths
15. The Rise of NoSQL
17. Microservices
18. State of Affairs
19. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write Monotonic Read Monotonic Write Write From Read Consistency Models
26. Can We Do Better?
27. Finance & Accounting
28. VisiCalc 1979
29. 1983 Lotus 123
31. 1987 Excel
33. & Standardization Computer Aided Computation
34. Standardization for Consistency in Distributed Systems
36. 1988 “We believe the form of communication that is needed is remote procedure call with at- most-once semantics”
37. 1988 “An adequate language must provide a modular, reasonably automatic method for achieving consistency ”
38. Branch B Account 789: $250 Branch A Account 123 : $100 Account 456: $50 SubAction: Deposit (123, 50) Transfer Action SubAction: Withdraw(789, 50) Transfer Deposit Withdraw enter topaction coenter action branchA.Deposit(123, 50) action branchB.Withdraw(789, 50) end end
39. What about Spanner?
41. 2015 “The biggest barrier…is that consistency mechanisms must integrate across many stateful services”
42. Microservices
43. CRDTs * Stolen from Chris Meiklejohn in practice
44. CRDTs * Stolen from Chris Meiklejohn in practice
45. There Is No One Size Fits All Consistency Model
46. Mixed Consistency Data Stores Manhattan
48. Computer Aided Computation for Consistency in Distributed Systems
50. Bank Application Bank Account must be > 0 Deposit Money Withdrawal Money
51. ‘Cause I’m Strong Enough: Reasoning About Consistency Choices in Distributed Systems
52. Define Invariants & Actions Deposit Money Withdrawal Money Determine Consistency Required Bank Account > 0 Eventual Consistency Strong Consistency Code Generation?
53. & Standardization Computer Aided Computation
54. Thank you @caitie