The Path Towards Simplifying Consistency in Distributed Systems

The Path Towards Simplifying Consistency in Distributed Systems

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

April 24, 2017
Tweet

Transcript

  1. The Path Towards Simplifying Consistency in Distributed Systems

  2. Caitie McCaffrey Distributed Systems Engineer caitiem.com @caitie

  3. Distributed Systems Engineer Feral Concurrency Control Programmer Caitie McCaffrey

  4. 2015

  5. 2015 “Application-level Mechanisms for maintaining database integrity”

  6. Halo 4 Statistics Service Halo 4 Statistics Service Record &

    Aggregate Statistics From Every Multiplayer Halo Game a User Plays
  7. Halo 4 Statistics Service

  8. Observability at Twitter Make Dashboards & Alerts Always Available

  9. Cmd Line Tool Viz / Dashboad Alerting Svc Cuckoo-Read Cuckoo-Write

    Indexing Svc Relay Svc Twitter Front End Twitter Svc Twitter Statsite Twitter Svc Twitter Svc Scribe Collection Agent HDFS Manhattan Database Public Cloud Observability at Twitter
  10. Notify Users When We Receive Abuse Reports

  11. Notify Users If An Action is Taken

  12. Report Events Stream Modification Events Stream Enrollment Events Stream Listener

    Service Daemons Service Executor Service Notification Events Stream Notification Service Manhattan Macaw
  13. Feral Concurrency Control

  14. Service Service Service Monoliths

  15. The Rise of NoSQL

  16. None
  17. Microservices

  18. State of Affairs

  19. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write

    Monotonic Read Monotonic Write Write From Read Consistency Models
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. Can We Do Better?

  27. Finance & Accounting

  28. VisiCalc 1979

  29. 1983 Lotus 123

  30. None
  31. 1987 Excel

  32. None
  33. & Standardization Computer Aided Computation

  34. Standardization for Consistency in Distributed Systems

  35. 1988

  36. 1988 “We believe the form of communication that is needed

    is remote procedure call with at- most-once semantics”
  37. 1988 “An adequate language must provide a modular, reasonably automatic

    method for achieving consistency ”
  38. Branch B Account 789: $250 Branch A Account 123 :

    $100 Account 456: $50 SubAction: Deposit (123, 50) Transfer Action SubAction: Withdraw(789, 50) Transfer Deposit Withdraw enter topaction coenter action branchA.Deposit(123, 50) action branchB.Withdraw(789, 50) end end
  39. What about Spanner?

  40. 2015

  41. 2015 “The biggest barrier…is that consistency mechanisms must integrate across

    many stateful services”
  42. Microservices

  43. CRDTs * Stolen from Chris Meiklejohn in practice

  44. CRDTs * Stolen from Chris Meiklejohn in practice

  45. There Is No One Size Fits All Consistency Model

  46. Mixed Consistency Data Stores Manhattan

  47. None
  48. Computer Aided Computation for Consistency in Distributed Systems

  49. 2016

  50. Bank Application Bank Account must be > 0 Deposit Money

    Withdrawal Money
  51. ‘Cause I’m Strong Enough: Reasoning About Consistency Choices in Distributed

    Systems
  52. Define Invariants & Actions Deposit Money Withdrawal Money Determine Consistency

    Required Bank Account > 0 Eventual Consistency Strong Consistency Code Generation?
  53. & Standardization Computer Aided Computation

  54. Thank you @caitie