Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Path Towards Simplifying Consistency in Distributed Systems

The Path Towards Simplifying Consistency in Distributed Systems

Caitie McCaffrey

April 24, 2017
Tweet

More Decks by Caitie McCaffrey

Other Decks in Technology

Transcript

  1. The Path Towards
    Simplifying
    Consistency in
    Distributed Systems

    View full-size slide

  2. Caitie McCaffrey
    Distributed Systems Engineer
    caitiem.com
    @caitie

    View full-size slide

  3. Distributed Systems Engineer
    Feral Concurrency
    Control Programmer
    Caitie McCaffrey

    View full-size slide

  4. 2015
    “Application-level
    Mechanisms for maintaining
    database integrity”

    View full-size slide

  5. Halo 4 Statistics Service
    Halo 4 Statistics Service
    Record & Aggregate Statistics
    From Every Multiplayer Halo
    Game a User Plays

    View full-size slide

  6. Halo 4 Statistics Service

    View full-size slide

  7. Observability at Twitter
    Make Dashboards & Alerts
    Always Available

    View full-size slide

  8. Cmd Line Tool Viz / Dashboad Alerting Svc
    Cuckoo-Read
    Cuckoo-Write
    Indexing Svc
    Relay Svc
    Twitter Front End
    Twitter
    Svc
    Twitter
    Statsite
    Twitter
    Svc
    Twitter
    Svc
    Scribe
    Collection
    Agent
    HDFS
    Manhattan Database
    Public Cloud
    Observability at Twitter

    View full-size slide

  9. Notify Users When
    We Receive Abuse
    Reports

    View full-size slide

  10. Notify Users If An
    Action is Taken

    View full-size slide

  11. Report
    Events Stream
    Modification
    Events Stream
    Enrollment
    Events Stream
    Listener Service
    Daemons Service
    Executor Service
    Notification Events
    Stream
    Notification
    Service
    Manhattan
    Macaw

    View full-size slide

  12. Feral Concurrency
    Control

    View full-size slide

  13. Service Service
    Service
    Monoliths

    View full-size slide

  14. The Rise of NoSQL

    View full-size slide

  15. Microservices

    View full-size slide

  16. State of Affairs

    View full-size slide

  17. Linearizable
    Sequential
    Causal
    Pipelined Random
    Access Memory
    Read Your
    Write
    Monotonic
    Read
    Monotonic
    Write
    Write From
    Read
    Consistency
    Models

    View full-size slide

  18. Can We Do
    Better?

    View full-size slide

  19. Finance & Accounting

    View full-size slide

  20. VisiCalc
    1979

    View full-size slide

  21. 1983 Lotus 123

    View full-size slide

  22. &
    Standardization
    Computer Aided
    Computation

    View full-size slide

  23. Standardization
    for Consistency in Distributed Systems

    View full-size slide

  24. 1988
    “We believe the form of
    communication that is needed is
    remote procedure call with at-
    most-once semantics”

    View full-size slide

  25. 1988
    “An adequate language must
    provide a modular, reasonably
    automatic method for achieving
    consistency ”

    View full-size slide

  26. Branch B
    Account 789: $250
    Branch A
    Account 123 : $100
    Account 456: $50
    SubAction: Deposit (123, 50)
    Transfer Action
    SubAction: Withdraw(789, 50)
    Transfer
    Deposit Withdraw
    enter topaction
    coenter
    action
    branchA.Deposit(123, 50)
    action
    branchB.Withdraw(789, 50)
    end
    end

    View full-size slide

  27. What about
    Spanner?

    View full-size slide

  28. 2015
    “The biggest barrier…is that
    consistency mechanisms must
    integrate across many
    stateful services”

    View full-size slide

  29. Microservices

    View full-size slide

  30. CRDTs
    * Stolen from Chris Meiklejohn
    in practice

    View full-size slide

  31. CRDTs
    * Stolen from Chris Meiklejohn
    in practice

    View full-size slide

  32. There Is No One
    Size Fits All
    Consistency Model

    View full-size slide

  33. Mixed Consistency Data Stores
    Manhattan

    View full-size slide

  34. Computer Aided
    Computation
    for Consistency in Distributed Systems

    View full-size slide

  35. Bank Application
    Bank Account must be > 0
    Deposit Money
    Withdrawal Money

    View full-size slide

  36. ‘Cause I’m Strong Enough: Reasoning About Consistency Choices in Distributed Systems

    View full-size slide

  37. Define Invariants
    & Actions
    Deposit Money
    Withdrawal Money
    Determine
    Consistency
    Required
    Bank Account > 0
    Eventual Consistency
    Strong Consistency
    Code
    Generation?

    View full-size slide

  38. &
    Standardization
    Computer Aided
    Computation

    View full-size slide

  39. Thank you
    @caitie

    View full-size slide