Upgrade to Pro — share decks privately, control downloads, hide ads and more …

We Hear You Like Papers: Eventual Consistency

We Hear You Like Papers: Eventual Consistency

Given at Women Who Code Sydney IT meetup

Resources: https://github.com/CaitieM20/Talks/tree/master/SoWeHearYouLikePapers

Caitie McCaffrey

December 07, 2016
Tweet

More Decks by Caitie McCaffrey

Other Decks in Technology

Transcript

  1. Papers
    We hear you like

    View full-size slide

  2. @Caitie
    Caitie 

    McCaffrey

    View full-size slide

  3. Distributed
    Systems

    View full-size slide

  4. Service Service
    Service
    We Are All
    Building
    Distributed
    Systems

    View full-size slide

  5. Twitter
    Services

    View full-size slide

  6. academic
    Papers

    View full-size slide

  7. Eventual
    Consistency

    View full-size slide

  8. 1983 1995
    Thinking Consistency
    Detection of
    Mutual
    Inconsistency
    in Distributed
    Systems
    Managing
    Update Conflicts
    in Bayou, a
    Weakly
    Connected
    Replicated
    Storage System
    Brewer's
    conjecture &
    the feasibility of
    consistent,
    available,
    partition-tolerant
    web services
    2002

    View full-size slide

  9. 2015
    2011
    Conflict-free
    replicated Data
    Types
    Feral Concurrency
    Control: An Empirical
    Investigation of
    Modern Application
    Integrity
    Thinking Consistency

    View full-size slide

  10. Service
    Service
    Service
    Applications Before

    View full-size slide

  11. Service
    Service
    Service
    Applications Before

    View full-size slide

  12. ApplicationsNow
    Service
    Service
    Service

    View full-size slide

  13. High
    availability

    View full-size slide

  14. High
    availability
    “In some environments
    it is desirable or
    necessary to permit
    users to continue
    modifying resources
    such as files when the
    network is partitioned”

    View full-size slide

  15. Origin
    Points
    &
    Version
    Vectors

    View full-size slide

  16. Version Vectors
    “A Version Vector for a file f is a sequence of n
    pairs, where n is the number of sites at which f
    is stored … the ith vector entry counts the
    number Vi of updates to f made at site Si”

    View full-size slide

  17. Compatible Version Vectors


    View full-size slide



  18. Incompatible Version Vectors

    View full-size slide

  19. ABCD
    Partition Graph
    C:0, D:0>

    View full-size slide

  20. ABCD
    AB CD
    Partition Graph
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>

    View full-size slide

  21. ABCD
    AB CD
    Partition Graph
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>

    View full-size slide

  22. ABCD
    AB CD
    D
    BC
    A
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>

    View full-size slide

  23. ABCD
    AB CD
    D
    BC
    A
    Partition Graph
    T2
    C:0, D:0>

    C:0, D:0>

    C:0, D:0>
    No Conflict!

    View full-size slide

  24. ABCD
    AB CD
    D
    BC
    A
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>

    View full-size slide

  25. ABCD
    AB CD
    D
    BC
    A
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:1, D:0>

    View full-size slide

  26. ABCD
    AB CD
    D
    BC
    A
    BCD
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:1, D:0>

    View full-size slide

  27. ABCD
    AB CD
    D
    BC
    A
    BCD
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>

    C:0, D:0>
    C:0, D:0>

    No Conflict!

    View full-size slide

  28. ABCD
    AB CD
    D
    BC
    A
    BCD
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:1, D:0>
    C:1, D:0>

    View full-size slide

  29. ABCD
    AB CD
    D
    BC
    A
    BCD
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:1, D:0>
    C:1, D:0>
    ABCD

    View full-size slide

  30. ABCD
    AB CD
    D
    BC
    A
    BCD
    ABCD
    Partition Graph
    T2
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>
    C:0, D:0>

    C:1, D:0>

    Conflict!

    View full-size slide

  31. Conflict Resolution
    “A conflict detection
    mechanism, while valuable,
    has increased effect if
    there is also a method for
    reconciling conflicts
    automatically”

    View full-size slide

  32. Bayou Summary
    System designed for weak connectivity
    Eventual consistency via application-
    defined dependency checks and
    developer defined merge procedures
    Epidemic algorithms to replicate state

    View full-size slide

  33. “Applications must be
    aware of and integrally
    involved in conflict
    detection and resolution”
    Terry et. al

    View full-size slide

  34. Bayou Take aways & thoughts
    “Humans would
    rather deal with
    the occasional
    unresolvable
    conflict than
    incur the
    adverse impact
    on availability”

    View full-size slide

  35. CAP Theorem
    PARTITION TOLERANCE
    CONSISTENCY AVAILABILITY

    View full-size slide

  36. Consistency
    Models
    Linearizable
    Sequential
    Causal
    Pipelined random
    access memory
    Read your write Monotonic read Monotonic write
    Write from read
    CP Consistency
    AP Consistency

    View full-size slide

  37. CRDTs Summary
    Mathematical properties & epidemic
    algorithms / gossip protocols
    Strong Eventual Consistency - apply
    updates immediately, no conflicts, or
    rollbacks via

    View full-size slide

  38. CRDTs
    * Stolen from Chris Meiklejohn
    in practice

    View full-size slide

  39. Applying rollbacks is
    hard
    Restrict operation
    space to get provably
    convergent systems
    Active area of research
    Resolving Conflicts

    View full-size slide

  40. Applying rollbacks is
    hard
    Restrict operation
    space to get provably
    convergent systems
    Active area of research
    Resolving Conflicts

    View full-size slide

  41. Feral mechanisms for keeping DB integrity
    Application-level mechanisms
    Analyzed 67 open source Ruby on
    Rails Applications
    Unsafe > 13% of the time 

    (uniqueness & foreign key constraint violations)

    View full-size slide

  42. Concurrency control is hard!
    Availability is important to application
    developers
    Home-rolling your own concurrency
    control or consensus algorithm is very
    hard and difficult to get correct!

    View full-size slide

  43. Eventual Consistency
    We want highly available systems so we must use
    weaker forms of consistency (remember CAP)
    Application semantics helps us make better
    tradeoffs
    Do not recreate the wheel, leverage existing
    research allows us to not repeat past mistakes
    Forced into a feral world but this may change soon!
    Tl;DR

    View full-size slide

  44. @Caitie
    Thank you!

    View full-size slide

  45. @Caitie
    Thank you!
    Follow
    your
    dreams!

    View full-size slide