We Hear You Like Papers: Eventual Consistency

We Hear You Like Papers: Eventual Consistency

Given at Women Who Code Sydney IT meetup

Resources: https://github.com/CaitieM20/Talks/tree/master/SoWeHearYouLikePapers

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

December 07, 2016
Tweet

Transcript

  1. Papers We hear you like

  2. @Caitie Caitie 
 McCaffrey

  3. Distributed Systems

  4. Service Service Service We Are All Building Distributed Systems

  5. Twitter Services

  6. None
  7. academic Papers

  8. Eventual Consistency

  9. 1983 1995 Thinking Consistency Detection of Mutual Inconsistency in Distributed

    Systems Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System Brewer's conjecture & the feasibility of consistent, available, partition-tolerant web services 2002
  10. 2015 2011 Conflict-free replicated Data Types Feral Concurrency Control: An

    Empirical Investigation of Modern Application Integrity Thinking Consistency
  11. Service Service Service Applications Before

  12. Service Service Service Applications Before

  13. ApplicationsNow Service Service Service

  14. 1983

  15. High availability

  16. High availability “In some environments it is desirable or necessary

    to permit users to continue modifying resources such as files when the network is partitioned”
  17. Origin Points & Version Vectors

  18. Version Vectors “A Version Vector for a file f is

    a sequence of n pairs, where n is the number of sites at which f is stored … the ith vector entry counts the number Vi of updates to f made at site Si” <A:9, B:7, C:22, D:3>
  19. Compatible Version Vectors <A:1, B:2, C:4, D:3> <A:0, B:2, C:2,

    D:3>
  20. <A:1, B:2, C:4, D:3> <A:1, B:2, C:3, D:4> Incompatible Version

    Vectors
  21. ABCD Partition Graph <A:0, B:0, C:0, D:0>

  22. ABCD AB CD Partition Graph <A:0, B:0, C:0, D:0> <A:0,

    B:0, C:0, D:0> <A:0, B:0, C:0, D:0>
  23. ABCD AB CD Partition Graph <A:0, B:0, C:0, D:0> <A:0,

    B:0, C:0, D:0> <A:1, B:0, C:0, D:0>
  24. ABCD AB CD D BC A Partition Graph T2 <A:0,

    B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:1, B:0, C:0, D:0>
  25. ABCD AB CD D BC A Partition Graph T2 <A:0,

    B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> No Conflict!
  26. ABCD AB CD D BC A Partition Graph T2 <A:0,

    B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:1, B:0, C:0, D:0>
  27. ABCD AB CD D BC A Partition Graph T2 <A:0,

    B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0>
  28. ABCD AB CD D BC A BCD Partition Graph T2

    <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0>
  29. ABCD AB CD D BC A BCD Partition Graph T2

    <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0> No Conflict!
  30. ABCD AB CD D BC A BCD Partition Graph T2

    <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0> <A:1, B:0, C:1, D:0>
  31. ABCD AB CD D BC A BCD Partition Graph T2

    <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0> <A:1, B:0, C:1, D:0> ABCD
  32. ABCD AB CD D BC A BCD ABCD Partition Graph

    T2 <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:0, B:0, C:0, D:0> <A:1, B:0, C:0, D:0> <A:2, B:0, C:0, D:0> <A:1, B:0, C:1, D:0> <A:1, B:0, C:1, D:0> Conflict!
  33. Conflict Resolution “A conflict detection mechanism, while valuable, has increased

    effect if there is also a method for reconciling conflicts automatically”
  34. 1995

  35. Bayou Summary System designed for weak connectivity Eventual consistency via

    application- defined dependency checks and developer defined merge procedures Epidemic algorithms to replicate state
  36. “Applications must be aware of and integrally involved in conflict

    detection and resolution” Terry et. al
  37. Bayou Take aways & thoughts “Humans would rather deal with

    the occasional unresolvable conflict than incur the adverse impact on availability”
  38. 2002

  39. CAP Theorem PARTITION TOLERANCE CONSISTENCY AVAILABILITY

  40. Consistency Models Linearizable Sequential Causal Pipelined random access memory Read

    your write Monotonic read Monotonic write Write from read CP Consistency AP Consistency
  41. 2011

  42. CRDTs Summary Mathematical properties & epidemic algorithms / gossip protocols

    Strong Eventual Consistency - apply updates immediately, no conflicts, or rollbacks via
  43. CRDTs * Stolen from Chris Meiklejohn in practice

  44. Applying rollbacks is hard Restrict operation space to get provably

    convergent systems Active area of research Resolving Conflicts
  45. Applying rollbacks is hard Restrict operation space to get provably

    convergent systems Active area of research Resolving Conflicts
  46. 2015

  47. Feral mechanisms for keeping DB integrity Application-level mechanisms Analyzed 67

    open source Ruby on Rails Applications Unsafe > 13% of the time 
 (uniqueness & foreign key constraint violations)
  48. Concurrency control is hard! Availability is important to application developers

    Home-rolling your own concurrency control or consensus algorithm is very hard and difficult to get correct!
  49. Eventual Consistency We want highly available systems so we must

    use weaker forms of consistency (remember CAP) Application semantics helps us make better tradeoffs Do not recreate the wheel, leverage existing research allows us to not repeat past mistakes Forced into a feral world but this may change soon! Tl;DR
  50. @Caitie Thank you!

  51. @Caitie Thank you! Follow your dreams!