We Hear You Like Papers: Eventual Consistency

Slide 1

Slide 1 text

Papers We hear you like

Slide 2

Slide 2 text

@Caitie Caitie   McCaffrey

Slide 3

Slide 3 text

Distributed Systems

Slide 4

Slide 4 text

Service Service Service We Are All Building Distributed Systems

Slide 5

Slide 5 text

Twitter Services

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

academic Papers

Slide 8

Slide 8 text

Eventual Consistency

Slide 9

Slide 9 text

1983 1995 Thinking Consistency Detection of Mutual Inconsistency in Distributed Systems Managing Update Conflicts in Bayou, a Weakly Connected Replicated Storage System Brewer's conjecture & the feasibility of consistent, available, partition-tolerant web services 2002

Slide 10

Slide 10 text

2015 2011 Conflict-free replicated Data Types Feral Concurrency Control: An Empirical Investigation of Modern Application Integrity Thinking Consistency

Slide 11

Slide 11 text

Service Service Service Applications Before

Slide 12

Slide 12 text

Service Service Service Applications Before

Slide 13

Slide 13 text

ApplicationsNow Service Service Service

Slide 14

Slide 14 text

1983

Slide 15

Slide 15 text

High availability

Slide 16

Slide 16 text

High availability “In some environments it is desirable or necessary to permit users to continue modifying resources such as ﬁles when the network is partitioned”

Slide 17

Slide 17 text

Origin Points & Version Vectors

Slide 18

Slide 18 text

Version Vectors “A Version Vector for a ﬁle f is a sequence of n pairs, where n is the number of sites at which f is stored … the ith vector entry counts the number Vi of updates to f made at site Si”

Slide 19

Slide 19 text

Compatible Version Vectors

Slide 20

Slide 20 text

Incompatible Version Vectors

Slide 21

Slide 21 text

ABCD Partition Graph

Slide 22

Slide 22 text

ABCD AB CD Partition Graph

Slide 23

Slide 23 text

ABCD AB CD Partition Graph

Slide 24

Slide 24 text

ABCD AB CD D BC A Partition Graph T2

Slide 25

Slide 25 text

ABCD AB CD D BC A Partition Graph T2 No Conflict!

Slide 26

Slide 26 text

ABCD AB CD D BC A Partition Graph T2

Slide 27

Slide 27 text

ABCD AB CD D BC A Partition Graph T2

Slide 28

Slide 28 text

ABCD AB CD D BC A BCD Partition Graph T2

Slide 29

Slide 29 text

ABCD AB CD D BC A BCD Partition Graph T2 No Conflict!

Slide 30

Slide 30 text

ABCD AB CD D BC A BCD Partition Graph T2

Slide 31

Slide 31 text

ABCD AB CD D BC A BCD Partition Graph T2 ABCD

Slide 32

Slide 32 text

ABCD AB CD D BC A BCD ABCD Partition Graph T2 Conflict!

Slide 33

Slide 33 text

Conflict Resolution “A conﬂict detection mechanism, while valuable, has increased effect if there is also a method for reconciling conﬂicts automatically”

Slide 34

Slide 34 text

1995

Slide 35

Slide 35 text

Bayou Summary System designed for weak connectivity Eventual consistency via application- defined dependency checks and developer defined merge procedures Epidemic algorithms to replicate state

Slide 36

Slide 36 text

“Applications must be aware of and integrally involved in conflict detection and resolution” Terry et. al

Slide 37

Slide 37 text

Bayou Take aways & thoughts “Humans would rather deal with the occasional unresolvable conflict than incur the adverse impact on availability”

Slide 38

Slide 38 text

2002

Slide 39

Slide 39 text

CAP Theorem PARTITION TOLERANCE CONSISTENCY AVAILABILITY

Slide 40

Slide 40 text

Consistency Models Linearizable Sequential Causal Pipelined random access memory Read your write Monotonic read Monotonic write Write from read CP Consistency AP Consistency

Slide 41

Slide 41 text

2011

Slide 42

Slide 42 text

CRDTs Summary Mathematical properties & epidemic algorithms / gossip protocols Strong Eventual Consistency - apply updates immediately, no conflicts, or rollbacks via

Slide 43

Slide 43 text

CRDTs * Stolen from Chris Meiklejohn in practice

Slide 44

Slide 44 text

Applying rollbacks is hard Restrict operation space to get provably convergent systems Active area of research Resolving Conflicts

Slide 45

Slide 45 text

Applying rollbacks is hard Restrict operation space to get provably convergent systems Active area of research Resolving Conflicts

Slide 46

Slide 46 text

2015

Slide 47

Slide 47 text

Feral mechanisms for keeping DB integrity Application-level mechanisms Analyzed 67 open source Ruby on Rails Applications Unsafe > 13% of the time   (uniqueness & foreign key constraint violations)

Slide 48

Slide 48 text

Concurrency control is hard! Availability is important to application developers Home-rolling your own concurrency control or consensus algorithm is very hard and difficult to get correct!

Slide 49

Slide 49 text

Eventual Consistency We want highly available systems so we must use weaker forms of consistency (remember CAP) Application semantics helps us make better tradeoffs Do not recreate the wheel, leverage existing research allows us to not repeat past mistakes Forced into a feral world but this may change soon! Tl;DR